Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA

The Judge-Generation Asymmetry: Why LLMs May Not Know What They Don't Know

A new preprint from arXiv (2606.28050v1) tackles a foundational assumption underpinning many modern AI evaluation pipelines: that LLMs are better at judging answers than generating them. The researchers test this "task asymmetry" hypothesis in a controlled in-context question-answering setup, where each model must evaluate its own output against a given context passage. The findings challenge a core premise of self-evaluation and LLM-as-a-Judge workflows.

What the Research Investigates

The study isolates the judge-versus-generator question by removing external variables like training data leakage or parametric knowledge. In their setup, the context passage is the sole information source, meaning the model cannot rely on memorized facts. Each model judges the answer it previously generated, creating a direct comparison between generation accuracy and evaluation accuracy. This controlled design allows the researchers to test whether evaluation truly is "easier" than generation—a claim that self-evaluation pipelines implicitly depend on.

Why This Matters

The implications cut to the heart of how AI systems are currently deployed and validated:

• Self-evaluation pipelines may be fundamentally flawed. If LLMs cannot reliably judge their own outputs in simple QA tasks, then more complex self-evaluation loops—used in RAG systems, agentic workflows, and iterative refinement—may compound errors rather than correct them.

• The assumption of asymmetry is not guaranteed. The paper suggests that the judge and generator roles may not be as distinct as practitioners assume. A model that generates incorrect answers may also misjudge those same answers, creating a blind spot where errors go undetected.

• Mechanistic interpretability insights. By analyzing how models evaluate versus generate, the research opens the door to understanding which internal representations are shared between these tasks—and which are not. This could inform future architectures that separate evaluation and generation more cleanly.

Implications for AI Practitioners

For teams building production systems, this research signals a need for caution:

• Do not rely solely on self-evaluation. Use external judges, human-in-the-loop validation, or ensemble methods where possible.
• Test for judge-generation correlation. Before deploying a self-evaluation pipeline, measure whether your model’s evaluation accuracy correlates with its generation accuracy. A high correlation indicates that the model may not catch its own mistakes.
• Consider task-specific judge models. The findings support the emerging practice of training separate, smaller models specifically for evaluation tasks, rather than using the same model for both generation and judgment.

The paper also raises a broader question: if evaluation is not inherently easier than generation, then what does this mean for alignment techniques that rely on self-critique? The answer may be that we need more robust, external validation mechanisms—and that the "LLM-as-a-Judge" paradigm has limits that practitioners must acknowledge.

Key Takeaways

• The assumption that LLMs are better at judging than generating is not universally true and may fail in controlled in-context settings.
• Self-evaluation pipelines risk undetected errors if the same model judges its own outputs without external validation.
• Practitioners should test for judge-generation correlation in their specific use cases and avoid over-reliance on self-evaluation.
• The research supports the use of separate, specialized judge models and external validation mechanisms in production AI systems.