Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs

The Validity Gap: When LLMs Get the Right Answer for the Wrong Reasons

A new arXiv paper (2606.28574v1) tackles a subtle but critical problem in social science research: LLMs can achieve high agreement with human coders on theoretical constructs while being fundamentally theory-naive. The authors argue that inter-rater reliability—the standard metric for coding quality—does not guarantee construct validity. An LLM might correctly label "anxiety" in text not because it understands the psychological construct, but because it has learned surface-level statistical patterns that correlate with anxiety language.

This distinction matters because reliability and validity are conceptually different. Reliability means consistent measurement; validity means measuring what you intend to measure. A broken thermometer that always reads 30°C is reliable but invalid. Similarly, an LLM that matches human coders on a training set may fail when the construct appears in novel contexts, nuanced language, or culturally specific forms.

Why This Matters for AI-Assisted Research

The implications extend far beyond academic social science. Organizations increasingly deploy LLMs to code customer sentiment, employee engagement, compliance risks, and market trends. If these models are merely pattern-matching rather than understanding underlying constructs, they will produce systematically biased results when applied to new populations, languages, or time periods.

Consider a model trained to detect "toxic behavior" in online forums. It might achieve high agreement with human raters by flagging profanity—but miss subtle harassment or over-flag sarcasm. The model is reliable (consistent) but invalid (not measuring the intended construct). This validity gap becomes dangerous when decisions—hiring, content moderation, clinical assessments—depend on these measurements.

Implications for AI Practitioners

First, reliability metrics are necessary but insufficient. Practitioners should not treat high Cohen's kappa or accuracy as proof that an LLM understands the construct. Second, construct validation requires theory-driven testing. Researchers must probe whether the model's coding aligns with the theoretical definition—not just with human ratings. This might involve adversarial examples, counterfactual tests, or probing the model's reasoning.

Third, domain expertise remains essential. The paper implicitly argues that LLMs cannot replace theoretical knowledge; they can only approximate human coding patterns. Practitioners should maintain human oversight for construct validation, especially in high-stakes domains.

Finally, the "right answer for wrong reasons" problem is not unique to LLMs—it applies to any automated measurement system. But LLMs' opacity makes it harder to detect. Practitioners should invest in explainability tools that reveal why a model assigned a particular code, not just what code it assigned.

Key Takeaways

• High inter-rater reliability between LLMs and human coders does not guarantee that the LLM is measuring the intended theoretical construct
• LLMs may achieve correct codes through surface-level pattern matching rather than genuine understanding, leading to validity failures in novel contexts
• Practitioners must supplement reliability metrics with theory-driven validation tests, adversarial examples, and human oversight
• Explainability tools that reveal model reasoning are essential for detecting when an LLM gets the right answer for the wrong reasons