From Texts to Scores: Tracing the Emergence of Essay Quality Representations in Large Language Models

The Black Box of Essay Scoring: Peeking Inside LLMs

A new preprint from arXiv (2606.20152v1) takes a significant step toward demystifying how Large Language Models assign scores to student essays. Rather than treating LLM-based Automated Essay Scoring (AES) as a simple input-output process, the researchers systematically analyze the hidden representations within these models to trace where and how essay quality emerges during processing.

This is not just another paper claiming LLMs can grade essays—that capability is already well-established. The novelty lies in the mechanistic analysis: by probing internal model states, the authors map the progression from raw text to numeric score across different layers of the neural network. They identify specific layers where representations shift from general language understanding to task-specific quality judgments, revealing that scoring is not a single "aha moment" but a gradual, distributed process.

Why This Matters Beyond Academia

For years, AES has been a black-box problem. Educators and administrators deploying LLM-based scoring systems have had to trust outputs without understanding why a particular score was assigned. This research begins to crack that box open, with three critical implications:

First, interpretability becomes actionable. If we know which layers encode quality judgments, we can design better prompts, fine-tuning strategies, or even layer-specific interventions to correct biases. A model that over-penalizes certain writing styles could be adjusted at the representation level rather than through brute-force retraining.

Second, reliability improves through transparency. Current AES systems often fail on edge cases—creative essays, non-native writing, or genre-bending submissions. Understanding the internal scoring mechanisms allows practitioners to identify when a model is likely to fail, enabling human-in-the-loop systems that flag uncertain scores for review.

Third, the approach generalizes. The methodology used here—tracing representation emergence across layers—can be applied to other LLM tasks beyond essay scoring, from medical diagnosis to legal document analysis. This paper offers a template for understanding how LLMs transform inputs into structured outputs.

Implications for AI Practitioners

For teams building or deploying LLM-based AES, this research provides a diagnostic toolkit. Instead of treating scoring errors as mysterious glitches, practitioners can now ask: At which layer did the representation diverge from expected quality signals? This shifts debugging from output-level analysis to model-internal inspection.

Additionally, the findings suggest that fine-tuning for AES should not treat all model layers equally. Targeted fine-tuning of the specific layers identified as critical for quality representation could yield better results with less data and compute than full-model fine-tuning.

Key Takeaways

• LLMs do not assign essay scores in a single step; quality representations emerge gradually across specific internal layers, offering new opportunities for interpretability and debugging.
• Understanding where and how scoring decisions form allows practitioners to design more transparent, fair, and reliable AES systems with targeted interventions.
• The layer-wise analysis methodology can be adapted to other LLM applications, providing a general framework for understanding model reasoning beyond just essay scoring.
• For production AES systems, this research supports moving from black-box trust to evidence-based confidence, enabling better human-AI collaboration in educational assessment.