JustDiag!: A Diagnostic Justification Engine for Accountable Root Cause Analysis

The paper “JustDiag!: A Diagnostic Justion Engine for Accountable Root Cause Analysis” addresses a critical blind spot in the deployment of large language models (LLMs) for operational tasks: the gap between fluency and accountability. While LLMs can generate plausible-sounding root cause analyses (RCAs) from system logs and incident reports, the research highlights that a convincing narrative is not the same as a verifiable one. In high-stakes environments—such as cloud infrastructure, healthcare IT, or financial trading systems—engineers cannot act on a diagnosis unless they can trace its logic back to specific evidence.

What Happened

The authors propose a framework that forces an LLM to produce not just a final diagnosis, but a structured “justification chain” linking each claim to a concrete piece of evidence (e.g., a log line, a metric spike, or a configuration change). This is achieved through a combination of retrieval-augmented generation (RAG) and a novel scoring mechanism that evaluates the sufficiency and relevance of each evidence step. The system then outputs a diagnostic report where every conclusion is explicitly supported by a traceable source, making the reasoning process auditable by human engineers.

Why It Matters

The core insight here is that trust in AI-generated RCAs is not binary—it is a function of transparency. Current LLM-based diagnostics often suffer from hallucination or “plausible but wrong” reasoning, which is especially dangerous when the cost of a misdiagnosis is extended downtime or incorrect remediation. By enforcing a chain-of-evidence structure, JustDiag! transforms the LLM from a black-box oracle into a collaborative tool that can be challenged and verified. This is a direct response to the growing frustration among DevOps and SRE teams who find that while LLMs can summarize logs quickly, they cannot yet be trusted without manual cross-checking.

For the broader AI industry, this work signals a shift from “model performance” to “process accountability.” It acknowledges that in operational contexts, the path to a conclusion is often more valuable than the conclusion itself. This aligns with emerging regulatory pressures (e.g., the EU AI Act’s requirements for explainability in high-risk systems) and with practical needs in incident management where post-mortems require documented reasoning.

Implications for AI Practitioners

First, practitioners building AI-assisted incident response tools should prioritize “justification fidelity” over raw accuracy metrics. A model that is 95% accurate but cannot explain its reasoning is less useful than one that is 90% accurate but provides a fully auditable chain of evidence. Second, the architecture described—RAG plus a justification scoring layer—is implementable today with existing open-source models and vector databases, meaning teams can adopt this approach without waiting for next-generation LLMs. Third, this work underscores the importance of interface design: the output format (structured evidence links vs. free-text prose) directly impacts how quickly engineers can trust or reject a diagnosis.

Key Takeaways

• JustDiag! introduces a structured “justification chain” that links each diagnostic claim to specific evidence, making LLM-generated RCAs auditable and accountable.
• The research addresses a critical trust deficit in high-stakes operations: fluency without traceability is insufficient for incident response.
• Practitioners should prioritize justification fidelity over raw accuracy when deploying LLMs for operational diagnostics, as verifiable reasoning reduces risk and manual overhead.
• The approach is immediately actionable using current RAG and scoring techniques, offering a practical path to accountable AI in production environments.