Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization

This paper, “Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization,” tackles a critical bottleneck in AI-driven mathematics: moving beyond proving a given theorem to actually generating the correct formal statement of that theorem from natural language. The researchers identify that current benchmarks for theorem-proving (like MiniF2F) assume the formal statement is already provided. The real challenge, however, lies in the “formalization” step—translating a human-written problem into Lean code that accurately captures its meaning.

What Happened

The core insight is that “compilation” (i.e., the Lean code type-checks without errors) is a necessary but insufficient condition for correctness. A Lean declaration may compile perfectly yet be semantically wrong—for example, it might prove a trivial variant of the intended theorem or contain a subtle off-by-one error in a definition. The authors propose a new evaluation framework that moves beyond binary compilation success. They introduce metrics that measure “faithfulness” between the natural language statement and the generated Lean code, likely involving semantic alignment checks, logical equivalence verification, or round-trip translation consistency. This shifts the goal from “does it compile?” to “does it mean the same thing?”

Why It Matters

This research directly addresses a practical barrier to using LLMs for formal mathematics. Current systems can often generate plausible-looking Lean code that compiles, but subtle errors in formalization render the proof useless for downstream tasks (e.g., verifying a new lemma or building a library). For AI practitioners, this means:

• Benchmarking becomes more realistic. Evaluating only compilation success overestimates model capability. A model that scores 80% on compilation might only be 50% faithful, misleading teams about deployment readiness.
• Training data quality improves. The “faithfulness” metric provides a signal for fine-tuning. Instead of rewarding any valid Lean code, practitioners can reward code that preserves the original problem’s semantics, leading to more reliable formalization agents.
• Debugging becomes tractable. When a model generates an unfaithful but compilable statement, the error is hidden. This framework provides a diagnostic tool to identify where the meaning diverged—whether in variable definitions, type mismatches, or logical quantifiers.

Implications for AI Practitioners

For teams building LLM-based theorem-proving assistants, this work implies a need to redesign evaluation pipelines. A simple “compiles? yes/no” metric should be replaced with a multi-dimensional score that includes syntactic validity, semantic faithfulness, and perhaps proof completeness. This is particularly relevant for applications like automated curriculum generation (where a wrong formalization teaches incorrect concepts) or large-scale library maintenance (where an unfaithful lemma corrupts dependent proofs).

Furthermore, the approach suggests a new training objective: contrastive learning between faithful and unfaithful formalizations. By collecting pairs of Lean statements that compile but differ in meaning, models can learn to discriminate subtle semantic shifts—a skill essential for robust formalization.

Key Takeaways

• Compilation is not correctness: A Lean declaration can type-check while being semantically wrong; the paper introduces “faithfulness” as a separate evaluation dimension.
• New benchmarks needed: Current theorem-proving benchmarks (e.g., MiniF2F) are insufficient for evaluating end-to-end formalization from natural language.
• Practical signal for training: Faithfulness metrics provide a better reward signal for fine-tuning LLMs, moving beyond superficial syntactic validity.
• Debugging tool for practitioners: The framework helps identify where semantic divergence occurs in generated formal statements, enabling targeted model improvements.