When CQs Go Wrong: Challenges in CQ Verification with OE-Assist

The Hidden Pitfalls of Ontology Verification

A new preprint from arXiv (2606.24619v1) shines a critical light on a seemingly straightforward process in knowledge engineering: verifying ontologies through Competency Questions (CQs). The research exposes significant challenges in CQ verification when assisted by Ontology Engineering (OE) tools, revealing that the process is far more brittle than many practitioners assume.

At its core, CQ-verification involves testing an ontology against a set of natural language questions to confirm it captures the intended domain knowledge. The paper documents systematic failures in this process—specifically, how OE-assist tools can produce false positives (appearing to verify CQs that the ontology does not actually address) and false negatives (failing to recognize valid CQ coverage). The root cause lies in the mismatch between natural language semantics and formal ontological representations. A CQ like "Which medications treat hypertension?" may be structurally answerable by the ontology, but subtle differences in phrasing, synonym usage, or implicit domain assumptions can derail verification.

Why This Matters

This research arrives at a crucial moment. Ontologies underpin everything from biomedical knowledge graphs to enterprise data catalogs and AI reasoning systems. If the verification process itself is flawed, downstream applications inherit silent errors. For instance, a clinical decision support system relying on an ontology that passed CQ verification might fail to retrieve relevant treatment options—not because the knowledge is missing, but because the verification tool misaligned the question's intent with the ontology's structure.

The implications extend beyond ontology engineering. As organizations increasingly adopt retrieval-augmented generation (RAG) and structured knowledge bases for LLM applications, the reliability of underlying ontological structures becomes paramount. A CQ-verification failure could mean an AI assistant confidently answers a user query with incomplete or incorrect information, eroding trust in the system.

Implications for AI Practitioners

First, practitioners must treat CQ verification as a probabilistic rather than deterministic process. OE-assist tools should be used as hypothesis generators, not final arbiters. Second, the research underscores the need for multi-modal verification—combining automated CQ checks with human review and, crucially, behavioral testing against real user queries. Third, ontology designers should document not just which CQs pass, but the specific linguistic and structural assumptions made during verification. This transparency enables debugging when downstream systems behave unexpectedly.

Finally, the findings suggest that the AI community should invest in more robust semantic alignment techniques. Current approaches often rely on shallow syntactic matching or fixed thesauri. Future work must incorporate contextual embeddings, domain-specific language models, and perhaps most importantly, explicit modeling of the verification process's uncertainty.

Key Takeaways

• CQ verification with OE-assist tools is prone to systematic false positives and negatives due to semantic mismatches between natural language and formal ontology structures
• Flawed verification cascades into unreliable downstream AI systems, particularly in knowledge-intensive applications like clinical decision support and enterprise RAG
• Practitioners should adopt multi-modal verification strategies combining automated checks, human review, and behavioral testing against real-world queries
• The AI field needs better semantic alignment techniques that go beyond syntactic matching to capture contextual and domain-specific nuances in verification