Study finds large language models can draft ontology competency questions — but quality and domain fit vary

What the paper did

A new arXiv preprint (arXiv:2604.16258) presents a cross-domain empirical study of whether large language models (LLMs) can generate Competency Questions (CQs), the natural-language requirements used to elicit and validate ontologies. The authors compare open-source and proprietary models across multiple subject areas, measuring not only linguistic fluency but also domain relevance, answerability, and utility for downstream ontology engineering. The headline result: LLMs can produce plausible CQs at scale, but their usefulness depends strongly on the model used and the target domain.

Key findings and methodology

The study evaluates both “open” and “closed” model families and reports systematic differences in output quality and factual grounding. Reportedly, open-source models produced more diverse phrasing but sometimes less precise domain semantics, while closed proprietary models often yielded tighter, more on-topic questions at higher cost and with access restrictions. The paper uses quantitative metrics and human-in-the-loop validation to show that many automatically generated CQs require refinement by subject-matter experts before they can be used reliably in ontology authoring.

Why this matters — and for whom

Why should readers care? Ontologies underpin knowledge graphs, search, regulatory compliance systems and much enterprise AI. Automating the laborious task of eliciting CQs could speed ontology projects dramatically. But there are risks: mistaken or underspecified competency questions can bake errors into knowledge representations. For practitioners in China and elsewhere, model choice is also political and practical. It has been reported that export controls and shifting trade policy have incentivized Chinese firms and researchers to lean on domestically developed or open models — for example Baidu (百度)’s ERNIE (文心), Alibaba (阿里巴巴)’s Tongyi (通义千问) and Huawei (华为)’s PanGu (盘古) — to ensure continued access and customization.

Bottom line

LLMs are a promising assist for ontology engineers, not a replacement. They can draft the scaffolding of requirements fast. But human curation remains essential to ensure domain fidelity and to guard against subtle, high-impact errors — especially in regulated or safety-critical domains. Will AI ever replace the domain expert at the heart of ontology work? Not yet.