Journal article · 2026 · Open access

Evaluating large language models for endodontic case difficulty assessment: accuracy, consistency, temporal stability, and agreement

Ekin Deniz Çatmabacak, İrem Çetinkaya

BMC Oral Health, 26(1) · Published 11 April 2026 · DOI 10.1186/s12903-026-08323-5

Open full textCite this paper

In brief

“Evaluating large language models for endodontic case difficulty assessment: accuracy, consistency, temporal stability, and agreement” is a 2026 journal article by Ekin Deniz Çatmabacak and İrem Çetinkaya, published in BMC Oral Health. OpenAlex counts 2 citations as of 18 November 2026, which puts it in the top 5% of works published in 2026 by citations. It is open access under a CC BY-NC-ND licence: the full text is free to read at BMC Oral Health.

Citations
2
Most cited in
2026 (2)
Field-weighted impact
2.40
References
44
Authors
2

Abstract

BACKGROUND: Large language models (LLMs) are increasingly explored as clinical decision-support tools in dentistry; however, their reliability in endodontic decision-making remains uncertain. Most previous studies have relied on single-response accuracy, providing limited insight into the reproducibility and stability of LLM outputs across repeated evaluations. This study aimed to evaluate LLM performance in endodontic case difficulty assessment using the American Association of Endodontists (AAE) framework, with emphasis on accuracy, consistency, temporal stability, and inter-model agreement. METHODS: Thirty standardised endodontic cases (10 low, 10 moderate, 10 high difficulty) were developed according to AAE criteria and presented in three linguistically distinct but clinically equivalent formats (n = 90 test items). Two LLMs were queried independently 30 times per item, yielding 5,400 classifications in a repeated-measures design. Accuracy was defined as agreement with expert-derived ground truth. Consistency and temporal stability were assessed across repeated queries, and inter-model agreement was evaluated using Cohen’s kappa statistics. RESULTS: Both models demonstrated a clear difficulty-dependent performance pattern. Accuracy and consistency were highest in low-difficulty cases, whereas moderate-difficulty cases showed the lowest accuracy and greatest variability. High-difficulty cases exhibited the highest response stability despite differences in accuracy between models. Linguistic variation had minimal impact on performance. Inter-model agreement was substantial, with disagreements largely confined to adjacent difficulty categories. CONCLUSIONS: LLM performance in endodontic case difficulty assessment is influenced by the clarity of clinical features more than case complexity alone. Moderate-difficulty cases represent a critical zone of decision instability, where reliance on single-response accuracy may be misleading. These findings support the need for uncertainty-aware evaluation frameworks and reinforce the role of LLMs as adjunctive clinical decision-support tools.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Çatmabacak, E. D., & Çetinkaya, İ. (2026). Evaluating large language models for endodontic case difficulty assessment: accuracy, consistency, temporal stability, and agreement. BMC Oral Health, 26(1). https://doi.org/10.1186/s12903-026-08323-5

MLA 9

Çatmabacak, Ekin Deniz, and İrem Çetinkaya. “Evaluating large language models for endodontic case difficulty assessment: accuracy, consistency, temporal stability, and agreement.” BMC Oral Health, vol. 26, no. 1, 2026. https://doi.org/10.1186/s12903-026-08323-5.

Chicago

Çatmabacak, Ekin Deniz and İrem Çetinkaya. 2026. “Evaluating large language models for endodontic case difficulty assessment: accuracy, consistency, temporal stability, and agreement.” BMC Oral Health 26 (1). https://doi.org/10.1186/s12903-026-08323-5.

Harvard

Çatmabacak, E. D. and Çetinkaya, İ. (2026) ‘Evaluating large language models for endodontic case difficulty assessment: accuracy, consistency, temporal stability, and agreement’, BMC Oral Health, 26(1). doi:10.1186/s12903-026-08323-5.

IEEE

E. D. Çatmabacak and İ. Çetinkaya, “Evaluating large language models for endodontic case difficulty assessment: accuracy, consistency, temporal stability, and agreement,” BMC Oral Health, vol. 26, no. 1, 2026, doi: 10.1186/s12903-026-08323-5.

BibTeX

@article{catmabacak2026evaluating,
  title     = {{Evaluating large language models for endodontic case difficulty assessment: accuracy, consistency, temporal stability, and agreement}},
  author    = {Çatmabacak, Ekin Deniz and Çetinkaya, İrem},
  journal   = {BMC Oral Health},
  year      = {2026},
  volume    = {26},
  number    = {1},
  doi       = {10.1186/s12903-026-08323-5},
  url       = {https://doi.org/10.1186/s12903-026-08323-5}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education

  • large language models
  • clinical decision support
  • temporal stability
  • inter-model agreement
  • language variation
  • case complexity

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Çatmabacak, E. D., & Çetinkaya, İ. (2026). Evaluating large language models for endodontic case difficulty assessment: accuracy, consistency, temporal stability, and agreement. BMC Oral Health, 26(1). https://doi.org/10.1186/s12903-026-08323-5

How many times has this paper been cited?

2 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at BMC Oral Health: sign in and open the full text from this page.

Who wrote this paper?

Ekin Deniz Çatmabacak, İrem Çetinkaya.

What is the DOI of this paper?

10.1186/s12903-026-08323-5, permanently at https://doi.org/10.1186/s12903-026-08323-5. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.