Journal article · 2026 · Open access
Evaluating large language models for endodontic case difficulty assessment: accuracy, consistency, temporal stability, and agreement
BMC Oral Health, 26(1) · Published 11 April 2026 · DOI 10.1186/s12903-026-08323-5
In brief
“Evaluating large language models for endodontic case difficulty assessment: accuracy, consistency, temporal stability, and agreement” is a 2026 journal article by Ekin Deniz Çatmabacak and İrem Çetinkaya, published in BMC Oral Health. OpenAlex counts 2 citations as of 18 November 2026, which puts it in the top 5% of works published in 2026 by citations. It is open access under a CC BY-NC-ND licence: the full text is free to read at BMC Oral Health.
- Citations
- 2
- Most cited in
- 2026 (2)
- Field-weighted impact
- 2.40
- References
- 44
- Authors
- 2
Abstract
BACKGROUND: Large language models (LLMs) are increasingly explored as clinical decision-support tools in dentistry; however, their reliability in endodontic decision-making remains uncertain. Most previous studies have relied on single-response accuracy, providing limited insight into the reproducibility and stability of LLM outputs across repeated evaluations. This study aimed to evaluate LLM performance in endodontic case difficulty assessment using the American Association of Endodontists (AAE) framework, with emphasis on accuracy, consistency, temporal stability, and inter-model agreement. METHODS: Thirty standardised endodontic cases (10 low, 10 moderate, 10 high difficulty) were developed according to AAE criteria and presented in three linguistically distinct but clinically equivalent formats (n = 90 test items). Two LLMs were queried independently 30 times per item, yielding 5,400 classifications in a repeated-measures design. Accuracy was defined as agreement with expert-derived ground truth. Consistency and temporal stability were assessed across repeated queries, and inter-model agreement was evaluated using Cohen’s kappa statistics. RESULTS: Both models demonstrated a clear difficulty-dependent performance pattern. Accuracy and consistency were highest in low-difficulty cases, whereas moderate-difficulty cases showed the lowest accuracy and greatest variability. High-difficulty cases exhibited the highest response stability despite differences in accuracy between models. Linguistic variation had minimal impact on performance. Inter-model agreement was substantial, with disagreements largely confined to adjacent difficulty categories. CONCLUSIONS: LLM performance in endodontic case difficulty assessment is influenced by the clarity of clinical features more than case complexity alone. Moderate-difficulty cases represent a critical zone of decision instability, where reliance on single-response accuracy may be misleading. These findings support the need for uncertainty-aware evaluation frameworks and reinforce the role of LLMs as adjunctive clinical decision-support tools.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Çatmabacak, E. D., & Çetinkaya, İ. (2026). Evaluating large language models for endodontic case difficulty assessment: accuracy, consistency, temporal stability, and agreement. BMC Oral Health, 26(1). https://doi.org/10.1186/s12903-026-08323-5
MLA 9
Çatmabacak, Ekin Deniz, and İrem Çetinkaya. “Evaluating large language models for endodontic case difficulty assessment: accuracy, consistency, temporal stability, and agreement.” BMC Oral Health, vol. 26, no. 1, 2026. https://doi.org/10.1186/s12903-026-08323-5.
Chicago
Çatmabacak, Ekin Deniz and İrem Çetinkaya. 2026. “Evaluating large language models for endodontic case difficulty assessment: accuracy, consistency, temporal stability, and agreement.” BMC Oral Health 26 (1). https://doi.org/10.1186/s12903-026-08323-5.
Harvard
Çatmabacak, E. D. and Çetinkaya, İ. (2026) ‘Evaluating large language models for endodontic case difficulty assessment: accuracy, consistency, temporal stability, and agreement’, BMC Oral Health, 26(1). doi:10.1186/s12903-026-08323-5.
IEEE
E. D. Çatmabacak and İ. Çetinkaya, “Evaluating large language models for endodontic case difficulty assessment: accuracy, consistency, temporal stability, and agreement,” BMC Oral Health, vol. 26, no. 1, 2026, doi: 10.1186/s12903-026-08323-5.
BibTeX
@article{catmabacak2026evaluating,
title = {{Evaluating large language models for endodontic case difficulty assessment: accuracy, consistency, temporal stability, and agreement}},
author = {Çatmabacak, Ekin Deniz and Çetinkaya, İrem},
journal = {BMC Oral Health},
year = {2026},
volume = {26},
number = {1},
doi = {10.1186/s12903-026-08323-5},
url = {https://doi.org/10.1186/s12903-026-08323-5}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- large language models
- clinical decision support
- temporal stability
- inter-model agreement
- language variation
- case complexity
Work with this paper
Most-cited papers in Medicine
- Yaşlı Kadınlarda Üreme SağlığıDemet Kokanalı, Yaprak Engin Üstün · Jinekoloji-Obstetrik ve Neonatoloji Tıp Dergisi · 2019 · 169,630 citations
- Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 CountriesHyuna Sung, Adalberto M. Filho, Mathieu Laversanne and 5 more · CA A Cancer Journal for Clinicians · 2021 · 119,885 citations
- “Mini-mental state”Marshal F. Folstein, Susan E. Folstein, Paul R. McHugh · Journal of Psychiatric Research · 1975 · 90,424 citations
- Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countriesFreddie Bray, Jacques Ferlay, Isabelle Soerjomataram and 3 more · CA A Cancer Journal for Clinicians · 2018 · 87,396 citations
- Hallmarks of Cancer: The Next GenerationDouglas Hanahan, Robert A. Weinberg · Cell · 2011 · 69,758 citations
Questions about this paper
How do I cite this paper in APA style?
Çatmabacak, E. D., & Çetinkaya, İ. (2026). Evaluating large language models for endodontic case difficulty assessment: accuracy, consistency, temporal stability, and agreement. BMC Oral Health, 26(1). https://doi.org/10.1186/s12903-026-08323-5
How many times has this paper been cited?
2 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at BMC Oral Health: sign in and open the full text from this page.
Who wrote this paper?
Ekin Deniz Çatmabacak, İrem Çetinkaya.
What is the DOI of this paper?
10.1186/s12903-026-08323-5, permanently at https://doi.org/10.1186/s12903-026-08323-5. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.