Journal article · 2026 · Open access
A statistical framework for evaluating the repeatability and reproducibility of large language models
Journal of the American Medical Informatics Association · Published 27 October 2026 · DOI 10.1093/jamia/ocag186
In brief
“A statistical framework for evaluating the repeatability and reproducibility of large language models” is a 2026 journal article by Cathy Shyr and 14 co-authors, published in Journal of the American Medical Informatics Association. OpenAlex counts 42 citations as of 17 November 2026, which puts it in the top 7% of works published in 2026 by citations. It is open access under a CC BY licence: the full text is free to read at Journal of the American Medical Informatics Association.
- Citations
- 42
- Most cited in
- 2026 (40)
- Field-weighted impact
- 41.97
- References
- 62
- Authors
- 15
Abstract
OBJECTIVES: Systematic evaluation of variability in large language model (LLM)-generated outputs is critical for assessing their reliability. We developed a general statistical framework and Python package (llmReproScore) for quantifying the repeatability and reproducibility of LLM outputs. MATERIALS AND METHODS: Following US Food and Drug Administration terminology, our framework establishes four metrics for quantifying LLM repeatability and reproducibility along two dimensions: semantic and internal. Semantic metrics quantify variability in the meaning of LLM-generated outputs, whereas internal metrics quantify variability in token-generation certainty. We validated these metrics, and as an illustrative use case, applied them to diagnostic reasoning outputs generated across LLMs and prompting strategies using US Medical Licensing Examination questions and 90 real-world rare disease cases from the Undiagnosed Diseases Network. RESULTS: Validation results showed that semantic scores aligned with human ratings (Spearman ρ = 0.88 for repeatability and 0.66 for reproducibility) and confirmed that internal metrics capture variability in token-generation certainty. In the illustrative use case, LLM repeatability was generally not associated with accuracy, suggesting that LLM output variability and accuracy capture distinct dimensions of model performance. DISCUSSION: The proposed metrics can be used to characterize LLM output variability and systematically compare models, prompts, and generation settings. These comparisons can inform more rigorous selection of LLM configurations based on repeatability and reproducibility. CONCLUSION: We developed and validated generalizable metrics for quantifying LLM output variability. The proposed metrics can complement accuracy-based evaluation to inform experimental design and support more rigorous comparison and selection of models, prompts, and generation settings.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Shyr, C., Ren, B., Hsu, C.-Y., Nguyen, M.-H., Enger, M., Yan, C., Tinker, R. J., Cassini, T. A., Hamid, R., Wright, A. T., Bastarache, L., Peterson, J. F., Malin, B., Xu, H., & Undiagnosed Diseases Network. (2026). A statistical framework for evaluating the repeatability and reproducibility of large language models. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocag186
MLA 9
Shyr, Cathy, et al. “A statistical framework for evaluating the repeatability and reproducibility of large language models.” Journal of the American Medical Informatics Association, 2026. https://doi.org/10.1093/jamia/ocag186.
Chicago
Shyr, Cathy, Boyu Ren, Chih-Yuan Hsu, Minh-Ha Nguyen, Mike Enger, Chao Yan, Rory James Tinker, et al. 2026. “A statistical framework for evaluating the repeatability and reproducibility of large language models.” Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocag186.
Harvard
Shyr, C. et al. (2026) ‘A statistical framework for evaluating the repeatability and reproducibility of large language models’, Journal of the American Medical Informatics Association. doi:10.1093/jamia/ocag186.
IEEE
C. Shyr et al., “A statistical framework for evaluating the repeatability and reproducibility of large language models,” Journal of the American Medical Informatics Association, 2026, doi: 10.1093/jamia/ocag186.
BibTeX
@article{shyr2026statistical,
title = {{A statistical framework for evaluating the repeatability and reproducibility of large language models}},
author = {Shyr, Cathy and Ren, Boyu and Hsu, Chih-Yuan and Nguyen, Minh-Ha and Enger, Mike and Yan, Chao and Tinker, Rory James and Cassini, Thomas A and Hamid, Rizwan and Wright, Adam T. and Bastarache, Lisa and Peterson, Josh F. and Malin, Bradley and Xu, Hua and Undiagnosed Diseases Network},
journal = {Journal of the American Medical Informatics Association},
year = {2026},
doi = {10.1093/jamia/ocag186},
url = {https://doi.org/10.1093/jamia/ocag186}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- large language models
- repeatability
- reproducibility
- output variability
- semantic variation
- diagnostic reasoning
- rare diseases
- Undiagnosed Diseases Network
Work with this paper
Most-cited papers in Medicine
- Yaşlı Kadınlarda Üreme SağlığıDemet Kokanalı, Yaprak Engin Üstün · Jinekoloji-Obstetrik ve Neonatoloji Tıp Dergisi · 2019 · 169,630 citations
- Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 CountriesHyuna Sung, Adalberto M. Filho, Mathieu Laversanne and 5 more · CA A Cancer Journal for Clinicians · 2021 · 119,885 citations
- “Mini-mental state”Marshal F. Folstein, Susan E. Folstein, Paul R. McHugh · Journal of Psychiatric Research · 1975 · 90,424 citations
- Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countriesFreddie Bray, Jacques Ferlay, Isabelle Soerjomataram and 3 more · CA A Cancer Journal for Clinicians · 2018 · 87,396 citations
- Hallmarks of Cancer: The Next GenerationDouglas Hanahan, Robert A. Weinberg · Cell · 2011 · 69,758 citations
Questions about this paper
How do I cite this paper in APA style?
Shyr, C., Ren, B., Hsu, C.-Y., Nguyen, M.-H., Enger, M., Yan, C., Tinker, R. J., Cassini, T. A., Hamid, R., Wright, A. T., Bastarache, L., Peterson, J. F., Malin, B., Xu, H., & Undiagnosed Diseases Network. (2026). A statistical framework for evaluating the repeatability and reproducibility of large language models. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocag186
How many times has this paper been cited?
42 times, according to OpenAlex on 17 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Journal of the American Medical Informatics Association: sign in and open the PDF from this page.
Who wrote this paper?
Cathy Shyr, Boyu Ren, Chih-Yuan Hsu, Minh-Ha Nguyen, Mike Enger, Chao Yan, Rory James Tinker, Thomas A Cassini, Rizwan Hamid, Adam T. Wright, Lisa Bastarache, Josh F. Peterson and 3 more.
What is the DOI of this paper?
10.1093/jamia/ocag186, permanently at https://doi.org/10.1093/jamia/ocag186. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 17 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.