Journal article · 2026 · Open access

A statistical framework for evaluating the repeatability and reproducibility of large language models

Cathy Shyr, Boyu Ren, Chih-Yuan Hsu, Minh-Ha Nguyen, Mike Enger, Chao Yan, Rory James Tinker, Thomas A Cassini, Rizwan Hamid, Adam T. Wright, Lisa Bastarache, Josh F. Peterson and 3 more

Journal of the American Medical Informatics Association · Published 27 October 2026 · DOI 10.1093/jamia/ocag186

Open PDFView at the publisherCite this paper

In brief

“A statistical framework for evaluating the repeatability and reproducibility of large language models” is a 2026 journal article by Cathy Shyr and 14 co-authors, published in Journal of the American Medical Informatics Association. OpenAlex counts 42 citations as of 17 November 2026, which puts it in the top 7% of works published in 2026 by citations. It is open access under a CC BY licence: the full text is free to read at Journal of the American Medical Informatics Association.

Citations
42
Most cited in
2026 (40)
Field-weighted impact
41.97
References
62
Authors
15

Abstract

OBJECTIVES: Systematic evaluation of variability in large language model (LLM)-generated outputs is critical for assessing their reliability. We developed a general statistical framework and Python package (llmReproScore) for quantifying the repeatability and reproducibility of LLM outputs. MATERIALS AND METHODS: Following US Food and Drug Administration terminology, our framework establishes four metrics for quantifying LLM repeatability and reproducibility along two dimensions: semantic and internal. Semantic metrics quantify variability in the meaning of LLM-generated outputs, whereas internal metrics quantify variability in token-generation certainty. We validated these metrics, and as an illustrative use case, applied them to diagnostic reasoning outputs generated across LLMs and prompting strategies using US Medical Licensing Examination questions and 90 real-world rare disease cases from the Undiagnosed Diseases Network. RESULTS: Validation results showed that semantic scores aligned with human ratings (Spearman ρ = 0.88 for repeatability and 0.66 for reproducibility) and confirmed that internal metrics capture variability in token-generation certainty. In the illustrative use case, LLM repeatability was generally not associated with accuracy, suggesting that LLM output variability and accuracy capture distinct dimensions of model performance. DISCUSSION: The proposed metrics can be used to characterize LLM output variability and systematically compare models, prompts, and generation settings. These comparisons can inform more rigorous selection of LLM configurations based on repeatability and reproducibility. CONCLUSION: We developed and validated generalizable metrics for quantifying LLM output variability. The proposed metrics can complement accuracy-based evaluation to inform experimental design and support more rigorous comparison and selection of models, prompts, and generation settings.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Shyr, C., Ren, B., Hsu, C.-Y., Nguyen, M.-H., Enger, M., Yan, C., Tinker, R. J., Cassini, T. A., Hamid, R., Wright, A. T., Bastarache, L., Peterson, J. F., Malin, B., Xu, H., & Undiagnosed Diseases Network. (2026). A statistical framework for evaluating the repeatability and reproducibility of large language models. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocag186

MLA 9

Shyr, Cathy, et al. “A statistical framework for evaluating the repeatability and reproducibility of large language models.” Journal of the American Medical Informatics Association, 2026. https://doi.org/10.1093/jamia/ocag186.

Chicago

Shyr, Cathy, Boyu Ren, Chih-Yuan Hsu, Minh-Ha Nguyen, Mike Enger, Chao Yan, Rory James Tinker, et al. 2026. “A statistical framework for evaluating the repeatability and reproducibility of large language models.” Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocag186.

Harvard

Shyr, C. et al. (2026) ‘A statistical framework for evaluating the repeatability and reproducibility of large language models’, Journal of the American Medical Informatics Association. doi:10.1093/jamia/ocag186.

IEEE

C. Shyr et al., “A statistical framework for evaluating the repeatability and reproducibility of large language models,” Journal of the American Medical Informatics Association, 2026, doi: 10.1093/jamia/ocag186.

BibTeX

@article{shyr2026statistical,
  title     = {{A statistical framework for evaluating the repeatability and reproducibility of large language models}},
  author    = {Shyr, Cathy and Ren, Boyu and Hsu, Chih-Yuan and Nguyen, Minh-Ha and Enger, Mike and Yan, Chao and Tinker, Rory James and Cassini, Thomas A and Hamid, Rizwan and Wright, Adam T. and Bastarache, Lisa and Peterson, Josh F. and Malin, Bradley and Xu, Hua and Undiagnosed Diseases Network},
  journal   = {Journal of the American Medical Informatics Association},
  year      = {2026},
  doi       = {10.1093/jamia/ocag186},
  url       = {https://doi.org/10.1093/jamia/ocag186}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education

  • large language models
  • repeatability
  • reproducibility
  • output variability
  • semantic variation
  • diagnostic reasoning
  • rare diseases
  • Undiagnosed Diseases Network

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Shyr, C., Ren, B., Hsu, C.-Y., Nguyen, M.-H., Enger, M., Yan, C., Tinker, R. J., Cassini, T. A., Hamid, R., Wright, A. T., Bastarache, L., Peterson, J. F., Malin, B., Xu, H., & Undiagnosed Diseases Network. (2026). A statistical framework for evaluating the repeatability and reproducibility of large language models. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocag186

How many times has this paper been cited?

42 times, according to OpenAlex on 17 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Journal of the American Medical Informatics Association: sign in and open the PDF from this page.

Who wrote this paper?

Cathy Shyr, Boyu Ren, Chih-Yuan Hsu, Minh-Ha Nguyen, Mike Enger, Chao Yan, Rory James Tinker, Thomas A Cassini, Rizwan Hamid, Adam T. Wright, Lisa Bastarache, Josh F. Peterson and 3 more.

What is the DOI of this paper?

10.1093/jamia/ocag186, permanently at https://doi.org/10.1093/jamia/ocag186. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 17 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.