Preprint · 2026 · Open access
RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine: a scoping review protocol
OSF Preprints (OSF Preprints) · Published 4 October 2026 · DOI 10.17605/osf.io/phrv2
In brief
“RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine: a scoping review protocol” is a 2026 preprint by Junbok Lee and 2 co-authors, published in OSF Preprints (OSF Preprints). OpenAlex counts 0 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at OSF Preprints (OSF Preprints).
- Citations
- 0
- Authors
- 3
Abstract
This scoping review maps the current landscape of studies applying large language models as automated evaluators (LLM-as-a-Judge, LaaJ) in medical and healthcare contexts. Despite the rapid growth of LaaJ applications in medicine — spanning tasks such as patient communication, clinical decision support, and clinical documentation — there is currently no standardized methodology or reporting framework for how these evaluator systems should be designed, validated, and reported, resulting in substantial heterogeneity across studies and limited reproducibility. The project proceeds in two stages. First, we conduct a scoping review of the published literature (PubMed, EMBASE, IEEE Xplore, ACM Digital Library; January 2024–April 2026) to characterize how LaaJ has been used in medicine, including the clinical task types addressed, the input data and evaluation criteria employed, the judge models and system architectures used, the prompting and output strategies adopted, and the extent and rigor of human validation and agreement reporting. Screening is performed in two stages (title/abstract and full text), with the title/abstract stage supported by an LLM-assisted screening pipeline that is calibrated and validated against human review before being applied to the full corpus. Second, building on the methodological gaps identified in the review, we develop RULE-LLM, a reporting framework intended to help researchers transparently document the design, data, output, and performance evaluation of LaaJ systems used in medicine. The framework is organized into four domains — (1) Role and Task Specification, (2) Underlying Data and Criteria, (3) LLM-Judge Design and Output, and (4) Evaluation of Judge Performance — comprising 24 items in total. Framework development follows established guidance for health research reporting-guideline development, and the finalized item set is evaluated for content validity (I-CVI, S-CVI) by an expert panel and for inter-rater reliability (Fleiss' kappa) using independent raters applying the items to a purposive sample of included studies. Expected outputs are: (1) a descriptive synthesis of current LaaJ practice and reporting gaps in the medical literature, and (2) a validated, ready-to-use reporting checklist (RULE-LLM) to improve methodological transparency and reproducibility in future LaaJ research in medicine.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Lee, J., Yun, H., & Bin Jung, Y. (2026). RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine: a scoping review protocol. OSF Preprints (OSF Preprints). https://doi.org/10.17605/osf.io/phrv2
MLA 9
Lee, Junbok, et al. “RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine: a scoping review protocol.” OSF Preprints (OSF Preprints), 2026. https://doi.org/10.17605/osf.io/phrv2.
Chicago
Lee, Junbok, Huiyeol Yun, and Yoon Bin Jung. 2026. “RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine: a scoping review protocol.” OSF Preprints (OSF Preprints). https://doi.org/10.17605/osf.io/phrv2.
Harvard
Lee, J., Yun, H. and Bin Jung, Y. (2026) ‘RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine: a scoping review protocol’, OSF Preprints (OSF Preprints). doi:10.17605/osf.io/phrv2.
IEEE
J. Lee, H. Yun, and Y. Bin Jung, “RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine: a scoping review protocol,” OSF Preprints (OSF Preprints), 2026, doi: 10.17605/osf.io/phrv2.
BibTeX
@misc{lee2026rule,
title = {{RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine: a scoping review protocol}},
author = {Lee, Junbok and Yun, Huiyeol and Bin Jung, Yoon},
howpublished = {OSF Preprints (OSF Preprints)},
year = {2026},
doi = {10.17605/osf.io/phrv2},
url = {https://doi.org/10.17605/osf.io/phrv2}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- large language models
- LLM-as-a-judge
- automated assessment
- reporting framework
- clinical decision support
- clinical documentation
- patient communication
- reproducibility
Work with this paper
Most-cited papers in Medicine
- Yaşlı Kadınlarda Üreme SağlığıDemet Kokanalı, Yaprak Engin Üstün · Jinekoloji-Obstetrik ve Neonatoloji Tıp Dergisi · 2019 · 169,630 citations
- Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 CountriesHyuna Sung, Adalberto M. Filho, Mathieu Laversanne and 5 more · CA A Cancer Journal for Clinicians · 2021 · 119,885 citations
- “Mini-mental state”Marshal F. Folstein, Susan E. Folstein, Paul R. McHugh · Journal of Psychiatric Research · 1975 · 90,424 citations
- Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countriesFreddie Bray, Jacques Ferlay, Isabelle Soerjomataram and 3 more · CA A Cancer Journal for Clinicians · 2018 · 87,396 citations
- Hallmarks of Cancer: The Next GenerationDouglas Hanahan, Robert A. Weinberg · Cell · 2011 · 69,758 citations
Questions about this paper
How do I cite this paper in APA style?
Lee, J., Yun, H., & Bin Jung, Y. (2026). RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine: a scoping review protocol. OSF Preprints (OSF Preprints). https://doi.org/10.17605/osf.io/phrv2
How many times has this paper been cited?
0 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at OSF Preprints (OSF Preprints): sign in and open the full text from this page.
Who wrote this paper?
Junbok Lee, Huiyeol Yun, Yoon Bin Jung.
What is the DOI of this paper?
10.17605/osf.io/phrv2, permanently at https://doi.org/10.17605/osf.io/phrv2. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.