Preprint · 2026 · Open access

RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine: a scoping review protocol

Junbok Lee, Huiyeol Yun, Yoon Bin Jung

OSF Preprints (OSF Preprints) · Published 4 October 2026 · DOI 10.17605/osf.io/phrv2

Open full textView at the publisherCite this paper

In brief

“RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine: a scoping review protocol” is a 2026 preprint by Junbok Lee and 2 co-authors, published in OSF Preprints (OSF Preprints). OpenAlex counts 0 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at OSF Preprints (OSF Preprints).

Citations
0
Authors
3

Abstract

This scoping review maps the current landscape of studies applying large language models as automated evaluators (LLM-as-a-Judge, LaaJ) in medical and healthcare contexts. Despite the rapid growth of LaaJ applications in medicine — spanning tasks such as patient communication, clinical decision support, and clinical documentation — there is currently no standardized methodology or reporting framework for how these evaluator systems should be designed, validated, and reported, resulting in substantial heterogeneity across studies and limited reproducibility. The project proceeds in two stages. First, we conduct a scoping review of the published literature (PubMed, EMBASE, IEEE Xplore, ACM Digital Library; January 2024–April 2026) to characterize how LaaJ has been used in medicine, including the clinical task types addressed, the input data and evaluation criteria employed, the judge models and system architectures used, the prompting and output strategies adopted, and the extent and rigor of human validation and agreement reporting. Screening is performed in two stages (title/abstract and full text), with the title/abstract stage supported by an LLM-assisted screening pipeline that is calibrated and validated against human review before being applied to the full corpus. Second, building on the methodological gaps identified in the review, we develop RULE-LLM, a reporting framework intended to help researchers transparently document the design, data, output, and performance evaluation of LaaJ systems used in medicine. The framework is organized into four domains — (1) Role and Task Specification, (2) Underlying Data and Criteria, (3) LLM-Judge Design and Output, and (4) Evaluation of Judge Performance — comprising 24 items in total. Framework development follows established guidance for health research reporting-guideline development, and the finalized item set is evaluated for content validity (I-CVI, S-CVI) by an expert panel and for inter-rater reliability (Fleiss' kappa) using independent raters applying the items to a purposive sample of included studies. Expected outputs are: (1) a descriptive synthesis of current LaaJ practice and reporting gaps in the medical literature, and (2) a validated, ready-to-use reporting checklist (RULE-LLM) to improve methodological transparency and reproducibility in future LaaJ research in medicine.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Lee, J., Yun, H., & Bin Jung, Y. (2026). RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine: a scoping review protocol. OSF Preprints (OSF Preprints). https://doi.org/10.17605/osf.io/phrv2

MLA 9

Lee, Junbok, et al. “RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine: a scoping review protocol.” OSF Preprints (OSF Preprints), 2026. https://doi.org/10.17605/osf.io/phrv2.

Chicago

Lee, Junbok, Huiyeol Yun, and Yoon Bin Jung. 2026. “RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine: a scoping review protocol.” OSF Preprints (OSF Preprints). https://doi.org/10.17605/osf.io/phrv2.

Harvard

Lee, J., Yun, H. and Bin Jung, Y. (2026) ‘RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine: a scoping review protocol’, OSF Preprints (OSF Preprints). doi:10.17605/osf.io/phrv2.

IEEE

J. Lee, H. Yun, and Y. Bin Jung, “RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine: a scoping review protocol,” OSF Preprints (OSF Preprints), 2026, doi: 10.17605/osf.io/phrv2.

BibTeX

@misc{lee2026rule,
  title     = {{RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine: a scoping review protocol}},
  author    = {Lee, Junbok and Yun, Huiyeol and Bin Jung, Yoon},
  howpublished = {OSF Preprints (OSF Preprints)},
  year      = {2026},
  doi       = {10.17605/osf.io/phrv2},
  url       = {https://doi.org/10.17605/osf.io/phrv2}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education

  • large language models
  • LLM-as-a-judge
  • automated assessment
  • reporting framework
  • clinical decision support
  • clinical documentation
  • patient communication
  • reproducibility

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Lee, J., Yun, H., & Bin Jung, Y. (2026). RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine: a scoping review protocol. OSF Preprints (OSF Preprints). https://doi.org/10.17605/osf.io/phrv2

How many times has this paper been cited?

0 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at OSF Preprints (OSF Preprints): sign in and open the full text from this page.

Who wrote this paper?

Junbok Lee, Huiyeol Yun, Yoon Bin Jung.

What is the DOI of this paper?

10.17605/osf.io/phrv2, permanently at https://doi.org/10.17605/osf.io/phrv2. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.