Preprint · 2026 · Open access

A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined

Zhangshu Joshua Jiang, Zina M. Ibrahim, James Teo

arXiv (Cornell University) · Published 1 October 2026 · DOI 10.48550/arxiv.2610.01938

Open PDFView at the publisherCite this paper

In brief

“A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined” is a 2026 preprint by Zhangshu Joshua Jiang and 2 co-authors, published in arXiv (Cornell University). OpenAlex counts 0 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at arXiv (Cornell University).

Citations
0
References
38
Authors
3

Abstract

Exam-style accuracy does not establish whether large language models (LLMs) reason well over clinical records. We define clinical reasoning as integrating and updating evidence across time and sources to form, revise and justify a patient's problem representation and a defensible plan. This structured narrative review maps three literatures: medical education assessment instruments, clinical LLM benchmarks published from 2023 onwards, and general-domain methods for evaluating long-form generation. We examine six dimensions: problem representation, temporal synthesis, differential and management reasoning, counterfactual reasoning, calibrated uncertainty, and reasoning faithfulness. Preprints are included and flagged. No single instrument covers all six dimensions. Problem representation and differential or management reasoning are reasonably covered, although reliability varies by instrument and setting. TIMER-Eval targets temporal synthesis, and ER-Reason assesses sequential diagnostic belief updating. Dedicated uncertainty and counterfactual evaluations are emerging, but their applicability to longitudinal free-text reasoning remains limited. Factual completeness is well theorised in general-domain evaluation, with early clinical evidence of important omissions. Faithfulness remains the weakest dimension, with one identified clinical causal-ablation study on multiple-choice questions. Existing tools should be combined through binary rubric items, separate completeness and correctness scores, case-specific importance weighting with non-compensable safety caps, temporal order-consistency checks, and chance-corrected reliability reporting. Further design work is needed for calibrated uncertainty, counterfactual reasoning and faithfulness over longitudinal free-text records. This review provides a design rationale, not a validated instrument.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Jiang, Z. J., Ibrahim, Z. M., & Teo, J. (2026). A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2610.01938

MLA 9

Jiang, Zhangshu Joshua, et al. “A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined.” arXiv (Cornell University), 2026. https://doi.org/10.48550/arxiv.2610.01938.

Chicago

Jiang, Zhangshu Joshua, Zina M. Ibrahim, and James Teo. 2026. “A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2610.01938.

Harvard

Jiang, Z. J., Ibrahim, Z. M. and Teo, J. (2026) ‘A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined’, arXiv (Cornell University). doi:10.48550/arxiv.2610.01938.

IEEE

Z. J. Jiang, Z. M. Ibrahim, and J. Teo, “A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined,” arXiv (Cornell University), 2026, doi: 10.48550/arxiv.2610.01938.

BibTeX

@misc{jiang2026rubric,
  title     = {{A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined}},
  author    = {Jiang, Zhangshu Joshua and Ibrahim, Zina M. and Teo, James},
  howpublished = {arXiv (Cornell University)},
  year      = {2026},
  doi       = {10.48550/arxiv.2610.01938},
  url       = {https://doi.org/10.48550/arxiv.2610.01938}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education

  • clinical reasoning
  • large language models
  • counterfactual reasoning
  • faithfulness
  • problem representation
  • medical records
  • medical education assessment

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Jiang, Z. J., Ibrahim, Z. M., & Teo, J. (2026). A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2610.01938

How many times has this paper been cited?

0 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.

Who wrote this paper?

Zhangshu Joshua Jiang, Zina M. Ibrahim, James Teo.

What is the DOI of this paper?

10.48550/arxiv.2610.01938, permanently at https://doi.org/10.48550/arxiv.2610.01938. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.