Dissertation · 2026 · Open access

ATOMIC CLAIM VERIFICATION FOR CLINICAL DIAGNOSTIC REASONING: DECOMPOSING AND VALIDATING LLM REASONING STEPS AGAINST MEDICAL EVIDENCE

Ali Nemati

Minds at UW (University of Wisconsin) · Published 1 August 2026

Open PDFView at the publisherCite this paper

In brief

“ATOMIC CLAIM VERIFICATION FOR CLINICAL DIAGNOSTIC REASONING: DECOMPOSING AND VALIDATING LLM REASONING STEPS AGAINST MEDICAL EVIDENCE” is a 2026 dissertation by Ali Nemati, published in Minds at UW (University of Wisconsin). OpenAlex counts 0 citations as of 19 November 2026. It is open access: the full text is free to read at Minds at UW (University of Wisconsin).

Citations
0
References
28
Authors
1

Abstract

Large Language Models (LLMs) are increasingly evaluated for, and piloted in, clinical settings for diagnostic support, and a growing literature uses LLMs themselves to grade other LLMs' medical reasoning. This dissertation set out to build an atomic-claim verification pipeline for LLM medical reasoning and to evaluate that pipeline against clinician-authored reasoning statements in the MedCaseReasoning corpus (Wu et al., 2025; N = 200 cases, 620 clinician statements). The principal result is a negative methodological one: across a rule-based decomposer, a single primary LLM verifier, and five additional vendor verifiers, no single-verifier configuration is operationally reliable as a stand-alone judge of medical reasoning quality. Seven empirical findings establish this conclusion. (1) DeepSeek-R1:14b reproduces only 25.5% of clinician reasoning statements (95% bootstrap CI [21.3, 29.8]) under a stricter qwen2.5-based gold-standard judge, against Stanford's published 64% baseline. (2) Reasoning recall and effective diagnostic accuracy are statistically dissociated (Pearson r = 0.09), so reaching the correct diagnosis does not require addressing the clinician's reasoning steps. (3) Rule-based atomic-claim decomposition achieves precision 0.88 but recall 0.15 against an independent qwen2.5:32b annotator, with Cohen's kappa = 0.02 for five-type classification, at chance for the four non-observation types. (4) A marginal-matched permutation null shows that the Agreement Bias Index is mathematically determined by the verifier's support rate: 0 of 11 verifier-by-prompt conditions are distinguishable from a random labeler with the same marginal. (5) Cross-verifier agreement on SUPPORTED versus NOT_SUPPORTED is negative, with Fleiss' kappa = -0.134 (standard prompt) and -0.109 (adversarial prompt) and 95% bootstrap confidence intervals of [-0.155, -0.109] and [-0.133, -0.081]. (6) Prompt wording flips identical-claim verdicts: Opus-Sonnet pairwise agreement collapses from kappa = 0.66 (standard) to kappa = -0.02 (adversarial) on the same 250 claims. (7) The dominant LLM failure mode is exclusion failure (63.3%) and factual omission (36.1%), together 99.4% of unrecalled clinician statements and invariant to whether the final diagnosis was reached. Findings 1 through 7 establish that single-verifier LLM-as-judge evaluation is not a defensible methodology for medical reasoning assessment. The contribution of this dissertation is the systematic demonstration of this limitation, the Agreement Bias Index as a descriptive marginal-rate index (not a bias quality metric), and a gold-standard-grounded evaluation alternative built against clinician-authored reasoning statements. Knowledge-grounded verification using UMLS and PubMed is positioned as the methodological response motivated by these findings, with empirical evaluation deferred to future work.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Nemati, A. (2026). ATOMIC CLAIM VERIFICATION FOR CLINICAL DIAGNOSTIC REASONING: DECOMPOSING AND VALIDATING LLM REASONING STEPS AGAINST MEDICAL EVIDENCE. Minds at UW (University of Wisconsin). https://digital.library.wisc.edu/1793/98220

MLA 9

Nemati, Ali. “ATOMIC CLAIM VERIFICATION FOR CLINICAL DIAGNOSTIC REASONING: DECOMPOSING AND VALIDATING LLM REASONING STEPS AGAINST MEDICAL EVIDENCE.” Minds at UW (University of Wisconsin), 2026. https://digital.library.wisc.edu/1793/98220.

Chicago

Nemati, Ali. 2026. “ATOMIC CLAIM VERIFICATION FOR CLINICAL DIAGNOSTIC REASONING: DECOMPOSING AND VALIDATING LLM REASONING STEPS AGAINST MEDICAL EVIDENCE.” Minds at UW (University of Wisconsin). https://digital.library.wisc.edu/1793/98220.

Harvard

Nemati, A. (2026) ‘ATOMIC CLAIM VERIFICATION FOR CLINICAL DIAGNOSTIC REASONING: DECOMPOSING AND VALIDATING LLM REASONING STEPS AGAINST MEDICAL EVIDENCE’, Minds at UW (University of Wisconsin). Available at: https://digital.library.wisc.edu/1793/98220.

IEEE

A. Nemati, “ATOMIC CLAIM VERIFICATION FOR CLINICAL DIAGNOSTIC REASONING: DECOMPOSING AND VALIDATING LLM REASONING STEPS AGAINST MEDICAL EVIDENCE,” Minds at UW (University of Wisconsin), 2026.

BibTeX

@phdthesis{nemati2026atomic,
  title     = {{ATOMIC CLAIM VERIFICATION FOR CLINICAL DIAGNOSTIC REASONING: DECOMPOSING AND VALIDATING LLM REASONING STEPS AGAINST MEDICAL EVIDENCE}},
  author    = {Nemati, Ali},
  year      = {2026},
  url       = {https://digital.library.wisc.edu/1793/98220}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education

  • adversarial prompts
  • Qwen2.5
  • diagnostic support tool

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Nemati, A. (2026). ATOMIC CLAIM VERIFICATION FOR CLINICAL DIAGNOSTIC REASONING: DECOMPOSING AND VALIDATING LLM REASONING STEPS AGAINST MEDICAL EVIDENCE. Minds at UW (University of Wisconsin). https://digital.library.wisc.edu/1793/98220

How many times has this paper been cited?

0 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Minds at UW (University of Wisconsin): sign in and open the PDF from this page.

Who wrote this paper?

Ali Nemati.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.