Journal article · 2026 · Open access
Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study
Journal of Medical Internet Research, 28, e100531 · Published 27 October 2026 · DOI 10.2196/100531
In brief
“Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study” is a 2026 journal article by Erik Cornelisse and 39 co-authors, published in Journal of Medical Internet Research. OpenAlex counts 1 citations as of 17 November 2026, which puts it in the top 9% of works published in 2026 by citations. It is open access under a CC BY licence: the full text is free to read at Journal of Medical Internet Research.
- Citations
- 1
- Most cited in
- 2026 (1)
- Field-weighted impact
- 1.20
- References
- 92
- Authors
- 40
Abstract
Background: The rapid adoption of generative AI in the community and health care necessitates reliable, human-led evaluation. Objective: This study aimed to develop a standardized rubric to support health care professionals, administrators, and researchers in evaluating generative AI outputs in health contexts. Methods: A rubric was developed through a 4-stage mixed methods study: literature synthesis, expert refinement, pilot application, and post-pilot feedback. The rubric was informed by 69 publications and refined through iterative feedback from a multidisciplinary expert panel (n=23). The pilot rubric was applied to 4 health care scenarios: responses to health-related Reddit questions, generation of health-related images and videos, responses to health professional continuing education questions, and automated physician-to-physician clinical documentation generation. Via these scenarios, 15 clinicians (medical practitioners, pharmacists, and medical oncologists) evaluated 250 input-output pairs using domain-level ratings and qualitative comments. The evaluated outputs included 200 AI-generated responses across the 4 health care scenarios and 50 physician-tagged replies in the health-related Reddit questions scenario. Results: The final rubric, OMNI, comprised 10 evaluation domains (Accuracy, Safety, Recognizes Uncertainty, Completeness, Communicates Clearly, Empathetic, Context Aware, Referencing, Follows Instructions, and Avoids Coerciveness) assessed on 4-point Likert scales, supported by detailed instructional and example materials. Via the 4 pilot scenarios, assessors generated 9600 domain-level ratings and 4010 qualitative comments. The mean pairwise agreement for the 4-point Likert groups ranged from 30.8% to 53.4%, increasing to 59.3% to 79.6% when collapsed into broad agreement categories; Gwet's AC2 ranged from 0.49 to 0.80. OMNI differentiated strengths and weaknesses between models within scenarios, while recurrent limitations in Referencing, Recognizes Uncertainty, Completeness, Empathetic, and Accuracy were identified across scenarios. Multimodal image and video outputs had the highest occurrence of negative rating domains. Assessors reported OMNI was usable and indicated they would reuse it. Conclusions: OMNI is a standardized, accessible rubric for expert-led evaluation of generative AI outputs in health contexts. Pilot application demonstrated that OMNI could differentiate AI performance across models and domains while generating structured feedback. Further work is needed to assess its applicability to purpose-built health AI tools, agentic systems, and organizational evaluation settings.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Cornelisse, E., Kousa, D., Menz, B. D., Murphy-Reuter, B., Shahnam, A., Chin-Yee, B. H., Calabro, R., Li, L. X., Ramsey, I., Abuhelwa, A. Y., Logan, J. M., Ng, H. S., Aravinthan, M., Goonan, J., Carthew, H., Molyneux, P. D., Tang, J., Hitchen, N., Hong, M., . . . Hopkins, A. M. (2026). Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study. Journal of Medical Internet Research, 28, e100531. https://doi.org/10.2196/100531
MLA 9
Cornelisse, Erik, et al. “Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study.” Journal of Medical Internet Research, vol. 28, 2026, p. e100531. https://doi.org/10.2196/100531.
Chicago
Cornelisse, Erik, Dib Kousa, Bradley D. Menz, Brinleigh Murphy-Reuter, Adel Shahnam, Benjamin Hull Chin-Yee, Ryan Calabro, et al. 2026. “Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study.” Journal of Medical Internet Research 28: e100531. https://doi.org/10.2196/100531.
Harvard
Cornelisse, E. et al. (2026) ‘Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study’, Journal of Medical Internet Research, 28, p. e100531. doi:10.2196/100531.
IEEE
E. Cornelisse et al., “Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study,” Journal of Medical Internet Research, vol. 28, p. e100531, 2026, doi: 10.2196/100531.
BibTeX
@article{cornelisse2026development,
title = {{Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study}},
author = {Cornelisse, Erik and Kousa, Dib and Menz, Bradley D. and Murphy-Reuter, Brinleigh and Shahnam, Adel and Chin-Yee, Benjamin Hull and Calabro, Ryan and Li, Lee Xingzhuo and Ramsey, Imogen and Abuhelwa, Ahmad Y. and Logan, Jessica M. and Ng, Huah Shin and Aravinthan, Mythily and Goonan, James and Carthew, Hannah and Molyneux, Paul David and Tang, Joanne and Hitchen, Nadia and Hong, Martin and Gandhi, Jimit and Fontanarosa, Stephanie and Scarfo, Nicholas L. and Daniel, Kezia and Marotti, Sally B. and Mariadas, Joy and van Diermen, Bianca and Manickam, Divya Joyce Susai and Ruanglertboon, Warit and Postma, David and McCradden, Melissa D. and others},
journal = {Journal of Medical Internet Research},
year = {2026},
volume = {28},
pages = {e100531},
doi = {10.2196/100531},
url = {https://doi.org/10.2196/100531}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- generative AI evaluation
- health care AI
- AI-generated health information
- interobserver agreement
- AI accuracy
- artificial empathy
Work with this paper
Most-cited papers in Medicine
- Yaşlı Kadınlarda Üreme SağlığıDemet Kokanalı, Yaprak Engin Üstün · Jinekoloji-Obstetrik ve Neonatoloji Tıp Dergisi · 2019 · 169,630 citations
- Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 CountriesHyuna Sung, Adalberto M. Filho, Mathieu Laversanne and 5 more · CA A Cancer Journal for Clinicians · 2021 · 119,885 citations
- “Mini-mental state”Marshal F. Folstein, Susan E. Folstein, Paul R. McHugh · Journal of Psychiatric Research · 1975 · 90,424 citations
- Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countriesFreddie Bray, Jacques Ferlay, Isabelle Soerjomataram and 3 more · CA A Cancer Journal for Clinicians · 2018 · 87,396 citations
- Hallmarks of Cancer: The Next GenerationDouglas Hanahan, Robert A. Weinberg · Cell · 2011 · 69,758 citations
Questions about this paper
How do I cite this paper in APA style?
Cornelisse, E., Kousa, D., Menz, B. D., Murphy-Reuter, B., Shahnam, A., Chin-Yee, B. H., Calabro, R., Li, L. X., Ramsey, I., Abuhelwa, A. Y., Logan, J. M., Ng, H. S., Aravinthan, M., Goonan, J., Carthew, H., Molyneux, P. D., Tang, J., Hitchen, N., Hong, M., . . . Hopkins, A. M. (2026). Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study. Journal of Medical Internet Research, 28, e100531. https://doi.org/10.2196/100531
How many times has this paper been cited?
1 times, according to OpenAlex on 17 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Journal of Medical Internet Research: sign in and open the full text from this page.
Who wrote this paper?
Erik Cornelisse, Dib Kousa, Bradley D. Menz, Brinleigh Murphy-Reuter, Adel Shahnam, Benjamin Hull Chin-Yee, Ryan Calabro, Lee Xingzhuo Li, Imogen Ramsey, Ahmad Y. Abuhelwa, Jessica M. Logan, Huah Shin Ng and 28 more.
What is the DOI of this paper?
10.2196/100531, permanently at https://doi.org/10.2196/100531. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 17 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.