Journal article · 2026 · Open access

Real-World Use and Expert-Guided Optimization of a Large Language Model–Based Pediatric Health Assistant: Retrospective Observational Study

Qiuhong Wei, Xiao Li, Xin Zhang, Hong Wei, Ruijue Wang, Suqiao Ou, Chao Zhang, Huarong Huo, Yu Peng, Lishen Liu, Ping Song, Ziyu Hua and 1 more

Journal of Medical Internet Research, 28, e105346 · Published 4 November 2026 · DOI 10.2196/105346

Open full textCite this paper

In brief

“Real-World Use and Expert-Guided Optimization of a Large Language Model–Based Pediatric Health Assistant: Retrospective Observational Study” is a 2026 journal article by Qiuhong Wei and 12 co-authors, published in Journal of Medical Internet Research. OpenAlex counts 0 citations as of 17 November 2026. It is open access under a CC BY licence: the full text is free to read at Journal of Medical Internet Research.

Citations
0
References
39
Authors
13

Abstract

BACKGROUND: Pediatric care resources are limited and unevenly distributed in many health systems. Caregivers increasingly turn to large language models (LLMs) for children's health information, but real-world evidence on LLM use and response quality in general pediatric care is limited. OBJECTIVE: This study aimed to characterize real-world use, consultation needs, and response quality of Xiaoyi, an LLM-based pediatric health assistant, and to evaluate the impact of expert-guided optimization on response quality. METHODS: This study collected all consultation sessions submitted between April 7 and December 31, 2025, to Xiaoyi, a pediatric health assistant co-developed by Children's Hospital of Chongqing Medical University and Zuoyi Tech. Temporal patterns of consultation activity were characterized; all sessions were annotated for pediatric subspecialty, reason for consultation, and presenting symptoms through LLM-assisted labeling with pediatrician validation. A multidisciplinary panel of 40 pediatricians evaluated Xiaoyi's response quality over 2 rounds using predefined criteria established by expert consensus: an initial evaluation informed system optimization (prompt revision, knowledge-base and tool expansion, and validation rules) and a follow-up evaluation assessed its impact. In each round, 1000 sessions randomly sampled from the corresponding pre- or postoptimization period (2000 in total) were rated at the session level (overall quality and clinical appropriateness) and the turn level (6 predefined deficiency categories). Categorical ratings were compared using the chi-square test, and turn-level deficiency rates were compared using generalized estimating equations to account for within-session correlation. RESULTS: Caregivers of 8266 children (median age 4.2, IQR 1.3-8 years; 4705/8266, 56.9% male) from 31 of 34 provincial-level divisions in China initiated 16,074 sessions, of which 7046 (43.8%) occurred outside routine clinic hours (08:00-17:30). Queries spanned 31 subspecialties, most frequently child health care (2025/16,074, 12.6%), pulmonology (1903/16,074, 11.8%), and gastroenterology (1833/16,074, 11.4%). The leading reasons for consultation were symptom consultation (7845/16,074, 48.8%), medical report interpretation (2531/16,074, 15.7%), and triage guidance (2515/16,074, 15.6%); among symptom consultation sessions, fever (2719/16,074, 34.7%) and cough (2349/16,074, 29.9%) were most common. After expert-guided optimization, the number of sessions rated good (n=1000) in overall quality (the response fully resolving the caregiver's question with actionable information) increased from 652 (65.2%) to 743 (74.3%), and the number rated clinically appropriate increased from 614 (61.4%) to 874 (87.4%, both P<.001). The turn-level deficiency rate declined from 19.4% (790/4073 turns) to 6.9% (236/3419 turns; odds ratio 0.30, 95% CI 0.24-0.37; P<.001), with suboptimal clinical recommendations (not optimally tailored to the child's age or clinical context) remaining the most frequent category. CONCLUSIONS: This real-world study suggests that an LLM-based health consultation assistant may complement existing pediatric care, with expert-guided optimization…

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Wei, Q., Li, X., Zhang, X., Wei, H., Wang, R., Ou, S., Zhang, C., Huo, H., Peng, Y., Liu, L., Song, P., Hua, Z., & Xu, X. (2026). Real-World Use and Expert-Guided Optimization of a Large Language Model–Based Pediatric Health Assistant: Retrospective Observational Study. Journal of Medical Internet Research, 28, e105346. https://doi.org/10.2196/105346

MLA 9

Wei, Qiuhong, et al. “Real-World Use and Expert-Guided Optimization of a Large Language Model–Based Pediatric Health Assistant: Retrospective Observational Study.” Journal of Medical Internet Research, vol. 28, 2026, p. e105346. https://doi.org/10.2196/105346.

Chicago

Wei, Qiuhong, Xiao Li, Xin Zhang, Hong Wei, Ruijue Wang, Suqiao Ou, Chao Zhang, et al. 2026. “Real-World Use and Expert-Guided Optimization of a Large Language Model–Based Pediatric Health Assistant: Retrospective Observational Study.” Journal of Medical Internet Research 28: e105346. https://doi.org/10.2196/105346.

Harvard

Wei, Q. et al. (2026) ‘Real-World Use and Expert-Guided Optimization of a Large Language Model–Based Pediatric Health Assistant: Retrospective Observational Study’, Journal of Medical Internet Research, 28, p. e105346. doi:10.2196/105346.

IEEE

Q. Wei et al., “Real-World Use and Expert-Guided Optimization of a Large Language Model–Based Pediatric Health Assistant: Retrospective Observational Study,” Journal of Medical Internet Research, vol. 28, p. e105346, 2026, doi: 10.2196/105346.

BibTeX

@article{wei2026real,
  title     = {{Real-World Use and Expert-Guided Optimization of a Large Language Model–Based Pediatric Health Assistant: Retrospective Observational Study}},
  author    = {Wei, Qiuhong and Li, Xiao and Zhang, Xin and Wei, Hong and Wang, Ruijue and Ou, Suqiao and Zhang, Chao and Huo, Huarong and Peng, Yu and Liu, Lishen and Song, Ping and Hua, Ziyu and Xu, Ximing},
  journal   = {Journal of Medical Internet Research},
  year      = {2026},
  volume    = {28},
  pages     = {e105346},
  doi       = {10.2196/105346},
  url       = {https://doi.org/10.2196/105346}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education

  • large language models
  • response quality
  • knowledge base expansion
  • clinical appropriateness
  • consultation patterns
  • real-world evidence
  • China
  • AI hallucination

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Wei, Q., Li, X., Zhang, X., Wei, H., Wang, R., Ou, S., Zhang, C., Huo, H., Peng, Y., Liu, L., Song, P., Hua, Z., & Xu, X. (2026). Real-World Use and Expert-Guided Optimization of a Large Language Model–Based Pediatric Health Assistant: Retrospective Observational Study. Journal of Medical Internet Research, 28, e105346. https://doi.org/10.2196/105346

How many times has this paper been cited?

0 times, according to OpenAlex on 17 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Journal of Medical Internet Research: sign in and open the full text from this page.

Who wrote this paper?

Qiuhong Wei, Xiao Li, Xin Zhang, Hong Wei, Ruijue Wang, Suqiao Ou, Chao Zhang, Huarong Huo, Yu Peng, Lishen Liu, Ping Song, Ziyu Hua and 1 more.

What is the DOI of this paper?

10.2196/105346, permanently at https://doi.org/10.2196/105346. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 17 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.