Journal article · 2026 · Open access
Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study
Journal of Medical Internet Research, 28, e93328 · Published 26 October 2026 · DOI 10.2196/93328
In brief
“Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study” is a 2026 journal article by Clara Frydman-Gani and 9 co-authors, published in Journal of Medical Internet Research. OpenAlex counts 3 citations as of 17 November 2026, which puts it in the top 4% of works published in 2026 by citations. It is open access under a CC BY licence: the full text is free to read at Journal of Medical Internet Research.
- Citations
- 3
- Most cited in
- 2026 (3)
- Field-weighted impact
- 4.19
- References
- 50
- Authors
- 10
Abstract
Background The accurate detection of clinical symptoms in electronic health records (EHRs) is pivotal for advancing large-scale genetic and longitudinal studies in psychiatry. Free-text clinical notes are an essential source of symptom-level information, particularly in psychiatry; however, the automated extraction of symptoms from clinical text remains challenging. Objective We aimed to evaluate the ability of open-source generative large language models (LLMs) to detect psychiatric symptom-level concepts from clinical text in Spanish and to develop a new fine-tuned, open-source LLM that can be shared without risk of exposing patient information. Methods We tested 11 open-source generative LLMs for their ability to detect 109 psychiatric concepts from clinical text, using annotated EHR notes from a psychiatric hospital in Colombia. The LLMs were evaluated both “out-of-the-box” and after fine-tuning and compared against a traditional natural language processing (tNLP) method developed from the same data. To develop a fine-tuned LLM that can be shared without risk of exposing patient information, we created a novel synthetic dataset, leveraging real concept annotations while generating fully synthetic clinical text, and fine-tuned a top-performing LLM on these data. Fine-tuned model generalizability was assessed through cross-site evaluation using data from a second independent hospital in Colombia. Results We show that while base LLM performance was poor to moderate (zero-shot: 0.2-0.6 macro-F1; few-shot: 0.2-0.74 macro-F1), it improved significantly after fine-tuning (0.75-0.86 macro-F1), with several fine-tuned LLMs outperforming the tNLP method. In total, 100 psychiatric concepts could be reliably detected (F1>0.8) using either a fine-tuned LLM or tNLP. Using the synthetic dataset, we developed Mistral-small-psych, a new LLM capable of detecting psychiatric concepts from Spanish clinical text with performance comparable to models trained on real EHR data (macro-F1=0.79) while remaining fully distributable. In cross-site evaluations, most LLMs generalized well, with small out-of-domain performance losses (macro-F1 decrease of 0.02-0.16). Conclusions This work provides the first large-scale, cross-institutional evaluation of state-of-the-art LLMs for psychiatric concept detection in Spanish EHRs, introduces a privacy-preserving pathway for building shareable clinical natural language processing models, and presents a new open model for extracting symptom-level psychiatric information from Spanish clinical text.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Frydman-Gani, C., Arias, A., Vallejo, M. P., Martínez, J. D. L., Valencia‐Echeverry, J., Castano, M., Bui, A. A. T., Freimer, N. B., López‐Jaramillo, C., & Loohuis, L. M. O. (2026). Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study. Journal of Medical Internet Research, 28, e93328. https://doi.org/10.2196/93328
MLA 9
Frydman-Gani, Clara, et al. “Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study.” Journal of Medical Internet Research, vol. 28, 2026, p. e93328. https://doi.org/10.2196/93328.
Chicago
Frydman-Gani, Clara, Alejandro Arias, Maria Perez Vallejo, John Daniel Londoño Martínez, Johanna Valencia‐Echeverry, Mauricio Castano, Alex A. T. Bui, Nelson B. Freimer, Carlos López‐Jaramillo, and Loes M. Olde Loohuis. 2026. “Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study.” Journal of Medical Internet Research 28: e93328. https://doi.org/10.2196/93328.
Harvard
Frydman-Gani, C. et al. (2026) ‘Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study’, Journal of Medical Internet Research, 28, p. e93328. doi:10.2196/93328.
IEEE
C. Frydman-Gani et al., “Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study,” Journal of Medical Internet Research, vol. 28, p. e93328, 2026, doi: 10.2196/93328.
BibTeX
@article{frydmangani2026generative,
title = {{Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study}},
author = {Frydman-Gani, Clara and Arias, Alejandro and Vallejo, Maria Perez and Martínez, John Daniel Londoño and Valencia‐Echeverry, Johanna and Castano, Mauricio and Bui, Alex A. T. and Freimer, Nelson B. and López‐Jaramillo, Carlos and Loohuis, Loes M. Olde},
journal = {Journal of Medical Internet Research},
year = {2026},
volume = {28},
pages = {e93328},
doi = {10.2196/93328},
url = {https://doi.org/10.2196/93328}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- clinical text
- electronic health records
- generative large language models
- cross-site generalization
- clinical natural language processing
- Colombia
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Frydman-Gani, C., Arias, A., Vallejo, M. P., Martínez, J. D. L., Valencia‐Echeverry, J., Castano, M., Bui, A. A. T., Freimer, N. B., López‐Jaramillo, C., & Loohuis, L. M. O. (2026). Generative Large Language Models for Psychiatric Concept Detection in Spanish-Language Electronic Health Records: Development and Evaluation Study. Journal of Medical Internet Research, 28, e93328. https://doi.org/10.2196/93328
How many times has this paper been cited?
3 times, according to OpenAlex on 17 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Journal of Medical Internet Research: sign in and open the full text from this page.
Who wrote this paper?
Clara Frydman-Gani, Alejandro Arias, Maria Perez Vallejo, John Daniel Londoño Martínez, Johanna Valencia‐Echeverry, Mauricio Castano, Alex A. T. Bui, Nelson B. Freimer, Carlos López‐Jaramillo, Loes M. Olde Loohuis.
What is the DOI of this paper?
10.2196/93328, permanently at https://doi.org/10.2196/93328. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 17 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.