Review article · 2021 · Open access
Availability and reporting quality of external validations of machine-learning prediction models with orthopedic surgical outcomes: a systematic review
Acta Orthopaedica, 92(4), 385–393 · Published 18 April 2021 · DOI 10.1080/17453674.2021.1910448
In brief
“Availability and reporting quality of external validations of machine-learning prediction models with orthopedic surgical outcomes: a systematic review” is a 2021 review article by Olivier Quinten Groot and 11 co-authors, published in Acta Orthopaedica. OpenAlex counts 82 citations as of 20 November 2026, which puts it in the top 7% of works published in 2021 by citations. It is open access under a CC BY-NC licence: the full text is free to read at Acta Orthopaedica.
- Citations
- 82
- Most cited in
- 2024 (21)
- Field-weighted impact
- 1.89
- References
- 39
- Authors
- 12
Abstract
Background and purpose - External validation of machine learning (ML) prediction models is an essential step before clinical application. We assessed the proportion, performance, and transparent reporting of externally validated ML prediction models in orthopedic surgery, using the Transparent Reporting for Individual Prognosis or Diagnosis (TRIPOD) guidelines.Material and methods - We performed a systematic search using synonyms for every orthopedic specialty, ML, and external validation. The proportion was determined by using 59 ML prediction models with only internal validation in orthopedic surgical outcome published up until June 18, 2020, previously identified by our group. Model performance was evaluated using discrimination, calibration, and decision-curve analysis. The TRIPOD guidelines assessed transparent reporting.Results - We included 18 studies externally validating 10 different ML prediction models of the 59 available ML models after screening 4,682 studies. All external validations identified in this review retained good discrimination. Other key performance measures were provided in only 3 studies, rendering overall performance evaluation difficult. The overall median TRIPOD completeness was 61% (IQR 43-89), with 6 items being reported in less than 4/18 of the studies.Interpretation - Most current predictive ML models are not externally validated. The 18 available external validation studies were characterized by incomplete reporting of performance measures, limiting a transparent examination of model performance. Further prospective studies are needed to validate or refute the myriad of predictive ML models in orthopedics while adhering to existing guidelines. This ensures clinicians can take full advantage of validated and clinically implementable ML decision tools.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Groot, O. Q., Bindels, B. J. J., Ogink, P. T., Kapoor, N. D., Twining, P. K., Collins, A. K., Bongers, M. E. R., Lans, A., Oosterhoff, J. H. F., Karhade, A. V., Verlaan, J., & Schwab, J. H. (2021). Availability and reporting quality of external validations of machine-learning prediction models with orthopedic surgical outcomes: a systematic review. Acta Orthopaedica, 92(4), 385–393. https://doi.org/10.1080/17453674.2021.1910448
MLA 9
Groot, Olivier Quinten, et al. “Availability and reporting quality of external validations of machine-learning prediction models with orthopedic surgical outcomes: a systematic review.” Acta Orthopaedica, vol. 92, no. 4, 2021, pp. 385–393. https://doi.org/10.1080/17453674.2021.1910448.
Chicago
Groot, Olivier Quinten, Bas J. J. Bindels, Paul T. Ogink, Neal D. Kapoor, Peter K Twining, Austin Keith Collins, Michiel E. R. Bongers, et al. 2021. “Availability and reporting quality of external validations of machine-learning prediction models with orthopedic surgical outcomes: a systematic review.” Acta Orthopaedica 92 (4): 385–393. https://doi.org/10.1080/17453674.2021.1910448.
Harvard
Groot, O. Q. et al. (2021) ‘Availability and reporting quality of external validations of machine-learning prediction models with orthopedic surgical outcomes: a systematic review’, Acta Orthopaedica, 92(4), pp. 385–393. doi:10.1080/17453674.2021.1910448.
IEEE
O. Q. Groot et al., “Availability and reporting quality of external validations of machine-learning prediction models with orthopedic surgical outcomes: a systematic review,” Acta Orthopaedica, vol. 92, no. 4, pp. 385–393, 2021, doi: 10.1080/17453674.2021.1910448.
BibTeX
@article{groot2021availability,
title = {{Availability and reporting quality of external validations of machine-learning prediction models with orthopedic surgical outcomes: a systematic review}},
author = {Groot, Olivier Quinten and Bindels, Bas J. J. and Ogink, Paul T. and Kapoor, Neal D. and Twining, Peter K and Collins, Austin Keith and Bongers, Michiel E. R. and Lans, Amanda and Oosterhoff, Jacobien H. F. and Karhade, Aditya V. and Verlaan, Jorrit‐Jan and Schwab, Joseph Hasbrouck},
journal = {Acta Orthopaedica},
year = {2021},
volume = {92},
number = {4},
pages = {385--393},
doi = {10.1080/17453674.2021.1910448},
url = {https://doi.org/10.1080/17453674.2021.1910448}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2021 | 2 | |
| 2022 | 14 | |
| 2023 | 13 | |
| 2024 | 21 | |
| 2025 | 13 | |
| 2026 | 19 |
Its field-weighted citation impact is 1.89: it has been cited 1.89 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- external validation
- machine learning prediction models
- orthopedic surgery
- surgical outcomes
- TRIPOD guidelines
- reporting quality
- model discrimination
- model calibration
Work with this paper
Most-cited papers in Mathematics
- Random ForestsLeo Breiman · Machine Learning · 2001 · 142,830 citations
- Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple TestingYoav Benjamini, Yosef Hochberg · Journal of the Royal Statistical Society Series B (Statistical Methodology) · 1995 · 113,923 citations
- Fitting Linear Mixed-Effects Models Using lme4Douglas M. Bates, Martin Maechler, Benjamin M. Bolker and 1 more · Journal of Statistical Software · 2015 · 100,328 citations
- The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations.Reuben M. Baron, David A. Kenny · Journal of Personality and Social Psychology · 1986 · 95,866 citations
- Using multivariate statisticsBarbara Gerson Tabachnick, Linda S. Fidell · 1983 · 79,867 citations
Questions about this paper
How do I cite this paper in APA style?
Groot, O. Q., Bindels, B. J. J., Ogink, P. T., Kapoor, N. D., Twining, P. K., Collins, A. K., Bongers, M. E. R., Lans, A., Oosterhoff, J. H. F., Karhade, A. V., Verlaan, J., & Schwab, J. H. (2021). Availability and reporting quality of external validations of machine-learning prediction models with orthopedic surgical outcomes: a systematic review. Acta Orthopaedica, 92(4), 385–393. https://doi.org/10.1080/17453674.2021.1910448
How many times has this paper been cited?
82 times, according to OpenAlex on 20 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Acta Orthopaedica: sign in and open the PDF from this page.
Who wrote this paper?
Olivier Quinten Groot, Bas J. J. Bindels, Paul T. Ogink, Neal D. Kapoor, Peter K Twining, Austin Keith Collins, Michiel E. R. Bongers, Amanda Lans, Jacobien H. F. Oosterhoff, Aditya V. Karhade, Jorrit‐Jan Verlaan, Joseph Hasbrouck Schwab.
What is the DOI of this paper?
10.1080/17453674.2021.1910448, permanently at https://doi.org/10.1080/17453674.2021.1910448. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 20 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.