Journal article · 2023 · Open access
Performance metrics for models designed to predict treatment effect
BMC Medical Research Methodology, 23(1), 165 · Published 8 July 2023 · DOI 10.1186/s12874-023-01974-w
In brief
“Performance metrics for models designed to predict treatment effect” is a 2023 journal article by Carolien C. H. M. Maas and 5 co-authors, published in BMC Medical Research Methodology. OpenAlex counts 26 citations as of 15 November 2026, which puts it in the top 3% of works published in 2023 by citations. It is open access under a CC BY licence: the full text is free to read at BMC Medical Research Methodology.
- Citations
- 26
- Most cited in
- 2026 (10)
- Field-weighted impact
- 3.87
- References
- 25
- Authors
- 6
Abstract
Background Measuring the performance of models that predict individualized treatment effect is challenging because the outcomes of two alternative treatments are inherently unobservable in one patient. The C-for-benefit was proposed to measure discriminative ability. However, measures of calibration and overall performance are still lacking. We aimed to propose metrics of calibration and overall performance for models predicting treatment effect in randomized clinical trials (RCTs). Methods Similar to the previously proposed C-for-benefit, we defined observed pairwise treatment effect as the difference between outcomes in pairs of matched patients with different treatment assignment. We match each untreated patient with the nearest treated patient based on the Mahalanobis distance between patient characteristics. Then, we define the E avg -for-benefit, E 50 -for-benefit, and E 90 -for-benefit as the average, median, and 90 th quantile of the absolute distance between the predicted pairwise treatment effects and local-regression-smoothed observed pairwise treatment effects. Furthermore, we define the cross-entropy-for-benefit and Brier-for-benefit as the logarithmic and average squared distance between predicted and observed pairwise treatment effects. In a simulation study, the metric values of deliberately “perturbed models” were compared to those of the data-generating model, i.e., “optimal model”. To illustrate these performance metrics, different modeling approaches for predicting treatment effect are applied to the data of the Diabetes Prevention Program: 1) a risk modelling approach with restricted cubic splines; 2) an effect modelling approach including penalized treatment interactions; and 3) the causal forest. Results As desired, performance metric values of “perturbed models” were consistently worse than those of the “optimal model” (E avg -for-benefit ≥ 0.043 versus 0.002, E 50 -for-benefit ≥ 0.032 versus 0.001, E 90 -for-benefit ≥ 0.084 versus 0.004, cross-entropy-for-benefit ≥ 0.765 versus 0.750, Brier-for-benefit ≥ 0.220 versus 0.218). Calibration, discriminative ability, and overall performance of three different models were similar in the case study. The proposed metrics were implemented in a publicly available R-package “HTEPredictionMetrics”. Conclusion The proposed metrics are useful to assess the calibration and overall performance of models predicting treatment effect in RCTs.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Maas, C. C. H. M., Kent, D. M., Hughes, M. C., Dekker, R., Lingsma, H. F., & Klaveren, D. V. (2023). Performance metrics for models designed to predict treatment effect. BMC Medical Research Methodology, 23(1), 165. https://doi.org/10.1186/s12874-023-01974-w
MLA 9
Maas, Carolien C. H. M., et al. “Performance metrics for models designed to predict treatment effect.” BMC Medical Research Methodology, vol. 23, no. 1, 2023, p. 165. https://doi.org/10.1186/s12874-023-01974-w.
Chicago
Maas, Carolien C. H. M., David M. Kent, Michael C. Hughes, R. Dekker, Hester Floor Lingsma, and David V. Klaveren. 2023. “Performance metrics for models designed to predict treatment effect.” BMC Medical Research Methodology 23 (1): 165. https://doi.org/10.1186/s12874-023-01974-w.
Harvard
Maas, C. C. H. M. et al. (2023) ‘Performance metrics for models designed to predict treatment effect’, BMC Medical Research Methodology, 23(1), p. 165. doi:10.1186/s12874-023-01974-w.
IEEE
C. C. H. M. Maas, D. M. Kent, M. C. Hughes, R. Dekker, H. F. Lingsma, and D. V. Klaveren, “Performance metrics for models designed to predict treatment effect,” BMC Medical Research Methodology, vol. 23, no. 1, p. 165, 2023, doi: 10.1186/s12874-023-01974-w.
BibTeX
@article{maas2023performance,
title = {{Performance metrics for models designed to predict treatment effect}},
author = {Maas, Carolien C. H. M. and Kent, David M. and Hughes, Michael C. and Dekker, R. and Lingsma, Hester Floor and Klaveren, David V.},
journal = {BMC Medical Research Methodology},
year = {2023},
volume = {23},
number = {1},
pages = {165},
doi = {10.1186/s12874-023-01974-w},
url = {https://doi.org/10.1186/s12874-023-01974-w}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2023 | 5 | |
| 2024 | 6 | |
| 2025 | 5 | |
| 2026 | 10 |
Its field-weighted citation impact is 3.87: it has been cited 3.87 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- treatment response prediction
- performance indicators
- calibration
- discriminatory power
- randomized controlled trials
- causal inference
- causal forest
- restricted cubic splines
Work with this paper
Most-cited papers in Mathematics
- Random ForestsLeo Breiman · Machine Learning · 2001 · 142,830 citations
- Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple TestingYoav Benjamini, Yosef Hochberg · Journal of the Royal Statistical Society Series B (Statistical Methodology) · 1995 · 113,923 citations
- Fitting Linear Mixed-Effects Models Using lme4Douglas M. Bates, Martin Maechler, Benjamin M. Bolker and 1 more · Journal of Statistical Software · 2015 · 100,328 citations
- The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations.Reuben M. Baron, David A. Kenny · Journal of Personality and Social Psychology · 1986 · 95,866 citations
- Using multivariate statisticsBarbara Gerson Tabachnick, Linda S. Fidell · 1983 · 79,867 citations
Questions about this paper
How do I cite this paper in APA style?
Maas, C. C. H. M., Kent, D. M., Hughes, M. C., Dekker, R., Lingsma, H. F., & Klaveren, D. V. (2023). Performance metrics for models designed to predict treatment effect. BMC Medical Research Methodology, 23(1), 165. https://doi.org/10.1186/s12874-023-01974-w
How many times has this paper been cited?
26 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access: read the free full text at BMC Medical Research Methodology.
Who wrote this paper?
Carolien C. H. M. Maas, David M. Kent, Michael C. Hughes, R. Dekker, Hester Floor Lingsma, David V. Klaveren.
What is the DOI of this paper?
10.1186/s12874-023-01974-w. The permanent link is https://doi.org/10.1186/s12874-023-01974-w.
Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.