Abstract
Methods for identifying heterogeneity of treatment effects in randomized trials have seen recent advances, yet applying these methods to health services intervention trials has not been well investigated. Our objective was to compare two approaches—predictive risk modeling and model-based recursive partitioning—for identifying subgroups of trial participants with potentially differential response to an intervention involving health risk assessment completion alone (n = 192) versus health risk assessment completion plus telephone-delivered health coaching (n = 173). Notably, these approaches have been developed by investigators from distinct disciplines and reported in separate literatures and have generally not been compared in prior work. Furthermore, these methods approach subgroup identification differently and answer related but slightly different questions. The primary outcome for both approaches was prevention health program enrollment by six months. The predictive risk model was developed in two steps, where, first, a single risk score was derived from a logistic regression model with 12 a priori chosen covariates by the scientific investigator team (c-statistic = 0.63). Then, the treatment effect was calculated within quartiles of risk via interaction in a logistic regression model (c-statistic = 0.69; c-for-benefit = 0.43). The greatest treatment effect was in the second quartile, in which 54% (22 of 41) of intervention patients and 10% (5 of 50) of control patients reported prevention program enrollment. In contrast, with the data-driven approach of model-based recursive partitioning, all 28 baseline covariates were considered, with the algorithm selecting covariates and optimal split points. Final model results had a c-statistic of 0.69 and a c-for-benefit of 0.55 (optimism-corrected c-statistic = 0.62 and c-for-benefit = 0.53) and identified 4 subgroups, with the greatest treatment effect among patients with lower mean numeracy, education less than a bachelor’s degree, and diabetes, in which 54% (15 of 28) of intervention patients reported prevention program enrollment versus 7% (3 of 41) of control patients. While there is increasing interest in discovering heterogeneity of treatment effects, our analyses highlight the important differences between these approaches, both from questions answered, model development, and results obtained. Specifying goals of treatment heterogeneity analyses, choosing the appropriate method to best address the goals, and external validation of results are important steps when applying these methods.
Clinicaltrials.gov identifier: NCT01828567
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Alemayehu, D., Chen, Y., Markatou, M.: A comparative study of subgroup identification methods for differential treatment effect: performance metrics and recommendations. Stat Methods Med. Res. 27(12), 3658–3678 (2018)
Alosh, M., Huque, M.F., Bretz, F., D’Agostino, R.B., Sr.: Tutorial on statistical considerations on subgroup analysis in confirmatory clinical trials. Stat. Med. 36(8), 1334–1360 (2017)
Baum, A., Scarpa, J., Bruzelius, E., Tamler, R., Basu, S., Faghmous, J.: Targeting weight loss interventions to reduce cardiovascular complications of type 2 diabetes: a machine learning-based post-hoc analysis of heterogeneous treatment effects in the Look AHEAD trial. The Lancet. Diabetes and Endocrinol. 5(10), 808–815 (2017)
D’Agostino, R.B., Sr., Vasan, R.S., Pencina, M.J., Wolf, P.A., Cobain, M., Massaro, J.M., Kannel, W.B.: General cardiovascular risk profile for use in primary care: the framingham heart study. Circulation 117(6), 743–753 (2008)
Doove, L.L., Dusseldorp, E., Van Deun, K., Van Mechelen, I.: A comparison of five recursive partitioning methods to find person subgroups involved in meaningful treatment–subgroup interactions. Adv. Data Anal. Classif. 8(4), 403–425 (2014)
Dusseldorp, E., Conversano, C., Jan Van Os, B.: Combining an additive and tree-based regression model simultaneously: STIMA. J. Comput. Gr. Stat. 19(3), 514–530 (2010)
Dusseldorp, E., Conversano, C., Van Os, B.J.: Combining an additive and tree-based regression model simultaneously: STIMA. J. Comput. Gr. Stat. 19(3), 514–530 (2010)
Dusseldorp, E., Doove, L., Mechelen, I.: Quint: an R package for the identification of subgroups of clients who differ in which treatment alternative is best for them. Behav. Res. Methods. 48(2), 650–663 (2016)
Harrell, F.E., Jr., Lee, K.L., Mark, D.B.: Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat. Med. 15(4), 361–387 (1996)
Hays, R.D., Stewart, A.L.: Sleep Measures. In: Stewart, A.L., Ware, J.E.J. (eds.) Measuring functioning and well-being; the medical outcomes study approach, pp. 235–259. Duke University Press, Durham (1992)
Hayward, R.A., Kent, D.M., Vijan, S., Hofer, T.P.: Reporting clinical trial results to inform providers, payers, and consumers. Health Aff. (Project Hope) 24, 1571–1581 (2005)
Hibbard, J.H., Mahoney, E.R., Stock, R., Tusler, M.: Do increases in patient activation result in improved self-management behaviors? Health Serv. Res. 42(4), 1443–1463 (2007)
Hibbard, J.H., Mahoney, E.R., Stockard, J., Tusler, M.: Development and testing of a short form of the patient activation measure. Health Serv. Res. 40(6), 1918–1930 (2005)
Hothorn, T., Zeileis, A.: partykit: a modular toolkit for recursive partytioning in R. J. Mach. Learn. Res. 16, 3905–3909 (2015)
Kent, D.M., Hayward, R.A.: Limitations of applying summary results of clinical trials to individual patients: the need for risk stratification. JAMA 298(10), 1209–1212 (2007)
Kent, D.M., Rothwell, P.M., Ioannidis, J.P., Altman, D.G., Hayward, R.A.: Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal. Trials 11, 85 (2010). https://doi.org/10.1186/1745-6215-11-85
Kent, D.M., Ruthazer, R., Selker, H.P.: Are some patients likely to benefit from recombinant tissue-type plasminogen activator for acute ischemic stroke even beyond 3 hours from symptom onset? Stroke 34(2), 464–467 (2003)
Kravitz, R.L., Duan, N., Braslow, J.: Evidence-based medicine, heterogeneity of treatment effects, and the trouble with averages. Milbank Q 82(4), 661–687 (2004). https://doi.org/10.1111/j.0887-378X.2004.00327.x
Kroenke, K., Spitzer, R.L., Williams, J.B.: The PHQ-9: validity of a brief depression severity measure. J. Gen. Intern. Med. 16(9), 606–613 (2001)
Lagakos, S.W.: The challenge of subgroup analyses–reporting without distorting. N. Engl. J. Med. 354(16), 1667–1669 (2006)
Lipkovich, I., Dmitrienko, A., Agostino, B.R.D.S.: Tutorial in biostatistics: data-driven subgroup identification and analysis in clinical trials. Stat. Med. 36(1), 136–196 (2017)
Loh, W.Y., He, X., Man, M.: A regression tree approach to identifying subgroups with differential treatment effects. Stat. Med. 34(11), 1818–1833 (2015)
McNaughton, C.D., Cavanaugh, K.L., Kripalani, S., Rothman, R.L., Wallston, K.A.: Validation of a short, 3-item version of the subjective numeracy scale. Med. Decis. Making 35(8), 932–936 (2015)
McVay, M.A., Yancy, W.S., Jr., Vijan, S., Van Scoyoc, L., Neelon, B., Voils, C.I., Maciejewski, M.L.: Obesity-related health status changes and weight-loss treatment utilization. Am J Prev Med 46(5), 465–472 (2014)
MyHealtheVet: HealtheLiving Assessment. https://www.myhealth.va.gov/mhv-portal-web/web/myhealthevet/ss20170509-birds-eye-view-of-your-wellness-and-your-health-risks (2018). Accessed 28 November, 2019
Nouri, S.S., Damschroder, L.J., Olsen, M.K., Gierisch, J.M., Fagerlin, A., Sanders, L.L., McCant, F., Oddone, E.Z.: Health coaching has differential effects on veterans with limited health literacy and numeracy: a secondary analysis of ACTIVATE. J. Gen. Intern. Med. 34(4), 552–558 (2019)
Oddone, E.Z., Gierisch, J.M., Sanders, L.L., Fagerlin, A., Sparks, J., McCant, F., May, C., Olsen, M.K., Damschroder, L.J.: A coaching by telephone intervention on engaging patients to address modifiable cardiovascular risk factors: a randomized controlled trial. J. Gen. Intern. Med. 33(9), 1487–1494 (2018)
Rothwell, P.M.: Treating individuals Subgroup analysis in randomised controlled trials: importance, indications, and interpretation. Lancet 365(9454), 176–186 (2005). https://doi.org/10.1016/S0140-6736(05)17709-5
Rothwell, P.M., Warlow, C.P.: Prediction of benefit from carotid endarterectomy in individual patients: a risk-modelling study. European carotid surgery trialists’ collaborative group. Lancet 353(9170), 2105–2110 (1999)
Seibold, H., Zeileis, A., Hothorn, T.: Model-based recursive partitioning for subgroup analyses. Int. J. Biostat. 12(1), 45–63 (2016)
Sies, A., Demyttenaere, K., Van Mechelen, I.: Studying treatment-effect heterogeneity in precision medicine through induced subgroups. J. Biopharm. Stat. 29(3), 491–507 (2019)
Strobl, C., Malley, J., Tutz, G.: An introduction to recursive partitioning: rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychol. Methods 14(4), 323–348 (2009). https://doi.org/10.1037/a0016973
Sussman, J.B., Kent, D.M., Nelson, J.P., Hayward, R.A.: Improving diabetes prevention with benefit based tailored treatment: risk based reanalysis of diabetes prevention program. BMJ 350, h454 (2015). https://doi.org/10.1136/bmj.h454
van Klaveren, D., Steyerberg, E.W., Serruys, P.W., Kent, D.M.: The proposed ‘concordance-statistic for benefit’provided a useful metric when modeling heterogeneous treatment effects. J. Clin. Epidemiol. 94, 59–68 (2018)
Zeileis, A., Hothorn, T., Hornik, K.: Model-based recursive partitioning. J. Comput. Graph Stat. 17, 492–514 (2008)
Acknowledgements
This work was supported by HSR&D funding (CRE 12-306, RCS 10-391) and by the Durham Center of Innovation to Accelerate Discovery and Practice Transformation (ADAPT), (CIN 13-410) at the Durham VA Health Care System. The views represented in this article represent those of the authors and not those of the VA or the United States Government. We are grateful for helpful comments on an earlier draft from Drs. Daniel Almirall, Sandeep Vijan, and David Kent.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix 1
See Table 5 .
Appendix 2: Statistical code for MoB analyses
Part 1: R code
Part 2. SAS Code for c-for-benefit Predicted Risk Model (PRM)(PREDICTED_EVRENR is from step 1 of the PRM model)
Rights and permissions
About this article
Cite this article
Olsen, M.K., Stechuchak, K.M., Oddone, E.Z. et al. Which patients benefit most from completing health risk assessments: comparing methods to identify heterogeneity of treatment effects. Health Serv Outcomes Res Method 21, 527–546 (2021). https://doi.org/10.1007/s10742-021-00243-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10742-021-00243-x