Skip to main content
SpringerLink
Log in

Machine Learning Prediction of Clinical Trial Operational Efficiency

  • Research Article
  • Theme: Applications of Machine Learning and AI to Drug Discovery, Development, and Regulations
  • Published:
The AAPS Journal Aims and scope Submit manuscript

Abstract

Clinical trials are the gatekeepers and bottlenecks of progress in medicine. In recent years, they have become increasingly complex and expensive, driven by a growing number of stakeholders requiring more endpoints, more diverse patient populations, and a stringent regulatory environment. Trial designers have historically relied on investigator expertise and legacy norms established within sponsor companies to improve operational efficiency while achieving study goals. As such, data-driven forecasts of operational metrics can be a useful resource for trial design and planning. We develop a machine learning model to predict clinical trial operational efficiency using a novel dataset from Roche containing over 2,000 clinical trials across 20 years and multiple disease areas. The data includes important operational metrics related to patient recruitment and trial duration, as well as a variety of trial features such as the number of procedures, eligibility criteria, and endpoints. Our results demonstrate that operational efficiency can be predicted robustly using trial features, which can provide useful insights to trial designers on the potential impact of their decisions on patient recruitment success and trial duration.

Graphical Abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. I
Fig. II
Fig. III

Similar content being viewed by others

Code Availability

The trained machine learning model is available at https://github.com/kevinwu23/clinical-trial-efficiency.

REFERENCES

  1. Kelly D, Spreafico A, Siu LL. Increasing operational and scientific efficiency in clinical trials. Br J Cancer. 2020;123(8):1207–8.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Rosenblatt M. The large pharmaceutical company perspective. N Engl J Med. 2017;376(1):52–60.

    Article  PubMed  Google Scholar 

  3. Martin L, Hutchens M, Hawkins C, Radnov A. How much do clinical trials cost. Nat Rev Drug Discov. 2017;16(6):381–2.

    Article  CAS  PubMed  Google Scholar 

  4. Hwang TJ, Carpenter D, Lauffenburger JC, Wang B, Franklin JM, Kesselheim AS. Failure of investigational drugs in late-stage clinical development and publication of trial results. JAMA Intern Med. 2016;176(12):1826–33.

    Article  PubMed  Google Scholar 

  5. Wong CH, Siah KW, Lo AW. Estimation of clinical trial success rates and related parameters. Biostatistics. 2019;20(2):273–86.

    Article  PubMed  Google Scholar 

  6. Getz K. Improving protocol design feasibility to drive drug development economics and performance. Int J Environ Res Public Health. 2014;11(5):5069–80.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Kaitin KI. Deconstructing the drug development process: the new face of innovation. Clin Pharmacol Ther. 2010;87(3):356–61.

    Article  CAS  PubMed  Google Scholar 

  8. Ledford H. Translational research: 4 ways to fix the clinical trial. Nat News. 2011;477(7366):526–8.

    Article  CAS  Google Scholar 

  9. Liu R, Rizzo S, Whipple S, Pal N, Pineda AL, Lu M, et al. Evaluating eligibility criteria of oncology trials using real-world data and AI. Nature. 2021;592(7855):629–33.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  10. Shenoy P. Multi-regional clinical trials and global drug development. Perspect Clin Res. 2016;7(2):62.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Song SY, Chee D, Kim E. Strategic inclusion of regions in multiregional clinical trials. Clin Trials. 2019;16(1):98–105.

    Article  PubMed  Google Scholar 

  12. Fogel DB. Factors associated with clinical trials that fail and opportunities for improving the likelihood of success: a review. Contemp Clin Trials Commun. 2018;11:156–64.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Allison M. Reinventing clinical trials. Nat Biotechnol. 2012 Jan 9;30(1):41–9.

    Article  CAS  PubMed  Google Scholar 

  14. Cavallo C, Labib MA, Honea N, Nakaji P. Enrollment-to-screening ratio: an undervalued data in randomized clinical trials. Neurosurgery. 2018.

  15. Craven BC, Balioussis C, Hitzig SL, Moore C, Verrier MC, Giangregorio LM, et al. Use of screening to recruitment ratios as a tool for planning and implementing spinal cord injury rehabilitation research. Spinal Cord. 2014;52(10):764–8.

    Article  CAS  PubMed  Google Scholar 

  16. Blanton S, Morris DM, Prettyman MG, McCulloch K, Redmond S, Light KE, et al. Lessons learned in participant recruitment and retention: the EXCITE trial. Phys Ther. 2006;86(11):1520–33.

    Article  PubMed  Google Scholar 

  17. Harris-Brown TM, Paterson DL. Reporting of pre-enrolment screening with randomized clinical trials: a small item that could impact a big difference. Perspect Clin Res. 2015;6(3):139.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Giffin RB, Lebovitz Y, English RA. Transforming clinical research in the United States: challenges and opportunities: workshop summary: National Academies Press; 2010.

    Google Scholar 

  19. Lamberti MJ, Smith Z, Henry R, Howe D, Goodwin M, Williams A, et al. Benchmarking patient recruitment and retention practices. Ther Innov Regul Sci. 2021;55(1):19–32.

    Article  PubMed  Google Scholar 

  20. Stergiopoulos S, Calvert SB, Brown CA, Awatin J, Tenaerts P, Holland TL, et al. Cost drivers of a hospital-acquired bacterial pneumonia and ventilator-associated bacterial pneumonia phase 3 clinical trial. Clin Infect Dis. 2018;66(1):72–80.

    Article  PubMed  Google Scholar 

  21. McDonald AM, Knight RC, Campbell MK, Entwistle VA, Grant AM, Cook JA, et al. What influences recruitment to randomised controlled trials? A review of trials funded by two UK funding agencies. Trials. 2006;7(1):1–8.

    Article  Google Scholar 

  22. Keith SJ. Evaluating characteristics of patient selection and dropout rates. J Clin Psychiatry. 2001;62:11–6.

    CAS  PubMed  Google Scholar 

  23. Dicesare J. Improve the clinical trial startup process with just-in-time site activation. Applied Clinical Trials [Internet]. 2014 Aug 19; Available from: https://www.appliedclinicaltrialsonline.com/view/improve-clinical-trial-startup-process-just-time-site-activation

  24. Getz KA. Characterizing the real cost of site regulatory compliance. Appl Clin Trials. 2015;24(6/7):18.

    Google Scholar 

  25. Eichler H-G, Sweeney F. The evolution of clinical trials: can we address the challenges of the future? Clin Trials. 2018;15(1_suppl):27–32.

  26. Smuck B, Bettello P, Berghout K, Hanna T, Kowaleski B, Phippard L, et al. Ontario protocol assessment level: clinical trial complexity rating tool for workload planning in oncology clinical trials. J Oncol Pract. 2011;7(2):80–4.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Cunanan KM, Gonen M, Shen R, Hyman DM, Riely GJ, Begg CB, et al. Basket trials in oncology: a trade-off between complexity and efficiency. J Clin Oncol. 2017;35(3):271.

    Article  PubMed  Google Scholar 

  28. Malik L, Lu D. Increasing complexity in oncology phase I clinical trials. Invest New Drugs. 2019;37(3):519–23.

    Article  PubMed  Google Scholar 

  29. Yuan G, Wang L, Li J, Feng H, Ji J, Gu W, et al. Complexity in clinical trials: blind spots, misleading criteria, winners and losers. Clin Cancer Drugs. 2020;7(1):3–15.

    Article  Google Scholar 

  30. Thadani SR, Weng C, Bigger JT, Ennever JF, Wajngurt D. Electronic screening improves efficiency in clinical trial recruitment. J Am Med Inform Assoc. 2009;16(6):869–73.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Frank G. Current challenges in clinical trial patient recruitment and enrollment. SoCRA Source. 2004;2(February):30–8.

    Google Scholar 

  32. Kadam RA, Borde SU, Madas SA, Salvi SS, Limaye SS. Challenges in recruitment and retention of clinical trial subjects. Perspect Clin Res. 2016;7(3):137.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Dilts DM, Sandler AB, Baker M, Cheng SK, George SL, Karas KS, et al. Processes to activate phase III clinical trials in a Cooperative Oncology Group: the Case of Cancer and Leukemia Group B. J Clin Oncol. 2006;24(28):4553–7.

    Article  PubMed  Google Scholar 

  34. Getz KA, Wenger J, Campo RA, Seguine ES, Kaitin KI. Assessing the impact of protocol design changes on clinical trial performance. Am J Ther. 2008;15(5):450–7.

    Article  PubMed  Google Scholar 

  35. Ross S, Grant A, Counsell C, Gillespie W, Russell I, Prescott R. Barriers to participation in randomised controlled trials: a systematic review. J Clin Epidemiol. 1999;52(12):1143–56.

    Article  CAS  PubMed  Google Scholar 

  36. Boericke K, Gwinn B. Planned to perfection. Int Clin Trials. 2010;17(8):26–30.

    Google Scholar 

  37. Andersen JW, Fass R, van der Horst C. Factors associated with early study discontinuation in AACTG studies, DACS 200. Contemp Clin Trials. 2007;28(5):583–92.

    Article  PubMed  Google Scholar 

  38. Tseo Y, Salkola MI, Mohamed A, Kumar A, Abnousi F. Information extraction of clinical trial eligibility criteria. ArXiv Prepr ArXiv200607296. 2020;

  39. Liu H, Chi Y, Butler A, Sun Y, Weng C. A knowledge base of clinical trial eligibility criteria. J Biomed Inform. 2021;117:103771.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Munos B, Niederreiter J, Riccaboni M. Improving the prediction of clinical success using machine learning. 2020;

  41. Feijoo F, Palopoli M, Bernstein J, Siddiqui S, Albright TE. Key indicators of phase transition for clinical trials through machine learning. Drug Discov Today. 2020;25(2):414–21.

    Article  PubMed  Google Scholar 

  42. Ke G, Meng Q, Finley T, Wang T, Chen W, Ma W, et al. Lightgbm: A highly efficient gradient boosting decision tree. Adv Neural Inf Process Syst. 2017;30:3146–54.

    Google Scholar 

  43. Feurer M, Klein A, Eggensperger K, Springenberg JT, Blum M, Hutter F. Auto-sklearn: efficient and robust automated machine learning. In: Automated Machine Learning. Springer, Cham; 2019. p. 113–34.

  44. Harrell FE, Califf RM, Pryor DB, Lee KL, Rosati RA. Evaluating the yield of medical tests. Jama. 1982;247(18):2543–6.

    Article  PubMed  Google Scholar 

  45. Apley DW, Zhu J. Visualizing the effects of predictor variables in black box supervised learning models. J R Stat Soc Ser B Stat Methodol. 2020;82(4):1059–86.

    Article  Google Scholar 

Download references

Acknowledgements

We thank Ernesto Guarin, Nicole Kim, Katelyn Bechler, James Harper, Jacek Basta, and Jennifer Copping for their helpful feedback.

Funding

This work was funded by Genentech, part of the Roche Group.

Author information

Authors and Affiliations

  1. Department of Biomedical Data Science, Stanford University, Stanford, California, USA

    Kevin Wu & James Zou

  2. Department of Electrical Engineering, Stanford University, Stanford, California, USA

    Eric Wu, Ruishan Liu & James Zou

  3. Genentech, South San Francisco, San Francisco, California, USA

    Michael DAndrea, Nandini Chitale, Melody Lim, Navdeep Pal, Shemra Rizzo & Ryan Copping

  4. Roche Pharmaceuticals, Warsaw, Poland

    Marek Dabrowski & Klaudia Kantor

  5. Roche Pharmaceuticals, Mississauga, Canada

    Hanoor Rangi

  6. Roche Pharmaceuticals, Basel, Switzerland

    Marius Garmhausen

  7. Roche Pharmaceuticals, Welwyn Garden City, UK

    Chris Harbron

Authors
  1. Kevin Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eric Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michael DAndrea
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nandini Chitale
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Melody Lim
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Marek Dabrowski
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Klaudia Kantor
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Hanoor Rangi
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Ruishan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Marius Garmhausen
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Navdeep Pal
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Chris Harbron
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Shemra Rizzo
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Ryan Copping
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. James Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

KW and EW were involved in the design, analysis, and interpretation of data. JZ was involved in the design and interpretation of the data. MD, NC, ML, MD, and KK were involved in feedback on content and interpretation. HR, RL, MG, NP, CH, SR, and RC were involved in the review and feedback on content.

Corresponding author

Correspondence to Kevin Wu.

Ethics declarations

Competing Interest

MDA, NC, ML, MD, KK, HR, MG, NP, CH, SR, and RC are employees of Roche.

Additional information

Guest Editors: Lawrence Yu, Hao Zhu and Qi Liu

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

ESM 1

(DOCX 21 kb)

ESM 2

(EPS 589 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, K., Wu, E., DAndrea, M. et al. Machine Learning Prediction of Clinical Trial Operational Efficiency. AAPS J 24, 57 (2022). https://doi.org/10.1208/s12248-022-00703-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1208/s12248-022-00703-3

KEY WORDS

Search

Navigation