Abstract
This article revisits 20 years of our work in developing evaluation tools adapted to non-linear mixed effect models. These hierarchical models involve a large number of assumptions concerning the structural evolution of the outcomes, the link between different outcomes, the variabilities in the parameters and model evaluation aims at assessing these various components, both to help guide the model building and to communicate on model adequacy for a given purpose. During our career, we have developed and extended simulation-based evaluation tools called normalised prediction discrepancies (npd) and normalised prediction distribution errors (npde), providing informative diagnostics through graphs and tests.
Graphical abstract
Similar content being viewed by others
References
Sheiner LB, Rosenberg B, Marathe VV. Estimation of population characteristics of pharmacokinetic parameters from routine clinical data. J Pharmacokin Biopharm. 1977;5(5):445–79.
Mentré F. Lewis Sheiner ISoP/UCSF lecturer award: from drug use to statistical models and vice versa. CPT Pharmacometrics Syst Pharmacol. 2014;3:e154.
Beal S, Sheiner L, Boeckmann A, Bauer R. NONMEM Version 7.2. Ellicott City; 1989-2011.
Lavielle M. Mixed effects models for the population approach: models, tasks, methods and tools. Chapman & Hall/CRC Biostatistics Series; 2014.
Carpenter B, Gelman A, Hoffman M, Lee D, Goodrich B, Betancourt M, et al. Stan: a probabilistic programming language. J Stat Softw. 2017;76:1–32.
Pinheiro J, Bates D, DebRoy S, Sarkar D, R Core Team. nlme: linear and nonlinear mixed effects models; 2020. R package version 3.1–150. Available from: https://CRAN.R-project.org/package=nlme.
Bates D, Mächler M, Bolker B, Walker S. Fitting linear mixed-effects models using lme4. J Stat Softw. 2015;67(1):1–48.
Comets E, Lavenu A, Lavielle M. Parameter estimation in nonlinear mixed effect models using saemix, an R implementation of the SAEM algorithm. J Stat Softw. 2017;80:1–41.
Fidler M, Xiong Y, Schoemaker R, Wilkins J, Trame M, Hooijmaijers R, et al.. nlmixr: nonlinear mixed effects models in population pharmacokinetics and pharmacodynamics; 2021. R package version 2.0.1. Available from: https://CRAN.R-project.org/package=nlmixr.
Mentré F, Mallet A, Baccar D. Optimal design in random-effects regression models. Biometrika. 1984;84:429–42.
Sheiner LB, Rosenberg B, Melmon KL. Modelling of individual pharmacokinetics for computer- aided drug dosage. Comput Biomed Res. 1972;5(5):441–59.
Sheiner LB, Beal SL. Some suggestions for measuring predictive performance. J Pharmacokin Biopharm. 1981;9:503–12.
European Medicines Agency. Guideline on reporting the results of population pharmacokinetic analysis (CHMP); 2007. Available from: http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2009/09/WC500003067.pdf.
Food and Drug Administration. Guidance for industry exposure-response relationships– study design, data analysis, and regulatory applications; 2019. Available from: https://www.fda.gov/media/128793/download.
Karlsson M, Savic R. Diagnosing model diagnostics. Clin Pharmacol Ther. 2007;82:17–20.
Mentré F, Escolano S. Prediction discrepancies for the evaluation of nonlinear mixed-effects models. J Pharmacokinet Pharmacodyn. 2006;33:345–67.
Brendel K, Comets E, Laffont C, Laveille C, Mentré F. Metrics for external model evaluation with an application to the population pharmacokinetics of gliclazide. Pharm Res. 2006;23:2036–49.
Mesnil F, Mentré F, Dubruc C, Thénot JP, Mallet A. Population pharmacokinetic analysis of mizolastine and validation from sparse data on patients using the nonparametric maximum likelihood method. J Pharmacokinet Pharmacodyn. 1998;26(2):133–61.
Dunn PK, Smyth GK. Randomized quantile residuals. J Comput Graph Stat. 1996;5:236–44.
Brendel K, Comets E, Laffont C, Mentré F. Evaluation of different tests based on observations for external model evaluation of population analyses. J Pharmacokinet Pharmacodyn. 2010;37:49–65.
Yano Y, Beal SL, Sheiner LB. Evaluating pharmacokinetic/pharmacodynamic models using the posterior predictive check. J Pharmacokinet Pharmacodyn. 2001;28(2):171–92.
Holford N. The visual predictive check—superiority to standard diagnostic (Rorschach) plots. PAGE 14. 2005;Abstr 738.
Bergstrand M, Hooker A, Wallin J, Karlsson M. Prediction-corrected visual predictive checks for diagnosing nonlinear mixed-effects models. AAPS J. 2011;13:143–51.
Hooker AC, Staatz CE, Karlsson MO. Conditional weighted residuals (CWRES): a model diagnostic for the FOCE method. Pharm Res. 2007;24(12):2187–97.
Nyberg J, Bauer RJ, Hooker AC. Investigations of the weighted residuals in NONMEM 7. PAGE 10. 2010;Abstr 1883.
Comets E, Brendel K, Mentré F. Computing normalised prediction distribution errors to evaluate nonlinear mixed-effect models: the npde add-on package for R. Comput Meth Prog Biomed. 2008;90:154–66.
R Core Team. R: a language and environment for statistical computing. Vienna, Austria; 2015. Available from: https://www.R-project.org/.
Keizer R, Karlsson M, Hooker A. Modeling and simulation workbench for NONMEM: tutorial on Pirana, PsN, and Xpose. CPT Pharmacometrics Syst Pharmacol. 2013;2:e50.
Nguyen T, Mouksassi MS, Holford N, Al-Huniti N, Freedman I, Hooker A, et al. Model evaluation of continuous data pharmacometric models: metrics and graphics. CPT Pharmacometrics Syst Pharmacol. 2017;6(2):87–109.
Bergstrand M, Karlsson M. Handling data below the limit of quantification in mixed effect models. AAPS J. 2009;11(2):371–80.
Nguyen THT, Comets E, Mentré F. Extension of NPDE for evaluation of nonlinear mixed effect models in presence of data below the quantification limit with applications to HIV dynamic model. J Pharmacokinet Pharmacodyn. 2012;39(5):499–518.
Cerou M, Lavielle M, Brendel K, Chenel M, Comets E. Development and performance of npde for the evaluation of time-to-event models. Pharm Res. 2018;35(2):30.
Mallet A. A maximum likelihood estimation method for random coefficient regression models. Biometrika. 1986;73:645–56.
Comets E, Brendel K, Mentré F. Model evaluation in nonlinear mixed effect models, with applications to pharmacokinetics. J Soc Fr Statistique. 2010;151:106–28.
Comets E, Nguyen THT, Mentré F. Additional features and graphs in the new npde library for R. PAGE 22. 2013;Abstr 2775.
Cerou M, Peigné S, Chenel M, Comets E. Performance of npde for the evaluation of joint model with time to event data. PAGE 28. 2019;Abstr 8940.
Wickham H. ggplot2: elegant graphics for data analysis. New York: Springer-Verlag; 2016. Available from: https://ggplot2.tidyverse.org
Savic R, Barrail-Tran A, Duval X, Nembot G, Panhard X, Descamps D, et al. Effect of adherence as measured by MEMS, ritonavir boosting, and CYP3A5 genotype on atazanavir pharmacokinetics in treatment-naive HIV-infected patients. Clin Pharmacol Ther. 2012;92:575–83. [39] box G. science and statistics. J Am Stat Assoc. 1976;71:791–9.
Brendel K, Dartois C, Comets E, Lemmenuel-Diot A, Laveille C, Tranchand B, et al. Are population PK and/or PD models adequately evaluated? A 2002 to 2004 literature survey. Clin Pharmacokin. 2007;46:221–34.
Dartois C, Brendel K, Comets E, Laffont C, Laveille C, Tranchand B, et al. Overview of model building strategies in population PK/PD analyses: 2002 to 2004 literature survey. Br J Clin Pharmacol. 2007;64:603–12.
Ioannidis J. Reproducible pharmacokinetics. J Pharmacokinet Pharmacodyn. 2019;46:111–6.
Author information
Authors and Affiliations
Contributions
The two (female) authors of this article (F. Mentré and E. Comets) have been the main architects for the development of npde. The original idea of prediction discrepancies was proposed by France Mentré with her PhD student Florence Mesnil for non-parametric mixed effect models and extended to general parametric models with Sylvie Escolano. Emmanuelle Comets got involved in the definition of npde and co-supervised the PhD of Karl Brendel, who also conducted in collaboration with Céline Dartois a survey of how population PK and PD analyses were evaluated. The extension of npde to BLQ data was the methodological topic of Tram (Thi Huyen) Nguyen, another female PhD student supervised by France Mentré. Tram was also the first author for the collaborative white paper by the ISoP group (29). Emmanuelle Comets supervised the PhD of Marc Cerou, our second male PhD student on the topic, who extended npd to models involving time-to-event and categorical outcomes (manuscript under preparation). Finally, we also would like to acknowledge the contribution since January 2020 of our engineer, Romain Leroux, for the latest version of the npde library. France Mentré was awarded the prestigious Lewis Sheiner lecturer award from the University of California and the International Society of Pharmacometrics in 2013, recalling in her lecture the prominent part of model evaluation in her career (2).
Corresponding author
Additional information
Guest Editors: Diane Burgess, Marilyn Morris and Meena Subramanyam
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
ESM 1
(PDF 806 kb)
Rights and permissions
About this article
Cite this article
Comets, E., Mentré, F. Developing Tools to Evaluate Non-linear Mixed Effect Models: 20 Years on the npde Adventure. AAPS J 23, 75 (2021). https://doi.org/10.1208/s12248-021-00597-7
Received:
Accepted:
Published:
DOI: https://doi.org/10.1208/s12248-021-00597-7