Some Statistical Considerations in Design and Analysis for Nonrandomized Comparative Studies Using Existing Data as Controls for Medical Device Premarket Evaluation

  • Nelson LuEmail author
  • Yunling Xu
  • Lilly Q. Yue
Part of the ICSA Book Series in Statistics book series (ICSABSS)


Nonrandomized comparative studies have been widely proposed and conducted in the premarket safety and effectiveness evaluation of medical devices. Sometimes, applicants conduct such studies in which the control data are formed from external sources, such as prior clinical studies. This type of study becomes even more popular with the availability of widely available real-world data such as patient registries of reasonable quality being sources for the control group. While such studies may potentially save time and money, bias can be introduced in every stage of such studies. To maintain the objectivity of study design and validity of study results, propensity score methodology is utilized and proper procedure of study design need to be implemented. In this article, considerations of the design and analysis of the non-randomized studies using the propensity score methodology are discussed from the statistical and regulatory perspectives.


Non-randomized study Real world data Propensity score Estimand Two-stage design 


  1. Austin, P.: Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples. Stat. Med. 28, 3083–3107 (2009)MathSciNetCrossRefGoogle Scholar
  2. Austin, P.: An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivar. Behav. Res. 46(3), 399–424 (2011a)CrossRefGoogle Scholar
  3. Austin, P.: A tutorial and case study in propensity score analysis: an application to estimating the effect of in-hospital smoking cessation counseling on mortality. Multivar. Behav. Res. 46(1), 119–151 (2011b)CrossRefGoogle Scholar
  4. Code of Federal Regulations 21CFR860.7(c)(2).: U.S. Government Publishing Office. (2012). Accessed 30 Nov 2017
  5. D’Agostino, R.: Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Stat. Med. 17, 2265–2281 (1998)CrossRefGoogle Scholar
  6. Hill, J.L., Reiter, J.P., Zanutto, E.L.: A comparison of experimental and observational data analyses. In: Gelman, A., Meng, X.-L. (eds.) Applied Bayesian Modeling and Causal Inference from an Incomplete-Data Perspective, pp. 44–56. Wiley, New York (2004)Google Scholar
  7. Imbens, G.W.: Nonparametric estimation of average treatment effects under exogeneity: a review. Rev. Econ. Stat. 86, 4–29 (2004)CrossRefGoogle Scholar
  8. International Conference on Harmonisation.: Final concept paper E9(R1): addendum to statistical principles for clinical trials on choosing appropriate estimands and defining sensitivity analyses in clinical trials. (2014). Accessed 13 Nov 2017
  9. Li, H., Mukhi, V., Lu, N., Xu, Y., Yue, Q.L.: A note on good practice of objective propensity score design for premarket nonrandomized medical device studies with an example. Stat. Biopharm. Res. 8(3), 282–286 (2016)CrossRefGoogle Scholar
  10. Mehrotra, D.V., Hemmings, R.J., Russek-Cohen, E., ICH E9/R1 Expert Working Group: Seeking harmony: estimands and sensitivity analyses for confirmatory clinical trials. Clin. Trials. 13(4), 456–458 (2016)CrossRefGoogle Scholar
  11. Rosenbaum, P.R.: Observational Studies, 2nd edn. Springer, New York (2002)CrossRefGoogle Scholar
  12. Rosenbaum, P.R., Rubin, D.B.: The central role of the propensity score in observational studies for causal effects. Biometrika. 70(1), 41–55 (1983)MathSciNetCrossRefGoogle Scholar
  13. Rosenbaum, P.R., Rubin, D.B.: Reducing bias in observational studies using subclassification on the propensity score. J. Am. Stat. Assoc. 79, 516–524 (1984)CrossRefGoogle Scholar
  14. Rubin, D.B.: Estimating causal effects of treatments in randomized and nonrandomized studies. J. Educ. Psychol. 66, 688–701 (1974)CrossRefGoogle Scholar
  15. Rubin, D.B.: For objective causal inference, design trumps analysis. Ann. Appl. Stat. 2(3), 808–840 (2008)MathSciNetCrossRefGoogle Scholar
  16. Rubin, D.B., Thomas, N.: Matching using estimated propensity scores: relating theory to practice. Biometrics. 52, 249–264 (1996)CrossRefGoogle Scholar
  17. Schafer, J.L., Kang, J.: Average causal effects from nonrandomized studies: a practical guide and simulated example. Psychol. Methods. 13, 279–313 (2008)CrossRefGoogle Scholar
  18. Stuart, E.A.: Developing practical recommendations for the use of propensity scores: discussion of “a critical appraisal of propensity score matching in the medical literature between 1996 and 2003”. Stat. Med. 27(12), 2062–2065 (2008)MathSciNetCrossRefGoogle Scholar
  19. Stuart, E.A.: Matching methods for causal inference: a review and a look forward. Stat. Med. 25, 1–21 (2010)MathSciNetzbMATHGoogle Scholar
  20. US Food and Drug Administration.: Use of real-world evidence to support regulatory decision making for medical devices. Guidance for industry and Food and Drug Administration staff. (2017). Accessed 18 Nov 2017
  21. Yue, L., Campbell, G., Lu, N., Xu, Y., Zuckerman, B.: Utilizing national and international registries to enhance pre-market medical device regulatory evaluation. J. Biopharm. Stat. 26(6), 1136–1145 (2016)CrossRefGoogle Scholar
  22. Yue, L., Lu, N., Xu, Y.: Designing premarket observational comparative studies using existing data as controls: challenges and opportunities. J. Biopharm. Stat. 24(5), 994–1010 (2014)MathSciNetCrossRefGoogle Scholar
  23. Yue, L.Q.: Regulatory considerations in the design of comparative observational studies using propensity scores. J. Biopharm. Stat. 22(6), 1272–1279 (2012)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.CDRH, U.S. Food and Drug AdministrationSilver SpringUSA

Personalised recommendations