Skip to main content
SpringerLink
Log in

Describing, Resampling, Validating, and Simplifying the Model

  • Chapter
Regression Modeling Strategies

Part of the book series: Springer Series in Statistics ((SSS))

Abstract

Before addressing issues related to describing and interpreting the model and its coefficients, one can never apply too much caution in attempting to interpret results in a causal manner. Regression models are excellent tools for estimating and inferring associations between an X and Y given that the “right” variables are in the model. Any ability of a model to provide causal inference rests entirely on the faith of the analyst in the experimental design, completeness of the set of variables that are thought to measure confounding and are used for adjustment when the experiment is not randomized, lack of important measurement error, and lastly the goodness of fit of the model.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 129.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The s.d. of a binary variable is, aside from a multiplier of \(\frac{n} {n-1}\), equal to \(\sqrt{a(1 - a)}\), where a is the proportion of ones.

  2. 2.

    There are decompositions of the Brier score into discrimination and calibration components.

  3. 3.

    For example, in the binary logistic model, there is a generalization of R 2 available, but no adjusted version. For logistic models we often validate other indexes such as the ROC area or rank correlation between predicted probabilities and observed outcomes. We also validate the calibration accuracy of \(\hat{Y }\) in predicting Y.

  4. 4.

    Using the rms package described in Chapter 6, such estimates and their confidence limits can easily be obtained, using for example contrast(fit, list(age=50, disease=’hypertension’, race=levels(race)), type=’average’, weights=table(race)).

References

  1. D. G. Altman and P. Royston. What do we mean by validating a prognostic model? Stat Med, 19:453–473, 2000.

    Article  Google Scholar 

  2. G. Ambler, A. R. Brady, and P. Royston. Simplifying a prognostic model: a simulation study based on clinical data. Stat Med, 21(24):3803–3822, Dec. 2002.

    Article  Google Scholar 

  3. P. C. Austin and E. W. Steyerberg. Events per variable (EPV) and the relative performance of different strategies for estimating the out-of-sample validity of logistic regression models. Statistical methods in medical research, Nov. 2014.

    Google Scholar 

  4. P. C. Austin and E. W. Steyerberg. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Stat Med, 33(3):517–535, Feb. 2014.

    Article  MathSciNet  Google Scholar 

  5. J. Banks. Nomograms. In S. Kotz and N. L. Johnson, editors, Encyclopedia of Stat Scis, volume 6. Wiley, New York, 1985.

    Google Scholar 

  6. S. E. Bleeker, H. A. Moll, E. W. Steyerberg, A. R. T. Donders, G. Derkson-Lubsen, D. E. Grobbee, and K. G. M. Moons. External validation is necessary in prediction research: A clinical example. J Clin Epi, 56:826–832, 2003.

    Article  Google Scholar 

  7. M. Blettner and W. Sauerbrei. Influence of model-building strategies on the results of a case-control study. Stat Med, 12:1325–1338, 1993.

    Article  Google Scholar 

  8. J. G. Booth and S. Sarkar. Monte Carlo approximation of bootstrap variances. Am Statistician, 52:354–357, 1998.

    Google Scholar 

  9. L. Breiman. The little bootstrap and other methods for dimensionality selection in regression: X-fixed prediction error. J Am Stat Assoc, 87:738–754, 1992.

    Article  MathSciNet  MATH  Google Scholar 

  10. D. Brownstone. Regression strategies. In Proceedings of the 20th Symposium on the Interface between Computer Science and Statistics, pages 74–79, Washington, DC, 1988. American Statistical Association.

    Google Scholar 

  11. A. J. Canty, A. C. Davison, D. V. Hinkley, and V. Venture. Bootstrap diagnostics and remedies. Can J Stat, 34:5–27, 2006.

    Article  MATH  Google Scholar 

  12. J. Carpenter and J. Bithell. Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians. Stat Med, 19:1141–1164, 2000.

    Article  Google Scholar 

  13. C. Chatfield. Model uncertainty, data mining and statistical inference (with discussion). J Roy Stat Soc A, 158:419–466, 1995.

    Article  Google Scholar 

  14. J. B. Copas. Cross-validation shrinkage of regression predictors. J Roy Stat Soc B, 49:175–183, 1987.

    MathSciNet  MATH  Google Scholar 

  15. A. C. Davison and D. V. Hinkley. Bootstrap Methods and Their Application. Cambridge University Press, Cambridge, 1997.

    Book  Google Scholar 

  16. B. Efron. Estimating the error rate of a prediction rule: Improvement on cross-validation. J Am Stat Assoc, 78:316–331, 1983.

    Article  MathSciNet  Google Scholar 

  17. B. Efron. How biased is the apparent error rate of a prediction rule? J Am Stat Assoc, 81:461–470, 1986.

    Article  MathSciNet  MATH  Google Scholar 

  18. B. Efron and G. Gong. A leisurely look at the bootstrap, the jackknife, and cross-validation. Am Statistician, 37:36–48, 1983.

    MathSciNet  Google Scholar 

  19. B. Efron and R. Tibshirani. Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical Sci, 1:54–77, 1986.

    Article  MathSciNet  Google Scholar 

  20. B. Efron and R. Tibshirani. An Introduction to the Bootstrap. Chapman and Hall, New York, 1993.

    Book  MATH  Google Scholar 

  21. B. Efron and R. Tibshirani. Improvements on cross-validation: The.632+ bootstrap method. J Am Stat Assoc, 92:548–560, 1997.

    Google Scholar 

  22. D. Faraggi, M. LeBlanc, and J. Crowley. Understanding neural networks using regression trees: an application to multiple myeloma survival data. Stat Med, 20:2965–2976, 2001.

    Article  Google Scholar 

  23. J. J. Faraway. The cost of data analysis. J Comp Graph Stat, 1:213–229, 1992.

    Google Scholar 

  24. J. Fox. Applied Regression Analysis and Generalized Linear Models. SAGE Publications, Thousand Oaks, CA, second edition, 2008.

    Google Scholar 

  25. D. Freedman, W. Navidi, and S. Peters. On the Impact of Variable Selection in Fitting Regression Equations, pages 1–16. Lecture Notes in Economics and Mathematical Systems. Springer-Verlag, New York, 1988.

    MATH  Google Scholar 

  26. A. Gelman. Scaling regression inputs by dividing by two standard deviations. Stat Med, 27:2865–2873, 2008.

    Article  MathSciNet  MATH  Google Scholar 

  27. M. Gönen and G. Heller. Concordance probability and discriminatory power in proportional hazards regression. Biometrika, 92(4):965–970, Dec. 2005.

    Article  MathSciNet  MATH  Google Scholar 

  28. G. Gong. Cross-validation, the jackknife, and the bootstrap: Excess error estimation in forward logistic regression. J Am Stat Assoc, 81:108–113, 1986.

    Article  Google Scholar 

  29. I. M. Graham and E. Clavel. Communicating risk — coronary risk scores. J Roy Stat Soc A, 166:217–223, 2003.

    Article  MathSciNet  Google Scholar 

  30. P. Hall and H. Miller. Using the bootstrap to quantify the authority of an empirical ranking. Ann Stat, 37(6B):3929–3959, 2009.

    Article  MathSciNet  MATH  Google Scholar 

  31. T. L. Hankins. Blood, dirt, and nomograms. Chance, 13(1):26–37, 2000.

    Article  Google Scholar 

  32. F. E. Harrell. Comparison of strategies for validating binary logistic regression models. Unpublished manuscript, 1991.

    Google Scholar 

  33. M. Julien and J. A. Hanley. Profile-specific survival estimates: Making reports of clinical trials more patient-relevant. CT, 5:107–115, 2008.

    Google Scholar 

  34. A. C. Justice, K. E. Covinsky, and J. A. Berlin. Assessing the generalizability of prognostic information. Ann Int Med, 130:515–524, 1999.

    Article  Google Scholar 

  35. J. Karvanen and F. E. Harrell. Visualizing covariates in proportional hazards model. Stat Med, 28:1957–1966, 2009.

    Article  MathSciNet  Google Scholar 

  36. M. W. Kattan and J. Marasco. What is a real nomogram? Sem Onc, 37(1): 23–26, Feb. 2010.

    Google Scholar 

  37. V. Kipnis. Relevancy criterion for discriminating among alternative model specifications. In K. Berk and L. Malone, editors, Proceedings of the 21st Symposium on the Interface between Computer Science and Statistics, pages 376–381, Alexandria, VA, 1989. American Statistical Association.

    Google Scholar 

  38. K. Larsen and J. Merlo. Appropriate assessment of neighborhood effects on individual health: integrating random and fixed effects in multilevel logistic regression. American journal of epidemiology, 161(1):81–88, Jan. 2005.

    Article  Google Scholar 

  39. M. A. H. Levine, A. I. El-Nahas, and B. Asa. Relative risk and odds ratio data are still portrayed with inappropriate scales in the medical literature. J Clin Epi, 63:1045–1047, 2010.

    Article  Google Scholar 

  40. K. Linnet. Assessing diagnostic tests by a strictly proper scoring rule. Stat Med, 8:609–618, 1989.

    Article  Google Scholar 

  41. J. Lubsen, J. Pool, and E. van der Does. A practical device for the application of a diagnostic or prognostic function. Meth Info Med, 17:127–129, 1978.

    Google Scholar 

  42. D. Paul, E. Bair, T. Hastie, and R. Tibshirani. “Preconditioning” for feature selection and regression in high-dimensional problems. Ann Stat, 36(4):1595–1619, 2008.

    Article  MathSciNet  MATH  Google Scholar 

  43. R. R. Picard and K. N. Berk. Data splitting. Am Statistician, 44:140–147, 1990.

    Google Scholar 

  44. R. R. Picard and R. D. Cook. Cross-validation of regression models. J Am Stat Assoc, 79:575–583, 1984.

    Article  MathSciNet  MATH  Google Scholar 

  45. E. B. Roecker. Prediction error and its estimation for subset-selected models. Technometrics, 33:459–468, 1991.

    Article  Google Scholar 

  46. W. Sauerbrei and M. Schumacher. A bootstrap resampling procedure for model building: Application to the Cox regression model. Stat Med, 11:2093–2109, 1992.

    Article  Google Scholar 

  47. J. Shao. Linear model selection by cross-validation. J Am Stat Assoc, 88:486–494, 1993.

    Article  MATH  Google Scholar 

  48. D. J. Spiegelhalter. Probabilistic prediction in patient management and clinical trials. Stat Med, 5:421–433, 1986.

    Article  Google Scholar 

  49. E. W. Steyerberg, S. E. Bleeker, H. A. Moll, D. E. Grobbee, and K. G. M. Moons. Internal and external validation of predictive models: A simulation study of bias and precision in small samples. Journal of Clinical Epi, 56(5):441–447, May 2003.

    Article  Google Scholar 

  50. E. W. Steyerberg, F. E. Harrell, G. J. J. M. Borsboom, M. J. C. Eijkemans, Y. Vergouwe, and J. D. F. Habbema. Internal validation of predictive models: Efficiency of some procedures for logistic regression analysis. J Clin Epi, 54:774–781, 2001.

    Article  Google Scholar 

  51. R. Tibshirani and K. Knight. The covariance inflation criterion for adaptive model selection. J Roy Stat Soc B, 61:529–546, 1999.

    Article  MathSciNet  MATH  Google Scholar 

  52. M. J. van Gorp, E. W. Steyerberg, M. Kallewaard, and Y. var der Graaf. Clinical prediction rule for 30-day mortality in Björk-Shiley convexo-concave valve replacement. J Clin Epi, 56:1006–1012, 2003.

    Google Scholar 

  53. H. C. van Houwelingen and J. Thorogood. Construction, validation and updating of a prognostic model for kidney graft survival. Stat Med, 14:1999–2008, 1995.

    Article  Google Scholar 

  54. J. C. van Houwelingen and S. le Cessie. Predictive value of statistical models. Stat Med, 9:1303–1325, 1990.

    Article  MATH  Google Scholar 

  55. Y. Vergouwe, E. W. Steyerberg, M. J. C. Eijkemans, and J. D. F. Habbema. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epi, 58:475–483, 2005.

    Article  Google Scholar 

  56. P. J. M. Verweij and H. C. van Houwelingen. Cross-validation in survival analysis. Stat Med, 12:2305–2314, 1993.

    Article  Google Scholar 

  57. C. F. J. Wu. Jackknife, bootstrap and other resampling methods in regression analysis. Ann Stat, 14(4):1261–1350, 1986.

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Biostatistics, School of Medicine Vanderbilt University, Nashville, TN, USA

    Frank E. Harrell Jr.

Authors
  1. Frank E. Harrell Jr.
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Harrell, F.E. (2015). Describing, Resampling, Validating, and Simplifying the Model. In: Regression Modeling Strategies. Springer Series in Statistics. Springer, Cham. https://doi.org/10.1007/978-3-319-19425-7_5

Download citation

Publish with us

Policies and ethics

Search

Navigation