Modeling by Combining Dimension Reduction and L2Boosting

  • Junlong Zhao
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7389)


Dimension reduction techniques are widely used in high dimensional modeling. The two stage approach, first making dimension reduction and then applying existing regression or classification method, is commonly used in practice. However, an important issue is that when two stage approach can lead to consistent estimate. In this paper, we focus on L2boosting and discuss the consistency of the two stage method-dimension reduction based L2boosting (briey DRL2B). We establish the conditions under which DRL2B method results in consistent estimate. This theoretical finding provides some useful guideline for practical application. In addition, we propose an iterative DRL2B approach and make some simulation study. Simulation results shows that iterative DRL2B method has good performance


dimension reduction L2boosting consistency 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Breiman, L., Friedman, J.H.: Estimating Optimal Transformations for Multiple Regression and Correlation. J. Amer. Statist. Assoc. 80, 580–598 (1985)MathSciNetzbMATHCrossRefGoogle Scholar
  2. 2.
    Bühlmann, P., Hothorn, T.: Boosting Algorithms: Regularization, Prediction and Model Fitting. Statist. Sci. 22(4), 477–505 (2007)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Bühlmann, P., Yu, B.: Boosting with the L2-loss: Regression and Classification. J. Amer. Statist. Assoc. 98, 324–339 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
  4. 4.
    Cook, R.D., Weisberg, S.: Sliced Inverse Regression for Dimension Reduction: Comment. J. Amer. Statist. Assoc. 86(414), 328–332 (1991)Google Scholar
  5. 5.
    Cook, R.D., Weisberg, S.: An Introduction to Regression Graphics. Wiley, New York (1994)zbMATHCrossRefGoogle Scholar
  6. 6.
    Chang, Y., Huang, Y., Huang, Y.: Early Stopping in L2 Boosting. Comp. Stat. & Data Anal. 54(11), 2203–2213 (2010)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Fukumizu, K., Bach, F.R., Jordan, M.I.: Kernel Dimension Reduction in Regression. Ann. Statist. 37(1), 1871–1905 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  8. 8.
    Friedman, J.: Greedy Function Approximation: A Gradient Boosting machine. Ann. Statist. 29, 1189–1232 (2001)MathSciNetzbMATHCrossRefGoogle Scholar
  9. 9.
    Li, K.C.: Sliced Inverse Regression for Dimension Reduction (with discussion). J. Amer. Statist. Assoc. 86, 316–327 (1991)MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    Li, L., Cook, R.D., Tsai, C.L.: Partial Inverse Regression Method. Biometrika 94, 615–625 (2007)MathSciNetzbMATHCrossRefGoogle Scholar
  11. 11.
    Jolliffe, I.T.: Principal Component Analysis, 2nd edn. Springer, New York (2002)zbMATHGoogle Scholar
  12. 12.
    Xia, Y., Tong, H., Li, W.K., Zhu, L.X.: An Adaptive Estimation of Dimension Reduction Space. J. Roy. Statist. Soc. Ser. B 64, 363–410 (2002)MathSciNetzbMATHCrossRefGoogle Scholar
  13. 13.
    Zhang, T., Yu, B.: Boosting with Early Stopping: Convergence and Consistency. Ann. Statist. 33, 1538–1579 (2005)MathSciNetzbMATHCrossRefGoogle Scholar
  14. 14.
    Zhao, W., Chellappa, R., Phillips, P.J., Rosenfeld, A.: Face Recognition: A Literature Survey. ACM Computing Surveys 35(4), 399–459 (2003)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Junlong Zhao
    • 1
  1. 1.School of Mathematics and System ScienceBeihang University, LMIB of the Ministry of EducationBeijingChina

Personalised recommendations