Empirical Software Engineering

, Volume 15, Issue 3, pp 277–295 | Cite as

Comparing the effectiveness of several modeling methods for fault prediction

  • Elaine J. Weyuker
  • Thomas J. Ostrand
  • Robert M. Bell


We compare the effectiveness of four modeling methods—negative binomial regression, recursive partitioning, random forests and Bayesian additive regression trees—for predicting the files likely to contain the most faults for 28 to 35 releases of three large industrial software systems. Predictor variables included lines of code, file age, faults in the previous release, changes in the previous two releases, and programming language. To compare the effectiveness of the different models, we use two metrics—the percent of faults contained in the top 20% of files identified by the model, and a new, more general metric, the fault-percentile-average. The negative binomial regression and random forests models performed significantly better than recursive partitioning and Bayesian additive regression trees, as assessed by either of the metrics. For each of the three systems, the negative binomial and random forests models identified 20% of the files in each release that contained an average of 76% to 94% of the faults.


Empirical study Fault prediction Negative binomial Recursive partitioning Random forests Bayesian trees Fault-percentile-average 


  1. Adams EN (1984) Optimizing preventive service of software products. IBM J Res Develop 28(1):2–14CrossRefGoogle Scholar
  2. Arisholm E, Briand LC (2006) Predicting fault-prone components in a java legacy system. In: Proc ACM/IEEE ISESE, Rio de JaneiroGoogle Scholar
  3. Basili VR, Perricone BT (1984) Software errors and complexity: an empirical investigation. Commun ACM 27(1):42–52CrossRefGoogle Scholar
  4. Bell RM, Ostrand TJ, Weyuker EJ (2006) Looking for bugs in all the right places. In: Proc ACM/international symposium on software testing and analysis (ISSTA2006), Portland, pp 61–71Google Scholar
  5. Breiman L (2001) Random forests. Mach Learn 45:5–32MATHCrossRefGoogle Scholar
  6. Breiman L, Friedman JH, Olshen RA, Stone CJ (1984) Classification and regression trees. Wadsworth, BelmontMATHGoogle Scholar
  7. Chipman HA, George EI, McCulloch RE (2008) BART: Bayesian additive regression trees. http://arxiv.org/abs/0806.3286v1
  8. Denaro G, Pezze M (2002) An empirical evaluation of fault-proneness models. In: Proc international conf on software engineering (ICSE2002), MiamiGoogle Scholar
  9. Eick SG, Graves TL, Karr AF, Marron JS, Mockus A (2001) Does code decay? Assessing the evidence from change management data. IEEE Trans Softw Eng 27(1):1–12CrossRefGoogle Scholar
  10. Fenton NE, Ohlsson N (2000) Quantitative analysis of faults and failures in a complex software system. IEEE Trans Softw Eng 26(8):797–814CrossRefGoogle Scholar
  11. Friedman JH (1991) Multivariate adaptive regression splines. Ann Stat 19:1–67MATHCrossRefGoogle Scholar
  12. Guo L, Ma Y, Cukic B, Singh H (2004) Robust prediction of fault-proneness by random forests. In: Proc ISSRE 2004, Saint-MaloGoogle Scholar
  13. Hatton L (1997) Reexamining the fault density—component size connection. IEEE Softw 14:89–97CrossRefGoogle Scholar
  14. Jiang Y, Cukic B, Ma Y (2008) Techniques for evaluating fault prediction models. Empir Softw Eng 13:561–595CrossRefGoogle Scholar
  15. Khoshgoftaar TM, Allen EB, Kalaichelvan KS, Goel N (1996) Early quality prediction: a case study in telecommunications. IEEE Softw 13:65–71CrossRefGoogle Scholar
  16. Khoshgoftaar TM, Allen EB, Deng J (2002) Using regression trees to classify fault-prone software modules. IEEE Trans Reliab 51(4):455–462CrossRefGoogle Scholar
  17. Koru AG, Liu H (2005) An investigation of the effect of module size on defect prediction using static measures. In: 2005 Promise workshop, 15 May 2005Google Scholar
  18. Lessmann S, Baesens B, Ues C, Pietsch S (2008) Benchmarking classification models for software defect prediction: a proposed framework and novel findings. IEEE Trans Softw Eng 34(4):485–496CrossRefGoogle Scholar
  19. McCullagh P, Nelder JA (1989) Generalized linear models, 2nd edn. Chapman and Hall, LondonMATHGoogle Scholar
  20. Menzies T, Di Stefano JS, Cunanan C, Chapman R (2004) Mining repositories to assist in project planning and resource allocation. In: International workshop on mining software repositoriesGoogle Scholar
  21. Moeller K-H, Paulish DJ (1993) An empirical investigation of software fault distribution. In: Proc IEEE first international software metrics symposium, Baltimore, 21–22 May 1993, pp 82–90Google Scholar
  22. Munson JC, Khoshgoftaar TM (1992) The detection of fault-prone programs. IEEE Trans Softw Eng 18(5):423–433CrossRefGoogle Scholar
  23. Ohlsson N, Alberg H (1996) Predicting fault-prone software modules in telephone switches. IEEE Trans Softw Eng 22(12):886–894CrossRefGoogle Scholar
  24. Ostrand T, Weyuker EJ (2002) The distribution of faults in a large industrial software system. In: Proc ACM/international symposium on software testing and analysis (ISSTA2002), Rome, pp 55–64Google Scholar
  25. Ostrand T, Weyuker EJ (2007) How to measure success of software prediction models. In: Proc fourth international workshop on software quality assurance, Dubrovnik, September 2007Google Scholar
  26. Ostrand TJ, Weyuker EJ, Bell RM (2005) Predicting the location and number of faults in large software systems. IEEE Trans Softw Eng 31(4):340–355CrossRefGoogle Scholar
  27. Ostrand TJ, Weyuker EJ, Bell RM (2007) Automating algorithms for the identification of fault-prone files. In: Proc ACM/international symposium on software testing and analysis (ISSTA07), LondonGoogle Scholar
  28. Pighin M, Marzona A (2003) An empirical analysis of fault persistence through software releases. In: Proc IEEE/ACM ISESE 2003, pp 206–212Google Scholar
  29. Succi G, Pedrycz W, Stefanovic M, Miller J (2003) Practical assessment of the models for identification of defect-prone classes in object-oriented commercial systems using design metrics. J Syst Softw 65(1):1–12CrossRefGoogle Scholar
  30. White H (1980) A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrics 48:817–838MATHCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  • Elaine J. Weyuker
    • 1
  • Thomas J. Ostrand
    • 1
  • Robert M. Bell
    • 1
  1. 1.AT&T Labs - ResearchFlorham ParkUSA

Personalised recommendations