Skip to main content
SpringerLink
Log in

Fault localization for automated program repair: effectiveness, performance, repair correctness

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Automated program repair (APR) tools apply fault localization (FL) techniques to identify the locations of likely faults to be repaired. The effectiveness, performance, and repair correctness of APR depends in part on the FL method used. If FL does not identify the location of a fault, the application of an APR tool will not be effective—it will fail to repair the fault. If FL assigns the actual faulty statement a low priority for repair, APR performance will be reduced by increasing the time required to find a potential repair. In addition, the correctness of a generated repair will be decreased since APR will modify fault-free statements that are assigned a higher priority for repair than an actual faulty statement. We conducted a controlled experiment to evaluate the impact of ten FL techniques on APR effectiveness, performance, and repair correctness using a brute force APR tool applied to faulty versions of the Siemens Suite and two other large programs: space and sed. All FL techniques were effective in identifying all faults; however, Wong3 and Ample1 were the least effective FL techniques since they assigned the lowest priority for repair in more than 26 % of the trials. We obtained the worst APR performance significantly when Ample1 was used since it generated a large number of variants in 29.11 % of the trials, and took the longest time to produce potential repairs. Jaccard FL improved repair correctness by generating more validated repairs–potential repairs that pass a set of regression tests, and generating potential repairs that failed fewer regression tests. Also Jaccard’s performance is noteworthy in that it never generated a large number of variants during the repair process compared to the alternatives.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  • Abreu, R., Zoeteweij, P., Golsteijn, R., & Van Gemund, A. J. (2009). A practical evaluation of spectrum-based fault localization. Journal of Systems and Software, 82(11), 1780–1792.

    Article  Google Scholar 

  • Abreu, R., Zoeteweij, P., & Van Gemund, A. J. (2006). An evaluation of similarity coefficients for software fault localization. In: 12th Pacific Rim international symposium on dependable computing, 2006 (PRDC’06) (pp. 39–46). IEEE.

  • Abreu, R., Zoeteweij, P., & Van Gemund, A. J. (2007). On the accuracy of spectrum-based fault localization. In Testing: Academic and industrial conference practice and research techniques—MUTATION, 2007. (TAICPART-MUTATION 2007) (pp. 89–98)

  • Ackling, T., Alexander, B., & Grunert, I. (2011). Evolving patches for software repair. In Proceedings of the 13th annual conference on Genetic and evolutionary computation, GECCO ’11 (pp. 1427–1434). New York, NY: ACM.

  • Agrawal, H., Horgan, J. R., London, S., & Wong, W. E. (1995) Fault localization using execution slices and dataflow tests. In Proceedings of the sixth international symposium on software reliability engineering, 1995 (pp. 143–151)

  • Arcuri, A. (2008). On the automation of fixing software bugs. Companion of the 30th international conference on software engineering, ICSE Companion ’08 (pp. 1003–1006). New York, NY: ACM.

  • Arcuri, A. (2011). Evolutionary repair of faulty software. Applied Soft Computing, 11(4), 3494–3514.

    Article  Google Scholar 

  • Assiri, F. Y. (2015) Assessment and improvement of automated program repair mechanisms and components. Ph.D. dissertation, Colorado State University.

  • Assiri, F. Y., & Bieman, J. M. (2014). An assessment of the quality of automated program operator repair. In Proceedings of the 2014 ICST conference, ICST ’14.

  • Baah, G. K., Podgurski, A., & Harrold, M. J. (2010). The probabilistic program dependence graph and its application to fault diagnosis. IEEE Transactions on Software Engineering, 36(4), 528–545.

    Article  Google Scholar 

  • Brown, H., & Prescott, R. (2006). Applied mixed models in medicine. Wiley Online Library.

  • Chen, M. Y., Kiciman, E., Fratkin, E., Fox, A., & Brewer, E. (2002). Pinpoint: Problem determination in large, dynamic Internet services. In Proceedings of the international conference on dependable systems and networks, 2002, DSN 2002 (pp. 595–604). IEEE.

  • Chilimbi, T. M., Liblit, B., Mehra, K., Nori, A. V., & Vaswani, K. (2009). Holmes: Effective statistical debugging via efficient path profiling. In IEEE 31st international conference on software engineering, 2009, ICSE 2009 (pp. 34–44). IEEE.

  • Dallmeier, V., Lindig, C., & Zeller, A. (2005). Lightweight defect localization for Java. In Proceedings of the european conference on object-oriented programming, ECOOP 2005 (pp. 528–550). Berlin: Springer.

  • Dallmeier, V., & Zimmermann, T. (2007). Extraction of bug localization benchmarks from history. In Proceedings of the twenty-second IEEE/ACM international conference on automated software engineering (pp. 433–436). ACM.

  • Debroy, V., & Wong, W. E. (2010). Using mutation to automatically suggest fixes for faulty programs. In: 2010 third international conference on software testing, verification and validation (ICST) (pp. 65–74).

  • Debroy, V., & Wong, W. E. (2014). Combining mutation and fault localization for automated program debugging. Journal of Systems and Software, 90, 45–60.

    Article  Google Scholar 

  • Delamaro, M. E., & Maldonado, J. C. (2001). Proteum/IM 2.0: An integrated mutation testing environment. In Mutation testing for the new century (pp. 91–101). Norwell, MA: Kluwer Academic Publishers.

  • Forrest, S., Nguyen, T., Weimer, W., & Le Goues, C. (2009). A genetic programming approach to automated software repair. Proceedings of the 11th annual conference on genetic and evolutionary computation, GECCO ’09 (pp. 947–954). New York, NY: ACM.

  • Hailpern, B., & Santhanam, P. (2002). Software debugging, testing, and verification. IBM Systems Journal, 41(1), 4–12.

    Article  Google Scholar 

  • Jones, J. A., & Harrold, M. J. (2005). Empirical evaluation of the Tarantula automatic fault-localization technique. In Proceedings of the 20th IEEE/ACM international conference on automated software engineering, ASE ’05 (pp. 273–282). New York, NY: ACM.

  • Jones, J. A., Harrold, M. J., & Stasko, J. (2002). Visualization of test information to assist fault localization. In Proceedings of the 24th international conference on software engineering (pp. 467–477). ACM.

  • Jones, J. A., Harrold, M. J., Stasko, J. T.(2001). Visualization for fault localization. In Proceedings of ICSE 2001 workshop on software visualization (pp. 71–75). Toronto, ON: Citeseer.

  • Kim, D., Nam, J., Song, J., & Kim, S. (2013). Automatic patch generation learned from human-written patches. In Proceedings of the 2013 international conference on software engineering (pp. 802–811). IEEE Press.

  • Le Goues, C., Nguyen, T., Forrest, S., & Weimer, W. (2012). GenProg: A generic method for automatic software repair. IEEE Transactions on Software Engineering, 38(1), 54–72.

    Article  Google Scholar 

  • Le Goues, C., Weimer, W., & Forrest, S. (2012). Representations and operators for improving evolutionary software repair. In Proceedings of the fourteenth international conference on genetic and evolutionary computation conference, GECCO ’12 (pp. 959–966). New York, NY: ACM.

  • Lee, H. J., Naish, L., & Ramamohanarao, K. (2009). Study of the relationship of bug consistency with respect to performance of spectra metrics. In 2nd IEEE international conference on computer science and information technology, 2009, ICCSIT 2009 (pp. 501–508).

  • Lo, D., Jiang, L., & Budi, A. (2010). Comprehensive evaluation of association measures for fault localization. In: 2010 IEEE international conference on software maintenance (ICSM) (pp. 1–10). IEEE.

  • Meyer, A. S., Garcia, A. A. F., Souza, A. P., & Souza Jr, C. L. (2004). Comparison of similarity coefficients used for cluster analysis with dominant markers in maize (Zea mays L.). Genetics and Molecular Biology, 27(1), 83–91.

    Article  Google Scholar 

  • Naish, L., Lee, H. J., & Ramamohanarao, K. (2011). A model for spectra-based software diagnosis. ACM Transactions on Software Engineering Methodology, 20(3), 11:1–11:32.

    Article  Google Scholar 

  • National Vulnerability Database. http://nvd.nist.org

  • Nguyen, H. D. T., Qi, D., Roychoudhury, A., & Chandra, S. (2013). Semfix: Program repair via semantic analysis. In Proceedings of the 2013 international conference on software engineering (pp. 772–781). IEEE Press.

  • Purushothaman, R., & Perry, D. E. (2005). Toward understanding the rhetoric of small source code changes. IEEE Transactions on Software Engineering, 31, 511–526.

    Article  Google Scholar 

  • Qi, Y., Mao, X., Lei, Y., & Wang, C. (2013). Using automated program repair for evaluating the effectiveness of fault localization techniques. Proceedings of the 2013 international symposium on software testing and analysis, ISSTA 2013 (pp. 191–201). New York, NY: ACM.

  • Renieres, M., & Reiss, S. P. (2003). Fault localization with nearest neighbor queries. In Proceedings of the 18th IEEE international conference on automated software engineering 2003 (pp. 30–39). IEEE.

  • Software-artifact Infrastructure Repository. http://sir.unl.edu/php/previewfiles.php.

  • Wei, Y., Pei, Y., Furia, C. A., Silva, L. S., Buchholz, S., Meyer, B., et al. (2010). Automated fixing of programs with contracts. In Proceedings of the 19th international symposium on software testing and analysis, ISSTA ’10 (pp. 61–72). New York, NY: ACM.

  • Weimer, W., Forrest, S., Le Goues, C., & Nguyen, T. (2010). Automatic program repair with evolutionary computation. Communications of the ACM, 53(5), 109–116.

    Article  Google Scholar 

  • Weimer, W., Nguyen, T., Le Goues, C., & Forrest, S. (2009). Automatically finding patches using genetic programming. In Proceedings of the 31st international conference on software engineering, ICSE ’09 (pp. 364–374). Washington, DC: IEEE Computer Society.

  • Xie, X., Chen, T. Y., Kuo, F. C., & Xu, B. (2013). A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization. ACM Transactions on Software Engineering Methodology, 22(4), 31:1–31:40.

    Article  Google Scholar 

Download references

Acknowledgments

This project was supported by the Ministry of Higher Education, Saudi Arabia. The authors would like to thank Westley Weimer and his research group for sharing GenProg tool and Qi et al. (2013) for sharing the GenProg-FL tool.

Author information

Authors and Affiliations

  1. CS Department, Colorado State University, Fort Collins, CO, 80523-1873, USA

    Fatmah Yousef Assiri & James M. Bieman

Authors
  1. Fatmah Yousef Assiri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. James M. Bieman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fatmah Yousef Assiri.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Assiri, F.Y., Bieman, J.M. Fault localization for automated program repair: effectiveness, performance, repair correctness. Software Qual J 25, 171–199 (2017). https://doi.org/10.1007/s11219-016-9312-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-016-9312-z

Keywords

Search

Navigation