Advertisement

Selective Bisection Debugging

  • Ripon SahaEmail author
  • Milos Gligoric
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10202)

Abstract

Bisection debugging, which is based on binary search over software version history, is widely used in practice to identify the bug introducing commit. However, this process can be expensive because it requires costly compilation and test execution at many commits. We introduce a novel technique—selective bisection (consisting of test selection and commit selection)— reduce the number of bisection steps, the number of compiler invocations, and the number of executed tests. We evaluated selective bisection on 10 popular open-source projects by performing 25,690 debugging sessions and measuring: (1) savings in number of compiler invocations obtained by commit selection, (2) savings in number of executed tests obtained by test selection, and (3) savings in overall debugging time by selective bisection. Our results show that, in 65% of debugging sessions, commit selection saves between 14% and 71% compiler invocations. Test selection saves 74% of testing effort on average (ranging from 42% to 95%) compared to when developers do not use any test selection. Finally, we demonstrate that one can save substantial time using selective bisection for large projects.

Keywords

Test Selection Source File Version Control System Stack Overflow Reference Version 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Notes

Acknowledgments

We thank Ahmet Celik, Julia Lawall, and Darko Marinov for their feedback on a draft of this paper. This research was partially supported by the NSF Grant No. CCF-1566363 and a Google Research Faculty Award.

References

  1. 1.
    Abreu, R., Zoeteweij, P., Golsteijn, R., Van Gemund, A.J.C.: A practical evaluation of spectrum-based fault localization. JSS 82(11), 1780–1792 (2009)Google Scholar
  2. 2.
    Finding culprits automatically in failing builds - i.e. who broke the build? GTAC (2013). https://www.youtube.com/watch?v=SZLuBYlq3OM
  3. 3.
    Couder, C.: Fighting regressions with git bisect. https://www.kernel.org/pub/software/scm/git/docs/git-bisect-lk2009.html
  4. 4.
    Dini, N., Sullivan, A., Gligoric, M., Rothermel, G.: The effect of test suite type on regression test selection. In: ISSRE, pp. 47–58 (2016)Google Scholar
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
    Gligoric, M., Eloussi, L., Marinov, D.: Ekstazi: lightweight test selection. In: ICSE Tool Demonstration Track, pp. 713–716 (2015)Google Scholar
  10. 10.
    Gligoric, M., Eloussi, L., Marinov, D.: Practical regression test selection with dynamic file dependencies. In: ISSTA, pp. 211–222 (2015)Google Scholar
  11. 11.
    Gligoric, M., Majumdar, R., Sharma, R., Eloussi, L., Marinov, D.: Regression test selection for distributed software histories. In: Biere, A., Bloem, R. (eds.) CAV 2014. LNCS, vol. 8559, pp. 293–309. Springer, Cham (2014). doi: 10.1007/978-3-319-08867-9_19 Google Scholar
  12. 12.
    Gligoric, M., Schulte, W., Prasad, C., van Velzen, D., Narasamdya, I., Livshits, B.: Automated migration of build scripts using dynamic analysis and search-based refactoring. In: OOPSLA, pp. 599–616 (2014)Google Scholar
  13. 13.
    Gross, T.R.: Bisection debugging. In: AADEBUG, pp. 185–191 (1997)Google Scholar
  14. 14.
    Herzig, K., Greiler, M., Czerwonka, J., Murphy, B.: The art of testing less without sacrificing quality. In: ICSE, pp. 483–493 (2015)Google Scholar
  15. 15.
  16. 16.
    Hovemeyer, D., Pugh, W.: Finding bugs is easy. SIGPLAN Note 39(12), 92–106 (2004)CrossRefGoogle Scholar
  17. 17.
    Jones, J.A., Harrold, M.J.: Empirical evaluation of the Tarantula automatic fault-localization technique. In: ASE, pp. 273–282 (2005)Google Scholar
  18. 18.
    Le, T.-DB., Oentaryo, R.J., Lo, D.: Information retrieval and spectrum based bug localization: better together. In: FSE, pp. 579–590 (2015)Google Scholar
  19. 19.
    Leung, H., White, L.: A cost model to compare regression test strategies. In: ICSM, pp. 201–208 (1991)Google Scholar
  20. 20.
    Leung, H.K.N., White, L.: Insights into regression testing. In: ICSM, pp. 60–69 (1989)Google Scholar
  21. 21.
    Liblit, B., Naik, M., Zheng, A.X., Aiken, A., Jordan, M.I.: Scalable statistical bug isolation. In: PLDI, pp. 15–26 (2005)Google Scholar
  22. 22.
    Lukins, S., Kraft, N., Etzkorn, L.: Bug localization using latent Dirichlet allocation. IST 52(9), 972–990 (2010)Google Scholar
  23. 23.
    Nanda, A., Mani, S., Sinha, S., Harrold, M.J., Orso, A.: Regression testing in the presence of non-code changes. In: ICST, pp. 21–30 (2011)Google Scholar
  24. 24.
    Ness, B., Ngo, V.: Regression containment through source change isolation. In: COMPSAC, pp. 616–621 (1997)Google Scholar
  25. 25.
    Nguyen, A.T., Nguyen, T.T., Al-Kofahi, J., Nguyen, H.V., Nguyen, T.: A topic-based approach for narrowing the search space of buggy files from a bug report. In: ASE, pp. 263–272 (2011)Google Scholar
  26. 26.
    Orso, A., Shi, N., Harrold, M.J.: Scaling regression testing to large software systems. ACM SIGSOFT Softw. Eng. Notes 29(6), 241–251 (2004)CrossRefGoogle Scholar
  27. 27.
    Pastore, F., Mariani, L., Goffi, A.: Radar: a tool for debugging regression problems in C/C++ software. In: ICSE Tool Demonstration Track, pp. 1335–1338 (2013)Google Scholar
  28. 28.
    Rao, S., Kak, A.: Retrieval from software libraries for bug localization: a comparative study of generic and composite text models. In: MSR, pp. 43–52 (2011)Google Scholar
  29. 29.
    Rothermel, G., Harrold, M.: A safe, efficient regression test selection technique. TOSEM 6(2), 173–210 (1997)CrossRefGoogle Scholar
  30. 30.
    Rothermel, G., Harrold, M.J.: A safe, efficient algorithm for regression test selection. In: ICSM, pp. 358–367 (1993)Google Scholar
  31. 31.
    Rothermel, G., Harrold, M.J.: A framework for evaluating regression test selection techniques. In: ICSE, pp. 201–210 (1994)Google Scholar
  32. 32.
    Rothermel, G., Harrold, M.J.: Analyzing regression test selection techniques. TSE 22(8), 529–551 (1996)Google Scholar
  33. 33.
    Saha, R.K., Lease, M., Khurshid, S., Perry, D.E.: Improving bug localization using structured information retrieval. In: ASE, pp. 345–355 (2013)Google Scholar
  34. 34.
    Shi, A., Gyori, A., Gligoric, M., Zaytsev, A., Marinov, D.: Balancing trade-offs in test-suite reduction. In: FSE, pp. 246–256 (2014)Google Scholar
  35. 35.
    Streamline testing process with test impact analysis. http://msdn.microsoft.com/en-us/library/ff576128%28v=vs.100%29.aspx
  36. 36.
  37. 37.
    Tools for continuous integration at Google scale. http://www.youtube.com/watch?v=b52aXZ2yi08
  38. 38.
    Yoo, S., Harman, M.: Pareto efficient multi-objective test case selection. In: ISSTA, pp. 140–150 (2007)Google Scholar
  39. 39.
    Yoo, S., Harman, M.: Regression testing minimization, selection and prioritization: a survey. STVR 22(2), 67–120 (2012)Google Scholar
  40. 40.
    Yu, Y., Dayani-Fard, H., Mylopoulos, J.: Removing false code dependencies to speedup software build processes. In: CASCON, pp. 343–352 (2003)Google Scholar
  41. 41.
    Zeller, A.: Yesterday, my program worked. Today, it does not. Why? In: Nierstrasz, O., Lemoine, M. (eds.) ESEC/SIGSOFT FSE -1999. LNCS, vol. 1687, pp. 253–267. Springer, Heidelberg (1999). doi: 10.1007/3-540-48166-4_16 CrossRefGoogle Scholar
  42. 42.
    Zhang, L., Kim, M., Khurshid, S.: Localizing failure-inducing program edits based on spectrum information. In: ICSM, pp. 23–32 (2011)Google Scholar
  43. 43.
    Zhang, X., He, H., Gupta, N., Gupta, R.: Experimental evaluation of using dynamic slices for fault location. In: AADEBUG, pp. 33–42 (2005)Google Scholar
  44. 44.
    Zheng, J., Robinson, B., Williams, L., Smiley, K.: An initial study of a lightweight process for change identification and regression test selection when source code is not available. In: ISSRE, pp. 225–234 (2005)Google Scholar
  45. 45.
    Zhou, J., Zhang, H., Lo, D.: Where should the bugs be fixed? More accurate information retrieval based bug localization based on bug reports. In: ICSE, pp. 14–24 (2012)Google Scholar
  46. 46.
    Zuddas, D., Jin, W., Pastore, F., Mariani, L., Orso, A.: Mimic: locating and understanding bugs by analyzing mimicked executions. In: ASE, pp. 815–826 (2014)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany 2017

Authors and Affiliations

  1. 1.Fujitsu Laboratories of AmericaSunnyvaleUSA
  2. 2.The University of Texas at AustinAustinUSA

Personalised recommendations