Mutation 2000: Uniting the Orthogonal

  • A. Jefferson Offutt
  • Roland H. Untch
Chapter
Part of the The Springer International Series on Advances in Database Systems book series (ADBS, volume 24)

Abstract

Mutation testing is a powerful, but computationally expensive, technique for unit testing software. This expense has prevented mutation from becoming widely used in practical situations, but recent engineering advances have given us techniques and algorithms for significantly reducing the cost of mutation testing. These techniques include a new algorithmic execution technique called schema-based mutation, an approximation technique called weak mutation, a reduction technique called selective mutation, heuristics for detecting equivalent mutants, and algorithms for automatic test data generation. This paper reviews experimentation with these advances and outlines a design for a system that will approximate mutation, but in a way that will be accessible to everyday programmers. We envision a system to which a programmer can submit a program unit and get back a set of input/output pairs that are guaranteed to form an effective test of the unit by being close to mutation adequate. We believe this system could be efficient enough to be adopted by leading-edge software developers. Full automation in unit testing has the potential to dramatically change the economic balance between testing and development, by reducing the cost of testing from the major part of the total development cost to a small fraction.

Keywords

IEEE Computer Society Software Test Mutation Testing Original Program Selective Mutation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [1]
    T. Budd and F. Sayward, “Users guide to the Pilot mutation system,” technical report 114, Department of Computer Science, Yale University, 1977.Google Scholar
  2. [2]
    R. G. Hamlet, “Testing programs with the aid of a compiler,” IEEE Transactions on Software Engineering, vol. 3, pp. 279–290, July 1977.Google Scholar
  3. [3]
    R. A. DeMillo, R. J. Lipton, and F. G. Sayward, “Hints on test data selection: Help for the practicing programmer,” IEEE Computer, vol. 11, pp. 34–41, April 1978.CrossRefGoogle Scholar
  4. [4]
    T. A. Budd, R. A. DeMillo, R. J. Lipton, and F. G. Sayward, “The design of a prototype mutation system for program testing,” in Proceedings NCC, AFIPS Conference Record, pp. 623–627, 1978.Google Scholar
  5. [5]
    R. J. Lipton and F. G. Sayward, “The status of research on program mutation,” in Digest for the Workshop on Software Testing and Test Documentation, pp. 355–373, December 1978.Google Scholar
  6. [6]
    A. T. Acree, T. A. Budd, R. A. DeMillo, R. J. Lipton, and F. G. Sayward, “Mutation analysis,” technical report GITICS-79/08, School of Information and Computer Science, Georgia Institute of Technology, Atlanta GA, September 1979.Google Scholar
  7. [7]
    A. J. Offutt and K. N. King, “A Fortran 77 interpreter for mutation analysis,” in 1987 Symposium on Interpreters and Interpretive Techniques, (St. Paul MN), pp. 177–188, ACM SIGPLAN, June 1987.Google Scholar
  8. [8]
    R. A. DeMillo, D. S. Guindi, K. N. King, W. M. McCracken, and A. J. Offutt, “An extended overview of the Mothra software testing environment,” in Proceedings of the Second Workshop on Software Testing, Verification, and Analysis, (Banff Alberta), pp. 142–151, IEEE Computer Society Press, July 1988.Google Scholar
  9. [9]
    A. J. Offutt, Automatic Test Data Generation. PhD thesis, Georgia Institute of Technology, Atlanta GA, 1988. Technical report GIT-ICS 88/28.Google Scholar
  10. [10]
    R. A. DeMillo and A. J. Offutt, “Constraint-based automatic test data generation,” IEEE Transactions on Software Engineering, vol. 17, pp. 900–910, September 1991.MathSciNetGoogle Scholar
  11. [11]
    R. A. DeMillo, E. W. Krauser, and A. P. Mathur, “Compiler-integrated program mutation,” in Proceedings of the Fifteenth Annual Computer Software and Applications Conference (COMPSAC’ 92), (Tokyo, Japan), Kogakuin University, IEEE Computer Society Press, September 1991.Google Scholar
  12. [12]
    R. H. Untch, M. J. Harrold, and J. Offutt, “Schema-based mutation analysis.” In preparation.Google Scholar
  13. [13]
    M. E. Delamaro and J. C. Maldonado, “Proteum-A tool for the assessment of test adequacy for c programs,” in Proceedings of the Conference on Performability in Computing Systems (PCS 96), (New Brunswick, NJ), pp. 79–95, July 1996.Google Scholar
  14. [14]
    A. J. Offutt, “Investigations of the software testing coupling effect,” ACM Transactions on Software Engineering Methodology, vol. 1, pp. 3–18, January 1992.Google Scholar
  15. [15]
    K. S. H. T. Wah, “Fault coupling in finite bijective functions,” The Journal of Software Testing, Verification, and Reliability, vol. 5, pp. 3–47, March 1995.CrossRefGoogle Scholar
  16. [16]
    K. S. H. T. Wah, “A theoretical study of fault coupling,” The Journal of Software Testing, Verification, and Reliability, vol. 10, pp. 3–46, March 2000.CrossRefGoogle Scholar
  17. (17]
    D. Wu, M. A. Hennell, D. Hedley, and I. J. Riddell, “A practical method for software quality control via program mutation,” in Proceedings of the Second Workshop on Software Testing, Verification, and Analysis, (Banff, Alberta, Canada), pp. 159–170, IEEE Computer Society Press, July 1988.Google Scholar
  18. [18]
    IEEE, IEEE Standard Glossary of Software Engineering Terminology. ANSI/IEEE Std 610.12–1990,1996.Google Scholar
  19. [19]
    R. Geist, A. J. Offutt, and F. Harris, “Estimation and enhancement of real-time software reliability through mutation analysis,” IEEE Transactions on Computers, vol. 41, pp. 550–558, May 1992. Special Issue on Fault-Tolerant Computing.Google Scholar
  20. [20]
    A. J. Offutt, A. Lee, G. Rothermel, R. Untch, and C. Zapf, “An experimental determination of sufficient mutation operators,” ACM Transactions on Software Engineering Methodology, vol. 5, pp. 99–118, April 1996.CrossRefGoogle Scholar
  21. [21]
    W. E. Wong, M. E. Delamaro, J. C. Maldonado, and A. P. Mathur, “Constrained mutation in C programs,” in Proceedings of the 8th Brazilian Symposium on Software Engineering, (Curitiba, Brazil), pp. 439–452, October 1994.Google Scholar
  22. [22]
    A. J. Offutt, G. Rothermel, and C. Zapf, “An experimental evaluation of selective mutation,” in Proceedings of the Fifteenth International Conference on Software Engineering, (Baltimore, MD), pp. 100–107, IEEE Computer Society Press, May 1993.Google Scholar
  23. [23]
    A. T. Acree, On Mutation. PhD thesis, Georgia Institute of Technology, Atlanta GA, 1980.Google Scholar
  24. [24]
    T. A. Budd, Mutation Analysis of Program Test Data. PhD thesis, Yale University, New Haven CT, 1980.Google Scholar
  25. [25]
    W. E. Wong, On Mutation and Data Flow. PhD thesis, Purdue University, December 1993. (Also Technical Report SERC-TR-149-P, Software Engineering Research Center, Purdue University, West Lafayette, IN).Google Scholar
  26. [26]
    M. Sahinoglu and E. H. Spafford, “A bayes sequential statistical procedure for approving software products,” in Proccedings of the IFIP Conference on Approving Software Products (ASP-90) (W. Ehrenberger, ed.), ( Garmisch-Partenkirchen, Germany ), pp. 43–56, Elsevier/North Holland, New York, Sept. 1990.Google Scholar
  27. [27]
    W. E. Howden, “Weak mutation testing and completeness of test sets,” IEEE Transactions on Software Engineering, vol. 8, pp. 371–379, July 1982.Google Scholar
  28. [28]
    L. J. Morel’, “Theoretical insights into fault-based testing,” in Proceedings of the Second Workshop on Software Testing, Verification, and Analysis, (Banff Alberta), pp. 45–62, IEEE Computer Society Press, July 1988.Google Scholar
  29. [29]
    M. R. Woodward and K. Halewood, “From weak to strong, dead or alive? An analysis of some mutation testing issues,” in Proceedings of the Second Workshop on Software Testing, Verification, and Analysis, (Banff Alberta), pp. 152–158, IEEE Computer Society Press, July 1988.Google Scholar
  30. [30]
    J. R. Horgan and A. P. Mathur, “Weak mutation is probably strong mutation,” technical report SERC-TR-83-P, Software Engineering Research Center, Purdue University, West Lafayette IN, December 1990.Google Scholar
  31. [31]
    M. R. Girgis and M. R. Woodward, “An integrated system for program testing using weak mutation and data flow analysis,” in Proceedings of the Eighth International Conference on Software Engineering, (London UK), pp. 313–319, IEEE Computer Society Press, August 1985.Google Scholar
  32. [32]
    B. Marick, “Two experiments in software testing,” technical report UIUCDCS-R-90–1644, Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana-Champaign Illinois, November 1990.Google Scholar
  33. [33]
    B. Marick, “The weak mutation hypothesis,” in Proceedings of the Fourth Symposium ors Software Testing, Analysis, and Verification, (Victoria, British Columbia, Canada), pp. 190–199, IEEE Computer Society Press, October 1991.Google Scholar
  34. [34]
    A. J. Offutt and S. D. Lee, “An empirical evaluation of weak mutation,” IEEE Transactions on Software Engineering, vol. 20, pp. 337–344, May 1994.Google Scholar
  35. [35]
    A. J. Offutt and S. D. Lee, “How strong is weak mutation?,” in Proceedings of the Fourth Symposium on Software Testing, Analysis, and Verification, (Victoria, British Columbia, Canada), pp. 200–213, IEEE Computer Society Press, October 1991.Google Scholar
  36. [36]
    A. P. Mathur and E. W. Krauser, “Mutant unification for improved vectorization,” technical report SERC-TR-14-P, Software Engineering Research Center, Purdue University, West Lafayette IN, April 1988.Google Scholar
  37. [37]
    E. W. Krauser, A. P. Mathur, and V. Rego, “High performance testing on SIMD machines,” in Proceedings of the Second Workshop on Software Testing, Verification, and Analysis, (Banff Alberta), pp. 171–177, IEEE Computer Society Press, July 1988.Google Scholar
  38. [38]
    A. J. Offutt, R. Pargas, S. V. Fichter, and P. Khambekar, “Mutation testing of software using a mimd computer,” in 1992 International Conference on Parallel Processing, (Chicago, Illinois), pp. II-257–266, August 1992.Google Scholar
  39. [39]
    B. Choi and A. P. Mathur, “High-performance mutation testing,” The Journal of Systems and Software, vol. 20, pp. 135–152, February 1993.CrossRefGoogle Scholar
  40. [40]
    C. N. Zapf, “Medusamothra - a distributed interpreter for the mothra mutation testing system,” M.S. thesis, Clemson University, Clemson, SC, August 1993.Google Scholar
  41. [41]
    V. N. Fleyshgakker and S. N. Weiss, “Efficient Mutation Analysis: A New Approach,” in Proceedings of the International Symposium on Software Testing and Analysis (ISSTA 94), (Seattle, WA), pp. 185–195, ACM SIGSOFT, ACM Press, Aug. 17–19 1994.Google Scholar
  42. [42]
    R. Untch, A. J. Offutt, and M. J. Harrold, “Mutation analysis using program schemata,” in Proceedings of the 1993 International Symposium on Software Testing, and Analysis, (Cambridge MA), pp. 139–148, June 1993.Google Scholar
  43. [43]
    R. A. DeMillo and A. J. Offutt, “Experimental results from an automatic test case generator,” ACM Transactions on Software Engineering Methodology, vol. 2, pp. 109–127, April 1993.Google Scholar
  44. [44]
    J. Offutt, Z. Jin, and J. Pan, “The dynamic domain reduction approach for test data generation: Design and algorithms,” technical report ISSE-TR-94–110, Department of Information and Software Systems Engineering, George Mason University, Fairfax VA, September 1994.Google Scholar
  45. [45]
    J. Offutt, Z. Jin, and J. Pan, “The dynamic domain reduction approach to test data generation,” Software-Practice and Experience, vol. 29, pp. 167–193, January 1999.CrossRefGoogle Scholar
  46. [46]
    B. Korel, “Automated software test data generation,” IEEE Transactions on Software Engineering, vol. 16, pp. 870–879, August 1990.Google Scholar
  47. [47]
    B. Korel, “Dynamic method for software test data generation,” The Journal of Software Testing, Verification, and Reliability, vol. 2, no. 4, pp. 203–213, 1992.CrossRefGoogle Scholar
  48. [48]
    L. A. Clarke, “A system to generate test data and symbolically execute programs,” IEEE Transactions on Software Engineering, vol. 2, pp. 215–222, September 1976.CrossRefGoogle Scholar
  49. [49]
    L. A. Clarke and D. J. Richardson, “Applications of symbolic evaluation,” The Journal of Systems and Software, vol. 5, pp. 15–35, January 1985.CrossRefGoogle Scholar
  50. [50]
    R. E. Fairley, “An experimental program testing facility,” IEEE Transactions on Software Engineering, vol. SE-1, pp. 350–3571, December 1975.Google Scholar
  51. [51]
    T. A. Budd and D. Angluin, “Two notions of correctness and their relation to testing,” Acta Informatica, vol. 18, pp. 31–45, November 1982.MathSciNetMATHCrossRefGoogle Scholar
  52. [52]
    D. Baldwin and F. Sayward, “Heuristics for determining equivalence of program mutations,” research report 276, Department of Computer Science, Yale University, 1979.Google Scholar
  53. [53]
    A. Tanaka, “Equivalence testing for fortran mutation system using data flow analysis,” Master’s thesis, School of Information and Computer Science, Georgia Institute of Technology, Atlanta GA, 1981.Google Scholar
  54. [54]
    A. J. Offutt and W. M. Craft, “Using compiler optimization techniques to detect equivalent mutants,” The Journal of Software Testing, Verification, and Reliability, vol. 4, pp. 131–154, September 1994.CrossRefGoogle Scholar
  55. [55]
    A. J. Offutt and J. Pan, “Detecting equivalent mutants and the feasible path problem,” in Proceedings of the 1996 Annual Conference on Computer Assurance (COMPASS 96), (Gaithersburg MD), pp. 224–236, IEEE Computer Society Press, June 1996.Google Scholar
  56. [56]
    A. J. Offutt and J. Pan, “Detecting equivalent mutants and the feasible path problem,” The Journal of Software Testing, Verification, and Reliability, vol. 7, pp. 165–192, September 1997.CrossRefGoogle Scholar
  57. [57]
    R. Hierons, M. Harman, and S. Danicic, “Using program slicing to assist in the detection of equivalent mutants,” Software Testing, Verification, and Reliability, vol. 9, pp. 233–262, December 1999.CrossRefGoogle Scholar
  58. [58]
    E. Mikk, “Compilation of z specifications into c for automatic test result evaluation,” in 9th International Conference of Z Users (ZUM’95), (Limerick, Ireland), pp. 167180, Springer-Verlag Lecture Notes in Computer Science Volume 967, J.P. Bowen and M.G. Hinchey (Eds.), September 1995.Google Scholar
  59. [59]
    D. K. Peters and D. L. Parnas, “Using test oracles generated from program documentation,” IEEE Transactions on Software Engineering, vol. 24, pp. 161–173, March 1998.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2001

Authors and Affiliations

  • A. Jefferson Offutt
    • 1
  • Roland H. Untch
    • 2
  1. 1.ISE Department, 4A4George Mason UniversityFairfaxUSA
  2. 2.Department of Computer ScienceMiddle Tennessee State UniversityMurfreesboroUSA

Personalised recommendations