Skip to main content
SpringerLink
Log in
Book cover

International Workshop on Abstract State Machines

ASM 2003: Abstract State Machines 2003 pp 1–21Cite as

Software Testing Research and Practice

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 2589))

Abstract

The paper attempts to provide a comprehensive view of the field of software testing. The objective is to put all the relevant issues into a unified context, although admittedly the overview is biased towards my own research and expertise. In view of the vastness of the field, for each topic problems and approaches are only briefly tackled, with appropriate references provided to dive into them. I do not mean to give here a complete survey of software testing. Rather I intend to show how an unwieldy mix of theoretical and technical problems challenge software testers, and that a large gap exists between the state of the art and of the practice.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Adams, E.: Optimizing Preventive Service of Software Products. IBM J. of Research and Development 281 (1984) 2–14 4

    Google Scholar 

  2. Agedis: Automated Generation and Execution of Test Suites for DIstributed Component-based Software. <http://www.agedis.de/index.shtml> 8

  3. Bache, R., Müllerburg, M.: Measures of Testability as a Basis for Quality Assurance. Software Engineering Journal. 5 (March 1990) 86–92 14

    Google Scholar 

  4. Baresi, L., Young, M.: Test Oracles. Tech. Report CIS-TR-01-02. On line at <http://www.cs.uoregon.edu/ michal/pubs/oracles.html> 11

  5. Basanieri, F., Bertolino, A., Marchetti, E.: The Cow Suite Approach to Planning and Deriving Test Suites in UML Projects. Proc. 5th Int. Conf. UML 2002, Dresden, Germany. LNCS 2460 (2002) 383–397 8

    Google Scholar 

  6. Basili, V.R., Selby, R.W.: Comparing the Effectiveness of Software Testing Strategies. IEEE Trans. Software Eng. 13(12) (1987) 1278–1296 5, 6

    Article  Google Scholar 

  7. Bernot, G., Gaudel M.C., Marre, B.: Software Testing Based On Formal Specifications: a Theory and a Tool. Software Eng. Journal 6 (1991) 387–405 8

    Article  Google Scholar 

  8. Bertolino, A.: ISSTA 2002 Panel: Is ISSTA Research Relevant to Industrial Users?. In [36] 201–202 (see also following entries 203-209) 6

    Google Scholar 

  9. Bertolino, A.: Knowledge Area Description of Software Testing. Guide to the SWEBOK, Joint IEEE-ACM Software Engineering Coordinating Committee. (2001) On-line at: <http://www.swebok.org> 2, 6

  10. Bertolino, A., Corradini, F, Inverardi, P., Muccini, H.: Deriving Test Plans from Architectural Descriptions. Proc. Int. ACM Conf. on Soft. Eng. (2000) 220–229 14

    Google Scholar 

  11. Bertolino, A., Inverardi, P., Muccini, H.: An Explorative Journey from Architectural Tests Definition downto Code Tests Execution. Proc. Int. Conf. on Soft. Eng. (2001) 211–220 9, 14

    Google Scholar 

  12. Bertolino, A., Marré, M.: A General Path Generation Algorithm for Coverage Testing. Proc. 10th Int. Soft. Quality Week, San Francisco, Ca. (1997) pap. 2T1 7

    Google Scholar 

  13. Bertolino, A., Polini, A.: Re-thinking the Development Process of Component-Based Software, ECBS 2002 Workshop on CBSE, Lund, Sweden (2002) 15

    Google Scholar 

  14. Bertolino, A., Polini, A.: WCT: a Wrapper for Component Testing. Proc. Fidji’2002, Luxembourg (to appear) (2002) 16

    Google Scholar 

  15. Bertolino, A., Strigini, L.: On the Use of Testability Measures for Dependability Assessment IEEE Trans. Software Eng. 22(2) (1996) 97–108 11, 14

    Article  Google Scholar 

  16. Binder, R. V.: Testing Object-Oriented Systems-Models, Patterns, and Tools. Addison-Wesley (2000) 14, 15

    Google Scholar 

  17. Bochmann, G. V., Petrenko, A.: Protocol Testing: Review of Methods and Relevance for Software Testing. Proc. Int. Symp. on Soft. Testing and Analysis (ISSTA), Seattle (1994) 109–124 8

    Google Scholar 

  18. Börger, E.: The Origins and the Development of the ASM Method for High Level System Design and Analysis. J. of Universal Computer Science 8(1) (2002) 2–74. On line at: <http://www.jucs.org/jucs_8_1/the_origins_and_the> 1, 2

    Google Scholar 

  19. Briand, L., Labiche, Y.: A UML-Based Approach to System Testing. Software and Systems Modeling 1(1) (2002) 10–42 8

    Google Scholar 

  20. Brinksma, E., Tretmans, J.: Testing Transition Systems: An Annotated Bibliography. Proc. of MOVEP’2k, Nantes (2000) 44–50 8

    Google Scholar 

  21. Carver, R.H., Tai, K.-C.: Use of Sequencing Constraints for Specification-Based Testing of Concurrent Programs. IEEE Trans. on Soft. Eng. 24(6) (1998) 471–490 10

    Article  Google Scholar 

  22. Choi, J-D., Zeller, A.: Isolating Failure-Inducing Thread Schedules. In [36] 210–220 4

    Google Scholar 

  23. Coward, P.D.: Symbolic Execution Systems-A Review. Software Eng. J. (1988) 229–239 7

    Google Scholar 

  24. Clarke, L.A., Richardson, D. J.: Applications of Symbolic Evaluation. The J. of Systems and Software 5 (1985) 15–35 2

    Article  Google Scholar 

  25. Crnkovic, I.: Component-based Software Engineering-New Challenges in Software Development. John Wiley&Sons (2001) 15

    Google Scholar 

  26. DeMillo, R.A., Offutt, A. J.: Constraint-Based Test Data Generation. IEEE Trans. Software Eng. 17(9) (1991) 900–910 8

    Article  Google Scholar 

  27. Dick, J., Faivre, A.: Automating The Generation and Sequencing of Test Cases From Model-Based Specifications. Proc. FME’93, LNCS 670 (1993) 268–284 8, 9

    Google Scholar 

  28. Dijkstra, E.W.: Notes on Structured Programming. T.H. Rep. 70-WSK03 (1970) On line at: <http://www.cs.utexas.edu/users/EWD/ewd02xx/EWD249.PDF> 1, 3

  29. Duesterwald, E., Gupta, R., Soffa, M. L.: Rigorous Data Flow Testing Through Output Influences. Proc. 2nd Irvine Software Symposium, Irvine, CA. (1992) 131–145 7

    Google Scholar 

  30. Duran, J. W., Ntafos, S.C.: An Evaluation of Random Testing. IEEE Trans. Software Eng. SE-10(4) (1984) 438–444 5

    Article  Google Scholar 

  31. Dunsmore, A., Roper, M., Wood, M.: Further Investigations into the Development and Evaluation of Reading Techniques for Object-Oriented Code Inspection. Proc. 24th Int. Conf. on Soft. Eng. Orlando, FL, USA (2002) 47–57 2

    Google Scholar 

  32. Edelstein, O., Farchi, E., Nir, Y., Ratsaby, G., Ur, S.: Multithreaded Java Program Test Generation. IBM Systems Journal 41 (2002) 111–125 10

    Article  Google Scholar 

  33. Fagan, M.R.: Design and Code Inspections to Reduce Errors in Program Development. IBM Systems Journal 15(3) (1976) 182–211 2

    Google Scholar 

  34. Fenton, N.E., Ohlsson, N.: Quantitative Analysis of Faults and Failures in a Complex Software System. IEEE Trans. Software Eng. 26(8) (2000) 797–814 4

    Article  Google Scholar 

  35. Forgacs, I., Bertolino, A.: Preventing Untestedness in Data-flow Based Testing. Soft. Testing, Verification and Reliability 12(1)(2001) 29–61 7

    Article  Google Scholar 

  36. Frankl, P.G. (Ed.): Proc. ACM Sigsoft Int. Symposium on Soft. Testing and Analysis ISSTA 2002, Soft. Engineering Notes 27(4) Roma, Italy (July 2002) 17, 18, 19

    Google Scholar 

  37. Frankl, P.G., Hamlet, R.G., Littlewood, B., Strigini, L.: Evaluating Testing Methods by Delivered Reliability. IEEE Trans. Software Eng. 24(8) (1998) 586–601 4, 5

    Article  Google Scholar 

  38. Freedman, R. S.: Testability of Software Components. IEEE Trans. Software Engineering 17(6) (1991) 553–564 14

    Article  Google Scholar 

  39. Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design Patterns-Elements of Reusable Object-Oriented Software. Addison-Wesley (1994) 15

    Google Scholar 

  40. Gao, J.: Component Testability and Component Testing Challenges, Proc. ICSE Workshop on Component-Based Soft. Eng. (2000) http://www.sei.cmu.edu/cbs/cbse2000/paper/ 15

  41. Gao, J., Gupta, K., Gupta, S., Shim, S.: On Building Testable Software Components. Proc. ICCBSS2002, LNCS 2255 (2002) 108–121 16

    Google Scholar 

  42. Gargantini, A., Riccobene, E.: ASM-Based Testing: Coverage Criteria and Automatic Test Sequence. J. of Universal Computer Science 7(11) (2001) 1050–1067. On line at: <http://www.jucs.org/jucs_7_11/asm_based_testing_coverage> 2

    Google Scholar 

  43. Grieskamp, W., Gurevich, Y., Schulte, W., Veanes, M.: Generating Finite State Machines from Abstract State Machines. In [36] 112–122 2

    Google Scholar 

  44. Hamlet, D.: Continuity in Software Systems. In [36] 196–200 1

    Google Scholar 

  45. Hamlet, D., Taylor, R.: Partition Testing Does Not Inspire Confidence. IEEE Trans. Software Eng. 16(12) (1990) 1402–1411 5, 12

    Article  MathSciNet  Google Scholar 

  46. Harrold, M. J.: Testing: A Roadmap. In A. Finkelstein (Ed.), The Future of Software Engineering, ACM Press (2000) 63–72 17

    Google Scholar 

  47. Harrold, M. J., Soffa, M. L.: Selection of Data for Integration Testing.IEEE Software (March 1991) 58–65 14

    Google Scholar 

  48. Hierons, R. M.: Testing from a Z Specification. Soft. Testing, Verification and Reliability 7(1997) 19–33 8

    Article  Google Scholar 

  49. Hierons, R., Derrick, J. (Eds): Special Issue on Specification-based Testing. Soft. Testing, Verification and Reliability 10 (2000) 8

    Google Scholar 

  50. Hiller, M., Jhumka, A., Suri, N.: PROPANE: An Environment for Examining the Propagation of Errors in Software. In [36] 81–85 4

    Google Scholar 

  51. Howden, W.E.: Weak Mutation Testing and Completeness of Test Sets. IEEE Trans. Software Eng. 8(4) (1982) 371–379 4

    Article  Google Scholar 

  52. ISO/IEC 9126, Information Technology-Software Product Evaluation-Quality Characteristics and Guidelines for Their Use. (1991) 13

    Google Scholar 

  53. Fernandez, J.-C., Jard, C., Jeron, T., Nedelka, L., Viho, C.: Using On-the-fly Verification Techniques for the Generation of Test Suites. Proc. of the 8th Int. Conf. on Computer Aided Verification (1996) 8

    Google Scholar 

  54. Korel, B.: Automated Software Test Data Generation. IEEE Trans. Software Eng. 16(8) (1990) 870–879 7

    Article  Google Scholar 

  55. Laprie, J.C.: Dependability-Its Attributes, Impairments and Means. In [66] 3–18 3

    Google Scholar 

  56. Latella, D., Massink, M.: On Testing and Conformance Relations for UML Statechart Diagrams Behaviours. In [36] 144–153 8

    Google Scholar 

  57. Littlewood, B., Popov, P. T., Strigini, L., Shryane, N.: Modeling the Effects of Combining Diverse Software Fault Detection Techniques. IEEE Trans. Software Eng. 26(12) (2000) 1157–1167 5

    Article  Google Scholar 

  58. Littlewood, B., Strigini, L.: Validation of Ultra-High Dependability for Softwarebased Systems. Communications of the ACM. 36(11) (1993) 69–80 13

    Article  Google Scholar 

  59. Lyu, M.R. (Ed.): Handbook of Software Reliability Engineering. IEEE Comp. Soc. Press/McGraw-Hill (1996) 8, 12

    Google Scholar 

  60. Morell, L. J.: A Theory of Fault-based Testing. IEEE Trans. Software Eng. 16(8) (1990) 844–857 4

    Article  Google Scholar 

  61. Morris, J., Lee, G., Parker, K., Bundell, G. A., Lam, C. P.: Software Component Certification. IEEE Computer (September 2001) 30–36 16

    Google Scholar 

  62. Myers, G. J.: The Art of Software Testing. Wiley. (1979) 7, 8

    Google Scholar 

  63. Orso, A., Harrold, M. J., Rosenblum, D.: Component Metadata for Software Engineering Tasks. Proc. EDO2000, LNCS 1999 (2000) 129–144 16

    Google Scholar 

  64. Ostrand, T. J., Balcer, M. J.: The Category-Partition Method for Specifying and Generating Functional Tests. ACM Comm. 31(6) (1988) 676–686 8

    Article  Google Scholar 

  65. Pargas, R., Harrold, M. J., Peck, R.: Test-Data Generation Using Genetic Algorithms. J. of Soft. Testing, Verifications, and Reliability 9 (1999) 263–282 7

    Article  Google Scholar 

  66. Randell, B., Laprie, J. C., Kopetz, H., Littlewood B., Eds.: Predictably Dependable Computing Systems, Springer (1995) 19, 20

    Google Scholar 

  67. Rapps, S., Weyuker, E. J.: Selecting Software Test Data Using Data Flow Information. IEEE Trans. Software Eng., SE-11 (1985) 367–375 7

    Article  Google Scholar 

  68. Richardson, D.J, Clarke, L.A.: Partition Analysis: A Method Combining Testing and Verification. IEEE Trans. Software Eng., SE-11 (1985) 1477–1490 5

    Article  Google Scholar 

  69. Richardson, D., Thompson, M.C.: The Relay Model for Error Detection and its Application. Proc. 2nd Wksp Soft. Testing, Verification, and Analysis. Ban. Alberta, ACM/Sigsoft and IEEE July 1988 223–230 4

    Google Scholar 

  70. Rothermel, G., Harrold, M. J.: Analyzing Regression Test Selection Techniques. IEEE Trans. Software Eng., 22(8) (1996) 529–551 15

    Article  Google Scholar 

  71. Stafford, J. A., Wolf, A. L.: Annotating Components to Support Component-Based Static Analyses of Software Systems, Proc. Grace Hopper Celeb. of Women in Computing (2001) 16

    Google Scholar 

  72. TESTNET-Integration of Testing Methodologies. <http://http://www-lor.int-evry.fr/testnet/> 13

  73. TGV-Test Generation from transitions systems using Verification techniques. On line at <http://www.inrialpes.fr/vasy/cadp/man/tgv.html> 8

  74. Thevenod-Fosse P., Waeselynck H., Crouzet Y.: Software Statistical Testing. In [66] 253–272 5

    Google Scholar 

  75. J. Tretmans. Conformance Testing with Labeled Transition Systems: Implementation Relations and Test Generation. Computer Networks and ISDN Systems 29 (1996) 49–79 8

    Article  Google Scholar 

  76. Vegas, S.: Characterisation Schema for Selecting Software Testing Techniques. Int. Software Engineering Research Network (ISERN’01) Strachclyde, Scotland (2001) 3, 6

    Google Scholar 

  77. Voas, J.M.: PIE: A Dynamic Failure-Based Technique. IEEE Trans. Software Eng. 18(8) (1992) 717–727 4

    Article  Google Scholar 

  78. Voas, J.M., Miller, K.W.: Software Testability: The New Verification. IEEE Software (May 1995) 17–28 14

    Google Scholar 

  79. Voas, J.: Certifying O.-the-Shelf Software Components. IEEE Computer (June 1998) 53–59 16

    Google Scholar 

  80. Voas, J.: Developing a Usage-Based Software Certification Process. IEEE Computer (August 2000) 32–37 16

    Google Scholar 

  81. Wang, Y., King, G., Wickburg, H.: A Method for Built-in Tests in Componentbased Software Maintenance, Proc. European Conference on Soft. Maintenance and Reengineering (1998) 186–189 16

    Google Scholar 

  82. Weyuker, E. J.: Translatability and Decidability Questions for Restricted Classes of Program Schemas. SIAM J. on Computers 8(4) (1979) 587–598 7

    Article  MATH  MathSciNet  Google Scholar 

  83. Weyuker, E. J.: On Testing Non-testable Programs. The Computer Journal 25(4) (1982) 465–470 10

    Google Scholar 

  84. Weyuker, E. J., Jeng, B.: Analyzing Partition Testing Strategies. IEEE Trans. Software Eng. 17(7) (1991) 703–711 5

    Article  Google Scholar 

  85. Weyuker, E. J., Ostrand, T. J.: Theories of Program Testing and the Application of Revealing Subdomains. IEEE Trans. Software Eng. SE-6(1980) 236–246 5

    Article  MathSciNet  Google Scholar 

  86. Wood, M., Roper, M., Brooks, A., Miller, J.: Comparing and Combining Software Defect Detection Techniques: A Replicated Empirical Study. Proc. ESEC/FSE, LNCS 1301 (1997) 262–277 5

    Google Scholar 

  87. Zhu, H., Hall, P.A.V., May, J.H.R.: Software Unit Test Coverage and Adequacy. ACM Computing Surveys, 29 (1997) 366–427 4

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. ISTI-CNR, Area della Ricerca CNR di Pisa, Italy

    Antonia Bertolino

Authors
  1. Antonia Bertolino
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Dipartimento di Informatica, Università di Pisa, Via Buonarotti 2, 56127, Pisa, Italy

    Egon Börger

  2. CEA, Università di Catania, Piazza Università 2, 95124, Catania, Italy

    Angelo Gargantini

  3. Dipartimento di Matematica e Informatica, Università di Catania, Città Universitaria, viale A. Doria 6, 95125, Catania, Italy

    Elvinia Riccobene

Rights and permissions

Reprints and permissions

Copyright information

© 2003 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Bertolino, A. (2003). Software Testing Research and Practice. In: Börger, E., Gargantini, A., Riccobene, E. (eds) Abstract State Machines 2003. ASM 2003. Lecture Notes in Computer Science, vol 2589. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-36498-6_1

Download citation

  • DOI: https://doi.org/10.1007/3-540-36498-6_1

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-00624-4

  • Online ISBN: 978-3-540-36498-6

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation