Skip to main content
Log in

Cooperative Software Testing and Analysis: Advances and Challenges

  • Survey
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

In recent years, to maximize the value of software testing and analysis, we have proposed the methodology of cooperative software testing and analysis (in short as cooperative testing and analysis) to enable testing and analysis tools to cooperate with their users (in the form of tool-human cooperation), and enable one tool to cooperate with another tool (in the form of tool-tool cooperation). Such cooperations are motivated by the observation that a tool is typically not powerful enough to address complications in testing or analysis of complex real-world software, and the tool user or another tool may be able to help out some problems faced by the tool. To enable tool-human or tool-tool cooperation, effective mechanisms need to be developed 1) for a tool to communicate problems faced by the tool to the tool user or another tool, and 2) for the tool user or another tool to assist the tool to address the problems. Such methodology of cooperative testing and analysis forms a new research frontier on synergistic cooperations between humans and tools along with cooperations between tools and tools. This article presents recent example advances and challenges on cooperative testing and analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Xie T, Tillmann N, de Halleux J, Schulte W. Future of developer testing: Building quality in code. In Proc. the 18th FSE/SDP Workshop on the Future of Software Engineering Research, Nov. 2010, pp.415-420.

  2. Xiao X, Thummalapenta S, Xie T. Advances on improving automation in developer testing. In Advances in Computers, volume 85, Memon A (ed.), Burlington: Academic Press, 2012, pp.165-212.

  3. D'Silva V, Kroening D, Weissenbacher G. A survey of automated techniques for formal software verification. IEEE Trans. Computer-Aided Design of Integrated Circuits and Systems, 2008, 27(7): 1165-1178.

    Article  Google Scholar 

  4. Xiao X, Xie T, Tillmann N, de Halleux J. Precise identification of problems for structural test generation. In Proc. the 33rd Int. Conf. Software Engineering, May 2011, pp.611-620.

  5. Dillig I, Dillig T, Aiken A. Automated error diagnosis using abductive inference. In Proc. the 33rd ACM SIGPLAN Conf. Programming Language Design and Implementation, June 2012, pp.181-192.

  6. Xie T. Cooperative testing and analysis: Human-tool, tool-tool, and human-human cooperations to get work done. In Proc. the 12th Int. Working Conf. Source Code Analysis and Manipulation, Sept. 2012, pp.1-3.

  7. Hellerstein J L, Diao Y, Parekh S, Tilbury D M. Feedback Control of Computing Systems. John Wiley & Sons, 2004.

  8. Godefroid P, Klarlund N, Sen K. DART: Directed automated random testing. In Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, June 2005, pp.213-223.

  9. Tillmann N, de Halleux J. Pex-white box test generation for .NET. In Proc. the 2nd Int. Conf. Tests and Proofs, April 2008, pp.134-153.

  10. Tillmann N, Schulte W. Mock-object generation with behavior. In Proc. the 21st Int. Conf. Automated Software Engineering, Sept. 2006, pp.365-368.

  11. Marri M R, Xie T, Tillmann N, de Halleux J, Schulte W. An empirical study of testing file-system-dependent software with mock objects. In Proc. the 4th Int. Workshop Automation of Software Test, May 2009, pp.149-153.

  12. Taneja K, Zhang Y, Xie T. MODA: Automated test generation for database applications via mock objects. In Proc. the 25th Int. Conf. Automated Software Engineering, Sept. 2010, pp.289-292.

  13. St-Amour V, Tobin-Hochstadt S, Felleisen M. Optimization coaching: Optimizers learn to communicate with programmers. In Proc. the 3rd ACM SIGPLAN Int. Conf. Object-Oriented Programming, Systems, Languages, and Applications, Oct. 2012, pp.163-178.

  14. Dincklage D, Diwan A. Explaining failures of program analyses. In Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, June 2008, pp.260-269.

  15. Lakhotia K, McMinn P, Harman M. An empirical investigation into branch coverage for C programs using CUTE and AUSTIN. Journal of Systems and Software, 2010, 83(12): 2379-2391.

    Article  Google Scholar 

  16. Godefroid P, Levin M Y, Molnar D A. Automated whitebox fuzz testing. In Proc. Network and Distributed System Security Symp., Feb. 2008.

  17. Xie T, Tillmann N, de Halleux P, Schulte W. Fitness-guided path exploration in dynamic symbolic execution. In Proc. the 39th IEEE/IFIP Int. Conf. Dependable Systems and Networks, July 2009, pp.359-368.

  18. Godefroid P, Luchaup D. Automatic partial loop summarization in dynamic test generation. In Proc. Int. Symp. Software Testing and Analysis, July 2011, pp.23-33.

  19. Saxena P, Poosankam P, McCamant S, Song D. Loop-extended symbolic execution on binary programs. In Proc. the 18th Int. Symp. Software Testing and Analysis, July 2009, pp.225-236.

  20. Strejček J, Trtík M. Abstracting path conditions. In Proc. Int. Symp. Software Testing and Analysis, July 2012, pp.155-165.

  21. Xiao X, Li S, Xie T, Tillmann N. Characteristic studies of loop problems for structural test generation via symbolic execution. In Proc. the 28th Int. Conf. Automated Software Engineering, Nov. 2013, pp.246-256.

  22. Tillmann N, de Halleux J, Xie T. Pex4Fun: Teaching and learning computer science via social gaming. In Proc. the 24th IEEE-CS Conf. Software Engineering Education and Training, May 2011, pp.546-548.

  23. Tillmann N, de Halleux J, Xie T, Gulwani S, Bishop J. Teaching and learning programming and software engineering via interactive gaming. In Proc. the 35th Int. Conf. Software Engineering, Software Engineering Education, May 2013, pp.1117-1126.

  24. Tillmann N, de Halleux J, Xie T, Bishop J. Pex4Fun: A web-based environment for educational gaming via automated test generation. In Proc. the 28th Int. Conf. Automated Software Engineering, Nov. 2013, pp.730-733.

  25. Hao D, Zhang L, Xie T, Mei H, Sun J. Interactive fault localization using test information. Journal of Computer Science and Technology, 2009, 24(5): 962-974.

    Article  Google Scholar 

  26. Hao D, Zhang L, Zhang L, Sun J, Mei H. VIDA: Visual interactive debugging. In Proc. the 31st Int. Conf. Software Engineering, May 2009, pp.583-586.

  27. Xie T, Tillmann N, de Halleux J. Educational software engineering: Where software engineering, education, and gaming meet. In Proc. the 3rd International Workshop on Games and Software Engineering, May 2013, pp.36-39.

  28. Fincher S, Cooper S, Kölling M, Maloney J. Comparing Alice, Greenfoot & Scratch. In Proc. the 41st ACM Technical Symp. Computer Science Education, March 2010, pp.192-193.

  29. Utting I, Cooper S, Kölling M, Maloney J, Resnick M. Alice, Greenfoot, and Scratch – A discussion. ACM Trans. Computing Education, 2010, 10(4): 17:1-17:11.

  30. Tillmann N, de Halleux J, Xie T, Bishop J. Code Hunt: Gamifying teaching and learning of computer science at scale. In Proc. the 1st ACM Conf. Learning at Scale, March 2014, pp.221-222.

  31. Tillmann N, Bishop J, Horspool N, Perelman D, Xie T. Code Hunt - Searching for secret code for fun. In Proc. the 7th Int. Workshop Search-Based Software Testing, June 2014.

  32. Dwyer M B, Elbaum S G. Unifying verification and validation techniques: Relating behavior and properties through partial evidence. In Proc. the 18th FSE/SDP Workshop on the Future of Software Engineering Research, Nov. 2010, pp.93-98.

  33. Ge X, Taneja K, Xie T, Tillmann N. DyTa: Dynamic symbolic execution guided with static verification results. In Proc. the 33rd Int. Conf. Software Engineering, May 2011, pp.992-994.

  34. Xie T, Notkin D. Tool-assisted unit-test generation and selection based on operational abstractions. Automated Software Engineering, 2006, 13(3): 345-371.

    Article  Google Scholar 

  35. Wang X, Zhang L, Xie T, Xiong Y, Mei H. Automating presentation changes in dynamic web applications via collaborative hybrid analysis. In Proc. the 20th ACM SIGSOFT Symp. Foundations of Software Engineering, Nov. 2012, pp.16:1-16:11.

  36. Csallner C, Smaragdakis Y, Xie T. DSD-Crasher: A hybrid analysis tool for bug finding. ACM Trans. Software Engineering and Methodology, 2008, 17(2): 8:1-8:37.

  37. Thummalapenta S, Xie T, Tillmann N, de Halleux J, Su Z. Synthesizing method sequences for high-coverage testing. In Proc. ACM SIGPLAN Int. Conf. Object-Oriented Programming, Systems, Languages, and Applications, Oct. 2011, pp.189-206.

  38. Inkumsah K, Xie T. Improving structural testing of object-oriented programs via integrating evolutionary testing and symbolic execution. In Proc. the 23rd Int. Conf. Automated Software Engineering, Sept. 2008, pp.297-306.

  39. Xiong Y, Liu D, Hu Z, Zhao H, Takeichi M, Mei H. Towards automatic model synchronization from model transformations. In Proc. the 22nd Int. Conf. Automated Software Engineering, Nov. 2007, pp.164-173.

  40. Weiser M. Program slicing. In Proc. the 5th Int. Conf. Software Engineering, March 1981, pp.439-449.

  41. Gong L, Lo D, Jiang L, Zhang H. Interactive fault localization leveraging simple user feedback. In Proc. the 28th IEEE Int. Conf. Software Maintenance, Sept. 2012, pp.67-76.

  42. Weimer W, Nguyen T, Le Goues C, Forrest S. Automatically finding patches using genetic programming. In Proc. the 31st Int. Conf. Software Engineering, May 2009, pp.364-374.

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Tao Xie.

Additional information

This work is supported in part by the National Natural Science Foundation of China under Grant Nos. 61228203, 61225007, and 61272157, the National Science Foundation of USA under Grant Nos. CCF-1349666, CNS-1434582, CCF-1434596, CCF-1434590, CNS-1439481, and a Microsoft Research award.

A preliminary version of the paper was published as a keynote paper in the Proceedings of the 12th International Working Conference on Source Code Analysis and Manipulation (SCAM 2012).

Electronic supplementary material

Below is the link to the electronic supplementary material.

ESM 1

(PDF 228 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xie, T., Zhang, L., Xiao, X. et al. Cooperative Software Testing and Analysis: Advances and Challenges. J. Comput. Sci. Technol. 29, 713–723 (2014). https://doi.org/10.1007/s11390-014-1461-6

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-014-1461-6

Keywords

Navigation