On Testing Data-Intensive Software Systems

  • Michael FeldererEmail author
  • Barbara Russo
  • Florian Auer


Today’s software systems like cyber-physical production systems or big data systems have to process large volumes and diverse types of data which heavily influences the quality of these so-called data-intensive systems. However, traditional software testing approaches rather focus on functional behavior than on data aspects. Therefore, the role of data in testing has to be rethought, and specific testing approaches for data-intensive software systems are required. Thus, the aim of this chapter is to contribute to this area by (1) providing basic terminology and background on data-intensive software systems and their testing and (2) presenting the state of the research and the hot topics in the area. Finally, the directions of research and the new frontiers on testing data-intensive software systems are discussed.


Data-intensive systems Data engineering Cyber-physical production systems Software testing Data quality 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Ammann, P., & Offutt, J. (2016). Introduction to software testing. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  2. Auer, F., & Felderer, M. (2018). Current state of research on continuous experimentation: A systematic mapping study. In 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) (pp. 335–344). Piscataway, NJ: IEEE.CrossRefGoogle Scholar
  3. de Bayser, M., Azevedo, L. G., & Cerqueira, R. (2015). Researchops: The case for devops in scientific applications. In 2015 IFIP/IEEE International Symposium on Integrated Network Management (IM) (pp. 1398–1404). Piscataway, NJ: IEEEGoogle Scholar
  4. Bourque, P., & Dupuis, R. (Eds.) (2014). Guide to the software engineering body of knowledge version 3.0 SWEBOK. Piscataway, NJ: IEEE.
  5. Casale, G., Ardagna, D., Artac, M., Barbier, F., Nitto, E. D., Henry, et al.: (2015). Dice: quality-driven development of data-intensive cloud applications. In Proceedings of the Seventh International Workshop on Modeling in Software Engineering (pp. 78–83). Piscataway, NJ: IEEE PressGoogle Scholar
  6. Corbin, J. M., & Strauss, A. (1990). Grounded theory research: Procedures, canons, and evaluative criteria (pp. 3–21). Berlin: Springer.Google Scholar
  7. Dersin, P. (2014). Systems of systems. Piscataway, NJ: IEEE.zbMATHGoogle Scholar
  8. Du, X., Xie, X., Li, Y., Ma, L., Zhao, J., & Liu, Y. (2018). Deepcruiser: Automated guided testing for stateful deep learning systems. arXiv preprint arXiv:1812.05339.Google Scholar
  9. Fagerholm, F., Guinea, A.S., Mäenpää, H., Münch, J. (2014). Building blocks for continuous experimentation. In Proceedings of the 1st International Workshop on Rapid Continuous Software Engineering (pp. 26–35). New York, NY: ACM.Google Scholar
  10. Felderer, M., Büchler, M., Johns, M., Brucker, A. D., Breu, R., & Pretschner, A. (2016). Security testing: A survey. In: Advances in Computers (Vol. 101, pp. 1–51). Amsterdam: Elsevier.Google Scholar
  11. Foidl, H., & Felderer, M. (2015). Research challenges of industry 4.0 for quality management. In International Conference on Enterprise Resource Planning Systems (pp. 121–137). Berlin: Springer.Google Scholar
  12. Gadler, D., Mairegger, M., Janes, A., & Russo, B. (2017). Mining logs to model the use of a system. In 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2017 (pp. 334–343). Piscataway, NJ: IEEE.Google Scholar
  13. Hummel, O., Eichelberger, H., Giloj, A., Werle, D., & Schmid, K. (2018). A collection of software engineering challenges for big data system development. In: 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) (pp. 362–369). Piscataway, NJ: IEEE.CrossRefGoogle Scholar
  14. ISO. (1994). ISO 8402:1994 Quality Management and Quality Assurance—Vocabulary. Tech. rep., ISO.Google Scholar
  15. ISO/IEC. (1994). Information Technology – Open Systems Interconnection – Conformance Esting Methodology and Framework. International ISO/IEC multi–part standard No. 9646.Google Scholar
  16. ISO/IEC. (2008). ISO/IEC 25012:2008 Software Engineering – Software Product Quality Requirements and Evaluation (Square) – Data Quality Model. Tech. Rep., ISO.Google Scholar
  17. ISO/IEC. (2011). ISO/IEC 25010:2011 Systems and Software Engineering – Systems and Software Quality Requirements and Evaluation (Square) – System and Software Quality Models. Tech. Rep., ISO.Google Scholar
  18. ISTQB. (2012). Standard Glossary of Terms Used in Software Testing. Version 2.2. Tech. Rep., ISTQB.Google Scholar
  19. Kim, J., Feldt, R., & Yoo, S. (2019). Guiding deep learning system testing using surprise adequacy. In: Proceedings of the 41st International Conference on Software Engineering (pp. 1039–1049). Piscataway, NJ: IEEE PressGoogle Scholar
  20. Kleppmann, M. (2017). Designing data-intensive applications: The big ideas behind reliable, scalable, and maintainable systems. Sebastopol, CA: O’Reilly.Google Scholar
  21. Mattmann, C. A., Crichton, D. J., Hart, A. F., Goodale, C., Hughes, J.S., Kelly, S., et al. (2011). Architecting data-intensive software systems. In: Handbook of data intensive computing (pp. 25–57). Berlin: Springer.CrossRefGoogle Scholar
  22. Nagy, C. (2013). Static analysis of data-intensive applications. In 2013 17th European Conference on Software Maintenance and Reengineering (pp. 435–438). Piscataway, NJ: IEEE.CrossRefGoogle Scholar
  23. Otero, C. E., & Peter, A. (2015). Research directions for engineering big data analytics software (pp. 13–19). Piscataway, NJ: IEEE.Google Scholar
  24. Pei, K., Cao, Y., Yang, J., & Jana, S. (2017). Deepxplore: Automated whitebox testing of deep learning systems. In Proceedings of the 26th Symposium on Operating Systems Principles (pp. 1–18). New York, NY: ACM.Google Scholar
  25. Radatz, J., Geraci, A., & Katki, F. (1990). IEEE Standard Glossary of Software Engineering Terminology. Tech. Rep., Piscataway, NJ: IEEE.Google Scholar
  26. Russo, B., Succi, G., & Pedrycz, W. (2015). Mining system logs to learn error predictors: a case study of a telemetry system (pp. 879–927). Berlin: Springer.Google Scholar
  27. Segura, S., Fraser, G., Sanchez, A. B., & Ruiz-Cortés, A. (2016). A survey on metamorphic testing (pp. 805–824). Piscataway, NJ: IEEE.Google Scholar
  28. Shull, F. (2013). Getting an intuition for big data (pp. 3–6). Piscataway, NJ: IEEE.Google Scholar
  29. Tian, Y., Pei, K., Jana, S., & Ray, B. (2018). Deeptest: Automated testing of deep-neural-network-driven autonomous cars. In: Proceedings of the 40th International Conference on Software Engineering (pp. 303–314). New York, NY: ACM.Google Scholar
  30. Tonella, P. (2019). 2019–2023 ERC project: Precrime: Self-assessment Oracles for Anticipatory Testing Retrieved 2018, January 7th, 2019, from Google Scholar
  31. Utting, M., & Legeard, B. (2007). Practical model-based testing: A tools approach. San Francisco, CA: Morgan Kaufmann Publishers.Google Scholar
  32. Wang, S., Wan, J., Li, D., & Zhang, C. (2016). Implementing smart factory of industrie 4.0: An outlook (p. 3159805). London, UK: Sage Publications.CrossRefGoogle Scholar
  33. Weyuker, E. J. (1982). On testing non-testable programs (pp. 465–470). London, UK: The British Computer Society.Google Scholar
  34. Xie, X., Ho, J. W., Murphy, C., Kaiser, G., Xu, B., & Chen, T. Y. (2011). Testing and validating machine learning classifiers by metamorphic testing (pp. 544–558). Amsterdam: Elsevier.Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.University of InnsbruckInnsbruckAustria
  2. 2.Free University of Bozen-BolzanoBolzanoItaly

Personalised recommendations