Test Reporting at a Large-Scale Austrian Logistics Organization: Lessons Learned and Improvement

  • Dietmar WinklerEmail author
  • Kristof Meixner
  • Daniel Lehner
  • Stefan Biffl
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11915)


Context and Background. Software testing and test automation are important activities in software development where frequent requirements changes and the fast delivery of software increments are supported by traditional and agile development processes. Test reports are often used as “proof of evidence” for executed software tests. However, the practical impact of test reports, such as decision making and quality assessment, requires structured information which might not be available in sufficient quality. Goal. In this paper we (a) report on needs of test reports of different stakeholders at a large-scale Austrian logistics organization, (b) develop candidate improvement actions based on the state of the practice, and (c) conceptually evaluate selected improvement actions. Method. We used surveys and interviews to elicit needs and expected capabilities for test reporting and developed candidate improvement. We used expert discussions prioritize improvement actions in the organization context for further implementation. Results. Based on 23 recommended improvement actions, 14 were initially selected for implementation. Most of these accepted improvement action focus on regular test status reports and visualization aspects of test reports. Conclusion. Although test reporting is systematically applied in development processes, there is still some potential to improve test reports to gain (additional) benefits for related stakeholder.


Software testing Test reporting Engineering process improvement Case study 



The financial support by the Austrian Federal Ministry for Digital, Business and Enterprise and the National Foundation for Research, Technology and Development is gratefully acknowledged.


  1. 1.
    Basili, V.R.: The experience factory and its relationship to other improvement paradigms. In: Sommerville, I., Paul, M. (eds.) ESEC 1993. LNCS, vol. 717, pp. 68–83. Springer, Heidelberg (1993). Scholar
  2. 2.
    Duvall, P.M., Matyas, S., Glover, A.: Continuous Integration: Improving Software Quality and Reducing Risk. Addison-Wesley, Boston (2007)Google Scholar
  3. 3.
    Humble, J., Farley, D.: Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation. Pearson Professional, Gurugram (2010)Google Scholar
  4. 4.
    ISO/IEC/IEEE 29119-3:2013: Software and Systems Engineering. Software Testing. Part 3: Test Documentation. International Standard, ISO/IEC/IEEE (2013)Google Scholar
  5. 5.
    Kelley, J.J.: Quality assurance reporting to the governing board. Trustee: J. Hospital Governing Boards 43(5), 10–12 (1990)Google Scholar
  6. 6.
    Laitenberger, O., DeBaud, J.-M.: An encompassing life cycle centric survey of software inspection. J. Syst. Softw. 50(1), 5–31 (2000)CrossRefGoogle Scholar
  7. 7.
    Molyneaux, I.: The Art of Application Performance Testing: From Strategy to Tools, 2nd edn. O’Reilly and Associates, Sebastopol (2014)Google Scholar
  8. 8.
    Myers, G.J., Sandler, C., Badgett, T.: The Art of Software Testing, 3rd edn. Wiley, Hoboken (2011)Google Scholar
  9. 9.
    Sommerville, I.: Software Engineering. Global Edition, 10th edn. Pearson Education Limited, Bengaluru (2015)Google Scholar
  10. 10.
    Spillner, A., Linz, T., Schaefer, H.: Software Testing Foundations: A Study Guide for the Certified Tester Exam, 4th edn. Rocky Nook, San Rafael (2014)Google Scholar
  11. 11.
    Winkler, D., Elberzhager, F., Biffl, S., Eschbach, R.: Software process improvement initiatives based on quality assurance strategies: a QATAM pilot application. In: Riel, A., O’Connor, R., Tichkiewitch, S., Messnarz, R. (eds.) EuroSPI 2010. CCIS, vol. 99, pp. 71–82. Springer, Heidelberg (2010). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Christian Doppler Laboratory for “Security and Quality Improvement in the Production System Lifecycle”, Institute of Information Systems EngineeringVienna University of TechnologyViennaAustria
  2. 2.Institute of Information Systems EngineeringVienna University of TechnologyViennaAustria

Personalised recommendations