Test Reporting at a Large-Scale Austrian Logistics Organization: Lessons Learned and Improvement
Context and Background. Software testing and test automation are important activities in software development where frequent requirements changes and the fast delivery of software increments are supported by traditional and agile development processes. Test reports are often used as “proof of evidence” for executed software tests. However, the practical impact of test reports, such as decision making and quality assessment, requires structured information which might not be available in sufficient quality. Goal. In this paper we (a) report on needs of test reports of different stakeholders at a large-scale Austrian logistics organization, (b) develop candidate improvement actions based on the state of the practice, and (c) conceptually evaluate selected improvement actions. Method. We used surveys and interviews to elicit needs and expected capabilities for test reporting and developed candidate improvement. We used expert discussions prioritize improvement actions in the organization context for further implementation. Results. Based on 23 recommended improvement actions, 14 were initially selected for implementation. Most of these accepted improvement action focus on regular test status reports and visualization aspects of test reports. Conclusion. Although test reporting is systematically applied in development processes, there is still some potential to improve test reports to gain (additional) benefits for related stakeholder.
KeywordsSoftware testing Test reporting Engineering process improvement Case study
The financial support by the Austrian Federal Ministry for Digital, Business and Enterprise and the National Foundation for Research, Technology and Development is gratefully acknowledged.
- 2.Duvall, P.M., Matyas, S., Glover, A.: Continuous Integration: Improving Software Quality and Reducing Risk. Addison-Wesley, Boston (2007)Google Scholar
- 3.Humble, J., Farley, D.: Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation. Pearson Professional, Gurugram (2010)Google Scholar
- 4.ISO/IEC/IEEE 29119-3:2013: Software and Systems Engineering. Software Testing. Part 3: Test Documentation. International Standard, ISO/IEC/IEEE (2013)Google Scholar
- 5.Kelley, J.J.: Quality assurance reporting to the governing board. Trustee: J. Hospital Governing Boards 43(5), 10–12 (1990)Google Scholar
- 7.Molyneaux, I.: The Art of Application Performance Testing: From Strategy to Tools, 2nd edn. O’Reilly and Associates, Sebastopol (2014)Google Scholar
- 8.Myers, G.J., Sandler, C., Badgett, T.: The Art of Software Testing, 3rd edn. Wiley, Hoboken (2011)Google Scholar
- 9.Sommerville, I.: Software Engineering. Global Edition, 10th edn. Pearson Education Limited, Bengaluru (2015)Google Scholar
- 10.Spillner, A., Linz, T., Schaefer, H.: Software Testing Foundations: A Study Guide for the Certified Tester Exam, 4th edn. Rocky Nook, San Rafael (2014)Google Scholar
- 11.Winkler, D., Elberzhager, F., Biffl, S., Eschbach, R.: Software process improvement initiatives based on quality assurance strategies: a QATAM pilot application. In: Riel, A., O’Connor, R., Tichkiewitch, S., Messnarz, R. (eds.) EuroSPI 2010. CCIS, vol. 99, pp. 71–82. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15666-3_7Google Scholar