Advertisement

To the attention of mobile software developers: guess what, test your app!

  • Luis CruzEmail author
  • Rui Abreu
  • David Lo
Article

Abstract

Software testing is an important phase in the software development lifecycle because it helps in identifying bugs in a software system before it is shipped into the hand of its end users. There are numerous studies on how developers test general-purpose software applications. The idiosyncrasies of mobile software applications, however, set mobile apps apart from general-purpose systems (e.g., desktop, stand-alone applications, web services). This paper investigates working habits and challenges of mobile software developers with respect to testing. A key finding of our exhaustive study, using 1000 Android apps, demonstrates that mobile apps are still tested in a very ad hoc way, if tested at all. However, we show that, as in other types of software, testing increases the quality of apps (demonstrated in user ratings and number of code issues). Furthermore, we find evidence that tests are essential when it comes to engaging the community to contribute to mobile open source software. We discuss reasons and potential directions to address our findings. Yet another relevant finding of our study is that Continuous Integration and Continuous Deployment (CI/CD) pipelines are rare in the mobile apps world (only 26% of the apps are developed in projects employing CI/CD) – we argue that one of the main reasons is due to the lack of exhaustive and automatic testing.

Keywords

Software testing Mobile applications Open source software Software quality Software metrics 

Notes

Acknowledgments

This work is financed by the ERDF — European Regional Development Fund through the Operational Program for Competitiveness and Internationalization - COMPETE 2020 Program and by National Funds through the Portuguese funding agency, FCT - Fundação para a Ciência e a Tecnologia with reference UID/CEC/50021/2019, and within projects GreenLab (POCI-01-0145-FEDER-016718) and FaultLockerRef (PTDC/CCI-COM/29300/2017). Luis Cruz is sponsored by an FCT scholarship grant number PD/BD/52237/2013.

References

  1. Amalfitano D, Amatucci N, Memon AM, Tramontana P, Fasolino AR (2017) A general framework for comparing automatic testing techniques of android mobile apps. J Syst Softw 125:322–343CrossRefGoogle Scholar
  2. Bavani R (2012) Distributed agile, agile testing, and technical debt. IEEE Softw 29(6):28–33CrossRefGoogle Scholar
  3. Beck K (2000) Extreme programming explained: embrace change. Addison-Wesley Professional, ReadingGoogle Scholar
  4. Bird C, Rigby PC, Barr ET, Hamilton DJ, German DM, Devanbu P (2009) The promises and perils of mining git. In: 2009 MSR’09. 6th IEEE International Working Conference on Mining Software Repositories. IEEE, pp 1–10Google Scholar
  5. Brooks ME, Dalal DK, Nolan KP (2014) Are common language effect sizes easier to understand than traditional effect sizes? J Appl Psychol 99(2):332CrossRefGoogle Scholar
  6. Choudhary SR, Gorla A, Orso A (2015) Automated test input generation for Android: Are we there yet? (E). In: 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, pp 429–440Google Scholar
  7. Coppola R (2017) Fragility and evolution of android test suites. In: Proceedings of the 39th International Conference on Software Engineering Companion. IEEE Press, pp 405–408Google Scholar
  8. Coppola R, Morisio M, Torchiano M (2017) Scripted gui testing of android apps: A study on diffusion, evolution and fragility. In: Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering. ACM, pp 22–32Google Scholar
  9. Corral L, Fronza I (2015) Better code for better apps: a study on source code quality and market success of android applications. In: Proceedings of the Second ACM International Conference on Mobile Software Engineering and Systems. IEEE Press, pp 22–32Google Scholar
  10. Cosentino V, Izquierdo JLC, Cabot J (2016) Findings from github: methods, datasets and limitations. In: 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR). IEEE, pp 137–141Google Scholar
  11. Cruz L, Abreu R (2018) Measuring the energy footprint of mobile testing frameworks. In: 2018 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-c). IEEEGoogle Scholar
  12. Das T, Di Penta M, Malavolta I (2016) A quantitative and qualitative investigation of performance-related commits in android apps. In: 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, pp 443–447Google Scholar
  13. de Langhe B, Fernbach PM, Lichtenstein DR (2016) Navigating by the stars: Investigating the actual and perceived validity of online user ratings. J Consum Res 42 (6):817–833CrossRefGoogle Scholar
  14. Gao J, Bai X, Tsai W-T, Uehara T (2014) Mobile application testing: a tutorial. Computer 47(2):46–55CrossRefGoogle Scholar
  15. Gao Z, Chen Z, Zou Y, Memon AM (2016) Sitar: Gui test script repair. IEEE Trans Softw Eng 42(2):170–186CrossRefGoogle Scholar
  16. Geiger F-X, Malavolta I, Pascarella L, Palomba F, Di Nucci D, Bacchelli A (2018) A graph-based dataset of commit history of real-world android apps. In: Proceedings of the 15th International Conference on Mining Software Repositories, MSR. ACM, New YorkGoogle Scholar
  17. Gousios G, Zaidman A, Storey M-A, Van Deursen A (2015) Work practices and challenges in pull-based development: the integrator’s perspective. In: Proceedings of the 37th International Conference on Software Engineering-Volume 1. IEEE Press, pp 358–368Google Scholar
  18. Gousios G, Storey M-A, Bacchelli A (2016) Work practices and challenges in pull-based development: The contributor’s perspective. In: 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE). IEEE, pp 285–296Google Scholar
  19. Hilton M, Tunnell T, Huang K, Marinov D, Dig D (2016) Usage, costs, and benefits of continuous integration in open-source projects. In: 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, pp 426–437Google Scholar
  20. Hu C, Neamtiu I (2011) Automating GUI testing for android applications. In: Proceedings of the 6th International Workshop on Automation of Software Test. ACM, pp 77–83Google Scholar
  21. Joorabchi ME, Mesbah A, Kruchten P (2013) Real challenges in mobile app development. In: 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. IEEE, pp 15–24Google Scholar
  22. Kaasila J, Ferreira D, Kostakos V, Ojala T (2012) Testdroid: Automated Remote UI Testing on Android. In: Proceedings of the 11th International Conference on Mobile and Ubiquitous Multimedia, ser. MUM ’12. ACM, New York, pp. 28:1–28:4Google Scholar
  23. Karvonen T, Behutiye W, Oivo M, Kuvaja P (2017) Systematic literature review on the impacts of agile release engineering practices. Information and Software TechnologyGoogle Scholar
  24. Kerby DS (2014) The simple difference formula: An approach to teaching nonparametric correlation, vol 3. [Online]. Available:  https://doi.org/10.2466/11.IT.3.1
  25. Khalid H, Nagappan M, Shihab E, Hassan AE (2014) Prioritizing the devices to test your app on: A case study of android game apps. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software EngineeringGoogle Scholar
  26. Kochhar PS (2013) An empirical study of adoption of software testing in open source projects. In: 2013 13th International Conference on Quality Software (QSIC). IEEE, pp 103–112Google Scholar
  27. Kochhar PS, Bissyandé TF, Lo D, Jiang L (2013) Adoption of software testing in open source projects — A preliminary study on 50,000 projects. In: 2013 17th European Conference on Software Maintenance and Reengineering (CSMR). IEEE, pp 353–356Google Scholar
  28. Kochhar PS, Thung F, Nagappan N, Zimmermann T, Lo D (2015) Understanding the test automation culture of app developers. In: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST). IEEE, pp 1–10Google Scholar
  29. Krutz DE, Mirakhorli M, Malachowsky SA, Ruiz A, Peterson J, Filipski A, Smith J (2015) A dataset of open-source android applications. In: 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories (MSR). IEEE, pp 522–525Google Scholar
  30. Leech NL, Onwuegbuzie AJ (2002) A call for greater use of nonparametric statisticsGoogle Scholar
  31. Li X, Chang N, Wang Y, Huang H, Pei Y, Wang L, Li X (2017) ATOM: Automatic maintenance of GUI test scripts for evolving mobile applications. In: 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE, pp 161–171Google Scholar
  32. Linares-Vásquez M, Bernal-Cárdenas C, Moran K, Poshyvanyk D (2017a) How do developers test android applications?” In: 33rd IEEE International Conference on Software Maintenance and Evolution (ICSME’17) page to appearGoogle Scholar
  33. Linares-Vásquez M, Moran K, Poshyvanyk D (2017b) Continuous, evolutionary and large-scale: A new perspective for automated mobile app testing. In: 33rd IEEE International Conference on Software Maintenance and Evolution (ICSME’17) page to appearGoogle Scholar
  34. Maji AK, Hao K, Sultana S, Bagchi S (2010) Characterizing failures in mobile oses: A case study with android and symbian. In: 2010 IEEE 21st International Symposium on Software Reliability Engineering (ISSRE). IEEE, pp 249–258Google Scholar
  35. Martin W, Sarro F, Jia Y, Zhang Y, Harman M (2017) A survey of app store analysis for software engineering. IEEE Trans Softw Eng 43(9):817–847CrossRefGoogle Scholar
  36. McGraw KO, Wong S (1992) A common language effect size statistic. Psychol Bullet 111(2):361CrossRefGoogle Scholar
  37. Moran K, Linares-Vásquez M, Poshyvanyk D (2017) Automated GUI testing of Android apps: from research to practice. In: Proceedings of the 39th International Conference on Software Engineering Companion. IEEE Press, pp 505–506Google Scholar
  38. Muccini H, Di Francesco A, Esposito P (2012) Software testing of mobile applications: Challenges and future research directions. In: Proceedings of the 7th International Workshop on Automation of Software Test. IEEE Press, pp 29–35Google Scholar
  39. Nagappan M, Shihab E (2016) Future trends in software engineering research for mobile apps. In: 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, vol 5, pp 21–32Google Scholar
  40. Nayebi M, Adams B, Ruhe G (2016) Release practices for mobile apps–what do users and developers think?” In: 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, vol 1, pp 552–562Google Scholar
  41. Nayebi M, Cho H, Ruhe G (2018) App store mining is not enough for app improvement. Empir Softw Eng 23(5):1–31CrossRefGoogle Scholar
  42. Pascarella L, Geiger F-X, Palomba F, Di Nucci D, Malavolta I, Bacchelli A (2018) Self-reported activities of android developers. In: 5th IEEE/ACM International Conference on Mobile Software Engineering and Systems, New YorkGoogle Scholar
  43. Picco GP, Julien C, Murphy AL, Musolesi M, Roman G-C (2014) Software engineering for mobility: reflecting on the past, peering into the future In: Proceedings of the on Future of Software Engineering. ACM, pp 13–28Google Scholar
  44. Ruiz IM, Nagappan M, Adams B, Berger T, Dienst S, Hassan A (2017) An examination of the current rating system used in mobile app stores. IEEE SoftwareGoogle Scholar
  45. Silva DB, Endo AT, Eler MM, Durelli VH (2016) An analysis of automated tests for mobile android applications. In: 2016 XLII Latin American Computing Conference (CLEI) . IEEE, pp 1–9Google Scholar
  46. Stolberg S (2009) Enabling agile testing through continuous integration. In: 2009 AGILE’09 Agile Conference. IEEE, pp 369–374Google Scholar
  47. Van Deursen A, risks Program comprehension (2001) Opportunities in extreme programming. In: Eighth Working Conference on Reverse Engineering Proceedings. IEEE, pp 176–185Google Scholar
  48. Visser J, Rigal S, van der Leek R, van Eck P, Wijnholds G (2016) Building maintainable software, java edition: Ten Guidelines for Future-Proof Code. ”O’Reilly Media Inc.”Google Scholar
  49. Wang Y, Alshboul Y (2015) Mobile security testing approaches and challenges. In: 2015 First Conference on Mobile and Secure Services (MOBISECSERV). IEEE, pp 1–5Google Scholar
  50. Zaeem RN, Prasad MR, Khurshid S (2014) Automated generation of oracles for testing user-interaction features of mobile apps. In: 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation (ICST). IEEE, pp 183–192 0Google Scholar
  51. Zeng X, Li D, Zheng W, Xia F, Deng Y, Lam W, Yang W, Xie T (2016) Automated test input generation for android: Are we really there yet in an industrial case?” In: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, pp 987–992Google Scholar
  52. Zhao Y, Serebrenik A, Zhou Y, Filkov V, Vasilescu B (2017) The impact of continuous integration on other software development practices: A large-scale empirical study 32nd IEEE/ACM International Conference on Automated Software EngineeringGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.INESC IDLisbonPortugal
  2. 2.INESC ID and ISTUniversity of LisbonLisbonPortugal
  3. 3.School of Information SystemsSingapore Management UniversitySingaporeSingapore

Personalised recommendations