Extending Mobile App Analytics for Usability Test Logging

  • Xavier FerreEmail author
  • Elena Villalba
  • Héctor Julio
  • Hongming Zhu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10515)


Mobile application development is characterized by reduced development cycles and high time-to-market pressure. Usability evaluation in mobile applications calls for the application of cost-effective methods, specially adapted to such constraints. We propose extending the Google Analytics for Mobile Applications basic service to store specific low-level user actions of interest for usability evaluation purposes. The solution can serve both for lab usability testing, automating quantitative data gathering, and for logging real use after application release. It is based on identification of relevant user tasks and the detailed events worth gathering, instrumentation of specific code for data gathering, and subsequent data extraction for calculating relevant usability–related variables. We validated our application in a real usability test by comparing the automatically gathered data with the information gathered by the human observer. Results shows both measurements are statistically exchangeable, opening promising new ways to perform usability testing cost-effectively and at greater scale.


Automated usability evaluation Usability testing Log file analysis Usability evaluation of mobile applications 



We would like to thank the anonymous reviewers for their valuable feedback that allowed us to greatly improve the paper.


  1. 1.
    IDC: Apple, Huawei, and Xiaomi Finish 2015 with Above Average Year-Over-Year Growth, as Worldwide Smartphone Shipments Surpass 1.4 Billion for the Year (2016).
  2. 2.
    Gartner: Worldwide Device Shipments to Grow 1.9 Percent in 2016, While End-User Spending to Decline for the First Time (2016).
  3. 3.
    Appcelerator/IDC: Voice of the Next-Generation Mobile Developer, Appcelerator/IDC Q3 2012 Mobile Developer Report (2012).
  4. 4.
    Flurry: Apps Solidify Leadership Six Years into the Mobile Revolution (2014).
  5. 5.
    Joorabchi, M.E., Mesbah, A., Kruchten, P.: Real challenges in mobile app development. In: 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2013, Baltimore, MD, USA, 10–11 October 2013, pp. 15–24. IEEE (2013). doi: 10.1109/ESEM.2013.9
  6. 6.
    Statista: Number of apps available in leading app stores as of June 2016 (2017).
  7. 7.
    Flora, H.K., Chande, S.V.: A review and analysis on mobile application development processes using agile method. Int. J. Res. Comput. Sci. 3(4), 9–18 (2013). doi: 10.7815/ijorcs.34.2013.068 CrossRefGoogle Scholar
  8. 8.
    ISO. 9241-210: Ergonomics of Human-System Interaction - Part 210: Human-Centred Design for Interactive Systems. ISO 9241-210 (2010)Google Scholar
  9. 9.
    Ferre, X., Juristo, N., Moreno, Ana M.: Improving software engineering practice with HCI aspects. In: Ramamoorthy, C.V., Lee, R., Lee, K.W. (eds.) SERA 2003. LNCS, vol. 3026, pp. 349–363. Springer, Heidelberg (2004). doi: 10.1007/978-3-540-24675-6_27 CrossRefGoogle Scholar
  10. 10.
    Coursaris, C., Kim, D.: A meta-analytical review of empirical mobile usability studies. J. Usabil. Stud. 6(3), 117–171 (2011)Google Scholar
  11. 11.
    Lettner, F., Holzmann, C.: Automated and unsupervised user interaction logging as basis for usability evaluation of mobile applications. In: Proceedings of 10th International Conference on Advances in Mobile Computing & Multimedia – MoMM 2012, vol. 118 (2012). doi: 10.1145/2428955.2428983
  12. 12.
    Ivory, M.Y., Hearst, M.A.: The state of the art in automating usability evaluation of user interfaces. ACM Comput. Surv. (CSUR) 33(4), 470–516 (2001). doi: 10.1145/503112.503114 CrossRefGoogle Scholar
  13. 13.
    Path, M.: Introducing Google Analytics for Mobile Apps. Google Mobile Ads Team (2009).
  14. 14.
    Google: Google Analytics for Mobile Apps. Accessed 2017
  15. 15.
    Tiedtke, T., Märtin, C., Gerth, N.: AWUSA–a tool for automated website usability analysis. In: Proceedings of 9th International Workshop on Design, Specification and Verification DSV-IS 2002 (2002)Google Scholar
  16. 16.
    Ivory, M.Y.: Automated Web Site Evaluation. Researchers’ and Practitioners’ Perspectives. Springer Science+Business Media, Dordrecht (2003)CrossRefzbMATHGoogle Scholar
  17. 17.
    Angulo, E., Ferre, X.: A case study on cross-platform development frameworks for mobile applications and UX. In: Proceedings of XV International Conference on Human Computer Interaction - Interacción 2014, Puerto de la Cruz, Tenerife, Spain, pp. 1–8 (2014). doi: 10.1145/2662253.2662280
  18. 18.
    Mendoza, A.: Mobile User Experience: Patterns to Make Sense of It All. Morgan Kaufmann, Waltham (2014)Google Scholar
  19. 19.
    Paternò, F., Russino, A., Santoro, C.: Remote evaluation of mobile applications. In: Winckler, M., Johnson, H., Palanque, P. (eds.) TAMODIA 2007. LNCS, vol. 4849, pp. 155–169. Springer, Heidelberg (2007). doi: 10.1007/978-3-540-77222-4_13 CrossRefGoogle Scholar
  20. 20.
    Carta, T., Paternò, F., Santana, V.: Support for remote usability evaluation of web mobile applications. In: Proceedings of 29th ACM International Conference on Design of Communication - SIGDOC 2011, pp. 129–136 (2011). doi: 10.1145/2038476.2038502
  21. 21.
    Balagtas-Fernandez, F., Hussmann, H.: A methodology and framework to simplify usability analysis of mobile applications. In: ASE2009 - 24th IEEE/ACM International Conference on Automated Software Engineering, pp. 520–524 (2009). doi: 10.1109/ASE.2009.12
  22. 22.
    Kluth, W., Krempels, K.H., Samsel, C.: Automated usability testing for mobile applications. In: WEBIST, vol. 2, pp. 149–156 (2014)Google Scholar
  23. 23.
    Feijó Filho, J., Valle, V., Prata, W.: Automated usability tests for mobile devices through live emotions logging. In: Proceedings of 17th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (MobileHCI 2015), pp. 636–643. ACM, New York (2105). doi: 10.1145/2786567.2792902
  24. 24.
    Leichtenstern, K., Erdmann, D., André, E.: EVAL-an evaluation component for mobile interfaces. In: Proceedings of 10th International Conference on Human-Computer Interaction with Mobile Devices and Services, pp. 483–484. ACM, September 2008Google Scholar
  25. 25.
    Porat, T., Schclar, A., Shapira, B.: Mate: a mobile analysis tool for usability experts. In: CHI 2013, Extended Abstracts on Human Factors in Computing Systems, pp. 265–270. ACM, April 2013Google Scholar
  26. 26.
    Hulo, A., To, J.: Developing real time tracking of user behavior, with Google analytics for mobile phone devices. Master thesis, Lund University (2015).
  27. 27.
    Rogers, Y., Sharp, H., Preece, J.: Interaction Design: Beyond Human-Computer Interaction, 3rd edn. Wiley, Chichester (2011)Google Scholar
  28. 28.
    Ferre, X., Juristo, N., Windl, H., Constantine, L.: Usability basics for software developers. IEEE Softw. 18, 22–29 (2001). doi: 10.1109/52.903160 CrossRefGoogle Scholar
  29. 29.
    Jacobson, I., Spence, I., Kerr, B.: Use-case 2.0. Commun. ACM 59(5), 61–69 (2016). doi: 10.1145/2890778 CrossRefGoogle Scholar
  30. 30.
    Constantine, L.L., Lockwood, L.A.D.: Software for Use: A Practical Guide to the Models and Methods of Usage-Centered Design. Addison-Wesley, New York (1999)Google Scholar
  31. 31.
    Rubin, J., Chisnell, D.: Handbook of Usability Testing. How to Plan, Design and Conduct Effective Tests, 2nd edn. Wiley, Indianapolis (2008)Google Scholar
  32. 32.
    Rettig, M.: Prototyping for tiny fingers. Commun. ACM 37(4), 21–27 (1994). doi: 10.1145/175276.175288 CrossRefGoogle Scholar
  33. 33.
    Mann, H.B., Whitney, D.R.: On a test of whether one of two random variables is stochastically larger than the other. Ann. Math. Stat. 18(1), 50–60 (1947). doi: 10.1214/aoms/1177730491 MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© IFIP International Federation for Information Processing 2017

Authors and Affiliations

  • Xavier Ferre
    • 1
    • 2
    Email author
  • Elena Villalba
    • 2
  • Héctor Julio
    • 2
  • Hongming Zhu
    • 3
  1. 1.High-end Expert at Tongji UniversityShanghaiChina
  2. 2.DLSIISUniversidad Politecnica de MadridMadridSpain
  3. 3.School of Software EngineeringTongji UniversityShanghaiChina

Personalised recommendations