Software Quality Journal

, Volume 27, Issue 1, pp 203–240 | Cite as

Usability improvement through A/B testing and refactoring

  • Sergio Firmenich
  • Alejandra GarridoEmail author
  • Julián Grigera
  • José Matías Rivero
  • Gustavo Rossi


Usability evaluation is an essential task in web application development. There have been several attempts to integrate user-centered design with agile methods, but it is hard to synchronize their practices. User testing is very valuable to learn from feedback of actual use, but it remains expensive to find and solve usability problems. Furthermore, the high cost of usability evaluation forces small/medium-sized companies to trust the first solution applied, without actually testing the success of the solution or considering a possible regression in usability, as could be highlighted by an iterative testing method. In this article, we advocate for a usability improvement cycle oriented by user feedback, and compatible with an agile development process. We propose an iterative method supported by a toolkit that allows usability experts to design user tests, run them remotely, analyze results, and assess alternative solutions to usability problems similarly to A/B testing. Each solution is created by applying client-side web refactorings, i.e., changes to the web pages in the client which are meant to improve usability. The main benefit of our approach is that it reduces the overall cost of user testing and particularly, A/B testing, by applying refactorings to create alternative solutions without modifying the application’s server code. By making it affordable for usability experts to apply the method in parallel with the development cycle, we aim to encourage them to incorporate user feedback and try different ideas to discover the best-performing solution in terms of the metrics of interest.


Usability evaluation A/B testing Web refactoring Agile methods External quality 



The authors acknowledge the support from the Argentinian National Agency for Scientific and Technical Promotion (ANPCyT), grant number PICT-2015-3000.


  1. Benigni, G., Gervasi, O., Passeri, F., & Kim, T. (2010). USABAGILE_web: a web agile usability approach for web site design. ICCSA (2) - Lecture Notes in Computer Science (LNCS), 6017, 422–431.CrossRefGoogle Scholar
  2. Brooke, J. (1996). SUS - a quick and dirty usability scale. Usability Evaluation in Industry, 189, 4–7.Google Scholar
  3. Burzacca, P., and Paternò, F. (2013) Remote usability evaluation of mobile web applications. In Proceedings of the 15th Int. Conf. on Human-Computer Interaction Vol. 1, pp. 241–248.Google Scholar
  4. Carta, T., Paternò, F., & De Santana, V. (2011). Web usability probe: A tool for supporting remote usability evaluation of web sites. In INTERACT 2011. LNCS, 6949, 349–357.Google Scholar
  5. Da Silva, T.S., Silveira, M.S., De O. Melo, C., and Parzianello, L.C. (2013) Understanding the UX designer’s role within agile teams. In LNCS Vol. 8012, pp. 599–609.Google Scholar
  6. Detweiler, M. (2007). Managing UCD within agile projects. Interactions, 14(3), 40.CrossRefGoogle Scholar
  7. Diaz, O., & Arellano, C. (2015). The augmented web: Rationales, opportunities, and challenges on browser-side transcoding. ACM Transaction on the Web, 9(8), 1–8:30.Google Scholar
  8. Distante, D., Garrido, A., Camelier-Carvajal, J., Giandini, R., & Rossi, G. (2014). Business processes refactoring to improve usability in E-commerce applications. Electronic Commerce Research, 14(4), 497–529.CrossRefGoogle Scholar
  9. Düchting, M., Zimmermann, D., and Nebe, K. (2007) Incorporating user centered requirement engineering into agile software development. In 12th Int. Conf. Human-computer interaction pp. 58–67.Google Scholar
  10. Fernandez, A., Insfran, E., & Abrahão, S. (2011). Usability evaluation methods for the web: A systematic mapping study. Information and Software Technology, 53(8), 789–817.CrossRefGoogle Scholar
  11. Fernandez, A., Abrahão, S., & Insfran, E. (2013). Empirical validation of a usability inspection method for model-driven web development. Journal of Systems and Software, 86(1), 161–186.CrossRefGoogle Scholar
  12. Firmenich, D., Firmenich, S., Rivero, J. M., Antonelli, L., & Rossi, G. (2016). CrowdMock: An approach for defining and evolving web augmentation requirements. Requirements Engineering, 1–29.Google Scholar
  13. Fowler, M. (1997) Analysis patterns: Reusable object models. Addison Wesley. Addison Wesley.Google Scholar
  14. Fowler, M. (1999) Refactoring: Improving the design of existing code, 431 p. Addison-Wesley.Google Scholar
  15. Garrido, A., Rossi, G., & Distante, D. (2011). Refactoring for usability in web applications. IEEE Software, 28(3), 60–67.CrossRefGoogle Scholar
  16. Garrido, A., Firmenich, S., Rossi, G., Grigera, J., Medina-Medina, N., & Harari, I. (2013). Personalized web accessibility using client-side refactoring. IEEE Internet Computing, 17(4), 58–66.CrossRefGoogle Scholar
  17. Garrido, A., Rossi, G., Medina, N. M., Grigera, J., & Firmenich, S. (2014). Improving accessibility of web interfaces: Refactoring to the rescue. Universal Access in the Information Society, 13(4), 387–399.Google Scholar
  18. Genov, and Alex. (2005). Iterative usability testing as continuous feedback: A control systems perspective. Journal of Usability Studies, 1, 18–27.Google Scholar
  19. George, C. A. (2005). Usability testing and design of a library website: An iterative approach. OCLC Systems & Services: International digital library perspectives, 21(3), 167–180.CrossRefGoogle Scholar
  20. Grigera, J., Garrido, A., Panach, J. I., Distante, D., & Rossi, G. (2016). Assessing refactorings for usability in e-commerce applications. Empirical Software Engineering, 21(3), 1224–1271.CrossRefGoogle Scholar
  21. Grigera, J., Garrido, A., Rivero, J. M., & Rossi, G. (2017). Automatic detection of usability smells in web applications. International Journal of Human-Computer Studies, 97, 129–148.CrossRefGoogle Scholar
  22. Hartson, H.R., and Castillo, J.C. (1998) Remote evaluation for post-deployment usability improvement. In Proceedings of the working conference on Advanced visual interfaces - AVI ‘98 pp. 22–29. ACM Press, New YorkGoogle Scholar
  23. Hartson, H. R., Andre, T. S., & Williges, R. C. (2003). Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction, 15(1), 145–181.CrossRefGoogle Scholar
  24. Hassenzahl, M. (2006). Hedonic, emotional, and experiential perspectives on product quality. In Encyclopedia of human computer interaction (pp. 266–268).CrossRefGoogle Scholar
  25. ISO, I (2011) ISO/IEC 25010 - Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - System and software quality models.Google Scholar
  26. Jurca, G., Hellmann, T.D., and Maurer, F. (2014) Integrating agile and user-centered design: A systematic mapping and review of evaluation and validation studies of agile-UX. In Proceedings - 2014 Agile Conference, AGILE 2014.Google Scholar
  27. Kohavi, R., and Longbotham, R. (2015) Online controlled experiments and a/B tests motivation and background. Encyclopedia of Machine Learning and Data Mining.Google Scholar
  28. Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R. M. (2009). Controlled experiments on the web: Survey and practical guide. Data Mining and Knowledge Discovery, 18(1), 140–181.MathSciNetCrossRefGoogle Scholar
  29. Lee, J.C., and McCrickard, D.S. (2007) Towards extreme(ly) usable software: Exploring tensions between usability and agile software development. In Agile 2007 pp. 59–71.Google Scholar
  30. Nielsen, J. (2000) Why You Only Need to Test with 5 Users. Last accessed June 16, 2018.
  31. Nielsen, J., and Loranger, H. (2006) Prioritizing Web Usability, 406 p. (C. Peri, Ed.). Pearson Education.Google Scholar
  32. Obendorf, H., and Finck, M. (2008) Scenario-based usability engineering techniques in agile development processes. Proceeding of the twenty-sixth annual CHI conference extended abstracts on Human factors in computing systems - CHI ‘08, 2159.Google Scholar
  33. Paganelli, L., & Paternò, F. (2003). Tools for remote usability evaluation of web applications through browser logs and task models. Behavior Research Methods, Instruments, & Computers : A Journal of the Psychonomic Society, Inc, 35(3), 369–378.CrossRefGoogle Scholar
  34. Panach, J. I., Juristo, N., Valverde, F., & Pastor, O. (2015). A framework to identify primitives that represent usability within model-driven development methods. Information and Software Technology, 58, 338–354.CrossRefGoogle Scholar
  35. Rubin, K. (2012) Essential scrum: A practical guide to the most popular agile process. Addison-Wesley.Google Scholar
  36. Rubin, J., and Chisnell, D. (2008) Handbook of usability testing: Howto plan, design, and conduct effective tests. Wiley.Google Scholar
  37. Salvador, C., Nakasone, A., and Pow-Sang, J.A. (2014) A systematic review of usability techniques in agile methodologies. In Proceedings of the 7th Euro American Conference on Telematics and Information Systems - EATIS ‘14.Google Scholar
  38. de Santana, V. F., & Baranauskas, M. C. C. (2015). WELFIT: A remote evaluation tool for identifying web usage patterns through client-side logging. International Journal of Human-Computer Studies, 76, 40–49.CrossRefGoogle Scholar
  39. Schissel, J. (2014) From MVP to MTP: Accelerating Development at Your Company. Last accessed June 16, 2018.
  40. Silva da Silva, T., Martin, A., Maurer, F., & Silveira, M. (2011). User-centered design and agile methods: A systematic review. Agile Conference (AGILE), 2011, 77–86.Google Scholar
  41. Speicher, M., Both, A., and Gaedke, M. (2014) Ensuring web Interface quality through usability-based split testing. In Icwe, LNCS 8541 pp. 93–110.Google Scholar
  42. UID (2018) AttrakDiff. User Interface Design GmbH. Last accessed June 16, 2018.
  43. Urbieta, M., Firmenich, S., Maglione, P., Rossi, G., and Olivero, M.A. (2017) A model-driven approach for empowering advance web augmentation - from client-side to server-side support. In WEBIST pp. 444–454.Google Scholar
  44. Williams, L., & Cockburn, A. (2003). Agile software development: It’s about feedback and change. Computer, 36(6), 39–43.CrossRefGoogle Scholar
  45. Yoder, J. W., Balaguer, F., & Johnson, R. (1999). From analysis to design of the observation pattern. Metadata and Active Object-Model Pattern Mining Workshop. Denver: OOPSLA'99.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.LIFIA, Fac. de InformáticaUniversidad Nacional de La PlataLa PlataArgentina
  2. 2.CONICETBuenos AiresArgentina
  3. 3.CICBuenos AiresArgentina

Personalised recommendations