Usability improvement through A/B testing and refactoring
Usability evaluation is an essential task in web application development. There have been several attempts to integrate user-centered design with agile methods, but it is hard to synchronize their practices. User testing is very valuable to learn from feedback of actual use, but it remains expensive to find and solve usability problems. Furthermore, the high cost of usability evaluation forces small/medium-sized companies to trust the first solution applied, without actually testing the success of the solution or considering a possible regression in usability, as could be highlighted by an iterative testing method. In this article, we advocate for a usability improvement cycle oriented by user feedback, and compatible with an agile development process. We propose an iterative method supported by a toolkit that allows usability experts to design user tests, run them remotely, analyze results, and assess alternative solutions to usability problems similarly to A/B testing. Each solution is created by applying client-side web refactorings, i.e., changes to the web pages in the client which are meant to improve usability. The main benefit of our approach is that it reduces the overall cost of user testing and particularly, A/B testing, by applying refactorings to create alternative solutions without modifying the application’s server code. By making it affordable for usability experts to apply the method in parallel with the development cycle, we aim to encourage them to incorporate user feedback and try different ideas to discover the best-performing solution in terms of the metrics of interest.
KeywordsUsability evaluation A/B testing Web refactoring Agile methods External quality
The authors acknowledge the support from the Argentinian National Agency for Scientific and Technical Promotion (ANPCyT), grant number PICT-2015-3000.
- Brooke, J. (1996). SUS - a quick and dirty usability scale. Usability Evaluation in Industry, 189, 4–7.Google Scholar
- Burzacca, P., and Paternò, F. (2013) Remote usability evaluation of mobile web applications. In Proceedings of the 15th Int. Conf. on Human-Computer Interaction Vol. 1, pp. 241–248.Google Scholar
- Carta, T., Paternò, F., & De Santana, V. (2011). Web usability probe: A tool for supporting remote usability evaluation of web sites. In INTERACT 2011. LNCS, 6949, 349–357.Google Scholar
- Da Silva, T.S., Silveira, M.S., De O. Melo, C., and Parzianello, L.C. (2013) Understanding the UX designer’s role within agile teams. In LNCS Vol. 8012, pp. 599–609.Google Scholar
- Diaz, O., & Arellano, C. (2015). The augmented web: Rationales, opportunities, and challenges on browser-side transcoding. ACM Transaction on the Web, 9(8), 1–8:30.Google Scholar
- Düchting, M., Zimmermann, D., and Nebe, K. (2007) Incorporating user centered requirement engineering into agile software development. In 12th Int. Conf. Human-computer interaction pp. 58–67.Google Scholar
- Firmenich, D., Firmenich, S., Rivero, J. M., Antonelli, L., & Rossi, G. (2016). CrowdMock: An approach for defining and evolving web augmentation requirements. Requirements Engineering, 1–29.Google Scholar
- Fowler, M. (1997) Analysis patterns: Reusable object models. Addison Wesley. Addison Wesley.Google Scholar
- Fowler, M. (1999) Refactoring: Improving the design of existing code, 431 p. Addison-Wesley.Google Scholar
- Garrido, A., Rossi, G., Medina, N. M., Grigera, J., & Firmenich, S. (2014). Improving accessibility of web interfaces: Refactoring to the rescue. Universal Access in the Information Society, 13(4), 387–399.Google Scholar
- Genov, and Alex. (2005). Iterative usability testing as continuous feedback: A control systems perspective. Journal of Usability Studies, 1, 18–27.Google Scholar
- Hartson, H.R., and Castillo, J.C. (1998) Remote evaluation for post-deployment usability improvement. In Proceedings of the working conference on Advanced visual interfaces - AVI ‘98 pp. 22–29. ACM Press, New YorkGoogle Scholar
- ISO, I (2011) ISO/IEC 25010 - Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - System and software quality models.Google Scholar
- Jurca, G., Hellmann, T.D., and Maurer, F. (2014) Integrating agile and user-centered design: A systematic mapping and review of evaluation and validation studies of agile-UX. In Proceedings - 2014 Agile Conference, AGILE 2014.Google Scholar
- Kohavi, R., and Longbotham, R. (2015) Online controlled experiments and a/B tests motivation and background. Encyclopedia of Machine Learning and Data Mining.Google Scholar
- Lee, J.C., and McCrickard, D.S. (2007) Towards extreme(ly) usable software: Exploring tensions between usability and agile software development. In Agile 2007 pp. 59–71.Google Scholar
- Nielsen, J. (2000) Why You Only Need to Test with 5 Users. https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/. Last accessed June 16, 2018.
- Nielsen, J., and Loranger, H. (2006) Prioritizing Web Usability, 406 p. (C. Peri, Ed.). Pearson Education.Google Scholar
- Obendorf, H., and Finck, M. (2008) Scenario-based usability engineering techniques in agile development processes. Proceeding of the twenty-sixth annual CHI conference extended abstracts on Human factors in computing systems - CHI ‘08, 2159.Google Scholar
- Rubin, K. (2012) Essential scrum: A practical guide to the most popular agile process. Addison-Wesley.Google Scholar
- Rubin, J., and Chisnell, D. (2008) Handbook of usability testing: Howto plan, design, and conduct effective tests. Wiley.Google Scholar
- Salvador, C., Nakasone, A., and Pow-Sang, J.A. (2014) A systematic review of usability techniques in agile methodologies. In Proceedings of the 7th Euro American Conference on Telematics and Information Systems - EATIS ‘14.Google Scholar
- Schissel, J. (2014) From MVP to MTP: Accelerating Development at Your Company. https://www.linkedin.com/pulse/20140710165313-4729218-from-mvp-to-mtp-accelerating-development-at-your-company/. Last accessed June 16, 2018.
- Silva da Silva, T., Martin, A., Maurer, F., & Silveira, M. (2011). User-centered design and agile methods: A systematic review. Agile Conference (AGILE), 2011, 77–86.Google Scholar
- Speicher, M., Both, A., and Gaedke, M. (2014) Ensuring web Interface quality through usability-based split testing. In Icwe, LNCS 8541 pp. 93–110.Google Scholar
- UID (2018) AttrakDiff. User Interface Design GmbH. http://attrakdiff.de/. Last accessed June 16, 2018.
- Urbieta, M., Firmenich, S., Maglione, P., Rossi, G., and Olivero, M.A. (2017) A model-driven approach for empowering advance web augmentation - from client-side to server-side support. In WEBIST pp. 444–454.Google Scholar
- Yoder, J. W., Balaguer, F., & Johnson, R. (1999). From analysis to design of the observation pattern. Metadata and Active Object-Model Pattern Mining Workshop. Denver: OOPSLA'99.Google Scholar