Abstract
Demands on more transparency of the backbox nature of machine learning models have led to the recent rise of human-in-the-loop in machine learning, i.e. processes that integrate humans in the training and application of machine learning models. The present work argues that this process requirement does not represent an obstacle but an opportunity to optimize the design process. Hence, this work proposes a new process framework, Human-in-the-loop Design Cycles – a design process that integrates the structural elements of agile and design thinking process, and controls the training of a machine learning model by the human in the loop.
The Human-in-the-loop Design Cycles process replaces the qualitative user testing by a quantitative psychometric measurement instrument for design perception. The generated user feedback serves to train a machine learning model and to instruct the subsequent design cycle along four design dimensions (novelty, energy, simplicity, tool).
Mapping the four-dimensional user feedback into user stories and priorities, the design sprint thus transforms the user feedback directly into the implementation process. The human in the loop is a quality engineer who scrutinizes the collected user feedback to prevent invalid data to enter machine learning model training.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Brown, T.: Design thinking. Harv. Bus. Rev. 86, 85–92 (2008)
Vredenburg, K., Isensee, S., Righi, C.: User-Centered Design: An Integrated Approach. Prentice Hall PTR, Upper Saddle River (2002)
Giacomin, J.: What is human centred design. Des. J. 17, 606–623 (2014). https://doi.org/10.2752/175630614X14056185480186
Maaike, K., Rianne, V., Janneke, S.: Capturing the value of design thinking in different innovation practices. Int. J. Des. 11, 25–40 (2017)
Gilpin, L.H., Bau, D., Yuan, B.Z., Bajwa, A., Specter, M., Kagal, L.: Explaining explanations: an overview of interpretability of machine learning. In: Proceedings - 2018 IEEE 5th International Conference on Data Science and Advanced Analytics DSAA 2018, pp. 80–89 (2019). https://doi.org/10.1109/DSAA.2018.00018
Zanzotto, F.M.: Viewpoint: human-in-the-loop artificial intelligence. J. Artif. Intell. Res. 64, 243–253 (2019). https://doi.org/10.1613/jair.1.11345
Gillies, M., et al.: Human-centered machine learning. In: Conference on Human Factors Computer System – Proceedings, 07–12 May 2016, pp. 3558–3565 (2016). https://doi.org/10.1145/2851581.2856492
IDEO: IDEO Human-Centered Design Toolkit, 2nd edn. (2008). https://doi.org/10.1002/ejoc.201200111
Beck, K., et al.: Manifesto for Agile Software Development (2001)
Mason, J.: Qualitative Researching. Sage Publications, Thousand Oaks/New Delhi (2002). https://doi.org/10.1159/000105503
Giles, D.C.: Advanced Research Methods in Psychology. Routledge, East Sussex (2002)
Bayazit, N.: Investigating design: a review of forty years of design research. Des. Issues 20, 16–29 (2004). https://doi.org/10.1162/074793604772933739
Nickerson, R.S.: Confirmation bias: a ubiquitous phenomenon in many guises. Rev. Gen. Psychol. 2, 175–220 (1998)
Powell, M.B., Hughes-Scholes, C.H., Sharman, S.J.: Skill in interviewing reduces confirmation bias. J. Invest. Psychol. Offender Profiling 9, 126–134 (2012). https://doi.org/10.1002/jip.1357
Kline, P.: The New Psychometrics: Science, Psychology, and Measurement. Routledge (1998)
Price, P.C.: Psychology Research Methods: Core Skills and Concepts v. 1.0. Psychology, pp. 215–260 (2012). https://doi.org/10.24926/8668.2201
Tabachnick, B.G., Fidell, L.S.: Using Multivariate Statistics. Pearson, Boston (2019)
Cohen, P.: Statistical Power Analysis for the Behavioral Sciences. Academic Press, New York (1977)
Nushi, B., Kamar, E., Horvitz, E., Kossmann, D.: On human intellect and machine failures: Troubleshooting integrative machine learning systems. In: 31st AAAI Conference on Artificial Intelligence, AAAI 2017, pp. 1017–1025 (2017)
Xin, D.D., Ma, L.L., Liu, J.J., Macke, S.S., Song, S.S., Parameswaran, A.A.: Accelerating human-in-the-loop machine learning: challenges and opportunities. In: Proceedings of 2nd Workshop on Data Management for End-to-End Machine Learning, DEEM 2018 - Conjunction with 2018 ACM SIGMOD/PODS Conference (2018). https://doi.org/10.1145/3209889.3209896
Cockburn, A.: Agile Software Development: Software Through People. Addison-Wesley (2001)
Fowler, M.: The New Methodology (2000)
Beck, K., Andres, C.: Extreme Programming Explained: Embrace Change. Addison-Wesley Longman, Amsterdam (2000)
Schwaber, K., Beedle, M.: Agile Software Development with Scrum. Prentice Hall (2002)
Knapp, J., Zeratsky, J., Kowitz, B.: Sprint: How to Solve Big Problems and Test New Ideas in Just Five Days. Simon & Schuster (2016)
Jalote, P.: Timeboxing: a process model for iterative software development. J. Syst. Softw. 70, 117–127 (2003)
Bittner, K., Spence, I.: Managing Iterative Software Development Projects. Addison-Wesley (2006)
Cohn, M.: User Stories Applied: For Agile Software Development. Addison Wesley Longman Publishing Co., Inc., Redwood City (2004)
So, C.: What makes good design? Revealing the predictive power of emotions and design dimensions in non-expert design vocabulary. Des. J. 22, 325–349 (2019). https://doi.org/10.1080/14606925.2019.1589204
Beck, K., Fowler, M.: Planning Extreme Programming. Addison-Wesley (2000)
Acknowledgment
This research was supported by the Yonsei University Faculty Research Fund of 2019-22-0199.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
So, C. (2020). Human-in-the-Loop Design Cycles – A Process Framework that Integrates Design Sprints, Agile Processes, and Machine Learning with Humans. In: Degen, H., Reinerman-Jones, L. (eds) Artificial Intelligence in HCI. HCII 2020. Lecture Notes in Computer Science(), vol 12217. Springer, Cham. https://doi.org/10.1007/978-3-030-50334-5_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-50334-5_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-50333-8
Online ISBN: 978-3-030-50334-5
eBook Packages: Computer ScienceComputer Science (R0)