Power to the Oracle? Design Principles for Interactive Labeling Systems in Machine Learning

  • Mario NadjEmail author
  • Merlin Knaeble
  • Maximilian Xiling Li
  • Alexander Maedche
Technical Contribution


Labeling is the process of enclosing information to some object. In machine learning it is required as ground truth to leverage the potential of supervised techniques. A key challenge in labeling is that users are not necessarily eager to behave as simple oracles, that is, repeatedly answering questions whether a label is right or wrong. In this respect, scholars acknowledge designing interactivity in labeling systems as a promising area for further improvements. In recent years, a considerable number of articles focusing on interactive labeling systems have been published. However, there is a lack of consolidated principles how to design these systems. In this article, we identify and discuss five design principles for interactive labeling systems based on a literature review and offer a frame for detecting common ground in the implementation of corresponding solutions. With these guidelines, we strive to contribute design knowledge for the increasingly important class of interactive labeling systems.


Interactive labeling Interactive machine learning Training data 


  1. 1.
    Chen NC, Drouhard M, Kocielnik R, Suh J, Aragon CR (2018) Using machine learning to support qualitative coding in social science. ACM Trans Interact Intell Syst 8(2):1–20. CrossRefGoogle Scholar
  2. 2.
    Watson H (2017) Preparing for the cognitive generation of decision support. MIS Q Exec 16(3):153–169.
  3. 3.
    Baccala M, Curran C, Garrett D, Likens S, Rao A, Ruggles A, Shehab M (2018) 2018 AI predictions—8 insights to shape business strategy. CrossRefGoogle Scholar
  4. 4.
    Anthes G (2017) Artificial intelligence poised to ride a new wave. Commun ACM 60(7):19–21. CrossRefGoogle Scholar
  5. 5.
    Liu S, Liu X, Liu Y, Feng L, Qiao H, Zhou J, Wang Y (2018) Perceptual visual interactive learning. CoRR abs/1810.10789:1–11. arXiv:1810.10789
  6. 6.
    Bernard J, Hutter M, Zeppelzauer M, Fellner D, Sedlmair M (2018) Comparing visual-interactive labeling with active learning: an experimental study. IEEE Trans Vis Comput Graph 24(1):298–308. CrossRefGoogle Scholar
  7. 7.
    Zhang L, Tong Y, Ji Q (2008) Active image abeling and its application to facial action labeling. In: Forsyth D, Torr P, Zisserman A (eds) Lecture notes in computer science (including subseries Lecture notes in artificial intelligence and lecture notes in bioinformatics). vol 5303 LNCS, Springer, Berlin, Heidelberg, pp 706–719. Google Scholar
  8. 8.
    Amershi S, Cakmak M, Knox WB, Kulesza T (2014) Power to the people: the role of humans in interactive machine learning. AI Mag 35(4):105–120. CrossRefGoogle Scholar
  9. 9.
    Cakmak M, Chao C, Thomaz AL (2010) Designing interactions for robot active learners. IEEE Trans Auton Ment Dev 2(2):108–118. CrossRefGoogle Scholar
  10. 10.
    Dudley JJ, Kristensson PO (2018) A review of user interface design for interactive machine learning. ACM Trans Interact Intell Syst 8(2):1–37. CrossRefGoogle Scholar
  11. 11.
    Nalisnik M, Gutman DA, Kong J, Cooper LAD (2015) An interactive learning framework for scalable classification of pathology images. In: International conference on big data, IEEE, pp 928–935.
  12. 12.
    Yimam SM, Biemann C, Majnaric L, Šabanović Š, Holzinger A (2016) An adaptive annotation approach for biomedical entity and relation recognition. Brain Inf 3(3):157–168. CrossRefGoogle Scholar
  13. 13.
    Gligic L, Kormilitzin A, Goldberg P, Nevado-Holgado AJ (2019) Named entity recognition in electronic health records using transfer learning bootstrapped neural networks. CoRR abs/1901.01592:1–11, arXiv:1901.01592
  14. 14.
    Kim B, Pardo B (2018) A human-in-the-loop system for sound event detection and annotation. ACM Trans Interact Intell Syst 8(2):1–23. CrossRefGoogle Scholar
  15. 15.
    Trivedi G (2016) On interactive machine learning. Available at: Accessed 9 Jan 2020
  16. 16.
    Settles B (2010) Active learning literature survey. Tech. rep. University of Wisconsin, MadisonGoogle Scholar
  17. 17.
    Fürnkranz J, Hüllermeier E (2011) Preference learning: an introduction, Springer, Berlin, pp 1–17. zbMATHGoogle Scholar
  18. 18.
    Sen S, Vig J, Riedl J (2009) Tagommenders: connecting users to items through tags. In: Proceedings of the 18th international conference on World Wide Web, ACM, New York, NY, USA, WWW ’09, pp 671–680.
  19. 19.
    Webster J, Watson RT (2002) Analyzing the past to prepare for the future: writing a literature review. MIS Q 26(2):13–23.
  20. 20.
    Fails JA, Olsen DR (2003) Interactive machine learning. In: Proceedings of the international conference on Intelligent user interfaces, ACM Press, New York, NY, USA, IUI ’03, pp 39–45.
  21. 21.
    Rashid AM, Ling K, Tassone RD, Resnick P, Kraut R, Riedl J (2006) Motivating participation by displaying the value of contribution. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, CHI ’06, pp 955–958.
  22. 22.
    Thomaz AL, Breazeal C (2008) Teachable robots: understanding human teaching behavior to build more effective robot learners. Artif Intell 172(6–7):716–737. CrossRefGoogle Scholar
  23. 23.
    Zikmund WG (2010) Business research methods. South-Western Cengage Learning, UKGoogle Scholar
  24. 24.
    Fogarty J, Tan D, Kapoor A, Winder S (2008) CueFlik: interactive concept learning in image search. In: Proceedings of the SIGCHI conference on human factors in computing systems, ACM, New York, NY, USA, CHI ’08. pp 29–38,
  25. 25.
    Xu Y, Zhang H, Miller K, Singh A, Dubrawski A (2017) Noise-tolerant interactive learning using pairwise comparisons. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R (eds) Advances in neural information processing systems 30, Curran Associates, Inc., pp 2431–2440.
  26. 26.
    Shivaswamy P, Joachims T (2012) Online structured prediction via coactive learning. CoRR abs/1205.4213:1–8. arXiv:1205.4213 MathSciNetCrossRefGoogle Scholar
  27. 27.
    Borovikov I, Harder J, Sadovsky M, Beirami A (2019) Towards interactive training of non-player characters in video games. CoRR abs/1906.00535:1–6. arXiv:1906.00535
  28. 28.
    Plummer BA, Kiapour MH, Zheng S, Piramuthu R (2018) Give me a hint! navigating image databases using human-in-the-loop feedback. CoRR abs/1809.08714:1–10. arXiv:1809.08714
  29. 29.
    Amershi S, Fogarty J, Weld D (2012) Regroup: interactive machine learning for on-demand group creation in social networks. Proceedings of the ACM annual conference on human factors in computing systems, pp 21–30.
  30. 30.
    Self JZ, Vinayagam RK, Fry JT, North C (2016) Bridging the gap between user intention and model parameters for human-in-the-loop data analytics. In: Proceedings of the Workshop on Human-In-the-Loop Data Analytics, ACM, New York, NY, USA, HILDA ’16, pp 1–6.
  31. 31.
    Dasgupta S, Poulis S, Tosh C (2019) Interactive topic modeling with anchor words. CoRR abs/1907.04919:1–7. arXiv:1907.04919
  32. 32.
    Cheng TY, Lin G, Gong X, Liu KJ, Wu SH (2016) Learning user perceived clusters with feature-level supervision. In: Lee DD, Sugiyama M, Luxburg UV, Guyon I, Garnett R (eds) Advances in neural information processing systems 29, Curran Associates, Inc., pp 532–540.
  33. 33.
    MacGlashan J, Ho MK, Loftin R, Peng B, Wang G, Roberts DL, Taylor ME, Littman ML (2017) Interactive learning from policy-dependent human feedback. In: Precup D, Teh YW (eds) Proceedings of the international conference on machine learning, PMLR, International Convention Centre, Sydney, Australia, Proceedings of Machine Learning Research, vol 70, pp 2285–2294.
  34. 34.
    Porter R, Theiler J, Hush D (2013) Interactive machine learning in data exploitation. Comput Sci Eng 15(5):12–20. CrossRefGoogle Scholar
  35. 35.
    Guo X, Wu H, Cheng Y, Rennie S, Tesauro G, Feris R (2018) Dialog-based interactive image retrieval. In: Bengio S, Wallach H, Larochelle H, Grauman K, Cesa-Bianchi N, Garnett R (eds) Advances in neural information processing systems 31, Curran Associates, Inc., pp 678–688.
  36. 36.
    Hebbalaguppe R, McGuinness K, Kuklyte J, Healy G, O’Connor N, Smeaton A (2013) How interaction methods affect image segmentation: User experience in the task. In: Workshop on user-centered computer vision, IEEE, pp 19–24.
  37. 37.
    Acuna D, Ling H, Kar A, Fidler S (2018) Efficient interactive annotation of segmentation datasets with polygon-rnn++. CoRR abs/1803.09693:1–21. arXiv:1803.09693
  38. 38.
    Lopresti D, Nagy G (2012) Optimal data partition for semi-automated labeling. In: Proceedings of the international conference on pattern recognition, IEEE, pp 286–289Google Scholar
  39. 39.
    Bryan N, Mysore G (2013) An efficient posterior regularized latent variable model for interactive sound source separation. In: Dasgupta S, McAllester D (eds) Proceedings of the international conference on machine learning, PMLR, Atlanta, Georgia, USA, Proceedings of Machine Learning Research, vol 28, pp 208–216.
  40. 40.
    Stumpf S, Rajaram V, Li L, Burnett M, Dietterich T, Sullivan E, Drummond R, Herlocker J (2007) Toward harnessing user feedback for machine learning. In: Proceedings of the international conference on intelligent user interfaces, ACM, New York, NY, USA, IUI ’07, pp 82–91.
  41. 41.
    Boyko A, Funkhouser T (2014) Cheaper by the dozen: group annotation of 3D Data. In: Proceedings of the annual ACM symposium on user interface software and technology, ACM, New York, NY, USA, pp 33–42.
  42. 42.
    Kim B, Glassman E, Johnson B, Shah J (2015) iBCM: Interactive Bayesian case model empowering humans via intuitive interaction.
  43. 43.
    Sun Q, DeJong G (2005) Explanation-augmented SVM. In: Proceedings of the international conference on machine learning, ACM, New York, NY, USA, ICML ’05, pp 864–871.
  44. 44.
    Early K, Fienberg SE, Mankoff J (2016) Test time feature ordering with FOCUS. In: Proceedings of the ACM international joint conference on pervasive and ubiquitous computing, ACM, New York, NY, USA, UbiComp ’16, pp 992–1003.
  45. 45.
    Weigl E, Walch A, Neissl U, Meyer-Heye P, Heidl W, Radauer T, Lughofer E, Eitzinger C (2016) MapView: graphical data representation for active learning. CEUR Workshop Proceedings, Sun SITE, Aachen, 1707:3–8Google Scholar
  46. 46.
    Datta S, Adar E (2018) CommunityDiff: visualizing community clustering algorithms. ACM Trans Knowl Discov Data 12(1):1–34. CrossRefGoogle Scholar
  47. 47.
    Jain S, Munukutla S, Held D (2019) Few-shot point cloud region annotation with human in the loop. CoRR abs/1906.04409:1–6. arXiv:1906.04409
  48. 48.
    Wallace BC, Small K, Brodley CE, Lau J, Trikalinos TA (2012) Deploying an interactive machine learning system in an evidence-based practice center: Abstrackr. In: Proceedings of the SIGHIT international health informatics symposium, ACM, New York, NY, USA, pp 819–824.
  49. 49.
    Zhu Y, Yang K (2019) Tripartite active learning for interactive anomaly discovery. IEEE Access 7:63195–63203. CrossRefGoogle Scholar
  50. 50.
    Yan Y, Rosales R, Fung G, Dy JG (2011) Active learning from crowds. In: Proceedings of the international conference on machine learning, Omnipress, USA, ICML’11, pp 1161–1168.
  51. 51.
    Cui S, Dumitru CO, Datcu M (2014) Semantic annotation in earth observation based on active learning. Int J Image Data Fusion 5(2):152–174. CrossRefGoogle Scholar
  52. 52.
    Burkovski A, Kessler W, Heidemann G, Kobdani H, Schütze H (2011) Self organizing maps in NLP: exploration of coreference feature space. In: Proceedings of the international conference on advances in self-organizing maps, Springer, Berlin, Heidelberg, pp 228–237. CrossRefGoogle Scholar
  53. 53.
    Rosenthal SL, Dey AK (2010) Towards maximizing the accuracy of human-labeled sensor data. In: Proceedings of the international conference on intelligent user interfaces, ACM, New York, NY, USA, pp 259–268.
  54. 54.
    Kagy J, Kayadelen T, Ma J, Rostamizadeh A, Strnadová J (2019) The practical challenges of active learning: Lessons learned from live experimentation. CoRR abs/1907.00038:1–7. arXiv:1907.00038
  55. 55.
    Benato BC, Telea AC, Falcão AX (2018) Semi-supervised learning with interactive label propagation guided by feature space projections. In: SIBGRAPI conference on graphics, patterns and images, pp 392–399.
  56. 56.
    Harvey N, Porter R (2016) User-driven sampling strategies in image exploitation. Inf Vis 15(1):64–74. CrossRefGoogle Scholar

Copyright information

© Gesellschaft für Informatik e.V. and Springer-Verlag GmbH Germany, part of Springer Nature 2020

Authors and Affiliations

  • Mario Nadj
    • 1
    Email author
  • Merlin Knaeble
    • 1
  • Maximilian Xiling Li
    • 1
  • Alexander Maedche
    • 1
  1. 1.Karlsruhe Institute of Technology (KIT)KarlsruheGermany

Personalised recommendations