Immersive Human-Centered Computational Analytics

  • Wolfgang StuerzlingerEmail author
  • Tim Dwyer
  • Steven Drucker
  • Carsten Görg
  • Chris North
  • Gerik Scheuermann
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11190)


In this chapter we seek to elevate the role of the human in human-machine cooperative analysis through a careful consideration of immersive design principles. We consider both strategic immersion through more accessible systems as well as enhanced understanding and control through immersive interfaces that enable rapid workflows. We extend the classic sensemaking loop from visual analytics to incorporate multiple views, scenarios, people, and computational agents. We consider both sides of machine/human collaboration: allowing the human to more fluidly control the machine process; and also allowing the human to understand the results, derive insights and continue the analytic cycle. We also consider system and algorithmic implications of enabling real-time control and feedback in immersive human-centered computational analytics.


Human-in-the-loop analytics Visual analytics Data visualization 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Adams, E.: The designer’s notebook: Postmodernism and the 3 types of immersion (2004).
  2. 2.
    Andrews, C., Endert, A., North, C.: Space to think: large high-resolution displays for sensemaking. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 55–64. ACM (2010)Google Scholar
  3. 3.
    Bavoil, L., et al.: Vistrails: enabling interactive multiple-view visualizations. In: IEEE Visualization, VIS 2005, pp. 135–142, October 2005.
  4. 4.
    Bradel, L., North, C., House, L., Leman, S.: Multi-model semantic interaction for text analytics. In: 2014 IEEE Conference on Visual Analytics Science and Technology (VAST), pp. 163–172, October 2014.
  5. 5.
    Card, S.K., Mackinlay, J.D., Shneiderman, B.: Readings in Information Visualization: Using Vision to Think. Morgan Kaufmann, San Francisco (1999)Google Scholar
  6. 6.
    Card, S.K., Robertson, G.G., Mackinlay, J.D.: The information visualizer, an information workspace. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 181–186. ACM (1991)Google Scholar
  7. 7.
    Carmack, J.: Latency mitigation strategies (2013).
  8. 8.
    Ceneda, D., et al.: Characterizing guidance in visual analytics. IEEE Trans. Vis. Comput. Graph. 23(1), 111–120 (2017). Scholar
  9. 9.
    Cernea, D., Ebert, A., Kerren, A.: A study of emotion-triggered adaptation methods for interactive visualization. In: UMAP 2013 Extended Proceedings: Late-Breaking Results, Project Papers and Workshop Proceedings of the 21st Conference on User Modeling, Adaptation, and Personalization. CEUR workshop proceedings, vol. 997, pp. 9–16. (2013)Google Scholar
  10. 10.
    Chen, X., Self, J.Z., House, L., North, C.: Be the data: a new approach for immersive analytics. In: IEEE Virtual Reality Workshop on Immersive Analytics (2016)Google Scholar
  11. 11.
    Choo, J., Lee, C., Reddy, C.K., Park, H.: Utopian: user-driven topic modeling based on interactive nonnegative matrix factorization. IEEE Trans. Vis. Comput. Graph. 19(12), 1992–2001 (2013)CrossRefGoogle Scholar
  12. 12.
    Chuang, J., Ramage, D., Manning, C., Heer, J.: Interpretation and trust: designing model-driven visualizations for text analysis. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 443–452. ACM (2012)Google Scholar
  13. 13.
    Chung, H., North, C., Joshi, S., Chen, J.: Four considerations for supporting visual analysis in display ecologies. In: 2015 IEEE Conference on Visual Analytics Science and Technology (VAST), pp. 33–40, October 2015Google Scholar
  14. 14.
    Collins, C., Carpendale, S.: Vislink: revealing relationships amongst visualizations. IEEE Trans. Vis. Comput. Graph. 13(6), 1192–1199 (2007). Scholar
  15. 15.
    Darragh, J.J., Witten, I.H.: Adaptive predictive text generation and the reactive keyboard. Interact. Comput. 3(1), 27–50 (1991)CrossRefGoogle Scholar
  16. 16.
    Doleisch, H.: SimVis: interactive visual analysis of large and time-dependent 3D simulation data. In: Proceedings of the 39th Conference on Winter Simulation: 40 Years! The Best Is Yet to Come, pp. 712–720. IEEE Press (2007)Google Scholar
  17. 17.
    Endert, A., Han, C., Maiti, D., House, L., Leman, S., North, C.: Observation-level interaction with statistical models for visual analytics. In: 2011 IEEE Conference on Visual Analytics Science and Technology (VAST), pp. 121–130, October 2011Google Scholar
  18. 18.
    Endert, A., Fiaux, P., North, C.: Semantic interaction for sensemaking: inferring analytical reasoning for model steering. IEEE Trans. Vis. Comput. Graph. 18(12), 2879–2888 (2012)CrossRefGoogle Scholar
  19. 19.
    Endert, A., Fox, S., Maiti, D., North, C.: The semantics of clustering: analysis of user-generated spatializations of text documents. In: Proceedings of the International Working Conference on Advanced Visual Interfaces, pp. 555–562. ACM (2012)Google Scholar
  20. 20.
    Endert, A., Hossain, M.S., Ramakrishnan, N., North, C., Fiaux, P., Andrews, C.: The human is the loop: new directions for visual analytics. J. Intell. Inf. Syst. 43(3), 411–435 (2014)CrossRefGoogle Scholar
  21. 21.
    Fisher, D., Popov, I., Drucker, S., Schraefel, M.: Trust me, I’m partially right: incremental visualization lets analysts explore large datasets faster. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1673–1682. ACM (2012)Google Scholar
  22. 22.
    Goodwin, S., Mears, C., Dwyer, T., de la Banda, M.G., Tack, G., Wallace, M.: What do constraint programming users want to see? Exploring the role of visualisation in profiling of models and search. IEEE Trans. Vis. Comput. Graph. 23(1), 281–290 (2017)CrossRefGoogle Scholar
  23. 23.
    Heer, J., Mackinlay, J., Stolte, C., Agrawala, M.: Graphical histories for visualization: supporting analysis, communication, and evaluation. IEEE Trans. Vis. Comput. Graph. 14(6), 1189–1196 (2008). Scholar
  24. 24.
    Heer, J., Shneiderman, B.: Interactive dynamics for visual analysis. Commun. ACM 55(4), 45–54 (2012). Scholar
  25. 25.
    Heine, C., et al.: A survey of topology-based methods in visualization. Comput. Graph. Forum 35(3), 643–667 (2016)CrossRefGoogle Scholar
  26. 26.
    Heun, V., von Kapri, A., Maes, P.: Perifoveal display: combining foveal and peripheral vision in one visualization. In: Proceedings of the 2012 ACM Conference on Ubiquitous Computing, UbiComp 2012, pp. 1150–1155. ACM (2012)Google Scholar
  27. 27.
    Hollan, J., Hutchins, E., Kirsh, D.: Distributed cognition: toward a new foundation for human-computer interaction research. ACM Trans. Comput. Hum. Interact. 7(2), 174–196 (2000)CrossRefGoogle Scholar
  28. 28.
    Isenberg, P., Elmqvist, N., Scholtz, J., Cernea, D., Ma, K.L., Hagen, H.: Collaborative visualization: definition, challenges, and research agenda. Inf. Vis. 10(4), 310–326 (2011). Scholar
  29. 29.
    Jänicke, H., Böttinger, M., Tricoche, X., Scheuermann, G.: Automatic detection and visualization of distinctive structures in 3D unsteady multi-fields. Comput. Graph. Forum 27(3), 767–774 (2008)CrossRefGoogle Scholar
  30. 30.
    Kerren, A., Schreiber, F.: Toward the role of interaction in visual analytics. In: Proceedings of the Winter Simulation Conference, WSC 2012, pp. 420:1–420:13 (2012).
  31. 31.
    Liu, J., Dwyer, T., Marriott, K., Millar, J., Haworth, A.: Understanding the relationship between interactive optimisation and visual analytics in the context of prostate brachytherapy. IEEE Trans. Vis. Comput. Graph. 24(1), 319–329 (2018)CrossRefGoogle Scholar
  32. 32.
    Liu, Y., Jin, R., Jain, A.K.: Boostcluster: boosting clustering by pairwise constraints. In: Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 450–459. ACM (2007)Google Scholar
  33. 33.
    MacKay, W.E.: Is paper safer? The role of paper flight strips in air traffic control. ACM Trans. Comput. Hum. Inter. 6(4), 311–340 (1999)CrossRefGoogle Scholar
  34. 34.
    Mahyar, N., Tory, M.: Supporting communication and coordination in collaborative sensemaking. IEEE Trans. Vis. Comput. Graph. 20(12), 1633–1642 (2014). Scholar
  35. 35.
    Makonin, S., McVeigh, D., Stuerzlinger, W., Tran, K., Popowich, F.: Mixed-initiative for big data: the intersection of human + visual analytics + prediction. In: 2016 49th Hawaii International Conference on System Sciences (HICSS), pp. 1427–1436. IEEE (2016)Google Scholar
  36. 36.
    McCrickard, D.S., Chewar, C.M., Somervell, J.P., Ndiwalana, A.: A model for notification systems evaluation-assessing user goals for multitasking activity. ACM Trans. Comput. Hum. Interact. (TOCHI) 10(4), 312–338 (2003)CrossRefGoogle Scholar
  37. 37.
    Meignan, D., Knust, S., Frayret, J.M., Pesant, G., Gaud, N.: A review and taxonomy of interactive optimization methods in operations research. ACM Trans. Interact. Intell. Syst. (TiiS) 5(3), 17 (2015)Google Scholar
  38. 38.
    Miller, R.B.: Response time in man-computer conversational transactions. In: Proceedings of the December 9–11, 1968, Fall Joint Computer Conference, Part I, pp. 267–277. ACM (1968)Google Scholar
  39. 39.
    Ng, A., Lepinski, J., Wigdor, D., Sanders, S., Dietz, P.: Designing for low-latency direct-touch input. In: Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology, pp. 453–464. ACM (2012)Google Scholar
  40. 40.
    Nielsen, J.: Usability Engineering. Elsevier, Amsterdam (1994)zbMATHGoogle Scholar
  41. 41.
    Nielsen, J.: Web-based application response time (2014).
  42. 42.
    North, C., et al.: Understanding multi-touch manipulation for surface computing. In: Gross, T., et al. (eds.) INTERACT 2009, Part II. LNCS, vol. 5727, pp. 236–249. Springer, Heidelberg (2009). Scholar
  43. 43.
    Peck, S.M., North, C., Bowman, D.: A multiscale interaction technique for large, high-resolution displays. In: 2009 IEEE Symposium on 3D User Interfaces, pp. 31–38, March 2009Google Scholar
  44. 44.
    Pirolli, P., Card, S.: The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In: Proceedings of International Conference on Intelligence Analysis, vol. 5, pp. 2–4 (2005)Google Scholar
  45. 45.
    Ragan, E.D., Endert, A., Sanyal, J., Chen, J.: Characterizing provenance in visualization and data analysis: an organizational framework of provenance types and purposes. IEEE Trans. Vis. Comput. Graph. 22(1), 31–40 (2016). Scholar
  46. 46.
    Ragan, E.D., Sowndararajan, A., Kopper, R., Bowman, D.A.: The effects of higher levels of immersion on procedure memorization performance and implications for educational virtual environments. Presence Teleop. Virt. Environ. 19(6), 527–543 (2010)CrossRefGoogle Scholar
  47. 47.
    Salzbrunn, T., Garth, C., Scheuermann, G., Meyer, J.: Pathline predicates and unsteady flow structures. Vis. Comput. 24(12), 1039–1051 (2008)CrossRefGoogle Scholar
  48. 48.
    Sauer, F., Zhang, Y., Wang, W., Ethier, S., Ma, K.L.: Visualization techniques for studying large-scale flow fields from fusion simulations. IEEE Comput. Sci. Eng. 18(2), 68–77 (2016)CrossRefGoogle Scholar
  49. 49.
    Shipman, F.M., Marshall, C.C.: Formality considered harmful: experiences, emerging themes, and directions on the use of formal representations in interactive systems. Comput. Support. Coop. Work (CSCW) 8(4), 333–352 (1999)CrossRefGoogle Scholar
  50. 50.
    Silva, J.A., Faria, E.R., Barros, R.C., Hruschka, E.R., de Carvalho, A.C., Gama, J.: Data stream clustering: a survey. ACM Comput. Surv. (CSUR) 46(1), 13 (2013)CrossRefGoogle Scholar
  51. 51.
    Simmhan, Y.L., Plale, B., Gannon, D., Marru, S.: Performance evaluation of the karma provenance framework for scientific workflows. In: Moreau, L., Foster, I. (eds.) IPAW 2006. LNCS, vol. 4145, pp. 222–236. Springer, Heidelberg (2006). Scholar
  52. 52.
    Stahnke, J., Dörk, M., Müller, B., Thom, A.: Probing projections: interaction techniques for interpreting arrangements and errors of dimensionality reductions. IEEE Trans. Vis. Comput. Graph. 22(1), 629–638 (2016)CrossRefGoogle Scholar
  53. 53.
    Streit, M., Schulz, H.J., Lex, A., Schmalstieg, D., Schumann, H.: Model-driven design for the visual analysis of heterogeneous data. IEEE Trans. Vis. Comput. Graph. 18(6), 998–1010 (2012). Scholar
  54. 54.
    Tatu, A., et al.: Subspace search and visualization to make sense of alternative clusterings in high-dimensional data. In: IEEE Conference on Visual Analytics Science and Technology (VAST), pp. 63–72. IEEE (2012)Google Scholar
  55. 55.
    Thieke, C., et al.: A new concept for interactive radiotherapy planning with multicriteria optimization: first clinical evaluation. Radiother. Oncol. 85(2), 292–298 (2007)CrossRefGoogle Scholar
  56. 56.
    Van Wijk, J.J., Nuij, W.A.A.: Smooth and efficient zooming and panning. In: Proceedings of the Ninth Annual IEEE Conference on Information Visualization, INFOVIS 2003, pp. 15–22. IEEE Computer Society (2003)Google Scholar
  57. 57.
    Wongsuphasawat, K., Moritz, D., Anand, A., Mackinlay, J., Howe, B., Heer, J.: Voyager: exploratory analysis via faceted browsing of visualization recommendations. IEEE Trans. Vis. Comput. Graph. 22(1), 649–658 (2016)CrossRefGoogle Scholar
  58. 58.
    Zaman, L., et al.: GEM-NI: a system for creating and managing alternatives in generative design. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1201–1210. ACM (2015)Google Scholar
  59. 59.
    Zimmer, B., Kerren, A.: Ongrax: a web-based system for the collaborative visual analysis of graphs. J. Graph Algorithm. Appl. 21(1), 5–27 (2017). Scholar
  60. 60.
    Cetin, G., Stuerzlinger, W., Dill, J.: Visual analytics on large displays: exploring user spatialization and how size and resolution affect task performance. In: IEEE Symposium on Big Data Visual Analytics (BDVA 2018), 10 p. (2018, to appear)Google Scholar
  61. 61.
    El Meseery, M., Wu, Y., Stuerzlinger, W.: Multiple workspaces in visual analytics In: IEEE Symposium on Big Data Visual Analytics (BDVA 2018), 12 p. (2018, to appear)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Wolfgang Stuerzlinger
    • 1
    Email author
  • Tim Dwyer
    • 2
  • Steven Drucker
    • 3
  • Carsten Görg
    • 4
  • Chris North
    • 5
  • Gerik Scheuermann
    • 6
  1. 1.School of Interactive Arts + Technology (SIAT)Simon Fraser UniversityBurnabyCanada
  2. 2.Monash UniversityMelbourneAustralia
  3. 3.Microsoft ResearchRedmondUSA
  4. 4.University of ColoradoDenverUSA
  5. 5.Virginia TechBlacksburgUSA
  6. 6.Leipzig UniversityLeipzigGermany

Personalised recommendations