Advertisement

Concepts and design space for a better understanding of multi-device user interfaces

  • Fabio PaternòEmail author
Long Paper
  • 72 Downloads

Abstract

This paper discusses the motivations behind and the characterising concepts of multi-device user interfaces by looking at the main design issues that have been addressed and the various solutions proposed. The discussion of relevant systems and frameworks highlights their main features, which are then used as the basis for comparative discussion. It compares different approaches and perspectives adopted in this area (e.g. responsive design, cross-device, distributed, migratory user interfaces). The features constitute a design space that can be used to facilitate analysis and comparison of tools and frameworks for multi-device user interfaces. Such aspects can be exploited by user interface designers and developers to analyse and compare various options when addressing existing and new applications. The analysis provided may inspire the design and development of new tools and frameworks as well.

Keywords

Multi-device environments Distributed and migratory user interfaces Cross-device user interfaces 

Notes

References

  1. 1.
    Arthur, R., Olsen, D.R.: Privacy-aware shared UI toolkit for nomadic environments. Softw. Pract. Exp. 42, 601–628 (2011)CrossRefGoogle Scholar
  2. 2.
    Barralon, N., Coutaz, J.: Coupling interaction resources in ambient spaces: there is more than meets the eye! In: Gulliksen, J., Harning, M.B., Palanque, P., van der Veer, G.C., Wesson, J. (eds.) Engineering Interactive Systems. Engineering Interactive Systems, vol. 4940. Springer, Berlin (2008)CrossRefGoogle Scholar
  3. 3.
    Barsotti, M., Paternò, F., Pulina, F.: A web framework for cross-device gestures between personal devices and public displays. In: The 16th International Conference on Mobile and Ubiquitous Multimedia (MUM2017), Suttgart, November 2017, pp. 69–78, ACM PressGoogle Scholar
  4. 4.
    Bellotti, V., Edwards, W.K.: Intelligibility and accountability: human considerations in context-aware systems. Hum. Comput. Interact. 16(2–4), 193–212 (2001)CrossRefGoogle Scholar
  5. 5.
    Bellucci, F., Ghiani, G., Paternò, F., Santoro, C.: Engineering JavaScript state persistence of web applications migrating across multiple devices. In: ACM EICS, pp. 105–110 (2011)Google Scholar
  6. 6.
    Brusilovsky, P.: Adaptive hypermedia. User Model. User-Adapt. Interact. 11(1–2), 87–110 (2001)CrossRefzbMATHGoogle Scholar
  7. 7.
    Buyukkokten, O., Kaljuvee, O., Garcia-Molina, H., Paepcke, A., Winograd, T.: Efficient web browsing on handheld devices using page and form summarization. ACM Trans. Inf. Syst. Secur. 20(1), 82–115 (2002)CrossRefGoogle Scholar
  8. 8.
    Cantera, J.M.: Model-Based UI XG Final Report https://www.w3.org/2005/Incubator/model-based-ui/XGR-mbui-20100504/ (2010)
  9. 9.
    Chang, T. H., Li, Y.: Deep shot: a framework for migrating tasks across devices using mobile phone cameras. In: Proceedings ACM CHI’11, pp. 2163–2172Google Scholar
  10. 10.
    Chen, X. A., Grossman, T., Wigdor, D. J., Fitzmaurice, G.W.: Duet: exploring joint interactions on a smart phone and a smart watch. In: CHI 2014, pp. 159–168Google Scholar
  11. 11.
    Chen, X., Li, Y.: Improv: an input framework for improvising cross-device interaction by demonstration. ACM Trans. Comput. Hum. Interact. 24(2), 15 (2017)CrossRefGoogle Scholar
  12. 12.
    Cheverst, K., Davies, N., Mitchell, K., Efstratiou, C.: Using context as a crystal ball: rewards and pitfalls. Pers. Ubiquitous Comput. 5(1), 8–11 (2001)CrossRefGoogle Scholar
  13. 13.
    Chi, P. Y. P., Li, Y.: Weave: scripting cross-device wearable interaction. In: ACM CHI 2015, Seoul, Korea, April 2015, pp. 3923–3932Google Scholar
  14. 14.
    Corporate Act-Net Consortium.:The active database management system manifesto: a rulebase of ADBMS features. In: ACM SIGMOD Record, vol. 25 (1996)Google Scholar
  15. 15.
    Coutaz, J., Lachenal, C., Dupuy-Chessa, S.: Ontology for multi-surface interaction. In: Proceedings of IFIP Conference on Human-Computer Interaction Interact’2003, IOS Press, Amsterdam, pp. 447–454 (2003)Google Scholar
  16. 16.
    Coutaz, J., Nigay, L., Salber, D., Blandford, A., May, J., Young, R.: Four easy pieces for assessing the usability of multimodal interaction: the CARE properties. In: Proceedings INTERACT, pp. 115–120 (1995)Google Scholar
  17. 17.
    Dearman, D., Pierce, J.: It’s on my other Computer!: computing with multiple devices. In: Proceedings of CHI’08, ACM Press, pp. 767–776 (2008)Google Scholar
  18. 18.
    Demeure, A., Sottet, J. S., Calvary, G., Coutaz, J., Ganneau, V., Vanderdonckt, J.: The 4C reference model for distributed user interfaces. In: Fourth International Conference on Autonomic and Autonomous Systems, ICAS 2008, pp. 61–69, (2008)Google Scholar
  19. 19.
    Dey, A.: Understanding and using context. Pers. Ubiquit. Comput. 5(1), 4–7 (2001)CrossRefGoogle Scholar
  20. 20.
    Di Geronimo, L., Bertarini, M., Badertscher, J., Husmann, M., Norrie, M. C.: Exploiting mid-air gestures to share data among devices. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI ‘17), vol. 35 (2017)Google Scholar
  21. 21.
    Dong, T., Churchill, E., Nichols, J.: Understanding the challenges of designing and developing multi-device experiences. In: Proceedings DIS 2016, pp. 62–72. ACM PressGoogle Scholar
  22. 22.
    Fischer, C., Gellersen, H., Gostner, R., Guinard, D., Kortuem, G., Kray, C., Rukzio, E., Strengin, S.: Supporting device discovery and spontaneous interaction with spatial references. Mobile Spat. Interact. Pers. Ubiquitous Comput. (2008).  https://doi.org/10.1007/s00779-008-0206-3 Google Scholar
  23. 23.
    Frosini, L., Paternò, F.: User interface distribution in multi-device and multi-user environments with dynamically migrating engines. ACM EICS 2014, 55–64 (2014)Google Scholar
  24. 24.
    Gajos, K.Z., Weld, D.S., Wobbrock, J.O.: Automatically generating personalized user interfaces with Supple. Artif. Intell. 174(12), 910–950 (2010)CrossRefGoogle Scholar
  25. 25.
    Ghiani, G., Isoni, L., Paternò, F.: Security in migratory interactive web applications. In: Proceedings of the 11th International Conference on Mobile and Ubiquitous Multimedia MUM 2012, vol. 15. ACM Press (2012)Google Scholar
  26. 26.
    Ghiani, G., Manca, M., Paternò, F., Porta, C.: Beyond responsive design: context-dependent multimodal augmentation of web applic. In: MobiWIS 2014, LNCS, vol. 8640, pp. 71–85. Springer (2014)Google Scholar
  27. 27.
    Ghiani, G., Paternò, F.: Supporting mobile users in selecting target devices. J. Univ. Comput. Sci. 16(15), 2019–2037 (2010)Google Scholar
  28. 28.
    Ghiani, G., Paternò, F., Santoro, C.: Push and pull of web user interfaces in multi-device environments. In: AVI 2012, pp. 10–17Google Scholar
  29. 29.
    Ghiani, G., Paternò, F., Santoro, C.: Interactive customization of ubiquitous Web applications. J. Vis. Lang. Comput. 24(1), 37–52 (2013)CrossRefGoogle Scholar
  30. 30.
    Ghiani, G., Polet, J., Antila, V.: Towards intelligent migration of user interfaces. In: MobiWIS 2013, pp. 203–217 (2013)Google Scholar
  31. 31.
    Ghiani, G., Manca, M., Paternò, F., Santoro, C.: Personalization of context-dependent applications through trigger-action rules. ACM Trans. Comput. Hum. Interact. 24(2), 14 (2017)CrossRefGoogle Scholar
  32. 32.
    Gjerlufsen, T., Klokmose, C. N., Eagan, J., Pillias, C., Beaudouin-Lafon, M.: Shared substance: developing flexible multi-surface applications. CHI 2011, pp. 3383–3392 (2011)Google Scholar
  33. 33.
    Google Research Report.: The New Multi-screen World: Understanding Cross-Platform Consumer Behavior. http://services.google.com/fh/files/misc/multiscreenworld_final.pdf (2012). Accessed 3 April 2019
  34. 34.
    Greenberg, S., Boring, S., Vermeulen, J., Dostal, J.: Dark patterns in proxemic interactions: a critical perspective. In Proceedings of DIS’14, pp. 523–532, ACMGoogle Scholar
  35. 35.
    Hamilton, P., Wigdor, D.J.: Conductor: enabling and understanding cross-device interaction. In Proceedings of CHI’14, pp. 2773–2782. ACM Press (2014)Google Scholar
  36. 36.
    Houben, S., Marquardt, N.: WatchConnect: a toolkit for prototyping smartwatch-centric cross-device applications. In: CHI 2015, pp. 1247–1256 (2015)Google Scholar
  37. 37.
    Houben, S., Tell, P., Bardram, J.E. Bardram: ActivitySpace: managing device ecologies in an activity-centric configuration space. In: ITS 2014, pp. 119–128 (2014)Google Scholar
  38. 38.
    Husmann, M., Nebeling, M., Pongelli, S., Norrie, M.C.: MultiMasher: providing architectural support and visual tools for multi-device mashups. WISE 2(2014), 199–214 (2013)Google Scholar
  39. 39.
    Husmann, M., Spiegel, M., Murolo, A., Norrie, M. C.: UI testing cross-device applications. In: Proceedings of the 2016 ACM on Interactive Surfaces and Spaces, ISS ‘16, pp. 179–188. New York. ACM (2016)Google Scholar
  40. 40.
    Johanson, B., Ponnekanti, S., Sengupta, C., Fox, A.: Multibrowsing: moving web content across multiple displays. In: Proceedings of Ubicomp 2001, Springe, LNCS, vol. 2201, pp. 346–353 (2001)Google Scholar
  41. 41.
    Johanson, B., Fox, A., Winograd, T.: The Interactive Workspaces Project: Experiences with Ubiquitous Computing Rooms. IEEE Pervasive Comput. 1(2), 67–74 (2002)CrossRefGoogle Scholar
  42. 42.
    Jokela, T., Ojala, J., Grassel, G., Piippo, P., Olsson, T.: A comparison of methods to move visual objects between personal mobile devices in different contexts of use. In: Proceedings of MobileHCI’15, pp. 172–181Google Scholar
  43. 43.
    Jokela, T., Ojala, J., Olsson, T.: A diary study on combining multiple information devices in everyday activities and tasks. In: CHI 2015, pp. 3903–3912 (2015)Google Scholar
  44. 44.
    Kulkarni, C., Klemmer, S.: Automatically adapting web pages to heterogeneous devices. In: CHI’11 Extended Abstracts on Human Factors in Computing Systems (CHI EA’11). ACM, New York, pp. 1573–1578 (2011)Google Scholar
  45. 45.
    Lieberman, H., Paternò, F., Klann, M., Wulf, V.: End-user development: an emerging paradigm. In: Lieberman, H., Paternò, F., Wulf, V. (eds.) End-user Development. Human-Computer Interaction Series, pp. 1–8. Springer, Berlin (2006)CrossRefGoogle Scholar
  46. 46.
    Lo, J., Wohlstadter, E., Mesbah, A.: Imagen: runtime migration of browser sessions for javascript web applications. In: Proceedings of the International World Wide Web Conference (WWW), pp. 815–825 (2013)Google Scholar
  47. 47.
    Manca, M. Paternò, F.: Customizable dynamic user interface distribution. In: Proceedings ACM EICS’16, Bruxells, pp. 27–37. ACM PressGoogle Scholar
  48. 48.
    Marcotte, E.: Responsive Web Design, A Book Apart. http://www.abookapart.com/products/responsive-web-design (2011). Accessed 3 April 2019
  49. 49.
    Marquardt, N., Diaz-Marino, R., Boring, S., Greenberg, S.: The proximity toolkit: prototyping proxemic interactions in ubiquitous computing ecologies. In: Proceedings of UIST 2011, pp. 315–326. ACM (2011)Google Scholar
  50. 50.
    Marquardt, N., Hinckley, K., Greenberg, S.: Cross-device interaction via micro-mobility and f-formations. In: Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology, pp. 13–22. ACM PressGoogle Scholar
  51. 51.
    Marquardt, N., Ballendat, T., Boring, S., Greenberg, S., Hinckley, K.: Gradual engagement: facilitating information exchange between digital devices as a function of proximity. In: Proceedings of the 2012 ACM International Conference on Interactive tabletops and surfaces, pp. 31–40. ACM PressGoogle Scholar
  52. 52.
    Melchior, J., Grolaux, D., Vanderdonckt, J., Van Roy, P.: A toolkit for peer-to-peer distributed user interfaces: concepts, implementation, and applications. In: Proceedings of EICS 2009, pp. 69–78. ACM, (2009)Google Scholar
  53. 53.
    Melchior, J., Vanderdonckt, J., Van Roy, P.: A model-based approach for distributed user interfaces. In: Proceedings of the 3rd ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS ‘11). ACM, pp. 11–20 (2009)Google Scholar
  54. 54.
    Meskens, J., Vermeulen, J., Luyten, K., Coninx, K.: Gummy for multi-platform user interface designs: shape me, multiply me, fix me, use me. In: Proceedings of the Working Conference on Advanced Visual Interfaces (AVI’08), pp. 233–240. ACMGoogle Scholar
  55. 55.
    Myers, B., Hudson, S.E., Pausch, R.: Past, present, and future of user interface software tools. ACM Trans. Comput. Hum. Interact. 7(1), 3–28 (2000)CrossRefGoogle Scholar
  56. 56.
    Nacenta, M.A., Aliakseyeu, D., Subramanian, S., Gutwin, C.: A comparison of techniques for multi-display reaching. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘05). ACM, New York, pp. 371–380 (2005). http://dx.doi.org/10.1145/1054972.1055024
  57. 57.
    Nebeling, M.: XDBrowser 2.0: semi-automatic generation of cross-device interfaces. In: Proceedings CHI 2017, pp. 4574–4584. ACM PressGoogle Scholar
  58. 58.
    Nebeling, M., Husmann, M., Zimmerli, C., Valente, G., Norrie, M. C.: XDSession: integrated development and testing of cross-device applications. In: EICS 2015, pp. 22–27 (2015)Google Scholar
  59. 59.
    Nebeling, M., Mintsi, T., Husmann, M., Norrie, M.C.: Interactive development of cross-device user interfaces, pp. 2793–2802. ACM CHI, Norrie (2014)Google Scholar
  60. 60.
    Nebeling, M., Speicher, M., Norrie, M. C.: CrowdAdapt: enabling crowdsourced web page adaptation for individual viewing conditions and preferences. In: EICS 2013, pp. 23–32 (2013)Google Scholar
  61. 61.
    Park, S., Gebhardt, C., Rädle, R., Feit, A.M., Vrzakova, H., Dayama, N.R., Yeo, H.S., Klokmose, C.N., Quigley, A., Oulasvirta, A., Hilliges, O.: AdaM: adapting multi-user interfaces for collaborative environments in real-time. In: Proceedings ACM CHI 2018, Monterey. ACM Press (2018)Google Scholar
  62. 62.
    Paternò,F., Santoro, C.: A logical framework for multi-device user interfaces. In: EICS 2012, pp. 45–50, ACM Press,Copenaghen (2012)Google Scholar
  63. 63.
    Paternò Santoro, C., Spano, L.D.: MARIA: a universal language for service-oriented applications in ubiquitous environment. ACM Trans. Comput. Hum. Interact. 16(4), 191–1930 (2009)Google Scholar
  64. 64.
    Paternò, F., Santoro, C., Spano, L.D.: Engineering the authoring of usable service front ends. J. Syst. Softw. 84(10), 1806–1822 (2011)CrossRefGoogle Scholar
  65. 65.
    S. Ponnekanti, B. Lee, A.Fox, and P.Hanrahan, Icrafter: A service framework for ubiquitous computing environments. Proc. ACM Ubiquitous Computing (UbiComp’01) (Jan 2001), pp.56–75Google Scholar
  66. 66.
    Ponnekanti, S. R., Johanson, B., Kiciman, E., Fox, A.: Portability, extensibility and robustness in iROS. In Proceedings of the 1st IEEE International Conference on Pervasive Computing and Communications (PERCOM ‘03)Google Scholar
  67. 67.
    Rädle, R., Jetter, H. C., Marquardt, N., Reiterer, H., Rogers, Y.: HuddleLamp: spatially-aware mobile displays for ad-hoc around-the-table collaboration. In Proceedings ITS’14, pp. 45–54. ACM (2014)Google Scholar
  68. 68.
    Rädle, R., Jetter, H. C., Schreiner, M., Lu, Z., Reiterer, H., Rogers, Y.: Spatially-aware or spatially-agnostic?: Elicitation and evaluation of user-defined cross-device interactions. In: CHI, pp. 3913–3922 (2015)Google Scholar
  69. 69.
    Rekimoto, J.: Pick-and-drop: a direct manipulation technique for multiple computer environments. In: ACM Symposium on User Interface Software and Technology, pp. 31–39 (1997)Google Scholar
  70. 70.
    Rekimoto J., Saitoh, M.: Augmented surfaces: a spatially continuous work space for hybrid computing environments. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘99). ACM, New York, pp. 378–385 (1999). Doi: http://dx.doi.org/10.1145/302979.303113
  71. 71.
    Sanctorum, A. Signer, B.: Towards end-user development of distributed user interfaces. In: UAIS, pp. 1–15. Springer (2017)Google Scholar
  72. 72.
    Santosa, S., Wigdor, D.: A field study of multi-device workflows in distributed workspaces. 2013. A field study of multi-device workflows in distributed workspaces. In: Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’13). ACM, New York, pp. 63–72Google Scholar
  73. 73.
    Schreiner, M., Rädle, R., Jetter, H. C., Reiterer, H.: Connichiwa: a framework for cross-device web applications. In: CHI Extended Abstracts 2015, pp. 2163–2168Google Scholar
  74. 74.
    Serrano, M., Ens, B., Yang, X. D., Irani, P: Developing a head-worn display interface to unify the interaction experience in distributed display environments. In: Proceedings of MobileHCI’15, pp. 16–171. ACM PressGoogle Scholar
  75. 75.
    Sohn, T., Li, F.C.Y., Battestini, A., Setlur, V., Mori, K. Horii, H.: Myngle: unifying and filtering web content for unplanned access between multiple personal devices. In: Proceedings UbiComp 2011, pp. 257–266. ACMGoogle Scholar
  76. 76.
    Tandler, P., Prante, T., Müller-Tomfelde, C., Streitz, N. and Steinmetz, R.: ConnecTables: dynamic coupling of displays for the flexible creation of shared workspaces. In: Proceedings of 14th ACM Symposium on UI Software and Technical UIST’01,, pp. 11–20. ACM Press, New York (2001)Google Scholar
  77. 77.
    Terrenghi, L., Quigley, A., Dix, A.: A taxonomy for and analysis of multi-person-display ecosystems. Pers Ubiquitous Comput 13(8), 583–598 (2009)CrossRefGoogle Scholar
  78. 78.
    Vandervelpen, C., Conix, K.: Towards model-based design support for distributed user interfaces. In: Proceedings of NordiCHI 2004, pp. 61–70. ACM (2004)Google Scholar
  79. 79.
    Wäljas, M., Segerståhl, K., Väänänen-Vainio-Mattila, K., Oinas-Kukkonen, H.: Cross-platform service user experience: a field study and an initial framework, In: Proceedings of MobileHCI 2010 Lisboa, Portugal, September 2010, pp. 219–228. ACM Press (2010)Google Scholar
  80. 80.
    Yang, J. Wigdor, D.: Panelrama: enabling easy specification of cross-device web applications. In: ACM CHI 2014, pp. 2783–2792, Toronto (2014)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.HIIS LaboratoryCNR-ISTIPisaItaly

Personalised recommendations