Software Support for Coherent Prototyping of 3D Gesture Interactions

  • Dominik RupprechtEmail author
  • Rainer Blum
  • Birgit Bomsdorf
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11262)


When prototyping applications that include touchless 3D gesture interaction three design matters must be taken into consideration: the gestures the user must execute, the visual representation and the dialog flow. Ideally, these aspects should be considered in parallel, to achieve a coherent design process, and avoid ineffective extra effort stemming from coordination between them. A flexible changeover of perspectives among the separate matters is needed. This paper proposes a software environment that enables the desired coherent rapid prototyping of applications with 3D gesture interactions. Its core consists of two types of mapping and a so-called co-simulation functionality. The environment facilitates combining existing software tools from industry and literature to cover the three design matters, i.e. specification and simulation of UI prototypes, gestures, and dialog models. It assists developers at design time in the specification of gestures and in binding them to UI prototypes as well as to statecharts used for defining dialog models. Relevant coherency information is used to offer the option to evaluate gestures at runtime in the context of UI prototype and dialog model. The co-simulator manages the synchronized simulation of all relevant artefacts once a gesture event occurs. Therefore, it enables quickly building prototypes that go beyond the capabilities of the individual tools. This paper describes the usage of the proposed environment in form of a case study with several software tools, each covering one of the three design aspects. It also shows its general applicability, meaning that it can be used with other tools too.


Gesture development Interaction design User interface design Model-based development Development support 



This research was financially supported by the German Federal Ministry of Education and Research within the framework of the program Forschung an Fachhochschulen - IngenieurNachwuchs (project no. 03FH007IX5).


  1. 1.
    Barboni, E., Ladry, J.-F., Navarre, D., Palanque, P., Winckler, M.: Beyond modelling: an integrated environment supporting co-execution of tasks and systems models. In: Proceedings of the 2nd ACM SIGCHI Symposium on Engineering Interactive Computing Systems, pp. 165–174. ACM, New York (2010)Google Scholar
  2. 2.
    Bomsdorf, B., Blum, R.: Early prototyping of 3D-gesture interaction within the presentation-gesture-dialog design space. In: Kurosu, M. (ed.) HCI 2014. LNCS, vol. 8511, pp. 12–23. Springer, Cham (2014). Scholar
  3. 3.
    Bomsdorf, B., Blum, R., Hesse, S., Heinz, P.: WeBewIn: Rapid Prototyping bewegungsbasierter Interaktionen. In: Boll, S., Maaß, S., Malaka, R. (eds.) Mensch & Computer 2013: Interaktive Vielfalt, pp. 251–260. Oldenbourg Verlag, München (2013)Google Scholar
  4. 4.
    Feuerstack, S., Anjo, M.D.S., Pizzolato, E.B.: Model-based design and generation of a gesture-based user interface navigation control. In: Gomes, A.S. (ed.) Proceedings of the 10th Brazilian Symposium on Human Factors in Computing Systems and the 5th Latin American Conference on Human-Computer Interaction, pp. 227–231. Brazilian Computer Society, Porto Alegre (2011)Google Scholar
  5. 5.
    Kistner, G., Nuernberger, C.: Developing user interfaces using SCXML statecharts. In: Schnelle-Walka, D., Radomski, S., Lager, T., Barnett, J., Dahl, D., Mühlhäuser, M. (eds.) Proceedings of the 1st EICS Workshop on Engineering Interactive Computer Systems with SCXML, pp. 5–11 (2014)Google Scholar
  6. 6.
    Martinie, C., Navarre, D., Palanque, P., Fayollas, C.: A generic tool-supported framework for coupling task models and interactive applications. In: Ziegler, J. (ed.) Proceedings of 7th ACM SIGCHI Symposium on Engineering Interactive Computing Systems, pp. 244–253 (2015)Google Scholar
  7. 7.
    Meixner, G., Paternó, F., Vanderdonckt, J.: Past, Present, and Future of Model-Based User Interface Development, i-com 10, 2–11 (2011)Google Scholar
  8. 8.
    Pintér, G., Micskei, Z.I., Majzik, I.: Supporting design and development of safety critical applications by model based tools. In: Annales Universitatis Scientiarum Budapestinensis de Rolando Eötvös nominatae Sectio Computatorica, pp. 61–78. ELTE (2009)Google Scholar
  9. 9.
    Spano, L.D., Cisternino, A., Paternò, F., Fenu, G.: GestIT: a declarative and compositional framework for multiplatform gesture definition. In: Forbrig, P., Dewan, P., Harrison, M., Luyten, K. (eds.) Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems - EICS 13, p. 187. ACM Press, New York (2013)Google Scholar
  10. 10.
    van Buskirk, R., Moroney, B.W.: Extending prototyping. IBM Syst. J. 42, 613–623 (2003)CrossRefGoogle Scholar
  11. 11.
    van den Bergh, J., Sahni, D., Haesen, M., Luyten, K., Coninx, K.: GRIP. In: Paternò, F., Luyten, K., Maurer, F. (eds.) Proceedings of the 3rd ACM SIGCHI Symposium on Engineering Interactive Computing Systems - EICS 2011, p. 143. ACM Press, New York (2011)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2019

Authors and Affiliations

  • Dominik Rupprecht
    • 1
    Email author
  • Rainer Blum
    • 1
  • Birgit Bomsdorf
    • 1
  1. 1.Fulda University of Applied SciencesFuldaGermany

Personalised recommendations