A Formal Description of Low Level Interaction and its Application to Multimodal Interactive Systems

  • Johnny Accot
  • Stéphane Chatty
  • Philippe Palanque
Part of the Eurographics book series (EUROGRAPH)


The lack of formal models for describing low-level interaction restricts programmers to interactors provided by toolkits. It impedes the construction of highly interactive systems and the design of new interaction styles, such as multimodal interaction. This article reports on our experience with formalising low-level graphical interaction. We propose primitives for event specification and handling that can be used along with Petri nets to model such interactions. We then show how multimodal interactions can be built from monomodal ones by combining those models. This is exemplified by an experimental two-handed graphical editor that has been built using the proposed model.


Interactive System Interaction Style Interaction Object Graphical Object Multimodal Interaction 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    M. Beaudouin-Lafon, Y. Berteaud, S. Chatty. Creating direct manipulation interfaces with XTV. EX’90. European conference on the X Window System. London 1990.Google Scholar
  2. 2.
    A. Beck, C. Janssen, A. Weisbecker, J. Ziegler. Integrating object-oriented analysis and graphical user interface design. In Coutaz J. & Taylor R. (Eds) LNCS Springer Verlag 1995.Google Scholar
  3. 3.
    E. Bier, M. Stone, K. Fishkin, W. Buxton, T. Baudel. A taxonomy of seethrough tools. In proceedings of the CHI’94 conference. ACM Press, 1994, p. 358–364.Google Scholar
  4. 4.
    P. Brun & M. Beaudouin-Lafon, A taxonomy and evaluation of formalisms for the specification of interactive systems. HCI’95, 1995, p. 197–212.Google Scholar
  5. 5.
    W. Buxton. A three state model of graphical input. In proceedings of the Interact’90 conference, p.449–456, North Holland 1990.Google Scholar
  6. 6.
    S. Chatty. Defining the behaviour of animated interfaces. Engineering for Human Computer Interfaces conference 1992. p. 95–109. North-Holland.Google Scholar
  7. 7.
    S. Chatty. Extending a graphical toolkit for two-handed interaction. In ACM UIST’94, pages 195–204. ACM Press, 1994.Google Scholar
  8. 8.
    S. Chatty, P. Lecoanet. Pen computing for air traffic control. CHI’96 conference proceedings ACM Press, 1996, p. 87–94.Google Scholar
  9. 9.
    S. Chatty, P. Girrad, S. Sire. Vers un support multimedia aux collecticiels synchrones. Techniques et Sciences Informatiques, vol. 15, n°9, 1996, 28p.Google Scholar
  10. 10.
    A. Dix. Formal Methods for Interactive Systems. Academic Press, 1991.Google Scholar
  11. 11.
    Proceedings of the First Eurographics workshop on Design, Specification and Verification of Interactive Systems, F. Paternó Ed. Springer Verlag 1995.Google Scholar
  12. 12.
    Proceedings of the Second Eurographics workshop on Design, Specification and Verification of Interactive Systems, P. Palanque & R. Bastide Eds. Springer Verlag 1995.Google Scholar
  13. 13.
    D. Duke, M. Harrison. Abstract Interaction Objects. Computer Graphics Forum 12(3), p. 25–36 1993. Eurographics 93.CrossRefGoogle Scholar
  14. 14.
    O. Esteban, S. Chatty, P. Palanque. Whizz’Ed: a visual environment for building highly interactive interfaces. Proceedings of the Interact’95 conference, p. 121–126.Google Scholar
  15. 15.
    F. Feldbrudge. Petri net tool overview 1992. Advances in Petri nets 1993. In G. Rozenberg (Ed.), Lecture Notes in Computer Science n° 674, p. 169–209. Springer Verlag 1993.Google Scholar
  16. 16.
    M. Harrison, H. Thimbleby. Formal Methods in Human Computer Interaction. Cambridge University Press, 1990.Google Scholar
  17. 17.
    K. Jensen. Coloured Petri nets and the invariant method. Theoretical Computer Science 14, 1981, North Holland, p. 317–336.Google Scholar
  18. 18.
    K. Jensen. Coloured Petri nets. Vol. 1 (Basic concepts) and Vol. 2 (Analysis methods and practical use) Springer Verlag, 1995.Google Scholar
  19. 19.
    B. Myers. Comprehensive support for graphical, highly interactive user interfaces. IEEE Computer, p. 71–85, Nov. 1990.Google Scholar
  20. 20.
    L. Nigay & J. Coutaz, A generic platform for addressing the multimodal challenge. Proceedings of CHI’95, 1995, p. 98–105.Google Scholar
  21. 21.
    P. Palanque & R. Bastide. Petri net based design of user-driven interfaces using the interactive cooperative object formalism. In [11], p. 383–401.Google Scholar
  22. 22.
    P. Palanque & R. Bastide. Formal specification and verification of CSCW using the Interactive Cooperative Object formalism. In HCI’95 conference, People and Computers X, p. 213–231. Cambridge University Press 1995.Google Scholar
  23. 23.
    P. Palanque & R. Bastide. Time modelling in Petri nets for the design of Interactive Systems. GIST workshop on Time in Interactive Systems. Glasgow, July 1995, and also SIGCHI bulletin vol 28 n°2, p. 43–46.Google Scholar
  24. 24.
    F. Paternó & A. Leonardi. A semantic-based approach for the design and implementation of interaction objects. Computer Graphics Forum 13(3) p. 195–204, 1994.CrossRefGoogle Scholar
  25. 25.
    J. T. Stasko. TANGO: A framework and System for Algorithm Animation, PhD thesis, Brown University, 1989.Google Scholar
  26. 26.
    W. Van Biljon. Extending Petri nets for specifying man-machine dialogues. Int. J. Man-Machine Studies (1988) 28, p. 437–455.CrossRefMATHGoogle Scholar
  27. 27.
    A. Wasserman. Extending state transition diagrams for the specification of human-computer interaction. IEEE Transactions on Software Engineering, 11(8), August 1985.Google Scholar

Copyright information

© Springer-Verlag/Wien 1996

Authors and Affiliations

  • Johnny Accot
    • 1
    • 2
  • Stéphane Chatty
    • 1
  • Philippe Palanque
    • 1
    • 3
  1. 1.CENAToulouse cedexFrance
  2. 2.D.G.P.University of TorontoTorontoCanada
  3. 3.LIS - IHMUniversité Toulouse IToulouse cedexFrance

Personalised recommendations