Multimedia Tools and Applications

, Volume 78, Issue 10, pp 13387–13433 | Cite as

Augmenting natural interaction with physical paper in ambient intelligence environments

  • George MargetisEmail author
  • Stavroula Ntoa
  • Margherita Antona
  • Constantine Stephanidis


Physical paper, in its various forms (e.g. books, leaflets, catalogues), is extensively used in everyday activities, despite any advancements in digital technology and ICT. Inspired by the popularity of this medium, several research efforts have envisioned and pursued a new era of interactive paper, however several challenges still remain to be addressed. On the other hand, recent advancements towards Ambient Intelligence (AmI) and Smart environments, bear the promise of seamless integration of the physical and digital worlds in an intuitive and user-friendly manner. This paper presents InPrinted, a systematic and generic framework supporting physical paper augmentation and user interaction in Ambient intelligence environments. InPrinted enables natural multimodal user interaction with any kind of printed matter in smart environments, providing context aware and anticipation mechanisms, as well as tools and interaction techniques that support the development of applications incorporating printed matter augmentation. InPrinted has been put to practice in the development of various systems, including the Interactive Maps system, facilitating interaction with printed maps as well as their augmentation with digital information. The results of an in-situ observation experiment of the Interactive Maps system are reported, highlighting that interacting with augmented paper is quite easy and natural, while the overall User Experience is positive.


Paper augmentation Natural interaction Ambient intelligence framework In-situ evaluation 



This work is supported by the FORTH-ICS internal RTD Programme ‘Ambient Intelligence Environments’.

Supplementary material

11042_2018_7088_MOESM1_ESM.doc (122 kb)
ESM 1 (DOC 121 kb)


  1. 1.
    Anoto (2018) Forms based on unique pattern. Accessed 8 May 2018
  2. 2.
    ARToolKit Introduction to ARToolKit. Accessed 8 May 2018
  3. 3.
    Benko H, Jota R, Wilson A (2012) MirageTable: freehand interaction on a projected augmented reality tabletop. Proceedings of the SIGCHI conference on human factors in computing systems (CHI '12), ACM, New York, p 199–208Google Scholar
  4. 4.
    Billinghurst M, Kato H, Poupyrev I (2001) The MagicBook: moving seamlessly between reality and Virtuality. IEEE Comput Graph 21(3):6–8Google Scholar
  5. 5.
    Billinghurst M, Clark A, Lee G (2015) A survey of augmented reality. Foundations and Trends® in Human–Computer Interaction 8(2–3):73–272CrossRefGoogle Scholar
  6. 6.
    Chen H, Finin T, Joshi A (2003) An ontology for context-aware pervasive computing environments. Knowl Eng Rev 18(3):197–207CrossRefGoogle Scholar
  7. 7.
    Crawford Technologies (2015) Paper vs. digital: two sides survey. Accessed 8 May 2018
  8. 8.
    Dachselt R, Al-Saiegh S (2011) Interacting with printed books using digital pens and smart mobile projection. Proceedings of the Workshop on Mobile and Personal Projection (MP2) @ CHI 2011, Vol. 2011Google Scholar
  9. 9.
    Fields B, Amaldi P, Wong W, Gill S (2007) In use, in situ: extending field research methods. Int J Hum-Comput Interact 22(1–2):1–6Google Scholar
  10. 10.
    Figueiredo AC, Pinto AL, Branco P, Zagalo N, Coquet E (2013) Bridging book: a not-so-electronic children's picturebook. Proceedings of the 12th International Conference on Interaction Design and Children (IDC '13), ACM, New York, p 569–572Google Scholar
  11. 11.
    Forsberg AS, LaViola JJ Jr, Zeleznik RC (1998) ErgoDesk: A Framework for Two- and Three-Dimensional Interaction at the ActiveDesk. Proceedings of the Second International Immersive Projection Technology Workshop, p 11–12Google Scholar
  12. 12.
    Georgalis I (2013) Architectures, methods and tools for creating ambient intelligence environments. Dissertation, Computer Science Department, University of Crete. Accessed 8 May 2018
  13. 13.
    Golemati M, Katifori A, Vassilakis C, Lepouras G, Halatsis C (2007). Creating an ontology for the user profile: Method and applications. Proceedings of the first RCIS conference, No 2007, p 407–412Google Scholar
  14. 14.
    Gu T, Wang XH, Pung HK, Zhang DQ (2004) An ontology-based context model in intelligent environments. Proceedings of communication networks and distributed systems modeling and simulation conference, Vol 2004, p 270–275Google Scholar
  15. 15.
    Holman D, Vertegaal R, Altosaar M, Troje N, Johns D (2005) Paper windows: interaction techniques for digital paper. Proceedings of the SIGCHI conference on Human factors in computing systems (CHI '05), ACM, New York, p 591–599Google Scholar
  16. 16.
    Kjeldsen R, Pinhanez C, Pingali G, Hartman J, Levas T, Podlaseck M (2002) Interacting with steerable projected displays. Proceedings of the fifth IEEE International Conference on Automatic Face and Gesture Recognition, IEEE, p 402–407Google Scholar
  17. 17.
    Kobayashi M, Koike H (1998) Enhanced desk: integrating paper documents and digital documents. Asia Pacific Computer Human Interaction (APCHI’98) IEEE CS, p 57–62Google Scholar
  18. 18.
    Korozi M, Leonidis A, Margetis G, Koutlemanis P, Zabulis X, Antona M, Stephanidis C (2012) Ambient educational mini-games. Proceedings of the International Working Conference on Advanced Visual Interfaces (AVI '12), ACM, New York, p 802–803Google Scholar
  19. 19.
    Lam A H, Chow K C, Yau E H, Lyu M R (2006) ART: augmented reality table for interactive trading card game. Proceedings of the 2006 ACM international conference on Virtual reality continuum and its applications (VRCIA '06), ACM, New York, p 357–360Google Scholar
  20. 20.
    Liao C, Guimbretière F, Hinckley K, Hollan J (2008) Papiercraft: a gesture-based command system for interactive paper. ACM T Comput-Hum Int 14(4):18 27 pagesGoogle Scholar
  21. 21.
    Liao C, Liu Q, Liew B, Wilcox L (2010) Pacer: fine-grained interactive paper via camera-touch hybrid gestures on a cell phone. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10), ACM, New York, p 2441–2450Google Scholar
  22. 22.
    Luff P, Heath C, Norrie M, Signer B, Herdman P (2004) Only touching the surface: creating affinities between digital content and paper. Proceedings of the 2004 ACM conference on Computer supported cooperative work (CSCW '04), ACM, New York, p 523–532Google Scholar
  23. 23.
    Mackay WE, Fayard AL (1999) Designing interactive paper: lessons from three augmented reality projects. Proceedings of International Workshop on Augmented Reality: placing artificial objects in real scenes (IWAR) 98:81–90Google Scholar
  24. 24.
    Margetis G, Koutlemanis P, Zabulis X, Antona M, Stephanidis C (2011) A smart environment for augmented learning through physical books. Proceedings of the 2011 Conference on Multimedia and Expo (ICME 2011), IEEE, p 1–6Google Scholar
  25. 25.
    Margetis G, Ntoa S, Bouhli M, Stephanidis C (2011) Study-buddy: improving the learning process through technology-augmented studying environments. Posters’ Extended Abstracts of the HCI International Conference 2011, Springer, Berlin, Heidelberg, p 504–508Google Scholar
  26. 26.
    Margetis G, Ntelidakis A, Zabulis X, Ntoa S, Koutlemanis P, Stephanidis C (2013). Augmenting physical books towards education enhancement. Proceedings of the 1st IEEE Workshop on User-Centered Computer Vision (UCCV '13), IEEE, p 43–49Google Scholar
  27. 27.
    Margetis G, Zabulis X, Koutlemanis P, Antona M, Stephanidis C (2013) Augmented interaction with physical books in an ambient intelligence learning environment. Multimed Tools Appl 67(2):473–495CrossRefGoogle Scholar
  28. 28.
    Margetis G, Antona M, Stephanidis C (2015) A framework for supporting natural interaction with printed matter in ambient intelligence environments. Proceeding of the Fifth International Conference on Ambient Computing, Applications, Services and Technologies (AMBIENT'15), IARIA, p 72–78Google Scholar
  29. 29.
    Margetis G, Zabulis X, Ntoa S, Koutlemanis P, Papadaki E, Antona M, Stephanidis C (2015) Enhancing education through natural interaction with physical paper. Univ Access Inf Soc 14(3):427–447CrossRefGoogle Scholar
  30. 30.
    Margetis G, Ntoa S, Antona M, Stephanidis C (2017) Interacting with augmented paper maps: a user experience study. Proceedings of the 12th Biannual Conference on Italian SIGCHI Chapter (CHItaly '17), ACM, New York, article no. 18Google Scholar
  31. 31.
    Marshall CC (2005) Reading and interactivity in the digital library: creating an experience that transcends paper. Proceedings of CLIR/Kanazawa institute of technology roundtable 5(4):1–20MathSciNetGoogle Scholar
  32. 32.
    Matsuo Y, Ishizuka M (2004) Keyword extraction from a single document using word co-occurrence statistical information. Int J Artif Intell T 13(01):157–169CrossRefGoogle Scholar
  33. 33.
    Matthews T, Rattenbury T, Carter S (2007) Defining, designing, and evaluating peripheral displays: an analysis using activity theory. Human–computer. Interaction 22(1–2):221–261Google Scholar
  34. 34.
    Microsoft (2006) Introduction to the windows workflow foundation rules engine. Accessed 8 May 2018
  35. 35.
    Microsoft (2018) Ink analysis overview. Accessed 8 May 2018
  36. 36.
    Milgram P, Kishino F (1994) A taxonomy of mixed reality visual displays. IEICE Trans Inf Syst 77(12):1321–1329Google Scholar
  37. 37.
    Miller G, Fellbaum C (1998) Wordnet: An electronic lexical database. Language 76(3):706–708zbMATHGoogle Scholar
  38. 38.
    Nielsen J (1994) Heuristic evaluation. In: Nielsen J, Mack RL (eds) Usability inspection methods. John Wiley & Sons, New York, pp 25–61Google Scholar
  39. 39.
    Nielsen C M, Overgaard M, Pedersen M B, Stage J, Stenild S (2006) It's worth the hassle!: the added value of evaluating the usability of mobile systems in the field. Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles (NordiCHI '06), ACM, New York, p 272–280Google Scholar
  40. 40.
    Ojala T, Kostakos V, Kukka H (2011) It’s a jungle out there: fantasy and reality of evaluating public displays in the wild. Large Displays in Urban Life 4:1–4Google Scholar
  41. 41.
    Papadaki E, Zabulis X, Ntoa S, Margetis G, Koutlemanis P, Karamaounas P, Stephanidis C (2013) The book of Ellie: an interactive book for teaching the alphabet to children. Proceedings of the 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW 2013), IEEE, p 1–6Google Scholar
  42. 42.
    Pinhanez C (2001) Using a steerable projector and a camera to transform surfaces into interactive displays. CHI'01 Extended Abstracts on Human Factors in Computing Systems, ACM, p 369–370Google Scholar
  43. 43.
    Preuveneers D, Van den Bergh J, Wagelaar D, Georges A, Rigole P, Clerckx T, De Bosschere K (2004) Towards an extensible context ontology for ambient intelligence. Proceedings of the European Symposium on Ambient Intelligence, Springer, Berlin, Heidelberg, p 148–159Google Scholar
  44. 44.
    Robertson C, Robinson J (1999) Live paper: video augmentation to simulate interactive paper. Proceedings of the seventh ACM international conference on Multimedia (Part 2) (MULTIMEDIA '99), ACM, New York, p 167–170Google Scholar
  45. 45.
    Rogers Y, Connelly K, Tedesco L, Hazlewood W, Kurtz A, Hall R E, Hursey J, Toscos T (2007) Why it’s worth the hassle: The value of in-situ studies when designing ubicomp. Proceedings of the International Conference on Ubiquitous Computing (UbiComp 2007), Springer, Berlin, Heidelberg, p 336–353Google Scholar
  46. 46.
    Rosson MB, Carroll JM (2009) Scenario based design. In: Sears A, Jacko J (eds) Human-computer interaction: development process, Boca Raton, pp 145–162Google Scholar
  47. 47.
    Schmalstieg D, Fuhrmann V, Hesina G, Szalav Z, Encarna LM, Gervautz M, Purgathofer W (2002) The studierstube augmented reality project. Presence: Teleoper. Virtual Environ 11:33–54Google Scholar
  48. 48.
    Sellen AJ, Harper RH (2003) The myth of the paperless office. MIT press, CambridgeGoogle Scholar
  49. 49.
    Signer B, Norrie M C (2007) PaperPoint: a paper-based presentation and interactive paper prototyping tool. Proceedings of the 1st international conference on Tangible and embedded interaction (TEI '07), ACM, New York, p 57–64Google Scholar
  50. 50.
    Signer B, Norrie MC (2010) Interactive paper: past, present and future. Proceedings of the 1st International Workshop on Paper Computing (PaperComp '2010), Copenhagen Denmark, September 2010Google Scholar
  51. 51.
    Smith R (2007) An overview of the Tesseract OCR engine. Proceedings of the Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), IEEE, Vol 2, p 629–633Google Scholar
  52. 52.
    Song H, Grossman T, Fitzmaurice G, Guimbretiere F, Khan A, Attar R, Kurtenbach G. (2009) PenLight: combining a mobile projector and a digital pen for dynamic visual overlay. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09), ACM, New York, p 143–152Google Scholar
  53. 53.
    The Bibliographic Ontology (2009) Bibliographic ontology specification. Accessed 8 May 2018
  54. 54.
    Valderrama Bahamóndez EDC, Kubitza T, Henze N, Schmidt A (2013) Analysis of children's handwriting on touchscreen phones. Proceedings of the 15th international conference on Human-computer interaction with mobile devices and services (MobileHCI '13), ACM, New York, p 171–174Google Scholar
  55. 55.
    Waldner M, Hauber J, Zauner J, Haller M, Billinghurst M (2006) Tangible tiles: design and evaluation of a tangible user interface in a collaborative tabletop setup. Proceedings of the 18th Australia conference on Computer-Human Interaction: Design: Activities, Artefacts and Environments (OZCHI '06), ACM, New York, p 151–158Google Scholar
  56. 56.
    Wellner P (1993) Interacting with paper on the DigitalDesk. Commun ACM 36(7):87–96CrossRefGoogle Scholar
  57. 57.
    Woods E, Billinghurst M, Looser J, Aldridge G, Brown D, Garrie B, Nelles C (2004) Augmenting the science centre and museum experience. Proceedings of the 2nd international conference on Computer graphics and interactive techniques in Australasia and South East Asia (GRAPHITE '04), ACM, New York, p 230–236Google Scholar
  58. 58.
    Wu C S A, Robinson S J, Mazalek A (2008) Turning a page on the digital annotation of physical books. Proceedings of the 2nd international conference on Tangible and embedded interaction (TEI '08), ACM, New York, p 109–116Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Foundation for Research and Technology – Hellas (FORTH)Institute of Computer ScienceCreteGreece
  2. 2.Department of Computer ScienceUniversity of CreteCreteGreece

Personalised recommendations