Abstract
Physical paper, in its various forms (e.g. books, leaflets, catalogues), is extensively used in everyday activities, despite any advancements in digital technology and ICT. Inspired by the popularity of this medium, several research efforts have envisioned and pursued a new era of interactive paper, however several challenges still remain to be addressed. On the other hand, recent advancements towards Ambient Intelligence (AmI) and Smart environments, bear the promise of seamless integration of the physical and digital worlds in an intuitive and user-friendly manner. This paper presents InPrinted, a systematic and generic framework supporting physical paper augmentation and user interaction in Ambient intelligence environments. InPrinted enables natural multimodal user interaction with any kind of printed matter in smart environments, providing context aware and anticipation mechanisms, as well as tools and interaction techniques that support the development of applications incorporating printed matter augmentation. InPrinted has been put to practice in the development of various systems, including the Interactive Maps system, facilitating interaction with printed maps as well as their augmentation with digital information. The results of an in-situ observation experiment of the Interactive Maps system are reported, highlighting that interacting with augmented paper is quite easy and natural, while the overall User Experience is positive.
Similar content being viewed by others
References
Anoto (2018) Forms based on unique pattern. https://support.anoto.com/article/283-forms-based-on-unique-pattern. Accessed 8 May 2018
ARToolKit Introduction to ARToolKit. http://www.hitl.washington.edu/artoolkit/documentation/userintro.htm. Accessed 8 May 2018
Benko H, Jota R, Wilson A (2012) MirageTable: freehand interaction on a projected augmented reality tabletop. Proceedings of the SIGCHI conference on human factors in computing systems (CHI '12), ACM, New York, p 199–208
Billinghurst M, Kato H, Poupyrev I (2001) The MagicBook: moving seamlessly between reality and Virtuality. IEEE Comput Graph 21(3):6–8
Billinghurst M, Clark A, Lee G (2015) A survey of augmented reality. Foundations and Trends® in Human–Computer Interaction 8(2–3):73–272
Chen H, Finin T, Joshi A (2003) An ontology for context-aware pervasive computing environments. Knowl Eng Rev 18(3):197–207
Crawford Technologies (2015) Paper vs. digital: two sides survey. http://www.crawfordtech.com/news-and-events/the-importance-of-paper. Accessed 8 May 2018
Dachselt R, Al-Saiegh S (2011) Interacting with printed books using digital pens and smart mobile projection. Proceedings of the Workshop on Mobile and Personal Projection (MP2) @ CHI 2011, Vol. 2011
Fields B, Amaldi P, Wong W, Gill S (2007) In use, in situ: extending field research methods. Int J Hum-Comput Interact 22(1–2):1–6
Figueiredo AC, Pinto AL, Branco P, Zagalo N, Coquet E (2013) Bridging book: a not-so-electronic children's picturebook. Proceedings of the 12th International Conference on Interaction Design and Children (IDC '13), ACM, New York, p 569–572
Forsberg AS, LaViola JJ Jr, Zeleznik RC (1998) ErgoDesk: A Framework for Two- and Three-Dimensional Interaction at the ActiveDesk. Proceedings of the Second International Immersive Projection Technology Workshop, p 11–12
Georgalis I (2013) Architectures, methods and tools for creating ambient intelligence environments. Dissertation, Computer Science Department, University of Crete. https://www.didaktorika.gr/eadd/handle/10442/29423. Accessed 8 May 2018
Golemati M, Katifori A, Vassilakis C, Lepouras G, Halatsis C (2007). Creating an ontology for the user profile: Method and applications. Proceedings of the first RCIS conference, No 2007, p 407–412
Gu T, Wang XH, Pung HK, Zhang DQ (2004) An ontology-based context model in intelligent environments. Proceedings of communication networks and distributed systems modeling and simulation conference, Vol 2004, p 270–275
Holman D, Vertegaal R, Altosaar M, Troje N, Johns D (2005) Paper windows: interaction techniques for digital paper. Proceedings of the SIGCHI conference on Human factors in computing systems (CHI '05), ACM, New York, p 591–599
Kjeldsen R, Pinhanez C, Pingali G, Hartman J, Levas T, Podlaseck M (2002) Interacting with steerable projected displays. Proceedings of the fifth IEEE International Conference on Automatic Face and Gesture Recognition, IEEE, p 402–407
Kobayashi M, Koike H (1998) Enhanced desk: integrating paper documents and digital documents. Asia Pacific Computer Human Interaction (APCHI’98) IEEE CS, p 57–62
Korozi M, Leonidis A, Margetis G, Koutlemanis P, Zabulis X, Antona M, Stephanidis C (2012) Ambient educational mini-games. Proceedings of the International Working Conference on Advanced Visual Interfaces (AVI '12), ACM, New York, p 802–803
Lam A H, Chow K C, Yau E H, Lyu M R (2006) ART: augmented reality table for interactive trading card game. Proceedings of the 2006 ACM international conference on Virtual reality continuum and its applications (VRCIA '06), ACM, New York, p 357–360
Liao C, Guimbretière F, Hinckley K, Hollan J (2008) Papiercraft: a gesture-based command system for interactive paper. ACM T Comput-Hum Int 14(4):18 27 pages
Liao C, Liu Q, Liew B, Wilcox L (2010) Pacer: fine-grained interactive paper via camera-touch hybrid gestures on a cell phone. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10), ACM, New York, p 2441–2450
Luff P, Heath C, Norrie M, Signer B, Herdman P (2004) Only touching the surface: creating affinities between digital content and paper. Proceedings of the 2004 ACM conference on Computer supported cooperative work (CSCW '04), ACM, New York, p 523–532
Mackay WE, Fayard AL (1999) Designing interactive paper: lessons from three augmented reality projects. Proceedings of International Workshop on Augmented Reality: placing artificial objects in real scenes (IWAR) 98:81–90
Margetis G, Koutlemanis P, Zabulis X, Antona M, Stephanidis C (2011) A smart environment for augmented learning through physical books. Proceedings of the 2011 Conference on Multimedia and Expo (ICME 2011), IEEE, p 1–6
Margetis G, Ntoa S, Bouhli M, Stephanidis C (2011) Study-buddy: improving the learning process through technology-augmented studying environments. Posters’ Extended Abstracts of the HCI International Conference 2011, Springer, Berlin, Heidelberg, p 504–508
Margetis G, Ntelidakis A, Zabulis X, Ntoa S, Koutlemanis P, Stephanidis C (2013). Augmenting physical books towards education enhancement. Proceedings of the 1st IEEE Workshop on User-Centered Computer Vision (UCCV '13), IEEE, p 43–49
Margetis G, Zabulis X, Koutlemanis P, Antona M, Stephanidis C (2013) Augmented interaction with physical books in an ambient intelligence learning environment. Multimed Tools Appl 67(2):473–495
Margetis G, Antona M, Stephanidis C (2015) A framework for supporting natural interaction with printed matter in ambient intelligence environments. Proceeding of the Fifth International Conference on Ambient Computing, Applications, Services and Technologies (AMBIENT'15), IARIA, p 72–78
Margetis G, Zabulis X, Ntoa S, Koutlemanis P, Papadaki E, Antona M, Stephanidis C (2015) Enhancing education through natural interaction with physical paper. Univ Access Inf Soc 14(3):427–447
Margetis G, Ntoa S, Antona M, Stephanidis C (2017) Interacting with augmented paper maps: a user experience study. Proceedings of the 12th Biannual Conference on Italian SIGCHI Chapter (CHItaly '17), ACM, New York, article no. 18
Marshall CC (2005) Reading and interactivity in the digital library: creating an experience that transcends paper. Proceedings of CLIR/Kanazawa institute of technology roundtable 5(4):1–20
Matsuo Y, Ishizuka M (2004) Keyword extraction from a single document using word co-occurrence statistical information. Int J Artif Intell T 13(01):157–169
Matthews T, Rattenbury T, Carter S (2007) Defining, designing, and evaluating peripheral displays: an analysis using activity theory. Human–computer. Interaction 22(1–2):221–261
Microsoft (2006) Introduction to the windows workflow foundation rules engine. https://msdn.microsoft.com/en-us/library/aa480193.aspx. Accessed 8 May 2018
Microsoft (2018) Ink analysis overview. http://msdn.microsoft.com/en-us/library/ms704040(v=VS.85).aspx. Accessed 8 May 2018
Milgram P, Kishino F (1994) A taxonomy of mixed reality visual displays. IEICE Trans Inf Syst 77(12):1321–1329
Miller G, Fellbaum C (1998) Wordnet: An electronic lexical database. Language 76(3):706–708
Nielsen J (1994) Heuristic evaluation. In: Nielsen J, Mack RL (eds) Usability inspection methods. John Wiley & Sons, New York, pp 25–61
Nielsen C M, Overgaard M, Pedersen M B, Stage J, Stenild S (2006) It's worth the hassle!: the added value of evaluating the usability of mobile systems in the field. Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles (NordiCHI '06), ACM, New York, p 272–280
Ojala T, Kostakos V, Kukka H (2011) It’s a jungle out there: fantasy and reality of evaluating public displays in the wild. Large Displays in Urban Life 4:1–4
Papadaki E, Zabulis X, Ntoa S, Margetis G, Koutlemanis P, Karamaounas P, Stephanidis C (2013) The book of Ellie: an interactive book for teaching the alphabet to children. Proceedings of the 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW 2013), IEEE, p 1–6
Pinhanez C (2001) Using a steerable projector and a camera to transform surfaces into interactive displays. CHI'01 Extended Abstracts on Human Factors in Computing Systems, ACM, p 369–370
Preuveneers D, Van den Bergh J, Wagelaar D, Georges A, Rigole P, Clerckx T, De Bosschere K (2004) Towards an extensible context ontology for ambient intelligence. Proceedings of the European Symposium on Ambient Intelligence, Springer, Berlin, Heidelberg, p 148–159
Robertson C, Robinson J (1999) Live paper: video augmentation to simulate interactive paper. Proceedings of the seventh ACM international conference on Multimedia (Part 2) (MULTIMEDIA '99), ACM, New York, p 167–170
Rogers Y, Connelly K, Tedesco L, Hazlewood W, Kurtz A, Hall R E, Hursey J, Toscos T (2007) Why it’s worth the hassle: The value of in-situ studies when designing ubicomp. Proceedings of the International Conference on Ubiquitous Computing (UbiComp 2007), Springer, Berlin, Heidelberg, p 336–353
Rosson MB, Carroll JM (2009) Scenario based design. In: Sears A, Jacko J (eds) Human-computer interaction: development process, Boca Raton, pp 145–162
Schmalstieg D, Fuhrmann V, Hesina G, Szalav Z, Encarna LM, Gervautz M, Purgathofer W (2002) The studierstube augmented reality project. Presence: Teleoper. Virtual Environ 11:33–54
Sellen AJ, Harper RH (2003) The myth of the paperless office. MIT press, Cambridge
Signer B, Norrie M C (2007) PaperPoint: a paper-based presentation and interactive paper prototyping tool. Proceedings of the 1st international conference on Tangible and embedded interaction (TEI '07), ACM, New York, p 57–64
Signer B, Norrie MC (2010) Interactive paper: past, present and future. Proceedings of the 1st International Workshop on Paper Computing (PaperComp '2010), Copenhagen Denmark, September 2010
Smith R (2007) An overview of the Tesseract OCR engine. Proceedings of the Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), IEEE, Vol 2, p 629–633
Song H, Grossman T, Fitzmaurice G, Guimbretiere F, Khan A, Attar R, Kurtenbach G. (2009) PenLight: combining a mobile projector and a digital pen for dynamic visual overlay. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09), ACM, New York, p 143–152
The Bibliographic Ontology (2009) Bibliographic ontology specification. http://bibliontology.com/. Accessed 8 May 2018
Valderrama Bahamóndez EDC, Kubitza T, Henze N, Schmidt A (2013) Analysis of children's handwriting on touchscreen phones. Proceedings of the 15th international conference on Human-computer interaction with mobile devices and services (MobileHCI '13), ACM, New York, p 171–174
Waldner M, Hauber J, Zauner J, Haller M, Billinghurst M (2006) Tangible tiles: design and evaluation of a tangible user interface in a collaborative tabletop setup. Proceedings of the 18th Australia conference on Computer-Human Interaction: Design: Activities, Artefacts and Environments (OZCHI '06), ACM, New York, p 151–158
Wellner P (1993) Interacting with paper on the DigitalDesk. Commun ACM 36(7):87–96
Woods E, Billinghurst M, Looser J, Aldridge G, Brown D, Garrie B, Nelles C (2004) Augmenting the science centre and museum experience. Proceedings of the 2nd international conference on Computer graphics and interactive techniques in Australasia and South East Asia (GRAPHITE '04), ACM, New York, p 230–236
Wu C S A, Robinson S J, Mazalek A (2008) Turning a page on the digital annotation of physical books. Proceedings of the 2nd international conference on Tangible and embedded interaction (TEI '08), ACM, New York, p 109–116
Acknowledgments
This work is supported by the FORTH-ICS internal RTD Programme ‘Ambient Intelligence Environments’.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
ESM 1
(DOC 121 kb)
Rights and permissions
About this article
Cite this article
Margetis, G., Ntoa, S., Antona, M. et al. Augmenting natural interaction with physical paper in ambient intelligence environments. Multimed Tools Appl 78, 13387–13433 (2019). https://doi.org/10.1007/s11042-018-7088-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-018-7088-9