Keywords

1 Introduction

The merging of augmented reality (AR) and picture book material has been explored for close to two decades, starting with Billinghurst et al.’s Magic Book [1]. Augmented reality adds virtual elements to a real scene, typically viewed through a headset or a phone’s camera. In this way, AR interfaces enhance real-world experiences of books [2]. Inspiring examples of extending the experience of books through AR are the haunted book enhancing poetry by Scherrer [6], the AR colouring book [3], and the Vivid Encyclopedia, an AR book of insects [4]. A number of AR-enhanced picture books are available, few of which focus on providing extended engagement over time as a child grows in understanding. This demonstration showcases our picture book AR app that provides multiple levels for children at different learning stages.

Traditional book enhancements in the form of questions, answers, and content sequencing can prompt children to interact and engage with a book at a deeper level [7]. While ‘touch and feel’ books promote learning for very young readers, ‘Search and Find’ or ‘Question and Answer’ books are often considered useful to teach children how to solve problems, make connections and follow cues because they require critical inspection of a page [11]. Pop-up books are often considered to make the learning experience interactive, playful, and memorable. By promoting a hands-on approach to learning – both figuratively and literally – interactive books allow the depiction of a written concept in visual form. This concept can be translated into using AR enhancements for children books to encourage exploration and learning.

While the use of AR in formal learning environments (see e.g., [2, 3, 8]) and digital book use and design (see e.g., [5, 7, 9]) has been extensively explored our application focusses on playful learning during shared reading at home. Yuen et al. [10] observe that most likely AR-enhanced books will provide a stepping stone for the public to engage in the cross-over between digital and physical worlds. We agree with their observation that AR-enhanced books have the potential to provide many paths for learners. In our application, we provide these interaction and learning paths within the same page of a book, addressing young readers at different levels of comprehension through the layering of AR features.

2 Demonstration Application

Our AR-enhanced book prototype is based on the children’s book ‘Hannah’s Favourite Place’ by Fiona Mason, consisting of the printed book source and an AR companion app. The mobile app provides AR animations, audio, and additional textual elements to enhance the content of the printed book. Elements of the original book were removed from the printed page to allow these to be augmented within the AR app. The app delivers the animations based on recognition of visual marker location anchors which are extracted from the books original illustrations. No further changes were made to the printed book source to allow for the augmentation provided by the app.

Scenario.

We propose a scenario where we first meet Maia as a 4-year-old pre-reader sat with her dad John. In front of them is the book Hannah’s Favourite Place which is open to a middle spread. At first, Maia holds dads smart-phone in landscape mode with her two small hands holding the camera quite close to the book. She moves the phone around and explores the interactions that occur when individual characters in the book are viewed filling the camera’s screen. John then helps Maia lift the phone a little further from the page. The two share the task of holding the phone and here the camera views more than one character or page feature at a time. Finally, Maia hands John the phone and he holds the camera farther from the page so that they can both see the entire spread through the phone.

A few years later we again meet Maia who as she grows older and more confident with books and reading and as her fine motor skills develop we find that she is able to read and interact with Hannah’s Favourite Place on her own. Yet there are features of this book that have held her interest for longer than some other books in her collection. Interactive, animated, textual, and audio features of the book have engaged her at different learning phases in her life as if to say the book has grown with her. The features that gripped her during the first read with John are less amusing, yet the opportunities to practice spelling and pronunciation, as well as to read along with the story carry interest for Maia today.

Implementation.

Exploration of multiple levels of interaction within a single page is achieved in our app through interactive features that become visible on the phone at three different camera heights, as shown on the left side of Fig. 1.

Fig. 1.
figure 1

Interacting with the book to encounter layered AR animations

Full size image

At Level 1, the camera is held close to the page and playful interactions of individual image elements are provided. One of these is an animation and audio of a panda sleeping in the tree (see top of Fig. 1).

At Level 2, the camera is held slightly higher from the page and simple educational interactions with two or more objects at a time result. For example, in Fig. 1, middle we illustrate the identification of the two chimps; here they are pointed to by large playful arrows and the spelling of the animals’ name is animated.

Level 3 requires the child or parent to hold the camera to view almost the entire page; the app provides both playful and educational interactions. In the example shown in Fig. 1, bottom, the story is narrated and animated bees fly across the page. These bees are objects that do not occur in the illustrated storybook, nor are they a part of the written text. However, the inclusion of such non-text illustration is common in children’s books and enhances opportunities for discussion, emersion, and realistic building of a scene or environment.

Further details of design thinking for our AR-enhancement is discussed in [8]. An interaction of the app with the book spread (engaging with level 2) is shown in Fig. 2.

Fig. 2.
figure 2

User interacting with the enhanced book via AR app (Level 2 animation)

Full size image

3 Summary

This paper discussed our mobile phone-based augmented reality picture book prototype that provides AR animations, audio, and additional textual elements to enhance the content of the printed book. We believe that enhanced picture books, be they with augmentation, physical interaction, or cognitive interaction will increase the learning possibilities and extend the useful lifespan of that book along with the child. The significant feature of our work is the investigation of the affordances of the 3-dimensional space between the camera and the book surface. At the time of our investigation, we are not aware of other children’s books that presently exploit this multi-level enhancement in their implementation. Our future work will extend this prototype based on insights from parents about the role of books and technology in their homes.