Keywords

1 Introduction

Recently, more and more electronic picture books are coming into people’s view, such as mixed-media picture books [1], touch-feel picture books [2], game books, etc. However, these electronic picture books have two major disadvantages: on the one hand, their fancy interaction style is generally designed for entertainment instead of children’s understanding; on the other hand, only the interaction for electronic picture book on appearance is taken into account in their design, but the interaction for electronic picture book on emotion is always ignored. Recently, with “affect and emotion in HCI” [3] being more and more significant, adding emotional design in electronic picture book may solve the existing problems.

Therefore, from the perspective of emotional design, we propose a novel PCE (Perception & Comprehension & Expression) model from the perspective of emotional design. And based on this, we propose an electronic picture book for children (5–8 years old), named E-book. First, at the level of perception, it provides excellent multi-sensory interaction; second, at the level of cognition, it builds some immersive cognitive interaction; third, at the level of affection, it creates some emotional interaction based on emotion recognition and emotion feedback. The PCE model provide clear guidance not only in the stage of designing electronic book from the prospective of emotional design, but also in the stage of user study. In the user study, we design experiments to verify whether electronic picture books added emotional design can improve children’s reading effect. The user study shows that emotional design for children picture books can induce positive sense experience in readers [4], and affect cognitive outcomes, e.g. excellent learning comprehension performance, as well as affective results, e.g., pleasant emotion.

2 Related Works

2.1 Traditional Picture Books and Electronic Picture Books

Traditional children’s picture books include a lot of illustrations and a few words, particularly for children age from 3 to 6. Picture books are beneficial for children in many ways, e.g., stimulating their interest, improving their cognition. However, traditional picture books are limited in adult’s guidance which will restrict the emotional expression of children if mother-child communicative interaction is inappropriate.

With the impact of digital products increasing, people may find electronic picture book cheaper, more convenient to carry and more environment-friendly. It goes beyond the text and pictures in the paper versions, and may thus broaden children’s imagination. Combining sound, animation and game, the electronic book is able to activate children in many aspects [5,6,7]. Compared to traditional picture books, electronic books might be useful in children understanding stories independently.

2.2 Emotional Design in Multimedia Learning

In the past years, research on multimedia learning begun to consider the influence of emotional design. Emotional design can be defined as the use of different design elements to impact learners’ emotion to enhance learning ability [8] (Plass & Kaplan). Lots of the researches has investigated on the emotional feedback of the users on the interaction with the design outcomes.

The literature shows that adding positive emotion into multimedia learning facilitates student learning performance [4, 9, 10]. Park, Plass, and Brünken [11] used an eye-tracking device to examine the influence of emotion on learning with multimedia. They found that learners who were in a positive emotional state before learning had better learning outcomes. Erb [12] assumed that sound is a method to stimulate involvement and motivation in learning situations, thereby bringing the learner to invest more mental effort into learning, which finally leads to better learning performance. Shavelson and Towne [13] conducted experiments on investigating the effects of emotional design into multimedia materials on problem-solving skills, attention and motivation levels in primary mathematics. The results show that the effects of emotional design into multimedia materials perform better on problem solving skill, attention and motivation levels.

2.3 Emotional Design in Electronic Picture Books Reading

Only a few studies investigate the impact of added the emotional design in the electronic picture books on children’ emotion and their reading outcome. In some point of view, introducing emotional design in picture books have good effects on children’s comprehension: Tsai’s study [14] investigated the impact of children’s picture books on the emotional understanding and emotion regulation of 5–6 years old Taiwanese students. They found that children’s emotional understanding was improved by picture books related to emotion elements. But emotional design in picture books may have adverse effects. For example, too many inappropriate emotional elements may distract their attention in reading. Visual effects may distract children and guide them to think of the story as a game, and interfere with comprehending the story as a motivation [15]. Too many exaggerated animation elements will reduce children’s attention. “Children seemed to enjoy the animated effects, but this affective motivation did not lead to understanding a whole story” [16].

2.4 Problem Statement

Previous researches on the emotional design in multimedia learning have been criticized for the following reasons. First, most of the studies on emotional design are conducted with adults rather on children. Second, in most of the previous research, they use subjective indicators such as the positive affective scale of PANAS [17] to detect emotion. But psychological scales may be not suitable for children. Third, few research is focus on the impact of emotional design for children’s electronic picture books. Finally, the research on the different levels of affective elements in electronic picture book is easily ignored.

Therefore, the purpose of this study is: Whether the utilization of different levels of emotional elements in electronic picture books has a positive influence on children’ perception, understandability, and emotion.

3 Emotional Design Models for Electronic Picture Book

3.1 The Theory of Emotional Design Models

We propose a novel PCE (Perception & Comprehension & Expression) model from the perspective of emotional design. In the following, we elaborate details of the emotional interaction between children and electronic books on three levels: perception level, comprehension level, and expression levels.

Perception level appeals to a first sense reaction when children encounter an electronic picture book. Perception is the simplest psychological phenomenon to sense the world. “Beauty Means Nice Effect” [18], that says excellent appearance can promote emotional relationships between users and products. The visual reaction to product’s appearance is about initial reactions that child takes one look and says “I want it.” Visual elements in picture book can stimulate children’s interest, affect children’s emotion and foster children’s learning [10].

Visual, auditory, tactile and olfactory senses are the basic factors that we need to consider. In details, the story structure, picture layout, graphics setting, animation elements, sound elements, touch interaction, and other elements should be considered to give user multi-sense experience in electronic picture books. The goal of perception level is achieving the harmonious among visual, auditory, tactile and olfactory senses.

Comprehension level refers to children’s cognitive psychological. Understandability comes first and foremost. If readers can read the words but not understand what they are reading, they may fail to construct the cognition through picture books. Emotional design should focus on understanding the cognitive system of the children. Nevertheless, numerous of studies reveal that the challenge for the emotional design is how to design a product suitable for the capacity of children’s cognitive system, which otherwise end up in creating a cognitive overload and decreasing learning performance [19].

When designing a picture book, we aim at improving comprehension for children, depicting an easy-to-understand world, and establishing a link for knowledge transmission. Children’s reading comprehension is related to language development, picture understanding, listening comprehension, emotion condition [20], personal interest, complex social skills [21, 22], etc. For instance, beautiful pictures are more likely to draw students’ attention and to help student remember the concept and knowledge longer [23]. Fredrickson and Thomas Joiner [24] describe that positive emotion broadens people’s thought and encourages them to discover novel lines of thought or action.

Expression level is about children’s emotion when they are reading—e.g., how books affect children feel, or what kind of feeling children want to express. Expression level extends design purpose from “aesthetics, function” to “emotion, experience.” For a child who reads a picture book, emotion is the meaning and the memory of the book. Emotions reflect personal experiences, associations, and memories. Kaptelinin [25], a well-known interaction design expert, once described the emotional experience with definite feelings including “satisfying, pleasant, interesting, effective, enlightening, beautiful, attractive, creative, achievable and emotional satisfaction.”

There are many ways to build the emotional relationship between picture books and readers. On the one hand, electronic picture books can express emotions through content, such as anthropomorphic cartoon characters or immersive interactive scene. On the other hand, electronic picture books can perceive emotions through artificial intelligence technology. For instance, electronic picture books can recognize children’s feeling through affective computing and then give children immediately feedback according to the change of emotional state.

3.2 Emotional Design Models in Practice

In the previous chapter, we propose a novel PCE (Perception & Comprehension & Expression) model from the perspective of emotional design. And based on this model, we establish an electronic e picture book for children (5–8 years old), named E-book, which depicts some typical scenarios that may happen in the kindergarten [26]. In the design of the main plot, commendatory, background music and user interface play important roles. And on this basis, some other designs of interactive functions in E-book provides entries for children to explore branches of the story-line by themselves. In the following, we elaborate details of three typical levels (see Fig. 1).

Fig. 1.
figure 1

Architecture of PCE model and corresponding implementation of E-book

Full size image

First, at the perception level, it provides multi-sensory interactions including clicking to animation (vision), clicking to sounds (hearing), finger painting (touch), etc. Perception level focus on the basic function of the electronic picture book, which includes the interface design (see Fig. 2), voice design, and tactile interaction, etc. It is an interactive book with anthropomorphic animations and sound effect that can be accessed by clicking on parts of pictures.

Fig. 2.
figure 2

Partially user interface of E-book

Full size image

Second, at the comprehension level, it builds some interactive scenes including “game of linking strokes” and “magic brush”. In “game of linking strokes”, children learn physics knowledge to help characters in the book to go through obstacles by linking strokes. For example, when children draw a bridge that meets the physical structural mechanics to connect both sides of the river, the hero of the story can successfully cross the river. And “magic brush” in which children need to imagine what they need in the story and draw it out through magic brush. In this way, children may have a better understanding in the process of “perception (what they need) – comprehension (how to draw) – expression (draw it out)”.

Third, at the expression level, it provides some emotional interaction modes including “reading poetry” and “coloring a picture book”. In “reading poetry”, we apply a multi-path deep neural network [27] to recognize four different emotions including happiness, sadness, impatience and peace, so that while children read poetry with different emotions (e.g., happy), a photo in a corresponding emotional color (e.g., orange) will be generated to encourage children to express emotions freely.

4 User Study

As mentioned in the previous section, the design of E-book is on account of the PCE (Perception & Comprehension & Expression) model. In this chapter, I design three experiments. The first experiment validates the sense organs level indicators, the second experiment validates the comprehension level indicators, and the third experiment validates the emotional level indicators. The following is the process of experiments including experiment participants, motivation, environment, materials, measures, and results.

4.1 Participants

To select children participants, we assessed their current level of story understanding using procedures recommended by Sulzby [28]. We recruited 20 children aged 4–6 (10 males and 10 females) whose reading ability is the level of 3–5 in Sulzby’s study. They can understand and retell the story independently. We randomly divided 20 children into test and control group by their gender and age, 10 children for each group.

4.2 Motivation

We design three experiments to verify whether children’s electronic picture book added emotional design can improve children’s reading effect from the following three perspectives [29]:

Hypothesis 1: Added the emotional design elements, whether the electronic picture book gives the children an excellent sensory experience during children learning?

Hypothesis 2: Added the emotional design elements, whether the electronic picture book enhances the understandability during children learning?

Hypothesis 3: Added the emotional design elements, whether the electronic picture book brings more positive emotional experience during children learning?

4.3 Environment

The experiment was conducted in a kindergarten. In some experiment conditions, children may be out of control due to tension or other reasons. To ensure the reliability of the whole experiment, we took the following measures: Firstly, the experimenter explained the process of the experiment to children so that children were familiar with the operation. Secondly, the experimenter gave children guidance if they were in trouble. Thirdly, the experiment site is a separate room without any interruptions. Fourthly, the children participant watched a 3 min relaxing video to neutralize their affective state before the experiment.

4.4 Materials

We have two groups, named the test group and the control group. The test group is taken from E-book which uses the emotional design method as a design principle. We sort and simplify the content of E-book to produce the material of test group. The test group’s material is an emotional electronic picture book with related anthropomorphic animations, colorful illustration, voice over and sound effect that can be accessed by clicking on any parts of pictures. As a comparison, the material of control group is electronic picture book which just consists of text and illustrations. The storyline in the control group is the same with the test group, which depicts Ganggang’s first day in the kindergarten. We divide the storyline into ten story chapters to ensure that each corresponding chapter of the test group and the control group is the same. The difference between the test group and the control group is that the control group has emotional design elements while the test group not.

4.5 Measures

The measures of the experiment are designed from three perspectives based on the PCE model. At the perception level, to verify children participant’s sensory experience in reading, we use subjective questionnaire to assess their evaluation. At the comprehension level, to ascertain the effect of children participant’s comprehension, we take the method of “story retelling [30].” At the expression level, to recognize children’s emotions condition, we design an emotion recognition system with facial expression signals. The following are the details of measures.

In the sensory evaluation experiment, child reads E-book independently for 15 min. When the children finish reading, we invited them to give a multi-sense evaluation on E-book. The design of the interview questionnaire is from the sensory experience evaluation including visual evaluation (e.g., interface), hearing evaluation (e.g., music), and touch evaluation (e.g., painting). A 5-point method [31] is used to evaluate the children’s satisfaction towards the questionnaire items. The current study uses the measurement for online store perception adopted from Kim, Fiore, and Lee, which includes 5 items to understand the customer’s perception of the website better.

The story retelling experiment is a comparative experiment. In the test group and control group, every child participant reads one chapter of ten corresponding materials. Each group of children engaged in all two processes: (a) Each group of children read one chapter of the materials independently in 5 min. (b) When the children finished the reading task, the experimenter has a 10-min-conversation with them. The experimenter guides each child to retell the story, e.g., “You read the story now.” “I am eager to hear you tell the story.” “Can you share me the story?” At the same time, the experimenter records the conversation. After that, the experimenter will translate the child’s voice into text.

In the emotion recognition experiment, the purpose of the experiment is to verify which group gets more positive emotion samples. We used an emotion detection device to recognize children’s emotion. Through analyzing facial expression signals, the device attempts to recognize the type of emotions (including positive emotion, negative emotion, and neutral emotion) and calculate the probability of emotion. In the test and control group, every child reads one chapter of ten corresponding materials. Each group reads one chapter of ten corresponding materials independently. And the emotion detection device records the children’ facial expression at the same time. The emotion will be used for data analysis in the next step. To facilitate statistical calculation, we map the probability of positive and negative results of emotion to the range of [−1–1]. The closer the emotion score is to 1, the higher the probability of positive emotion is. The closer the emotion score is to −1, the higher the probability of negative emotion is. The closer the emotion score is to 0, the higher the probability of neutral emotion is. We can find that emotion score of the test and control group over time in Fig. 3.

Fig. 3.
figure 3

Emotion score of the test group and the control group over time

Full size image

4.6 Result

The outcome of the sensory evaluation experiment shows that the score of children’s sensory evaluation in reading. See Fig. 4, in which we can find that the mean score of children’s sensory evaluation is 4.26. It may reflect that E-book can bring children excellent sensory experience especially on visual experience and touch experience.

Fig. 4.
figure 4

The mean score of children’s sensory evaluation in reading.

Full size image

To verify the effect of emotional design on children’s understanding, we take the method of “story retelling”. Table 1 is the examples of coding system proposed by Sulzby [27]. With the help of verbatim transcriptions, we calculate how many words appear in the original text, and how many phrases (at least three words in a row) were derived from the original text. The examples of transcriptions and coding for selected chapter are in Table 1. And Table 2 is the story retelling accuracy of each chapter. Story retelling accuracy of the test group and the control group is in Table 3, which proves that test group achieves a significant improvement. Through the comparison experiment, we find that adding emotional design in an electronic picture book can promote children’s ability of understanding.

Table 1. Examples of transcriptions and coding for selected chapter
Full size table
Table 2. Story retelling accuracy of each chapter
Full size table
Table 3. Story retelling accuracy of the test group and the control group
Full size table

In this chapter, three delicate experiments are introduced. We calculate the ratio of positive emotion samples to total emotion samples (see Table 4) and average score and standard deviation of emotion samples in the test and control group over time (see Table 5). In the test group, the ratio of positive emotion samples to total emotion samples is 0.587, the mean score of emotion samples is 0.5, and the standard deviation value is 0.47. Obviously, the proportion of positive emotion samples is greater than the neutral and negative emotion samples. In the control group, the ratio of positive emotion samples to total emotion samples is 0.307, the average score of emotion samples is 0.1, and the standard deviation value is 0.37. The experimental data shows that the proportion of neutral emotion samples are greater than the positive and negative emotion samples.

Table 4. The ratio of positive emotion samples to total emotion samples
Full size table
Table 5. The statistics of emotion score over time
Full size table

Generally, the number of positive emotion samples and the mean score of emotion samples in the test group are all higher than the control group. Results reveal that positive emotions generally increase as the amount of emotional design features increase in the electronic picture book.

5 Conclusion

In this work, we proposed a novel PCE model from the perspective of emotional design. It provides clear guidance not only in the stage of designing electronic book from the prospective of emotional design, but also in the stage of user study. And based on this, we propose E-book which makes significant impact on children’s reading effects. In the user study, we find that emotional design for children’s electronic picture books can strongly influence children’ sensory experience, enhance comprehension and create a positive emotional resonance for the children.