Keywords

1 Introduction

In communication, humans use body motions and actions, such as nodding, body and hands gestures as well as spoken language [1]. Such motions and actions are referred to as nonverbal communication. Body motions play various roles and are classified as Emblems, Regulators, Illustrators, Affect Displays, and Adaptors. Hand raising is a typical body motion and indicates one’s intention—such as wanting to say something, calling people, and greeting people. Especially, hand raising is used as a communication channel between teachers and students in classrooms, and it reflects active participation [2]. To actively participate in class, students must raise their hands [3]. Similar situations are observed outside classrooms as well because settings with group communication—such as meetings, group work, etc.—have many opportunities for raising hands. However, Japanese people tend not to their raise hands, even when they want to say something, because they feel ashamed.

Analyzing hand-raising communication and interaction is not easy because they involve so many factors. Fujiu performed one of a few systematic studies on this topic [4]. He analyzed the behavior mechanism of hand raising and showed that self-efficiency, outcome expectancy, and outcome value are the predicting factors. Sawabe et al. extended the study of hand-raising to students’ cognitive factors, which the surrounding environment influences [5]. These studies are based on case studies in real classrooms. On the other hand, few people outside our research conduct engineering research about how the surrounding environment, such as embodied interaction affects these factors in hand raising. The authors have used a constructive approach. Kawabe et al. measured various instances of hand raising and classified them by height, angle, and speed [3]. It became clear that people’s confidence affected the movements of their hands raising. Aoyagi et al. introduced the classified hand-raising motions into a hand-raising robot and clarified hand raising’s effectiveness in active participation in classrooms [6]. However, these studies focused on single-person interactions.

To analyze hand raising in groups, Aoyagi evaluated participants’ impressions of hand-raising actions by generating a Computer Graphics (CG) classroom with multiple CG characters [7]. In that case, a participant formed impressions and did not raise his/her hand in front of a large screen. In this study, we developed a system for analyzing hand-raising communication and interaction by using Virtual Reality (VR) headsets to synthesize human-CG character interaction.

2 Related Studies

2.1 Constructive Studies on Group Communication and Interaction

Robots and CG characters have been used in studies with a constructive approach because they can physically or visually express human embodiment. For example, Watanabe developed a speech-driven group-entrainment system called SAKURA by using five CG characters and focusing on nodding and its effect on entrainment [8] (Fig. 1). Fukushima et al. developed robots as “Flatterers” laugh-enhancer systems [9].

Fig. 1.
figure 1

CG and Robot versions of Group-entrainment system SAKURA [9].

Full size image

Takahashi et al. developed-hand clapping machines to transmit clapping remotely from the audience to the performer.

Such an approach is effective for analyzing interactions between talkers and listeners in group communication and interaction. Ishii and Watanabe synthesized CG listeners around a talker’s video image and clarified their effectiveness as listeners [10]. Fukuda et al. developed a virtual classroom with multiple CG characters who perform various daily actions in classroom for teacher training [11].

In this study, we used a VR headset, which can express CG realistically. VR headsets are already used as tools for analyzing group interactions. For example, Tanaka developed an immersive presentation-experience system in which audience characters nod in response to lecturers’ utterances [12].

2.2 Previous Studies on Group Hand-Raising Interactions

In previous studies, Kawabe et al. measured hand-raising motions by using a motion-capture system and extracted five typical motions with a cluster analysis. They converted the motions into CG characters from motion-captured data, as shown in Fig. 2 [3]. Here, Motions A and B were fast and went to high points. Motion B went the highest and fastest. Motions D and E were slow and went to low points. Motion C was in the middle position. The sensory evaluation of these motions revealed that the impressions of hand raising changed based on the motions’ height and speed, and Motion B, which went the highest and fastest, was most positively rated [6].

Fig. 2.
figure 2

Typical hand-raising motions of Japanese students [3].

Full size image

Based on this, Aoyagi et al. developed a system for analyzing the impressions of group hand-raising. The CG classroom and characters that raised their hands are shown in Fig. 3. A projector projected this CG image on a large screen. Here, the authors introduced Motions B and E from Fig. 2 into the CG characters. All five CG characters raised their hands. The authors compared four scenes in to consider cases B or E and all five characters. The scene in which everyone raised their hands to a high point was rated as active and exciting, and the scene in which a small number raised their hands and only did so to a low point was given a low rating. By constructing interactions using CG, we could control the experimental conditions.

Fig. 3.
figure 3

Typical hand-raising motions of Japanese students.

Full size image

However, this system could not subjectively evaluate the hand-raising communications that occurred in the same place. In this study, we developed a system that uses a VR headset that allows us to arrange the CG characters and users at the same place with the same group. By changing the behavior characteristics and the number of people that raise their hands, we analyzed the communication effect of hand-raising motions.

3 Developing a System for Analyzing Hand-Raising Communication

To develop a VR headset–based system, we configured the system as shown in Fig. 4. We used a PC (CPU: AMD, Ryzen 5 1600; video card: NVIDIA, GeForce GTX 1070), VR headset (Oculus VR, Oculus Rift), and two tracking sensors attached to the VR headset and controllers for both hands. For the analysis, we used a video camera (SONY, HDR-PJ790V) and a display (EIZO, FlexScan EV2450). While adapting the CGs to VR, we encountered some problems in terms of the size and reality of the objects, so we remodeled the virtual classroom to mimic the actual classroom’s size and illumination.

Fig. 4.
figure 4

System configuration.

Full size image

4 Analysis of Hand-Raising Communication

4.1 Method

In the experiment, we showed the scene with a classroom and 20 CG characters in a VR headset. We explained to the participants that they were in a classroom with a presenter asking for opinions. We requested that they form impressions for each scene in five items described later. Four situations are constructed for evaluating the characteristics and the number of the hands raised and height of hand-raising as shown in Fig. 5: (a) a scene in which all CG characters raise their hands to a high point (high, all), (b) a scene in which all CG characters raise their hands to a low point (low, all), (c) a scene in which one-third of the CG characters raise their hands to a high point (high, one-third), and (d) a scene in which one-third of the CG characters raise their hands to a low point (low, one-third).

Fig. 5.
figure 5

Situations for evaluation of hand-raising motions in classroom situation.

Full size image

In our previous studies in front of a screen, we randomly chose five CG characters, one-third of which would raise their hands. However, the number of CG characters in the VR field of view changed as the direction of user’s face changed. To keep at least one-third of the CG characters in view at all times, we arranged the characters as shown in Fig. 6. Here, because the angle of the field of view was about 90°, the participant could see about nine CG characters at once. In the field of view, we divided the CG characters into a new group every 30° (Groups A to C), and raised the hand of one character in each group. To the sides of the field of view, we divided the CG characters into a new group every 45° (Groups D and E) and raised two characters’ hands in each group. Seven CG characters raised their hands in total.

Fig. 6.
figure 6

Groups of CG characters

Full size image

We instructed the participants to take the perspective of a member of the class and answer a questionnaire. They evaluated five items on a seven-point scale. The items were (1) “The scene gave me a positive impression”, (2) “The classroom was animated”, (3) “Everyone looked confident”, (4) “Raising my hand felt easy”, and (5) “I felt that I had to raise my hand”. The experiment’s environment is shown in Fig. 7. We adapted the desk in VR from the desk in reality. The participants (12 males and 12 females) comprised 24 people aged between 21 and 24 years old.

Fig. 7.
figure 7

Experimental environment.

Full size image

4.2 Results

The results of the questionnaire are shown in Fig. 8. We analyzed the results by using a Wilcoxon signed-rank test and Bonferroni’s test.

Fig. 8.
figure 8

Results of Wilcoxon signed-rank test of impression evaluation of hand-raising motions.

Full size image

The results for Items 1 and 2 had significant differences, except for those in the comparison between Situations B (low, all) and C (high, one-third). The number of CG character who raised their hands and the height to which they raised their hands affected the participants’ impressions. The results of Item 3 had significant differences, except for those in the comparisons between Situations B (low, all) and C (high, one-third) and between Situations B (low, all) and D (low, one-third). In the scene in which many people raised their hands to a high point, everyone in the classroom looked confidence. The results of Items 4 and 5, which address the ease and necessity of raising one’s hand, showed no significant differences in the comparison between the heights of the raised hands. However, the comparison between the numbers of CG characters who raised their hands showed significant differences. Only the number of CG characters who raised their hands affected the impressions regarding the ease and necessity of raising one’s hand.

5 Discussion

This study clearly showed that the number of CG characters who raise their hands and the height of the raised hands gave the participants positive impressions in the VR space. When we compared the results of the on-screen and VR experiments, Items 1 and 2 had the similar results. On the other hand, in Item 3 in the screen experiment, Situation B (low, all) was rated lower than Situation C (high, one-third), and in Item 3 in the case of VR, Situation B (low, all) and Situation C (high, one-third) were rated the same. We determined that the effect of the number of people who raise their hands is larger than the effect of the hand-raising height in VR. In Items 4 and 5, Situation B (low, all) was rated higher than they were in previous studies. From this, we concluded that VR increases immersion and provides participants with an impression of the whole scene. In the case of the screen experiment, the participants saw five of the 15 CG characters who raised their hands. In this study with VR, the participants could see the same number of CG characters who raised their hands, regardless of which direction they turned. However, only three CG characters raised their hands in the participants’ field of view. In other words, the proportion of raised hands was the same in both experiments, but the numbers of CG characters who raised their hands in the participants’ field of view differed. We need to further consider how participants looked around and used specific VR functions.

In recent years, the number of VR communication software such as VRChat [13] and rumii [14] has increased. Remote communication, which is currently done through the flat display, will change the communication in the VR space. For example, by using VR in distance education—as the school corporation “N High School” at Kadokawa Dwango Gakuen did—students will receive education while feeling mutual existence [15]. One of advantage of VR communication is that we can freely design the appearance of the communication settings. By utilizing the findings of this study, we would like create the impression of excitement in VR space and make an index to create a place where we can speak positively.

In this experiment, we asked the participants to raise their hands to express opinions [16]. In a future experiment on the subject, we should ask the participants to raise their hands to answer questions. In the current experiment, the ease of the hand raising varied depending on confidence. For example, raising hands is difficult for people if they are not confident. Therefore, it is necessary to analyze the effect of hand-raising communication in various scenes to evaluate different levels of problems, change the situations, and conduct experiments that can change participants’ motivations. In this study, one person evaluated the impressions of the classroom. By introducing communication functions and developing a system in which multiple people share the same VR space, we analyzed hand-raising communication among multiple people.

6 Summary

In this study, we developed a system for using a VR headset to analyze hand-raising communication. We developed a virtual classroom and CG characters who raised their hands, and made introducing the hand-raising motions measured in previous studies for a constructive analysis possible. Using our system, we performed an evaluation experiment with participants. The participants’ impressions of the hand-raising communication changed as the heights of the raised hands and numbers of characters who raised their hands differed. In detail, the scene in which all CG characters raised their hands to a low point and the scene in which one-third of the CG characters raised their hands to a high point were rated equally. This means that the effect of the number of participants who raised their hands was larger in this study than it was when we projected CGs on a large screen. We think the reality of a VR headset caused this increase.