Keywords

1 Introduction

Imagine if you are having a device and monitor. If you touch specific part of the device, that action reflects to CG. Children might be fascinated more than just playing with ordinary device without action reflecting to CG. We have proposed a novel device that has this feature. The Digital Dollhouse (Fig. 1) enhanced traditional psychological play therapy with digital sensors and computer graphics on this mechanism [1]. It obtains significant difference in reaction of children that compared with normal dollhouse especially in taking the responsive actions are twice as much as the normal device, and for the abstract concept three times, the number of miss-communications are 80 % less than the normal one. Even more, children spent playing with this device twice as longer time as the normal device.

Fig. 1.
figure 1

Digital Dollhouse and correspondence room [1]

Full size image

Based on the result, we, psychologists, brain scientist and human computer interaction researchers, have an assumption that there is a possibility for enhancing biological effect on collaboration of touching and viewing, compared with just touching, and just viewing. Based on this assumption, we define the cooperation of the action of touching the device (Haptic), and reflection of action to CG (Virtual) for viewing the result as a feedback of their action, and propose “Haptic Virtual Approach”. It includes not only the haptic and virtual collaboration method, but also present the novel human-computer interaction technology that collecting touching information of humans with several sensors and present it for some representative image for the display. This immersive space should have lots of discussion that how to achieve the immersive play, real-time, virtual expression one-to-one mapping, and how to illustrate abstract image, and applications etc. In this paper, a first step of our research, we will present an experimental evaluation that uses biological responses of subjects with brainwave and heartbeat. We find out that our presented “Haptic Virtual Approach” has various results, different from each participant.

This paper is organized as follows. Section 2 explains related works. Section 3 explains our proposal. Section 4 shows system design and implementation. Section 5 introduces the experiment we conducted and the results earned from the experiment. Section 6 discusses about subjects we obtained from the experiment. At last, Sect. 7 draws the conclusion.

2 Related Works

There are several studies that apply the method of measuring biological information to estimate human’s consciousness.

In the study of estimating “Wakuwaku (Japanese word mainly represents excitement)”, Galvanic Skin Reflex (GSR), Electrocardiogram (ECG) and breathing rate are measured [2]. This study concluded that the use of heart rate, earned from ECG and GSR might have showed the measurement of “Wakuwaku” feeling.

In the study of measuring attention and relaxation, Electro Encephalogram (EEC) are used to measure these consciousness [3]. This study uses its own formula to calculate stress from values earned from brainwave sensor using. In this study, several experiments that are often used to generate stress deliberately were conducted and the brainwave was measured. From the results, the value of brainwave changes as the participants participated the experiment numerous times. From the results shown, participants generated lower stress as they participated on the experiment many times.

3 Proposal

Based on the preliminary experiment that had conducted with the Digital Dollhouse, we found out results that might be useful for therapy. For example, children who experienced the device increased the number of words spoken, comparing to normal device [1]. Somehow, the result was abstractive, and difficult to say that the device is suitable for enhancing speaking.

From the results, we presumed that combination of house-shaped device and CG enhanced speaking. From the control of concrete object and visualizing invisible and abstract information, we named the effect as “Haptic Virtual”.

3.1 Combination of Touching and Viewing

We defined Haptic as controlling and touching concrete object, and Virtual as expressing the results of controlling and touching by changing information displayed on CG. Combination of these elements makes Haptic Virtual.

For example, there are functions to control character on the device. On the view of Haptic, person who experience the device control doll and placing it. On the view of Virtual, person will control the character inside CG. Combining these together to make Haptic Virtual, character inside CG can be controlled by using doll and placing it on the specific part of the device.

3.2 Use of Biological Information

To evaluate our interactive device, we consider the results earned from previous study that increase the number of speaking and the longer continuation on the device. We assumed that enhancing concentration and reducing stress were aroused from the device compared with the just only CG, touching the concrete object. To know the effect correctly, biological information as the measurement of stress and concentration can explain the background of these results. We measured heartbeat and brainwave to compare the effect between Haptic only, Virtual only and Haptic Virtual.

We used “WHS-1” for measuring heartbeat [6]. From this heartbeat sensor, we obtained LF/HF. This is a ratio of revitalization of the sympathetic nerve (LF), which revitalize when stressed and the parasympathetic nerve (HF), which revitalize when relaxed. This means that when the wearer of the heartbeat sensor feels more stress, the value of LF/HF increases [4].

Brainwave is obtained from “MindWave Mobile” [7]. From this brainwave, we obtained Attention, which is the value calculated with specific algorithm by this brainwave. Its value is from 0 to 100, and it is used to measure concentration [5]. In another word, the value will be lowered when the user with the sensor attached is having less concentration.

4 System Design and Implementation

To know the effect of the “Haptic Virtual Approach”, we firstly focus on the topics that provided with abstract image, since we assume that virtual expression would be more effective with understanding the abstract things. Time and seasons has no actual things that can touch. However, it can be understanding only with the observable evidences that clock number and different color of leaves viewing in the window. We add the functions to change time and season. The other abstract thing, we present an emotion with displaying character, with moving around the actual doll in the house (Fig. 2). We also implemented the function to control temperature and brightness of CG, by using buttons and knob.

Fig. 2.
figure 2

Playing with Digital Dollhouse with design implemented

Full size image

To compare the effect of Haptic only, Virtual only and Haptic Virtual, devices with Haptic only and Virtual only are required. We used the device we implemented with several functions changed to make these two devices. On device with Haptic only, there is no CG information, and user need to recognize the change of the situation by themselves. On device with Virtual only, there is no dollhouse to control, and user uses keyboard to control object.

5 Preliminary Experiment and Evaluation

We held a preliminary experiment to compare biological information earned from Haptic, Virtual and Haptic Virtual.

5.1 Objective

The experiment has held to know what kind of effect will appear not only on device with Haptic Virtual implemented, but also with Haptic implemented only and Virtual implemented only. There were 5 participants who participated on the experiment (Age of 21 to 22, Male: 4, Female: 1), and used biological information to evaluate the effect.

5.2 Method

The subject will proceed the experiment as follows.

  1. 1.

    Sit on the chair and attach heartbeat and brainwave sensor

  2. 2.

    Getting instructed the way how to control the device

  3. 3.

    Stay still for two minutes.

  4. 4.

    Experience devices with Haptic, Virtual and Haptic Virtual implemented, in order.

We started to measure biological information as soon as we started the process (3). We measured the biological information while staying still in the use to settling criteria value, since the biological information’s value is going to be different by the individuals. We used this biological information to calculate the changing rate. The formula is listed below.

$$ \begin{aligned} Changing\,rate \,of\,biological\,information\,while\,experiencing\,device\left( \% \right) \hfill \\ \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, = \frac{{\left( {\left( {Staying\,still} \right) - \left( {device\,experiencing} \right)} \right)*100}}{(staying\,still)} \hfill \\ \end{aligned} $$
(1)

This calculation method was used to calculate the changing rate of heartbeat at previous research [8].

After the experiment was held and calculating changing rate, we conducted hearing research to all participants who participated to the experiment. This research was conducted to know the influence between individual’s thought and biological information. Content of the research was discussed by analyzing each participant’s biological information recorded.

5.3 Environment

Experiment was held at silent room with only participants and explainer. We settled the situation to reject factors such as noises caused by human activities, participants getting strained from other person’s existence to minimize an influence to biological information. Participants sit on a chair and attach heartbeat and brainwave sensor. They experience the device in that situation, and biological information is obtained (Fig. 3). Moreover, process of the experiment was recorded by video camera, to analyze the experiment in detail.

Fig. 3.
figure 3

System of experiment

Full size image

5.4 Result

Experiment was conducted as the way explained at Sect. 5.3. From these results, we compared the difference of biological information of Haptic (H), Virtual (V) and Haptic Virtual (HV). First, we analyzed the pattern of biological information’s transition each participant recorded. Next, we analyzed the dimension of biological information each participant recorded. At last, we analyzed the summary of hearing research. The transition of biological information each participant recorded is listed on Fig. 4.

Fig. 4.
figure 4

Result of biological information earned from the preliminary experiment

Full size image

As the result shown from all the participant’s biological information on Fig. 4, we found out that transition of biological information showed some similar patterns with plural groups of participants. The pattern of these biological information is listed below.

  1. (1)

    Measurement of LF/HF (stress) with heartbeat

There were three patterns of transition on heartbeat. One of the pattern was seen from participant 1, 2 and 5. The value of Heartbeat recorded from H to V increased its value, and the value recorded from V to HV decreased its value.　From this result, participant 1, 2 and 5 increased LF/HF, or enhanced stress on V. Somehow, comparing to V, H and HV recorded lower LF/HF. This means that touching house-shaped device may had lead to decreasing stress. But just viewing at CG enhanced stress.

Participant 3 showed next pattern. The value of heartbeat recorded from H to V increased its value, same as first pattern. But the value of heartbeat recorded from V to HV also increased. We suggested that this participant may had increased stress, or enhanced stress because of time taking to conduct experiment. The longer the experiment held, more stress the participant may had enhanced.

Last pattern was seen from participant 4. The value of heartbeat recorded from H to V decreased its value, and the value of heartbeat recorded from V to HV increased its value. With the heartbeat recorded the lowest on V, elements which this participant only has might have lead this result.

  1. (2)

    Measurement of Attention (concentration) with brainwave

All participants made the same pattern of transition on brainwave. They first experienced Haptic, and increased the value of the brainwave from H when experiencing V. However, value of the brainwave decreased from V when experiencing HV. From this result, we can suggest that V enhances the value of brainwave, meaning that displaying CG which changes enhances concentration. On the other hand, H and HV, which controls house-shaped device, couldn’t have enhanced concentration enough, comparing to V.

Next, we analyzed the dimension of biological information. Dimension of heartbeat and brainwave each participant recorded is listed on Table 1.

Table 1. Dimension of biological information each subject recorded
Full size table

Same as transition of biological information listed on Fig. 4, there were several patterns of dimension appeared from Table 1. Here are the patterns that can be seen from this table.

  1. (1)

    Measurement of LF/HF (stress) with heartbeat

The result from participant 1, 2 and 5, V was the device with the highest value of heartbeat recorded. Looking the result in detail, HV recorded the second highest value in participant 1 and 5. What we can consider about this result is that V, which controls and views CG information increases stress. And H, which only controls house-shaped device decreases stress. With Haptic Virtual, it recorded the intermediate value of H and V. As the result shown from participant 2, HV recorded the lowest value of heartbeat. This means, for participant 2, experiencing HV felt lower stress than experiencing any other devices.

As the result shown from participant 3 and 4, HV recorded the highest value of heartbeat. Looking the result in detail, heartbeat recorded from participant 3 increased its value as the participant experienced the device. Meaning that time elapsing may had caused the enhancement of stress. Participant 4 recorded the lowest value of heartbeat on V. Somehow, the value of heartbeat on HV, which also uses CG, recorded the highest value.

  1. (2)

    Measurement of Attention (concentration) with brainwave

All participants recorded the highest value of brainwave on V. As the result shown from remaining H and HV, participant 1, 4 and 5, HV recorded higher brainwave than H. From this pattern of dimension, presence of CG may have enhanced concentration. Beside, presence of control of house-shaped device may have lead to cutting down concentration. We can assume that result earned from HV is a merge of results earned from H only and V only.

As the result shown from participant 2 and 3, HV recorded lower brainwave than H. What can be assumed from this result is that the participant might had felt tedious on experiment.

As we analyzed transition and dimension of biological information, some patterns were recorded. However, discussion about the result was completely subjective, and participant’s impressions were required to substantiate the results earned from the experiment. For this problem, we conducted hearing research from all participants. The content of research was considered by results analyzed by the result shown from the transition and dimension of biological information. Several answers were earned as a result of the research. The correspondence of the result of the research and corresponding participants who answered the research is listed on Table 2.

Table 2. Summary of hearing research
Full size table

As the result shown from Table 2, four types of answers were mainly heard from five participants. The detail of the answer is that whether the participants like to play video games, role-played character’s role, felt tedious on the experiment and felt some problems on the device. We analyzed the relationship between hearing research, and transition and dimension of biological information.

All participants answered “Likes playing with video games”. Relevance of biological information and the answer appeared at brainwave. All participants recorded the highest value in V. This means that V, which controls and views the change of CG enhanced the attention of the participants, and their partial of playing video games may had lead this result.

Participant 2 and 3 answered “Role-played character’s role”. Relevance of biological information and the answer appeared at brainwave. Value of the brainwave was significantly high comparing to other participants. On the other hands, participants who answered “Tedious experiment”, which was participant 1, 4 and 5, recorded lower brainwave than the one who answered “Role-played character’s role”. From this result, person who imagined the character’s role enhanced attention. On the other hand, person who felt the experiment tedious, which was participant 1, 4 and 5, couldn’t have enhanced the attention that much.

The answer “Problems on device” was heard from participant 1 and 3. Detail of this answer is that participant felt difficult to control the device because of bared wires and control with no descriptions. There was no relationship between the answer and biological information found. Somehow, both participants recorded the lowest heartbeat at H. This means that they felt the least stress through all the devices, even thought they felt that controlling the Haptic device was a problem.

6 Discussion About Experiment

From the experiment, value of biological information obtained from 5 participants showed several patterns. Haptic Virtual and other devices. Much as the result of hearing research, answers obtained from participant and the result of biological information showed specific relationship. Although, the method of the experiment we conducted was not well discussed. To obtain reliable and better data, more discussions about the method of the experiment is required. Here is the list of the problems that should be discussed at next experiment we’re going to conduct.

  1. (1)

    Experiencing order of the devices

At preliminary experiment, experiencing order of the device was constant to all participants. We are planning to prepare several patterns of experiencing order, to know the difference appearing by order of experiencing the devices.

  1. (2)

    Inducting detailed questionnaire

From results earned from preliminary experiment, we earned the knowledge that elements each participants have may influence the results earned by experiencing the devices.

  1. (3)

    Changing method of the experiment

We prepared a method to ask the participant to stay still at the beginning of the experiment. Somehow, while conducting the experiment, behavior while staying still was different by every participant. We’re planning to involve the method to close their eyes and stay still during this situation.

  1. (4)

    Ask participants to participate the experiment plural times

At experiment we’re planning to held, we’re going to ask some of the participants to participate the experiment plural times. This is to know the effect obtained from plural experiments.

7 Conclusion

In this paper, we examined “Haptic Virtual” as an approach that combines concrete object with sensors attached (Haptic) and CG (Virtual). With this combination, CG can be controlled by touching the objects inside the concrete device. We also discussed the way to evaluate “Haptic Virtual” by using biological information. From the result of the experiment, we found out that specific patterns were found on the transition and dimension of biological information. Hearing research also made progress to reveal the relationship between biological information each participant recorded and answer obtained from the research. To ensure the result earned from the preliminary experiment, more data is required. Furthermore, to obtain more reliable data, revision of the method of experiment is required.