Keywords

1 Introduction

Autism Europe defines autism as a “lifelong disability… People on the autism spectrum experience persistent difficulties with social communication and social interaction, and might display restricted and repetitive patterns of behaviours, activities or interests.” Autism Spectrum Disorders (ASDs) are a group of pervasive developmental disorders characterised by core deficits in three domains, i.e. “social interaction, communication and repetitive or stereotypical behaviour” [1], occurring more often in boys then in girls (4:1). While the level of impairment of suffering from autism may differ among different cases of individuals, the bearing on the individuals themselves and the people around them can be considered as being “life-changing” [2]. Parents, for example, describe how children on the autism spectrum may sometimes be left out of social events in schools simply because the environment is not adapted to their needs. This lack of adaptation is the result of a combination of numerous factors. One of these challenges is the lack of knowledge of how to deal and how to react when interacting with individuals suffering from ASDs. Teachers face enormous challenges when attempting to integrate children on the autism spectrum in the daily classroom experience. The biggest of these challenges lies in the fact that it is very difficult for teachers to fully comprehend the situations, the triggers and the stress levels that an autistic child may be subject to when interacting in class.

Most studies of ASDs in relation to technology focus on the way different mediums can help the student learn and communicate. They create technology-driven media to aid the student directly by changing the way learning is presented. However little work has been done to explore how such media can use narrative to support empathy. Following on our previous study ‘Autism VR’ [17], we propose the use of Virtual Reality (VR) coupled with Haptic feedback technology as a new approach that could recreate and repurpose a narrative to immerse adult teachers into the world of a child on the autism spectrum to a degree higher than what was already achieved. VR may be a simulation of the real world or the creation of a completely new world based on computer graphics to provide an experience that allows the user to understand concepts [3]. The realism of the virtual environment allows the user to acquire important skills, increasing the probability to transfer them into real life behaviour [4,5,6]. Building on ‘Autism VR’, this study explores various avenues in creating increased immersion in the virtual environment by exploring innovative communication mediums. While virtual reality implementations mostly focus on the visual and audio sensory elicited emotions, the addition of the sense of touch as a continuation of our previous study, should give an increased level of immersion and a provide a more realistic performance to anyone using the system.

The project’s scope is that of building upon the use of Virtual Reality as a tool with which to understand better the case of autism in children. More specifically, the research aims at bringing together VR and Haptic Feedback. As the project unfolds and carers experience this VR immersion into autism, we want to ask questions such as Will the VR have a visible impact on the carers’ empathy towards children with autism? Also, we want to note how the combination of these two technologies improves one’s ability to empathise with children when compared to the results achieved when the standard approach to implementing VR was adopted in earlier research. The next section provides an overview of the research that has already been carried out in the area, highlighting work that has already been done in ‘Autism VR’.

2 Literature Review

2.1 Autism and Virtual Reality

Oliver Sacks (1995) describes autism as “an impenetrable world, one which makes it difficult to unveil. The reason for this difficulty lies in the need to truly understand and explain what the relationship is with an autistic child, who, to a certain extent is a strange being who lives on a different plane of existence. In most cases an autistic child is a person whom we cannot connect with” [7]. While it is only recently that Virtual Reality technology applications have delved in autism therapy [13], the use of other technology as a means of helping individuals on the ASD scale has for long been an area of interest for many researchers. Interventions outside the area of Virtual Reality have quantified social skills and cognition over time using a variety of techniques and measures leading to mixed success rates [9]. The use of computers and computer technology for children with autism has been shown by research to have positive and beneficial effects [10]. Due to the increase in diagnosed cases of ASD, software and hardware dedicated to persons with autism have been developed for several decades. These solutions reinforce ASD sufferers’ strong points and work on their weaknesses, helping them to increase their vocabulary and communication [12].

A number of studies have been done about the potential benefits of VR in supporting the learning process, particularly in relation to social situations, in children with autism [8]. Among the studies that have been carried out to teach children how to behave in social situation and how to understand social conventions are “Virtual Café” [14] and “Virtual Supermarket” [15]. These studies reported that the user improved in social skills and their performance also increased as the child was able to transfer the learn skills from the VE to real life situations. There have been discussions that these virtual environments can aid the educators, especially with children with special needs. This is because the VE has content which can be controlled. For example by allowing wheelchair users “.. to see how the world looks like from a standing perspective.. [and] to take part in activities or visit places that are inaccessible to real life” [16]. It is on these lines that “Autism VR” [17] drew its inspiration. The experiments carried out in “Autism VR”, which brought VR into a classroom environment, prove that Virtual Reality can in fact be used as an effective teaching tool for carers of individuals on the ASD spectrum. Among the participants taking part in the study 50% of the participants reported that they were now able to better empathise with the children.

2.2 Participative Narrative

What differs Virtual Reality form conventional media outlets is the fact that users in a 3D virtual environment play a very important role in the building of the story and their own overall experience since this depends on where they move and look, and their reactions within the world itself [18]. This form of narration is therefore a participative one. Audience participation within virtual reality is an idea recently referred to as ‘presence’ [19]. Presence has been defined “as the experience of one’s physical environment; it refers not to one’s surroundings as they exist in the physical world, but to the perception of those surroundings as mediated by both automatic and controlled mental processes [20]”. Perception is often discussed in relation to another crucial aspect in the context of effective VR systems [21]. Immersion in the context of virtual reality refers to the amount of sensory input that the virtual reality system creates. An increase in immersion often leads to an increase in the presence felt by the user [22, 23] and while immersion levels in the traditional approaches to VR systems usually involves the use of the visual and audio sensory mechanisms, this research looks at the possibility of incorporating the use of haptic feedback and haptic feedback devices.

2.3 Haptic Technology

Haptic Technology is defined as the technology of virtually touching and feeling objects and forces. It is a new emerging technology from the area of Virtual Reality that allows computer users to use their sense of touch to feel three-dimensional virtual objects using haptic devices. Touch is a very powerful sense and the sensation of touch is the brain’s most effective learning mechanism that has for long been neglected in the field of VR. Essentially what this technology gives in terms of benefits, is the ability for a person to feel a simulation of a solid object as if it is really in front of him. Haptic feedback groups together three modalities of feedback, force feedback, tactile feedback, and proprioceptive feedback [24]. More recent research tends to couple force feedback and proprioceptive feedback under the category of kinaesthetic feedback.

2.4 Tactile Feedback

The second type of Haptic feedback is tactile feedback. While kinesthetic feedback is feedback received through the arousal of muscles, joints or tendons, tactile feedback is feedback received on the surface of the skin. Skint issue has a number of different receptors embedded in the skin and right underneath it and these allow the brain to feel things such as vibration, changes in pressure, and touch, for example. The nature of this type of feedback means that in order to trigger such feelings, one does not necessarily require hardware to be worn on the user’s person as is the case with kinesthetic feedback. [34] list three types of conventional strategies for tactile feedback. The first strategy is similar to the strategy used for kinesthetic feedback, i.e. attaching devices on user’s fingers or palms. This strategy has been explored in research through implementations such as CyberTouch [27], GhostGlove [28], and SaLT [29]. In implementing these approaches researchers made use of numerous pieces of equipment such as vibrotactile simulators, motor driven belts and pin-array units. However, the result of this strategy would mean that the system degrades tactile feelings due to the contact between the skin and the device occurring even when there is no need to provide tactile sensation. A second strategy for tactile feedback is also presented. An implementation of this strategy is seen in [30]. Sato et al. implement a strategy whereby the system controls the positions of tactile devices so that they only make contact with the skin when tactile feedback is required. To achieve this, Sato et al. make use of an exoskeleton master hand that generates encounter-type force feedback. The last is strategy aims at providing tactile feedback without any direct contact to a user’s person. In [31], Suzuki and Kobayashi use air-jet to realize noncontact force feedback. However, when it comes to producing detailed texture sensation rather than kinetic feedback, Drif points out [32] that there are at least two major drawbacks. First, air-jet cannot produce localized force due to diffusion. Second, it also suffers from limited bandwidth. In addition, even if multiple air-jet nozzles are used, the variation of the spatial distribution of the pressure is quite limited.

2.5 Tactile Feedback Using Ultrasound

Of the three tactile feedback strategies perhaps the one which promises to take Virtual Reality to the next level of immersion and user interaction is the strategy which requires no actual contact to the user’s person. Having already mentioned the use of air-jet for this purpose, other researchers have looked at eliciting tactile feedback through the use of radiation pressure of airborne ultrasound. The use of airborne ultrasound for a tactile display was pioneered by Iwamoto et al. [33]. The potential of using ultrasound for tactile feedback lies in the fact that such feedback can be applied to bare hands in free space with a high spatial and temporal resolution [33]. Whereas other strategies restrict the user in one form or another, in this scenario the user is able to move freely without any hindrance while receiving high fidelity tactile feedback with 3D visual objects.

The principle by which this method is applied in a real-world environment is based on the phenomenon of acoustic radiation pressure. In their study, Iwamoto et al. fabricated a device that comprised of 91 airborne ultrasound transducers set up in a hexagonal arrangement and could produce sufficient vibrations of up to 1 kHz. When the airborne ultrasound is applied on the surface of the skin, almost all the incident ultrasound is reflected back meaning that almost all of the incident acoustic energy is reflected on the surface of the skin. This in turn results in the tactile feedback felt by the end user.

Previous literature in the areas that are to be tackled by this study have shown potential in the approach being proposed. Having reviewed this literature, the next sections introduce the methodology behind the proposed VR implementation using tactile feedback.

3 Methodology

The scope of this project is to implement a novel tactile feeling within a VR experience of the daily life of an autistic child at a school. The suggested approach aims mainly to enhance the interaction between the designed VR environment and potential users. As a result, users are more immersed in the VR experience and can feel empathy towards these disordered children. This section contains three subsections; Haptic interaction, gesturing interaction, and interaction feedback technique to measure the different reactions of users.

3.1 Haptic Interaction

During the first phase of the VR autism application, we encountered a number of challenges when we needed to simulate some human interactions and feelings. Autism is a unique disorder and autistic children behave differently upon the same situations. One of the main interesting findings was that they share tactile frightening issues. During the first stage of the project, we managed to simulate some emotions namely fear, panic, and confusion within 3D sound and make the sound of the thoughts as it comes from the head. Even though the feedback was positive, there is still room for development to improve the experience in general. Simulating the autistic child’s thoughts using 3D sound techniques will be more realistic if we manage to make users feel as if they were poked. By synchronizing both interactions, users can definitely better understand the Autistic Children hurt feelings when they are touched. Figure 1 illustrates the integration between the implemented elements that used to develop the VR experience.

Fig. 1.
figure 1

(Source: Self).

The VR environment main elements

Full size image

3.2 Haptic Interaction

In VR autism experience, users interact with a virtual designed object. The suggested scene includes two virtual hands that simulate the action when the autistic child will be flipping a small ball. In addition, the virtual object was added to the VR environment to help users feel more immersion. While the flipping scene plays, the virtual hands start acting as if the child is actually touching a physical object. In the meantime, the user will be feeling the visualised object in addition to feeling the movement of the actual virtual simulated object. Figure 2 shows a diagram that explains the two actions; the virtual and the physical actions of the user during the VR experience. Haptic interaction will provide an artificial touch sensation that simulates natural human touch. We intend to use more virtual interaction tools to improve users’ experience through increased immersion levels.

Fig. 2.
figure 2

(Source: Self).

Virtual and physical interaction in the VR feel autism experience

Full size image

3.3 Interaction Feedback

VR technology has shown great potential for simulating how autistic children’s private life looks like. The first version of the application was presented in different conferences and global events and showed a great impact on users including parents who have autistic children. During the feedback collection stage, there were some difficulties with measuring direct facial reactions due to the size of the used head-mounted display hardware. Using haptic devices can easily make the feedback measuring process easier. The interaction with the virtual object allows designers observing the gesturing interaction and detect any design flows. Using different sensors, to measure the hand movement, handshaking, and hand sweating, we can collect additional insights about user interaction. Figure 3 illustrates the main used elements to mimic and measure the reactions of autistic children when they are poked.

Fig. 3.
figure 3

(Source: Self).

Tactile virtual and physical scenario

Full size image

In addition, using a haptic device to create some artificial touch sensations will help developers when measuring user’s physical interactions. Autistic children are extremely sensitive to situations when they are being touched. Such a situation can easily scare them, and for the purpose of this study, it can be simulated by integrating the hepatic interaction, visual effects, and 3D sounds in the head. In addition, autistic children live in repetitive life. One of the observations made when carrying out the previous study was that they like playing with the same object. This could be a toy, a tool, a remote control…etc. This proposed approach is investigating the use of the Haptogram Ultrasound device to simulate playing with a ball scenario. While in the virtual environment there will be virtual hands and a virtual ball object, the user will be able to interact with the same virtual object in reality. The capability of using Ultrasound waves to simulate the tactile experience will add more immersion feeling to the designed VR environment. In addition, users’ reactions will be observed and measured during the experience. The Haptogram device will measure the movements of the hands. This will be used as a key factor for measuring the users’ reactions when the autistic child responds to the poking action. Figure 4 shows the used feedback measuring technique of the physical actions.

Fig. 4.
figure 4

(Source: Self).

VR feel autism feedback scenario

Full size image

4 Results and Finding

Although the project is still ongoing, the collected results out of the developed application have shown promising results. During the testing stage, a prototype included only the binaural and the image effects interactions elements. Thus, only five selected feelings were considered to measure the final feedback about the designed application. The results showed that users were immersed in the experience and strongly felt: excited, sad, anxious, or surprised. Figure 5 shows the results of users’ measured reactions during the experience. Participants stated that it was a very useful application and that it was a high-quality VR experience. At the time of writing, initial observations have indicated that the used haptic tool is still expensive, and it still has some limited functions. In addition, it requires some extra integration steps to design suitable VR objects. The next section highlights a future plan for the project to overcome the pricing and the scalability of the suggested haptic tool.

Fig. 5.
figure 5

(Source: Self)

Measured feedback physical reactions

Full size image

5 Future Work

The second phase of the VR Autism project has enabled a new tactile interaction technique to enhance users experience and immersion. Based on the results out of the two designed versions, it is clear that there is a great opportunity to improve the developed VR experience. These improvements should ensure that the experience is capable of simulating more real-life situations and challenges that autistic children might go through when they are at school. The next stage of the project aims to design a Cave VR experience for the different situations of a normal daily life for an autistic child when at school. In addition, work has started on a collaboration endeavour with another group of researches to design a suitable hardware system. This ultrasonic system will be used mainly to simulate real-life tactile actions. The suggested novel approach will investigate the possibility of designing cheaper hardware to be installed in a VR Cave experience and implement real tactile actions. Thus, this will pave the way into adding tactile feedback capabilities in any designed VR application when tactile actions are required.

6 Conclusion

This study presents a new approach into the creation of novel VR systems targeted with the scope of helping autistic children in mind. Previous studies have shown that what we are setting out to achieve can in fact be obtained, and results have been encouraging. Through this study we are aiming to go a step further and introduce haptic technology as a means to further help these individuals in their daily lives. ‘Autism VR’ provided a first step into what can be achieved through the use of virtual reality as a tool to help teachers adjust their methods to better cope with autistic children. With ‘Feel Autism VR’ we aim to make an unprecedented leap into the creation of such tools.