Keywords

1 Background

Co-creation refers to creative activity wherein people with different backgrounds and values share thoughts and dreams and come together to achieve them. To this end, it is necessary for the participants to resolve for themselves the contradiction of not violating the other person’s independence or individuality while sharing the same context with him or her [1]. Therefore, a space is created wherein there is a sense that “I” and “we” coexist, and each person must be positioned there. This is mediated by “expressive bodies.” That is, it could be said that when the participants express themselves through mutual bodies, an inclusive sense of coexistence and an awareness of such, that is, an awareness of “we,” is created. As a concrete method of realizing this, the authors have focused in the past on “hand contact improvisation,” which involves the participants joining the palms of their hands together and carrying out improvised bodily expression together. Experiments by one of the authors, Nishi, have confirmed that in this hand contact improvisation, as the expression grows deeper, the relationship between the participants transforms, and the expressions deepen from expressions between individuals that is between “you” and “me,” into “our” expressions shown in Fig. 1 [2]. In other words, the main distinguishing feature of hand contact improvisation is that it leads to equal reciprocity. Also, as the authors continued to carry out workshops whose main activity was hand contact improvisation in areas stricken by the Great East Japan Earthquake, it became clear on the basis of experience that hand contact improvisation is an extremely useful means of supporting communication with children with developmental disabilities for whom verbal communication is difficult or who tend to avoid interpersonal interaction, in particular children with autism spectrum disorders [3].

Fig. 1.
figure 1

(picture; Ken Yabuno)

Changes of relationship in hand contact improvisation

Full size image

Also, most existing communication support technologies are designed based on the condition that linguistic or symbolic communication is possible. Thus, in the past, there have been almost no cases in which a co-creative expression interface was developed that deepens the relationship between the participants and creates an awareness of coexistence through mutual bodily expression regardless of the presence or absence of disabilities, their severity, or the age of the participants. Therefore, although more attention is being paid to support for communication among children with developmental disabilities (autism) [4–6], this is occurring at a very late stage.

Thus, based on the knowledge gained from the authors’ research on hand contact improvisation, in this study, two mutual expression interfaces that encourage children with autism to express themselves bodily through hand contact improvisation, and that enable them to express themselves together with those around them, were developed. Also, these interfaces were brought to sites where communication occurs with children with autism, and their usefulness was investigated; the results are reported below.

2 Tabletop Single-Axis Hand Contact Improvisation System and Co-Creative Expression Interface

The authors carried out workshops whose main activity was hand contact improvisation once per month beginning in April 2012 in the Great East Japan Earthquake-stricken cities of Higashimatsushima and Ishinomaki, Miyagi Prefecture, Japan. A diverse range of people of various ages and both sexes, who had been affected by the disaster to different degrees, some of whom were afflicted with developmental disabilities such as autism spectrum disorders and some of whom were not, took part in the workshops. During this time, most of the children with autism, for whom verbal communication was difficult, were able to continue taking part on their own, and as connections were established with them, problematic behavior towards others lessened, a surprising result. However, it was observed that some of the children with autism were still unable to put their hands together on their own or to continue with hand contact improvisation for a long period of time.

Also, in the past, to research the dynamics of hand contact improvisation, the authors developed a single-axis hand contact improvisation system for jointly creating expressions that uses a slide board with a single degree of freedom to the front and rear shown in Fig. 2 [7–9]. The results confirmed that with this system, co-creative expression was possible for those who were skilled at expressing themselves even if the hands of the participants did not touch directly. Also, it was found that there were differences in the measured data when co-creative expression occurred and when it did not occur. Specifically, when co-creative expression did occur, changes in the movements of the body overall in which awareness did not participate directly (changes in COP [center of foot pressure]) occurred prior to conscious hand movements (movements in the slide board), and chaos attractor-like structures occurred in the return map of the movements of the slide board, and further, intermittent chaos-like structures occurred in the temporal changes of the differences in the degree of force exerted on the slide board on both sides (expressional jerk), and so on. These experimental findings suggest that this system can be used to measure and evaluate the process of the deepening of expression as displayed in Fig. 1. Thus, not only is it likely that using the system with children with developmental disabilities will help to support hand contact improvisation, but also, the participants’ bodily and emotional state as well as the process of the deepening of relationships with others may be reflected in the system, and it may be possible to use it to obtain objective data regarding these factors.

Fig. 2.
figure 2

Single-axis hand contact improvisation system (Color figure online)

Full size image

Thus, in this study, first of all, the size of the existing single-axis hand contact improvisation system was reduced, and a “tabletop hand contact improvisation system” that can be used at sites such as child welfare facilities was developed. However, it is not possible to use this system at workshops where hand contact improvisation is performed through three-dimensional bodily movements. Next, therefore, a “hand contact improvisation interface” that consists of cylindrical interfaces equipped with visual, auditory, and force feedback interfaces, and in which participants carry out mutual hand contact improvisation by freely moving in three dimensions, was developed.

3 Tabletop Hand Contact Improvisation System

3.1 Design Principle

Considering the sensitivity of people with autism to changes in the environment, a interface that could be brought into areas where children with autism live was needed. Therefore, it was determined that the interface would use a tabletop design. The design requirements were as follows.

  1. 1.

    A portable, tabletop-size, compact interface

  2. 2.

    Possible to measure exchanges of force, the positions of the hands in the context of the expressions created by the two participants, and the force on the hands

  3. 3.

    Possible to measure the unconscious movements of COP

  4. 4.

    Equipped with force feedback interfaces to encourage bodily awareness of the hands of both participants

  5. 5.

    Equipped with mechanisms with a high degree of backdrivability so as not to interfere in the movements of the participants

(4) and (5) were included in the design requirements in view of possible future application of the previously developed telecommunication-based hand contact improvisation system [10] and the single-person hand contact improvisation system [11] to children with autism. To fulfill the above requirements, the interface had to be no more than 400 mm wide and 500 [mm] deep so that it could sit on a tabletop. Also, in light of the mobility of the hands of an adult male facing directly ahead, the stroke of the slide board had to be at least 300 [mm].

3.2 System Design

The system is made up of a sensor interface, a force feedback interface, and a measurement interface that measure the position of the slide board, the force on it, and the COP shown in Fig. 3. In this system, to make the interface small and to simultaneously lengthen the stroke of the slide board, the sensors and the force feedback mechanism were all built into the unit on which the slide board was installed. In specific terms, a linear encoder was installed on the side of this unit, with the position of the slide board measured at a resolution of 10 [µm]. It is possible to independently measure the force on the slide board from both sides within ± 100 [N] because an aluminum post is built into the center of the unit and load cells are installed on the front and rear ends. When doing so, it is possible to choose between a large grip that keeps the hands of the two participants from touching directly and a small grip that permits their hands to touch. Regarding the force feedback mechanism, a DC motor is installed beneath the unit, and force feedback of up to 40 [N] is provided through torque control. When doing so, the combination of a decrease in speed by low gear ratio and a rack and pinion mechanism that reduces the tooth contact ratio provides backdrivability at 1.2 [N]. COP measurement is achieved by installing a stabilometer on the surface of the participants’ seats and on the floor. Also, to ease onsite usage of the interface, the system can run on two 12 [V] lead batteries. In the above system, data measurement and control is achieved at a maximum of 200 [Hz] using a measurement control interface comprised of a PC and a DAQ.

Fig. 3.
figure 3

Overview of tabletop hand contact improvisation system

Full size image

3.3 Application of the System to Autistic Settings

We brought this system to a child welfare service, “Mirai” (Ishinomaki, Miyagi Prefecture, Japan) in order to investigate its usefulness (Fig. 4). In our 5-h trial, we observed a strong interest in the program that exceeded most of our expectations: with children lining up and waiting for their turn to participate. Of particular note was Boy A, who exhibited strong interpersonal avoidance behavior - usually by throwing himself on the ground repeatedly - who sat in the chair of his own volition and utilized our system to engage in hand contact improvisation. In addition, we observed Girl A engaged in hand contact improvisation even when using a small grip that involved touching hands, who even urged other children to participate. Further, Boy H stated, “I want the special school teachers to see my expression.”

Fig. 4.
figure 4

Situation of using tabletop hand contact improvisation system

Full size image

Figure 5 provides an example of our test results. As it shows, there was significant difference in participation duration and hand movements between each participant. This suggests that the bodily expression of children with autism is not uniform, but varied, and communication may be possible despite difficulties with linguistic interaction. Notably, the intermittency structure that is accentuated in co-creative expression occurred in the hand contact improvisation between autistic participant R and the perspective of expression expert N (Fig. 6). In addition, expression expert N subjectively judged this expression as co-creative expression. These results suggest that there may be a rich sensitivity inherent to autistic children similar to that seen in expression expert. We also received comments from parents and staff members such as, “I was surprised to see them engage with the interface without hesitation,” “The system did a good job of communicating the feelings of participants,” and “I saw facial expressions and emotions that I’ve never seen before.” These results indicate that use of this system can support efforts at communication by facilitating co-creative expression among autistic children with a tendency towards avoiding interpersonal contact.

Fig. 5.
figure 5

Profiles of tabletop hand contact improvisation system in autistic settings (Color figure online)

Full size image
Fig. 6.
figure 6

Profiles from the experiment of expert N and boy R (Color figure online)

Full size image

4 Development of a Hand Contact Improvisation Interface

4.1 Design Plan

Hand contact improvisation, as a method for non-restrictive improvisational co-creative expression, is possible in a variety of positions - standing, lying down, or seated. Our goal was to develop a “hand contact improvisation interface” that supports 3-dimensional hand contact improvisation by encouraging bodily awareness in the participants. The design requirements to achieve this were as follows:

  1. 1.

    The use of objects no heavier than what two children can pick up and carry

  2. 2.

    A mechanism of engagement that does not interfere with expressive behavior

  3. 3.

    A framework for introducing incentives that encourage bodily awareness

  4. 4.

    The ability to collect data on motion and exchange of force

4.2 Interface Design and Development

In order to satisfy the above requirements, we developed an interface constructed with 3 types of blocks - a cylindrical block, a joint block, and a contact block shown in Fig. 7. We then developed the cylindrical block, which was made of acrylic pipe, with a variety of sensors. This enabled a force measurement via the film sensor or load cell and acceleration, gyroscopic and geomagnetic measurements via the 9-axis sensor. In addition, the cylindrical block was mounted with LEDs that changed color and intensity in correspondence with the collected data. By sending the data wirelessly to a PC, we could save the data we had acquired. The joint block, made with ABS, connected to the cylindrical block. For our experiment, we equipped one of these ABS joint blocks with a small speaker. In order to make the contact block - also made with ABS - easy for two people to support with their hands, we gave it a curved surface that fits easily into one’s palm.

Fig. 7.
figure 7

Overview of hand contact improvisation interface

Full size image

4.3 Application of the Interface to Autistic Settings

In order to determine whether or not our interface would be useful in encouraging 3-dimensional hand contact improvisation, we conducted experiments both with the “expert pair” and “novice pair”. Participants of expert pair have experience in the hand contact improvisation with autistic children at the workshops. On the other hand, Participants of novice pair don’t have such experience. Specifically, we held 1-min bodily expression trials in a 2.0 × 2.0 × 2.5 [m] space, both with and without the interface. Our results shown in Fig. 8(a) showed that the vertical range of movement increased when the novice pair used the interface compared to their movement range (of midpoint of the line formed between their hands) when using only their hands. These results indicate that the interface has the ability to naturally draw out bodily expressions from two novices, expanding the expressive space. In addition, in contrast to the novice pair, we observed that the expert pair produced compressive and tensile force when using the interface in contrast to the novice pair, as shown in Fig. 8(b). It is our impression that it may be possible to evaluate changes in relationship that accompany the deepening of the expressions based on these differences.

Fig. 8.
figure 8

Results of experiments using hand contact improvisation interfaces

Full size image

Next, we used this hand contact improvisation interfaces during the workshop that included children with developmental disabilities at the disaster-stricken areas. Here we observed that developmentally impaired children who showed an interest in the interface interact with one another by using it shown in Fig. 9(a). Use of the interface was not limited to the children, and we observed parents and facility staff members conducting hand contact improvisation using it in a proactive fashion as well. Surprisingly to the authors, we also observed severely autistic children who generally are unable to engage in workshop activities for an extended period of time participating in hand contact improvisation for a longer period - with a greater range of motion - by utilizing the sound-making interface as shown in Fig. 9(b). We received comments from parents and staff such as, “This enables movements that you normally can’t make in hand contact improvisation,” “Even though you are separated, I feel connected to you,” and “Based on the movement of the interface the sound changes, which was fun.” One of the expression experts told us, “I don’t just connect with my partner by dots or a line - there are many different faces by which I can connect with my partner.” These results show that our interface has the potential to be useful in facilitating hand contact improvisation with autistic children at workshops.

Fig. 9.
figure 9

Application of hand contact improvisation interfaces in the workshop

Full size image

5 Conclusion

Our study was aimed at the development of a bodily interface to facilitate co-creative expression inclusive of human diversity. Specifically, we developed a miniaturized tabletop version of our existing hand contact improvisation system, suitable for use in places such as child welfare institutions. In addition, we developed a hand contact improvisation interface capable of facilitating 3-dimensional bodily expression, and introduced it at a hand contact improvisation workshop in areas afflicted by the affected areas of the Great East Japan Earthquake. Via use of the interface, we observed voluntary engagement in hand contact improvisation from autistic children who tend to avoid a face-to-face contact, who freely utilized the expressive space to express themselves in three dimensions, for longer periods than previously observed. Furthermore, there appears to be potential to track hand contact improvisations between autistic children and the accompanying changes in their relationships via repeated use of the interface. The above demonstrates that hand contact improvisation interfaces such as these show promise as a new physical communication support technology that enables the establishment of sensitive connections with autistic children who struggle with linguistic exchange.