User Stress Measurement of Remote Operation Supporting System with Hand Gesture Transmission Function

Abstract

In this paper, we report the experimental results that try to measure user stress when the user communicates with a remote operation supporting system that applies a wearable camera and a wearable display. The remote operation supporting system refers to an audiovisual communication system that enables communication between a worker in real fields and a director in a remote site with sharing eyesight of the worker with wearable devices and mobile communication network. Various systems have been proposed such as early research by Kuzuoka [1] and a more recent project by Kasahara [2]. Some of these systems are commercialized, however, have not been widely used yet. Our developing system has a feature that enables experts (matured workers) send their hand movements and shapes to guide workers (non-matured workers) by using a gesture sensor in addition to conventional audiovisual communication systems. As the evaluation criteria of the effect of introducing worker support systems, the task completion time reduced by the system has been generally used. Reducing work completion time is an important measurement; however, that is not necessarily beneficial for the system users. Only applying this explicit measuring index could hinder finding further development of the system and popularizing of the system. In this report, we focus on stress as the evaluation measurement of the system users’ situation. We expect that the feature of our system that enables the expert’s natural hands movements as a directional method would reduce users’ stress. To verify the effect, we conducted experiments that measure users (experts and operators) physiological index data, i.e. the heart rate of users with wearable devices, while they are operating tasks using the proposing remote operating support system. Through that experiment, we report that physiological effect of hand gestures transmission function instead of task completion time. Moreover, we discuss the required functions or a development process to the remote operation supporting system to be used widely.

1 Introduction

These days, first-line workers or on-site practitioners who work in fields such as maintenance and manufacturing industries are facing problems for aging society including lack of labor force especially declining number of matured workers and difficulty of quality control of jobs. Because the number of matured workers is limited, the time required to arrange the matured worker for the job directly decides response time when sending the workers to the workplace are necessary. To shorten the response time, Non-mature workers would be sent to the field. That sometimes could sacrifices quality of works due to lack of experience of the workers.

At the same time, hardware improvement of wearable cameras and head-mounted displays and the development of a mobile communication network enable a remote operation supporting system to use in real fields, not in research laboratories.

In this paper, the remote operation supporting system refers to an audiovisual communication system that enables communication between a worker in real fields and a director in a remote site with sharing eyesight of the worker with wearable devices and mobile communication network.

Figure 1 shows the typical structure of the system. The worker(s) in the field wears a camera, a display, and an audio headset. The camera captures image and video of the environment where the worker is conducting their jobs. The image and the video are sent to the computer of the expert(s) who are staying in the dedicated office. Then experts superimposed guidance on the image or video then send back the guidance to the remote worker. The worker can watch the guidance on their wearing display. Of course, voice communication like that of mobile phones is also available throughout the operation.

Fig. 1.

Overview of remote operation supporting system.

These systems could solve the problem mentioned above. Matured workers (from now on, experts) would stay a dedicated center and can support non-matured workers (from now on, workers) work at the remote field with the system. That would save time for the expert to visit the field because they always stay at the center and workers who located on the closest site can be sent regardless of their experiences. Moreover, the system guarantees the quality of work by providing proper guidance and checks to the non-experienced worker from that center.

Many support systems have been proposed, and some of them are commercialized. They, however, have not been widely used yet.

Our developing system has a feature that enables experts to send their hand movements and shapes to guide workers by using a gesture sensor called Leap Motion in addition to conventional audiovisual communication systems [5]. That features provides merits including;

• 1. The experts can reflect their natural hand movements to directions for the operators.

• 2. The experts can send real-time continuous directions without interruption by taking a snapshot or a still image.

After we developed a conceptual model of the system, we iteratively have conducted trial tests with real field workers and have improved the system to reflect opinions from the users (User-Centered development). The examples of functions reflecting the users’ opinions include the system should be “easy to wear,” and the camera on the head-mounted display should be detachable to make the system useful for workers.

In general, the evaluation criteria of these systems focus on the task completion time reduced by using the system. Reducing work completion time is an important measurement; however, it is not necessarily beneficial for system users. Only applying these explicit measuring indexes might hinder finding further development of the system and popularizing of the system. So, in this paper, we focus on stress as the measurement of the system user’s situation.

Communication on the remote operation supporting system is different from face to face communication. Losing embodiments [9], losing expressions or directions made by users body which are available inface to face communication, limits directional methods and makes directions more difficult. Communication under the limitation of available directing methods is stressful. Both experts and operators might feel stress when they cannot communicate using media freely available in case of face to face communication.

We expect that the feature of our system that enables the expert to use natural hands movements as directional methods would reduce that stress. To verify the effect, we conduct experiments that measure users (experts and workers) physiological index data, i.e. the heart rate of users with wearable devices, while they are operating tasks using the remote operating support system. Through this experiment, we report that physiological effect of hand gestures transmission from different than task completion index. Moreover, we discuss the required development process to the remote operation supporting system to be used widely.

2 Related Research

Remote operation supporting system has a relatively long history, and various researches have been consecutively proposed since the 1990s. One of the pioneering researches “Shared-View System” has been proposed by Kuzuoka [1]. More recently, Kasahara et al. [2] proposed a “JackIn” system that redefined a remote operation supporting system from the concept of Human Augmentation approach. Ou et al. [3] showed that superimposing computer graphics lines on the captured image of remote site enabled a variety of guidance in the remote operation supporting system for a robot assembling task as an instance case.

Various remote operation supporting systems place importance on hands-free operation: enabling workers to receive or watch the visual information without holding a display by their hands and keeping their hands open and free for dedicating for their jobs. To provide this hands-free working environment to the worker, such system commonly uses head-mounted display (HMD); however, Kurata et al. [4] proposed the WACL system that applies shoulder worn display system to achieve the same goal.

Whereas, Microsoft developed a Hololens system that is all-in-one type Mixed Reality (MR) device: self-supporting operation is available without additional computing devices. Chen et al. [5] also provided a remote supporting system as a Skype application.

As we can see various systems have been proposed in accordance with the development of hardware, and some of these research results have been commercialized.

3 Feature of Our Proposing System

In this section, we describe the detail of our developing remote support operating system. Our system has a base on typical audiovisual communication. Using a WebRTC technology, peer to peer media (audio and video) data communication is enabled between browsers working on one site terminal and remote site wearable devices respectively.

3.1 Gesture Transmission Function

Adding to that basic function, our system has a gesture transmission function. The function enables the remote experts to send hand shapes and movements information that he/she behaves in front of a terminal of remove site using a gesture sensor. Present system applies a gesture sensor called Leap Motion. The gesture sensor captures the shapes and movements of the expert’s hand and sends captured data over the WebRTC data transmission.

We expect our feature: gesture sending function could provide two additional merits to the system such as;

• 1. The experts can reflect their natural hand movements to a direction for the operators. Pointing important part on the image as if the expert were pointing the real object in the actual field, Sending hand shapes to tell how to grab a target object are example usage of hand gesture.

• 2. The experts can send real-time directions without interruption by taking a snapshot. Taking snapshots and drawing directions on them are widely used directing method in many remote support systems [5].

Figure 2 shows an overview of our proposed system. The gesture recognition device is added on the expert’s side, and synthesized computer graphics hand images are superimposed on the captured image of the worker wearing a camera. Fig. 3 shows actual image of an expert’s side setup (left) and captured superimposed image (right).

Fig. 2.

Overview of our proposed system.

Fig. 3.

Setup of the expert site (left), an image superimposed on the remote workers’ eye-sight (right).

3.2 Other Functions

Like as the prior proposed system [3], our system also provides a drawing transmission function. The expert can draw lines on the captured image with the touch screen, and the worker can watch that image simultaneously on his wearable display. The expert can record and store what he draws and resend them as guidance. These data transmissions are also over the WebRTC [6].

4 User-Centered Development

After we have developed our primary concept model system, we iteratively improved the system iteratively under the principle of User-Centered design; we have been working with on-site workers and ask them to use our system and reflect their compliments and opinions to improve the system.

Figure 4 shows an early stage of the prototype of our proposed system. At this stage, a wearable display and a wearable camera were not light enough (approx. 160 g), and the worker must wear the helmet to sustain the weight of the wearable device (Fig. 4 left). Moreover, the required computational performance and a form factor of the notebook computer for the worker side forced the worker to wear the backpack contained notebook computer (Fig. 4 right).

Fig. 4.

Worker side appliances (Early stage of development).

We believe that the fact is worthy of mention that we were able to confirm the effectiveness of the system worked even at this very early stage; Through the experiment, we could observe that the inexperienced workers could complete their tasks (assembling the electronics system and test of the system) under the guidance of the remote expert. Through the experiments with this prototype, requirements for introducing our proposing system into the real field were clarified as the areas for improvement including;

• 1. The wearable device of the system for the worker should be “easy to wear.”

• 2. The wearable camera should be “detachable.”

4.1 Ease of Wearing

The reasons ‘“ease of wearing” of the system matters’ include the following points;

• A remote operating support system will be used for quickening response to the problem occurred, and therefore, if the device system is too bulky and difficult to put on by workers themselves, the system cannot satisfy the basic needs.

• If the device is difficult to wear, the psychological barrier to use that system hinder workers/users to use the system, and they could choose a conventional method of communication such as mobile phones.

Recently, hardware advancement has been faster, and through the developing process, we were able to replace the hardware components to better and lighter ones. The wearable display and camera became smaller and lighter, then wearing helmet became unnecessary. Now the worker wears the headband type wearable display and camera on his/her work cap. Figure 5 left shows the appearance of the present system. The display is AirScouter MD-300A by Brother, Inc. and the camera is HX-A1H by Panasonic, Inc. Now the system utilizes a small computer dynaedge DE-100, by Toshiba, Inc., the whole system became so small that can be stored in a small fanny pack (Fig. 5 right).

Fig. 5.

Worker side appliances (Present system).

With the improvements of the system mentioned above, the present system requires the users to take just two actionsto use the system; wearing the display and the camera over the cap and fastening the belt of the pack around their waist.

4.2 Detachable Camera System

The “detachable” camera system is a mechanism that is for a wearable camera fixed on a headband should be used as a temporary hand carry camera. Figure 6 shows that image. Workers can slide the camera forward and release the camera from the attachment of the headband. The attachment part has a magnet inside and holds parts when workers put the camera part back to the attachment part.

Fig. 6.

Detachable wearable camera system.

This function is required in such cases when the worker works in a dedicated server room and maintains network facilities. In such cases, more than one similar facility: possible working targets for the worker should exist in the same room. The workers, therefore, sometimes has to capture a detailed image of labels that show machine’s names or ID’s to ask the remote expert confirm the workers is working on a proper facility. The labels could be put on the inner part of the machine the other side of the ceiling of that. If the wearable camera is fixed on the head part, the worker has to crouch, bend down deeply and poke his/her head into the narrow place. These movements or postures could be stressful for the worker. With the detachable camera, the worker, instead, detach and hold the camera and poke it to take a closer look at the labels.

Digital or analog zooming might be another possible solution to the problem, however, to apply that solution, we have to develop another user interface for controlling zooming of the fixed wearable camera and ask the worker to get used to it. We think the “detachable” mechanism is a reasonable solution in these points.

• 1. That is easy to understand and easy to control the captured image.

• 2. The worker is not required to train themselves for a new interface.

We have been reporting the effectiveness of the proposed system for toy problems in the laboratory and trial in the field of our company’s factory [7].

5 User-Centered System Evaluation

Evaluation criteria of proposed remote operation supporting systems generally focus on efficiency: the task completion time reduced by using the system [8]. Reducing work completion time is an important measurement; however, that is not necessarily beneficial for the system users: both workers and experts in this case. Only applying that explicit measuring index might hinder finding further improvement or development of the system. Also, we think that the lack of this kind of perspective might be a one of the reasons why this remote operation supporting system not widely accepted in the real fields. In this paper, as a first step for setting up new “user-centered” criteria to understand the user, we focus on stress as a measurement of the system users’ situation.

Communication over the remote operation supporting system is different from face to face communication (from now on, F2F) in various aspects. Losing embodiments [9], losing expressions or directions made by users body parts and their movements available in case of F2F (such as pointing by the finger and showing proper hand shape and movements how to grab the handle of the machine) restrict directional methods and make the guidance more difficult. Communication under these limitations could be stressful. Both workers and experts feel stress when they cannot communicate using media freely available in case of F2F.

We expect that the main feature of our system that enables the expert’s natural hands movements as a directional method could reduce that stress, because, at least the expert can behave and communicate similarly as in the F2F situation.

To verify the effect, we conduct experiments that measure users (experts and workers) physiological index data, concretely, the heart rate of users with wearable devices while they are operating tasks over communication with the proposed remote operation support system. Through this experiment, we report that physiological influence of hand gestures transmission function instead of task completion index.

5.1 Stress Measurement with Bio-Medical Measurement

Stress measurement and stress evaluation with biomedical measurement has widely been studied, and various are practically and clinically used. Measuring heart ratio, electrodermal activity (EDA), and skin temperature are examples that require wearable or fixed monitors. Besides, measuring eye movement with a particular eye tracking camera and collecting users’ saliva to check the amount of cortisol are also used.

We applied the heart rate monitor for our research because we find that method is well balanced between ease of measurement and validity and it can obtain continuous data. We also highly evaluated that because that method provides a lower load during the work operation.

To estimate user stress from the heart rate signal, the LF (Low Frequency)/HF (High Frequency) ratio is commonly used after Pagani’s research [11]. The LF is power spectral density of the low frequency bands and the HF is that of high frequency bands. The LF suggests activities the sympathetic nerve systems and the HF reflects that of the parasympathetic nerve system. Therefore, the value ratio is commonly used to estimate the extent’s of user stress. We applied that LF ratio as a measurement of the experiment.

To measure heart rate, we adopted a chest-worn heart rate monitor (my Beat WHS-1 and chest strap with electrodes by Union tool Corp. Figure 7 shows the appearance.). The monitor works on a coin cell (CR2032) and can store the measured data on an inner memory.

Fig. 7.

Chest strap type heart rate monitor

RRI (R-R interval) of workers and an expert were measured. The sampling rate for Electrocardiogram (ECG) measurement was 1000 Hz. After measurement, data was processed for further analysis. Autocorrelations were used to obtain frequency components and squared power of each frequency was obtained. Moreover, the LF/HF ratios were calculated. In the experiments, LF was defined as ranges from 0.04 Hz to 0.02 Hz and HF from 0.15 to 0.4 Hz. These parameters are commonly used after the research by Malki et al. [10].

6 Experiment

The experiment reported in this paper is a preliminary study for a small group. The number of experiment participants is three. Two are for workers, and one is for the expert. All of them are male and the ages of them range from their forties to fifties.

The expert and the worker are arranged on different rooms (Fig. 8 shows the environment of the experiment). Under the guidance of the expert, the worker assembles a communication apparatus. The worker wears the wearable devices, and audio communication headset and all the communication between the worker and the expert is conducted over the proposed remote support system.

Fig. 8.

Environment of experiment.

In addition to that, both the expert and the worker wear a heart rate monitor and heart rate is recorded throughout the operation (providing guidance and assembling an apparatus respectively).

Video images are also recorded for further analysis. The expert stands still and does not walk around; therefore, two fixed cameras on tripods are used. Whereas the worker walks around the space to conduct a task; therefore, one camera is fixed, and an experimenter holds one portable camera, and behavior of the worker are captured.

The worker’s target objecti.e., the communication apparatus is a kind of shelf in the form of a cuboid, and the size is approximately 1500*800*600 mm. Electronics circuit boards (from now on, ECBs) that should be inserted to that shelf are packed inside cardboard boxes and are arranged nearby thatapparatus.

The worker follows guidance by the experts and conduct process following,

• 1. Move around and find the cardboard box contains required ECB.

• 2. Open the box with a cutter.

• 3. Unwrap the ECB.

• 4. Confirm the ECB is the required one.

• 5. Insert the ECB to the proper position in the right direction.

These processes repeated several times until its completion. One set of experiment is a work for one shelf each. Two sets of the experiment are conducted by each worker and expert pair. One experiment is conducted on the remote operation supporting system with gesture transmission function, and the other is conducted without that function. In each experimental set, the order of the function’s availability has changed. One set of experiments requires about 20 min. In total 40 min are required for one participant’s group (an expert and a worker).

We analyze the psychological effect of the difference in available communication media. We also do a brief structured interview and ask the participants to answer the 5 point Likert scales questionnaires.

7 Results and Discussion

Because the experiment is a preliminary study, we have not applied detailed statistical analysis for the data. Instead, we observe the LF Ratio data. Figure 9 shows the graphs of the experimental result on. Upper charts show the results of first group. The left is the first trial’s result. The right isthe second. The gesture transmission is available on the second trial (right one). Lower charts show result of the second group (the same expert and the other worker). The gesture is available on the first (left).

Fig. 9.

LF Ratio graphs of participants.

In each graph, vertical axis shows percentage of the LF ratio over the HF. That shows stress level of the users. Horizontal axis shows time and note that that does not apply same scale for each case.

The expert and the first worker’s result did not show clear differences between conditions. The second worker’s stress levels, however, are higher than the first worker we can observe from the chart. Also, his mean LF ratio increased for the second trial: the case that gesture transmission is not available. The worker B’s Mean LF ratio for the first trial was 83.60% (S.D. was 16.44) and that for the second trial was 90.29% (S.D. was 7.03).

Through the whole experiments the expert seemed to be relaxed and his answer for the questionaries’ and the interview result showed the consistent tendency. Availability of situation awareness that means the user of the system can obtain the environmental information of the remote place and contextual information of the user, the expert might feel less stress even with very basic function of the remote operation supporting system.

Second worker’s LF ratio result suggests the hypothesis that our proposed method might influence the stress of the users might be supported. We still cannot negate the order effect because the number of experience is limited. Further analysis and investigation are required.

8 Conclusion

In this paper, we introduced our developed system and its features that user can send his hand shape and movements. Also, we conducted the experiment that seek the possibilities of the feature can alleviate stress of remote operations supporting system. The experiment was preliminary and the result still did not clearly support our expectation. We believe, however, human-centered evaluation and human-centered improvement of the system are essential. We will continue iterative developing process with actual users of the system. We also try to find a suitable index that can properly evaluate authentic user-centered remote operation supporting system. That evaluation index could help to find a new function that support workers and experts collaboration.