Keywords

1 Introduction

With the development of technology and science, wearable devices such as smart watch and smart bracelet become more popular. And the increasing use of wearable devices has promoted interest in gesture input techniques for interaction. Hand gesture as one of the contact-free input techniques allows users interact with systems more naturally compared with other traditional contact input techniques such as keyboard and mouse. However, hand gestures also cause obvious fatigue problem during operating device due to the diversity and large movements range of hand gestures. Therefore, it’s very important to evaluate and measure the fatigue of hand gesture interaction, which can provide a reasonable basis for gesture interaction design.

There are mainly five methods to evaluate the fatigue degree [1,2,3,4,5]. The first method is the subjective evaluation method which is using users’ sense of self to evaluate the fatigue degree. The second method is that the biomechanical evaluation method which is conducting biomechanical analysis of joints according to the force constrains of operations. The third is the physiological signal evaluation method which is analyzing the fatigue state by measuring ECG, EEG and surface EMG (Surface electromyography, sEMG) and other physiological signals. The forth is the energy metabolism evaluation method which is evaluating the functions of a human body by measuring the oxygen consumption, respiratory rate and other biochemical indexes of the operator. The fifth is the transmitter fatigue evaluation method of muscle tissue which is evaluating human muscle fatigue degree by measuring the density of neurotransmitters.

Surface electromyography (sEMG) is a non-invasive and simple method to record the biological electric signals during neuromuscular activities by electrodes attached to human skeletal muscle surface. Surface electromyography(sEMG) contains a great deal of information of status of neuromuscular contraction function which can quantitatively reflect the degree of fatigue muscle activity. It has high academic value in the field of human engineering. Brog [7] subjective evaluation is generally acknowledged to medical community that can reflect human fatigue condition by reflecting heart rate condition to a large extent. Ge Shuwang et al. [8] conducted the research about right arm in the static posture and found out that sEMG and Borg scores showed a significant correlation. Song Haiyan et al. [9] studied the characteristics of surface electromyography of human upper limb muscles in daily life. Wang Lejun et al. [10] conducted the research about the fatigue of finger extensor caused by clicking mouse quickly by monitoring sEMG. Liu Chang et al. [11] studied the users’ fatigue of non-contact human-computer interactive gestures of large screen device. Due to the differences such as hardware equipment, usage, usage scenarios, recognition patterns, the gesture interaction of wearable devices is very different with other products, yet there is no research on the user fatigue of gesture interaction of wearable devices (Fig. 1).

Fig. 1.
figure 1

The SEMG signals before, during and after an action.

Full size image

By comparing the difference of these five methods, we decided to combine surface electromyography (sEGM) and the subjective fatigue evaluation method (Brog) to reflect the fatigue condition of left arm after performing a basic interactive gesture of wearable devices. Our experiments analyzed the characteristics of gesture interaction of wearable device by combining the subjective and objective data, and established the fatigue evaluation model, which can provide the basis and reference for analyzing and monitoring the operating load of gesture interaction of wearable device, designing interactive gestures and planning tasks.

2 Method

2.1 Experimental Subject

We selected 10 participated (5 males and 5 females, the average age is 23.3) who were in good condition and didn’t perform any strenuous physical activities in 24 h before the experiment. They were all at good mental state and they weren’t in any kinds of bad conditions like lack of sleep or listlessness. And they all worn watches on left wrest in daily life. Before the experiment, all the subjects were familiar with the experimental procedures and agreed to participate in the experiment voluntarily.

2.2 Experimental Equipment

Ticwatch smart watch was worn by the participants to simulate the use of smart watches.16 guide wireless physiological recorder was used for collecting surface EMG. Notebook computer was used to connect the wireless physiological recorder which can record test data. This experiment was carried out in the usability laboratory of Beijing University of Posts and Telecommunications, and the laboratory temperature was tested under the condition of 20–26.

2.3 Experimental Method

  1. 1.

    Selecting gestures. We selected 15 basic interactive gestures through the smart watch gestures intuitive research [12]. And we gave each gesture a unified number, specification and description, as shown in Fig. 2. And we classified these gestures into 9 categories and analyzed the main objects of the actions, as shown in Table 1.

    Fig. 2.
    figure 2

    Basic interactive hand gestures

    Full size image
    Table 1. Gesture classification
    Full size table
  2. 2.

    Training participates. Before the experiment, in order to eliminate tension, we ensured every participates to know the purpose, the process and the attentions of the experiment. And every participate was asked to learn and become familiar with the 15 basic gestures.

  3. 3.

    Placing electrodes. Before placing the electrodes, we cleaned the surface of the subjects’ skin with 75% alcohol cotton ball to reduce impedance. The surface electrodes were attached to the left arm finger extensor (M1), biceps (M2), middle deltoid (M3). And these three muscles had the most direct correlation and monitored the EMG status of the shoulder and the upper arm, the elbow and the wrist muscles. Two of the three recording electrodes were attached along the longitudinal axis of the muscle fibers, and another reference electrode formed triangle with the two electrodes, and the distance between the electrodes is 2–3 cm.

  4. 4.

    Conducting experiment. During the experiment, participants were asked to choose a comfortable sitting position in a resting for state 3–5 min and we collected the EMG signal of the three muscles in the state of rest for 1–2 min. At the beginning of each test, participates were asked to perform the same gesture 10 times in a normal speed. We recorded the sEMG signals in the implementation process. And the Brog 10 level subjective fatigue table was hanging in the front of participates during the experiment. And participates were asked to report the score of the fatigue after performing a gesture 10 times. The interval between each gesture was 30 s (Table 2).

    Table 2. Borg subjective fatigue scale
    Full size table

3 Discussions

We collected 450 groups sEMG signals (4500 gestures) and 150 Brog user subjective fatigue scores in total. And the data were normalized by ErogLAB to exclusive individual differences of participants. The physiological signals (sEMG) is regarded as a function of time in time domain analysis. The statistical characteristics of physiological signals are obtained by the analysis, as shown in Table 3. And we use the average electromyography (AEMG) to indicate the energy consumption of muscle actions by reflecting the average level of the amplitude of physiological signals. The bigger the AEMG value is, the more muscles are involved, and the more energy is spent; the smaller the AEMG value is, the fewer muscles are involved, and the less energy is spent.

Table 3. AEMG and Borg score of the 15 gestures
Full size table

3.1 AEMG Value Analysis of the Basic Gestures

We got the mean of EMG and the Borg score of each gestures by SPSS, while we summed the AEMG value of the three muscles of each gesture up, as shown in Table 4.

Table 4. Mean of AEMG and mean of Borg score
Full size table

The data in Table 4 showed that five of the gestures have the highest figure extensor fatigue degree, which are flipping arm, turning the wrist up, raising and putting down arm, turning the wrist down, and swinging arm up. Five of the gestures have the lowest figure extensor fatigue degree, which are moving arm up 10 cm, moving arm down 10 cm, moving arm to the right, tilting 45 degrees to the left, and moving arm to the left. Five of the gestures have the highest biceps fatigue degree, which are flipping arm, swinging arm up, moving arm forward 10 cm, moving arm down 10 cm, shaking arm up and down two times. Five of the gestures have the lowest biceps fatigue degree, which are moving arm up 10 cm, turning the wrist up, moving arm to the left 10 cm, turning the wrist down, and making a fist. Five of the gestures have the highest middle deltoid fatigue degree, which are raising and putting down arm, swinging arm up, shaking arm up and down two times, moving arm to the right 10 cm, and flipping arm. Five of the gestures have the lowest middle deltoid fatigue degree, which are moving arm up 10 cm, moving arm down 10 cm, turning the wrist down, turning the wrist up, and making a fist. So we found that two gestures have the relatively high fatigue degree of all three muscles, which are flipping arm and swinging arm up.

The AEMG value of figure extensor, biceps and middle deltoid represent the left forearm, arm and shoulder muscles fatigue status, therefore we sum the AEMG value of three muscles up to evaluate the fatigue situation of human upper limb. And we found that three gestures have significantly higher fatigue degree than other gestures, which are flipping arm, raising and putting up arm, and swinging arm up. So we should avoid the frequent use of these three gestures in interaction design and task planning. And we found that three gestures have relatively lower fatigue degree than other gestures, which are turning the wrist down, moving arm to the left, and making a fist. So these three gestures have a lower fatigue degree and a better experience.

3.2 Gesture Types and Fatigue Analysis

We analyzed the 15 basic gestures according to the types of gestures (Fig. 3). And we found that three types of gestures have high fatigue degree because of the large action range and the number of involved muscle groups, which are flipping army, raising and putting down arm, and swinging arm up. And the shaking gesture has a small action range but have a high frequency. So the fatigue degree of this gesture is also in the upper level. There are four types of gestures have relatively low and very close fatigue degree due to small action range and mainly using the biceps, which are moving horizontally, moving vertically, tilting and turning. Making a fist has significantly lower fatigue degree than others because of the mainly using the figure extensor muscles. Combined with the operation objects of different types of gestures in Table 1, we can see that the gestures the operation objects of which are physical device (raising and putting down, shaking) and structural level (flipping) have higher degree of fatigue. Thus, it should be considered to use a flat information architecture to reduce the operation of the physical device and of the structure level in the process of wearable device task planning and interaction designing. And place more emphasis about the user experience on the operation of contents of the interface information.

Fig. 3.
figure 3

AEMG value of gesture types in descending order

Full size image

3.3 Gesture Direction and Fatigue Analysis

There are 4 kinds of basic gestures which are tilting, turning, moving horizontally, and moving vertically. Subdivision gestures of these types are the gestures moving in the same mode but in the symmetrical direction. From the Table 4, we found that there is a difference in the energy consumption between the same type gestures in different directions because of the asymmetry of body movements. In the Fig. 4, for the human left arm, turning down the wrist is superior to turning up(P = 0.00); tilting to the left is superior to tilting to the right(P = 0.29); moving up is superior to moving down(P = 0.14); moving to the left is superior to moving to the right(P = 0.25); and moving backward is superior to moving forward(P = 0.28). In the design process of gesture interaction, it should be considered that choosing a more appropriate direction can reduce users’ fatigue degree. For example, we can select turning the wrist down and tilting to the lift as the gestures which can operate the content lists.

Fig. 4.
figure 4

Gesture direction recommendation

Full size image

3.4 Gesture Fatigue Model

It can be seen from Table 4 that with the increase of the fatigue degree, the corresponding Borg value of the fatigue degree is also increased, and it can be speculated that there may be a functional relationship between them. The AEMG value and Borg value of G1(lifting and putting down left arm) have a big difference because that this gesture is one of the daily arm actions which might have a influence on the Borg score. So we excluded the data of G1 as singular data. We used fitting method to process data. And the correlation between the AEMG value and the Borg value is mainly dependent on the correlation coefficient of the results. The Borg score is taken as the independent variable x, and the AEMG summation value of each gesture is taken as the dependent variable y. And we use the quadratic regression model, cubic regression model and exponential regression model to fit the data in the Table 3. And according to the comparison, it is best to use the cubic regression model to fit the data, and the fitting model is Formula 1.

$$ y = 0.1693x^{3} - 1.162x^{2} + 2.6387x - 1.6859. $$
(1)

The Correlation coefficient R2 is 0.7945, which are shown in Fig. 5. And it is generally believed that the model can be considered as a fatigue evaluation model when the R is 0.85.

Fig. 5.
figure 5

User fatigue evaluation model of gesture interaction

Full size image

In this model, the subjective fatigue degree and surface EMG are mapped. Through the model, we can know that the relationship between user’s subjective fatigue and muscle energy consumption is nonlinear. The user’s subjective fatigue degree score ranged from 1.5 to 3.8. With x = 3 as the critical point, when the energy consumption is less, the user’s subjective fatigue sensitivity is higher; when the energy consumption is more, the user’s subjective fatigue sensitivity decreases. We can observe the variation of the user’s fatigue directly and evaluate the user’s fatigue state by this model.

4 Conclusions

This experiment was designed to study the user fatigue of wearable devices what are based on hand gestures interaction such as smart watch. In this study, the sEMG surface electromyography and Borg subjective fatigue assessment were used to measure and evaluate the user’s fatigue degree of gestures.

By comparing the AEMG value of basic gestures, we can acknowledge the energy consumption characteristics of different muscle groups and of each gesture. By analyzing the relationship between the gesture types and the objects they operate, it can be known that gestures for operating equipment and information hierarchy consume more energy than gestures for operating contents. Thus, it should be considered to use a flat information architecture to reduce the operation of the physical device and of the structure level in the process of wearable device task planning and interaction designing. And place more emphasis about the user experience on the operation of contents of the interface information. And via the cross analysis of movement direction and fatigue of gestures, we find that there is a difference in the energy consumption between the same type gestures in different directions because of the asymmetry of body movements. Therefore, in order to reduce user fatigue during the usage of device, gesture direction should be taken into consideration when designers select interactive methods for devices. All these analyses provide references for wearable devices task-planning and gesture design.

This study also establishes a user fatigue evaluation model of gesture interaction of wearable devices, which can reflect the relationship between objective energy consumption and subjective fatigue. In the practical application, we can use the fatigue evaluation model to predict the change of surface electromyography (sEMG) by obtaining the subjective fatigue evaluation of users, providing intuitive technical method for hand gesture fatigue monitoring.