Keywords

1 Introduction

A future vision of the use of autonomous and intelligent robots in dismounted military operations has Soldiers interacting with robots as teammates, with an interim goal of having the robot able to execute tactics much like a military working dog or robotic wingman [2,3,4], with increasing levels of autonomy—and corresponding issues [5]. Soldiers would no longer have to continuously tele-operate every movement of the robot. Instead, Soldier–robot interactions would be more tactical, bidirectional, and naturalistic [6]. Gesture-based commands to robots are one such means to more naturalistic control, and have been used in a variety of different settings, such as assisting users with special needs [7], assisting in grocery stores [8], and home assistance [9]. Examples of gestural commands in these settings include “Follow me”, “Go there”, or “Hand me that”.

In this report, we describe progress regarding use of an instrumented glove to detect and transmit hand and arm signals to maneuver and manipulate a small ground robot and robotic arm, without the use of speech or visual interface icons. This research effort was performed to examine the concept of instrumented gloves as a means for gesture based HRI control. Instrumented gloves were used to investigate aspects of gesture-based controls compared to a handheld controller. The system was developed through an Army Small Business Innovative Research (SBIR) program, led by Army Research Laboratory Human Research and Engineering Directorate in collaboration with AnthroTronix, Inc (ATinc), building upon previous efforts regarding wearable computers and robotic platforms [1, 10].

1.1 Purpose

The current effort seeks to investigate an advanced concept in intuitive interfaces to reduce cognitive, physical, and temporal demands and enhance robot control. Instrumented gloves were adapted to aid in robot control for driving and robotic arm manipulation. In this experiment, soldiers used the instrumented glove or a handheld controller to navigate around obstacles and manipulate the robotic arm.

2 Robot Control

2.1 Gestures for Robot Control

Instrumented gloves are the most common instantiation of wearable, instrumented systems for robot control [11]. The glove concept is congruent for many work situations where operators may already have to wear gloves. For robot control, glove-based approaches are usually stand-alone, with the glove sending signals to robotic intelligence software for recognition, interpretation, and translation into computationally understandable and executable robotic behaviors.

Gesture recognition is accomplished through the mathematical interpretation of human body movements. Hand and body gestures can be transmitted from a controller mechanism that contains inertial measurement unit (IMU) sensors to sense rotation and acceleration of movement, or in other instances via camera vision-based technologies. Inertial measurement unit (IMU) sensor technologies placed on the body provide an alternative, technically-feasible, near-term approach to gesture recognition within uncontrolled environments. ATinc has demonstrated IMU-based hand and arm signal gesture recognition accuracy of 100% [10] via a custom instrumented glove interface. They have previously integrated with unmanned ground vehicles (UGVs) to demonstrate intuitive forms of control and communication.

2.2 Robot Control

Robot control is traditionally accomplished using handheld controllers, much like a gamepad or joystick form factor. Use of instrumented gloves to accomplish simple movement commands have been demonstrated across a number of situations [11]. There are several examples of using gestures for control of a robot [12, 13]. A strong advantage to a multi-use instrumented glove to a dismount Soldier is that sensors can be embedded within a standard Army field glove normally worn by Soldiers, thus eliminating the need to carry a handheld controller, and allowing easier access to their weapon.

While it is easy to think of single commands (e.g., stop, move forward, turn left) as simple commands, one should keep in mind it is not the command per se, but the distinguishability and the intuitive nature of the gesture that determines ease of use and recognition. When the gesture set is small, recognition rates have been high, across many glove-based approaches [11].

2.3 Remote Manipulation

Ground-based mobile robots are often used for remote manipulation of objects. In combat situations, this capability is often used for explosive ordinance disposal (EOD) [14]. Several efforts have been reported where gestures have been developed for remote manipulation. Several of these regard the development of service robots designed to assist people in locations such as offices, supermarkets, hospitals, and households. Other efforts focus on assisting users in more dangerous environments such as hazardous areas or space, using telepresence and teleoperation (see [15] for a review of teleoperation issues).

3 Equipment

3.1 Instrumented Glove for Robot Control

Participants used a single instrumented glove for robotic maneuvering and manipulation developed by ATinc, called the NuGlove. The single-glove configuration allowed for the switching between robot driving and robotic arm manipulation. The glove contained ten 9-axis sensors (3-axis accelerometer, 3-axis gyroscope, and 3-axis magnetometer). Data from the glove sampled at a rate of 100 Hz. The glove was tethered to a smartphone, which was used to transmit the wireless command signal (Fig. 1).

Fig. 1.
figure 1

AnthroTronix NuGlove

Full size image

In addition to the instrumented glove, the robot was controlled via a traditional gamepad controller commonly used for robotic control and gaming. This gamepad, the Xbox 360 controller (shown in Fig. 2), is familiar to most video gamers and was integrated with the handheld computer via wireless protocols.

Fig. 2.
figure 2

X-box controller

Full size image

3.2 Robot

The robot used for this evaluation was a Jaguar V2 Robot implemented with a three degree of freedom (DOF) manipulator arm (shown in Fig. 3), which is a commercially available off-the-shelf mobile robotic platform. It is rugged, lightweight (<25 kg), and compact, as well as weather and water resistant. It has a chassis with two flippers for completing mobility tasks and had a commercial-off-the-shelf (COTS) manipulator arm mounted to the upper chassis housing. A felt tip marker was attached to the end of the manipulator arm to enable accuracy measurements for the dexterous task of touching a paper target.

Fig. 3.
figure 3

Robotic platform & manipulator arm

Full size image

4 Experimental Methods

4.1 Participants

Twenty-four Soldiers participated in this study. They were recruited from the Officer Candidate School at Fort Benning, GA. All participants had a BS degree or higher—two had PhDs. Age ranged from 22 to 32 (average = 26.04). Twelve were female. Three participants were left-handed. Uniform size ranged from XS to L.

4.2 Robot Control Procedures

Soldier-participants were briefed on the purpose of the robot control experiment. They were told they would be trained on two controllers (i.e., gamepad, instrumented glove). After two training runs, each Soldier accomplished robot navigation and manipulation twice, once with each controller. Performance data was collected through trained observers. After each performance session, each Soldier filled out a NASA TLX self-report of workload. After both performance sessions were complete, they filled out a questionnaire pertaining to each controller.

4.3 Robot Control Training

Participants were trained on the different controllers prior to completing the task. The trainers described the general task demands throughout the robot control course, and explained that the goal of the task was to tele-operate the robot through the course while avoiding all obstacles and staying within the barriers. Participants were then shown the robot that they would be operating, including the chassis and three degree of freedom arm. They were told to navigate using only the camera for visual feedback, and for the task they must drive the chassis through the course and touch the target using the arm. They were told that they would be using two different control methods to operate the robot.

For the handheld controller condition, participants were shown the layout of the joystick and button controls. They were told to regard the arm as they would a human finger, given that it had the same number of joints and segments. They were shown how to use the handheld controller buttons to move the joints of the arm. They were also shown how to control robot maneuvers and movements, via the camera feedback. Participants then completed a test run once all questions of theirs were addressed.

For the glove condition, participants were shown the Android interface that was used in conjunction with the glove to operate the robot. The app interface consisted of “Drive”, “Arm”, and “Lock” buttons. This was used to toggle between driving (Drive) mode and manipulation (Arm) mode, as well as the option to completely stop operation of the robot (Lock). As in the controller condition, participants were encouraged to regard the manipulator arm as they would the human finger. Movement of the arm mapped directly to the index finger on the glove. Participants were then shown how to use the glove to control the chassis drive. They were asked if they had any questions and asked to complete a test run.

4.4 Robot Control Route and Task Demands

There were two options for robot control setup: single-glove control in which control was switched from the chassis to the manipulator arm and gamepad control in which one joystick controlled the chassis and the button pad was used to control the manipulator arm. A marker was attached to the end of the robotic arm to indicate where the participant planted the target. A camera was attached to the robot chassis for video feedback during teleoperation.

Obstacle locations were systematically varied for the three performance conditions (e.g., training, glove, gamepad) to minimize practice effects. To begin each trial, the robot is placed at the start point. The operator maneuvered the robot along the path, taking care to avoid obstacles and stay within line boundaries. At the end of the route, they deployed the manipulator arm, and made contact with a target on the door. The target was clearly visible via the robot’s camera. Figure 4 shows the robot system along with the simulated path. Soldiers were given 25 min to complete the building-clearing task. Each condition took approximately 1 h to train, perform and provide feedback.

Fig. 4.
figure 4

Robotic platform & manipulator arm & course

Full size image

4.5 Robot Control Performance Measures

For each of the conditions, drive time was collected as the total recorded time to complete navigation of the robot from the starting position to the intended target. This only included the task time to drive the robot chassis, and not to manipulate the robot arm. Touch time was also recorded and was the total time recorded to complete the manipulation portion of the task. It was the total time that participants spend within the manipulation mode to manipulate the robot arm to the placed target. Distance in inches of the final mark made by the operator from the intended target was noted, as well as the number of times the robot hit or crossed one of three aspects of the course: boundary lines, boundary posts (a table), or a simulated IED obstacle.

Mechanical Failures

Due to power draw issues, the unmanned ground vehicle used for the experiment sporadically dropped wireless connectivity. During one of the participants’ runs, the robot collided with one of the barriers, causing a gear to snap. The motor was switched out for a spare motor. These issues delayed a few experimental runs, but were quickly resolved.

4.6 Robot Control Subjective Measures (Workload and Feedback)

Subjective measures included:

NASA TLX - the NASA TLX is a multi-dimensional rating scale for operators to report their mental workload. It uses six dimensions of workload to provide diagnostic information about the nature and relative contribution of each dimension in influencing overall operator workload. Operators rate the contribution made by each of six dimensions of workload to identify the intensity of the perceived workload [16]. Unweighted scores for each dimension were used in analyses.

Robot Control Questionnaire -

participants were asked to provide open feedback on the following aspects of the overall system and experiment.

  • Ease of training of the two controllers

  • Comfort/fit of the glove

  • Any problems experienced with the glove

  • Control scheme of the gamepad controller

  • Any problems experienced with the controller

  • Which controller was preferred

  • Overall glove controller concept

  • Ways to improve the glove system

5 Results

5.1 Touch Time

Touch time (seconds) was recorded as the amount of time spent by the operator manipulating the robotic arm to ‘touch’ the intended target at the end of the pathway. Paired-comparison t-test of this difference was significant (t = 2.394, df = 36, p = 0.022). Figure 5 and Table 1 show the results for touch time (Fig. 7).

Fig. 5.
figure 5

Mean touch time (seconds)

Full size image
Table 1. Touch time for both conditions
Full size table

5.2 Drive Time

The amount of time, in seconds, spent by the operator driving the robot before switching to manipulator mode was recorded during each run. Drive times for the Xbox and glove control conditions are indicated in Figs. 6, 7 and Table 2. Paired-comparison t-test of this difference was significant (t = − 3.14, df = 36, p = 0.003).

Fig. 6.
figure 6

Drive times (seconds) by controller type across subjects

Full size image
Fig. 7.
figure 7

Graph of mean drive time (seconds)

Full size image
Table 2. Drive time for both conditions
Full size table

5.3 Distance

Difference in inches of the mark made by the operator via the robotic arm to the intended target was recorded. Paired-comparison t-test of this difference was significant (t = − 4.035, df = 37, p = 0.000). Figure 8 and Table 3 shows the results for distance.

Fig. 8.
figure 8

Mean distance

Full size image
Table 3. Distance for both conditions
Full size table

Other measures collected related to the number of times the robot hit either the barriers or obstacles of the course. However, the difference in means of the two conditions was not statistically significant.

5.4 Subjective Feedback

After completion of the robotic control task, participants were asked to provide feedback of the training, glove, gamepad controller, system preference, glove controller as a concept, and suggestions to improve the system. Overall, the gamepad controller had greater positive feedback, mostly due to familiarity with the system in commercial applications (i.e. home video-gaming). However, the glove was viewed as an intuitive interface for maneuver, but less for arm manipulation. Negative feedback relating to either system corresponded mostly to how the controller (glove or gamepad) was implemented with the robot. When asked which system participants preferred, 9 responded that they prefer the gamepad controller, mainly due to their familiarity with the system. Of the remaining 9 participants whose responses were recorded, 8 preferred the glove and reported it as easier to use and quicker to learn. One participant responded that they did not have a preference of system. Other comments on the overall system included ones regarding the camera for feedback.

6 Conclusion

Given that the glove condition showed better performance, in the form of faster completion times, for the touch time than for the drive time, we can conclude that the glove is better suited for faster completion of manipulation tasks. However, it should be noted that the accuracy for the task was lower in the glove condition. Since the gamepad style controllers are a common consumer product, additional user training with the glove controller may increase the glove controller performance compared to the gamepad for chassis control and/or manipulator arm accuracy. Additionally, alternate control mappings of glove sensor input to robot motor activation may show higher performance. The index finger mapping was selected as the most intuitive, however, other approaches might be more effective for manipulation speed and accuracy.

The results of the drive time by controller type show similar distributions, with the exception of two outlier data points for the glove control condition which if eliminated, would significantly reduce the difference between the mean values of the Xbox and glove controller. Depending on the task, speed may take priority over accuracy, or vice-versa. For example, during a building reconnaissance, deployment of a camera payload that is mounted to a manipulator arm may require speed over accuracy due to the nature of the mission in progress.

Soldier feedback on the instrumented glove was of most interest in this preliminary evaluation. Some issues were anticipated due to having a single glove size. Additional feedback from the Soldiers with regard to glove-robot-camera integration will aid in further refinement of the glove as a viable option in operational settings.

Because the glove technology integrates into existing combat attire, the glove control solution provides an overall weight reduction to the soldier’s combat load as it eliminates the need for a dedicated controller. Current ruggedized operator control units (OCUs) are bulky and add extra weight to the soldier’s load. The sensors in the glove controller add approximately 70 grams of weight, in comparison to about 205 grams for an Xbox controller. Additionally, unlike holding a game controller, the soldier using the glove controller can quickly and easily transition from robot control to individual rifle deployment, thus maintaining a higher level of defensive posture.