Usability Evaluation of Newly Developed Three-Dimensional Input Device for Drone Operation

Abstract

As an input device suitable for operation in VR space, we developed a prototype in 2003. Comparison experiment results with other input devices, including game controllers, showed that our prototype had learnability and memorability advantages; that is, this device is suitable for intuitive operation in VR space. After the improvement of the device, it was placed on the market under the name Cyberbird. Moreover, we experimentally proposed the best combinations between a 3-DOF analog stick and two buttons on Cyberbird and six movements of drones: pitch, roll, yaw, throttle, take-off and landing. This article introduces the results of the experiment evaluating the usability of Cyberbird for older people and students unfamiliar with playing TV games using a game controller.

1 Introduction

As an input device suitable for VR spaces, we developed a prototype in 2003. Comparison experiment results with other input devices (including game controllers) showed that our prototype had learnability and memorability advantages; that is, this device is suitable for intuitive operation in VR space [1]. Then, the device, which was named Cyberbird [2], was improved and placed on the market.

At the same time, drones used for rescue operations emerged, with advantages of no spatial restrictions. They have also begun to be used for entertainment. The Drone Race was held in California in 2015 [3], and a Bebop Drone [4] was developed for use with Oculus Rift. However, when operators control the drones that are currently available on the market, their controllers must be grasped with both hands and operators must be trained to control them.

We addressed this training requirement with Cyberbird, as it can provide intuitive drone operation. Then, we began to study how to apply Cyberbird to the intuitive operation of a drone.

Generally, there are six movements involved in flying drones: pitch, roll, throttle, yaw, take-off and landing. On the other hand, Cyberbird has 3-DOF movements with an analog stick and two buttons. Since drone operation is impossible by just using an analog stick, we had to examine drone-operation methods using Cyberbird to discover the best combination of drone movements and Cyberbird operations. Therefore, we performed an experiment to determine the best combination.

During the experiment, we recognized the necessity of improvement for better operability of Cyberbird. Thus, we improved its shape, and performed a comparative experiment to evaluate the usability of the improved version of Cyberbird. This article describes these experiments and improvements in detail.

2 Cyberbird Operation Methods

Figure 1 shows how to hold Cyberbird. The operator’s thumb is placed on the analog stick, and both of her pointer and middle fingers are on specific buttons. Operators can control drones with these analog sticks and buttons.

Fig. 1.

Holding cyberbird

A drone has four flight control inputs: pitch, roll, yaw, and throttle (Fig. 2). The pitch input is the horizontally front and back movement parameters, and the roll is the horizontally left and right movement parameters. The yaw turns the drone left or right. The throttle moves it up or down.

Fig. 2.

Drone flight movements

First, we designed operation methods 1 and 2. Their differences are shown in Fig. 3. In operation method 1, when the operator turns the analog stick left or right, the drone moves horizontally left or right. When the operator pushes the buttons, the drone turns left or right. In operation method 2, when the operator turns the analog stick, the drone rotates. When the operator pushes the button, it moves horizontally.

Fig. 3.

Differences of operation methods 1 and 2

We designed operation methods (a) and (b). For example, in operation method 1(a), if the operator pushes button 1, the drone turns left, and it turns right when button 1 is quickly pushed twice. In operation method 1(b), when the operator pushes button 1, the drone turns left, and it turns right when button 2 is pushed. We designed four operation methods: 1(a), 1(b), 2(a), and 2(b). Their common operation methods are shown in Fig. 4.

Fig. 4.

Common operation in all methods

3 Experimentto Determine Combination of Operations

3.1 Experimental System

A diagram of our system is shown in Fig. 5. The drone we used is the Parrot AR. Drone 2.0 [5]. The PC is connected to Cyberbird via USB and to the drone by Wi-Fi. The drone 2.0 has a front- and bottom- mounted cameras; we used the front camera and those images appeared on the PC. The operator manipulates the drone using Cyberbird while looking at the image on the PC. The system sends control commands to the drone 2.0 based on input from Cyberbird. The library we used to control the drone is ARDroneForP5 [6].

Fig. 5.

System diagram

3.2 Experimental Method

The operator sequentially captured four markers using the drone camera. Figure 6 shows the numbered markers targeted on a pole extending to the ceiling. The height of each marker was between 70 to 200 cm, and the markers shown in Fig. 6 were used for both operation methods. The operator also confirmed the position of the markers before starting the capture operation.

Fig. 6.

Marker positions and orientations

First, the operator controls the take-off of the drone from 2.1 m away. A few seconds later, the operator starts to capture the markers. In the normal state, the image from the drone’s inner camera with the black square is shown on the PC (Fig. 7). When the marker is in the black square and the drone is in the marker’s range, a green box is displayed on the marker (Fig. 7). This is the successful-capture state. After the operator successfully captures the marker, he starts to capture the next marker. After capturing the fourth marker, he lands the drone. This experiment’s methodology is based on Higuchi and Rekimoto [7].

Fig. 7.

Capturing markers (Color figure online)

An example of our experimental procedure is shown in Fig. 8. We divided our experiment into parts 1 and 2. At the end of part 1, the participants answered questionnaire 1. After finishing part 2, they answered questionnaire 2. Table 1 shows the items in questionnaire 1. We also changed the combination of operation methods in each part for every participant. In questionnaire 2, participants explained which operation method they preferred and explained why.

Fig. 8.

Experimental procedure

Table 1. Items for questionnaire 1

3.3 Experimental Results

Our participants were 12 male students in their 20 s at the Shibaura Institute of Technology. Figure 9 shows the results of questionnaire 1. Operation method 1(b) showed the best score among the four methods because it is more intuitive for rolling with an analog stick than with buttons. It is also easier to assess left-right orientation using buttons 1 and 2.

Fig. 9.

Questionnaire 1 result

The questionnaire 2 results are shown in Fig. 10. All participants chose either operation method 1(b) or 2(b). Nine of 12 chose 1(b). Operation method 1(b) has more general versatility than 2(b).

Fig. 10.

Questionnaire2 result

From the results of questionnaires 1 and 2, we chose operation method 1(b) as the best combination of drone movements and Cyberbird operations. In operation method 1(b), when the operator turns the analog stick left or right, the drone moves horizontally left or right. When the operator moves the analog stick forward or backward, the drone moves horizontally forward or backward. When the operator moves the analog stick upward or downward, the drone moves horizontally upward or downward. Also if the operator pushes button 1, the drone turns left, and it turns right when button 2 is pushed. Also, if the operator pushes buttons 1 and 2 simultaneously, the drone takes off or lands.

4 Improvement of Cyberbird

During the above experiment, we noticed a problem with Cyberbird in that it was difficult to transmit the force of the fingers when pressing down the analog stick or pushing buttons. Based on the knowledge that the grasping force increases as the contact area between the hand and the cylindrical object increases [8], we made new parts with a 3D printer and attached them to Cyberbird to increase its contact area with the hand. We will call this “improved Cyberbird” from here on (Figs. 11 and 12).

Fig. 11.

Cyberbird with new attachments

Fig. 12.

Top view of holding Cyberbird with new attachments

5 Experiment for Usabilityevaluation

5.1 Experimental Method

To evaluate the usability of improved Cyberbird for drone operation, we performed an experiment, in which we compared two devices, the improved Cyberbird and a game controller (Fig. 13). In addition, we employed a third device, Cyberbird Mk-II (Fig. 14), which was developed independently by the third author. The experimental system and content were the same as the experiment to determine the combination of operations described in Sect. 3. The operator captured four markers while operating a drone with the improved Cyberbird, and then with a game controller. The order of using the two devices was counter-balanced.

Fig. 13.

Game controller and operations of drone

Fig. 14.

Cyberbird Mk-II

A questionnaire and required time for capturing markers were employed for evaluation. Operators were asked six questions on a five-point Likert scale as shown in Table 2 after operation. Required time was measured from take-off to landing of the drone.

Table 2. Items for questionnaire

5.2 Experimental Results

The experiment was performed employing eight older people in their 60’s and 70’s with no video game experience, eight male students in their 20’s with less experience of video game, and eight male students in their 20’s with more video game experience. “With less experience” means that they play video games fewer than five times a month, and “with more experience” means that they play video games often. Figure 15 shows an experimental scene. In the experiment for older users, only Cyberbird and the game controller were employed for comparison because Cyberbird Mk-II had not been produced at that time.

Fig. 15.

Experimental setting

The questionnaire results for older users are shown in Fig. 16. Because of the significant differences between the two devices in Q2 and Q3 (p < 0.05, p < 0.01), the improved Cyberbird has significantly better learnability and memorability than the game controller for older users. Figure 17 shows the required time for each task. The time in the case of using the improved Cyberbird is significantly shorter than that in the case of using the game controller.

Fig. 16.

Questionnaire results for older users

Fig. 17.

Required time for each device for older users

The questionnaire results for the students are shown in Figs. 18 and 19. Although there are no significant main effects of devices for “with less game experience” students, there are significant main effects in Q1, Q4, and Q5 for “with more game experience” students. From the results of multiple comparisons, the game controller is significantly simpler, has significantly better operability, and is significantly more enjoyable for “with more game experience” students. Figures 20 and 21 show the required times. There are no significant main effects of devices for both student groups.

Fig. 18.

Questionnaire results for students with less game experience

Fig. 19.

Questionnaire results for students with more game experience

Fig. 20.

Required time for each device for students with less game experience

Fig. 21.

Required time for each device for students with more game experience

We obtained the following results.

• The improved Cyberbird has better usability than the game controller for older users with no game experience.

• The improved Cyberbirdand Mk-II show no difference with the game controller for students unfamiliar with the game controller.

• The game controller has better usability than the improved Cyberbird and Mk-IIfor students familiar with the game controller.

In addition, the students familiar with the game controllers commented in the questionnaire as follows.

• The analog stick of the improved Cyberbird has better operability than that of Mk-II.

• The buttons on Mk-II are easier to push than those of the improved Cyberbird.

6 Discussion

We designed an experiment to determine drone operation methods with our new device called Cyberbird; its improvement and an experiment on usability are described. From the experimental results, it was clarified that Cyberbird is suitable for intuitive operation of a drone for older users with no game experience and those who are unfamiliar with the game controller.

In addition, the comparative advantages and disadvantages for operability for the improved Cyberbird and Mk-II were clarified. Therefore, employing the advantages of both devices, we should be able to achieve a new device with better operability.

7 Conclusions

To apply the device we developed for intuitive operation in VR space to drone operation, we performed the following.

• Experiment to decide the combinations of drone movements and Cyberbird operations.

• Improvement of Cyberbird by new attachments for better transmission of force from fingers.

• Evaluation experiment to compare the usability of improved Cyberbird with the game controller.

Finally, we reached the conclusion that the improved Cyberbird is suitable for intuitive operation of a drone for older users with no game experience and those who are unfamiliar with the game controller.

In addition, combining the advantages of the improved Cyberbird and Mk-II should allow us to achieve a new device with better operability.