Game of Drones: How to Control a UAV?

Abstract

Unmanned aerial vehicles (UAVs) so-called drones are getting more and more popular in the civil sector as well as in the military scope. One of the main challenges is the interaction of the pilot and the drone. This paper explores the usage of different game controllers as input device to control an UAV. In an explorative study, participants fulfill a predefined flying task and report their expectations before and experiences after performing the flight with different gaming controllers. The resulting insights are a basis for further interaction research activities.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Since the beginning of aviation, airplanes are controlled by the help of a control stick. It was already used in the first motorized aircrafts and is still standard in actual aviation. So, unmanned aerial vehicles (UAVs) so-called drones are typically operated by a control stick as well. Also virtual airplanes in the gaming section are often controlled by Joysticks too. But for gaming purposes, many other interaction devices exist beside the Joystick and a huge community is familiar in using them.

Since drones are getting more and more popular, this paper explores the usage of different game controllers for operating a leisure sector drone. Therefore, three wide-spread input devices are selected:

• 1. A Joystick is a quite usual way to interact with a drone

• 2. A Xbox console gamepad represents a very common interaction devices for console gamer but it is rather unusual to control UAVs

• 3. A Wii Balance Board is very unusual to control a UAV and it exposes the user to a physical challenging operating situation

These input devices are used to control a drone to fulfill a predefined task. This task is carried out in an empirical study and reviews different aspects including gaming know-how, gender, sportiness and age.

1.1 Technical Realization

The following technical adaptions are conducted to connect different input devices with the drone. So, the user is enabled to control the direction and speed of the UAV. The flight intelligence and stabilization algorithms of the drone are still active and working.

The used technical setup is based on a student’s work in the lecture “Advanced Interactive Systems” in 2016. As UAV, the drone Bebop 2 of the French manufacturer Parrot is used. It was chosen, since it is a comparatively compact, stable and powerful quadcopter with an open programming interface. As SDK, the currently newest software version ARDroneSDK 3 is utilized. A special feature of Bepop 2 is the connection via Wireless Lan, which allows an easy and simple connection of self-programmed control software. The central element of the used setup is a laptop or PC, which acts as a control station (tower). It serves as a central interface between the controlled drone and the used input devices. This experimental setup, in combination with geofencing and the continuous intervention facility of the ‘tower’-PC, allows safe flights even with inexperienced participants. The application logic is written in JavaScript and executed with NodeJS. Both, the drone and the various input devices are embedded by corresponding node modules. As input devices, a joystick, a gamepad, a keyboard and a Wii Balance Board are used. Thereby the joystick and the gamepad are connected via USB-ports and the Balance Board over Bluetooth. Due to the missing NodeJS implementation for the Balance Board, a Java application is used to connect the main program and the board via UDP sockets.

The used software is publicly available under github.com/fog1992/AIS- Drone. A schematic representation of the technical implementation is shown in Fig. 1.

Fig. 1.

Technical realization

2 Related Work

Due to the importance of the pilot and drone interaction, there are several studies that focus on the question, how a UAV can be controlled. Higuchi et al. introduced a control mechanism that transforms head motion to UAV movements [HR1]. A Brain Computer Interface is used by Kos’myna et al. in their research about an optimal BCI architecture for controlling a UAV [KT1].

Ballas evaluated different methods of direct manipulation in aviation. In modern airplanes fly-by-wire systems separate the pilot from direct control of the wing surfaces [BH1]. This is also suitable for most UAVs: the control device allows to steer in several directions and the UAV reacts appropriately, but the user does not need to know how this motion is achieved.

Some studies try to design new interfaces and/or controllers for airplanes, drones, etc. For example, Won et al. created GUI guidelines for UAV ground control stations in [JH1]. Witheside concluded that usability depends more on a specific interface design than the interface style [WJ1].

This paper explores and discusses how suitable different game controllers are to interact with a drone. Among others, Skalski discussed the naturalness of game controllers and the use of different input devices [ST1]. Recent work in the area of games has shown that beyond simply performing a task, users want to engage with an interactive system, allowing them to have a playful and joyful experience [B1]. User experience in games considers that the users should enjoy their activities while interacting with the computer and that this enjoyment can be one of the main goals they want to achieve [BB1].

3 Study Method and Participants

To achieve an evaluation before and after the flight task, the study was divided into three parts:

• pre-questionnaire,

• field study, and

• post-questionnaire.

3.1 Pre-questionnaire

Before starting the flight task, the participants were interviewed by using a questionnaire. In this questionnaire both general information of the person as well as possible factors which could influence the ability to control a drone were questioned. The objective of the questionnaire was to elaborate whether the participants have previous experience with the relevant input devices and whether synergy effects for the drone control can be demonstrated. To identify synergy effects, some questions deals with sport activities, driving license, gender, and age of the participants. In addition, the expectations regarding the personal knowledge and their usefulness during the execution of a flight task were asked. The participants estimated furthermore how suitable the different devices for flying the drone will be.

3.2 Field Study

The field study took place at the university sports ground by moderate weather. While carrying out the study, participants had to cope with various flight tasks. Among other challenges, the participants had to direct the drone several times through a predefined course (cf. Fig. 1), using each of the available input devices. The task was to maneuver the drone around three bars, fly through a gate and return in a straight line to the starting point. The flight altitude was predefined and could not be altered by the participants. This simplification reduced the risk of crashes and improved the comparability (Fig. 2).

Fig. 2.

Course sketch

The flight time was measured between a predefined start area and end area. The elapsed time for the pass with each device was measured separately with the order Joystick, Gamepad, Balanced Board. This order was chosen to minimize learning effects by ordering the devices from familiar to unfamiliar. The distance of the drone to each obstacle during the flights was not determined, since the resulting time for conducting the task describes the performance of each pass sufficiently.

3.3 Post-questionnaire

In the post-questionnaire a subjective assessment of the task difficulty with each input device was determined. The participants answered how suitable each input devices for the control of the drone was. Furthermore, the question was asked whether the participants believed that the personal knowledge had a positive effect on the performance in the study task.

3.4 Participants

For the study, 25 adults were selected as participants. The group consisted of 18 males and 6 females, aged from 21 to 64 years (cf. Fig. 3).

Fig. 3.

Age of participants

The participants had different gaming experiences with one or more of the following gaming platforms: Xbox, Playstation, PC, Wii or others. This is shown in Fig. 4b. The time spent for playing per week varies from 0 to 25 h per platform. There were almost no participants without any gaming experience.

Fig. 4.

Participant experience

4 Analysis

This section explains the executed statistics in more detail. The data is analyzed concerning the expected and actual performance of the three controls, the measured course times and correlations in gaming experience and drone flight performance.

4.1 Ratings of Expected and Actual Performance of the Three Controls

All three controls (gamepad, joystick, Balance Board) were rated regarding their expected performance (before task execution) and the actual performance (after execution). The differences of the two ratings for each control were compared with a paired two-sample t-test. The gamepad was rated significantly better after the testing than before. The two other controls ratings also improved, however the differences were not significant (cf. Fig. 5). The gamepad’s mean rating improved, with a mean rating of 2.8 ($MD=1.3$) before and a mean rating of 3.7 ($MD=0.2$) after the testing ($t(23) =- 4.2, p < .001$). The joystick received a mean rating of 2.4 ($M=1.3$) before, and a mean rating of 2.7 ($MD=0.7$) after the testing ($t(23) = -1.0, p = n.s.$). The Balance Board received a mean rating of .9 ($MD=0.8$) before, and a mean rating of 1.3 ($MD=1.2$) after the testing ($t(23) = -1.2, p = n.s.$). The ratings for the three controls were compared to each other with two one-way ANOVAs, one for the ratings of expected performance and one for the ratings of actual performance. The assumption of sphericity was violated, therefore degrees of freedom were corrected using Greenhouse-Geisser correction. The ratings significantly differed between the three conditions before ($F(1.5,345) = 31.4, p < 0.001$) and after the testing ($F(1.5,345) = 55, p < 0.001$). Planned contrasts revealed, that the gamepad’s performance ratings after the actual testing were significantly better than those for the joystick and the Balance Board ($F(23) = 142.3, p < 0.001$) Furthermore, the Balance Board was rated significantly worse than the joystick ($F(23) = 24.8, p < 0.001$). Furthermore the joystick and the gamepad’s ratings of expected performance did not significantly differ ($F(23) = 1.9, p = n.s.$). However the Balance Board was rated as significantly worse than the gamepad and the joystick ($F(23) = 57.1, p < 0.001$). This can be interpreted in a way that the gamepad’s actual performance outperformed initial expectations, while the other two controls performed as expected. When rating the expected performance, the participants expected the Balance Board to perform worse than the two other controls, and this expectation was confirmed in the actual testing.

Fig. 5.

Participant expectations

4.2 Performance Differences Between the Controls

The performance differences were analyzed with a between-subjects repeated measures ANOVA. The assumption of sphericity was violated, therefore degrees of freedom were corrected using Greenhouse-Geisser correction. The performance significantly differed between the three conditions ($F(1.6,34.8) =73.0, p < 0.001$). Planned contrasts revealed a nonsignificant difference between the joystick condition and the gamepad condition ($F(1) =1.4, p = n.s.$). The joystick condition showed a mean performance time of 30.5 ($SD=4.3$), while the gamepad condition revealed a mean performance time of 29.4 ($SD=5.5$). This is shown in Fig. 6. However, the Balance Board condition showed a significantly decreased performance with a mean performance time of 40.8 ($SD=5.8$), which was significantly longer than in the two other conditions ($F(1) = 97, p < 0.001$). While the joystick and gamepad seem to be equally suitable for controlling a drone, the Balance Board seems to be less efficient when it comes to performance times.

Fig. 6.

Course times

4.3 Connection Between Gaming Experience and Drone Flight Performance

The correlation between the participants’ weekly time spend gaming and their drone flight performance was not significant for any of the three controls (Balance Board: $r =-0.24$, Joystick: $r= -0.18$, Gamepad: $R= -0.19$, $p = n.s.$). Although a null-effect should be interpreted with care, the result suggests that previous gaming performance might not have a strong influence on one’s ability to fly a drone.

4.4 Comments of Participants After Flight

Below, some of the most meaningful comments of the participants after flying with the controllers are listed:

Comments on the Gamepad

P4 :

‘Very good control, reacts fast and precise.’

P5 :

‘Familiar precision, nice usage as controller, fits better pilots as well’

P8 :

‘Very enjoyable. The only controller that is portable. Furthermore has many usable buttons.’

P15 :

‘Astonishing precise flight behavior.’

P18 :

‘The Controller was the easiest to handle, because the type of controlling is formally known. Doesn’t matter if XBOX- or Playstation Controller, they are all used alike.’

P25 :

‘Very intuitive.’

Comments on the Joystick

P6 :

‘Very intuitive. Forgives controlling mistakes. Very good for beginners.’

P8 :

‘Flying with one hand is advantageous.’

P18 :

‘The Joystick is the most difficult to use, what actually surprised me.’

P20 :

‘Needs to get used to, but works very well after that.’

P21 :

‘Some kind of imprecise.’

P22 :

‘Easy to handle, but it feels delayed. Reacts very sensible.’

Comments on the Balance Board

P12 :

‘Required getting used to. You have be careful to not fall down.’

P15 :

‘Very imprecise.’

P16 :

‘Easy to balance, but delayed.’

P18 :

‘Takes a lot sense of balance, After a short testing time you quickly get how it’s done.’

P19 :

‘The way of controlling was actually pretty good. If the board would have been more precise controlling would have been easier.’

P20 :

‘Needs a lot of practice. Very unintuitive.’

P22 :

‘Easier than expected but the coordination was difficult, you have to get used to it.’

5 Conclusion

The main conclusion of the conducted study is that both, the gamepad and the joystick are equal suitable to control a drone. The third tested input device, the Wii Balance Board is rather unsuitable to execute sophisticated flight tasks. This result emerged directly from the collected data as shown in the analysis section. In addition to the worst elapsed times for the course, the Balance Board has earned low suitability rates before as well as after the flight.

The joystick and the Balance Board satisfied the expectation of the participants, whereby the gamepad surpassed its expectation. It was assumed that the joystick was rated comparatively high before the field study, because the participants knew of its usage in aviation. This expectation was met by the actual flight experience. The Balance Board was significantly less familiar in comparison to the other devices. This and the fact that the Balance Board is usually not used to control motion supposedly leaded to its poor rating and actual flight performance. The gamepad is one of the most known gaming controllers, especially in racing games, therefore most participants already knew it (cf. Fig. 4a) and were aware of its purpose to control first or third person motion. This implies that using a gaming controller for controlling a highly interactive device is a promising topic for further investigations.

As mentioned in Sect. 4 there is no evidence, that the gaming experience (respectively gaming hours per week) of the participants have any impact on their UAV flight skills. Though the collected data show interesting possibilities for further investigation. Two of the participants have no driving license and performed very badly controlling the UAV. This suggests that the driving experience enables the participants to control the UAV more precise. In a follow-up study will be tested a group consisting of 50% with and 50% without a driver license.

The influence of a first person view compared to observing the drone is another thread for further investigation.