Gestural Transmission of Tasking Information to an Airborne UAV

  • Alexander SchelleEmail author
  • Peter Stütz
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10904)


A system is presented that enables an authorized person on ground to transmit mission information to an airborne UAV within line of sight by using gestural expressions of both arms without the need for additional devices on ground. A miniaturized processing board with a discrete GPU is used to detect the body movements via a high resolution onboard camera and to translate them into relevant tasking information. Individual task elements are transmitted consecutively, including numerical and non-numerical information. A context aware gesture recognition approach is implemented to enable the reuse of gestures for different contexts in order to maintain a small gesture set. The system further features a bidirectional communication which allows to dispatch visual feedbacks and to query missing information visually via a LED matrix. Two experiments with different briefing contents in a static and dynamic setup have been conducted to proof the feasibility under real-life conditions.


Visual communication Gesture recognition Human-UAV-interaction 


  1. 1.
    Schelle, A., Stütz, P.: Modelling visual communication with UAS. In: Hodicky, J. (ed.) MESAS 2016. LNCS, vol. 9991, pp. 81–98. Springer, Cham (2016). Scholar
  2. 2.
    Monajjemi, M., Mohaimenianpour, S., Vaughan, R.: UAV, come to me. End-to-end, multi-scale situated HRI with an uninstrumented human and a distant UAV. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4410–4417 (2016).
  3. 3.
    Monajjemi, V.M., et al.: HRI in the sky: creating and commanding teams of UAVs with a vision-mediated gestural interface. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 03 November 2013, pp. 617–623. IEEE (2013)Google Scholar
  4. 4.
    Wang, T., Wang, M.: Remote control method and terminal Patent US9493232B2 (2016)Google Scholar
  5. 5.
    Nagi, J., et al.: HRI in the sky: controlling UAVs using face poses and hand gestures. In: HRI, pp. 252–253 (2014)Google Scholar
  6. 6.
    Schelle, A., Stütz, P.: Visual communication with UAS: recognizing gestures from an airborne platform. In: Lackey, S., Chen, J. (eds.) VAMR 2017. LNCS, vol. 10280, pp. 173–184. Springer, Cham (2017). Scholar
  7. 7.
    Cao, Z., et al.: Realtime multi-person 2D pose estimation using part affinity fields. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  8. 8.
    Wei, S.-E., et al.: Convolutional pose machines. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4724–4732 (2016)Google Scholar
  9. 9.
    Simon, T., et al.: Hand keypoint detection in single images using multiview bootstrapping. arXiv preprint arXiv:1704.07809 (2017)
  10. 10.
    Cao, Z., Simon, T., Wei, S.-E., Sheikh, Y.: OpenPose: real-time multi-person keypoint detection library for body, face, and hands estimation. Carnegie Mellon University, Perceptual Computing Laboratory (2018).
  11. 11.
    Kirk, D.: NVIDIA CUDA software and GPU parallel computing architecture. In: Proceedings of the 6th International Symposium on Memory management, Montreal, Quebec, Canada, pp. 103–104. ACM, New York (2007).
  12. 12.
    Andriluka, M., et al.: 2D human pose estimation. new benchmark and state of the art analysis. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3686–3693 (2014).
  13. 13.
    Liu, W., et al.: SphereFace. Deep hypersphere embedding for face recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  14. 14.
    Godil, A., Grother, P., Ressler, S.: Human identification from body shape. In: Proceedings of the Fourth International Conference on 3-D Digital Imaging and Modeling, 3DIM 2003, pp. 386–392 (2003)Google Scholar
  15. 15.
    Bouchrika, I.: On using gait biometrics for re-identification in automated visual surveillance. In: Computer Vision: Concepts, Methodologies, Tools, and Applications, pp. 2363–2386. IGI Global (2018)Google Scholar
  16. 16.
    King, D.E.: Dlib-ml: a machine learning toolkit. J. Mach. Learn. Res. 10, 1755–1758 (2009)Google Scholar
  17. 17.
    Shen, C., Kim, J., Wang, L.: A scalable dual approach to semidefinite metric learning. In: CVPR 2011, Providence, RI, USA, pp. 2601–2608. IEEE (2011)Google Scholar
  18. 18.
    Berndt, D.J., Clifford, J.: Using dynamic time warping to find patterns in time series. In: Proceedings of the 3rd International Conference on Knowledge Discovery and Data Mining, vol. 10, pp. 359–370. AAAI Press, Seattle (1994)Google Scholar
  19. 19.
    Gillian, N.E., Paradiso, J.A.: The gesture recognition toolkit. J. Mach. Learn. Res. 15(1), 3483–3487 (2014)MathSciNetGoogle Scholar
  20. 20.
    Yeo, H.-S., Lee, B.-G., Lim, H.: Hand tracking and gesture recognition system for human-computer interaction using low-cost hardware. Multimed. Tools Appl. (2015). Scholar
  21. 21.
    bin Abdul Rahman, N.A., Wei, K.C., See, J.: RGB-H-CBCR skin colour model for human face detection. Faculty of Information Technology, Multimedia University, vol. 4 (2007)Google Scholar
  22. 22.
    Quigley, M., et al.: ROS: an open-source robot operating system. In: ICRA Workshop on Open Source Software (2009)Google Scholar
  23. 23.
    Munaro, M., et al.: OpenPTrack: people tracking for heterogeneous networks of color-depth cameras. In: IAS-13 Workshop Proceedings of the 1st International Workshop on 3D Robot Perception with Point Cloud Library, Padova, Italy, pp. 235–247 (2014)Google Scholar
  24. 24.
    Garcia, D., Kriegsman, M.: FastLED (2015).

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.University of the Bundeswehr MunichNeubibergGermany

Personalised recommendations