Research Considerations and Tools for Evaluating Human-Automation Interaction with Future Unmanned Systems

  • Ciara SibleyEmail author
  • Joseph Coyne
  • Sarah Sherwood


Advances in automation will soon enable a single operator to supervise multiple unmanned aerial vehicles. Successfully implementing this new supervisory control paradigm not only requires improvements in automation capability and reliability, but also an understanding of the human performance issues associated with concurrent management of several automated systems. Research in this area has generally focused on topics such as trust, reliability, and levels of automation. The goal of automating systems is generally to minimize the human’s need to directly interact with the system; despite this objective, the majority of current supervisory control research emphasizes situations in which the human must frequently interact with the automation. This is typically done to provide researchers with a clear means of assessing human performance, but ultimately limits the generalizability of the research since it only applies to a limited mission context. The current chapter discusses a model of assessing human-automation interaction that emphasizes not only the traditional outcome-based measures of assessing performance (e.g., speed and accuracy), but also addresses measures of operator state. Such measures include those obtained from subjective workload and fatigue probes, situation awareness (SA) probes, and continuous measures from eye tracking systems. The chapter closes by discussing a new testbed developed by the authors that enables the assessment of human-automation interaction across a broad range of mission contexts.


  1. Beatty J, Lucero-Wagoner B (2000) The pupillary system. Handbook of psychophysiology 2:142-162Google Scholar
  2. Button K (2009) Different courses-New-style UAV trainees edge toward combat. C4isr 8 (10):34Google Scholar
  3. Caffier PP, Erdmann U, Ullsperger P (2003) Experimental evaluation of eye-blink parameters as a drowsiness measure. European journal of applied physiology 89 (3-4):319-325CrossRefGoogle Scholar
  4. Calhoun GL, Draper MH, Ruff HA Effect of level of automation on unmanned aerial vehicle routing task. In: Proceedings of the Human Factors and Ergonomics Society 53rd Annual Meeting, San Antonio, TX, 2009. vol 4. SAGE Publications, pp 197-201Google Scholar
  5. Calhoun GL, Ruff HA, Draper MH, Wright EJ (2011) Automation-level transference effects in simulated multiple unmanned aerial vehicle control. Journal of Cognitive Engineering and Decision Making 5 (1):55-82CrossRefGoogle Scholar
  6. Chanda M, DiPlacido J, Dougherty J, Egan R, Kelly J, Kingery T, Liston D, Mousseau D, Nadeau J, Rothman T, Smith L, Supko M (2010) Proposed functional architecture and associated benefits analysis of a common ground control station for Unmanned Aircraft Systems.Google Scholar
  7. Chen JY, Barnes MJ, Harper-Sciarini M (2011) Supervisory control of multiple robots: Human-performance issues and user-interface design. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 41 (4):435-454CrossRefGoogle Scholar
  8. Coyne J, Sibley C Investigating the Use of Two Low Cost Eye Tracking Systems for Detecting Pupillary Response to Changes in Mental Workload. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 2016. vol 1. SAGE Publications, pp 37-41Google Scholar
  9. Cummings ML, Bertucelli LF, Macbeth J, Surana A (2014) Task versus vehicle-based control paradigms in multiple unmanned vehicle supervision by a single operator. IEEE Transactions on Human-Machine Systems 44 (3):353-361CrossRefGoogle Scholar
  10. Cummings ML, Nehme CE Modeling the impact of workload in network centric supervisory control settings. In: 2nd Annual Sustaining Performance Under Stress Symposium, 2009.Google Scholar
  11. Defense Science Board (2016) Summer Study on Autonomy. Office of the Under Secretary of Defense for Acquisition, Technology and Logistics, Washington, D.C.Google Scholar
  12. Department of Defense (2013) Unmanned systems integrated roadmap: FY2013-2038. Washington, DC, USAGoogle Scholar
  13. Durso FT, Dattel AR (2004) SPAM: The real-time assessment of SA. In: Banbury S, Tremblay S (eds) A cognitive approach to situation awareness: Theory and application, vol 1. Ashgate Publishing Ltd, Hampshire, pp 137-154Google Scholar
  14. Endsley MR Situation awareness global assessment technique (SAGAT). In: Proceedings of the National Aerospace and Electronics Conference, New York, 1988. IEEE, pp 789-795Google Scholar
  15. Endsley MR, Kaber DB (1999) Level of automation effects on performance, situation awareness and workload in a dynamic control task. Ergonomics 42 (3):462-492. doi: 10.1080/001401399185595 CrossRefGoogle Scholar
  16. Funke G, Greenlee E, Carter M, Dukes A, Brown R, Menke L Which Eye Tracker Is Right for Your Research? Performance Evaluation of Several Cost Variant Eye Trackers. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 2016. vol 1. SAGE Publications, pp 1240-1244Google Scholar
  17. Gertler J (2012) US Unmanned Aerial Systems. Congressional Research Service,Google Scholar
  18. Giese S, Carr D, Chahl J Implications for unmanned systems research of military UAV mishap statistics. In: Intelligent Vehicles Symposium (IV), 2013 IEEE, 2013. IEEE, pp 1191-1196Google Scholar
  19. Holmqvist K, Nyström M, Andersson R, Dewhurst R, Jarodzka H, Van de Weijer J (2011) Eye tracking: A comprehensive guide to methods and measures. Oxford University Press, New YorkGoogle Scholar
  20. Holmqvist K, Nyström M, Mulvey F Eye tracker data quality: What it is and how to measure it. In: ACM, Proceedings of the Symposium on Eye Tracking Research and Applications, Santa Barbara, CA, 2012. pp 45-52Google Scholar
  21. Johnson R, Leen M, Goldberg D (2007) Testing adaptive levels of automation (ALOA) for UAV supervisory control (Technical Report AFRL-HE-WP-TR-2007-0068). Air Force Research LaboratoryGoogle Scholar
  22. Kahneman D (1973) Attention and effort. Prentice-Hall, Kahneman, Daniel. Attention and effort. Englewood Cliffs, NJGoogle Scholar
  23. Kidwell B, Calhoun GL, Ruff HA, Parasuraman R Adaptable and adaptive automation for supervisory control of multiple autonomous vehicles. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 2012. vol 1. SAGE Publications, pp 428-432Google Scholar
  24. NATO (2012) Standard Interfaces of UAV Control System (UCS) for NATO UAV Interoperability. NATO Standardization Agency, Brussels, BelgiumGoogle Scholar
  25. Nehme CE (2009) Modeling human supervisory control in heterogeneous unmanned vehicle systems. Massachusetts Institute of Technology, Cambridge, MAGoogle Scholar
  26. Office of Naval Research (2015) Naval Science & Technology Strategy. Department of the Navy, Arlington, VAGoogle Scholar
  27. Office of the Secretary of Defense (2012) Unmanned Aircraft Systems Ground Control Station Human-Machine Interface: Development and Standardization Guide. Washington, DCGoogle Scholar
  28. Ooms K, Dupont L, Lapon L, Popelka S (2015) Accuracy and precision of fixation locations recorded with the low-cost Eye Tribe tracker in different experimental set-ups. Journal of Eye Movement Research 8:1-24Google Scholar
  29. Parasuraman R, Manzey DH (2010) Complacency and bias in human use of automation: An attentional integration. Human Factors: The Journal of the Human Factors and Ergonomics Society 52 (3):381-410CrossRefGoogle Scholar
  30. Parasuraman R, Riley V (1997) Humans and automation: Use, misuse, disuse, abuse. Human Factors 39:230-253CrossRefGoogle Scholar
  31. Parasuraman R, Sheridan TB, Wickens CD (2000) A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans 30 (3):286-297. doi: 10.1109/3468.844354 CrossRefGoogle Scholar
  32. Parasuraman R, Sheridan TB, Wickens CD (2008) Situation awareness, mental workload, and trust in automation: Viable, empirically supported cognitive engineering constructs. Journal of Cognitive Engineering and Decision Making 2 (2):140-160CrossRefGoogle Scholar
  33. Ratwani RM, McCurry JM, Trafton JG Single operator, multiple robots: an eye movement based theoretic model of operator situation awareness. In: Proceedings of the 5th ACM/IEEE international conference on Human-robot interaction, 2010. IEEE Press, pp 235-242Google Scholar
  34. Samn SW, Perelli LP (1982) Estimating aircrew fatigue: a technique with application to airlift operations. DTIC Document, USAF School of Aerospace Medicine, Brooks Air Force Base, TXGoogle Scholar
  35. Sheridan T, Verplank W (1978) Human and computer control of undersea teleoperators. MIT Man-Machine Systems Laboratory, Cambridge, MAGoogle Scholar
  36. Sheridan TB (2000) Function allocation: algorithm, alchemy or apostasy? International Journal of Human-Computer Studies 52 (2):203-216CrossRefGoogle Scholar
  37. Sheridan TB (2012) Human Supervisory Control. In: Handbook of Human Factors and Ergonomics. John Wiley & Sons, Inc., pp 990-1015. doi: 10.1002/9781118131350.ch34
  38. Sibley C, Coyne J, Avvari GV, Mishra M, Pattipati KR (2016a) Supporting Multi-objective Decision Making Within a Supervisory Control Environment. In: Schmorrow DD, Fidopiastis CM (eds) Foundations of Augmented Cognition: Neuroergonomics and Operational Neuroscience: 10th International Conference, AC 2016, Held as Part of HCI International 2016, Toronto, ON, Canada, July 17-22, 2016, Proceedings, Part II. Springer International Publishing, Cham, pp 210-221. doi: 10.1007/978-3-319-39952-2_21 Google Scholar
  39. Sibley C, Coyne J, Baldwin C Pupil dilation as an index of learning. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 2011. vol 1. pp 237-241Google Scholar
  40. Sibley C, Coyne J, Thomas J Demonstrating the Supervisory Control Operations User Testbed (SCOUT). In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 2016b. vol 1. SAGE Publications, pp 1324-1328Google Scholar
  41. Tsai Y-F, Viirre E, Strychacz C, Chase B, Jung T-P (2007) Task performance and eye activity: predicting behavior relating to cognitive workload. Aviation, space, and environmental medicine 78 (Supplement 1):B176-B185Google Scholar
  42. van de Merwe K, van Dijk H, Zon R (2012) Eye movements as an indicator of situation awareness in a flight simulator experiment. The International Journal of Aviation Psychology 22 (1):78-95CrossRefGoogle Scholar
  43. Williams KW (2004) A summary of unmanned aircraft accident/incident data: Human factors implications. DTIC Document, OKLAHOMA CITY, OKGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Naval Research LaboratoryWashington, DCUSA
  2. 2.Embry-Riddle Aeronautical UniversityDaytona BeachUSA

Personalised recommendations