Untangling Operator Monitoring Approaches When Designing Intelligent Adaptive Systems for Operational Environments

  • Ming Hou
  • Cali M. Fidopiastis
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8534)


An Intelligent Adaptive System (IAS) is a synergy between an intelligent interface and adaptive automation technologies capable of context sensitive interaction with operators. A well-designed IAS should enable flexible task allocation between the operator and the machine. Research suggests that the integration of real-time operator state assessment (e.g., performance, psychophysiology) can create a true ‘human-in-the-loop’ system, thereby minimizing deleterious performance effects such as overlooking automation failures and slowly reorienting to tasks. However, these research approaches apply a variety of methodologies to determine sensors, metrics, and overall system design when applied to real world tasks. This paper seeks to untangle these issues such that a more comprehensive framework for systematically evaluating the utility of cognitive state detection methods is attainable.


Intelligent tutoring systems adaptive automation augmented cognition psychophysiological measures cognitive state 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Hou, M., Kobierski, R., Brown, M.: Intelligent adaptive interfaces for the control of multiple UAVs. J. Cogn. Eng. Decis. Making 1(3), 327–362 (2007)CrossRefGoogle Scholar
  2. 2.
    Kaber, D.B., Endsley, M.R.: The Effects of Level of Automation and Adaptive Automation on Human Performance, Situation Awareness and Workload in a Dynamic Control Task. Theor. Iss. Ergon. 5(2), 113–153 (2004)CrossRefGoogle Scholar
  3. 3.
    McCraty, R., Atkinson, M., Tiller, W.A., Rein, G., Watkins, A.D.: The Effects of Emotions on Short-term Power Spectrum Analysis of Heart Rate Variability. Am. J. Cardiol. 76(14), 1089–1093 (1995)CrossRefGoogle Scholar
  4. 4.
    Schnell, T., Keller, M., Macuda, T.: Pilot State Classification and Mitigation in a Fixed and Rotary Wing Platform. Aviat. Space Environ. Med. 78(3), 377 (2007)Google Scholar
  5. 5.
    Figner, B., Murphy, R.O.: Using Skin Conductance in Judgment and Decision Making Research. In: Schulte-Mecklenbeck, M., Kuehberger, A., Ranyard, R. (eds.) A Handbook of Process Tracing Methods for Decision Research, pp. 163–184. Psychology Press, New York (2011)Google Scholar
  6. 6.
    Barr, L., Howrach, H., Popkin, S., Carroll, R.J.: A Review and Evaluation of Emerging Driver Fatigue Detection, Measures and Technologies, US department of Transportation, Washington DC, USA (2009)Google Scholar
  7. 7.
    Dinges, D.F., Mallis, M., Maislin, G., Powell, J.W.: Final Report: Evaluation of Techniques for Ocular Measurement as an Index of Fatigue and as the Basis for Alertness Management. Report No. DOT HS 808 762. National Highway Traffic Safety Administration, Washington, D.C. (1998)Google Scholar
  8. 8.
  9. 9.
    Chen, J.Y.C., Terrence, P.I.: Effects of Tactile Cueing on Concurrent Performance of Military and Robotics Tasks in a Simulated Multitasking Environment. Ergon. 51(8), 1137–1152 (2008)CrossRefGoogle Scholar
  10. 10.
    Fitts, P.M., Jones, R.E., Milton, J.L.: Eye Movements of Aircraft Pilots During Instrument-landing Approaches. Aero Engin. Rev. 9(2), 24–29 (1950)Google Scholar
  11. 11.
    Mathan, S., Whitlow, S., Dorneich, M., Ververs, P., Davis, G.: Neurophysiological Estimation of Interruptibility: Demonstrating Feasibility in a Field Context. In: Schmorrow, D.D., Nicholson, D.M., Drexler, J.M., Reeves, L.M. (eds.) Foundations of Augmented Cognition, 4th edn., pp. 51–58. Strategic Analysis, Arlington (2007)Google Scholar
  12. 12.
    Cummings, M.L.: Technology Impedances to Augmented Cognition. Ergon. Des., 25–27 (2010)Google Scholar
  13. 13.
    St. John, M., Kobus, D.A., Morrison, J.G., Schmorrow, D.: Overview of the DARPA augmented cognition technical integration experiment. International Journal of Human-Computer Interaction 17(2), 131–149 (2004)Google Scholar
  14. 14.
    Fidopiastis, C.M., Drexler, J., Barber, D., Cosenzo, K., Barnes, M., Chen, J.Y., Nicholson, D.: Impact of Automation and Task Load on Unmanned System Operator’s Eye Movement Patterns. In: Schmorrow, D.D., Estabrooke, I.V., Grootjen, M. (eds.) FAC 2009. LNCS (LNAI), vol. 5638, pp. 229–238. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  15. 15.
    Fidopiastis, C.M., Nicholson, D.M.: Neuroergonomics: From Theory to Practice. In: Marek, T., Karwowski, W., Rice, V. (eds.) Advancing the Understanding of Human Performance: Neuroergonomics, Human Factors Design, and Special Populations, ch. 36, pp. 354–359. CRC Press, Boca Raton (2010)Google Scholar
  16. 16.
    Fidopiastis, C.M.: Theoretical Transpositions in Brain Function and the Underpinnings of Augmented Cognition. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) FAC 2011. LNCS (LNAI), vol. 6780, pp. 153–158. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  17. 17.
    Schnell, T., Cornwall, R., Walwanis, M., Grubb, J.: The Quality of Training Effectiveness Assessment (QTEA) Tool Applied to the Naval Aviation Training Context. In: Schmorrow, D.D., Estabrooke, I.V., Grootjen, M. (eds.) FAC 2009. LNCS (LNAI), vol. 5638, pp. 640–649. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  18. 18.
    Stanney, K.M., Hale, K.S.: Today’s Competitive Objective: Augmenting Human Performance. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) FAC 2011. LNCS (LNAI), vol. 6780, pp. 628–635. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  19. 19.
  20. 20.
    Dorneich, M.C., Mathan, S., Ververs, P.M., Whitlow, S.D.: Cognitive State Estimation in Mobile Environments. In: Schmorrow, D.D., Stanney, K.M. (eds.) Augmented Cognition: A Practitioner’s Guide, pp. 75–111 (2008)Google Scholar
  21. 21.
    Scerbo, M.W.: Adaptive Automation. In: Parasuraman, R., Rizzo, M. (eds.) Neuroergonomics: The Brain at Work, ch. 26, pp. 239–252. Oxford University Press, New York (2007)Google Scholar
  22. 22.
    Fairclough, S.H.: Fundamentals of Physiological Computing. Interact. Comput. 21(1-2), 133–145 (2009)CrossRefGoogle Scholar
  23. 23.
    Sauer, J., Nickel, P., Wastell, D.: Designing Automation for Complex Work Environments Under Different Levels of Stress. App. Ergo. 44(1), 119–127 (2013)CrossRefGoogle Scholar
  24. 24.
    Berka, C., Levendowski, D.J., Lumicao, M.N., Yau, A., Davis, G., Zivkovic, V.T., Craven, P.L.: EEG Correlates of Task Engagement and Mental Workload in Vigilance, Learning, and Memory Tasks. Aviat. Space Environ. Med. 78(suppl. 1), B231–B244 (2007)Google Scholar
  25. 25.
    Wilson, G.F., Russell, C.A.: Performance Enhancement in an Uninhabited Air Vehicle Task using Psychophysiologically Determined Adaptive Aiding. Hum. Fact. 49(6), 1005–1018 (2007)Google Scholar
  26. 26.
    Boucsein, W., Haarmann, A., Schaefer, F.: Combining Skin Conductance and Heart Rate Variability for Adaptive Automation During Simulated IFR Flight. In: Harris, D. (ed.) HCII 2007 and EPCE 2007. LNCS (LNAI), vol. 4562, pp. 639–647. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  27. 27.
    Schaefer, F., Haarmann, A., Boucscein, W.: The Usability of Cardiovascular and Electrodermal Measures for Adaptive Automation. In: Westerink, J., Ouwerkerk, M., Overbeek, T.J.M., Pasveer, W.F. (eds.) Probing Experience: From Assessment of User Emotions and Behaviour to Development of Products, vol. ch. 20, pp. 235–243. Springer, Netherlands (2008)CrossRefGoogle Scholar
  28. 28.
    Scerbo, M.W., Freeman, F.G., Mikulka, P.J., Parasuraman, R., Di Nocero, F., Prinzel III, L.J.: The efficacy of psychophysiological measures for implementing adaptive technology. In: National Aeronautics and Space Administration, vol. 211018, Langley Research Center (2001)Google Scholar
  29. 29.
    Fuchs, C., Aschenbruck, N., Martini, P., Wieneke, M.: Indoor Tracking for Mission Critical Scenarios: A Survey. J. of Perv. and Mobi. Comp. 7(1), 1–15 (2011)Google Scholar
  30. 30.
    Schnell, T., Macuda, T., Keller, M.: Sensor Integration to Characterize Operator State. In: Schmorrow, D.D., Stanney, K.M. (eds.) Augmented Cognition: A Practitioner’s Guide, pp. 41–74. Human Factors and Ergonomics Society (HFES), Santa Monica (2008)Google Scholar
  31. 31.
    Sarter, N., Sarter, M.: Neuroergonomics: Opportunities and Challenges of Merging Cognitive Neuroscience with Cognitive Ergonomics. Theor. Iss. Ergon. Sci. 4(1-2), 142–150 (2003)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Ming Hou
    • 1
  • Cali M. Fidopiastis
    • 2
  1. 1.Defence Research & Development Canada-TorontoTorontoCanada
  2. 2.University of Alabama at BirminghamBirminghamUSA

Personalised recommendations