Performance Measurement and Evaluation in Human-in-the-Loop Simulations

  • Ling Rothrock


A prerequisite for designers of the complex systems is a proper understanding of human performance characteristics. While human factor texts provide some insights into basic performance issues, the emergence of highly automated computing systems have fundamentally altered the way humans work. The purpose of this paper is to present an approach to quantify and analyze human performance in human-in-the-loop simulations based on over ten years of research experience. The approach is centered on a measurement construct, called a time window, which enables a functional relationship between constraints on operator activities and time availability. A blackboard model is presented as the mechanism to generate, maintain, and complete time windows. To demonstrate the utility of time windows, an existing implementation in a real-time human-in-the-loop simulation is also described. An extension of time windows to measure team performance is also discussed. Using time window outcomes, samples of previous analyses are presented to exhibit the potential of the construct.


Time Window Operator Action Call Center Team Performance Teamwork Skill 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. Adeli H, Yu G (1995) An integrated computing environment for solution of complex engineering problems using the object-oriented programming paradigm and a blackboard architecture. Comput Struct 54(2):255–265CrossRefGoogle Scholar
  2. Aksin O, Armony M, Mehrotra V (2007) The modern call-center: a multi-disciplinary perspective on operations management research. Prod Operat Manag 16(6):655–688Google Scholar
  3. Allen JF (1983) Maintaining Knowledge about Temporal Intervals. Commun ACM 26(11):832–843MATHCrossRefGoogle Scholar
  4. Anton A (1997) Call center management: by the numbers. Purdue University Press, West LafayetteGoogle Scholar
  5. Atlason J, Epelman MA, Henderson SG (2004) Call center staffing with simulation and cutting plane methods. Ann Oper Res 127:333–358MathSciNetMATHCrossRefGoogle Scholar
  6. Avramidis A, Chan W, L’Ecuyer P (2009) Staffing multi-skill call centers via search methods and a performance approximation. IIE Trans 41(6):483–497CrossRefGoogle Scholar
  7. Beach LR, Lipshitz R (1993) Why classical decision theory is an inappropriate standard for evaluating and aiding most human decision making. In: Klein GA, Orasanu J, Calderwood R, Zsambok C (eds) Decision making in action: models and methods. Ablex Publishing Corp, Norwood, pp 21–35Google Scholar
  8. Booch G, Rumbaugh J, Jacobson I (1999) The unified modeling language user guide. Addison-Wesley, ReadingGoogle Scholar
  9. Caldeira JD (1980) Parametric assumptions of some “nonparametric” measures of sensory efficiency. Hum Factors 22(1):119–120Google Scholar
  10. Cannon-Bowers JA, Salas E (1997) A framework for developing team performance measures in training. In: Brannick M, Salas E, Prince C (eds) Team performance assessment and measurement: theory, methods, and applications. Erlbaum, Hillsdale, pp 45–62Google Scholar
  11. Cattell RB (1966) The meaning and strategic use of factor analysis. In: Cattell RB (ed) Handbook of multivariate experimental psychology. Rand McNally, ChicagoGoogle Scholar
  12. Cezik MT, L’Ecuyer P (2008) Staffing Multiskill Call Centers via Linear Programming and Simulation. Manag Sci 54(2):310–323CrossRefGoogle Scholar
  13. Chesbrough H, Spohrer J (2006) A research manifesto for services science. Commun ACM 49(7):35–40CrossRefGoogle Scholar
  14. Chu RW, Jones PM, Mitchell CM (1991) The Georgia tech payload operations control center simulation: design and implementation in C++. In: Proceedings of the object-oriented simulation track of the 1991 society for computer simulation multiconference, Anaheim, CAGoogle Scholar
  15. Coovert MD, Cannon-Bowers JA, Salas E (1990) Applying mathematical modeling technology to the study of team training and performance. Paper presented at the 12th annual interservice/industry training systems conferenceGoogle Scholar
  16. Craig A (1979) Nonparametric measures of sensory efficiency for sustained monitoring tasks. Hum Factors 21(1):69–78MathSciNetGoogle Scholar
  17. Endsley MR (2006) Situation awareness. In: Salvendy G (ed) Handbook of human factors and ergonomics, 3rd edn. John Wiley and Sons, Inc, New York, pp 528–542CrossRefGoogle Scholar
  18. Erman DL, Hayes-Roth F, Lesser VR, Reddy D (1980) The HEARSAY-II speech understanding system: integrating knowledge to resolve uncertainty. ACM Comput Surv 12:213–253CrossRefGoogle Scholar
  19. Fleming JH, Coffman C, Harter JK (2005) Manage your human sigma. Harv Bus Rev 83(7/8):106–114Google Scholar
  20. Gabbay DM, Hodkinson I, Reynolds M (1994) Temporal logic: mathematical foundations and computational aspects. Oxford University Press, OxfordMATHGoogle Scholar
  21. Gans N, Koole G, Mandelbaum A (2003) Telephone call centers: Tutorial, review, and research prospects. Manuf Serv Oper Manag 5(2):79–142CrossRefGoogle Scholar
  22. Garnet O, Mandelbaum A, Reiman M (2002) Designing a call center with impatient customers. Manuf Serv Oper Manag 4(3):208–227CrossRefGoogle Scholar
  23. Gilmore A, Moreland L (2000) Call centres: how can service quality be managed? Ir Market Rev 13(1):3–11Google Scholar
  24. Green PE (1978) Analyzing multivariate data. Dryden Press, HinsdaleGoogle Scholar
  25. Green DM, Swets JA (1966) Signal detection theory and psychophysics. Wiley, New YorkGoogle Scholar
  26. Greeno JG (1994) Gibson’s affordances. Psychol Rev 101(2):336–342CrossRefGoogle Scholar
  27. Greeno JG (1998) The situativity of knowing, learning, and research. Am Psychol 53(1):5–26CrossRefGoogle Scholar
  28. Greeno JG, Moore JL (1993) Situativity and symbols: response to Vera and Simon. Cogn Sci 17:49–60CrossRefGoogle Scholar
  29. Hammond KR (1986) Generalization in operational contexts: what does it mean? can it be done? IEEE Trans Syst Man Cybern, SMC 16(3):428–433MathSciNetCrossRefGoogle Scholar
  30. Howie DE, Vicente KJ (1998) Measures of operator performance in complex, dynamic microworlds: advancing the state of the art. Ergonomics 41(4):485–500CrossRefGoogle Scholar
  31. Jones PM, Chu RW, Mitchell CM (1995) A methodology for human-machine systems research: knowledge engineering, modeling, and simulation. IEEE Trans Syst Man Cybern 25(7)Google Scholar
  32. Kirlik A (1995) Requirements for psychological models to support design: toward ecological task analysis. In: Flach JM, Hancock P, Caird J, Vicente KJ (eds) Global perspectives on the ecology of human-machine systems. Erlbaum, Hillsdale, pp 68–120Google Scholar
  33. Laudeman IV, Palmer EA (1995) The quantitative measurement of observed workload in the analysis of aircrew performance. Int J Aviat Psychol 5(2):187–197CrossRefGoogle Scholar
  34. Ma J, Kim N, Rothrock L (2011) Performance assessment in an interactive call center workforce simulation. Simul Model Pract Theory 19:227–238CrossRefGoogle Scholar
  35. Marks MA, Sabella MJ, Burke CS, Zaccaro SJ (2002) The impact of cross-training on team effectiveness. J Appl Psychol 87(1):3–13CrossRefGoogle Scholar
  36. Mehrotra V, Fama J (2003) Call center simulation modeling: methods, challenges, and opportunities. Paper presented at the 2003 winter simulation conferenceGoogle Scholar
  37. Militello LG, Hutton RJB (1998) Applied cognitive task analysis (acta): a practitioner’s toolkit for understanding task demands. Ergonomics 41(11):1618–1641CrossRefGoogle Scholar
  38. Moray N, Dessouky MI, Kijowski BA, Adapathya R (1991) Strategic behavior, workload, and performance in task scheduling. Hum Factors 33(6):607–629Google Scholar
  39. Nii HP (1986) Blackboard systems: the blackboard model of problem solving and the evolution of blackboard architectures. Part I. AI Mag 7:38–53Google Scholar
  40. Raby M, Wickens CD (1994) Strategic workload management and decision biases in aviation. Int J Aviat Psychol 4(3):211–240CrossRefGoogle Scholar
  41. Rothrock L (2001) Using time windows to evaluate operator performance. Int J Cogn Ergonomics 5(1):1–21CrossRefGoogle Scholar
  42. Rothrock L, Cohen A, Yin J, Thiruvengada H, Nahum-Shani I (2009) Analyses of team performance in a dynamic task environment. Appl Ergonomics 40(4):699–706CrossRefGoogle Scholar
  43. Rubin KS, Jones PM, Mitchell CM (1988) OFMspert: inference of operator intentions in supervisory control using a blackboard architecture. IEEE Trans Syst Man Cybern SMC 18(4):618–637CrossRefGoogle Scholar
  44. Sanderson PM, Verhage AG, Fuld RB (1989) State-space and verbal protocol methods for studying the human operator in process control. Ergonomics 32:1343–1372CrossRefGoogle Scholar
  45. Sharp D (2003) Call center operation: design. Digital Science Operation, and Maintenance, New YorkGoogle Scholar
  46. Smith-Jentsch KA, Johnston JH, Payne SC (1998) Measuring team-related expertise in complex environments. In: Cannon-Bowers JA, Salas E (eds) Making decisions under stress: implications for individual and team training. APA, Washington, D.C., pp. 61–87CrossRefGoogle Scholar
  47. Smith-Jentsch KA, Zeisig RL, Acton B, McPherson JA (1998) Team dimensional training: a strategy for guided team self-correction. In: Cannon-Bowers JA, Salas E (eds) Making decisions under stress: implications for individual and team training. APA, Washington, D.C., pp 271–297CrossRefGoogle Scholar
  48. Suchman LA (1987) Plans and situated actions: the problem of human machine communication. Cambridge University Press, CambridgeGoogle Scholar
  49. Swets JA (1996) Signal detection theory and ROC analysis in psychology and diagnostics: collected papers. Lawrence Erlbaum Assoc, MahwahMATHGoogle Scholar
  50. Tanir O, Booth RJ (1999) Call center simulation in Bell Canada. Paper presented at the 1999 winter simulation conference. Phoenix, AZGoogle Scholar
  51. Thiruvengada H, Rothrock L (2007) Time window-based performance measures: a framework to measure team performance in dynamic environments. Cogn Technol Work 9(2):99–108CrossRefGoogle Scholar
  52. Vicente KJ, Rasmussen JR (1992) Ecological interface design: theoretical foundations. IEEE Trans Syst, Man Cybern 22(4):589–606CrossRefGoogle Scholar
  53. Vranes S, Lucin M, Stanojevic M, Stevanovic V, Subasic P (1991) Blackboard metaphor in tactical decision making. Eur J Oper Res 61:86–97CrossRefGoogle Scholar
  54. Wallace CM, Eagleson G, Waldersee R (2000) The sacrificial HR strategy in call centres. Int J Serv Ind Manag 11(2):174–184CrossRefGoogle Scholar
  55. Wickens CD, Hollands JG (2000) Engineering psychology and human performance, 3rd edn. Prentice Hall, Upper Saddle RiverGoogle Scholar
  56. Wickens CD, Kessel C (1979) The effects of participatory mode and task workload on the detection of dynamic system failures. IEEE Trans Syst Man Cybern 9(1):24–34CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  1. 1.The Harold and Inge Marcus Department of Industrial and Manufacturing EngineeringThe Pennsylvania State UniversityUniversity ParkUSA

Personalised recommendations