A mixed integer programming (MIP) model for evaluating navigation and task planning of human–robot interactions (HRI)

  • Mehmet Burak ŞenolEmail author
Original Research Paper


Exercise of robotics in many applications brings in concerns of human–robot interaction. This paper offers a mathematical model-based mission planning tool for optimizing operator workload and platform utilization in human/multi-robot (H/M-R) teams. None of the earlier methods consistently predicts fan-out (number and configuration of robots that can be operated simultaneously and effectively, a critical H/M-R design decision). In this research, a mixed integer programming (MIP) model and solution framework are proposed to provide better estimates of fan-out while explicitly considering the performance, mission characteristics, objective and task/environment complexity. The extent of each robot’s waiting time is restricted by a utilization threshold in the MIP model. The effect of environment’s complexity on the task effectiveness is considered, where robots’ performances deteriorate during switch and neglect times. Simulation results show that fan-out effect is dependent on interaction efficiency, neglect tolerance, as well as other parameters. Performance is most sensitive to environment’s complexity and least sensitive to utilization threshold. In addition, the MIP model reveals optimal control sequence of robots to prevent switching confusions and maximize team performance. Empirical evaluations show that this approach holds great promise for real-world scenarios.


Fan-out Human–robot interaction (HRI) Robot effectiveness Mixed integer programming (MIP) Optimization Optimal control sequence Ergonomics 



This research did not receive any specific grant from funding agencies in the public, commercial or not-for-profit sectors.


  1. 1.
    Liu C, Tomizuka M (2014) Modeling and controller design of cooperative robots in workspace sharing human–robot assembly teams. In: Proceedings of the 2014 IEEE/RSJ international conference on intelligent robots and systems, Chicago, IL, USA, pp 1386–1391Google Scholar
  2. 2.
    Chai T, Ding J, Yu G, Wang H (2014) Integrated optimization for the automation systems of mineral processing. IEEE Trans Autom Sci Eng 11(4):965–982CrossRefGoogle Scholar
  3. 3.
    Kim HJ, Lee JH, Lee TE (2014) Non-cyclic scheduling of a wet station. IEEE Trans Autom Sci Eng 11(4):1262–1274MathSciNetCrossRefGoogle Scholar
  4. 4.
    Shibata T, Wada K, Saito T, Tanie K (2005) Human interactive robot for psychological enrichment and therapy. In: Proceedings of the AISB 2005 symposium on robot companion hard problem and open challenges in human robot interaction, pp 98–109Google Scholar
  5. 5.
    Husain A, Jones H, Kannan B, Wong U, Pimentel T, Tang S, Daftry S, Huber S, Whittaker WL (2012) Mapping planetary caves with an autonomous, heterogeneous robot team. IEEE, pp 1–13Google Scholar
  6. 6.
    Chen F, Sekiyama K, Cannella F, Fukuda T (2014) Optimal subtask allocation for human and robot collaboration within hybrid assembly system. IEEE Trans Autom Sci Eng 11(4):1065–1075CrossRefGoogle Scholar
  7. 7.
    Chen F, Sekiyama K, Cannella F, Fukuda T (2013) Cooperative human robot interaction systems: IV. Communication of shared plans with Naive humans using gaze and speech. In: Proceedings of the 2013 IEEE/RSJ international conference on intelligent robots and systems (IROS), Tokyo, Japan. IEEE, pp 129–136Google Scholar
  8. 8.
    Abbott KA, Slotte SM, Stimson DK (1996) Federal Aviation Administration Human Factors team report on: the interfaces between flightcrews and modern flight deck systems. Federal Aviation Administration, Washington DC, Technical Report, June 1996, pp D1–D3.
  9. 9.
    Espinosa JA, Cadiz J, Rico-Gutierrez L, Kraut RE, Scherlis W, Lautenbacher G (2000) Coming to the wrong decision quickly: why awareness tools must be matched with appropriate tasks. In: Proceedings of the human factors in computing systems, computer–human interaction (CHI) conference. ACM PressGoogle Scholar
  10. 10.
    Pires JN (2005) Semi-autonomous manufacturing systems: the role of human–machine interface software and of the manufacturing tracking software. Mechatron Int J 15(10):1191–1205CrossRefGoogle Scholar
  11. 11.
    Neto P, Pires JN, Moreira AP (2009) Accelerometer-based control of an industrial robotic arm. In: Proceedings of the 18th IEEE international symposium on robot and human interactive communication, Toyama, Japan, pp 1192–1197Google Scholar
  12. 12.
    Hillenbrand U, Brunner B, Borst C, Hirzinger G (2004) The robutler: a vision-controlled hand-arm system for manipulating bottles and glasses. In: Proceedings of the 35th international symposium on robotics, Paris, FranceGoogle Scholar
  13. 13.
    Yamamoto S, Valin JM, Nakadai K, Rouat J, Michaud F, Ogata T, Okuno HG (2005) Enhanced robot speech recognition based on microphone array source separation and missing feature theory. In: Proceedings of the 2005 ieee international conference on robotics and automation (ICRA 2005), Barcelona, Spain, pp 1477–1482Google Scholar
  14. 14.
    Burke JL, Murphy RR, Rogers E, Lumelsky VJ, Scholtz J (2004) Final report for the DARPA/NSF interdisciplinary study on human–robot interaction. IEEE Syst Man Cybern C Appl Rev Spec Issue Hum Robot Interact 34(2):103–112Google Scholar
  15. 15.
    Schreckenghost DK (1999) Checklists for human–robot collaboration during space operations. In: Proceedings of the 43rd annual meeting of the human factors and ergonomics society, Santa Monica, CA, pp 46–50Google Scholar
  16. 16.
    Scholtz J, Bahrami S (2003) Human–Robot interaction: development of an evaluation methodology for the bystander role of interaction. Retrieved 23 Feb 2004, from
  17. 17.
    Murphy RR (2004) Human–robot interaction in rescue robotics. IEEE Syst Man Cybern C Appl Rev Spec Issue Hum Robot Interact 34(2):138–153MathSciNetGoogle Scholar
  18. 18.
    Fong T, Kaber D, Lewis M, Scholtz J, Shultz A, Steinfeld A (2004) Common metrics for human–robot interaction. In: Proceedings of IEEE international conference on intelligent robots and systems, Sendai, JapanGoogle Scholar
  19. 19.
    Chen JYC, Haas EC, Pillalamarri K, Jacobson CN (2006) Human–robot interface: issues in operator performance interface design, and technologies. Army Research Laboratory, ARL-TR-3834Google Scholar
  20. 20.
    Goodrich M, Olsen D (2003) Seven principles of efficient human–robot interaction. In: Proceedings of the 2003 IEEE international conference on systems, man and cybernetics, Washington DC, pp 3943–3948Google Scholar
  21. 21.
    Dixon SR, Wickens CD, Chang D (2003) Comparing quantitative model predictions to experimental data in multiple-UAV flight control. In: Proceedings of the human factors and ergonomics society 47th annual meeting Santa Monica, CA, pp 104–108Google Scholar
  22. 22.
    Schipani SP (2003) An evaluation of operator workload during partially-autonomous vehicle operations. In: Proceedings of PerMISGoogle Scholar
  23. 23.
    Gombolay M, Bair A, Huang C, Shah J (2017) Computational design of mixed-initiative human–robot teaming that considers human factors: situational awareness, workload, and workflow preferences. Int J Robot Res 36(5–7):597–617CrossRefGoogle Scholar
  24. 24.
    H. Tempelmeier,”Simultaneous buffer and workload optimization for asynchronous flow production systems. Department of Production Management, University of Cologne, Koeln, Germany.
  25. 25.
    Crandall JW, Cummings M, Penna MD, de Jong PMA (2011) Computing the effects of operator attention allocation in human control of multiple robots. IEEE Trans Syst Man Cybern A Syst Hum 41(3):385–397CrossRefGoogle Scholar
  26. 26.
    Crandall JW, Cummings M (2007) Developing performance metrics for the supervisory control of multiple robots. In: Proceedings of the 2007 ACM/IEEE conference on human robot interaction, Arlington, VAGoogle Scholar
  27. 27.
    Olsen D, Goodrich M (2003) Metrics for evaluating human–robot interactions. In: Proceedings of PERMISGoogle Scholar
  28. 28.
    Olsen D, Wood S (2004) Fan-out: measuring human control of multiple robots. In: Proceedings of the SIGCHI conference on human factors in computer systems, Vienna, Austria, pp 231–238Google Scholar
  29. 29.
    Crandall JW, Goodrich M, Olsen D, Nielsen C (2005) Validating human–robot interaction schemes in multitasking environments. IEEE Trans Syst 35(4):438–448Google Scholar
  30. 30.
    Crandall JW, Cummings M (2007) Identifying predictive metrics for supervisory control of multiple robots. IEEE Trans Robot Autom 23(5):942–951CrossRefGoogle Scholar
  31. 31.
    Mau S, Dolan J (2006) Scheduling to minimize downtime in human–multirobot supervisory control. In: Proceedings of the 5th international workshop on planning and scheduling for space, pp 268–277Google Scholar
  32. 32.
    Mau S, Dolan J (2007) Scheduling for humans in multirobot supervisory control. In: Proceedings of the 2007 IEEE/RSJ international conference on intelligent robots and systems, San Diego, CA, pp 1637–1643Google Scholar
  33. 33.
    Donmez B, Pina P, Cummings M (2008) Evaluation criteria for human automation performance metrics. In: Proceedings of the PerMISGoogle Scholar
  34. 34.
    Geoffrion AM (1972) Generalized benders decomposition. J Optim Theory Appl 10(4):237–260MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Duran MA, Grossmann LE (1986) An outer-approximation algorithm for a class of mixed-integer non-linear programs. Math Program 36(3):307–339CrossRefzbMATHGoogle Scholar
  36. 36.
    Fletcher R, Leyffer S (1994) Solving mixed integer nonlinear programs by outer approximation. Math Program 66(1–3):327–349MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    Westerlund T, Petersson F (1995) An extended cutting plane method for solving convex MINLP problems. Comput Chem Eng 19(1):131–136CrossRefGoogle Scholar
  38. 38.
    Borchers B, Mitchell JE (1991) An improved branch and bound algorithm for mixed integer non-linear programs.RPI Math Report No 200Google Scholar
  39. 39.
    Leyffer S (2001) Integrating SQP and branch-and-bound for mixed integer nonlinear programming. Comput Optim Appl 18:295–309MathSciNetCrossRefzbMATHGoogle Scholar
  40. 40.
    Floudas CA, Gounaris CE (2008) A review of recent advances in global optimization. J Glob Optim 45:3–38MathSciNetCrossRefzbMATHGoogle Scholar
  41. 41.
    Floudas CA, Akrotirianakis IG, Caratzoulas S, Meyer CA, Kallrath J (2005) Global optimization in the 21st century: advances and challenges. Comput Chem Eng 29:1185–1202CrossRefGoogle Scholar
  42. 42.
    Liao T, Socha K, Montes de Oca MA, Stutzle T, Dorigo M (2014) Ant colony optimization for mixed-variable optimization problems. IEEE Trans Evol Comput 18(4):503–518CrossRefGoogle Scholar
  43. 43.
    IBM ILOG CPLEX Optimization Studio 12.8. Accessed 11 Dec 2018
  44. 44.
    Hart SG, Staveland LE (1988) Development of the NASA-TLX (Task Load Index): results of empirical and theoretical research. In: Hancock PA, Meshkati N (eds) Human mental workload. North Holland, Amsterdam, pp 139–183CrossRefGoogle Scholar
  45. 45.
    Sert O, Freedy E, McDonough J, Weltman G, Tambe M, Gupta T, Grayson W, Cabrera P (2008) Multiagent adjustable autonomy framework (MAAF) for multi-robot, multi-human teams. IEEE, pp 498–505Google Scholar
  46. 46.
    Sherali HD, Adams WP (1998) Reformulation–linearization techniques for discrete optimization problems. In: Du DZ, Pardalos PM (eds) Handbook of combinatorial optimization. Springer, BostonGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Industrial EngineeringGazi UniversiyMaltepe, AnkaraTurkey

Personalised recommendations