Collaborative Autonomy Between High-Level Behaviors and Human Operators for Control of Complex Tasks with Different Humanoid Robots

  • David C. Conner
  • Stefan Kohlbrecher
  • Philipp Schillinger
  • Alberto Romay
  • Alexander Stumpf
  • Spyros Maniatopoulos
  • Hadas Kress-Gazit
  • Oskar von Stryk
Chapter
Part of the Springer Tracts in Advanced Robotics book series (STAR, volume 121)

Abstract

This chapter discusses the common reactive high-level behavioral control system used by Team ViGIR and Team Hector on separate robots in the 2015 DARPA Robotics Challenge (DRC) Finals. We present an approach that allows one or more human operators to share control authority with a high-level behavior controller in the form of a finite state machine (automaton). This collaborative autonomy leverages the relative strengths of the robotic system and the (remote) human operators; it increases reliability of the human-robot team performance and decreases the task completion time. This approach is well-suited to disaster scenarios due to the unstructured nature of the environment. The system allows the operators to adjust the robotic system’s autonomy on-the-fly in response to changing circumstances, and to modify pre-defined behaviors as needed. To enable these high-level behaviors, we introduce our system designs for several of the lower-level system capabilities such as footstep planning and template-based object manipulation. We evaluate the proposed approach in the context of our two teams’ participation in the DRC Finals using two different humanoid platforms, and in systematic experiments conducted in the lab afterward. We present a discussion about the lessons learned during the DRC, especially those related to transitioning between operator-centered control and behavior-centered control during competition. Finally, we describe ongoing research beyond the DRC that extends the systems developed during the DRC. All of our described software is available as open source software.

Notes

Acknowledgements

This project was supported in part by the Defense Advanced Research Projects Agency (DARPA) under Air Force Research Lab (AFRL) contract FA8750-12-C-0337 to TORC Robotics; Team ViGIR would like to thank TORC Robotics for their support and overall project management. Team Hector would like to thank TORC Robotics for opening their doors and providing logistical support during final testing and transportation. Team ViGIR and Team Hector would like to thank all team members; their contribution and support enabled the realization of this work. We would also like to thank DARPA and its support staff for a well run competition, Boston Dynamics, Inc. for their support with the Atlas robot, ROBOTIS, Inc. for their support with THORMANG robot, and the Open Source Robotics Foundation (OSRF) for their support of ROS and Gazebo. The teams would also like to thank the contributors and maintainers of MoveIt! and the SMACH High-level Executive.

References

  1. Banerjee, N., Long, X., Du, R., Polido, F., Feng, S., Atkeson, C. G., et al. (2015). Human-supervised control of the atlas humanoid robot for traversing doors. In 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids) (pp. 722–729). http://dx.doi.org/10.1109/HUMANOIDS.2015.7363442.
  2. Berenson, D., Srinivasa, S. S., & Kuffner, J. (2011). Task space regions: A framework for pose-constrained manipulation planning. The International Journal of Robotics Research, 30(12), 1435–1460. http://dx.doi.org/10.1177/0278364910396389.
  3. Birkenkampf, P., Leidner, D., & Borst, C. (2014). A knowledge-driven shared autonomy human-robot interface for tablet computers. In 2014 14th IEEE-RAS International Conference on Humanoid Robots (Humanoids) (pp. 152–159). http://dx.doi.org/10.1109/HUMANOIDS.2014.7041352.
  4. Bloem, R., Jobstmann, B., Piterman, N., Pnueli, A., & Saar, Y. (2012). Synthesis of reactive(1) designs. Journal of Computer and System Sciences, 78(3), 911–938. http://dx.doi.org/10.1016/j.jcss.2011.08.007.
  5. Bohren, J., & Cousins, S. (2010). The SMACH high-level executive [ROS News]. IEEE Robotics Automation Magazine, 17(4), 18–20. http://dx.doi.org/10.1109/MRA.2010.938836.
  6. Cheng, G., & Zelinsky, A. (2001). Supervised autonomy: A framework for human-robot systems development. Autonomous Robots, 10(3), 251–266. http://dx.doi.org/10.1023/A:1011231725361.
  7. Chitta, S., Sucan, I., & Cousins, S. (2012). MoveIt! [ros topics]. IEEE Robotics Automation Magazine, 19(1), 18–19. http://dx.doi.org/10.1109/MRA.2011.2181749.
  8. Conner, D. C., & Willis, J. (2017). Flexible navigation: Finite state machine-based integrated navigation and control for ROS enabled robots. In IEEE SoutheastCon 2017 (pp. 1–8). http://dx.doi.org/10.1109/SECON.2017.7925266.
  9. Crandall, J., & Goodrich, M. (2001). Experiments in adjustable autonomy. In 2001 IEEE International Conference on Systems, Man, and Cybernetics (Vol. 3, pp. 1624–1629). http://dx.doi.org/10.1109/ICSMC.2001.973517.
  10. Desai, M., & Yanco, H. (2005). Blending human and robot inputs for sliding scale autonomy. In IEEE International Workshop on Robot and Human Interactive Communication, 2005. ROMAN 2005 (pp. 537–542). http://dx.doi.org/10.1109/ROMAN.2005.1513835.
  11. DRC Teams. (2015). What happened at the DARPA Robotics Challenge? Retrieved September 23, 2015, from http://www.cs.cmu.edu/~cga/drc/events.
  12. Ehlers, R., Raman, V., & Finucane, C. (2013). Slugs GR(1) synthesizer. https://github.com/VerifiableRobotics/slugs.
  13. Fallon, M., & Marion, P. (2016). Private communication.Google Scholar
  14. Fallon, M., et al. (2015a). An architecture for online affordance-based perception and whole-body planning. Journal of Field Robotics, 32(2), 229–254. http://dx.doi.org/10.1002/rob.21546.
  15. Fallon, M. F., Marion, P., Deits, R., Whelan, T., Antone, M., McDonald, J., et al. (2015b). Continuous humanoid locomotion over uneven terrain using stereo fusion. In 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids) (pp. 881–888). https://doi.org/10.1109/HUMANOIDS.2015.7363465.
  16. Feix, T., Bullock, I., & Dollar, A. (2014). Analysis of human grasping behavior: Correlating tasks, objects and grasps. IEEE Transactions on Haptics, 7(4), 430–441. http://dx.doi.org/10.1109/TOH.2014.2326867.
  17. Fong, T. W., & Nourbakhsh, I. (2004). Peer-to-peer human-robot interaction for space exploration. In AAAI Fall Symposium. AAAI. http://ri.cmu.edu/pub_files/pub4/fong_terrence_w_2004_1/fong_terrence_w_2004_1.pdf.
  18. Fong, T. W., Thorpe, C., & Baur, C. (1999). Collaborative control: A robot-centric model for vehicle teleoperation. In AAAI 1999 Spring Symposium: Agents with Adjustable Autonomy. http://ri.cmu.edu/pub_files/pub1/fong_terry_1999_1/fong_terry_1999_1.pdf.
  19. Gibson, J. J. (1977). The theory of affordances. In R. Shaw & J. Bransford (Eds.), Perceiving, acting, and knowing: Toward an ecological psychology, Hilldale, USA (pp. 67–82).Google Scholar
  20. Goodrich, M. A., Crandall, J. W., & Barakova, E. (2013). Teleoperation and beyond for assistive humanoid robots. Reviews of Human Factors and Ergonomics, 9(1), 175–226. https://doi.org/10.1177/1557234X13502463.
  21. Hart, S., Dinh, P., Yamokoski, J., Wightman, B., & Radford, N. (2014). Robot task commander: A framework and IDE for robot application development. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2014) (pp. 1547–1554). http://dx.doi.org/10.1109/IROS.2014.6942761.
  22. Hart, S., Dinh, P., & Hambuchen, K. (2015). The affordance template ROS package for robot task programming. In 2015 IEEE International Conference on Robotics and Automation (ICRA) (pp. 6227–6234). http://dx.doi.org/10.1109/ICRA.2015.7140073.
  23. Hebert, P., et al. (2015). Mobile manipulation and mobility as manipulation—design and algorithms of robosimian. Journal of Field Robotics, 32(2), 255–274.  https://doi.org/10.1002/rob.21566.
  24. Hornung, A., Dornbush, A., Likhachev, M., & Bennewitz, M. (2012). Anytime search-based footstep planning with suboptimality bounds. In 2012 12th IEEE-RAS International Conference on Humanoid Robots (Humanoids) (pp. 674–679). IEEE.Google Scholar
  25. Hornung, A., Wurm, K. M., Bennewitz, M., Stachniss, C., & Burgard, W. (2013). OctoMap: An efficient probabilistic 3D mapping framework based on octrees. Autonomous Robots. http://dx.doi.org/10.1007/s10514-012-9321-0, software available at http://octomap.github.com.
  26. Huang, H. M., Messina, E., & Albus, J. (2007). Autonomy levels for unmanned systems (alfus) framework volume II: Framework models version 1.0. NIST Special Publication 1011-II-1.0, NIST. URL http://www.nist.gov/el/isd/ks/autonomy_levels.cfm.
  27. Johnson, M., Bradshaw, J., Feltovich, P., Jonker, C., van Riemsdijk, B., & Sierhuis, M. (2011a). The fundamental principle of coactive design: Interdependence must shape autonomy. In M. De Vos, N. Fornara, J. Pitt & G. Vouros (Eds.), Coordination, Organizations, Institutions, and Norms in Agent Systems VI (Vol. 6541, pp. 172–191). Lecture notes in computer science. Berlin, Heidelberg: Springer. http://dx.doi.org/10.1007/978-3-642-21268-0_10.
  28. Johnson, M., Bradshaw, J. M., Feltovich, P. J., Jonker, C. M., Van Riemsdijk, B., & Sierhuis, M. (2011b). The fundamental principle of coactive design: Interdependence must shape autonomy. In Coordination, Organizations, Institutions, and Norms in Agent Systems VI (pp. 172–191). Berlin, Heidelberg: Springer. https://doi.org/10.1007/978-3-642-21268-0_10.
  29. Johnson, M., Shrewsbury, B., Bertrand, S., Wu, T., Duran, D., Floyd, M., et al. (2015). Team IHMC’s lessons learned from the DARPA robotics challenge trials. Journal of Field Robotics, 32(2), 192–208. http://dx.doi.org/10.1002/rob.21571.
  30. Johnson, M., Shrewsbury, B., Bertrand, S., Calvert, D., Wu, T., Duran, D., et al. (2017). Team IHMC’s lessons learned from the DARPA robotics challenge: Finding data in the rubble. Journal of Field Robotics, 34(2), 241–261. http://dx.doi.org/10.1002/rob.21674.
  31. Klasing, K., Althoff, D., Wollherr, D., & Buss, M. (2009). Comparison of surface normal estimation methods for range sensing applications. In 2009 IEEE International Conference on Robotics and Automation (pp. 3206–3211). https://doi.org/10.1109/ROBOT.2009.5152493.
  32. Klien, G., Woods, D., Bradshaw, J., Hoffman, R., & Feltovich, P. (2004). Ten challenges for making automation a “team player” in joint human-agent activity. IEEE Intelligent Systems, 19(6), 91–95. http://dx.doi.org/10.1109/MIS.2004.74.
  33. Knabe, C., Griffin, R., Burton, J., Cantor-Cooke, G., Dantanarayana, L., Day, G., Ebeling-Koning, O., Hahn, E., Hopkins, M., Neal, J., Newton, J., Nogales, C., Orekhov, V., Peterson, J., Rouleau, M., Seminatore, J., Sung, Y., Webb, J., Wittenstein, N., Ziglar, J., Leonessa, A., Lattimer, B., & Furukawa, T. (2017). Team VALOR’s ESCHER: a novel electromechanical biped for the DARPA robotics challenge. Journal of Field Robotics, 34(5), 912–939. http://dx.doi.org/10.1002/rob.21697.
  34. Kohlbrecher, S., & Von Stryk, O. (2016). From robocup rescue to supervised autonomous mobile robots for remote inspection of industrial plants. KI-Künstliche Intelligenz, 30(3–4), 311–314. https://doi.org/10.1007/s13218-016-0446-8.
  35. Kohlbrecher, S., Conner, D. C., Romay, A., Bacim, F., Bowman, D. A., & von Stryk, O. (2013). Overview of team ViGIR’s approach to the virtual robotics challenge. In 2013 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR) (pp. 1–2). IEEE. https://doi.org/10.1109/SSRR.2013.6719382.
  36. Kohlbrecher, S., Romay, A., Stumpf, A., Gupta, A., von Stryk, O., Bacim, F., et al. (2015). Human-robot teaming for rescue missions: Team ViGIR’s approach to the 2013 DARPA robotics challenge trials. Journal of Field Robotics, 32(3), 352–377. http://dx.doi.org/10.1002/rob.21558.
  37. Kohlbrecher, S., Stumpf, A., Romay, A., Schillinger, P., von Stryk, O., & Conner, D. C. (2016). A comprehensive software framework for complex locomotion and manipulation tasks applicable to different types of humanoid robots. Frontiers in Robotics and AI, 3, 31. https://doi.org/10.3389/frobt.2016.00031.
  38. Koolen, T., Smith, J., Thomas, G., Bertrand, S., Carff, J., Mertins, N., et al. (2013). Summary of team IHMC’s virtual robotics challenge entry. In 2013 13th IEEE-RAS International Conference on Humanoid Robots (Humanoids) (pp. 307–314). http://dx.doi.org/10.1109/HUMANOIDS.2013.7029992.
  39. Krotkov, E., Hackett, D., Jackel, L., Perschbacher, M., Pippine, J., Strauss, J., et al. (2017). The DARPA robotics challenge finals: Results and perspectives. Journal of Field Robotics, 34(2), 229–240. http://dx.doi.org/10.1002/rob.21683.
  40. Leidner, D., Borst, C., & Hirzinger, G. (2012). Things are made for what they are: Solving manipulation tasks by using functional object classes. In 2012 12th IEEE-RAS International Conference on Humanoid Robots (Humanoids) (pp. 429–435). http://dx.doi.org/10.1109/HUMANOIDS.2012.6651555.
  41. Maniatopoulos, S., Schillinger, P., Pong, V., Conner, D. C., & Kress-Gazit, H. (2016). Reactive high-level behavior synthesis for an atlas humanoid robot. In 2016 IEEE International Conference on Robotics and Automation (ICRA) (pp. 4192–4199). http://dx.doi.org/10.1109/ICRA.2016.7487613.
  42. Marder-Eppstein, E., Berger, E., Foote, T., Gerkey, B., & Konolige, K. (2010). The office marathon: Robust navigation in an indoor office environment. In 2010 IEEE International Conference on Robotics and Automation (pp. 300–307). http://dx.doi.org/10.1109/ROBOT.2010.5509725.
  43. Morrow, J., & Khosla, P. (1997). Manipulation task primitives for composing robot skills. In 1997 IEEE International Conference on Robotics and Automation, 1997. Proceedings (Vol. 4, pp. 3354–3359). http://dx.doi.org/10.1109/ROBOT.1997.606800.
  44. Murphy, R. R. (2015). Meta-analysis of autonomy at the DARPA robotics challenge trials. Journal of Field Robotics, 32(2), 189–191. http://dx.doi.org/10.1002/rob.21578.
  45. Nagatani, K., Kiribayashi, S., Okada, Y., Otake, K., Yoshida, K., Tadokoro, S., et al. (2013). Emergency response to the nuclear accident at the Fukushima Daiichi Nuclear Power Plants using mobile rescue robots. Journal of Field Robotics, 30(1), 44–63. http://dx.doi.org/10.1002/rob.21439.
  46. Norton, A., Ober, W., Baraniecki, L., McCann, E., Scholtz, J., Shane, D., et al. (2017). Analysis of human-robot interaction at the DARPA robotics challenge finals. The International Journal of Robotics Research, 36(5–7), 483–513. http://dx.doi.org/10.1177/0278364916688254.
  47. Radford, N. A., et al. (2015). Valkyrie: NASA’s first bipedal humanoid robot. Journal of Field Robotics, 32(3), 397–419. http://dx.doi.org/10.1002/rob.21560.
  48. Raman, V., Piterman, N., & Kress-Gazit, H. (2013). Provably correct continuous control for high-level robot behaviors with actions of arbitrary execution durations. In 2013 IEEE International Conference on Robotics and Automation (pp. 4075–4081). http://dx.doi.org/10.1109/ICRA.2013.6631152.
  49. Raman, V., Piterman, N., Finucane, C., & Kress-Gazit, H. (2015). Timing semantics for abstraction and execution of synthesized high-level robot control. IEEE Transactions on Robotics, 31(3), 591–604. http://dx.doi.org/10.1109/TRO.2015.2414134.
  50. Romay, A., Kohlbrecher, S., Conner, D. C., Stumpf, A., & von Stryk, O. (2014). Template-based manipulation in unstructured environments for supervised semi-autonomous humanoid robots. In 2014 14th IEEE-RAS International Conference on Humanoid Robots (Humanoids) (pp. 979–986). http://dx.doi.org/10.1109/HUMANOIDS.2014.7041482.
  51. Romay, A., Kohlbrecher, S., Conner, D. C., & von Stryk, O. (2015a). Achieving versatile manipulation tasks with unknown objects by supervised humanoid robots based on object templates. In 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids) (pp. 249–255). http://dx.doi.org/10.1109/HUMANOIDS.2015.7363543.
  52. Romay, A., Stein, A., Oehler, M., Stumpf, A., Kohlbrecher, S., von Stryk, O., et al. (2015b). Open source driving controller concept for humanoid robots: Teams hector and ViGIR at 2015 DARPA robotics challenge finals. In 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids) (pp. 1147–1147). https://doi.org/10.1109/HUMANOIDS.2015.7363497.
  53. Romay, A., Kohlbrecher, S., & von Stryk, O. (2016). An object template approach to manipulation for humanoid avatar robots for rescue tasks. KI—Künstliche Intelligenz, 30(3), 279–287. https://doi.org/10.1007/s13218-016-0445-9.
  54. Romay, A., Kohlbrecher, S., Stumpf, A., von Stryk, O., Maniatopoulos, S., Kress-Gazit, H., et al. (2017). Collaborative autonomy between high-level behaviors and human operators for remote manipulation tasks using different humanoid robots. Journal of Field Robotics, 34(2), 333–358. http://dx.doi.org/10.1002/rob.21671.
  55. Şahin, E., Çakmak, M., Doğar, M. R., Uğur, E., & Üçoluk, G. (2007). To afford or not to afford: A new formalization of affordances toward affordance-based robot control. Adaptive Behavior, 15(4), 447–472. https://doi.org/10.1177/1059712307084689.
  56. Schappler, M., Vorndamme, J., Tödtheide, A., Conner, D. C., von Stryk, O., & Haddadin, S. (2015). Modeling, identification and impedance control of the atlas arms. In 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids) (pp. 1052–1059). http://dx.doi.org/10.1109/HUMANOIDS.2015.7363499.
  57. Schillinger, P. (2015). An Approach for Runtime-Modifiable Behavior Control of Humanoid Rescue Robots. Master’s thesis, Technische Universität Darmstadt. https://www.sim.informatik.tu-darmstadt.de/publ/da/2015_Schillinger_MA.pdf.
  58. Schillinger, P., Kohlbrecher, S., & von Stryk, O. (2016). Human-robot collaborative high-level control with application to rescue robotics. In 2016 IEEE International Conference on Robotics and Automation (ICRA) (pp. 2796–2802). http://dx.doi.org/10.1109/ICRA.2016.7487442.
  59. Stilman, M., Zafar, M., Erdogan, C., Hou, P., Reynolds-Haertle, S., & Tracy, G. (2014). Robots using environment objects as tools the “MacGyver” paradigm for mobile manipulation. In 2014 IEEE International Conference on Robotics and Automation (ICRA) (pp. 2568–2568). http://dx.doi.org/10.1109/ICRA.2014.6907225.
  60. Stumpf, A., Kohlbrecher, S., Conner, D. C., & von Stryk, O. (2014). Supervised footstep planning for humanoid robots in rough terrain tasks using a black box walking controller. In 2014 IEEE-RAS International Conference on Humanoid Robots (pp. 287–294). IEEE. http://dx.doi.org/10.1109/HUMANOIDS.2014.7041374.
  61. Stumpf, A., Kohlbrecher, S., von Stryk, O., & Conner, D. C. (2016). Open source integrated 3D footstep planning framework for humanoid robots. In 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids), Cancun, Mexico (pp. 938–945). https://doi.org/10.1109/HUMANOIDS.2016.7803385.
  62. Wisely Babu, B. P., Du, R., Padir, T., & Gennert, M. A. (2015). Improving robustness in complex tasks for a supervisor operated humanoid. Retrieved August 9, 2017, from https://www.cs.cmu.edu/~cga/drc/drill.pdf.
  63. Yanco, H. A., Norton, A., Ober, W., Shane, D., Skinner, A., & Vice, J. (2015). Analysis of human-robot interaction at the DARPA robotics challenge trials. Journal of Field Robotics, 32(3), 420–444. http://dx.doi.org/10.1002/rob.21568.

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • David C. Conner
    • 1
  • Stefan Kohlbrecher
    • 2
  • Philipp Schillinger
    • 3
  • Alberto Romay
    • 2
  • Alexander Stumpf
    • 2
  • Spyros Maniatopoulos
    • 4
  • Hadas Kress-Gazit
    • 4
  • Oskar von Stryk
    • 2
  1. 1.Capable Humanitarian Robotics & Intelligent Systems Lab, Department of Physics, Computer Science and EngineeringChristopher Newport UniversityNewport NewsUSA
  2. 2.Simulation, Systems Optimization and Robotics Group, Department of Computer ScienceTechnische Universität DarmstadtDarmstadtGermany
  3. 3.Bosch Center for Artificial IntelligenceRenningenGermany
  4. 4.Verifiable Robotics Research Group, School of Mechanical and Aerospace EngineeringCornell UniversityIthacaUSA

Personalised recommendations