Abstract
Operating a high degree of freedom mobile manipulator, such as a humanoid, in a field scenario requires constant situational awareness, capable perception modules, and effective mechanisms for interactive motion planning and control. A well-designed operator interface presents the operator with enough context to quickly carry out a mission and the flexibility to handle unforeseen operating scenarios robustly. By contrast, an unintuitive user interface can increase the risk of catastrophic operator error by overwhelming the user with unnecessary information. With these principles in mind, we present the philosophy and design decisions behind Director—the open-source user interface developed by Team MIT to pilot the Atlas robot in the DARPA Robotics Challenge (DRC). At the heart of Director is an integrated task execution system that specifies sequences of actions needed to achieve a substantive task, such as drilling a wall or climbing a staircase. These task sequences, developed a priori, make online queries to automated perception and planning algorithms with outputs that can be reviewed by the operator and executed by our whole-body controller. Our use of Director at the DRC resulted in efficient high-level task operation while being fully competitive with approaches focusing on teleoperation by highly-trained operators. We discuss the primary interface elements that comprise the Director and provide analysis of its successful use at the DRC.
A version of this article was previously published in the Journal of Field Robotics, vol. 34, issue 2, pp. 262–280, ©Wiley 2017.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Although in Sect. 4 we will discuss our successful performance when robot damage lowered that precision.
- 2.
- 3.
References
Bagnell, J. A., Cavalcanti, F., Cui, L., Galluzzo, T., Hebert, M., Kazemi, M., Klingensmith, M., et al. (2012). An integrated system for autonomous robotics manipulation. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 2955–2962). IEEE.
Bohren, J., Rusu, R. B., Jones, E. G., Marder-Eppstein, E., Pantofaru, C., Wise, M., et al. (2011). Towards autonomous robotic butlers: Lessons learned with the PR2. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 5568–5575). IEEE.
Burstein, M. H., Beranek, B., Inc, N., & Mcdermott, D. V. (1996). Issues in the development of human-computer mixed-initiative planning. In Cognitive technology (pp. 285–303). Elsevier.
Conway, L., Volz, R. A., & Walker, M. W. (1990). Teleautonomous systems: Projecting and coordinating intelligent action at a distance. IEEE Transaction on Robotics, 6(2), 146–158.
Deits, R., & Tedrake, R. (2014). Footstep planning on uneven terrain with mixed-integer convex optimization. In IEEE/RSJ International Conference on Humanoid Robots (pp. 279–286). IEEE.
Diankov, R. (2010). Automated construction of robotic manipulation programs. Ph.D thesis, Robotics Institute, Carnegie Mellon University.
Dorais, G., Bonasso, R. P., Kortenkamp, D., Pell, B., & Schreckenghost, D. (1999). Adjustable autonomy for human-centered autonomous systems. In Working Notes of the Sixteenth International Joint Conference on Artificial Intelligence Workshop on Adjustable Autonomy Systems (pp. 16–35).
Dragan, A., & Srinivasa, S. (2012). Formalizing assistive teleoperation. Science and Systems (RSS): In Robotics.
Dragan, A. D., & Srinivasa, S. S. (2013). A policy-blending formalism for shared control. International Journal of Robotics Research, 32(7), 790–805.
Enes, A. R. (2010). Shared control of hydraulic manipulators to decrease cycle time. Ph.D thesis, Georgia Institute of Technology.
Fallon, M. F., Antone, M., Roy, N., & Teller, S. (2014). Drift-free humanoid state estimation fusing kinematic, inertial and lidar sensing. In IEEE/RSJ International Conference on Humanoid Robots, Madrid, Spain.
Fallon, M., Kuindersma, S., Karumanchi, S., Antone, M., Schneider, T., Dai, H., et al. (2015a). An architecture for online affordance-based perception and whole-body planning. Journal of Field Robotics, 32(2), 229–254.
Fallon, M. F., Marion, P., Deits, R., Whelan, T., Antone, M., McDonald, J., et al. (2015b). Continuous humanoid locomotion over uneven terrain using stereo fusion. In IEEE/RSJ International Conference on Humanoid Robots, Seoul, Korea. IEEE.
Finzi, A., & Orlandini, A. (2005). Human-robot interaction through mixed-initiative planning for rescue and search rovers. In AI* IA 2005: Advances in artificial intelligence (pp. 483–494). Springer.
Gibson, J. J. (1977). The theory of affordances. In R. Shaw & J. Bransford (Eds.), Perceiving, acting, and knowing. Wiley.
Hart, S., Dinh, P., & Hambuchen, K. (2015). The affordance template ROS package for robot task programming. In 2015 IEEE International Conference on Robotics and Automation (ICRA) (pp. 6227–6234).
Hart, S., Dinh, P., Yamokoski, J. D., Wightman, B., & Radford, N. (2014). Robot task commander: A framework and IDE for robot application development. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2014) (pp. 1547–1554). IEEE.
He, K., Lahijanian, M., Kavraki, L. E., & Vardi, M. Y. (2015). Towards manipulation planning with temporal logic specifications. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 346–352). IEEE.
Huang, A., Olson, E., & Moore, D. (2010). LCM: Lightweight communications and marshalling. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Taipei, Taiwan.
Jain, S., Farshchiansadegh, A., Broad, A., Abdollahi, F., Mussa-Ivaldi, F., & Argall, B. (2015). Assistive robotic manipulation through shared autonomy and a body-machine interface. In 2015 IEEE International Conference on Rehabilitation Robotics (ICORR) (pp. 526–531).
Kaelbling, L. P., & Lozano-Pérez, T. (2011). Hierarchical task and motion planning in the now. In 2011 IEEE International Conference on Robotics and Automation (ICRA) (pp. 1470–1477). IEEE.
Kress-Gazit, H., Fainekos, G. E., & Pappas, G. J. (2007). Where’s Waldo? Sensor-based temporal logic motion planning. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 3116–3121). IEEE.
Kuindersma, S., Deits, R., Fallon, M. F., Valenzuela, A., Dai, H., Permenter, F., et al. (2016). Optimization-based locomotion planning, estimation, and control design for atlas. Autonomous Robots, 40, 429–455.
Marion, P., Kwitt, R., Davis, B., & Gschwandtner, M. (2012). PCL and ParaView—Connecting the dots. In 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (pp. 80–85). IEEE.
Martin, C. E., Macfadzean, R. H., & Barber, K. S. (1996). Supporting dynamic adaptive autonomy for agent-based systems. In Proceedings of the Artificial Intelligence and Manufacturing Research Planning Workshop (pp. 112–120).
Muelling, K., Venkatraman, A., Valois, J.-S., Downey, J., Weiss, J., Javdani, S., et al. (2015). Autonomy infused teleoperation with application to BCI manipulation. In Robotics Science and Systems (RSS).
Murphy, R. (2004). Human-robot interaction in rescue robotics. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 34(2), 138–153.
Nehaniv, C. L., & Dautenhahn, K. (2002). The correspondence problem. In Imitation in animals and artifacts (pp. 41–61). Cambridge, MA, USA: MIT Press.
O’Brien, B., Stump, E., & Pierce, C. (2010). Effects of increasing autonomy on tele-operation performance. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 1792–1798).
Rusu, R. B., & Cousins, S. (2011). 3D is here: Point Cloud Library (PCL). In IEEE International Conference on Robotics and Automation (ICRA) (pp. 1–4). IEEE.
Rusu, R. B., Şucan, I. A., Gerkey, B., Chitta, S., Beetz, M., & Kavraki, L. E. (2009). Real-time perception-guided motion planning for a personal robot. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 4245–4252). IEEE.
Schroeder, W. J., Lorensen, B., & Martin, K. (2008). The visualization toolkit: An object-oriented approach to 3D graphics (4th ed.). Kitware.
Sellner, B., Heger, F. W., Hiatt, L. M., Simmons, R., & Singh, S. (2006). Coordinated multiagent teams and sliding autonomy for large-scale assembly. Proceedings of the IEEE, 94(7), 1425–1444.
Sentis, L., Park, J., & Khatib, O. (2010). Compliant control of multicontact and center-of-mass behaviors in humanoid robots. IEEE Transaction on Robotics, 26(3), 483–501.
Sheridan, T. B. (1992). Telerobotics, automation, and human supervisory control. Cambridge, MA, USA: MIT Press.
Srivastava, S., Fang, E., Riano, L., Chitnis, R., Russell, S., & Abbeel, P. (2014). Combined task and motion planning through an extensible planner-independent interface layer. In 2014 IEEE International Conference on Robotics and Automation (ICRA) (pp. 639–646). IEEE.
Stentz, A., Herman, H., Kelly, A., Meyhofer, E., Haynes, G. C., Stager, D., et al. (2015). CHIMP, the CMU highly intelligent mobile platform. Journal of Field Robotics, 32(2), 209–228.
Tambe, M., Scerri, P., & Pynadath, D. V. (2002). Adjustable autonomy for the real world. Journal of Artificial Intelligence Research, 17(1), 171–228.
Tedrake, R. (2014). Drake: A planning, control, and analysis toolbox for nonlinear dynamical systems. http://drake.mit.edu.
Tedrake, R., Fallon, M., Karumanchi, S., Kuindersma, S., Antone, M., Schneider, T., et al. (2014). A summary of Team MIT’s approach to the virtual robotics challenge. IEEE International Conference on Robotics and Automation (ICRA) (pp. 2087–2087). Hong Kong: China.
Tedrake, R., Kuindersma, S., Deits, R., & Miura, K. (2015). A closed-form solution for real-time ZMP gait generation and feedback stabilization. In IEEE/RSJ International Conference on Humanoid Robots, Seoul, Korea.
Wang, A., Ramos, J., Mayo, J., Ubellacker, W., Cheung, J., & Kim, S. (2015). The HERMES humanoid system: A platform for full-body teleoperation with balance feedback. In IEEE/RSJ International Conference on Humanoid Robots, Seoul, Korea.
Wolfe, J., Marthi, B., & Russell, S. J. (2010). Combined task and motion planning for mobile manipulation. In ICAPS (pp. 254–258).
Yamauchi, B. M. (2004). Packbot: A versatile platform for military robotics. In Proceedings of SPIE 5422. Unmanned Ground Vehicle Technology (Vol. 5422, pp. 228–237).
Acknowledgements
We gratefully acknowledge the support of the Defense Advanced Research Projects Agency via Air Force Research Laboratory award FA8750-12-1-0321, and the Office of Naval Research via award N00014-12-1-0071. We are also grateful to the team’s many supporters both inside and outside MIT (listed at http://drc.mit.edu), including our families and friends. We are also grateful to Boston Dynamics, Carnegie Robotics, the Open Source Robotics Foundation, Robotiq, iRobot Corporation, and Sandia National Laboratories for their support during the DRC. Included photos from the DRC Finals are credit to Jason Dorfman of MIT CSAIL. Finally, we acknowledge our late colleague, advisor, and friend, Seth Teller, whose leadership and ideas contributed immensely to this work.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this chapter
Cite this chapter
Marion, P. et al. (2018). Director: A User Interface Designed for Robot Operation with Shared Autonomy. In: Spenko, M., Buerger, S., Iagnemma, K. (eds) The DARPA Robotics Challenge Finals: Humanoid Robots To The Rescue. Springer Tracts in Advanced Robotics, vol 121. Springer, Cham. https://doi.org/10.1007/978-3-319-74666-1_7
Download citation
DOI: https://doi.org/10.1007/978-3-319-74666-1_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-74665-4
Online ISBN: 978-3-319-74666-1
eBook Packages: EngineeringEngineering (R0)