Virtually Adapted Reality and Algorithm Visualization for Autonomous Robots

  • Danny ZhuEmail author
  • Manuela Veloso
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9776)


Autonomous mobile robots are often videotaped during operation, whether for later evaluation by their developers or for demonstration of the robots to others. Watching such videos is engaging and interesting. However, clearly the plain videos do not show detailed information about the algorithms running on the moving robots, leading to a rather limited visual understanding of the underlying autonomy. Researchers have resorted to following the autonomous robots algorithms through a variety of methods, most commonly graphical user interfaces running on offboard screens and separated from the captured videos. Such methods enable considerable debugging, but still have limited effectiveness, as there is an inevitable visual mismatch with the video capture. In this work, we aim to break this disconnect, and we contribute the ability to overlay visualizations onto a video, to extract the robot’s algorithms, in particular to follow its route planning and execution. We further provide mechanisms to create and visualize virtual adaptations of the real environment to enable the exploration of the behavior of the algorithms in new situations. We demonstrate the complete implementation with an autonomous quadrotor navigating in a lab environment using the rapidly-exploring random tree algorithm. We briefly motivate and discuss our follow-up visualization work for our complex small-size robot soccer team.


  1. 1.
    Amstutz, P., Fagg, A.H.: Real time visualization of robot state with mobile virtual reality. In: The IEEE International Conference on Robotics and Automation, Proceedings of ICRA 2002, vol. 1, pp. 241–247. IEEE (2002)Google Scholar
  2. 2.
    Azuma, R.T.: A survey of augmented reality. Presence 6(4), 355–385 (1997)CrossRefGoogle Scholar
  3. 3.
    Bradski, G.: OpenCV. Dr. Dobb’s J. Softw. Tools (2000)Google Scholar
  4. 4.
    Brown, M.H., Sedgewick, R.: A system for algorithm animation, vol. 18. ACM (1984)Google Scholar
  5. 5.
    Bruce, J., Veloso, M.: Real-time randomized path planning for robot navigation. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, vol. 3, pp. 2383–2388. IEEE (2002)Google Scholar
  6. 6.
    Chadalavada, R.T., Andreasson, H., Krug, R., Lilienthal, A.J.: That’s on my mind! Robot to human intention communication through on-board projection on shared floor space. In: European Conference on Mobile Robots (ECMR) (2015)Google Scholar
  7. 7.
    Collett, T.H.J.: Augmented reality visualisation for mobile robot developers. Ph.D. thesis, University of Auckland (2007)Google Scholar
  8. 8.
    Forsyth, D.A., Ponce, J.: Computer Vision: A Modern Approach. Prentice-Hall, Englewood Cliffs (2003)Google Scholar
  9. 9.
    Hayet, J.-B., Piater, J., Verly, J.: Robust incremental rectification of sports video sequences. In: British Machine Vision Conference (BMVC 2004), pp. 687–696 (2004)Google Scholar
  10. 10.
    Kim, W.S.: Virtual reality calibration and preview/predictive displays for telerobotics. Presence: Teleoper. Virtual Environ. 5(2), 173–190 (1996)MathSciNetCrossRefGoogle Scholar
  11. 11.
    LaValle, S.M.: Rapidly-exploring random trees: a new tool for path planning. Technical report 98–11, Iowa State University (1998)Google Scholar
  12. 12.
    Mendoza, J.P., Biswas, J., Cooksey, P., Wang, R., Klee, S., Zhu, D., Veloso, M.: Selectively reactive coordination for a team of robot soccer champions. In: The Thirtieth AAAI Conference on Artificial Intelligence, Proceedings of AAAI 2016 (2016)Google Scholar
  13. 13.
    Mendoza, J.P., Biswas, J., Zhu, D., Wang, R., Cooksey, P., Klee, S., Veloso, M.: CMDragons 2015: coordinated offense and defense of the SSL champions. In: Almeida, L., Ji, J., Steinbauer, G., Luke, S. (eds.) RoboCup 2015. LNCS, vol. 9513, pp. 106–117. Springer, Cham (2015). Scholar
  14. 14.
    Riley, P., Stone, P., Veloso, M.: Layered disclosure: revealing agents’ internals. In: Castelfranchi, C., Lespérance, Y. (eds.) ATAL 2000. LNCS, vol. 1986, pp. 61–72. Springer, Heidelberg (2001). Scholar
  15. 15.
    Serra, J.: Image Analysis and Mathematical Morphology. Academic Press Inc., Cambridge (1983)Google Scholar
  16. 16.
    Zickler, S., Laue, T., Birbach, O., Wongphati, M., Veloso, M.: SSL-vision: the shared vision system for the RoboCup small size league. In: Baltes, J., Lagoudakis, M.G., Naruse, T., Ghidary, S.S. (eds.) RoboCup 2009. LNCS, vol. 5949, pp. 425–436. Springer, Heidelberg (2010). Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Computer Science DepartmentCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations