Abstract
Robot navigation is one of the most studied problems in robotics and the key capability for robot autonomy. Navigation techniques have become more and more reliable, but evaluation mainly focused on individual navigation components (i.e., mapping, localization, and planning) using datasets or simulations. The goal of this paper is to define an experimental protocol to evaluate the whole navigation system, deployed in a real environment. To ensure repeatability and reproducibility of experiments, our benchmark protocol provides detailed definitions and controls the environment dynamics. We define standardized environments and introduce the concept of a reference robot to allow comparison between different navigation systems at different experimentation sites. We present applications of our protocol in experiments in two different research groups, showing the usefulness of the benchmark.
C. Sprunk, J. Röwekämper and G. Parent contributed equally to this work.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Bache, K., Lichman, M.: UCI machine learning repository. University of California, Irvine (2013). http://archive.ics.uci.edu/ml
Bennett, J., Lanning, S.: The netflix prize. In: KDD Cup and Workshop at the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Minig (2007)
Borenstein, J., Feng, L.: Umbmark: a benchmark test for measuring odometry errors in mobile robots. Proc. SPIE 2591, 113-124 (1995)
Burgard, W., Stachniss, C., Grisetti, G., Steder, B., Kümmerle, R., Dornhege, C., Ruhnke, M., Kleiner, A., Tardós, J.D.: A comparison of SLAM algorithms based on a graph of relations. In: International Conference on Intelligent Robots and Systems (2009)
Calisi, D., Iocchi, L., Nardi, D.: A unified benchmark framework for autonomous mobile robots and vehicles motion algorithms (MoVeMA benchmarks). In: RSS-Wksp. on Experimental Methodology and Benchmarking in Robotics Research (2008)
CAVIAR data sets. http://homepages.inf.ed.ac.uk/rbf/CAVIAR
DARPA urban challenge rules. http://archive.darpa.mil/grandchallenge/docs/Urban_Challenge_Rules_102707.pdf (2007)
Del Pobil, A.P., Madhavan, R., Messina, E.: Benchmarks in robotics research. In: IROS Workshop on Benchmarks in Robotics Research (2007)
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition (2009)
Dillmann, R.: Ka 1.10 benchmarks for robotics research (2004). http://www.cas.kth.se/euron/euron-deliverables/ka1-10-benchmarking.pdf
Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis. 88(2) (2010)
Gutmann, J.-S., Burgard, W., Fox, D., Konolige, K.: An experimental comparison of localization methods. In: International Conference on Robotics and Automation (1998)
Kikkeri, H., Parent, G., Jalobeanu, M., Birchfield, S.: An inexpensive methodology for evaluating the performance of a mobile robot navigation system. In: International Conference on Robotics and Automation (2014)
Knotts, R., Nourbakhsh, I., Morris, R.: Navigates: a benchmark for indoor navigation. In: International Conference and Experiments on Robotics for Challenging, Environments (1998)
Nowak, W., Zakharov, A., Blumenthal, S., Prassler, E.: Benchmarks for mobile manipulation and robust obstacle avoidance and navigation. BRICS Deliverable D3.1 (2010)
Pang, B., Lee, L.: Opinion mining and sentiment analysis. Found. Trends Inf. Retr. 2(1-2), 1-135 (2008)
PETS 2009 data set. http://pets2009.net/
Röwekämper, J., Sprunk, C., Tipaldi, G., Stachniss, C., Pfaff, P., Burgard, W.: On the position accuracy of mobile robot localization based on particle filters combined with scan matching. In: International Conference on Intelligent Robots and Systems (2012)
Sprunk, C., Lau, B., Pfaff, P., Burgard, W.: Online generation of kinodynamic trajectories for non-circular omnidirectional robots. In: International Conference on Robotics and Automation (2011)
Sturm, J., Engelhard, N., Endres, F., Burgard, W., Cremers, D.: A benchmark for the evaluation of RGB-D SLAM systems. In: International Conference on Intelligent Robots and Systems (2012)
Acknowledgments
This work has partly been supported by the EC under FP7-260026-TAPAS, FP7-610917-STAMINA, FP7-610603-EUROPA2, and FP7-267686-LIFENAV. The authors thank all members of the AIS Lab, the Microsoft Robotics Team, Studio99 and the Building 99 Hardware Lab for their patient help with the experiments.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Sprunk, C. et al. (2016). An Experimental Protocol for Benchmarking Robotic Indoor Navigation. In: Hsieh, M., Khatib, O., Kumar, V. (eds) Experimental Robotics. Springer Tracts in Advanced Robotics, vol 109. Springer, Cham. https://doi.org/10.1007/978-3-319-23778-7_32
Download citation
DOI: https://doi.org/10.1007/978-3-319-23778-7_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-23777-0
Online ISBN: 978-3-319-23778-7
eBook Packages: EngineeringEngineering (R0)