Abstract
Field robotics applications have some unique and unusual data requirements—the curating, organisation and management of which are often overlooked. An emerging theme is the use of large corpora of spatiotemporally indexed sensor data which must be searched and leveraged both offline and online. Increasingly we build systems that must never stop learning. Every sortie requires swift, intelligent read-access to gigabytes of memories and the ability to augment the totality of stored experiences by writing new memories. This however leads to vast quantities of data which quickly become unmanageable, especially when we want to find what is relevant to our needs. The current paradigm of collecting data for specific purposes and storing them in ad-hoc ways will not scale to meet this challenge. In this paper we present the design and implementation of a data management framework that is capable of dealing with large datasets and provides functionality required by many offline and online robotics applications. We systematically identify the data requirements of these applications and design a relational database that is capable of meeting their demands. We describe and demonstrate how we use the system to manage over 50TB of data collected over a period of 4 years.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The definition of ‘nearest’ may differ between queries but note that it is beyond the scope of the system to, for example, interpolate between records or verify annotation correctness—how that is handled is up to individual client applications.
- 2.
- 3.
- 4.
- 5.
References
Churchill, W., Newman, P.: Experience-based navigation for long-term localisation. Int. J. Robot. Res. 32(14), 1645–1661 (2013)
Smith, M., Baldwin, I., Churchill, W., Paul, R., Newman, P.: The New College vision and laser data set. Int. J. Robot. Res. 28, 595–599 (2009)
Huang, A.S., Antone, M., Olson, E., Fletcher, L., Moore, D., Teller, S., Leonard, J.: A high-rate, heterogeneous data set from the DARPA urban challenge. Int. J. Robot. Res. 29, 1595–1601 (2010)
Waibel, M., Beetz, M., Civera, J., D’Andrea, R., Elfring, J., Galvez-Lopez, D., Haussermann, K., Janssen, R., Montiel, J., Perzylo, A., et al.: Roboearth. IEEE Robot. Autom. Mag. 18(2), 69–82 (2011)
Haklay, M., Weber, P.: OpenStreetMap: user-generated street maps. IEEE Pervasive Comput. 7(4), 12–18 (2008)
Ramakrishnan, R., Gehrke, J.: Database Management Systems. Osborne/McGraw-Hill (2000)
Bayer, R.: Organization and maintenance of large ordered indexes. Acta Informatica 1(3), 173–189 (1972)
Guttman, A.: R-trees: A Dynamic Index Structure for Spatial Searching, vol. 14. ACM (1984)
Linegar, C., Churchill, W., Newman, P.: Work smart, not hard: recalling relevant experiences for vast-scale but time-constrained localisation. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA2015) (2015)
Acknowledgments
Peter Nelson is supported by an EPSRC Doctoral Training Account. Chris Linegar is supported by the Rhodes Trust. Paul Newman is supported by EPSRC Leadership Fellowship EP/I005021/1.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Nelson, P., Linegar, C., Newman, P. (2016). Building, Curating, and Querying Large-Scale Data Repositories for Field Robotics Applications. In: Wettergreen, D., Barfoot, T. (eds) Field and Service Robotics. Springer Tracts in Advanced Robotics, vol 113. Springer, Cham. https://doi.org/10.1007/978-3-319-27702-8_34
Download citation
DOI: https://doi.org/10.1007/978-3-319-27702-8_34
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-27700-4
Online ISBN: 978-3-319-27702-8
eBook Packages: EngineeringEngineering (R0)