Indoor Environment RGB-DT Mapping for Security Mobile Robots

  • Lijun ZhaoEmail author
  • Yu Liu
  • Xinkai Jiang
  • Ke Wang
  • Zigeng Zhou
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11743)


Many robot applications, such as environmental monitoring, security and surveillance help people to do tasks in day-to-day scenarios. However, the growing security demand for environment perception is a key issue of mapping or frequent updating in the long term, such as fire detection in early stage. A hybrid mapping method is proposed based on fusing RGB, depth and thermal (DT) information from Kinect and infrared sensors equipped in the mobile robot. Firstly, the proposed pipeline will estimate the robot’s pose by extracting and matching ORB features in RGB images successively. Then Poses corresponding to each depth and thermal- Infrared image are estimated through a combination of timestamp synchronization and the result of the extrinsic calibration of the system, and the map with both appearance and the temperature of environment is generated by the combination of The RGB and temperature information. Finally, the depth information is used to project the pixel points to the world coordinate system to generate the RGB-DT map. Extensive results verify the effectiveness of the proposed RGB-DT mapping for environments perception.


SLAM Environment mapping ORB-SLAM Multi-sensor fusing 


  1. 1.
    Tan, Y.: A survey on visual perception for firefighting robots. J. Mianyang Teach. Coll. 2, 40–45 (2018)Google Scholar
  2. 2.
    Newcombe, R.A., Izadi, S., et al.: KinectFusion: real-time dense surface mapping and tracking. In: IEEE International Symposium on Mixed and Augmented Reality. pp. 127–136. IEEE, Basel (2011)Google Scholar
  3. 3.
    Dai, A., Izadi, S., Theobalt, C.: BundleFusion: real-time globally consistent 3D reconstruction using on-the-fly surface re-integration. ACM Trans. Graph. 36(4), 76a (2017)CrossRefGoogle Scholar
  4. 4.
    Mur-Artal, R., Tardós, J.D.: ORB-SLAM2: an open-source SLAM system for monocular, stereo, and RGB-D cameras. IEEE Trans. Robot. 33(5), 1255–1262 (2017)CrossRefGoogle Scholar
  5. 5.
    Vidas, S., Moghadam, P., Bosse, M.: 3D thermal mapping of building interiors using an RGB-D and thermal camera. In: IEEE International Conference on Robotics and Automation, pp. 2311–2318. IEEE, Karlsruhe (2013)Google Scholar
  6. 6.
    Borrmann, D., Nüchter, A., Djakulovi’C, M., et al.: The project thermal mapper-thermal 3D mapping of indoor environments for saving energy. In: International IFAC Symposium on Robot Control, pp. 31–38 (2012)CrossRefGoogle Scholar
  7. 7.
    Nagatani, K., Otake, K., Yoshida, K.: Three-dimensional thermography mapping for mobile rescue robots. In: Yoshida, K., Tadokoro, S. (eds.) Field and Service Robotics, pp. 49–63. Springer, Heidelberg (2014). Scholar
  8. 8.
    Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000)CrossRefGoogle Scholar
  9. 9.
    Zhang, Q., Pless, R.: Extrinsic calibration of a camera and laser range finder (improves camera calibration). In: IEEE/RSJ International Conference on Intelligent Robots & Systems. IEEE, Sendai (2005)Google Scholar
  10. 10.
    Moghadam, P., Bosse, M., Zlot, R.: Line-based extrinsic calibration of range and image sensors. In: 2013 IEEE International Conference on Robotics and Automation. IEEE, Karlsruhe (2013)Google Scholar
  11. 11.
    Hwang, S., Choi, Y., Kim, N., et al.: Low-cost synchronization for multispectral cameras. In: International Conference on Ubiquitous Robots & Ambient Intelligence. IEEE, Goyang (2015)Google Scholar
  12. 12.
    Baar, J.V., Beardsley, P., Pollefeys, M., et al.: Sensor fusion for depth estimation, including TOF and thermal sensors. In: 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission, pp. 472–478. IEEE Computer Society, Zurich (2012)Google Scholar
  13. 13.
    Ge, P., Yang, B., Han, Q., et al.: Infrared image detail enhancement algorithm based on hierarchical processing by guided image filter. Infrared Technol. 40(12), 45–53 (2018)Google Scholar
  14. 14.
    Chang, H., Chen, C.: Image fusion based on HSV color space model and wavelet transform. Comput. Eng. Des. 28(23), 5682–5684 (2007)Google Scholar
  15. 15.
    Lepetit, V., Moreno-Noguer, F., Fua, P.: EPnP: an accurate O (n) solution to the PnP problem. Int. J. Comput. Vis. 81(2), 155–166 (2009)CrossRefGoogle Scholar
  16. 16.
    Galvez-Lo, P.D., Tardos, J.D.: Bags of binary words for fast place recognition in image sequences. IEEE Trans. Robot. 28(5), 1188–1197 (2012)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Lijun Zhao
    • 1
    Email author
  • Yu Liu
    • 1
  • Xinkai Jiang
    • 1
  • Ke Wang
    • 1
  • Zigeng Zhou
    • 2
  1. 1.State Key Laboratory of Robotics and SystemHarbin Institute of TechnologyHarbinChina
  2. 2.Kunming Power Supply BureauKunmingChina

Personalised recommendations