Abstract
The self-diving vehicle is a critical revolution in the automobile industry, transportation and people’s daily life. Generally, self-driving vehicles utilize the LIDAR, IMU, radar along with traditional optical cameras for sensing surrounding environment. Therefore, on road object detection can be improved by utilizing multiview of LIDAR and camera instead of single sensor. In this paper, we propose a sensor data fusion-based multiview CNN model for on road vehicle detection. We first up-sample the sparse LIDAR point cloud as gray images, and then combine the up-sampled gray image (F channel) with the camera image (RGB channels) as four-channel fusion data to train a CNN model. To reduce the redundant data, we tested the performance of CNN models by different fusion schemes. In this way, we can reduce the consumption of the CNN model in both training and on board testing for vehicular environment. Our final fusion model (RBF) can reach an average accuracy of 82% on the real trace dataset of KITTI.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Borenstein, J., Herkert, J., Miller, K.: Self-driving cars: ethical responsibilities of design engineers. IEEE Technol. Soc. Mag. 36(2), 67–75 (2017)
Li, X., Chen, M., Nie, F., et al.: A multiview-based parameter free framework for group detection. In: AAAI Conference on Artificial Intelligence (2017)
Chen, Z., Huang, X.: End-to-end learning for lane keeping of self-driving cars. In: 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 1856–1860 (2017)
Ouyang, Z., Niu, J., Liu, Y., et al.: Multiwave: a novel vehicle steering pattern detection method based on smartphones. In: IEEE International Conference on Communications, pp. 1–7. IEEE (2016)
Jafari, O.H., Yang, M.Y.: Real-time RGB-D based template matching pedestrian detection. In: IEEE International Conference on Robotics and Automation. IEEE (2016)
Tung, F., Chen, J., Meng, L., et al.: The raincouver scene parsing benchmark for self-driving in adverse weather and at night. IEEE Robot. Autom. Lett. 2(4), 2188–2193 (2017)
Song, S., Chandraker, M.: Joint SFM and detection cues for monocular 3D localization in road scenes. In: Computer Vision and Pattern Recognition, pp. 3734–3742. IEEE (2015)
Geiger, A.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354–3361. IEEE Computer Society (2012)
Hwang, S., Park, J., Kim, N., et al.: Multispectral pedestrian detection: benchmark dataset and baseline. In: Computer Vision and Pattern Recognition, pp. 1037–1045. IEEE (2015)
Xue, J.R., Wang, D., Shao-Yi, D.U., et al.: A vision-centered multi-sensor fusing approach to self-localization and obstacle perception for robotic cars. Front. Inf. Technol. Electron. Eng. 18(1), 122–138 (2018)
Dollar, P., Appel, R., et al.: Fast feature pyramids for object detection. IEEE Trans. Pattern Anal. Mach. Intell. 36, 1532–1545 (2014)
Lin, Y., Abdelfatah, K., Zhou, Y., et al.: Co-interest person detection from multiple wearable camera videos, pp. 4426–4434 (2015)
Huang, Y., Zhu, F., Shao, L., et al.: Color object recognition via cross-domain learning on RGB-D images. In: IEEE International Conference on Robotics and Automation, pp. 1672–1677. IEEE (2016)
Gu, X., Zang, A., Huang, X., et al.: Fusion of color images and LIDAR data for lane classification. In: Proceedings of the 23rd SIGSPATIAL International Conference on Advances in Geographic Information Systems, pp. 1–4 (2015)
Zhang, F., Clarke, D., Knoll, A.: Vehicle detection based on LIDAR and camera fusion. In: IEEE International Conference on Intelligent Transportation Systems, pp. 1620–1625. IEEE (2014)
Premebida, C., Carreira, J., Batista, J., et al.: Pedestrian detection combining RGB and dense LIDAR data. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4112–4117. IEEE (2014)
Ouyang, Z., Niu, J., Liu, Y., et al.: A cGANs-based scene reconstruction model using LIDAR point cloud. In: Proceedings of the 15th IEEE International Symposium on Parallel and Distributed Processing with Applications (2017)
Liu, W., Ji, R., Li, S.: Towards 3D object detection with bimodal deep Boltzmann machines over RGBD imagery. In: Computer Vision and Pattern Recognition, pp. 3013–3021. IEEE (2015)
Qi, C.R., Su, H., Nießner, M., et al.: Volumetric and multi-view CNNs for object classification on 3D data, pp. 5648–5656 (2016)
Girshick, R., Donahue, J., Darrell, T., Malik J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: CVPR (2014)
Girshick R.: Fast R-CNN. Computer Science (2015)
Ren, S., Girshick, R., Girshick, R., et al.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2015)
He, K., Zhang, X., Ren, S., et al.: Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(9), 1904–1916 (2015)
Redmon, J., Divvala, S., Girshick, R., et al.: You only look once: unified, real-time object detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788. IEEE Computer Society (2016)
Redmon, J., Farhadi, A.: YOLO9000: better, faster, stronger, pp. 6517–6525 (2016)
Li, B., Zhang, T., Xia, T.: Vehicle detection from 3D lidar using fully convolutional network. In: Robotics: Science and Systems (2016)
Asvadi, A., Garrote, L., Premebida, C., Peixoto, P., Nunes, U.J.: Real-time deep convnet-based vehicle detection using 3D-LIDAR reflection intensity data. In: Ollero, A., Sanfeliu, A., Montano, L., Lau, N., Cardeira, C. (eds.) ROBOT 2017. AISC, vol. 694, pp. 475–486. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-70836-2_39
Asvadi, A., Garrote, L., Premebida, C., et al.: DepthCN: vehicle detection using 3D-LIDAR and ConvNet. In: IEEE International Conference on Intelligent Transportation Systems. IEEE (2017)
Wang, X., Nie, L., Song, X., et al.: Unifying virtual and physical worlds: learning toward local and global consistency. ACM Trans. Inf. Syst. 36(1), 1–26 (2017)
Chen, J., Song, X., Nie, L., et al.: Micro tells macro: predicting the popularity of micro-videos via a transductive model. In: ACM on Multimedia Conference, pp. 898–907. ACM (2016)
Acknowledgement
This work has been supported by National Key R&D Program of China (2017YFB1301100), National Natural Science Foundation of China (61572060, 61772060, 61728201), State Key Laboratory of Software Development Environment(No. 2016YFC1300205), and CERNET Innovation Project (NGII20160316, NGII20170315).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Ouyang, Z., Wang, C., Liu, Y., Niu, J. (2018). Multiview CNN Model for Sensor Fusion Based Vehicle Detection. In: Hong, R., Cheng, WH., Yamasaki, T., Wang, M., Ngo, CW. (eds) Advances in Multimedia Information Processing – PCM 2018. PCM 2018. Lecture Notes in Computer Science(), vol 11166. Springer, Cham. https://doi.org/10.1007/978-3-030-00764-5_42
Download citation
DOI: https://doi.org/10.1007/978-3-030-00764-5_42
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00763-8
Online ISBN: 978-3-030-00764-5
eBook Packages: Computer ScienceComputer Science (R0)