Multi-view X-Ray R-CNN

  • Jan-Martin O. Steitz
  • Faraz SaeedanEmail author
  • Stefan Roth
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11269)


Motivated by the detection of prohibited objects in carry-on luggage as a part of avionic security screening, we develop a CNN-based object detection approach for multi-view X-ray image data. Our contributions are two-fold. First, we introduce a novel multi-view pooling layer to perform a 3D aggregation of 2D CNN-features extracted from each view. To that end, our pooling layer exploits the known geometry of the imaging system to ensure geometric consistency of the feature aggregation. Second, we introduce an end-to-end trainable multi-view detection pipeline based on Faster R-CNN, which derives the region proposals and performs the final classification in 3D using these aggregated multi-view features. Our approach shows significant accuracy gains compared to single-view detection while even being more efficient than performing single-view detection in each view.



The authors gratefully acknowledge support by Smiths Heimann GmbH.


  1. 1.
    Akçay, S., Kundegorski, M.E., Devereux, M., Breckon, T.P.: Transfer learning using convolutional neural networks for object classification within X-ray baggage security imagery. In: ICIP, pp. 1057–1061 (2016).
  2. 2.
    Aubry, M., Maturana, D., Efros, A.A., Russell, B., Sivic, J.: Seeing 3D chairs: exemplar part-based 2D-3D alignment using a large dataset of CAD models. In: CVPR, pp. 3762–3769 (2014).
  3. 3.
    Baştan, M.: Multi-view object detection in dual-energy X-ray images. Mach. Vis. Appl. 26(7–8), 1045–1060 (2015). Scholar
  4. 4.
    Brudy, T., Schilhab, S.: Projection of hazardous items into X-ray images of inspection objects. Patent WO 2016/001282 AI, January 2016Google Scholar
  5. 5.
    Chen, X., Ma, H., Wan, J., Li, B., Xia, T.: Multi-view 3D object detection network for autonomous driving. In: CVPR, pp. 1907–1915 (2017).
  6. 6.
    Everingham, M., Eslami, S.M.A., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PASCAL visual object classes challenge: a retrospective. Int. J. Comput. Vis. 111(1), 98–136 (2014). Scholar
  7. 7.
    Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PASCAL visual object classes (VOC) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010). Scholar
  8. 8.
    Franzel, T., Schmidt, U., Roth, S.: Object detection in multi-view X-Ray images. In: Pinz, A., Pock, T., Bischof, H., Leberl, F. (eds.) DAGM/OAGM 2012. LNCS, vol. 7476, pp. 144–154. Springer, Heidelberg (2012). Scholar
  9. 9.
    Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: CVPR, pp. 3354–3361 (2012).
  10. 10.
    Girshick, R.: Fast R-CNN. In: ICCV, pp. 1440–1448. IEEE (2015).
  11. 11.
    Girshick, R., Donahue, J., Darrell, T., Malik, J.: Region-based convolutional networks for accurate object detection and segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 38(1), 142–158 (2016). Scholar
  12. 12.
    Goyal, P., et al.: Accurate, large minibatch SGD: training ImageNet in 1 hour. arXiv:1706.02677 (2017)
  13. 13.
    Gupta, S., Arbeláez, P., Girshick, R., Malik, J.: Aligning 3D models to RGB-D images of cluttered scenes. In: CVPR, pp. 4731–4740 (2015).
  14. 14.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016).
  15. 15.
    He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 630–645. Springer, Cham (2016). Scholar
  16. 16.
    Jaccard, N., Rogers, T.W., Morton, E.J., Griffin, L.D.: Automated detection of smuggled high-risk security threats using deep learning. In: ICDP 2016, pp. 1–6 (2016).
  17. 17.
    Jaccard, P.: The distribution of the Flora in the Alpine Zone. New Phytol. 11(2), 37–50 (1912). Scholar
  18. 18.
    Li, B.: 3D fully convolutional network for vehicle detection in point cloud. In: IROS, pp. 1513–1518 (2016).
  19. 19.
    Mery, D., Svec, E., Arias, M., Riffo, V., Saavedra, J.M., Banerjee, S.: Modern computer vision techniques for X-ray testing in baggage inspection. IEEE Trans. Syst. Man Cybern. Syst. 15(2), 682–692 (2016). Scholar
  20. 20.
    Mousavian, A., Anguelov, D., Flynn, J., Kos̆ecká, J.: 3D bounding box estimation using deep learning and geometry. In: CVPR, pp. 5632–5640 (2017).
  21. 21.
    Qi, C.R., Liu, W., Wu, C., Su, H., Guibas, L.J.: Frustum pointnets for 3D object detection from RGB-D data. In: CVPR, pp. 918–927 (2018)Google Scholar
  22. 22.
    Redmon, J., Farhadi, A.: YOLO9000: better, faster, stronger. CVPR (2017).
  23. 23.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2017). Scholar
  24. 24.
    Romea, A.C., Torres, M.M., Srinivasa, S.: The MOPED framework: object recognition and pose estimation for manipulation. Int. J. Robot. Res. 30(10), 1284–1306 (2011). Scholar
  25. 25.
    Rothganger, F., Lazebnik, S., Schmid, C., Ponce, J.: 3D object modeling and recognition using local affine-invariant image descriptors and multi-view spatial constraints. Int. J. Comput. Vis. 66(3), 231–259 (2006). Scholar
  26. 26.
    Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015). Scholar
  27. 27.
    Song, S., Xiao, J.: Sliding shapes for 3D object detection in depth images. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 634–651. Springer, Cham (2014). Scholar
  28. 28.
    Song, S., Xiao, J.: Deep sliding shapes for amodal 3D object detection in RGB-D images. In: CVPR, pp. 808–816 (2016).
  29. 29.
    Su, H., Maji, S., Kalogerakis, E., Learned-Miller, E.G.: Multi-view convolutional neural networks for 3D shape recognition. In: ICCV, pp. 945–953 (2015).
  30. 30.
    Tulsiani, S., Malik, J.: Viewpoints and keypoints. In: CVPR, pp. 1510–1519 (2015)Google Scholar
  31. 31.
    Vatti, B.R.: A generic solution to polygon clipping. Commun. ACM 35(7), 56–63 (1992). Scholar
  32. 32.
    Xiang, Y., Choi, W., Lin, Y., Savarese, S.: Data-driven 3D voxel patterns for object category recognition. In: CVPR, pp. 1903–1911 (2015).
  33. 33.
    Xiang, Y., Choi, W., Lin, Y., Savarese, S.: Subcategory-aware convolutional neural networks for object proposals and detection. In: WACV, pp. 924–933 (2017).
  34. 34.
    Zhu, M., et al.: Single image 3D object detection and pose estimation for grasping. In: ICRA, pp. 3936–3943 (2014).

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Computer ScienceTU DarmstadtDarmstadtGermany

Personalised recommendations