Deep Learning for 3D Data Processing

  • Zhenbao Liu
  • Zhizhong Han
  • Shuhui BuEmail author


Extracting local features from raw 3D data is a nontrivial and challenging task that requires carefully designed 3D shape descriptors. In conventional methods, these descriptors are handcrafted and require intensive human intervention and prior knowledge. To tackle this issue, we propose a novel deep learning model, namely, Circle Convolutional Restricted Boltzmann Machine (CCRBM), for unsupervised 3D local feature learning. CCRBM is specially designed for 3D shapes which effectively resolves the obstacles in the hierarchical learning process that existing deep learning models cannot resolve, such as irregular topology of vertices, orientation ambiguity on the 3D surface, and rigid or slightly nonrigid transformation invariance. Specially, by introducing the novel circle convolution, CCRBM holds a novel ring-like multilayer structure to learn 3D local features in a manner of structure preservation. Circle convolution convolves across 3D local regions with a novel circular sector convolution window by rotating itself along a xed circle direction. In the process of circle convolution, extra points are sampled on each 3D local region and projected onto the tangent plane of the center of the region. By this way, the projection distances in each sector window are employed to constitute the raw 3D feature called projection distance distribution (PDD). In addition, to eliminate the ambiguity of the initial location of a sector window, Fourier Transform Modulus (FTM) is used to transform the PDD into Fourier domain which is then conveyed to CCRBM. Experiments using the learned local features are conducted on three aspects: global shape retrieval, partial shape retrieval, and shape correspondence. The experimental results show that the learned local features outperform other state-of-the-art 3D shape descriptors.


  1. 1.
    Y. Gao, M. Wang, D. Tao, R. Ji, and Q. Dai, 3D object retrieval and recognition with hypergraph analysis, IEEE Transactions on Image Processing, vol. 21, no. 9, pp. 4290–4303, 2012.MathSciNetCrossRefGoogle Scholar
  2. 2.
    O. van Kaick, H. Zhang, G. Hamarneh, and D. Cohen-Or, A survey on shape correspondence, Computer Graphics Forum, vol. 30, no. 6, pp. 1681–1707, Sep 2011.CrossRefGoogle Scholar
  3. 3.
    X. Qian and C. Ye, Ncc-ransac: A fast plane extraction method for 3D range data segmentation, IEEE Transactions on Cybernetics, vol. 44, pp. 2771–2783, 2014.CrossRefGoogle Scholar
  4. 4.
    K.-C. Chan, C.-K. Koh, and C. S. G. Lee, A 3-d-point-cloud system for human-pose estimation, IEEE Transactions on Cybernetics, vol. 44, pp. 1486–1497, 2014.Google Scholar
  5. 5.
    O. S. Gedik and A. A. Alatan, 3D rigid body tracking using vision and depth sensors, IEEE Transaction on Cybernetics, vol. 43, no. 5, pp. 1395–1405, 2013.CrossRefGoogle Scholar
  6. 6.
    M. Liang, H. Min, R. Luo, and J. Zhu, Simultaneous recognition and modeling for learning 3D object models from everyday scenes, IEEE Transactions on Cybernetics, vol. PP, pp. 1–12, 2014.Google Scholar
  7. 7.
    F.-F. Li and P. Perona, A bayesian hierarchical model for learning natural scene categories, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, 2005, pp. 524–531.Google Scholar
  8. 8.
    A. M. Bronstein, M. M. Bronstein, L. J. Guibas, and M. Ovsjanikov, Shape google: geometric words and expressions for invariant shape retrieval, ACM Transactions on Graphics, vol. 30, no. 1, pp. 1–20, 2011.CrossRefGoogle Scholar
  9. 9.
    L. Shapira, A. Shamir, and D. Cohen-Or, Consistent mesh partitioning and skeletonisation using the shape diameter function, The Visual Computer, vol. 24, no. 4, pp. 249–259, 2008.CrossRefGoogle Scholar
  10. 10.
    A. Johnson and M. Hebert, Using spin images for efficient object recognition in cluttered 3D scenes, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 21, no. 5, pp. 433–449, 1999.CrossRefGoogle Scholar
  11. 11.
    T. Darom and Y. Keller, Scale-invariant features for 3D mesh models. IEEE Transactions on Image Processing, vol. 21, no. 5, pp. 2758–2769, 2012.MathSciNetCrossRefGoogle Scholar
  12. 12.
    Y. Bengio, A. Courville, and P. Vincent, Representation learning: A review and new perspectives, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, pp. 1798–1828, 2013.CrossRefGoogle Scholar
  13. 13.
    H. Yan, J. Lu, and X. Zhou, Prototype-based discriminative feature learning for kinship verification, IEEE Transactions on Cybernetics, vol. PP, pp. 1–13, 2014.Google Scholar
  14. 14.
    X. Lu, Y. Yuan, and P. Yan, Alternatively constrained dictionary learning for image superresolution, IEEE Transactions on Cybernetics, vol. 44, pp. 366–377, 2014.CrossRefGoogle Scholar
  15. 15.
    H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng, Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations, in The Annual International Conference on Machine Learning, 2009, pp. 609–616.Google Scholar
  16. 16.
    A. Krizhevsky, I. Sutskever, and G. E. Hinton, Imagenet classification with deep convolutional neural networks, in Advances in Neural Information Processing Systems, 2012, vol. 25, pp. 1097–1105.Google Scholar
  17. 17.
    A. Graves, A. rahman Mohamed, and G. E. Hinton, Speech recognition with deep recurrent neural networks, in IEEE International Conference on Acoustics, Speech, and Signal Processing, 2013, pp. 6645–6649.Google Scholar
  18. 18.
    S. Bu, Z. Liu, J. Han, J. Wu, and R. Ji, Learning high-level feature by deep belief networks for 3D model retrieval and recognition, IEEE Transactions on Multimedia, vol. 16, no. 8, pp. 2154–2167, 2014.CrossRefGoogle Scholar
  19. 19.
    Z. Liu, S. Chen, S. Bu, and K. Li, High-level semantic feature for 3D shape based on deep belief networks, Proceeding of IEEE International Conference on Multimedia and Expo, 2014, pp. 1–6.Google Scholar
  20. 20.
    B. Leng, X. Zhang, M. Yao, and Z. Xiong, A 3D model recognition mechanism based on deep boltzmann machines, Neurocomputing, vol. 151, pp. 593–602, 2014.CrossRefGoogle Scholar
  21. 21.
    P. Heider, A. Pierre-Pierre, R. Li, and C. Grimm, Local shape descriptors, a survey and evaluation, in Eurographics Workshop on 3D Object Retrieval, 2011, pp. 49–57.Google Scholar
  22. 22.
    J. Lodder, Curvature in the calculus curriculum. The American Mathematical Monthly, vol. 110, no. 7, pp. 593–605, 2003.MathSciNetCrossRefGoogle Scholar
  23. 23.
    J. Sun, M. Ovsjanikov, and L. J. Guibas, A concise and provably informative multi-scale signature based on heat diffusion, Computer Graphics Forum, vol. 28, no. 5, pp. 1383–1392, 2009.CrossRefGoogle Scholar
  24. 24.
    M. M. Bronstein and I. Kokkinos, Scale-invariant heat kernel signatures for non-rigid shape recognition, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2010, pp. 1704–1711.Google Scholar
  25. 25.
    M. Aubry, U. Schlickewei, and D. Cremers, The wave kernel signature: A quantum mechanical approach to shape analysis, in International Conference on Computer Vision Workshops, 2011, pp. 1626–1633.Google Scholar
  26. 26.
    R. Socher, B. Huval, B. Bhat, C. D. Manning, and A. Y. Ng, Convolutional-recursive deep learning for 3D object classification, in Advances in Neural Information Processing Systems, 2012, vol. 25, pp. 665–673.Google Scholar
  27. 27.
    Z. Wu, S. Song, A. Khosla, X. Tang, and J. Xiao, 3D ShapeNets for 2.5D object recognition and Next-Best-View prediction, arXiv: 1406.5670, 2014.Google Scholar
  28. 28.
    G. E. Hinton, S. Osindero, and Y.-W. Teh, A fast learning algorithm for deep belief nets, Neural Computation, vol. 18, no. 7, pp. 1527–1554, 2006.MathSciNetCrossRefGoogle Scholar
  29. 29.
    G. E. Hinton and R. R. Salakhutdinov, Reducing the dimensionality of data with neural networks, Science, vol. 313, pp. 504–507, 2006.MathSciNetCrossRefGoogle Scholar
  30. 30.
    H. Ackley, E. Hinton, and J. Sejnowski, A learning algorithm for boltzmann machines, Cognitive Science, pp. 147–169, 1985.Google Scholar
  31. 31.
    G. E. Hinton, Training products of experts by minimizing contrastive divergence, Neural Computation, vol. 14, no. 8, pp. 1771–1800, 2002.CrossRefGoogle Scholar
  32. 32.
    G. Casella and E. I. George, Explaining the gibbs sampler, The American Statistician, vol. 46, no. 3, pp. 167–174, 1992.MathSciNetGoogle Scholar
  33. 33.
    R. Osada, T. Funkhouser, B. Chazelle, and D. Dobkin, Shape distributions, ACM Transactions on Graphics, vol. 21, no. 4, pp. 807–832, 2002.MathSciNetCrossRefGoogle Scholar
  34. 34.
    G. Peyre and L. D. Cohen, Geodesic remeshing using front propagation, International Journal of Computer Vision, vol. 69, no. 1, pp. 145–156, 2006.CrossRefGoogle Scholar
  35. 35.
    S. Zokai and G. Wolberg, Image registration using log-polar mappings for recovery of large-scale similarity and projective transformations, vol. 14, pp. 1422–1434, 2005.Google Scholar
  36. 36.
    I. Kokkinos and A. L. Yuille, Scale invariance without scale selection, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2008, pp. 1–8.Google Scholar
  37. 37.
    I. Kokkinos, M. M. Bronstein, R. Litman, and A. M. Bronstein, Intrinsic shape context descriptors for deformable shapes, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2012, pp. 159–166.Google Scholar
  38. 38.
    K. Siddiqi, J. Zhang, D. Macrini, A. Shokoufandeh, S. Bouix, and S. Dickinson, Retrieving articulated 3D models using medial surfaces, Machine Vision and Applications, vol. 19, no. 4, pp. 261–275, 2008.CrossRefGoogle Scholar
  39. 39.
    D. Giorgi, S. Biasotti, and L. Paraboschi, Shape retrieval contest 2007: Watertight models track, 2007.Google Scholar
  40. 40.
    S. Marini, L. Paraboschi, and S. Biasotti, Shape retrieval contest 2007: Partial matching track, in SHREC in conjunction with IEEE Shape Modelling International, 2007, pp. 13–16.Google Scholar
  41. 41.
    D. Anguelov, P. Srinivasan, H.-C. Pang, D. Koller, S. Thrun, and J. Davis, The correlated correspondence algorithm for unsupervised registration of nonrigid surfaces, Proceeding of the Neural Information Processing Systems, 2004, pp. 33–40.Google Scholar
  42. 42.
    V. G. Kim, Y. Lipman, and T. Funkhouser, Blended intrinsic maps, ACM Transaction on Graphics, vol. 30, no. 4, pp. 79:1–79:12, 2011.Google Scholar
  43. 43.
    A. Elad and R. Kimmel, On bending invariant signatures for surfaces, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, no. 10, pp. 1285–1295, 2003.CrossRefGoogle Scholar
  44. 44.
    R. Martin, W. F. Erich, and P. Niklas, Laplace-spectra as fingerprints for shape matching, in Symposium on Solid and Physical Modeling, 2005, pp. 101–106.Google Scholar
  45. 45.
    X. Wang, Y. Liu, and H. Zha, Intrinsic spin images: A subspace decomposition approach to understanding 3D deformable shapes, in 5th International Symposium 3D Data Processing, Visualization and Transmission, 2010, pp. 17–20.Google Scholar
  46. 46.
    D. Chen, X. Tian, Y. Shen, and M. Ouhyoung, On visual similarity based 3D model retrieval, Computer Graphics Forum, vol. 22, no. 3, pp. 223–232, 2003.CrossRefGoogle Scholar
  47. 47.
    M. Kazhdan, T. Funkhouser, and S. Rusinkiewicz, Rotation invariant spherical harmonic representation of 3D shape descriptors, Proceedings of Eurographics Symposium on Geometry Processing, 2003, pp. 156–165.Google Scholar
  48. 48.
    G. Lavou, Combination of Bag-of-Words Descriptors for Robust Partial Shape Retrieval, The Visual Computer, vol. 28, no. 9, pp. 931–942, 2012.CrossRefGoogle Scholar
  49. 49.
    H. Tabia, M. Daoudi, J.-P. Vandeborre, and O. Colot, A new 3D-matching method of non-rigid and partially similar models using curve analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 4, pp. 852–858, 2011.CrossRefGoogle Scholar
  50. 50.
    R. Toldo, U. Castellani, and A. Fusiello, Visual vocabulary signature for 3D object retrieval and partial matching. in Eurographics Workshop on 3D Object Retrieval, 2009, pp. 21–28.Google Scholar
  51. 51.
    H. Tabia, H. Laga, D. Picard, and P.-H. Gosselin, Covariance descriptors for 3D shape matching and retrieval, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 4815–4192, 2014.Google Scholar
  52. 52.
    S. Biasotti, S. Marini, M. Spagnuolo, and B. Falcidieno, Sub-part correspondence by structural descriptors of 3D shapes. Computer-Aided Design, vol. 38, pp. 1002–1019, 2006.CrossRefGoogle Scholar
  53. 53.
    N. D. Cornea, M. F. Demirci, D. Silver, A. Shokoufandeh, S. J. Dickinson, and P. B. Kantor, 3D object retrieval using many-to-many matching of curve skeletons, Proceedings of the International Conference on Shape Modeling and Applications, 2005, pp. 368–373.Google Scholar
  54. 54.
    J. Tierny, J.-P. Vandeborre, and M. Daoudi, Partial 3D shape retrieval by reeb pattern unfolding. Computer Graphics Forum, vol. 28, pp. 41–55, 2009.CrossRefGoogle Scholar
  55. 55.
    H. W. Kuhn, The hungarian method for the assignment problem, Naval Research Logistics Quarterly, vol. 2, pp. 83–97, 1955.MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.Northwestern Polytechnical UniversityXi’anChina

Personalised recommendations