3D Model-Based 6D Object Pose Tracking on RGB Images

  • Mateusz Majcher
  • Bogdan KwolekEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12033)


In this paper, we present a 3D-model based algorithm for 6D object pose estimation and tracking on segmented RGB images. The object of interest is segmented by U-Net neural network trained on a set of manually delineated images. A Particle Swarm Optimization is used to estimate the 6D object pose by projecting the 3D object model and then matching the projected image with the image acquired by the camera. The tracking of 6D object pose is formulated as a dynamic optimization problem. In order to keep necessary human intervention minimal, we use an automated turntable setup to prepare a 3D object model and to determine the ground-truth poses. We compare the experimental results obtained by our algorithm with results achieved by PWP3D algorithm.


Tracking 6D pose of object Image segmentation Optimization 



This work was supported by Polish National Science Center (NCN) under a research grant 2017/27/B/ST6/01743.


  1. 1.
    Brachmann, E., Krull, A., Michel, F., Gumhold, S., Shotton, J., Rother, C.: Learning 6D object pose estimation using 3D object coordinates. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8690, pp. 536–551. Springer, Cham (2014). Scholar
  2. 2.
    Hinterstoisser, S., et al.: Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes. In: Lee, K.M., Matsushita, Y., Rehg, J.M., Hu, Z. (eds.) ACCV 2012. LNCS, vol. 7724, pp. 548–562. Springer, Heidelberg (2013). Scholar
  3. 3.
    Marchand, E., Uchiyama, H., Spindler, F.: Pose estimation for augmented reality: a hands-on survey. IEEE Trans. Vis. Comput. Graph. 22(12), 2633–2651 (2016)CrossRefGoogle Scholar
  4. 4.
    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Hinterstoisser, S., et al.: Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes. In: International Conference on Computer Vision, pp. 858–865 (2011)Google Scholar
  6. 6.
    Model-Based Training, Detection and Pose Estimation of Texture-less 3D Objects in Heavily Cluttered Scenes. In: Proceedings of the Asian Conference on Computer Vision (ACCV) (2012)Google Scholar
  7. 7.
    Brachmann, E., Michel, F., Krull, A., Yang, M., Gumhold, S., Rother, C.: Uncertainty-driven 6D pose estimation of objects and scenes from a single RGB image. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3364–3372 (2016)Google Scholar
  8. 8.
    Pouyanfar, S., et al.: A survey on deep learning: algorithms, techniques, and applications. ACM Comput. Surv. 51(5), 1–36 (2018)CrossRefGoogle Scholar
  9. 9.
    Wohlhart, P., Lepetit, V.: Learning descriptors for object recognition and 3D pose estimation. In: Conference on Computer Vision and Pattern Recognition, pp. 1–10 (2015)Google Scholar
  10. 10.
    Xiang, Y., Schmidt, T., Narayanan, V., Fox, D.: PoseCNN: a convolutional neural network for 6D object pose estimation in cluttered scenes. In: Robotics: Science and Systems XIV (RSS) (2018)Google Scholar
  11. 11.
    Rad, M., Lepetit, V.: BB8: a scalable, accurate, robust to partial occlusion method for predicting the 3D poses of challenging objects without using depth. In: IEEE International Conference on Computer Vision, ICCV, pp. 3848–3856 (2017)Google Scholar
  12. 12.
    Deng, X., Mousavian, A., Xiang, Y., Xia, F., Bretl, T., Fox, D.: PoseRBPF: a rao-blackwellized particle filter for 6D object pose tracking. In: Robotics: Science and Systems (RSS) (2019)Google Scholar
  13. 13.
    Gritsenko, P., Gritsenko, I., Seidakhmet, A., Kwolek, B.: Plane-based humanoid robot navigation and object model construction for grasping. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11129, pp. 649–664. Springer, Cham (2019). Scholar
  14. 14.
    Kennedy, J., Eberhart, R.: Particle Swarm Optimization. In: Proceedings of IEEE International Conference on Neural Networks, pp. 1942–1948. IEEE Press, Piscataway (1995)Google Scholar
  15. 15.
    Sengupta, S., Basak, S., Peters, R.A.: Particle swarm optimization: a survey of historical and recent developments with hybridization perspectives. Mach. Learn. Knowl. Extr. 1(1), 157–191 (2019)CrossRefGoogle Scholar
  16. 16.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  17. 17.
    Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICML, vol. 37, pp. 448–456 (2015)Google Scholar
  18. 18.
    Ugolotti, R., Nashed, Y.S.G., Mesejo, P., Ivekovič, V., Mussi, L., Cagnoni, S.: Particle swarm optimization and differential evolution for model-based object detection. Appl. Soft Comput. 13(6), 3092–3105 (2013)CrossRefGoogle Scholar
  19. 19.
    Zabulis, X., Lourakis, M.I.A., Stefanou, S.S.: 3D pose refinement using rendering and texture-based matching. In: Chmielewski, L.J., Kozera, R., Shin, B.-S., Wojciechowski, K. (eds.) ICCVG 2014. LNCS, vol. 8671, pp. 672–679. Springer, Cham (2014). Scholar
  20. 20.
    Rymut, B., Kwolek, B., Krzeszowski, T.: GPU-accelerated human motion tracking using particle filter combined with PSO. In: Blanc-Talon, J., Kasinski, A., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2013. LNCS, vol. 8192, pp. 426–437. Springer, Cham (2013). Scholar
  21. 21.
    Tan, Y.: GPU-based parallel implementation of swarm intelligence algorithms. Morgan Kaufmann, San Francisco (2016)Google Scholar
  22. 22.
    Izadi, S., et al.: KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, New York, NY, USA, pp. 559–568. ACM (2011)Google Scholar
  23. 23.
    Bradski, G., Kaehler, A.: Learning OpenCV: Computer Vision in C++ with the OpenCV Library, 2nd edn. O’Reilly Media Inc, Sebastopol (2013)Google Scholar
  24. 24.
    Prisacariu, V.A., Reid, I.D.: PWP3D: real-time segmentation and tracking of 3D objects. Int. J. Comput. Vision 98(3), 335–354 (2012)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.AGH University of Science and TechnologyKrakowPoland

Personalised recommendations