Skip to main content
SpringerLink
Log in

Robust and real-time pose tracking for augmented reality on mobile devices

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper addresses robust and ultrafast pose tracking on mobile devices, such as smartphones and small drones. Existing methods, relying on either vision analysis or inertial sensing, are either too computational heavy to achieve real-time performance on a mobile platform, or not sufficiently robust to address unique challenges in mobile scenarios, including rapid camera motions, long exposure time of mobile cameras, etc. This paper presents a novel hybrid tracking system which utilizes on-device inertial sensors to greatly accelerate the visual feature tracking process and improve its robustness. In particular, our system adaptively resizes each video frame based on inertial sensor data and applies a highly efficient binary feature matching method to track the object pose in each resized frame with little accuracy degradation. This tracking result is revised periodically by a model-based feature tracking method (Hare et al. 2012) to reduce accumulated errors. Furthermore, an inertial tracking method and a solution of fusing its results with the feature tracking results are employed to further improve the robustness and efficiency. We first evaluate our hybrid system using a dataset consisting of 16 video clips with synchronized inertial sensing data and then assess its performance in a mobile augmented reality application. Experimental results demonstrated our method’s superior performance to a state-of-the-art feature tracking method (Hare et al. 2012), a direct tracking method (Engel et al. 2014) and the Vuforia SDK (Ibañez and Figueras 2013), and can run at more than 40 Hz on a standard smartphone. We will release the source code with the pubilication of this paper.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Chliveros G, Pateraki M, Trahanias P (2013) Robust multi-hypothesis 3d object pose tracking International conference on computer vision systems. Springer, pp 234–243

  2. Chum O, Matas J (2005) Matching with prosac-progressive sample consensus 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), vol 1. IEEE, pp 220–226

  3. Crivellaro A, Rad M, Verdie Y, Moo Yi K, Fua P, Lepetit V (2015) A novel representation of parts for accurate 3d object detection and tracking in monocular images Proceedings of the IEEE international conference on computer vision, pp 4391–4399

  4. Daniel W, Dieter S (2007) Artoolkitplus for pose tracking on mobile devices Proceedings of 12th computer vision winter workshop

  5. DiVerdi S, Hollerer T (2007) Groundcam: a tracking modality for mobile mixed reality 2007 IEEE virtual reality conference. IEEE, pp 75–82

  6. Engel J, Schöps T, Cremers D (2014) Lsd-slam: large-scale direct monocular slam European conference on computer vision. Springer, pp 834–849

  7. Engel J, Sturm J, Cremers D (2013) Semi-dense visual odometry for a monocular camera Proceedings of the IEEE international conference on computer vision, pp 1449–1456

  8. Fiala M (2010) Designing highly reliable fiducial markers. IEEE Trans Pattern Anal Mach Intell 32(7):1317–1324

    Article  Google Scholar 

  9. Hallaway D, Feiner S, Höllerer T (2004) Bridging the gaps: hybrid tracking for adaptive mobile augmented reality. Appl Artif Intell 18(6):477–500

    Article  Google Scholar 

  10. Hare S, Saffari A, Torr PH (2012) Efficient online structured output learning for keypoint-based object tracking IEEE conference on computer vision and pattern recognition (CVPR), 2012. IEEE, pp 1894–1901

  11. Ibañez AS, Figueras JP (2013) Vuforia v1. 5 sdk: analysis and evaluation of capabilities. Ph.D. thesis, Master Thesis, Universitat Politecnica De Catalunya

  12. Kato H, Billinghurst M (1999) Marker tracking and hmd calibration for a video-based augmented reality conferencing system 2nd IEEE and ACM international workshop on augmented reality, 1999.(IWAR’99) proceedings. IEEE, pp 85–94

  13. Klein G, Murray D (2007) Parallel tracking and mapping for small ar workspaces 6th IEEE and ACM international symposium on mixed and augmented reality, 2007. ISMAR 2007. IEEE, pp 225–234

  14. Klein G, Murray D (2009) Parallel tracking and mapping on a camera phone 8th IEEE international symposium on mixed and augmented reality, 2009. ISMAR 2009. IEEE, pp 83–86

  15. Klein G, Murray DW (2006) Full-3d edge tracking with a particle filter BMVC, pp 1119–1128

  16. Lepetit V, Fua P (2006) Keypoint recognition using randomized trees. IEEE Trans Pattern Anal Mach Intell 28(9):1465–1479

    Article  Google Scholar 

  17. Li M, Kim BH, Mourikis AI (2013) Real-time motion tracking on a cellphone using inertial sensing and a rolling-shutter camera Robotics and automation (ICRA), 2013 IEEE international conference on. IEEE, pp 4712–4719

  18. Ligorio G, Sabatini AM (2013) Extended kalman filter-based methods for pose estimation using visual, inertial and magnetic sensors: comparative analysis and performance evaluation. Sensors 13(2):1919–1941

    Article  Google Scholar 

  19. Morgan DL (2014) Improved orientation estimation for smart phone indoor localization. Ph.D. thesis, UNIVERSITY OF CALIFORNIA SANTA BARBARA

  20. Obeidy WK, Arshad H, Chowdhury SA, Parhizkar B, Huang J (2013) Increasing the tracking efficiency of mobile augmented reality using a hybrid tracking technique International visual informatics conference. Springer, pp 447–457

  21. Ozuysal M, Calonder M, Lepetit V, Fua P (2010) Fast keypoint recognition using random ferns. IEEE Trans Pattern Anal Mach Intell 32(3):448–461

    Article  Google Scholar 

  22. Payet N, Todorovic S (2011) From contours to 3d object detection and pose estimation 2011 International conference on computer vision. IEEE, pp 983–990

  23. Prisacariu VA, Reid ID (2012) Pwp3d: real-time segmentation and tracking of 3d objects. Int J Comput Vis 98(3):335–354

    Article  MathSciNet  Google Scholar 

  24. Prisacariu VA, Segal AV, Reid I (2012) Simultaneous monocular 2d segmentation, 3d pose recovery and 3d reconstruction Asian conference on computer vision. Springer, pp 593–606

  25. Rolland JP, Davis L, Baillot Y (2001) A survey of tracking technology for virtual environments. Fundamentals of Wearable Computers and Augmented Reality 1 (1):67–112

    Google Scholar 

  26. Rosten E, Porter R, Drummond T (2010) Faster and better: a machine learning approach to corner detection. IEEE Trans Pattern Anal Mach Intell 32(1):105–119

    Article  Google Scholar 

  27. Rublee E, Rabaud V, Konolige K, Bradski G (2011) Orb: an efficient alternative to sift or surf 2011 International conference on computer vision. IEEE, pp 2564–2571

  28. Schöps T, Engel J, Cremers D (2014) Semi-dense visual odometry for ar on a smartphone IEEE international symposium on mixed and augmented reality (ISMAR), 2014. IEEE, pp 145–150

  29. Wagner D, Langlotz T, Schmalstieg D (2008) Robust and unobtrusive marker tracking on mobile phones 7th IEEE/ACM international symposium on mixed and augmented reality, 2008. ISMAR 2008. IEEE, pp 121–124

  30. Wagner D, Reitmayr G, Mulloni A, Drummond T, Schmalstieg D (2008) Pose tracking from natural features on mobile phones Proceedings of the 7th IEEE/ACM international symposium on mixed and augmented reality. IEEE Computer Society, pp 125–134

  31. Wagner D, Reitmayr G, Mulloni A, Drummond T, Schmalstieg D (2010) Real-time detection and tracking for augmented reality on mobile phones. IEEE Trans Vis Comput Graph 16(3):355–368

    Article  Google Scholar 

  32. White S, Feiner S, Kopylec J (2006) Virtual vouchers: prototyping a mobile augmented reality user interface for botanical species identification 3D user interfaces (3DUI’06). IEEE, pp 119– 126

  33. Yang X, Cheng KTT (2014) Local difference binary for ultrafast and distinctive feature description. IEEE Trans Pattern Anal Mach Intell 36(1):188–194

    Article  Google Scholar 

  34. Yang X, Huang C, Cheng KTT (2014) libldb: a library for extracting ultrafast and distinctive binary feature description Proceedings of the 22nd ACM international conference on multimedia. ACM, pp 671–674

  35. Yang X, Si X, Xue T, Cheng KTT (2015) [Poster] Fusion of vision and inertial sensing for accurate and efficient pose tracking on smartphones IEEE international symposium on mixed and augmented reality (ISMAR), 2015. IEEE, pp 68–71

  36. Yang X, Si X, Xue T, Zhang L, Cheng KTT (2015) Vision-inertial hybrid tracking for robust and efficient augmented reality on smartphones Proceedings of the 23rd ACM international conference on multimedia. ACM, pp 1039–1042

Download references

Acknowledgements

This work is funded by the National Science Foundation of China (grant no. 61502188).

Author information

Authors and Affiliations

  1. School of Electronics Information and Communications, Huazhong University of Science and Technology, 1037 Luoyu Road, 1st South Building, Rm. W205, Wuhan, 430074, China

    Xin Yang, Jiabin Guo & Tangli Xue

  2. Department of Electrical and Computer Engineering, University of California, Santa Barbara, CA, USA

    Kwang-Ting (Tim) Cheng

  3. Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong, China

    Kwang-Ting (Tim) Cheng

Authors
  1. Xin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiabin Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tangli Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kwang-Ting (Tim) Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xin Yang.

Electronic supplementary material

Below is the link to the electronic supplementary material.

(MP4 76.2 MB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, X., Guo, J., Xue, T. et al. Robust and real-time pose tracking for augmented reality on mobile devices. Multimed Tools Appl 77, 6607–6628 (2018). https://doi.org/10.1007/s11042-017-4575-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-4575-3

Keywords

Search

Navigation