Skip to main content
SpringerLink
Log in

Video stitching using interacting multiple model based feature tracking

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper, we propose a novel video stitching algorithm for videos from multiple cameras using interacting multiple model feature tracking to maintain spatial-temporal consistency. Apart from image alignment challenges while stitching a video, inter frame consistency, video jitter due to moving object and camera movement also need to be addressed. To address these challenges, feature point detected in the initial frame is tracked in the subsequent frames to maintain spatial-temporal consistency and reduce computation complexity in feature point detection. Firstly, the feature points are detected using Features from Accelerated Segment Test algorithm. Secondly, using Binary Robust Invariant Scalable Keypoints descriptor values are obtained from detected feature points and matched using hamming distance. The outliers are removed by Random Sample Consensus Algorithm. Once, the first frame is stitched, feature points detected from first frame are tracked using kalman filter with interacting multiple model. The tracked feature points are descripted and homography between the frames are found. This will maintain the spatio-temporal consistency by reducing jitter effect between frames after stitching, and since the frames are neglected from feature point detection, computation complexity is reduced. From the experimental results, we observed that the execution time of the proposed method is less and the performance of structural similarity is better than the existing methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Bar-Shalom Y (1990) Multitarget-multisensor tracking: advanced applications. Artech House, Norwood, p 391

    Google Scholar 

  2. Bishop G, Welch G (2001) An introduction to the kalman filter. Proc of SIGGRAPH, Course 8(27599-23175):41

  3. Blom HA, Bar-Shalom Y (1988) The interacting multiple model algorithm for systems with markovian switching coefficients. IEEE Trans Autom Control 33 (8):780–783

    Article  Google Scholar 

  4. Brown M, Lowe DG (2007) Automatic panoramic image stitching using invariant features. Int J Comput Vis 74(1):59–73

    Article  Google Scholar 

  5. Brown M, Lowe DG et al. (2003) Recognising panoramas. In: ICCV, vol 3, pp 1218–1227

  6. Cui J, Liu Y, Xu Y, Zhao H, Zha H (2013) Tracking generic human motion via fusion of low-and high-dimensional approaches. IEEE Trans Syst Man Cybern Syst Hum 43(4):996–1002

    Article  Google Scholar 

  7. El-Saban M, Izz M, Kaheel A (2010) Fast stitching of videos captured from freely moving devices by exploiting temporal redundancy. In: 2010 17th IEEE international conference on image processing (ICIP), IEEE, pp 1193–1196

  8. Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395

    Article  MathSciNet  Google Scholar 

  9. Guo H, Liu S, He T, Zhu S, Zeng B, Gabbouj M (2016) Joint video stitching and stabilization from moving cameras. IEEE Trans Image Process 25 (11):5491–5503

    Article  MathSciNet  Google Scholar 

  10. Hamza A, Hafiz R, Khan MM, Cho Y, Cha J (2015) Stabilization of panoramic videos from mobile multi-camera platforms. Image Vis Comput 37:20–30

    Article  Google Scholar 

  11. Harris C, Stephens M (1988) A combined corner and edge detector. In: Alvey vision conference, Manchester, UK, vol 15, pp 10–5244

  12. He B, Yu S (2015) Parallax-robust surveillance video stitching. Sensors 16 (1):7

    Article  MathSciNet  Google Scholar 

  13. Jia J, Tang CK (2005) Eliminating structure and intensity misalignment in image stitching. In: 10th IEEE international conference on computer vision, 2005. ICCV 2005. IEEE, vol 2, pp 1651-1658

  14. Jia J, Tang CK (2008) Image stitching using structure deformation. IEEE Trans Pattern Anal Mach Intell 30(4):617–631

    Article  Google Scholar 

  15. Jiang W, Gu J (2015) Video stitching with spatial-temporal content-preserving warping. In: 2015 IEEE conference on computer vision and pattern recognition workshops (CVPRW). IEEE, pp 42-48

  16. Joshi H, Sinha MK (2013) A survey on image mosaicing techniques. International Journal of Advanced Research in Computer Engineering & Technology (IJARCET), vol 2

  17. Khan N, McCane B, Mills S (2015) Better than sift? Mach Vis Appl 26(6):819–836

    Article  Google Scholar 

  18. Kirubarajan T, Bar-Shalom Y (2003) Kalman filter versus imm estimator: when do we need the latter? IEEE Trans Aerosp Electron Syst 39(4):1452–1457

    Article  Google Scholar 

  19. Kwon OS, Ha YH (2010) Panoramic video using scale-invariant feature transform with embedded color-invariant values. IEEE transactions on Consumer Electronics 56(2)

  20. Li J, Yang T, Yu J, Lu Z, Lu P, Jia X, Chen W (2014) Fast aerial video stitching. Int J Adv Robot Syst 11(10):167

    Article  Google Scholar 

  21. Li J, Xu W, Zhang J, Zhang M, Wang Z, Li X (2015) Efficient video stitching based on fast structure deformation. IEEE Trans Cybern 45(12):2707–2719

    Article  Google Scholar 

  22. Liu L, Cheng L, Liu Y, Jia Y, Rosenblum DS (2016a) Recognizing complex activities by a probabilistic interval-based model. In: AAAI, vol 30, pp 1266–1272

  23. Liu Y, Cui J, Zhao H, Zha H (2012) Fusion of low-and high-dimensional approaches by trackers sampling for generic human motion tracking. In: 2012 21st international conference on pattern recognition (ICPR), IEEE, pp 898–901

  24. Liu Y, Nie L, Han L, Zhang L, Rosenblum DS (2015) Action2activity: Recognizing complex activities from sensor data. In: IJCAI, vol 2015, pp 1617-1623

  25. Liu Y, Nie L, Liu L, Rosenblum DS (2016) From action to activity: sensor-based activity recognition. Neurocomputing 181:108–115

    Article  Google Scholar 

  26. Liu Y, Zhang L, Nie L, Yan Y, Rosenblum DS (2016) Fortune teller: Predicting your career path. In: AAAI, vol 2016, pp 201–207

  27. Liu Y, Zheng Y, Liang Y, Liu S, Rosenblum DS (2016) Urban water quality prediction based on multi-task multi-view learning

  28. Mills A, Dudek G (2009) Image stitching with dynamic elements. Image Vis Comput 27(10):1593–1602

    Article  Google Scholar 

  29. Perazzi F, Sorkine-Hornung A, Zimmer H, Kaufmann P, Wang O, Watson S, Gross M (2015) Panoramic video from unstructured camera arrays. In: Computer graphics forum, Wiley online library, vol 34, pp 57–68

  30. Rosten E, Porter R, Drummond T (2010) Faster and better: A machine learning approach to corner detection. IEEE Trans Pattern Anal Mach Intell 32(1):105–119

    Article  Google Scholar 

  31. Shi J et al. (1994) Good features to track. In: 1994 IEEE computer society conference on computer vision and pattern recognition, 1994. Proceedings CVPR’94. IEEE, pp 593-600

  32. Sinha SN, Frahm JM, Pollefeys M, Genc Y (2011) Feature tracking and matching in video using programmable graphics hardware. Mach Vis Appl 22(1):207–217

    Article  Google Scholar 

  33. Song T, Jeon C, Ko H (2015) Image stitching using chaos-inspired dissimilarity measure. Electron Lett 51(3):232–234

    Article  Google Scholar 

  34. Szeliski R (1996) Video mosaics for virtual environments. IEEE Comput Graph Appl 16:22–30

    Article  Google Scholar 

  35. Szeliski R (2006) Image alignment and stitching: A tutorial. Foundations and Trends®, in Computer Graphics and Vision 2(1):1–104

  36. Xu W, Mulligan J (2010) Performance evaluation of color correction approaches for automatic multi-view image and video stitching. In: 2010 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 263-270

  37. Xu W, Mulligan J (2013) Panoramic video stitching from commodity hdtv cameras. Multimedia Syst 19(5):407–426

    Article  Google Scholar 

  38. Zaragoza J, Chin TJ, Brown MS, Suter D (2013) As-projective-as-possible image stitching with moving dlt. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2339–2346

Download references

Author information

Authors and Affiliations

  1. Department of Electronics Engineering, Madras Institute of Technology, Anna University, Chrompet, Chennai, 600044, India

    K. Krishnakumar & S. Indira Gandhi

Authors
  1. K. Krishnakumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. Indira Gandhi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to K. Krishnakumar.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Krishnakumar, K., Gandhi, S.I. Video stitching using interacting multiple model based feature tracking. Multimed Tools Appl 78, 1375–1397 (2019). https://doi.org/10.1007/s11042-018-6116-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6116-0

Keywords

Search

Navigation