Multimedia Tools and Applications

, Volume 78, Issue 2, pp 1375–1397 | Cite as

Video stitching using interacting multiple model based feature tracking

  • K. KrishnakumarEmail author
  • S. Indira Gandhi


In this paper, we propose a novel video stitching algorithm for videos from multiple cameras using interacting multiple model feature tracking to maintain spatial-temporal consistency. Apart from image alignment challenges while stitching a video, inter frame consistency, video jitter due to moving object and camera movement also need to be addressed. To address these challenges, feature point detected in the initial frame is tracked in the subsequent frames to maintain spatial-temporal consistency and reduce computation complexity in feature point detection. Firstly, the feature points are detected using Features from Accelerated Segment Test algorithm. Secondly, using Binary Robust Invariant Scalable Keypoints descriptor values are obtained from detected feature points and matched using hamming distance. The outliers are removed by Random Sample Consensus Algorithm. Once, the first frame is stitched, feature points detected from first frame are tracked using kalman filter with interacting multiple model. The tracked feature points are descripted and homography between the frames are found. This will maintain the spatio-temporal consistency by reducing jitter effect between frames after stitching, and since the frames are neglected from feature point detection, computation complexity is reduced. From the experimental results, we observed that the execution time of the proposed method is less and the performance of structural similarity is better than the existing methods.


Video stitching Video registration Feature tracking Kalman filter Interacting multiple model 


  1. 1.
    Bar-Shalom Y (1990) Multitarget-multisensor tracking: advanced applications. Artech House, Norwood, p 391Google Scholar
  2. 2.
    Bishop G, Welch G (2001) An introduction to the kalman filter. Proc of SIGGRAPH, Course 8(27599-23175):41Google Scholar
  3. 3.
    Blom HA, Bar-Shalom Y (1988) The interacting multiple model algorithm for systems with markovian switching coefficients. IEEE Trans Autom Control 33 (8):780–783CrossRefGoogle Scholar
  4. 4.
    Brown M, Lowe DG (2007) Automatic panoramic image stitching using invariant features. Int J Comput Vis 74(1):59–73CrossRefGoogle Scholar
  5. 5.
    Brown M, Lowe DG et al. (2003) Recognising panoramas. In: ICCV, vol 3, pp 1218–1227Google Scholar
  6. 6.
    Cui J, Liu Y, Xu Y, Zhao H, Zha H (2013) Tracking generic human motion via fusion of low-and high-dimensional approaches. IEEE Trans Syst Man Cybern Syst Hum 43(4):996–1002CrossRefGoogle Scholar
  7. 7.
    El-Saban M, Izz M, Kaheel A (2010) Fast stitching of videos captured from freely moving devices by exploiting temporal redundancy. In: 2010 17th IEEE international conference on image processing (ICIP), IEEE, pp 1193–1196Google Scholar
  8. 8.
    Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395MathSciNetCrossRefGoogle Scholar
  9. 9.
    Guo H, Liu S, He T, Zhu S, Zeng B, Gabbouj M (2016) Joint video stitching and stabilization from moving cameras. IEEE Trans Image Process 25 (11):5491–5503MathSciNetCrossRefGoogle Scholar
  10. 10.
    Hamza A, Hafiz R, Khan MM, Cho Y, Cha J (2015) Stabilization of panoramic videos from mobile multi-camera platforms. Image Vis Comput 37:20–30CrossRefGoogle Scholar
  11. 11.
    Harris C, Stephens M (1988) A combined corner and edge detector. In: Alvey vision conference, Manchester, UK, vol 15, pp 10–5244Google Scholar
  12. 12.
    He B, Yu S (2015) Parallax-robust surveillance video stitching. Sensors 16 (1):7MathSciNetCrossRefGoogle Scholar
  13. 13.
    Jia J, Tang CK (2005) Eliminating structure and intensity misalignment in image stitching. In: 10th IEEE international conference on computer vision, 2005. ICCV 2005. IEEE, vol 2, pp 1651-1658Google Scholar
  14. 14.
    Jia J, Tang CK (2008) Image stitching using structure deformation. IEEE Trans Pattern Anal Mach Intell 30(4):617–631CrossRefGoogle Scholar
  15. 15.
    Jiang W, Gu J (2015) Video stitching with spatial-temporal content-preserving warping. In: 2015 IEEE conference on computer vision and pattern recognition workshops (CVPRW). IEEE, pp 42-48Google Scholar
  16. 16.
    Joshi H, Sinha MK (2013) A survey on image mosaicing techniques. International Journal of Advanced Research in Computer Engineering & Technology (IJARCET), vol 2Google Scholar
  17. 17.
    Khan N, McCane B, Mills S (2015) Better than sift? Mach Vis Appl 26(6):819–836CrossRefGoogle Scholar
  18. 18.
    Kirubarajan T, Bar-Shalom Y (2003) Kalman filter versus imm estimator: when do we need the latter? IEEE Trans Aerosp Electron Syst 39(4):1452–1457CrossRefGoogle Scholar
  19. 19.
    Kwon OS, Ha YH (2010) Panoramic video using scale-invariant feature transform with embedded color-invariant values. IEEE transactions on Consumer Electronics 56(2)Google Scholar
  20. 20.
    Li J, Yang T, Yu J, Lu Z, Lu P, Jia X, Chen W (2014) Fast aerial video stitching. Int J Adv Robot Syst 11(10):167CrossRefGoogle Scholar
  21. 21.
    Li J, Xu W, Zhang J, Zhang M, Wang Z, Li X (2015) Efficient video stitching based on fast structure deformation. IEEE Trans Cybern 45(12):2707–2719CrossRefGoogle Scholar
  22. 22.
    Liu L, Cheng L, Liu Y, Jia Y, Rosenblum DS (2016a) Recognizing complex activities by a probabilistic interval-based model. In: AAAI, vol 30, pp 1266–1272Google Scholar
  23. 23.
    Liu Y, Cui J, Zhao H, Zha H (2012) Fusion of low-and high-dimensional approaches by trackers sampling for generic human motion tracking. In: 2012 21st international conference on pattern recognition (ICPR), IEEE, pp 898–901Google Scholar
  24. 24.
    Liu Y, Nie L, Han L, Zhang L, Rosenblum DS (2015) Action2activity: Recognizing complex activities from sensor data. In: IJCAI, vol 2015, pp 1617-1623Google Scholar
  25. 25.
    Liu Y, Nie L, Liu L, Rosenblum DS (2016) From action to activity: sensor-based activity recognition. Neurocomputing 181:108–115CrossRefGoogle Scholar
  26. 26.
    Liu Y, Zhang L, Nie L, Yan Y, Rosenblum DS (2016) Fortune teller: Predicting your career path. In: AAAI, vol 2016, pp 201–207Google Scholar
  27. 27.
    Liu Y, Zheng Y, Liang Y, Liu S, Rosenblum DS (2016) Urban water quality prediction based on multi-task multi-view learningGoogle Scholar
  28. 28.
    Mills A, Dudek G (2009) Image stitching with dynamic elements. Image Vis Comput 27(10):1593–1602CrossRefGoogle Scholar
  29. 29.
    Perazzi F, Sorkine-Hornung A, Zimmer H, Kaufmann P, Wang O, Watson S, Gross M (2015) Panoramic video from unstructured camera arrays. In: Computer graphics forum, Wiley online library, vol 34, pp 57–68Google Scholar
  30. 30.
    Rosten E, Porter R, Drummond T (2010) Faster and better: A machine learning approach to corner detection. IEEE Trans Pattern Anal Mach Intell 32(1):105–119CrossRefGoogle Scholar
  31. 31.
    Shi J et al. (1994) Good features to track. In: 1994 IEEE computer society conference on computer vision and pattern recognition, 1994. Proceedings CVPR’94. IEEE, pp 593-600Google Scholar
  32. 32.
    Sinha SN, Frahm JM, Pollefeys M, Genc Y (2011) Feature tracking and matching in video using programmable graphics hardware. Mach Vis Appl 22(1):207–217CrossRefGoogle Scholar
  33. 33.
    Song T, Jeon C, Ko H (2015) Image stitching using chaos-inspired dissimilarity measure. Electron Lett 51(3):232–234CrossRefGoogle Scholar
  34. 34.
    Szeliski R (1996) Video mosaics for virtual environments. IEEE Comput Graph Appl 16:22–30CrossRefGoogle Scholar
  35. 35.
    Szeliski R (2006) Image alignment and stitching: A tutorial. Foundations and Trends®, in Computer Graphics and Vision 2(1):1–104Google Scholar
  36. 36.
    Xu W, Mulligan J (2010) Performance evaluation of color correction approaches for automatic multi-view image and video stitching. In: 2010 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 263-270Google Scholar
  37. 37.
    Xu W, Mulligan J (2013) Panoramic video stitching from commodity hdtv cameras. Multimedia Syst 19(5):407–426CrossRefGoogle Scholar
  38. 38.
    Zaragoza J, Chin TJ, Brown MS, Suter D (2013) As-projective-as-possible image stitching with moving dlt. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2339–2346Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Electronics Engineering, Madras Institute of TechnologyAnna UniversityChrompet, ChennaiIndia

Personalised recommendations