Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

An IR and visible image sequence automatic registration method based on optical flow

  • 536 Accesses

  • 9 Citations

Abstract

IR–visible camera registration is required for multi-sensor fusion and cooperative processing. Image sequences can provide motion information, which is useful for sequence registration. The existing methods mainly focus on registration using moving objects which are observed by both cameras. However, accurate motion feature extraction for a whole moving object is difficult, because of the complex environment and different imaging mechanism of two sensors. To overcome this problem, we use motion features associated with single pixels in the two image sequences to carry out automatic registration. A normalized optical flow time sequence for each image pixel is constructed. The matching of pixels between the IR image and the visible light image is carried out using a fast similarity measurement and a three stage correspondence selection method. Finally cascaded random sample consensus is adopted to remove outlying matches, and least-square method and Levenberg–Marquardt method are used to estimate the transformation from the IR image to the visible image. The effectiveness of our method is demonstrated using several real datasets and simulated datasets.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

References

  1. 1.

    Lu, X., Zhang, S., Su, H., Chen, Y.: Mutual information-based multimodal image registration using a novel joint histogram estimation. Comput. Med. Imaging Graph. 32(3), 202–209 (2008)

  2. 2.

    Luan, H.X., Qi, F.H., Xue, Z.H.: Multimodality image registration by maximization of quantitative–qualitative measure of mutual information. Pattern Recognit. 41(1), 285–298 (2008)

  3. 3.

    Istenic, R., Heric, D., Ribaric, S.: Thermal and visual image registration in hough parameter space. In: Proceedings of the 14th International Workshop on Systems, Signals and Image Processing and 6th EURASIP Conference focused on Speech and Image Processing, Multimedia Communications and Services, pp. 106–109 (2007)

  4. 4.

    Coiras, E., Santamaria, J., Miravet, C.: Segment-based registration technique for visual-infrared images. Opt. Eng. 39(1), 282–289 (2000)

  5. 5.

    Lee, H.J., Kim, S.Y., Lee, D.: Robust CCD and IR image registration using gradient-based statistical information. IEEE Signal Process. Lett. 17(4), 347–350 (2010)

  6. 6.

    Kim, Y.S., Lee, J.H., Ra, J.B.: Multi-sensor image registration based on intensity and edge orientation information. Pattern Recognit. 41(11), 3356–3365 (2008)

  7. 7.

    Hrkac, T., Kalafatic, Z., Krapac, J.: Infrared-visual image registration based on corners and hausdorff distance. Lect. Notes Comput. Sci. 4522, 383–392 (2007)

  8. 8.

    Irani, M., Anandan, P.: Robust multi-sensor image alignment. In: Proceedings of International Conference on Computer Vision, pp. 959–966 (1998)

  9. 9.

    Lee, L., Romano, R., Stein, G.P.: Monitoring activities from multiple video streams: establishing a common coordinate frame. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 758–767 (2000)

  10. 10.

    Caspi, Y., Simakov, D., Irani, M.: Feature-based sequence-to-sequence matching. Int. J. Comput. Vis. 68(1), 53–64 (2006)

  11. 11.

    Ju, H., Bhanu, B.: Fusion of color and infrared video for moving human detection. Pattern Recognit. 40(6), 1771–1784 (2007)

  12. 12.

    Verstockt, S., Poppe, C., Potter, D.P., Hollemeersch, C.: Silhouette coverage analysis for multi-modal video surveillance. PIERS Online 7(5), 401–405 (2011)

  13. 13.

    Torabi, A., Masse, G., Bilodeau, G-A.: Feedback scheme for thermal-visible video registration, sensor fusion, and people tracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition-Workshop, pp. 15–22 (2010)

  14. 14.

    Bilodeau, A.G., Torabi, A., Morin, F.: Visible and infrared image registration using trajectories and composite foreground images. Image Vis. Comput. 29(1), 41–50 (2011)

  15. 15.

    Levenberg, K.: A method for the solution of certain nonlinear problems in least squares. Q. Appl. Math. 2(2), 164–168 (1994)

  16. 16.

    Marquardt, D.W.: An algorithm for least-squares estimation of nonlinear parameters. SIAM J. Appl. Math. 11(2), 431–441 (1963)

  17. 17.

    Bouguet, J.Y.: Pyramidal implementation of the Lucas Kanade feature tracker: description of the algorithm. Technical report, Intel Corporation, Microprocessor Research Labs 2000, OpenCV documentation (2001)

  18. 18.

    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)

  19. 19.

    Davis, J., Sharma, V.: Background-subtraction using contour-based fusion of thermal and visible imagery. Comput. Vis. Image Underst. 106(23), 162–182 (2007)

Download references

Acknowledgments

This work is supported by the National Natural Science Foundation of China (Nos. 60872145, 60903126), China Postdoctoral Special Science Foundation (No. 201003685), and China  Post doctoral Science Foundation (No. 20090451397).

Author information

Correspondence to Xiuwei Zhang.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Zhang, Y., Zhang, X., Maybank, S.J. et al. An IR and visible image sequence automatic registration method based on optical flow. Machine Vision and Applications 24, 947–958 (2013). https://doi.org/10.1007/s00138-012-0465-x

Download citation

Keywords

  • Image sequences registration
  • Visible image sequence
  • IR image sequence
  • Optical flow