Advertisement

Multidimensional Systems and Signal Processing

, Volume 30, Issue 1, pp 119–143 | Cite as

A visible-light and infrared video database for performance evaluation of video/image fusion methods

  • Andreas Ellmauthaler
  • Carla L. PagliariEmail author
  • Eduardo A. B. da Silva
  • Jonathan N. Gois
  • Sergio R. Neves
Article
  • 212 Downloads

Abstract

In general, the fusion of visible-light and infrared images produces a composite representation where both data are pictured in a single image. The successful development of image/video fusion algorithms relies on realistic infrared/visible-light datasets. To the best of our knowledge, there is a particular shortage of databases with registered and synchronized videos from the infrared and visible-light spectra suitable for image/video fusion research. To address this need we recorded an image/video fusion database using infrared and visible-light cameras under varying illumination conditions. Moreover, different scenarios have been defined to better challenge the fusion methods, with various contexts and contents providing a wide variety of meaningful data for fusion purposes, including non-planar scenes, where objects appear on different depth planes. However, there are several difficulties in creating datasets for research in infrared/visible-light image fusion. Camera calibration, registration, and synchronization can be listed as important steps of this task. In particular, image registration between imagery from sensors of different spectral bands imposes additional difficulties, as it is very challenging to solve the correspondence problem between such images. Motivated by these challenges, this work introduces a novel spatiotemporal video registration method capable of generating registered and temporally aligned infrared/visible-light video sequences. The proposed workflow improves the registration accuracy when compared to the state-of-the art. By applying the proposed methodology to the recorded database we have generated the visible-light and infrared video database for image fusion, a publicly available database to be used by the research community to test and benchmark fusion schemes.

Keywords

Infrared/visible image/video database Image registration Image fusion Camera calibration 

Notes

Acknowledgements

This work was partially supported by CAPES/Pro-Defesa under Grant No. 23038.009094/2013-83.

References

  1. Aslantas, R. K. V., Bendes, E., & Toprak, A. (2014). New optimised region-based multi-scale image fusion method for thermal and visible images. IET Image Processing, 8(5), 289–299.CrossRefGoogle Scholar
  2. Bilodeau, G.-A., et al. (2014). Thermal-visible registration of human silhouettes: A similarity measure performance evaluation. Infrared Physics and Technology, 64, 79–86.CrossRefGoogle Scholar
  3. Bouguet, J.-Y. (2012). Camera calibration toolbox for Matlab. http://www.vision.caltech.edu/bouguetj/calib_doc/. Accessed December 12, 2012.
  4. Bradski, G., Kaehler, A., & Pisarevsky, V. (2005). Learning-based computer vision with intel’s open source computer vision library. Intel Technology Journal, 9, 119–130.Google Scholar
  5. Campo, F. B., Ruiz, F. L., & Sappa, A. (2012). Multimodal stereo vision system: 3d data extraction and algorithm evaluation. IEEE Journal of Selected Topics in Signal Processing, 6, 437–446.CrossRefGoogle Scholar
  6. Caspi, Y., & Irani, M. (2000). A step towards sequence-to-sequence alignment. In Proceedings of the 2000 IEEE conference on computer vision and pattern recognition (Vol. 2, pp. 682–689).Google Scholar
  7. Caspi, Y., & Irani, M. (2002). Spatio-temporal alignment of sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(11), 1409–1424.CrossRefGoogle Scholar
  8. Datta, A., Kim, J.-S., & Kanade, T. (2009). Accurate camera calibration using iterative refinement of control points. In Proceedings of the 2009 IEEE international conference on computer vision workshops (ICCV workshops) (pp. 1201–1208).Google Scholar
  9. Davis, J., & Sharma, V. (2007). Background-subtraction using contour-based fusion of thermal and visible imagery. Computer Vision and Image Understanding, 106(2–3), 162–182.CrossRefGoogle Scholar
  10. Debevec, P. E., Taylor, C. J., & Malik, J. (1996). Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In Proceedings of the 23rd annual conference on computer graphics and interactive techniques (pp. 11–20).Google Scholar
  11. Ellmauthaler, A., da Silva, E. A. B., Pagliari, C. L., & Gois, J. N. (2013a). A robust temporal alignment technique for infrared and visible-light video sequences. In XXXI Simp. Brasileiro de Telecomunicações.Google Scholar
  12. Ellmauthaler, A., da Silva, E., Pagliari, C., Gois, J., & Neves, S. R. (2013b). A novel iterative calibration approach for thermal infrared cameras. In Proceedings of the 2013 IEEE international conference on image processing.Google Scholar
  13. Ellmauthaler, A., da Silva, E., Pagliari, C., & Neves, S. (2012). Infrared-visible image fusion using the undecimated wavelet transform with spectral factorization and target extraction. In Proceedings of the 2012 IEEE international conference on image processing (pp. 2661–2664).Google Scholar
  14. Ellmauthaler, A., Pagliari, C., & da Silva, E. (2013). Multiscale image fusion using the undecimated wavelet transform with spectral factorization and nonorthogonal filter banks. IEEE Transactions on Image Processing, 22(3), 1005–1017.MathSciNetCrossRefzbMATHGoogle Scholar
  15. Faugeras, O. (1993). Three dimensional computer vision: A geometric viewpoint. Cambridge: The MIT Press.Google Scholar
  16. Faugeras, O., & Lustman, F. (1988). Motion and structure from motion in a piecewise planar environment. International Journal of Pattern Recognition and Artificial Intelligence, 2(3), 485–508.CrossRefGoogle Scholar
  17. Fitzgibbon, A., Pilu, M., & Fisher, R. (1999). Direct least-squares fitting of ellipses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21, 476–480.CrossRefGoogle Scholar
  18. Gschwandtner, M., Kwitt, R., Uhl, A., & Pree, W. (2011). Infrared camera calibration for dense depth map construction. In Proceedings of the 2011 intelligent vehicles symposium (pp. 857–862).Google Scholar
  19. Han, J., & Bhanu, B. (2007). Fusion of color and infrared video for moving human detection. Pattern Recognition, 40(6), 1771–1784.CrossRefzbMATHGoogle Scholar
  20. Hartley, R., & Zisserman, A. (2004). Multiple view geometry in computer vision (2nd ed.). Cambridge: Cambridge University Press.CrossRefzbMATHGoogle Scholar
  21. Heikkilä, J. (2000). Geometric camera calibration using circular control points. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22, 1066–1077.CrossRefGoogle Scholar
  22. Heikkilä, J., & Silvén, O. (1997). A four-step camera calibration procedure with implicit image correction. In Proceedings of the 1997 IEEE computer society conference on computer vision and pattern recognition (pp. 1106–1112).Google Scholar
  23. http://adas.cvc.uab.es/projects/simeve/. Accessed January 11, 2015.
  24. http://vcipl-okstate.org/pbvs/bench/. Accessed January 11, 2015.
  25. http://www.ino.ca/en/video-analytics-dataset/. Accessed February 08, 2017.
  26. http://www.polymtl.ca/litiv/vid/. Accessed January 11, 2015.
  27. http://www.vap.aau.dk/stereo-thermal-dataset/. Accessed February 08, 2017.
  28. Int. society on information fusion. http://isif.org. Accessed April 07, 2015.
  29. Lagüela, S., González-Jorge, H., Armesto, J., & Arias, P. (2011). Calibration and verification of thermographic cameras for geometric measurements. Infrared Physics and Technology, 54, 92–99.CrossRefGoogle Scholar
  30. Lewis, J. J., et al. (2006). The Eden Project multi-sensor data set. In Technical report TR-UoB-WS-Eden-Project-Data-Set (University of Bristol and Waterfall Solutions Ltd, London).Google Scholar
  31. Li, S. L. S., Chu, R., & Zhang, L. (2007). A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(4), 627–639.CrossRefGoogle Scholar
  32. Loop, C., & Zhang, Z. (1999). Computing rectifying homographies for stereo vision. In IEEE computer society conference on computer vision and pattern recognition (Vol. 1).Google Scholar
  33. Morris, N. J. W., Avidan, S., Matusik, W., & Pfister, H. (2007). Statistics of infrared images. In Proceedings of the 2007 IEEE computer society conference on computer vision and pattern recognition (pp. 1–7).Google Scholar
  34. Morris, N. J. W., Avidan, S., & Pfister, H. Statistics of infrared images. http://www.dgp.toronto.edu/~nmorris/IR/. Accessed January 11, 2015.
  35. Padua, F., Carceroni, R., Santos, G., & Kutulakos, K. (2010). Linear sequence-to-sequence alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(2), 304–320.CrossRefGoogle Scholar
  36. Prakash, C., & Karam, L. (2012). Camera calibration using adaptive segmentation and ellipse fitting for localizing control points. In Proceedings of the 2012 IEEE international conference on image processing (pp. 341–344).Google Scholar
  37. Prakash, S., Lee, P., Caelli, T., & Raupach, T. (2006). Robust thermal camera calibration and 3d mapping of object surface temperatures. In Proceedings of the XXVIII SPIE conference on thermosense (Vol. 6205).Google Scholar
  38. Ravichandran, A., & Vidal, R. (2011). Video registration using dynamic textures. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1), 158–171.CrossRefGoogle Scholar
  39. Saponaro, P., Sorensen, S., Rhein, S., & Kambhamettu, C. (2015). Improving calibration of thermal stereo cameras using heated calibration board. In Proceedings of the 2015 IEEE international conference on image processing.Google Scholar
  40. St-Charles, P. -L., Bilodeau, G.-A., & Bergevin, R. (2015). Online multimodal video registration based on shape matching. In 2015 IEEE conference on computer vision and pattern recognition workshops (CVPRW) (pp. 26–34).Google Scholar
  41. Torabi, A., & Bilodeau, G.-A. (2013). Local self-similarity-based registration of human ROIs in pairs of stereo thermal-visible videos. Pattern Recognition, 46(2), 578–589.CrossRefGoogle Scholar
  42. Torabi, A., Massé, G., & Bilodeau, G.-A. (2012). An iterative integrated framework for thermal-visible image registration, sensor fusion, and people tracking for video surveillance applications. Computer Vision and Image Understanding, 116(2), 210–221.CrossRefGoogle Scholar
  43. Ukrainitz, Y., & Irani, M. (2006). Aligning sequences and actions by maximizing space-time correlations. In Proceedings of the 2006 European conference on computer vision (Vol. 3953, pp. 538–550).Google Scholar
  44. Vidas, S., et al. (2012). A mask-based approach for the geometric calibration of thermal-infrared cameras. IEEE Transactions on Instrumentation and Measurement, 61(6), 1625–1635.CrossRefGoogle Scholar
  45. Visible-light and infrared video database for image fusion. http://www.smt.ufrj.br/~fusion/. Accessed November 11, 2017.
  46. Yang, R., Yang, W., Chen, Y., & Wu, X. (2011). Geometric calibration of IR camera using trinocular vision. Journal of Lightwave Technology, 29, 3797–3803.CrossRefGoogle Scholar
  47. Zhang, M. M. E. A. (2015). Vais: A dataset for recognizing maritime imagery in the visible and infrared spectrums. In The IEEE conference on computer vision and pattern recognition (CVPR) workshops.Google Scholar
  48. Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11), 1330–1334.CrossRefGoogle Scholar
  49. Zhang, Z., & Blum, R. S. (1999). A categorization of multiscale-decomposition-based image fusion schemes with a performance study for a digital camera application. Proceedings of the IEEE, 87(8), 1315–1326.CrossRefGoogle Scholar
  50. Zhang, Z., & Hanson, A. (1996). 3D reconstruction based on homography mapping. In ARPA image understanding workshop (pp. 249–399).Google Scholar
  51. Zhao, J., & Cheung, S.-C. (2014). Human segmentation by geometrically fusing visible-light and thermal imageries. Multimedia Tools and Applications, 73(1), 61–89.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2017

Authors and Affiliations

  • Andreas Ellmauthaler
    • 1
  • Carla L. Pagliari
    • 2
    Email author
  • Eduardo A. B. da Silva
    • 3
  • Jonathan N. Gois
    • 4
  • Sergio R. Neves
    • 5
  1. 1.Halliburton Technology Center, Rua Paulo Emidio Barbosa, 485Ilha da Cidade UniversitariaRio de JaneiroBrazil
  2. 2.Instituto Militar de EngenhariaRio de JaneiroBrazil
  3. 3.Universidade Federal do Rio de JaneiroRio de JaneiroBrazil
  4. 4.Centro Federal de Educação Tecnológica Celso Suckow da FonsecaRio de JaneiroBrazil
  5. 5.Instituto de Pesquisas da MarinhaRio de JaneiroBrazil

Personalised recommendations