Skip to main content
SpringerLink
Log in

Multimodal Sensor Fusion in Single Thermal Image Super-Resolution

  • Conference paper
  • First Online:
Computer Vision – ACCV 2018 Workshops (ACCV 2018)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 11367))

Included in the following conference series:

Abstract

With the fast growth in the visual surveillance and security sectors, thermal infrared images have become increasingly necessary in a large variety of industrial applications. This is true even though IR sensors are still more expensive than their RGB counterpart having the same resolution. In this paper, we propose a deep learning solution to enhance the thermal image resolution. The following results are given: (I) Introduction of a multimodal, visual-thermal fusion model that addresses thermal image super-resolution, via integrating high-frequency information from the visual image. (II) Investigation of different network architecture schemes in the literature, their up-sampling methods, learning procedures, and their optimization functions by showing their beneficial contribution to the super-resolution problem. (III) A benchmark ULB17-VT dataset that contains thermal images and their visual images counterpart is presented. (IV) Presentation of a qualitative evaluation of a large test set with 58 samples and 22 raters which shows that our proposed model performs better against state-of-the-arts.

This work was supported by the European Regional Development Fund (ERDF) and the Brussels-Capital Region within the framework of the Operational Programme 2014–2020 through the ERDF-2020 project F11-08 ICITY-RDI.BRU. The Titan X Pascal used for this research was donated by the NVIDIA Corporation. We are grateful to Thermal Focus BVBA for their help and support.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Université Libre de Bruxelles.

References

  1. Chen, X., Zhai, G., Wang, J., Hu, C., Chen, Y.: Color guided thermal image super resolution. In: Visual Communications and Image Processing (VCIP), pp. 1–4. IEEE (2016)

    Google Scholar 

  2. Chen, Z., Tong, Y.: Face super-resolution through Wasserstein GANs. arXiv preprint arXiv:1705.02438 (2017)

  3. Cho, H., Seo, Y.W., Kumar, B.V., Rajkumar, R.R.: A multi-sensor fusion system for moving object detection and tracking in urban driving environments. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 1836–1843. IEEE (2014)

    Google Scholar 

  4. Choi, Y., Kim, N., Hwang, S., Kweon, I.S.: Thermal image enhancement using convolutional neural network. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 223–230. IEEE (2016)

    Google Scholar 

  5. Clevert, D.A., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by exponential linear units (ELUs). arXiv preprint arXiv:1511.07289 (2015)

  6. Dong, C., Change Loy, C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. ArXiv e-prints, December 2015

    Google Scholar 

  7. Goodfellow, I.: NIPS 2016 tutorial: generative adversarial networks. arXiv preprint arXiv:1701.00160 (2016)

  8. Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)

    Google Scholar 

  9. Gross, S., Wilber, M.: Training and investigating residual nets (2016). https://torch.ch/blog/2016/02/04/resnets.html

  10. Huang, Y., Qin, M.: Densely connected high order residual network for single frame image super resolution. arXiv preprint arXiv:1804.05902 (2018)

  11. Hwang, S., Park, J., Kim, N., Choi, Y., Kweon, I.S.: Multispectral pedestrian detection: benchmark dataset and baselines. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

    Google Scholar 

  12. Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)

  13. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. arXiv preprint (2017)

    Google Scholar 

  14. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43

    Chapter  Google Scholar 

  15. Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)

    Google Scholar 

  16. Kim, J., Kwon Lee, J., Mu Lee, K.: Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1637–1645 (2016)

    Google Scholar 

  17. Kiran, Y., Shrinidhi, V., Hans, W.J., Venkateswaran, N.: A single-image super-resolution algorithm for infrared thermal images. Int. J. Comput. Sci. Netw. Secur. 17(10), 256–261 (2017)

    Google Scholar 

  18. Lai, W.S., Huang, J.B., Ahuja, N., Yang, M.H.: Deep Laplacian pyramid networks for fast and accurate super-resolution. In: Proceedings IEEE Conference Computer Vision and Pattern Recognition, pp. 624–632 (2017)

    Google Scholar 

  19. Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. ArXiv e-prints, September 2016

    Google Scholar 

  20. Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. arXiv preprint (2016)

    Google Scholar 

  21. Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, vol. 1, p. 4 (2017)

    Google Scholar 

  22. Panagiotopoulou, A., Anastassopoulos, V.: Super-resolution reconstruction of thermal infrared images. In: Proceedings of the 4th WSEAS International Conference on Remote Sensing (2008)

    Google Scholar 

  23. Qu, Y., Zhang, G., Zou, Z., Liu, Z., Mao, J.: Active multimodal sensor system for target recognition and tracking. Sensors 17(7), 1518 (2017)

    Article  Google Scholar 

  24. Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)

    Google Scholar 

  25. Wu, B., Duan, H., Liu, Z., Sun, G.: SRPGAN: perceptual generative adversarial network for single image super resolution. arXiv preprint arXiv:1712.05927 (2017)

  26. Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., Li, H.: High-resolution image inpainting using multi-scale neural patch synthesis. ArXiv e-prints, November 2016

    Google Scholar 

  27. Zhang, H., et al.: StackGAN: text to photo-realistic image synthesis with stacked generative adversarial networks. In: IEEE International Conference on Computer Vision (ICCV), pp. 5907–5915 (2017)

    Google Scholar 

  28. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint arXiv:1703.10593 (2017)

Download references

Author information

Authors and Affiliations

  1. Department LISA - Laboratory of Image Synthesis and Analysis, Université Libre de Bruxelles, CPI 165/57, Avenue Franklin Roosevelt 50, 1050, Brussels, Belgium

    Feras Almasri & Olivier Debeir

Authors
  1. Feras Almasri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Olivier Debeir
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Olivier Debeir .

Editor information

Editors and Affiliations

  1. School of Computer Science, University of Adelaide, Adelaide, Australia

    Gustavo Carneiro

  2. Data61, Commonwealth Scientific and Industrial Research Organization, Canberra, Australia

    Shaodi You

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Almasri, F., Debeir, O. (2019). Multimodal Sensor Fusion in Single Thermal Image Super-Resolution. In: Carneiro, G., You, S. (eds) Computer Vision – ACCV 2018 Workshops. ACCV 2018. Lecture Notes in Computer Science(), vol 11367. Springer, Cham. https://doi.org/10.1007/978-3-030-21074-8_34

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-21074-8_34

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-21073-1

  • Online ISBN: 978-3-030-21074-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation