DenseNetFuse: a study of deep unsupervised DenseNet to infrared and visual image fusion

Abstract

Visual images have visual features that are convenient for human observation, while infrared images have thermal features. Image fusion of visual image and the infrared image is the integration of most conductive intrinsic features by still preserving its original visual information concealed in. Image fusion has significant importance in different areas application in special environment (e.g. military, security and in surveillance) and human vision. In recent years, due to the-state-of-the-art discriminative performance, deep learning has been greatly applied in field of image fusion due to its excellent feature extraction ability. This paper employs an innovative approach for image fusion for infrared and visual images using a pre-trained dense block neural network. The model uses an isolated encoder-decoder bi-layered deep learning models, each for visual images and infrared images. An intermediary novel fusion protocols was designed using a hybrid of fusion and L1_norm strategy to infuse both visual and infrared image features. Finally, the resulting images of decoders are made subject to a weighting average to obtained fusion resulting image. Based on subjective evaluation, the proposed model, outperform all existing state of the art literature studies, via different objective evaluation indicators.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

References

  1. Babenko A, Slesarev A, Chigorin A, Lempitsky V (2014). Neural codes for image retrieval. In: European conference on computer vision (pp. 584–599). Springer, Cham

  2. Bavirisetti DP, Xiao G, Liu G (2017). Multi-sensor image fusion based on fourth order partial differential equations. In: 2017 20th International conference on information fusion (Fusion) (pp. 1–9). IEEE

  3. Broussard RP, Rogers SK, Oxley ME, Tarr GL (1999) Physiologically motivated image fusion for object detection using a pulse coupled neural network. IEEE Trans Neural Netw 10(3):554–563

    Article  Google Scholar 

  4. Deshmukh M, Bhosale U (2010) Image fusion and image quality assessment of fused images. Int J Image Proces (IJIP) 4(5):484

    Google Scholar 

  5. Dogra A, Goyal B, Agrawal S (2017) From multi-scale decomposition to non-multi-scale decomposition methods: a comprehensive survey of image fusion techniques and its applications. IEEE Access 5:16040–16067

    Article  Google Scholar 

  6. Eskicioglu AM, Fisher PS (1995) Image quality measures and their performance. IEEE Trans Commun 43(12):2959–2965

    Article  Google Scholar 

  7. Guo Y, Li C, Liu Q (2019) R ~ 2n: a novel deep learning architecture for rain removal from single image. Comput Mater Continua 58(3):829–843

    Article  Google Scholar 

  8. James AP, Dasarathy BV (2014) Medical image fusion: a survey of the state of the art. Inf Fusion 19:4–19

    Article  Google Scholar 

  9. Lewis JJ, O’Callaghan RJ, Nikolov SG, Bull DR, Canagarajah N (2007) Pixel-and region-based image fusion with complex wavelets. Inf Fusion 8(2):119–130

    Article  Google Scholar 

  10. Li H, Wu XJ (2018) DenseFuse: a fusion approach to infrared and visible images. IEEE Trans Image Process 28(5):2614–2623

    MathSciNet  Article  Google Scholar 

  11. Li S, Kang X, Hu J (2013) Image fusion with guided filtering. IEEE Trans Image Process 22(7):2864–2875

    Article  Google Scholar 

  12. Li S, Kang X, Fang L, Hu J, Yin H (2017) Pixel-level image fusion: a survey of the state of the art. Inf Fusion 33:100–112

    Article  Google Scholar 

  13. Li Y, Sun Y, Huang X, Qi G, Zheng M, Zhu Z (2018) An image fusion method based on sparse representation and sum modified-Laplacian in NSCT domain. Entropy 20(7):522

    Article  Google Scholar 

  14. Liu Y, Chen X, Peng H, Wang Z (2017) Multi-focus image fusion with a deep convolutional neural network. Inf Fusion 36:191–207

    Article  Google Scholar 

  15. Liu X, Mei W, Du H (2018) Multi-modality medical image fusion based on image decomposition framework and nonsubsampled shearlet transform. Biomed Signal Process Control 40:343–350

    Article  Google Scholar 

  16. Liu Z, Xiang B, Song Y, Lu H, Liu Q (2019) An improved unsupervised image segmentation method based on multi-objective particle swarm optimization clustering algorithm. Comput Mater Continua 58(2):451–461

    Article  Google Scholar 

  17. Lu GX, Zhou DW, Wang JL (2002) Geological information extracting from remote sensing image in complex area: based on wavelet analysis for automatic image segmentation. Earth Science-Journal of China University of Geosciences 27(1):50–54

    Google Scholar 

  18. Ma J, Chen C, Li C, Huang J (2016a) Infrared and visible image fusion via gradient transfer and total variation minimization. Inf Fusion 31:100–109

    Article  Google Scholar 

  19. Ma Y, Chen J, Chen C, Fan F, Ma J (2016b) Infrared and visible image fusion using total variation model. Neurocomputing 202:12–19

    Article  Google Scholar 

  20. Ma J, Zhou Z, Wang B, Zong H (2017) Infrared and visible image fusion based on visual saliency map and weighted least square optimization. Infrared Phys Technol 82:8–17

    Article  Google Scholar 

  21. Ma J, Yu W, Liang P, Li C, Jiang J (2019) FusionGAN: a generative adversarial network for infrared and visible image fusion. Information Fusion 48:11–26

    Article  Google Scholar 

  22. Mohanapriya N, Kalaavathi B (2019) Adaptive image enhancement using hybrid particle swarm optimization and watershed segmentation. Intell Autom soft comput 25(4):663–672

    Google Scholar 

  23. Nejad MB, Shiri ME (2019) A new enhanced learning approach to automatic image classification based on salp swarm algorithm. Int J Comput Syst Sci Eng 34(2):91–100

    Google Scholar 

  24. Pan L, Qin J, Chen H, Xiang X, Li C, Chen R (2019) Image augmentation-based food recognition with convolutional neural networks. Comput Mater Continua 59(1):297–313

    Article  Google Scholar 

  25. Qin J, Huang Y, Wen W (2020) Multi-scale feature fusion residual network for single image super-resolution. Neurocomputing 379:334–342

    Article  Google Scholar 

  26. Ram Prabhakar K, Sai Srikar V, Venkatesh Babu R (2017) DeepFuse: a deep unsupervised approach for exposure fusion with extreme exposure image pairs. In: Proceedings of the IEEE international conference on computer vision (pp. 4714-4722)

  27. Shachor Y, Greenspa H, Goldberger J (2019) A mixture of views network with applications to the classification of breast microcalcifications. In: 2019 IEEE 16th International symposium on biomedical imaging (ISBI 2019) (pp. 1065–1069). IEEE

  28. Wang J, Peng J, Feng X, He G, Fan J (2014) Fusion method for infrared and visible images by using non-negative sparse representation. Infrared Phys Technol 67:477–489

    Article  Google Scholar 

  29. Wang Q, Lai J, Yang Z, Xu K, Kan P, Liu W, Lei L (2019) Improving cross-dimensional weighting pooling with multi-scale feature fusion for image retrieval. Neurocomputing 363:17–26

    Article  Google Scholar 

  30. Wu L, Liu Q, Lou P (2019) Image classification using optimized mkl for sspm. Intell Autom Soft Comput 25(2):249–257

    Google Scholar 

  31. Yin M, Liu X, Liu Y, Chen X (2018) Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain. IEEE Trans Instrum Meas 68(1):49–64

    Article  Google Scholar 

  32. Zhang X, Ma Y, Fan F, Zhang Y, Huang J (2017) Infrared and visible image fusion via saliency analysis and local edge-preserving multi-scale decomposition. JOSA A 34(8):1400–1410

    Article  Google Scholar 

  33. Zhao J, Cui G, Gong X, Zang Y, Tao S, Wang D (2017) Fusion of visible and infrared images using global entropy and gradient constrained regularization. Infrared Phys Technol 81:201–209

    Article  Google Scholar 

  34. Zhu Z, Zheng M, Qi G, Wang D, Xiang Y (2019) A phase congruency and local Laplacian energy based multi-modality medical image fusion method in NSCT domain. IEEE Access 7:20811–20824

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Yue Pan.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest with any person(s) or organization(s).

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Pan, Y., Pi, D., Khan, I. et al. DenseNetFuse: a study of deep unsupervised DenseNet to infrared and visual image fusion. J Ambient Intell Human Comput (2021). https://doi.org/10.1007/s12652-020-02820-3

Download citation

Keywords

  • Infrared
  • Image
  • Visual image
  • Image fusion
  • Deep learning