Advertisement

Multi-modal Spectral Image Super-Resolution

  • Fayez Lahoud
  • Ruofan ZhouEmail author
  • Sabine Süsstrunk
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11133)

Abstract

Recent advances have shown the great power of deep convolutional neural networks (CNN) to learn the relationship between low and high-resolution image patches. However, these methods only take a single-scale image as input and require large amount of data to train without the risk of overfitting. In this paper, we tackle the problem of multi-modal spectral image super-resolution while constraining ourselves to a small dataset. We propose the use of different modalities to improve the performance of neural networks on the spectral super-resolution problem. First, we use multiple downscaled versions of the same image to infer a better high-resolution image for training, we refer to these inputs as a multi-scale modality. Furthermore, color images are usually taken at a higher resolution than spectral images, so we make use of color images as another modality to improve the super-resolution network. By combining both modalities, we build a pipeline that learns to super-resolve using multi-scale spectral inputs guided by a color image. Finally, we validate our method and show that it is economic in terms of parameters and computation time, while still producing state-of-the-art results (Code at https://github.com/IVRL/Multi-Modal-Spectral-Image-Super-Resolution).

Keywords

Spectral reconstruction Spectral image super-resolution Residual learning Image completion Multi-modality 

References

  1. 1.
    Achanta, R., Arvanitopoulos, N., Susstrunk, S.: Extreme image completion. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2017).  https://doi.org/10.1109/icassp.2017.7952373
  2. 2.
    Chang, C.I.: An information-theoretic approach to spectral variability, similarity, and discrimination for hyperspectral image analysis. IEEE Trans. Inf. Theor. 46(5), 1927–1932 (2000)CrossRefGoogle Scholar
  3. 3.
    Damodaran, B.B., Kellenberger, B., Flamary, R., Tuia, D., Courty, N.: Deepjdot: deep joint distribution optimal transport for unsupervised domain adaptation. arXiv preprint arXiv:1803.10081 (2018)
  4. 4.
    Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 184–199. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10593-2_13CrossRefGoogle Scholar
  5. 5.
    Dosovitskiy, A., et al.: Flownet: Learning optical flow with convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2758–2766 (2015)Google Scholar
  6. 6.
    Ganin, Y., Lempitsky, V.: Unsupervised domain adaptation by backpropagation. arXiv preprint arXiv:1409.7495 (2014)
  7. 7.
    Guo, T., Mousavi, H.S., Vu, T.H., Monga, V.: Deep wavelet prediction for image super-resolution. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2017)Google Scholar
  8. 8.
    Gupta, S., Hoffman, J., Malik, J.: Cross modal distillation for supervision transfer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2827–2836 (2016)Google Scholar
  9. 9.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  10. 10.
    Hu, Y., Zhang, D., Ye, J., Li, X., He, X.: Fast and accurate matrix completion via truncated nuclear norm regularization. IEEE Trans. Pattern Anal. Mach. Intell. 1 (2012)Google Scholar
  11. 11.
    Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)Google Scholar
  12. 12.
    Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1637–1645 (2016)Google Scholar
  13. 13.
    Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016),  https://doi.org/10.1109/cvpr.2016.182
  14. 14.
    Levin, A., Zomet, A., Weiss, Y.: Learning how to inpaint from global image statistics. In: Null, p. 305. IEEE (2003)Google Scholar
  15. 15.
    Li, W., Zhao, L., Lin, Z., Xu, D., Lu, D.: Non-local image inpainting using low-rank matrix completion. In: Computer Graphics Forum, vol. 34, pp. 111–122. Wiley Online Library (2015)Google Scholar
  16. 16.
    Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2017).  https://doi.org/10.1109/cvprw.2017.151
  17. 17.
    Liu, Q., Lai, Z., Zhou, Z., Kuang, F., Jin, Z.: A truncated nuclear norm regularization method based on weighted residual error for matrix completion. IEEE Trans. Image Process. 25(1), 316–330 (2016)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016).  https://doi.org/10.1109/cvpr.2016.207
  19. 19.
    Shoeiby, M., et al.: PIRM2018 challenge on spectral image super-resolution: methods and resultsGoogle Scholar
  20. 20.
    Shoeiby, M., Robles-Kelly, A., Wei, R., Timofte, R.: PIRM2018 challenge on spectral image super-resolution: dataset and studyGoogle Scholar
  21. 21.
    Sun, J., Yuan, L., Jia, J., Shum, H.Y.: Image completion with structure propagation. In: ACM Transactions on Graphics (ToG), vol. 24, pp. 861–868. ACM (2005)Google Scholar
  22. 22.
    Wei, Q., Dobigeon, N., Tourneret, J.Y.: Fast fusion of multi-band images based on solving a sylvester equation. IEEE Trans. Image Process. 24(11), 4109–4121 (2015)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Wycoff, E., Chan, T.H., Jia, K., Ma, W.K., Ma, Y.: A non-negative sparse promoting algorithm for high resolution hyperspectral imaging. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1409–1413. IEEE (2013)Google Scholar
  24. 24.
    Yokoya, N., Yairi, T., Iwasaki, A.: Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Trans. Geosci. Remote Sens. 50(2), 528–537 (2012)CrossRefGoogle Scholar
  25. 25.
    Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26(7), 3142–3155 (2017).  https://doi.org/10.1109/tip.2017.2662206MathSciNetCrossRefGoogle Scholar
  26. 26.
    Zhao, H., Gallo, O., Frosio, I., Kautz, J.: Loss functions for image restoration with neural networks. IEEE Trans. Comput. Imaging 3(1), 47–57 (2017)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.School of Computer and Communication SciencesÉcole Polytechnique Fédérale de LausanneLausanneSwitzerland

Personalised recommendations