Advertisement

Tensor Super-Resolution with Generative Adversarial Nets: A Large Image Generation Approach

  • Zihan DingEmail author
  • Xiao-Yang Liu
  • Miao Yin
  • Linghe Kong
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1072)

Abstract

Deep generative models have been successfully applied to many applications. However, existing methods experience limitations when generating large images (the literature usually generates small images, e.g., \(32 \times 32\) or \(128 \times 128\)). In this paper, we propose a novel scheme using tensor super-resolution with adversarial generative nets (TSRGAN), to generate large high-quality images by exploring tensor structures. Essentially, the super resolution process of TSRGAN is based on tensor representation. First, we impose tensor structures for concise image representation, which is superior in capturing the pixel proximity information and the spatial patterns of elementary objects in images, over the vectorization preprocess in existing works. Secondly, we propose TSRGAN that integrates deep convolutional generative adversarial networks and tensor super-resolution in a cascading manner, to generate high-quality images from random distributions. More specifically, we design a tensor super-resolution process that consists of tensor dictionary learning and tensor coefficients learning. Finally, on three datasets, the proposed TSRGAN generates images with more realistic textures, compared with state-of-the-art adversarial autoencoders and super-resolution methods. The size of the generated images is increased by over 8.5 times, namely \(374\times 374\) in PASCAL2.

Keywords

GAN Generative model Super-resolution Tensor sparse coding Tensor representation 

References

  1. 1.
    Brock, A., Donahue, J., Simonyan, K.: Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 (2018)
  2. 2.
    Denton, E.L., Chintala, S., Fergus, R., et al.: Deep generative image models using a laplacian pyramid of adversarial networks. In: Advances in Neural Information Processing Systems, pp. 1486–1494 (2015)Google Scholar
  3. 3.
    Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 184–199. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10593-2_13CrossRefGoogle Scholar
  4. 4.
    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)CrossRefGoogle Scholar
  5. 5.
    Everingham, M., Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010)CrossRefGoogle Scholar
  6. 6.
    Fei, J., Liu, X.-Y., Lu, H., Shen, R.: Efficient multi-dimensional tensor sparse coding using t-linear combinations. In: Association for the Advancement of Artificial Intelligence (2018)Google Scholar
  7. 7.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar
  8. 8.
    Gregor, K., Danihelka, I., Graves, A., Rezende, D.J., Wierstra, D.: Draw: a recurrent neural network for image generation. arXiv preprint arXiv:1502.04623 (2015)
  9. 9.
    Gu, S., Sang, N., Ma, F.: Fast image super resolution via local regression. In: Proceedings of the 21st International Conference on Pattern Recognition (ICPR 2012), pp. 3128–3131. IEEE (2012)Google Scholar
  10. 10.
    Hao, N., Kilmer, M.E., Braman, K., Hoover, R.C.: Facial recognition using tensor-tensor decompositions. SIAM J. Imaging Sci. 6(1), 437–463 (2013)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 (2017)
  12. 12.
    Kingma, D.P., Welling, M.: Auto-encoding variational bayes. In: International Conference on Learning Representations (2014)Google Scholar
  13. 13.
    Kolda, T.G., Bader, B.W.: Tensor decompositions and applications. SIAM Rev. 51(3), 455–500 (2009)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report. Citeseer (2009)Google Scholar
  15. 15.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  16. 16.
    Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR, vol. 2, p. 4 (2017)Google Scholar
  17. 17.
    Liu, X.-Y., Aeron, S., Aggarwal, V., Wang, X.: Low-tubal-rank tensor completion using alternating minimization. arXiv preprint arXiv:1610.01690 (2017)
  18. 18.
    Liu, X.-Y., Aeron, S., Aggarwal, V., Wang, X., Wu, M.-Y.: Adaptive sampling of RF fingerprints for fine-grained indoor localization. IEEE Trans. Mob. Comput. 15(10), 2411–2423 (2015)CrossRefGoogle Scholar
  19. 19.
    Liu, X.-Y., Wang, X.: Fourth-order tensors with multidimensional discrete transforms. arXiv preprint arXiv:1705.01576 (2017)
  20. 20.
    Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., Frey, B.: Adversarial autoencoders. In: International Conference on Learning Representations (2016)Google Scholar
  21. 21.
    Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 (2018)
  22. 22.
    Parikh, N., Boyd, S., et al.: Proximal algorithms. Found. Trends® Optim. 1(3), 127–239 (2014)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)
  24. 24.
    Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training GANs. In: Advances in Neural Information Processing Systems, pp. 2234–2242 (2016)Google Scholar
  25. 25.
    She, B., Wang, Y., Liang, J., Liu, Z., Song, C., Hu, G.: A data-driven avo inversion method via learned dictionaries and sparse representation. Geophysics 83(6), 1–91 (2018)CrossRefGoogle Scholar
  26. 26.
    Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)Google Scholar
  27. 27.
    Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)Google Scholar
  28. 28.
    Xu, B., Wang, N., Chen, T., Li, M.: Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853 (2015)
  29. 29.
    Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Zhang, H., et al.: StackGAN: text to photo-realistic image synthesis with stacked generative adversarial networks. In: IEEE International Conference on Computer Vision, pp. 5907–5915 (2017)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  • Zihan Ding
    • 1
    Email author
  • Xiao-Yang Liu
    • 2
  • Miao Yin
    • 3
  • Linghe Kong
    • 4
  1. 1.Imperial College LondonLondonUK
  2. 2.Columbia UniversityNew YorkUSA
  3. 3.University of Electronic Science and Technology of ChinaChengduChina
  4. 4.Shanghai Jiao Tong UniversityShanghaiChina

Personalised recommendations