Perceptual image quality using dual generative adversarial network

  • Masoumeh Zareapoor
  • Huiyu Zhou
  • Jie YangEmail author
Deep Learning Approaches for RealTime Image Super Resolution (DLRSR)


Generative adversarial networks have received a remarkable success in many computer vision applications for their ability to learn from complex data distribution. In particular, they are capable to generate realistic images from latent space with a simple and intuitive structure. The main focus of existing models has been improving the performance; however, there is a little attention to make a robust model. In this paper, we investigate solutions to the super-resolution problems—in particular perceptual quality—by proposing a robust GAN. The proposed model unlike the standard GAN employs two generators and two discriminators in which, a discriminator determines that the samples are from real data or generated one, while another discriminator acts as classifier to return the wrong samples to its corresponding generators. Generators learn a mixture of many distributions from prior to the complex distribution. This new methodology is trained with the feature matching loss and allows us to return the wrong samples to the corresponding generators, in order to regenerate the real-look samples. Experimental results in various datasets show the superiority of the proposed model compared to the state of the art methods.


Image processing Perceptual quality Data distribution Generative adversarial network Classification 



This research is partly supported by NSFC, China (U1803261, 61876107, 61572315); 973 Plan, China (2015CB856004). H. Zhou was supported by UK EPSRC under Grant EP/N011074/1, Royal Society-Newton Advanced Fellowship under Grant NA160342 and European Union’s Horizon 2020 research and innovation program under the Marie-Sklodowska-Curie grant agreement No 720325.

Compliance with ethical standards

Conflict of interest

We have no conflict of interest to declare.


  1. 1.
    Zareapoor M, Zhang J, Yang J (2019) Towards realistic image via function learning. Multimed Tools Appl. Google Scholar
  2. 2.
    Zareapoor M, Shamsolmoali P, Yang J (2019) Learning depth super-resolution by using multi-scale convolutional neural network. J Intell Fuzzy Syst 36(2):1773–1783CrossRefGoogle Scholar
  3. 3.
    Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Proceeding of advances in neural information processing systems, pp 2672–2680Google Scholar
  4. 4.
    Ledig C, Theis L, Huszar F, Caballero J, Aitken AP, Tejani A, Totz J, Wang Z, Shi W (2016) Photo-realistic single image super-resolution using a generative adversarial network. CoRR, vol. abs/1609.04802, 2016. [Online].
  5. 5.
    Reed S, Akata Z, Yan X, Logeswaran L, Schiele B, Lee H (2016) Generative adversarial text-to-image synthesis. In: Proceedings of ICML, pp 1060–1069Google Scholar
  6. 6.
    Zhang H, Xu T, Li H, Zhang S, Huang X, Wang X, Metaxas DN (2017) StackGAN: text to photo-realistic image synthesis with stacked generative adversarial networks. In: Proceeding of the ICCV, pp 5907–5915Google Scholar
  7. 7.
    Durugkar IP, Gemp I, Mahadevan S (2016) Generative multi-adversarial networks. ICLR. CoRR, abs/1611.01673Google Scholar
  8. 8.
    Zareapoor M, Celebi ME, Yang J (2019) Diverse adversarial network for image super-resolution. Signal Process Image Commun 74:191–200. CrossRefGoogle Scholar
  9. 9.
    Ding L, Zhang H, Xiao J et al (2018) An improved image mixed noise removal algorithm based on super-resolution algorithm and CNN. Neural Comput Appl. Google Scholar
  10. 10.
    Wang Z, Liu D, Yang J, Han W, Huang T (2015) Deep networks for image super-resolution with sparse prior. In: ICCVGoogle Scholar
  11. 11.
    Dong C, Loy CC, He K, Tang X (2016) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307CrossRefGoogle Scholar
  12. 12.
    Zareapoor M, Jain DK, Yang J (2018) Local spatial information for image super-resolution. Cogn Syst Res 52:49–57CrossRefGoogle Scholar
  13. 13.
    Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with deep convolutional generative adversarial networks. In: Proceeding of international conference on learning representations arXiv:1511.06434
  14. 14.
    Salimans T, Goodfellow I, Zaremba W, Cheung V, Radford A, Chen X (2016) Improved techniques for training GANs. In: Proceeding of the NIPS, pp 2234–2242Google Scholar
  15. 15.
    Odena A, Olah C, Shlens J (2017) Conditional image synthesis with auxiliary classifier GANs. In: International conference on machine learning (PMLR), pp 2642–2651Google Scholar
  16. 16.
    Chen X, Duan Y, Houthooft R, Schulman J, Sutskever I, Abbeel P (2016) Infogan: interpretable representation learning by information maximizing generative adversarial nets. In: Advances in neural information processing systemsGoogle Scholar
  17. 17.
    Arjovsky M, Chintala S, Bottou L (2017) Wasserstein generative adversarial networks. In: Proceedings of the 34th international conference on machine learning, pp 214–223Google Scholar
  18. 18.
    Nguyen TD, Le T, Vu H, Phung D (2017) Dual discriminator generative adversarial nets. In: Advances in neural information processing systems 29 (NIPS) (accepted)Google Scholar
  19. 19.
    Arora S, Ge R, Liang Y, Ma T, Zhang Y (2017) Generalization and equilibrium in generative adversarial nets (gans). arXiv preprint arXiv:1703.00573
  20. 20.
    Tolstikhin I, Gelly S, Bousquet O, Simon-Gabriel C-J, Sch¨olkopf B (2017) Adagan: boosting generative models. arXiv preprint arXiv:1701.02386
  21. 21.
    Ghosh A, Kulharia V, Namboodiri VP, Torr PHS, Dokania PK (2017) Multi-agent diverse generative adversarial networks. In: Proceeding of the CVPR, pp 8513–8521Google Scholar
  22. 22.
    Wang X, Gupta A (2016) Generative image modeling using style and structure adversarial networks. arXiv preprint arXiv:1603.05631
  23. 23.
    Yang J, Kannan A, Batra D, Parikh D (2017) Lr-gan: layered recursive generative adversarial networks for image generation. arXiv preprint arXiv:1703.01560
  24. 24.
    Denton E, Chintala S, Szlam A, Fergus R (2015) Deep generative image models using a Laplacian pyramid of adversarial networks. In: Proceeding the NIPS, pp 1486–1494Google Scholar
  25. 25.
    Burt PJ, Adelson EH (1987) The Laplacian pyramid as a compact image code. In: Readings in computer vision. Elsevier, pp 671–679Google Scholar
  26. 26.
    Chen R, Qu Y, Li C et al (2018) Single-image super-resolution via joint statistic models-guided deep auto-encoder network. Neural Comput Appl. Google Scholar
  27. 27.
    Liu M-Y, Tuzel O (2016) Coupled generative adversarial networks. In: Proceedings of the advances in neural information processing systems (NIPS 2016), Barcelona, Spain, pp 469–477Google Scholar
  28. 28.
    Kliger M, Fleishman S (2018) Novelty detection with GAN. arXiv:1802.10560v1 [cs.CV]
  29. 29.
    He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: ICCVGoogle Scholar
  30. 30.
    Maas A, Hannun AY, Ng AY (2013) Rectifier nonlinearities improve neural network acoustic modelsGoogle Scholar
  31. 31.
    Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M et al (2016) TensorFlow: a system for large-scale machine learning. In: OSDI, vol 16, pp 265–283Google Scholar
  32. 32.
    Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. CoRR, vol. abs/1412.6980Google Scholar
  33. 33.
    Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the CVPR, pp 1646–1654Google Scholar
  34. 34.
    Lai WS, Huang J-B, Ahuja N, Yang M-H (2017) Deep Laplacian pyramid networks for fast and accurate superresolution. In: CVPR, pp 624–632Google Scholar
  35. 35.
    Wang Y, Perazzi F, Williams BM, Hornung AS, Hornung OS, Schroers C (2017) A fully progressive approach to single-image super-resolution. arXiv:1804.02900v2
  36. 36.
    Berthelot D, Schumm T, Metz L (2017) Began: boundary equilibrium generative adversarial networks. CoRR, abs/1703.10717Google Scholar
  37. 37.
    Juefei-Xu F, Boddeti VN, Savvides M (2017) Gang of gans: generative adversarial networks with maximum margin ranking. arXiv preprint arXiv:1704.04865
  38. 38.
    Metz L, Poole B, Pfau D, Sohl-Dickstein J (2016) Unrolled generative adversarial networks. arXiv preprint arXiv:1611.02163
  39. 39.
    Wang R, Cully A, Chang HJ, Demiris Y (2017) Magan: Margin adaptation for generative adversarial networks. arXiv preprint arXiv:1704.03817
  40. 40.
    Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: ECCVGoogle Scholar
  41. 41.
    Huang G, Liu Z, van der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: CVPRGoogle Scholar
  42. 42.
    Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network. In: Proceedings of the CVPR, pp 2790–2798Google Scholar
  43. 43.
    Wu H, Zheng S, Zhang J, Huang K (2017) GP-GAN: towards realistic high-resolution image blending. arXiv:1703.07195v2
  44. 44.
    Wang X, Yu K, Dong C, Loy CC (2018) Recovering realistic texture in image super-resolution by deep spatial feature transform. In: CVPR. arXiv:1804.02815v1
  45. 45.
    Dong C, Loy CC, Tang X (2016) Accelerating the super-resolution convolutional neural network. In: European conference on computer vision (ECCV), pp 391–407Google Scholar
  46. 46.
    Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y (2018) Residual dense network for image super-resolution. In: CVPRGoogle Scholar
  47. 47.
    Tai Y, Yang J, Liu X, Xu C (2017) Memnet: a persistent memory network for image restoration. In: ICCVGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  1. 1.School of Electronic Information and Electrical EngineeringShanghai Jiao Tong UniversityShanghaiChina
  2. 2.Department of InformaticsUniversity of LeicesterLeicesterUK

Personalised recommendations