Advertisement

Convolutional Photomosaic Generation via Multi-scale Perceptual Losses

  • Matthew TesfaldetEmail author
  • Nariman SaftarliEmail author
  • Marcus A. BrubakerEmail author
  • Konstantinos G. DerpanisEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11131)

Abstract

Photographic mosaics (or simply photomosaics) are images comprised of smaller, equally-sized image tiles such that when viewed from a distance, the tiled images of the mosaic collectively resemble a perceptually plausible image. In this paper, we consider the challenge of automatically generating a photomosaic from an input image. Although computer-generated photomosaicking has existed for quite some time, none have considered simultaneously exploiting colour/grayscale intensity and the structure of the input across scales, as well as image semantics. We propose a convolutional network for generating photomosaics guided by a multi-scale perceptual loss to capture colour, structure, and semantics across multiple scales. We demonstrate the effectiveness of our multi-scale perceptual loss by experimenting with producing extremely high resolution photomosaics and through the inclusion of ablation experiments that compare with a single-scale variant of the perceptual loss. We show that, overall, our approach produces visually pleasing results, providing a substantial improvement over common baselines.

Keywords

Photomosaic ASCII text Deep learning Perceptual loss Multi-scale analysis 

Notes

Acknowledgements

MT is supported by a Natural Sciences and Engineering Research Council of Canada (NSERC) Canadian Graduate Scholarship. KGD and MAB are supported by NSERC Discovery Grants. This research was undertaken as part of the Vision: Science to Applications program, thanks in part to funding from the Canada First Research Excellence Fund.

References

  1. 1.
    Abadi, M., et al.: TensorFlow: a system for large-scale machine learning. In: 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2016), pp. 265–283 (2016). https://www.usenix.org/system/files/conference/osdi16/osdi16-abadi.pdf
  2. 2.
    Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv:1607.06450 (2016)
  3. 3.
    Burt, P.J., Adelson, E.H.: The Laplacian pyramid as a compact image code. IEEE Trans. Commun. 31, 532–540 (1983)CrossRefGoogle Scholar
  4. 4.
    Dalí, S.: Gala Contemplating the Mediterranean Sea which at Twenty Meters Becomes the Portrait of Abraham Lincoln Exhibited in 1976. Guggenheim Museum, New YorkGoogle Scholar
  5. 5.
    Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: CVPR, pp. 2414–2423 (2016)Google Scholar
  6. 6.
    Harmon, L., Knowlton, K., Hay, D.: Studies in Perception I Exhibited at The Machine as Seen at the End of the Mechanical Age, 27 November 1968– 9 February 1969, The Museum of Modern Art, New YorkGoogle Scholar
  7. 7.
    Harmon, L.D.: The recognition of faces. Sci. Am. 229(5), 70–83 (1973)CrossRefGoogle Scholar
  8. 8.
    Jetchev, N., Bergmann, U., Seward, C.: GANosaic: mosaic creation with generative texture manifolds. In: NIPS Workshop (2017)Google Scholar
  9. 9.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46475-6_43CrossRefGoogle Scholar
  10. 10.
    Ke, T.W., Maire, M., Yu, S.X.: Multigrid neural architectures. In: CVPR, pp. 6665–6673 (2017)Google Scholar
  11. 11.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2014)Google Scholar
  12. 12.
    Kornmesser, M.: Top 100 images. https://www.spacetelescope.org/images/archive/top100 (2015), images by ESA/Hubble (M. Kornmesser)
  13. 13.
    Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10602-1_48CrossRefGoogle Scholar
  14. 14.
    Martins, D.: Photo-mosaic (2014). https://github.com/danielfm/photo-mosaic. Accessed 15 July 2018
  15. 15.
    Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. IJCV 115(3), 211–252 (2015)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)Google Scholar
  17. 17.
    Snelgrove, X.: High-resolution multi-scale neural texture synthesis. In: SIGGRAPH ASIA Technical Briefs (2017)Google Scholar
  18. 18.
    Tran, N.: Generating photomosaics: an empirical study. In: SAC, pp. 105–109 (1999)Google Scholar
  19. 19.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Signal Process 13, 600–612 (2004)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Electrical Engineering and Computer ScienceYork UniversityTorontoCanada
  2. 2.Vector InstituteTorontoCanada
  3. 3.Department of Computer ScienceRyerson UniversityTorontoCanada

Personalised recommendations