Advertisement

Learning Style Compatibility for Furniture

  • Divyansh Aggarwal
  • Elchin Valiyev
  • Fadime SenerEmail author
  • Angela Yao
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11269)

Abstract

When judging style, a key question that often arises is whether or not a pair of objects are compatible with each other. In this paper we investigate how Siamese networks can be used efficiently for assessing the style compatibility between images of furniture items. We show that the middle layers of pretrained CNNs can capture essential information about furniture style, which allows for efficient applications of such networks for this task. We also use a joint image-text embedding method that allows for the querying of stylistically compatible furniture items, along with additional attribute constraints based on text. To evaluate our methods, we collect and present a large scale dataset of images of furniture of different style categories accompanied by text attributes.

Supplementary material

480455_1_En_38_MOESM1_ESM.pdf (1.8 mb)
Supplementary material 1 (pdf 1828 KB)

References

  1. 1.
    Bell, S., Bala, K.: Learning visual similarity for product design with convolutional neural networks. ACM Trans. Graph. 34, 98 (2015)CrossRefGoogle Scholar
  2. 2.
    Bromley, J., Guyon, I., LeCun, Y., Säckinger, E., Shah, R.: Signature verification using a “siamese” time delay neural network. In: Cowan, J.D., Tesauro, G., Alspector, J. (eds.) Advances in Neural Information Processing Systems 6, pp. 737–744. Morgan-Kaufmann (1994). http://papers.nips.cc/paper/769-signature-verification-using-a-siamese-time-delay-neural-network.pdf
  3. 3.
    Datta, R., Joshi, D., Li, J., Wang, J.Z.: Studying aesthetics in photographic images using a computational approach. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3953, pp. 288–301. Springer, Heidelberg (2006).  https://doi.org/10.1007/11744078_23CrossRefGoogle Scholar
  4. 4.
    Dhar, S., Ordonez, V., Berg, T.L.: High level describable attributes for predicting aesthetics and interestingness. In: Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2011 (2011)Google Scholar
  5. 5.
    Doersch, C., Singh, S., Gupta, A., Sivic, J., Efros, A.: What makes paris look like paris? ACM Trans. Graph. 31(4) (2012)CrossRefGoogle Scholar
  6. 6.
    Frome, A., et al.: DeViSE: a deep visual-semantic embedding model. In: Advances in Neural Information Processing Systems 26 (2013)Google Scholar
  7. 7.
    Garces, E., Agarwala, A., Gutierrez, D., Hertzmann, A.: A similarity measure for illustration style. ACM Trans. Graph. 33, 93 (2014)CrossRefGoogle Scholar
  8. 8.
    Gatys, L.A., Ecker, A.S., Bethge, M.: A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576 (2015)
  9. 9.
    Goel, A., Juneja, M., Jawahar, C.V.: Are buildings only instances?: exploration in architectural style categories. In: Proceedings of the Eighth Indian Conference on Computer Vision, Graphics and Image Processing, ICVGIP 2012 (2012)Google Scholar
  10. 10.
    Han, X., Wu, Z., Jiang, Y., Davis, L.S.: Learning fashion compatibility with bidirectional LSTMs. CoRR (2017)Google Scholar
  11. 11.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  12. 12.
    Hu, Z., et al.: Visual classification of furniture styles. ACM Trans. Intell. Syst. Technol. (2017)Google Scholar
  13. 13.
    Isola, P., Xiao, J., Parikh, D., Torralba, A., Oliva, A.: What makes a photograph memorable? IEEE Trans. Pattern Anal. Mach. Intell. 36, 1469–1482 (2014)CrossRefGoogle Scholar
  14. 14.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46475-6_43CrossRefGoogle Scholar
  15. 15.
    Karayev, S., et al.: Recognizing image style. arXiv preprint arXiv:1311.3715 (2013)
  16. 16.
    Keren, D.: Painter identification using local features and naive Bayes. In: Proceedings of Pattern Recognition Conference. IEEE (2002)Google Scholar
  17. 17.
    Kiapour, M.H., Han, X., Lazebnik, S., Berg, A.C., Berg, T.L.: Where to buy it: matching street clothing photos in online shops. In: Proceedings of the 2015 International Conference on Computer Vision. IEEE Press (2015)Google Scholar
  18. 18.
    Kiapour, M.H., Yamaguchi, K., Berg, A.C., Berg, T.L.: Hipster wars: discovering elements of fashion styles. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 472–488. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10590-1_31CrossRefGoogle Scholar
  19. 19.
  20. 20.
    Li, C., Chen, T.: Aesthetic visual quality assessment of paintings. IEEE J. Sel. Top. Sign. Proces. 3, 236–252 (2009)CrossRefGoogle Scholar
  21. 21.
    Lin, T.Y., Cui, Y., Belongie, S., Hays, J.: Learning deep representations for ground-to-aerial geolocalization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015)Google Scholar
  22. 22.
    Liu, T., Hertzmann, A., Li, W., Funkhouser, T.: Style compatibility for 3D furniture models. ACM Trans. Graph. 34, 85 (2015)Google Scholar
  23. 23.
    McAuley, J., Targett, C., Shi, Q., Van Den Hengel, A.: Image-based recommendations on styles and substitutes. In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM (2015)Google Scholar
  24. 24.
    Řehůřek, R., Sojka, P.: Software framework for topic modelling with large corpora. In: Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks. ELRA (2010)Google Scholar
  25. 25.
    Sangkloy, P., Burnell, N., Ham, C., Hays, J.: The sketchy database: learning to retrieve badly drawn bunnies. ACM Trans. Graph. 35, 119 (2016). Proceedings of SIGGRAPHCrossRefGoogle Scholar
  26. 26.
    Shamir, L., Macura, T., Orlov, N., Eckley, D.M., Goldberg, I.G.: Impressionism, expressionism, surrealism: automated recognition of painters and schools of art. ACM Trans. Appl. Percept. 7, 8 (2010)CrossRefGoogle Scholar
  27. 27.
    Simo-Serra, E., Fidler, S., Moreno-Noguer, F., Urtasun, R.: Neuroaesthetics in fashion: modeling the perception of fashionability. In: Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR 2015). IEEE Press (2015)Google Scholar
  28. 28.
    Simo-Serra, E., Ishikawa, H.: Fashion style in 128 floats: joint ranking and classification using weak data for feature extraction. In: Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  29. 29.
    Szegedy, C., et al.: Going deeper with convolutions. In: Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  30. 30.
    Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  31. 31.
    Veit, A., Kovacs, B., Bell, S., McAuley, J., Bala, K., Belongie, S.: Learning visual clothing style with heterogeneous dyadic co-occurrences. In: Proceedings of the IEEE International Conference on Computer Vision (2015)Google Scholar
  32. 32.
    Wang, J., et al.: Learning fine-grained image similarity with deep ranking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014)Google Scholar
  33. 33.
    Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10590-1_53CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Divyansh Aggarwal
    • 1
  • Elchin Valiyev
    • 2
  • Fadime Sener
    • 2
    Email author
  • Angela Yao
    • 2
  1. 1.IIT JodhpurJodhpurIndia
  2. 2.University of BonnBonnGermany

Personalised recommendations