Advertisement

The SUN Attribute Database: Organizing Scenes by Affordances, Materials, and Layout

  • Genevieve Patterson
  • James Hays
Chapter
Part of the Advances in Computer Vision and Pattern Recognition book series (ACVPR)

Abstract

One of the core challenges of computer vision is understanding the content of a scene. Often, scene understanding is demonstrated in terms of object recognition, 3D layout estimation from multiple views, or scene categorization. In this chapter we instead reason about scene attributes—high-level properties of scenes related to affordances (‘shopping,’ ‘studying’), materials (‘rock,’ ‘carpet’), surface properties (‘dirty,’ ‘dry’), spatial layout (‘symmetrical,’ ‘enclosed’), lighting (‘direct sun,’ ‘electric lighting’), and more (‘scary,’ ‘cold’). We describe crowd experiments to first determine a taxonomy of 102 interesting attributes and then to annotate binary attributes for 14,140 scenes. These scenes are sampled from 707 categories of the SUN database and this lets us study the interplay between scene attributes and scene categories. We evaluate attribute recognition with several existing scene descriptors. Our experiments suggest that scene attributes are an efficient feature for capturing high-level semantics in scenes.

Keywords

Convolutional Neural Network Attribute Classifier Scene Category Binary Attribute Attribute Label 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Notes

Acknowledgements

We thank our collaborators Chen Xu and Hang Su for their significant contributions as co-authors on the IJCV submission of our work with Scene Attributes [23]. We also thank Vazheh Moussavi for his insights and contributions in the data annotation process. Genevieve Patterson was supported by the Department of Defense (DoD) through the National Defense Science & Engineering Graduate Fellowship (NDSEG) Program. This work was also funded by NSF CAREER Award 1149853 to James Hays.

References

  1. 1.
    Berg, T., Berg, A., Shih, J.: Automatic attribute discovery and characterization from noisy web data. In: European Conference on Computer Vision (ECCV) (2010)Google Scholar
  2. 2.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2009)Google Scholar
  3. 3.
    Ehinger, K.A., Xiao, J., Torralba, A., Oliva, A.: Estimating scene typicality from human ratings and image features. In: 33rd Annual Conference of the Cognitive Science Society (2011)Google Scholar
  4. 4.
    Endres, I., Farhadi, A., Hoiem, D., Forsyth, D.: The benefits and challenges of collecting richer object annotations. In: Advancing Computer Vision with Humans in the Loop (ACVHL) (in conjunction with CVPR) (2010)Google Scholar
  5. 5.
    Farhadi, A., Endres, I., Hoiem, D., Forsyth, D.: Describing objects by their attributes. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2009)Google Scholar
  6. 6.
    Farhadi, A., Endres, I., Hoiem, D.: Attribute-centric recognition for cross-category generalization. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2010)Google Scholar
  7. 7.
    Ferrari, V., Zisserman, A.: Learning visual attributes. In: Conference on Neural Information Processing Systems (NIPS) (2008)Google Scholar
  8. 8.
    Greene, M., Oliva, A.: Recognition of natural scenes from global properties: seeing the forest without representing the trees. Cogn. Psychol. 58(2), 137–176 (2009)CrossRefGoogle Scholar
  9. 9.
    Kovashka, A., Grauman, K.: Attribute adaptation for personalized image search. In: International Conference on Computer Vision (ICCV) (2013)Google Scholar
  10. 10.
    Kumar, N., Berg, A., Belhumeur, P., Nayar, S.: Attribute and simile classifiers for face verification. In: International Conference on Computer Vision (ICCV) (2009)Google Scholar
  11. 11.
    Lampert, C.H., Nickisch, H., Harmeling, S.: Learning to detect unseen object classes by between-class attribute transfer. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2009)Google Scholar
  12. 12.
    Lampert, C.H., Nickisch, H., Harmeling, S.: Attribute-based classification for zero-shot visual object categorization. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 36(3), 453–465 (2014)CrossRefGoogle Scholar
  13. 13.
    Lasecki, W.S., Murray, K.I., White, S., Miller, R.C., Bigham, J.P.: Real-time crowd control of existing interfaces. In: User Interface Software and Technology Symposium (UIST) (2011)Google Scholar
  14. 14.
    Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2006)Google Scholar
  15. 15.
    Liu, J., Kuipers, B., Savarese, S.: Recognizing human actions by attributes. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2011)Google Scholar
  16. 16.
    Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. J. Mach. Learn. Res. (JMLR) 9(2579–2605), 85 (2008)zbMATHGoogle Scholar
  17. 17.
    Mason, R., Charniak, E.: Nonparametric method for data-driven image captioning. In: Annual meeting of the Association for Computational Linguistics (ACL) (2014)Google Scholar
  18. 18.
    Oliva, A., Torralba, A.: Modeling the shape of the scene: a holistic representation of the spatial envelope. Int. J. Comput. Vision (IJCV) 42(3), 145–175 (2001)CrossRefzbMATHGoogle Scholar
  19. 19.
    Oliva, A., Torralba, A.: Scene-centered description from spatial envelope properties. In: 2nd Workshop on Biologically Motivated Computer Vision (BMCV) (2002)Google Scholar
  20. 20.
    Palatucci, M., Pomerleau, D., Hinton, G.E., Mitchell, T.M.: Zero-shot learning with semantic output codes. In: Conference on Neural Information Processing Systems (NIPS) (2009)Google Scholar
  21. 21.
    Parikh, D., Grauman, K.: Interactively building a discriminative vocabulary of nameable attributes. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2011)Google Scholar
  22. 22.
    Patterson, G., Hays, J.: Sun attribute database: discovering, annotating, and recognizing scene attributes. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2012)Google Scholar
  23. 23.
    Patterson, G., Xu, C., Su, H., Hays, J.: The sun attribute database: beyond categories for deeper scene understanding. Int. J. Comput. Vision (IJCV) 108(1–2), 59–81 (2014)CrossRefGoogle Scholar
  24. 24.
    Rohrbach, M., Stark, M., Schiele, B.: Evaluating knowledge transfer and zero-shot learning in a large-scale setting. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2011)Google Scholar
  25. 25.
    Russakovsky, O., Fei-Fei, L.: Attribute learning in largescale datasets. In: ECCV Workshop on Parts and Attributes (2010)Google Scholar
  26. 26.
    Russell, B.C., Torralba, A., Murphy, K.P., Freeman, W.T.: Labelme: a database and web-based tool for image annotation. In: International Conference on Computer Vision (ICCV) (2008)Google Scholar
  27. 27.
    Sorokin, A., Forsyth, D.: Utility data annotation with amazon mechanical turk. In: First IEEE Workshop on Internet Vision at CVPR (2008)Google Scholar
  28. 28.
    Su, Y., Allan, M., Jurie, F.: Improving object classification using semantic attributes. In: British Machine Vision Conference (BMVC) (2010)Google Scholar
  29. 29.
    Torralba, A., Fergus, R., Freeman, W.T.: 80 million tiny images: a large dataset for non-parametric object and scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 30(11), 1958–1970 (2008)CrossRefGoogle Scholar
  30. 30.
    Wang, S., Joo, J., Wang, Y., Zhu, S.C.: Weakly supervised learning for attribute localization in outdoor scenes. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2013)Google Scholar
  31. 31.
    Xiao, J., Hays, J., Ehinger, K., Oliva, A., Torralba, A.: SUN database: Large-scale scene recognition from abbey to zoo. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2010)Google Scholar
  32. 32.
    Yao, B., Jiang, X., Khosla, A., Lin, A.L., Guibas, L., Fei-Fei, L.: Human action recognition by learning bases of action attributes and parts. In: International Conference on Computer Vision (ICCV) (2011)Google Scholar
  33. 33.
    Zhou, B., Lapedriza, A., Xiao, J., Torralba, A., Oliva, A.: Learning deep features for scene recognition using places database. In: Conference on Neural Information Processing Systems (NIPS) (2014)Google Scholar
  34. 34.
    Zhou, B., Liu, L., Oliva, A., Torralba, A.: Recognizing city identity via attribute analysis of geo-tagged images. In: European Conference on Computer Vision (ECCV) (2014)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Brown UniversityProvidenceUSA
  2. 2.Georgia Institute of TechnologyAtlantaUSA

Personalised recommendations