Skip to main content
SpringerLink
Log in
Book cover

Visual Attributes pp 269–297Cite as

The SUN Attribute Database: Organizing Scenes by Affordances, Materials, and Layout

  • Chapter
  • First Online:

  • 1465 Accesses

Part of the book series: Advances in Computer Vision and Pattern Recognition ((ACVPR))

Abstract

One of the core challenges of computer vision is understanding the content of a scene. Often, scene understanding is demonstrated in terms of object recognition, 3D layout estimation from multiple views, or scene categorization. In this chapter we instead reason about scene attributes—high-level properties of scenes related to affordances (‘shopping,’ ‘studying’), materials (‘rock,’ ‘carpet’), surface properties (‘dirty,’ ‘dry’), spatial layout (‘symmetrical,’ ‘enclosed’), lighting (‘direct sun,’ ‘electric lighting’), and more (‘scary,’ ‘cold’). We describe crowd experiments to first determine a taxonomy of 102 interesting attributes and then to annotate binary attributes for 14,140 scenes. These scenes are sampled from 707 categories of the SUN database and this lets us study the interplay between scene attributes and scene categories. We evaluate attribute recognition with several existing scene descriptors. Our experiments suggest that scene attributes are an efficient feature for capturing high-level semantics in scenes.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    SUN attribute Classifiers along with the full SUN attribute dataset and associated code are available at www.cs.brown.edu/~gen/sunattributes.html.

  2. 2.

    The images in the SUN attribute dataset were originally taken from the whole SUN dataset, which includes more than 900 scene categories. Thus, some portion of the SUN attribute images also appear in the SUN 397 dataset, which is also a subset of the full SUN dataset. The scene classifiers using low-level and predicted attribute features were trained and tested on the SUN397 dataset minus any overlapping images from the SUN attribute dataset to avoid testing scene classification on the same images used to train attribute classifiers.

References

  1. Berg, T., Berg, A., Shih, J.: Automatic attribute discovery and characterization from noisy web data. In: European Conference on Computer Vision (ECCV) (2010)

    Google Scholar 

  2. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2009)

    Google Scholar 

  3. Ehinger, K.A., Xiao, J., Torralba, A., Oliva, A.: Estimating scene typicality from human ratings and image features. In: 33rd Annual Conference of the Cognitive Science Society (2011)

    Google Scholar 

  4. Endres, I., Farhadi, A., Hoiem, D., Forsyth, D.: The benefits and challenges of collecting richer object annotations. In: Advancing Computer Vision with Humans in the Loop (ACVHL) (in conjunction with CVPR) (2010)

    Google Scholar 

  5. Farhadi, A., Endres, I., Hoiem, D., Forsyth, D.: Describing objects by their attributes. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2009)

    Google Scholar 

  6. Farhadi, A., Endres, I., Hoiem, D.: Attribute-centric recognition for cross-category generalization. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2010)

    Google Scholar 

  7. Ferrari, V., Zisserman, A.: Learning visual attributes. In: Conference on Neural Information Processing Systems (NIPS) (2008)

    Google Scholar 

  8. Greene, M., Oliva, A.: Recognition of natural scenes from global properties: seeing the forest without representing the trees. Cogn. Psychol. 58(2), 137–176 (2009)

    Article  Google Scholar 

  9. Kovashka, A., Grauman, K.: Attribute adaptation for personalized image search. In: International Conference on Computer Vision (ICCV) (2013)

    Google Scholar 

  10. Kumar, N., Berg, A., Belhumeur, P., Nayar, S.: Attribute and simile classifiers for face verification. In: International Conference on Computer Vision (ICCV) (2009)

    Google Scholar 

  11. Lampert, C.H., Nickisch, H., Harmeling, S.: Learning to detect unseen object classes by between-class attribute transfer. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2009)

    Google Scholar 

  12. Lampert, C.H., Nickisch, H., Harmeling, S.: Attribute-based classification for zero-shot visual object categorization. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 36(3), 453–465 (2014)

    Article  Google Scholar 

  13. Lasecki, W.S., Murray, K.I., White, S., Miller, R.C., Bigham, J.P.: Real-time crowd control of existing interfaces. In: User Interface Software and Technology Symposium (UIST) (2011)

    Google Scholar 

  14. Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2006)

    Google Scholar 

  15. Liu, J., Kuipers, B., Savarese, S.: Recognizing human actions by attributes. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2011)

    Google Scholar 

  16. Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. J. Mach. Learn. Res. (JMLR) 9(2579–2605), 85 (2008)

    MATH  Google Scholar 

  17. Mason, R., Charniak, E.: Nonparametric method for data-driven image captioning. In: Annual meeting of the Association for Computational Linguistics (ACL) (2014)

    Google Scholar 

  18. Oliva, A., Torralba, A.: Modeling the shape of the scene: a holistic representation of the spatial envelope. Int. J. Comput. Vision (IJCV) 42(3), 145–175 (2001)

    Article  MATH  Google Scholar 

  19. Oliva, A., Torralba, A.: Scene-centered description from spatial envelope properties. In: 2nd Workshop on Biologically Motivated Computer Vision (BMCV) (2002)

    Google Scholar 

  20. Palatucci, M., Pomerleau, D., Hinton, G.E., Mitchell, T.M.: Zero-shot learning with semantic output codes. In: Conference on Neural Information Processing Systems (NIPS) (2009)

    Google Scholar 

  21. Parikh, D., Grauman, K.: Interactively building a discriminative vocabulary of nameable attributes. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2011)

    Google Scholar 

  22. Patterson, G., Hays, J.: Sun attribute database: discovering, annotating, and recognizing scene attributes. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2012)

    Google Scholar 

  23. Patterson, G., Xu, C., Su, H., Hays, J.: The sun attribute database: beyond categories for deeper scene understanding. Int. J. Comput. Vision (IJCV) 108(1–2), 59–81 (2014)

    Article  Google Scholar 

  24. Rohrbach, M., Stark, M., Schiele, B.: Evaluating knowledge transfer and zero-shot learning in a large-scale setting. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2011)

    Google Scholar 

  25. Russakovsky, O., Fei-Fei, L.: Attribute learning in largescale datasets. In: ECCV Workshop on Parts and Attributes (2010)

    Google Scholar 

  26. Russell, B.C., Torralba, A., Murphy, K.P., Freeman, W.T.: Labelme: a database and web-based tool for image annotation. In: International Conference on Computer Vision (ICCV) (2008)

    Google Scholar 

  27. Sorokin, A., Forsyth, D.: Utility data annotation with amazon mechanical turk. In: First IEEE Workshop on Internet Vision at CVPR (2008)

    Google Scholar 

  28. Su, Y., Allan, M., Jurie, F.: Improving object classification using semantic attributes. In: British Machine Vision Conference (BMVC) (2010)

    Google Scholar 

  29. Torralba, A., Fergus, R., Freeman, W.T.: 80 million tiny images: a large dataset for non-parametric object and scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 30(11), 1958–1970 (2008)

    Article  Google Scholar 

  30. Wang, S., Joo, J., Wang, Y., Zhu, S.C.: Weakly supervised learning for attribute localization in outdoor scenes. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2013)

    Google Scholar 

  31. Xiao, J., Hays, J., Ehinger, K., Oliva, A., Torralba, A.: SUN database: Large-scale scene recognition from abbey to zoo. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2010)

    Google Scholar 

  32. Yao, B., Jiang, X., Khosla, A., Lin, A.L., Guibas, L., Fei-Fei, L.: Human action recognition by learning bases of action attributes and parts. In: International Conference on Computer Vision (ICCV) (2011)

    Google Scholar 

  33. Zhou, B., Lapedriza, A., Xiao, J., Torralba, A., Oliva, A.: Learning deep features for scene recognition using places database. In: Conference on Neural Information Processing Systems (NIPS) (2014)

    Google Scholar 

  34. Zhou, B., Liu, L., Oliva, A., Torralba, A.: Recognizing city identity via attribute analysis of geo-tagged images. In: European Conference on Computer Vision (ECCV) (2014)

    Google Scholar 

Download references

Acknowledgements

We thank our collaborators Chen Xu and Hang Su for their significant contributions as co-authors on the IJCV submission of our work with Scene Attributes [23]. We also thank Vazheh Moussavi for his insights and contributions in the data annotation process. Genevieve Patterson was supported by the Department of Defense (DoD) through the National Defense Science & Engineering Graduate Fellowship (NDSEG) Program. This work was also funded by NSF CAREER Award 1149853 to James Hays.

Author information

Authors and Affiliations

  1. Brown University, 112 Waterman St., Providence, RI, USA

    Genevieve Patterson

  2. Georgia Institute of Technology, 801 Atlantic Dr NW, Atlanta, GA, USA

    James Hays

Authors
  1. Genevieve Patterson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. James Hays
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Genevieve Patterson .

Editor information

Editors and Affiliations

  1. IBM T.J. Watson Research Center, Yorktown Heights, New York, USA

    Rogerio Schmidt Feris

  2. IST Austria Computer Vision and Machine Learning, Klosterneuburg, Austria

    Christoph Lampert

  3. Virginia Tech Electrical and Computer Engineering, Blacksburg, Virginia, USA

    Devi Parikh

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Cite this chapter

Patterson, G., Hays, J. (2017). The SUN Attribute Database: Organizing Scenes by Affordances, Materials, and Layout. In: Feris, R., Lampert, C., Parikh, D. (eds) Visual Attributes. Advances in Computer Vision and Pattern Recognition. Springer, Cham. https://doi.org/10.1007/978-3-319-50077-5_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-50077-5_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-50075-1

  • Online ISBN: 978-3-319-50077-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation