Computer Vision

Living Edition

Facial Attribute Recognition: A Survey

  • Nathan Thom
  • Emily M. HandEmail author
Living reference work entry
DOI: https://doi.org/10.1007/978-3-030-03243-2_815-1
  • 1 Downloads

Definition

We present a survey of attribute recognition research in the computer vision community over the past decade. Most of our attention is given to facial attributes, but attributes of objects, pedestrians, and actions are considered as well.

Background

Facial attributes – human-describable features of faces – were introduced to the computer vision community in 2008, with their first application being image search [1]. Kumar et al. identified a problem with the image search engines of the time, realizing that simple descriptive search terms would not produce expected face image results. Attributes were then used for face recognition and verification as well as, again, for image search and retrieval [2, 3, 4] before attribute recognition itself became the focus of research. Facial attribute recognition is related to the problem of soft biometrics [5], which is focused on identification using these so-called soft traits rather than recognizing them for purely descriptive purposes.

W...

This is a preview of subscription content, log in to check access.

References

  1. 1.
    Kumar N, Belhumeur P, Nayar S (2008) Facetracer: a search engine for large collections of images with faces. In: European conference on computer vision. Springer, pp 340–353Google Scholar
  2. 2.
    Kumar N, Berg A, Belhumeur P, Nayar S (2009) Attribute and simile classifiers for face verification. In: International conference on computer vision. IEEE, pp 365–372Google Scholar
  3. 3.
    Kumar N, Berg A, Belhumeur PN, Nayar S (2011) Describable visual attributes for face verification and image search. IEEE Trans Pattern Anal Mach Intell 33(10):1962–1977CrossRefGoogle Scholar
  4. 4.
    Siddiquie B, Feris RS, Davis LS (2011) Image ranking and retrieval based on multi-attribute queries. In: CVPR 2011, pp 801–808Google Scholar
  5. 5.
    Jain AK, Dass SC, Nandakumar K (2004) Can soft biometric traits assist user recognition? In: Biometric technology for human identification. International Society for Optics and Photonics, vol 5404, pp 561–573Google Scholar
  6. 6.
    Ng CB, Tay YH, Goi B-M (2012) Recognizing human gender in computer vision: a survey. In: Pacific rim international conference on artificial intelligence. Springer, pp 335–346Google Scholar
  7. 7.
    Ramanathan N, Chellappa R, Biswas S (2009) Computational methods for modeling facial aging: a survey. J Vis Lang Comput 20(3):131–144CrossRefGoogle Scholar
  8. 8.
    Fu Y, Guo G, Huang TS (2010) Age synthesis and estimation via faces: a survey. IEEE Trans Pattern Anal Mach Intell 32(11):1955–1976CrossRefGoogle Scholar
  9. 9.
    Golomb BA, Lawrence DT, Sejnowski TJ (1990) Sexnet: a neural network identifies sex from human faces. In: NIPS, vol 1, p 2Google Scholar
  10. 10.
    Cottrell GW, Metcalfe J (1991) EMPATH: face, emotion, and gender recognition using holons. In: Advances in neural information processing systems, pp 564–571Google Scholar
  11. 11.
    Gao W, Ai H (2009) Face gender classification on consumer images in a multiethnic environment. In: International conference on biometrics. Springer, pp 169–178Google Scholar
  12. 12.
    BenAbdelkader C, Griffin P (2005) A local region-based approach to gender classification from face images. In: 2005 IEEE computer society conference on computer vision and pattern recognition-workshops, CVPR Workshops. IEEE, p 52Google Scholar
  13. 13.
    Guo G, Dyer CR, Fu Y, Huang TS (2009) Is gender recognition affected by age. In: 2009 IEEE 12th international conference on computer vision workshops (ICCV Workshops). IEEE, pp 2032–2039Google Scholar
  14. 14.
    Kwon YH, da Vitoria Lobo N (1999) Age classification from facial images. Comput Vis Image Underst 74(1):1–21CrossRefGoogle Scholar
  15. 15.
    Lanitis A, Taylor CJ, Cootes TF (2002) Toward automatic simulation of aging effects on face images. IEEE Trans Pattern Anal Mach Intell 24(4):442–455CrossRefGoogle Scholar
  16. 16.
    Geng X, Zhou Z-H, Zhang Y, Li G, Dai H (2006) Learning from facial aging patterns for automatic age estimation. In: Proceedings of the 14th ACM international conference on multimedia. ACM, pp 307–316Google Scholar
  17. 17.
    Ramanathan N, Chellappa R (2006) Face verification across age progression. IEEE Trans Image Process 15(11):3349–3361CrossRefGoogle Scholar
  18. 18.
    Zhang N, Paluri M, Ranzato M, Darrell T, Bourdev L (2014) Panda: pose aligned networks for deep attribute modeling. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1637–1644Google Scholar
  19. 19.
    Farhadi A, Endres I, Hoiem D, Forsyth D (2009) Describing objects by their attributes. In: 2009 IEEE conference on computer vision and pattern recognition CVPR 2009. IEEE, pp 1778–1785Google Scholar
  20. 20.
    Berg T, Berg A, Shih J (2010) Automatic attribute discovery and characterization from noisy web data. Eur Conf Comput Vis 6311:663–676Google Scholar
  21. 21.
    Liu J, Kuipers B, Savarese S (2011) Recognizing human actions by attributes. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 3337–3344Google Scholar
  22. 22.
    Scheirer WJ, Kumar N, Belhumeur PN, Boult TE (2012) Multi-attribute spaces: calibration for attribute fusion and similarity search. In: 2012 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 2933–2940Google Scholar
  23. 23.
    Newton C (2016) Facebook begins using artificial intelligence to describe photos to blind users. [Online; posted 5-April-2016].Google Scholar
  24. 24.
    Samangouei P, Patel VM, Chellappa R (2015) Attribute-based continuous user authentication on mobile devices. In: 2015 IEEE 7th international conference on biometrics theory, applications and systems (BTAS). IEEE, pp 1–8Google Scholar
  25. 25.
    Samangouei P, Chellappa R (2016) Convolutional neural networks for attribute-based active authentication on mobile devices. In: 2016 IEEE 8th international conference on biometrics theory, applications and systems (BTAS). IEEE, pp 1–8Google Scholar
  26. 26.
    Samangouei P, Hand E, Patel VM, Chellappa R (2017) Active authentication using facial attributes. Mobile Biometrics 3:131Google Scholar
  27. 27.
    Russakovsky O, Fei-Fei L (2012) Attribute learning in large-scale datasets. In: Kutulakos KN (ed) Trends and topics in computer vision. Springer, Berlin/Heidelberg, pp 1–14Google Scholar
  28. 28.
    Jojic N, Caspi Y (2004) Capturing image structure with probabilistic index maps. In: Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, CVPR 2004, vol 1, pp I–IGoogle Scholar
  29. 29.
    Lazebnik S, Schmid C, Ponce J (2005) A sparse texture representation using local affine regions. IEEE Trans Pattern Anal Mach Intell 27:1265–1278CrossRefGoogle Scholar
  30. 30.
    Liu Y, Tsin Y, Lin W-C (2005) The promise and perils of near-regular texture. Int J Comput Vis 62:145–159CrossRefGoogle Scholar
  31. 31.
    Ferrari V, Zisserman A (2008) Learning visual attributes. In: Platt JC, Koller D, Singer Y, Roweis ST (eds) Advances in neural information processing systems 20. Curran Associates, Inc., pp 433–440Google Scholar
  32. 32.
    Parikh D, Grauman K (2011) Interactively building a discriminative vocabulary of nameable attributes. In: CVPR 2011, pp 1681–1688Google Scholar
  33. 33.
    Wang X, Ji Q (2013) A unified probabilistic approach modeling relationships between attributes and objects. In: 2013 IEEE international conference on computer vision, pp 2120–2127Google Scholar
  34. 34.
    Sun Y, Bo L, Fox D (2013) Attribute based object identification. In: 2013 IEEE international conference on robotics and automation, pp 2096–2103Google Scholar
  35. 35.
    Vedaldi A, Mahendran S, Tsogkas S, Maji S, Girshick R, Kannala J, Rahtu E, Kokkinos I, Blaschko MB, Weiss D, Taskar B, Simonyan K, Saphra N, Mohamed S (2014) Understanding objects in detail with fine-grained attributes. In: 2014 IEEE conference on computer vision and pattern recognition, pp 3622–3629Google Scholar
  36. 36.
    Wang X, Ji Q (2016) Object recognition with hidden attributes. In: Proceedings of the twenty-fifth international joint conference on artificial intelligence, IJCAI’16. AAAI Press, pp 3498–3504Google Scholar
  37. 37.
    Jhuang H, Serre T, Wolf L, Poggio T (2007) A biologically inspired system for action recognition. In: 2007 IEEE 11th international conference on computer vision, pp 1–8Google Scholar
  38. 38.
    Yao B, Jiang X, Khosla A, Lin AL, Guibas L, Fei-Fei L (2011) Human action recognition by learning bases of action attributes and parts. In: 2011 International conference on computer vision, pp 1331–1338Google Scholar
  39. 39.
    Sharma G, Jurie F, Schmid C (2013) Expanded parts model for human attribute and action recognition in still images. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition (CVPR)Google Scholar
  40. 40.
    Zhang Z, Wang C, Xiao B, Zhou W, Liu S (2013) Attribute regularization based human action recognition. IEEE Trans Inf Forensics Secur 8(10):1600–1609CrossRefGoogle Scholar
  41. 41.
    Tahmoush D (2015) Applying action attribute class validation to improve human activity recognitionGoogle Scholar
  42. 42.
    Cai X, Zhou W, Li H (2015) Attribute mining for scalable 3D human action recognition. In: Proceedings of the 23rd ACM international conference on multimediaGoogle Scholar
  43. 43.
    Chen K, Ding G, Han J (2017) Attribute-based supervised deep learning model for action recognition. Front Comput Sci. abs/1707.09468, pp 219–229Google Scholar
  44. 44.
    Khan FS, van de Weijer J, Anwer RM, Bagdanov AD, Felsberg M, Laaksonen J (2018) Scale coding bag of deep features for human attribute and action recognition. Mach Vis Appl 29:55–71CrossRefGoogle Scholar
  45. 45.
    Zaki MH, Sayed T (2014) Using automated walking gait analysis for the identification of pedestrian attributes. Transp Res Part C: Emerg Technol 48: 16–36CrossRefGoogle Scholar
  46. 46.
    Deng Y, Luo P, Loy CC, Tang X (2014) Pedestrian attribute recognition at far distance. In: Proceedings of the 22nd ACM international conference on multimedia, MM ’14, pp 789–792Google Scholar
  47. 47.
    Deng Y, Luo P, Loy CC, Tang X (2015) Learning to recognize pedestrian attribute. CoRR abs/1501.00901Google Scholar
  48. 48.
    Zhu J, Liao S, Yi D, Lei Z, Li SZ (2015) Multi-label CNN based pedestrian attribute learning for soft biometricsGoogle Scholar
  49. 49.
    Yu K, Leng B, Zhang Z, Li D, Huang K (2016) Weakly-supervised learning of mid-level features for pedestrian attribute recognition and localization. CoRR, vol abs/1611.05603, pp 224–229Google Scholar
  50. 50.
    Lu Y, Kumar A, Zhai S, Cheng Y, Javidi T, Feris RS (2016) Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification. CoRR, vol abs/1611.05377Google Scholar
  51. 51.
    Zhao X, Sang L, Ding G, Guo Y, Jin X (2018) Grouping attribute recognition for pedestrian with joint recurrent learning. In: Proceedings of the twenty-seventh international joint conference on artificial intelligence, IJCAI-18, International joint conferences on artificial intelligence organization, 7, pp 3177–3183Google Scholar
  52. 52.
    Chen Z, Li A, Wang Y (2019) Video-based pedestrian attribute recognition. CoRR, vol abs/1901.05742Google Scholar
  53. 53.
    Vaquero DA, Feris RS, Tran D, Brown L, Hampapur A, Turk M (2009) Attribute-based people search in surveillance environments. In: 2009 workshop on applications of computer vision (WACV). IEEE, pp 1–8Google Scholar
  54. 54.
    Huang GB, Mattar M, Berg T, E. Learned-Miller (2008) Labeled faces in the wild: a database forstudying face recognition in unconstrained environments. In: Workshop on faces in ‘Real-Life ’Images: detection, alignment, and recognitionGoogle Scholar
  55. 55.
    Parikh D, Grauman K (2011) Relative attributes. In: 2011 IEEE international conference on computer vision (ICCV). IEEE, pp 503–510Google Scholar
  56. 56.
    Berg T, Belhumeur PN (2012) Tom-vs-pete classifiers and identity-preserving alignment for face verification. In: BMVC, Citeseer, vol 2, p 7Google Scholar
  57. 57.
    Liu Z, Luo P, Wang X, Tang X (2015) Deep learning face attributes in the wild. In: Proceedings of the ieee international conference on computer vision, pp 3730–3738Google Scholar
  58. 58.
    Hand EM, Chellappa R (2017) Attributes for improved attributes: a multi-task network utilizing implicit and explicit relationships for facial attribute classification. In: AAAI, pp 4068–4074Google Scholar
  59. 59.
    Wang J, Cheng Y, R. Schmidt Feris (2016) Walk and learn: facial attribute representation learning from egocentric video and contextual data. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2295–2304Google Scholar
  60. 60.
    Rudd EM, Günther M, Boult TE (2016) Moon: a mixed objective optimization network for the recognition of facial attributes. In: European conference on computer vision. Springer, pp 19–35Google Scholar
  61. 61.
    Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105Google Scholar
  62. 62.
    Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 248–255Google Scholar
  63. 63.
    Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556Google Scholar
  64. 64.
    Ehrlich M, Shields TJ, Almaev T, Amer MR (2016) Facial attributes classification using multi-task representation learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 47–55Google Scholar
  65. 65.
    Rozsa A, Günther M, Rudd EM, Boult TE (2016) Are facial attributes adversarially robust? In: 2016 23rd International conference on pattern recognition (ICPR). IEEE, pp 3121–3127Google Scholar
  66. 66.
    Günther M, Rozsa A, Boult TE (2017) Affact: alignment-free facial attribute classification technique. In: 2017 IEEE international joint conference on biometrics (IJCB). IEEE, pp 90–99Google Scholar
  67. 67.
    He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778Google Scholar
  68. 68.
    Ding H, Zhou H, Zhou SK, Chellappa R (2018) A deep cascade network for unaligned face attribute classification. In: Thirty-second AAAI conference on artificial intelligenceGoogle Scholar
  69. 69.
    Kalayeh MM, Gong B, Shah M (2017) Improving facial attribute prediction using semantic segmentation. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 4227–4235Google Scholar
  70. 70.
    Le V, Brandt J, Lin Z, Bourdev L, Huang TS (2012) Interactive facial feature localization. In: European conference on computer vision. Springer, pp 679–692Google Scholar
  71. 71.
    He K, Fu Y, Zhang W, Wang C, Jiang Y-G, Huang F, Xue X (2018) Harnessing synthesized abstraction images to improve facial attribute recognition. In: IJCAI, pp 733–740Google Scholar
  72. 72.
    Hand EM, Castillo C, Chellappa R (2018) Doing the best we can with what we have: multi-label balancing with selective learning for attribute prediction. AAAI, New OrleansGoogle Scholar
  73. 73.
    Hand EM, Castillo CD, Chellappa R (2018) Predicting facial attributes in video using temporal coherence and motion-attention. In: 2018 IEEE winter conference on applications of computer vision (WACV). IEEE, pp 84–92Google Scholar
  74. 74.
    Wolf L, Hassner T, Maoz I (2011) Face recognition in unconstrained videos with matched background similarity. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 529–534Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.University of NevadaRenoUSA

Section editors and affiliations

  • Rama Chellappa
    • 1
  1. 1.University of MarylandCollege ParkUSA