Face Recognition by Humans and Machines

  • Alice J. O’TooleEmail author


By the standards of automated face recognition systems, human performance is notable in its ability to operate robustly across changes in illumination, pose, and expression. This chapter presents a comparative examination of face recognition by humans and machines. The first part contains an overview of the basic characteristics of human face representations, with emphasis both on the strengths and weaknesses of this code. The second part of the chapter, considers recent comparisons between humans and automated face recognition algorithms. These include benchmarking the performance of algorithms against humans, the fusion of human and machine judgments of identity, and the robustness of algorithms operating in ethnically diverse environments. The evaluation of machine performance in the context of human skills can give insight into the challenges of face recognition in uncontrolled environments and into the strategies humans have evolved to overcome these challenges.


Face Recognition Human Face Average Face Face Recognition System Face Pair 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work was supported by funding to A.J. O’Toole from the Technical Support Working Group of the Department of Defense, USA.


  1. 1.
    Adolphs, R., Tranel, D., Damsio, H., Damasio, A.R.: Impaired recognition of emotion in facial expressions following bilateral damage to the human amygdala. Nature 372, 669–672 (1994) CrossRefGoogle Scholar
  2. 2.
    Antonakis, J., Dalgas, O.: Predicting elections: Child’s play! Science 323(5918), 1183–1327 (2007) CrossRefGoogle Scholar
  3. 3.
    Blanz, V., Vetter, T.: A morphable model for the synthesis of 3d faces. In: Proceedings, SIGGRAPH’99, pp. 187–194 (1999) Google Scholar
  4. 4.
    Bower, G.H., Karlin, M.B.: Depth of processing pictures of faces and recognition memory. J. Exp. Psychol. 103(4), 267–278 (1974) CrossRefGoogle Scholar
  5. 5.
    Braje, W.: Illumination encoding in face recognition: effect of position shift. J. Vis. 3, 161–170 (2003) CrossRefGoogle Scholar
  6. 6.
    Braje, W., Kersten, D., Tarr, M.J., Troje, N.F.: Illumination effects in face recognition. Psychobiology 26, 371–380 (1999) Google Scholar
  7. 7.
    Bruce, V., Young, A.W.: Understanding face recognition. Br. J. Psychol. 77(3), 305–327 (1986) CrossRefGoogle Scholar
  8. 8.
    Bryatt, G., Rhodes, G.: Recognition of own-race and other-race caricatures: Implications for models of face recognition. Vis. Res. 38, 2455–2468 (1998) CrossRefGoogle Scholar
  9. 9.
    Burton, A.M., Bruce, V., Hancock, P.J.B.: From pixels to people: A model of familiar face recognition. Trends Cogn. Sci. 23, 1–31 (1999) Google Scholar
  10. 10.
    Burton, A.M., Wilson, S., Cowan, M., Bruce, V.: Face recognition in poor-quality video. Psychol. Sci. 10, 243–248 (1999) CrossRefGoogle Scholar
  11. 11.
    Calder, A., Young, A.: Understanding the recognition of facial identity and facial expression. Nat. Rev., Neurosci. 6, 641–651 (2005) CrossRefGoogle Scholar
  12. 12.
    Doddington, G., Ligget, W., Martin, A., Przybocki, M., Reynolds, D.: Sheeps, goats, lambs, and wolves: A statistical analysis of speaker performance in the NIST 1998 recognition evaluation. In: Proceedings ICSLP ’98, 1998 Google Scholar
  13. 13.
    Ekman, P., Friesen, W.V.: Pictures of Facial Affect. Consulting Psychologists Press, Palo Alto (1976) Google Scholar
  14. 14.
    Gross, R., Baker, S., Matthews, I., Kanade, T.: Face recognition across pose and illumination. In: Li, S.Z., Jain, A.K. (eds.) Handbook of Face Recognition, pp. 193–216. Springer, Berlin (2005) CrossRefGoogle Scholar
  15. 15.
    Haxby, J.V., Hoffman, E.A., Gobbini, M.I.: The distributed human neural system for face perception. Trends Cogn. Sci. 20(6), 223–233 (2000) CrossRefGoogle Scholar
  16. 16.
    Hill, H., Bruce, V.: Effects of lighting on the perception of facial surface. J. Exp. Psychol. 22, 986–1004 (1996) Google Scholar
  17. 17.
    Kelly, D.J., Quinn, P..C., Slater, A.M., Lee, K., Ge, L., Pascalis, O.: The other-race effect develops during infancy: Evidence of perceptual narrowing. Psychol. Sci. 18, 1084–1089 (2007) CrossRefGoogle Scholar
  18. 18.
    Kuhl, P.K., Williams, K.H., Lacerdo, F.: Linguistic experience alters phonetic perception in infants by 6 months of age. Science 225, 606–608 (1992) CrossRefGoogle Scholar
  19. 19.
    Leopold, D.A., O’Toole, A.J., Vetter, T., Blanz, V.: Prototype-referenced shape encoding revealed by high-level aftereffects. Nat. Neurosci. 4, 89–94 (2001) CrossRefGoogle Scholar
  20. 20.
    Levin, D.: Classifying faces by race: The structure of face categories. J. Exp. Psychol. Learn. Mem. Cogn. 22, 1364–1382 (1996) CrossRefGoogle Scholar
  21. 21.
    Light, L., Kayra-Stuart, F., Hollander, S.: Recognition memory for typical and unusual faces. J. Exp. Psychol. Hum. Learn. Mem. 5, 212–228 (1979) CrossRefGoogle Scholar
  22. 22.
    Macmillan, N.A., Creelman, C.D.: Detection Theory: A User’s Guide. Cambridge University Press, Cambridge (1991) Google Scholar
  23. 23.
    Malpass, R.S., Kravitz, J.: Recognition for faces of own and other race faces. J. Pers. Soc. Psychol. 13, 330–334 (1969) CrossRefGoogle Scholar
  24. 24.
    O’Toole, A.J., Roark, D., Abdi, H.: Recognition of moving faces: A psychological and neural perspective. Trends Cogn. Sci. 6, 261–266 (2002) CrossRefGoogle Scholar
  25. 25.
    O’Toole, A.J., Abdi, H., Jiang, F., Phillips, P.J.: Fusing face recognition algorithms and humans. IEEE Trans. Syst. Man Cybern. 37(5), 1149–1155 (2007) CrossRefGoogle Scholar
  26. 26.
    O’Toole, A.J., Phillips, P.J., Jiang, F., Ayyad, J., Pénard, N., Abdi, H.: Face recognition algorithms surpass humans matching faces across changes in illumination. IEEE Trans. Pattern Anal. Mach. Intell. 29, 1642–1646 (2007) CrossRefGoogle Scholar
  27. 27.
    O’Toole, A.J., Phillips, P.J., Narvekar, A.: Humans versus algorithms: Comparisons from the FRVT 2006. In: Eighth International Conference on Automatic Face and Gesture Recognition, 2008 Google Scholar
  28. 28.
    Phillips, P.J., Moon, H., Rizvi, S., Rauss, P.: The FERET evaluation methodology for face-recognition algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 22, 1090–1104 (2000) CrossRefGoogle Scholar
  29. 29.
    Phillips, P.J., Grother, P.J., Micheals, R.J., Blackburn, D.M., Tabassi, E., Bone, J.M.: Face recognition vendor test 2002: Evaluation report. Technical Report NISTIR 6965, National Institute of Standards and Technology (2003).
  30. 30.
    Phillips, P.J., Narvekar, A., Jiang, F., O’Toole, A.J.: An other-race effect for face recognition algorithms. ACM Trans. Appl. Percept. (2010) Google Scholar
  31. 31.
    Phillips, P.J., Scruggs, W.T., O’Toole, A.J., Bowyer, K.W., Schott, C.L., Sharpe, M.: Frvt 2006 and ice 2006 large-scale experimental results. IEEE Trans. Pattern Anal. Mach. Intell. 32(5), 831–846 (2010) CrossRefGoogle Scholar
  32. 32.
    Rhodes, G.: Superportraits: Caricatures and Recognition. The Psychology Press, Hove (1996) CrossRefGoogle Scholar
  33. 33.
    Todorov, A., Mandisodza, A.N., Gren, A., Hall, C.C.: Inferences of competence from faces predict election outcomes. Science 308, 1623–1626 (2005) CrossRefGoogle Scholar
  34. 34.
    Valentine, T.: A unified account of the effects of distinctiveness, inversion, and race in face recognition. Q. J. Exp. Psychol., A Hum. Exp. Psychol. 43, 161–204 (1991) Google Scholar
  35. 35.
    Webster, M.A., MacLin, O.H..: Figural after-effects in the perception of faces. Psychon. Bull. Rev. 6, 647–653 (1999) CrossRefGoogle Scholar
  36. 36.
    Webster, M.A., Kaping, D., Mizokami, Y., Duhamel, P.: Adaptation to natural facial categories. Nature 428, 557–561 (2004) CrossRefGoogle Scholar
  37. 37.
    Young, A.W., Newcombe, F., de Haan, E.H.F., Small, M., Hay, D.C.: Face perception after brain injury: Selective impairments affecting identity and expression. Brain 116(4), 941–959 (1993) CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  1. 1.School of Behavioral and Brain SciencesThe University of Texas at DallasRichardsonUSA

Personalised recommendations