Skip to main content

Advertisement

SpringerLink
Log in

Facial Similarity Learning with Humans in the Loop

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

Similarity learning has always been a popular topic in computer vision research. Among this, facial similarity is especially important and difficult due to its wide applications and the nonrigid nature of human faces. The large gap between feature representations and human perceptual descriptions makes the problem even harder. In this paper, we learn facial similarity through human-computer interactions. To learn perceptual similarities of faces in a gallery set, we ask users to label some candidate images with their similarities to a probe image. Based on users’ responses, a sampling algorithm actively generates a probe image and a set of candidates for the next query. Assisted with human efforts, the algorithm embeds all the images into a space where the distance between two subjects conforms to their dissimilarity in human perception. We apply the learned embedding to face retrieval and compare our method with some feature-based methods on a dataset we collect from social network sites (SNS). Experimental results demonstrate that incorporating human efforts can ensure retrieval accuracy. At the same time, the active sampling algorithm reduces human efforts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Yuen P C, Man C. Human face image searching system using sketches. IEEE Trans. Systems, Man and Cybernetics, Part A: Systems and Humans, 2007, 37(4): 493-504.

    Article  Google Scholar 

  2. Kumar N, Berg A C, Belhumeur P N, Nayar S K. Attribute and simile classifiers for face verification. In Proc. the 12th IEEE International Conference on Computer Vision (ICCV), Sept. 29-Oct. 2, 2009, pp.365-372.

  3. Jain A K, Vailaya A. Image retrieval using color and shape. Pattern Recognition, 1996, 29(8): 1233-1244.

    Article  Google Scholar 

  4. Ahonen T, Hadid A, Pietikäinen M. Face description with local binary patterns: Application to face recognition. IEEE Trans. Pattern Analysis and Machine Intelligence, 2006, 28(12): 2037-2041.

    Article  Google Scholar 

  5. Wiskott L, Fellous J M, Krüger N et al. Face recognition by elastic bunch graph matching. IEEE Trans. Pattern Analysis and Machine Intelligence, 1997, 19(7): 775-779.

    Article  Google Scholar 

  6. Berg T, Belhumeur P N. Tom-vs-pete classifiers and identity-preserving alignment for face verification. In Proc. the British Machine Vision Conference, September 2012.

  7. Chopra S, Hadsell R, LeCun Y. Learning a similarity metric discriminatively, with application to face verification. In Proc. the IEEE Conf. Computer Vision and Pattern Recognition (CVPR), June 2005, pp.539-546.

  8. Schroff F, Treibitz T, Kriegman D, Belongie S. Pose, illumination and expression invariant pairwise face-similarity measure via doppelgänger list comparison. In Proc. the 13th IEEE ICCV, Nov. 2011, pp.2494-2501.

  9. Yin Q, Tang X, Sun J. An associate-predict model for face recognition. In Proc. the IEEE CVPR, June 2011, pp.497-504.

  10. Scheirer W J, Kumar N, Belhumeur P N, Boult T E. Multi-attribute spaces: Calibration for attribute fusion and similarity search. In Proc. the IEEE CVPR, June 2012, pp.2933-2940.

  11. Siddiquie B, Feris R S, Davis L S. Image ranking and retrieval based on multi-attribute queries. In Proc. the IEEE CVPR, June 2011, pp.801-808.

  12. Parikh D, Grauman K. Relative attributes. In Proc. the 13th IEEE Int. Conf. Computer Vision, Nov. 2011, pp.503-510.

  13. Biswas A, Parikh D. Simultaneous active learning of classifiers & attributes via relative feedback. In Proc. the IEEE CVPR, June 2013, pp.644-651.

  14. Liang L, Grauman K. Beyond comparing image pairs: Set-wise active learning for relative attributes. In Proc. the IEEE CVPR, June 2014, pp.208-215.

  15. Agarwal S, Wills J, Cayton L et al. Generalized non-metric multidimensional scaling. In Proc. the 11th Int. Conf. Artificial Intelligence and Statistics, March 2007, pp.11-18.

  16. Tamuz O, Liu C, Belongie S, Shamir O, Kalai A T. Adaptively learning the crowd kernel. arXiv:1105.1033, 2011. http://arxiv.org/abs/1105.1033, March 2015.

  17. van der Maaten L, Weinberger K. Stochastic triplet embedding. In Proc. IEEE International Workshop on Machine Learning for Signal Processing, September 2012.

  18. Garces E, Agarwala A, Gutierrez D, Hertzmann A. A similarity measure for illustration style. ACM Transactions on Graphics, 2014, 33(4): 93:1-93:9.

    Article  Google Scholar 

  19. Deng J, Krause J, Li F F. Fine-grained crowdsourcing for fine-grained recognition. In Proc. IEEE CVPR, June 2013, pp.580-587.

  20. Holub A, Liu Y H, Perona P. On constructing facial similarity maps. In Proc. IEEE CVPR, June 2007.

  21. Kruskal J B. Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika, 1964, 29(1): 1-27.

    Article  MATH  MathSciNet  Google Scholar 

  22. Fang Y, Geman D. Experiments in mental face retrieval. In Lecture Notes in Computer Science Volume 3546, Kanade T, Jain A, Ratha N K (eds.), Springer, 2005, pp.637-646.

  23. Ruiz-del-Solar J, Navarrete P. FACERET: An interactive face retrieval system based on self-organizing maps. In Proc. Int. Conf. Image and Video Retrieval, July 2002, pp.157-164.

  24. Yang Z, Laaksonen J. Partial relevance in interactive facial image retrieval. In Proc. the 3rd Int. Conf. Advances in Pattern Recognition, Part II, August 2005, pp.216-225.

  25. Cao C, Kwak S, Belongie S, Kriegman D, Ai H. Adaptive ranking of facial attractiveness. In Proc. the IEEE International Conference on Multimedia and Expo, July 2014.

  26. Huang G B, Ramesh M, Berg T, Learned-Miller E. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical Report 07-49, University of Massachusetts, Amherst, Oct. 2007.

  27. Huang C, Ai H, Li Y, Lao S. High-performance rotation invariant multiview face detection. IEEE Trans. Pattern Analysis and Machine Intelligence, 2007, 29(4): 671-686.

    Article  Google Scholar 

  28. Yan D, Huang L, Jordan M I. Fast approximate spectral clustering. In Proc. the 15th ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, June 2009, pp.907-916.

Download references

Author information

Authors and Affiliations

  1. Tsinghua National Laboratory for Information Science and Technology, Department of Computer Science and Technology Tsinghua University, Beijing, 100084, China

    Chong Cao & Hai-Zhou Ai

Authors
  1. Chong Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hai-Zhou Ai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chong Cao.

Additional information

This work was partly supported by the National Basic Research 973 Program of China under Grant No. 2011CB302203.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cao, C., Ai, HZ. Facial Similarity Learning with Humans in the Loop. J. Comput. Sci. Technol. 30, 499–510 (2015). https://doi.org/10.1007/s11390-015-1540-3

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-015-1540-3

Keywords

Search

Navigation