Multimedia Tools and Applications

, Volume 61, Issue 2, pp 321–352 | Cite as

A data association approach to detect and organize people in personal photo collections

  • Liliana Lo PrestiEmail author
  • Marco Morana
  • Marco La Cascia


In this paper we present a method to automatically segment a photo sequence in groups containing the same persons. Many methods in literature accomplish to this task by adopting clustering techniques. We model the problem as the search for probable associations between faces detected in subsequent photos considering the mutual exclusivity constraint: a person can not be in a photo two times, nor two faces in the same photo can be assigned to the same group. Associations have been found considering face and clothing descriptions. In particular, a two level architecture has been adopted: at the first level, associations are computed within meaningful temporal windows (situations); at the second level, the resulting clusters are re-processed to find associations across situations. Experiments confirm our technique generally outperforms clustering methods. We present an analysis of the results on a public dataset, enabling future comparison, and on private collections.


Digital library Personal photo album Data association Re-identification 



We thank all the anonymous reviewers whose insightful comments led to significant improvements of the manuscript.


  1. 1.
    Ahonen T, Hadid A, Pietikäinen M (2004) Face recognition with local binary patterns. In: Proc. of European Conference on Computer Vision (ECCV 2004), pp 469–481Google Scholar
  2. 2.
    Androutsos D, Plataniotiss K, Venetsanopoulos A (1998) Distance measures for color image retrieval. In: Proc. of International Conference on Image Processing (ICIP 98), vol 2. IEEE, pp 770–774Google Scholar
  3. 3.
    Ardizzone E, Cascia ML, Morana M, Vella F (2009) Clustering techniques for personal photo album management. J Electron Imaging 18(4):043014Google Scholar
  4. 4.
    Choi J, Yang S, Ro Y, Plataniotis K (2008) Face annotation for personal photos using context-assisted face recognition. In: Proc. of international conference on Multimedia Information Retrieval (MIR 2008), pp 44–51Google Scholar
  5. 5.
    Chu WT, Lee YL, Yu JY (2009) Using context information and local feature points in face clustering for consumer photos. In: Proc. of International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2009), pp 1141–1144Google Scholar
  6. 6.
    Davis M, Smith M, Stentiford F, Bamidele A, Canny J, Good N, King S, Janakiraman R (2006) Using context and similarity for face and location identification. In: Proceedings of the IS&T/SPIE 18th annual symposium on electronic imaging science and technology, vol 6061. pp 119–127Google Scholar
  7. 7.
    Edmonds J, Karp R (1972) Theoretical improvements in algorithmic efficiency for network flow problems. JACM 19(2):248–264zbMATHCrossRefGoogle Scholar
  8. 8.
    El-Khoury E, Senac C, Joly P (2010) Face-and-clothing based people clustering in video content. In: Proc. of international conference on Multimedia Information Retrieval (MIR 2010), ACM, pp 1–10Google Scholar
  9. 9.
    Gallagher A, Chen T (2008) Clothing cosegmentation for recognizing people. In: Proc. of Computer Vision and Pattern Recognition (CVPR 2008). IEEEGoogle Scholar
  10. 10.
    Gross J, Yellen J (2006) Graph theory and its applications. CRC pressGoogle Scholar
  11. 11.
    Heyer LJ, Kruglyak S, Yooseph S (1999) Exploring expression data: Identification and analysis of coexpressed genes. Genome Res 9(11):1106–1115. doi: 10.1101/gr.9.11.1106 CrossRefGoogle Scholar
  12. 12.
    Huang GB, Jain V, Learned-Miller E (2007) Unsupervised joint alignment of complex images. In: Proc. of international conference on computer vision (ICCV 2007)Google Scholar
  13. 13.
    Huang T, Russell S (1997) Object identification in a bayesian context. Int Joint Conf on Artificial Intel 15:1276–1283Google Scholar
  14. 14.
  15. 15.
    Jain A, Murty M, Flynn P (1999) Data clustering: a review. ACM Comput Surv (CSUR) 31(3):264–323CrossRefGoogle Scholar
  16. 16.
    Jain R, Sinha P (2010) Content without context is meaningless. In: Proc. of conference on Multimedia (MM 2010)Google Scholar
  17. 17.
    Kakumanu P, Makrogiannis S, Bourbakis N (2007) A survey of skin-color modeling and detection methods. Pattern Recogn 40(3):1106–1122zbMATHCrossRefGoogle Scholar
  18. 18.
    Kang H, Shneiderman B (2000) Visualization methods for personal photo collections: Browsing and searching in the photofinder. In: Proc. of International Conference on Multimedia & Expo (ICME 2000)Google Scholar
  19. 19.
    Kuhn H (1955) The Hungarian method for the assignment problem. Nav Res Logist Q 2(1–2):83–97CrossRefGoogle Scholar
  20. 20.
    Lawless J (1982) Statistical models and methods for lifetime data. Wiley, New YorkzbMATHGoogle Scholar
  21. 21.
    Lo Presti L, Morana M, La Cascia M (2010) A data association algorithm for people re-identification in photo sequences. In: Proc. of IEEE International Symposium on Multimedia (ISM 2010). IEEEGoogle Scholar
  22. 22.
    Ojala T, Pietikäinen M, Mäenpää T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987Google Scholar
  23. 23.
    Phillips P, Wechsler H, Huang J, Rauss P (1998) The FERET database and evaluation procedure for face-recognition algorithms. Image and Vision Computing 16(5):295–306CrossRefGoogle Scholar
  24. 24.
    Phillips P, Moon H, Rizvi S, Rauss P (2002) The FERET evaluation methodology for face-recognition algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(10):1090–1104CrossRefGoogle Scholar
  25. 25.
  26. 26.
    Rubner Y, Tomasi C, Guibas L (1998) A metric for distributions with applications to image databases. Proc of International Conference on Computer Vision, (ICCV 1998)Google Scholar
  27. 27.
    Sivic J, Zitnick C, Szeliski R (2006) Finding people in repeated shots of the same scene. In: Proc. of British Machine Vision Conference (BMVC 2006) 3:909–918Google Scholar
  28. 28.
    Song Y, Leung T (2006) Context-aided human recognition—clustering. In: Computer Vision—ECCV 2006, Lecture Notes in Computer Science, vol 3953. Springer, Berlin, Heidelberg, pp 382–395CrossRefGoogle Scholar
  29. 29.
    Tan X, Triggs B (2007) Enhanced local texture feature sets for face recognition under difficult lighting conditions. In: Proc. of international conference on analysis and modeling of faces and gestures. Springer, pp 168–182Google Scholar
  30. 30.
    Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3(1):71–86CrossRefGoogle Scholar
  31. 31.
    Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154CrossRefGoogle Scholar
  32. 32.
    Zhang L, Chen L, Li M, Zhang H (2003) Automated annotation of human faces in family albums. In: Proc. of Conference on Multimedia (MM 2003), pp 355–358Google Scholar
  33. 33.
    Zhang L, Hu Y, Li M, Ma W, Zhang H (2004) Efficient propagation for face annotation in family albums. In: Proc of Conference on Multimedia (MM 2004), pp 716–723Google Scholar
  34. 34.
    Zhao M, Teo Y, Liu S, Chua TS, Jain R (2006) Automatic person annotation of family photo album. In: Image and video retrieval. Lecture notes in computer science, vol 4071. Springer, Berlin, Heidelberg, pp 163–172CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Liliana Lo Presti
    • 1
    Email author
  • Marco Morana
    • 1
  • Marco La Cascia
    • 1
  1. 1.University of PalermoPalermoItaly

Personalised recommendations