Being John Malkovich

  • Ira Kemelmacher-Shlizerman
  • Aditya Sankar
  • Eli Shechtman
  • Steven M. Seitz
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6311)


Given a photo of person A, we seek a photo of person B with similar pose and expression. Solving this problem enables a form of puppetry, in which one person appears to control the face of another. When deployed on a webcam-equipped computer, our approach enables a user to control another person’s face in real-time. This image-retrieval-inspired approach employs a fully-automated pipeline of face analysis techniques, and is extremely general—we can puppet anyone directly from their photo collection or videos in which they appear. We show several examples using images and videos of celebrities from the Internet.


Image Retrieval Local Binary Pattern Target Image Target Face Mouth Region 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Pighin, F., Hecker, J., Lischinski, D., Szeliski, R., Salesin, D.H.: Synthesizing realistic facial expressions from photographs. In: SIGGRAPH, pp. 75–84 (1998)Google Scholar
  2. 2.
    Zhang, Z., Liu, Z., Adler, D., Cohen, M.F., Hanson, E., Shan, Y.: Robust and rapid generation of animated faces from video images: A model-based modeling approach. Int. J. Comput. Vision 58, 93–119 (2004)CrossRefGoogle Scholar
  3. 3.
    Vlasic, D., Brand, M., Pfister, H., Popović, J.: Face transfer with multilinear models. ACM Trans. Graph. 24, 426–433 (2005)CrossRefGoogle Scholar
  4. 4.
    Weise, T., Li, H., Gool, L.V., Pauly, M.: Face/off: Live facial puppetry. In: Symposium on Computer Animation (2009)Google Scholar
  5. 5.
    Kumar, N., Belhumeur, P., Nayar, S.: Facetracer: A search engine for large collections of images with faces. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part IV. LNCS, vol. 5305, pp. 340–353. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  6. 6.
    Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., Nayar, S.K.: Face swapping: automatically replacing faces in photographs. In: SIGGRAPH, pp. 1–8 (2008)Google Scholar
  7. 7.
    Goldman, D.B., Gonterman, C., Curless, B., Salesin, D., Seitz, S.M.: Video object annotation, navigation, and composition. In: UIST, pp. 3–12 (2008)Google Scholar
  8. 8.
    Saragih, J.M., Lucey, S., Cohn, J.: Face alignment through subspace constrained mean-shifts. In: ICCV (2009)Google Scholar
  9. 9.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: CVPR, pp. 511–518 (2001)Google Scholar
  10. 10.
    Everingham, M., Sivic, J., Zisserman, A.: Hello! My name is. Buffy – automatic naming of characters in TV video. In: BMVC (2006)Google Scholar
  11. 11.
    Zhang, L., Snavely, N., Curless, B., Seitz, S.M.: Spacetime faces: high resolution capture for modeling and animation. In: SIGGRAPH, pp. 548–558 (2004)Google Scholar
  12. 12.
    Ojala, T., Pietikinen, M., Menp, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. PAMI 24, 971–987 (2002)Google Scholar
  13. 13.
    Ahonen, T., Hadid, A., Pietikinen, M.: Face description with local binary patterns: Application to face recognition. IEEE Trans. PAMI 28, 2037–2041 (2006)Google Scholar
  14. 14.
    Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical Report 07-49, University of Massachusetts, Amherst (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Ira Kemelmacher-Shlizerman
    • 1
  • Aditya Sankar
    • 1
  • Eli Shechtman
    • 2
  • Steven M. Seitz
    • 1
    • 3
  1. 1.University of Washington 
  2. 2.Adobe Systems 
  3. 3.Google Inc 

Personalised recommendations