Generalization to Novel Views from a Single Face Image
When only a single image of a face is available, can we generate new images of the face across changes in viewpoint or illumination? The approach presented in this paper acquires its knowledge about possible image changes from other faces and transfers this prior knowledge to a novel face image. In previous work we introduced the concept of linear object classes (Vetter and Poggio, 1997; Vetter, 1997): In an image based approach, a flexible image model of faces was used to synthesize new images of a face when only a single 2D image of that face is available.
In this paper we describe a new general flexible face model which is now “learned” from examples of individual 3D-face data (Cyberware-scans). In an analysis-by-synthesis loop the flexible 3D model is matched to the novel face image. Variation of the model parameters, similar to multidimensional morphing, allows for generating new images of the face where viewpoint, illumination or even the expression is changed.
The key problem for generating a flexible face model is the computation of dense correspondence between all given example faces. A new correspondence algorithm is described, which is a generalization of existing algorithms for optic flow computation to 3D-face data.
KeywordsOptical Flow Face Image View Synthesis Shape Vector Optic Flow Computation
Unable to display preview. Download preview PDF.
- Barron, J., Fleet, D., and Beauchemin, S. (1994). Performance of optical flow techniques. Int. Journal of Computer Vision, pages 43–77.Google Scholar
- Bergen, J., Anandan, P., Hanna, K., and Hingorani, R. (1992). Hierarchical model-based motion estimation. In Proceedings of the European Conference on Computer Vision, pages 237–252, Santa Margherita Ligure, Italy.Google Scholar
- Bergen, J. and Hingorani, R. (1990). Hierarchical motion-based frame rate conversion. Technical report, David Sarnoff Research Center Princeton NJ 08540.Google Scholar
- Beymer, D., Shashua, A., and Poggio, T. (1993). Example-based image analysis and synthesis. A.L Memo No. 1431, Artificial Intelligence Laboratory, Massachusetts Institute of Technology.Google Scholar
- Choi, C, Okazaki, T., Harashima, H., and Takebe, T. (1991). A system of analyzing and synthesizing facial images. In Proc. IEEE Int. Symposium of Circuit and Syatems (ISCAS91), pages 2665–2668.Google Scholar
- Grenander, U. (1978). Pattern Analysis, Lectures in Pattern Theory. Springer, New York, 1 edition.Google Scholar
- Herpers, R., Michaelis, M., Lichtenauer, K. H., and Sommer, G. (1996). Edge and keypoint detection in facial regions. In Proc. International Conference on Automatic Face and Gesture Recognition, pages 22–27, Killington, VT.Google Scholar
- Jones, M. and Poggio, T. (1996). Model-based matching by linear combination of prototypes. A.i. memo no., Artificial Intelligence Laboratory, Massachusetts Institute of Technology.Google Scholar
- Kosslyn, S. M. (1994). Image and Brain. MIT Press, Cambridge, MA.Google Scholar
- Lanitis, A., Taylor, C., Gootes, T., and Ahmad, T. (1995). Automatic interpretation of human faces and hand gestures using flexible models. In M. Bichsel, editor, Proc. International Workshop on Face and Gesture Recognition, pages 98–103, Zurich, Switzerland.Google Scholar
- Marr, D. (1982). Vision,. W. H. Freeman, San Fancisco.Google Scholar
- Mumford, D. (1996). Pattern theory: A unifying perspective. In Knill, D. and Richards, W., editors, Perception as Bayesian Inference. Cambridge University Press.Google Scholar
- O’Toole, A., Vetter, T., Volz, H., and Salter, E. (1997). Three-dimensional caricatures of human heads: distinctiveness and the perception of facial age. Perception, in press.Google Scholar
- Parke, F. (1974). A parametric model of human faces. Doctoral thesis, University of Utah, Salt Lake City.Google Scholar
- Poggio, T. and Vetter, T. (1992). Recognition and structure from one 2D model view: observations on prototypes, object classes, and symmetries. A.I. Memo No. 1347, Artificial Intelligence Laboratory, Massachusetts Institute of Technology.Google Scholar
- Vetter, T. (1997). Synthestis of novel views from a single face image. International Journal of Computer Vision, (in press).Google Scholar
- Vetter, T., Jones, M. J., and Poggio, T. (1997). A bootstrapping algorithm for learning linear models of object classes. In IEEE Conference on Computer Vision and Pattern Recognition - CVPR’97, Puerto Rico, USA. IEEE Computer Society Press.Google Scholar
- Viola, P. (1995). Alignment by maximization of mutual information. A.I. Memo No. 1548, Artificial Intelligence Laboratory, Massachusetts Institute of Technology.Google Scholar