Abstract
We present a robust framework for estimating non-rigid 3D shape and motion in video sequences. Given an input video sequence, and a user-specified region to reconstruct, the algorithm automatically solves for the 3D time-varying shape and motion of the object, and estimates which pixels are outliers, while learning all system parameters, including a PDF over non-rigid deformations. There are no user-tuned parameters (other than initialization); all parameters are learned by maximizing the likelihood of the entire image stream. We apply our method to both rigid and non-rigid shape reconstruction, and demonstrate it in challenging cases of occlusion and variable illumination.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Irani, M.: Multi-Frame Correspondence Estimation Using Subspace Constraints. Int. J. of Comp. Vision 48, 173–194 (2002)
Torresani, L., Yang, D., Alexander, G., Bregler, C.: Tracking and Modeling Non- Rigid Objects with Rank Constraints. In: Proc. CVPR (2001)
Brand, M.: Morphable 3D models from video. In: Proc. CVPR (2001)
Soatto, S., Yezzi, A.J.: DEFORMOTION: Deforming Motion, Shape Averages, and the Joint Registration and Segmentation of Images. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2352, pp. 32–47. Springer, Heidelberg (2002)
Torresani, L., Hertzmann, A., Bregler, C.: Learning Non-Rigid 3D Shape from 2D Motion. In: Proc. NIPS 16 (2003) (to appear)
Jojic, N., Frey, B.: Learning Flexible Sprites in Video Layers. In: Proc. CVPR (2001)
Horn, B.K.P.: Robot Vision. McGraw-Hill, New York (1986)
Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proc. 7th IJCAI (1981)
Irani, M., Anandan, P.: About Direct Methods. In: Triggs, B., Zisserman, A., Szeliski, R. (eds.) ICCV-WS 1999. LNCS, vol. 1883, pp. 267–277. Springer, Heidelberg (2000)
Bregler, C., Hertzmann, A., Biermann, H.: Recovering Non-Rigid 3D Shape from Image Streams. In: Proc. CVPR (2000)
Torresani, L., Bregler, C.: Space-Time Tracking. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2350, pp. 801–812. Springer, Heidelberg (2002)
Forsyth, D.A., Ponce, J.: Computer Vision: A Modern Approach. Prentice-Hall, Englewood Cliffs (2003)
Dellaert, F., Seitz, S.M., Thorpe, C.E., Thrun, S.: EM, MCMC, and Chain Flipping for Structure from Motion with Unknown Correspondence. Machine Learning 50, 45–71 (2003)
Black, M.J., Anandan, P.: The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Computer Vision and Image Understanding 63, 75–104 (1996)
Jepson, A., Black, M.J.: Mixture models for optical flow computation. In: Proc. CVPR, pp. 760–761 (1993)
Jepson, A.D., Fleet, D.J., El-Maraghi, T.F.: Robust Online Appearance Models for Visual Tracking. IEEE Trans. PAMI 25, 1296–1311 (2003)
Wang, J.Y.A., Adelson, E.H.: Representing moving images with layers. IEEE Trans. Image Processing 3, 625–638 (1994)
Weiss, Y., Adelson, E.H.: Perceptually organized EM: A framework for motion segmentation that combines information about form and motion. Technical Report TR 315, MIT Media Lab Perceptual Computing Section (1994)
Jordan, M.I., Ghahramani, Z., Jaakkola, T.S., Saul, L.K.: An introduction to variational methods for graphical models. In: Jordan, M.I. (ed.) Learning in Graphical Models, Kluwer Academic Publishers, Dordrecht (1998)
Morris, D.D., Kanade, T.: A Unified Factorization Algorithm for Points, Line Segments and Planes with Uncertainty Models. In: Proc. ICCV, pp. 696–702 (1998)
Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: A factorization method. Int. J. of Computer Vision 9, 137–154 (1992)
Shi, J., Tomasi, C.: Good Features to Track. In: Proc. CVPR, pp. 593–600 (1994)
Zhang, L., Curless, B., Hertzmann, A., Seitz, S.M.: Shape and Motion under Varying Illumination: Unifying Structure from Motion, Photometric Stereo, and Multi-view Stereo. In: Proc. ICCV, pp. 618–625 (2003)
Gruber, A., Weiss, Y.: Factorization with Uncertainty and Missing Data: Exploiting Temporal Coherence. In: Proc. NIPS 16 (2003) (to appear)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Torresani, L., Hertzmann, A. (2004). Automatic Non-rigid 3D Modeling from Video. In: Pajdla, T., Matas, J. (eds) Computer Vision - ECCV 2004. ECCV 2004. Lecture Notes in Computer Science, vol 3022. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-24671-8_24
Download citation
DOI: https://doi.org/10.1007/978-3-540-24671-8_24
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-21983-5
Online ISBN: 978-3-540-24671-8
eBook Packages: Springer Book Archive