Skip to main content
SpringerLink
Log in

Photo-Realistic Conversation Agent

  • Chapter
Integrated Image and Graphics Technologies

Part of the book series: The International Series in Engineering and Computer Science ((SECS,volume 762))

  • 241 Accesses

Abstract

Image-based facial animation (IBFA) techniques allow us to build photorealistic talking heads. In this chapter, we address two important issues in creating a conversation agent using IBFA. First, we show how to make a conversation agent appear to be alive during the conversation. Second, we present techniques for lip-syncing different languages in the same IBFA system. Using these techniques, we built two conversation agents, an English speaker and a Mandarin Chinese speaker, in our E-Partner system.

We also present a geometry-driven facial expression synthesis system, which is very important to make talking head more naturally in the future. Given the feature point positions (geometry) of a facial expression, our system automatically synthesizes the corresponding expression image which has photo-realistic and natural looking expression details. This technique can be used to enhance the traditional expression mapping technique, or let user edit facial expression directly by moving feature points in our expression editing system.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. N. Badler and S. Platt. “Animating acial expressions”. In Computer Graphics, Annual Conference Series, pp. 245–252. Siggraph, August 1981.

    Google Scholar 

  2. T. Beier and S. Neely. “Feature-based image metamorphosis”. In Computer Graphics, Annual Conference Series, pp. 35–42. Siggraph, July 1992.

    Google Scholar 

  3. M. Brand. “Voice Puppetry”. In Computer Graphics, Annual Conference Series, pp. 22–28. Siggraph, August 1999.

    Google Scholar 

  4. C. Bregler, M. Covell, and M. Slaney. “Video rewrite: driving visual speech with audio”. In Computer Graphics, Annual Conference Series, pp. 353–360. Siggraph, August 1997.

    Google Scholar 

  5. S. E. Brennan. “Caricature generator. M.S. visual studies”. Dept of Architecture, Massachusetts Institute of Technology, Cambridge, MA., September 1982.

    Google Scholar 

  6. E. Cossatto and H. P. Graf. “Photo-realistic talking-heads from image samples”. IEEE Trans. on Multimedia, vol. 2, no. 3, September 2000.

    Google Scholar 

  7. P. Ekman and W. V. Friesen. “Manual for the facial action coding system”. Consulting Psychologists Press, Inc., 1978.

    Google Scholar 

  8. T. Ezzat, G. Geiger, and T. Poggio. “Trainable video realistic speech animation”. In Computer Graphics, Annual Conference Series, pp. 388–398. Siggraph, August 2002.

    Google Scholar 

  9. T. Ezzat and T. Poggio. “Miketalk: a talking facial display based on morphing visemes”. In Proc. Computer Animation, pp. 75–84. June 1998.

    Google Scholar 

  10. B. Guenter, C. Grimm, D. Wood, H Malvar, and F. Pighin. “Making faces”. In Computer Graphics, Annual Conference Series, pp. 55–66. Siggraph, July 1998.

    Google Scholar 

  11. Y. Lee, D. Terzopoulos, and K. Waters. “Realistic modeling for facial animation”. In Computer Graphics, Annual Conference Series, pp. 55–62. Siggraph, August 1995.

    Google Scholar 

  12. S. Z. Li and L. Gu. “Real-time multi-view face detection, tracking, pose estimation, alignment, and recognition”. In IEEE Conf. on Computer Vision and Pattern Recognition Demo Summary, 2001.

    Google Scholar 

  13. P. Litwinowicz and L. Williams. “Animating images with drawings”. In Computer Graphics, Annual Conference Series, pp. 235–242. Siggraph, August 1990.

    Google Scholar 

  14. Z. Liu, Y. Shan, and Z. Zhang. “Expressive expression mapping with ratio images”. In Computer Graphics, Annual Conference Series, pp. 271–276. Siggraph, August 2001.

    Google Scholar 

  15. D. G. Luenberger. “Linear and nonlinear programming”. Addison-Wesley Publishing Company, 1984.

    Google Scholar 

  16. N. Magneneat-Thalmann, N. E. Primeau, and D. Thalmann. “Abstract muscle actions procedures for human face animation”. Visual Computer, vol. 3, no. 5, pp. 290–297, 1988.

    Google Scholar 

  17. J. yong Noh and U. Neumann. “Expression cloning”. In Computer Graphics, Annual Conference Series, pp. 277–288. Siggraph, August 2001.

    Google Scholar 

  18. I. Pandzic, J. Ostermann, and D. Millen. “User evaluation: synthetic talking faces for interactive services”. The Visual Computer, vol. 15, pp. 330–340, 1999.

    Article  Google Scholar 

  19. F. I. Parke and K. Waters. Computer Facial Animation. AKPeters, Wellesley, Massachusetts, 1996.

    Google Scholar 

  20. K. Perlin and A. Goldberg. Improv: “A system for scripting interactive actors in virtual worlds”. In Computer Graphics, Annual Conference Series, pp. 205–216. Siggraph, August 1996.

    Google Scholar 

  21. F. Pighin, J. Hecker, D. Lischinski, R. Szeliski, and D. H. Salesin. “Synthesizing realistic facial expressions from photographs”. In Computer Graphics, Annual Conference Series, pp. 75–84. Siggraph, July 1998.

    Google Scholar 

  22. S. M. Seize and C. R. Dyer. “View morphing”. In Computer Graphics, Annual Conference Series, pp. 21–30. Siggraph, August 1996.

    Google Scholar 

  23. D. Terzopoulos and K. Waters. “Physically-based facial modelling and animation”. Journal of Visualization and Computer Animation, vol. 1, no. 4, pp. 73–80, March, 1990.

    Google Scholar 

  24. S. Toelg and T. Poggio. “Towards an example-based image compression architecture for video-conferencing”. In MIT Technical Report No. 1494, 1994.

    Google Scholar 

  25. Y. Wang. “A robust parser for spoken language understanding”. In Proceedings of Eurospeech’99, 1999.

    Google Scholar 

  26. K. Waters. “A muscle model for animating three-dimensional facial expression”. Computer Graphics, vol. 22, no. 4, pp.17–24, 1987.

    Google Scholar 

  27. L. Williams. “Performace-driven facial animation”. In Computer Graphics, Annual Conference Series, pp. 235–242. Siggraph, August 1990.

    Google Scholar 

  28. Y. Ye. Interior Point Algorithms: Theory and Analysis. John Wiley, 1997.

    Google Scholar 

  29. B. Zhang, C. Hu, Q. Cai, B. Guo, H. Shum. “E-partner: a photo-realistic conversation agent”. In Proc. of Second IEEE Pacific-Rim Conference on Multimedia (PCM), 2001.

    Google Scholar 

  30. Q. Zhang, Z. Liu, B. Guo, and H. Shum. “Geometry-driven photorealistic facial expression synthesis”. In Proc. of ACM Symp. on Computer Animation, San Diego, CA, July, 2003.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. 3/F Sigma Center, Microsoft Research Asia, 49 Zhichun Road, Haidian District, Beijing, 100080, P.R.China

    Bo Zhang & Baining Guo

  2. One Microsoft Way Redmond, Microsoft Research Redmond, WA, 98052-6399, USA

    Zicheng Liu

Authors
  1. Bo Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zicheng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Baining Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Hong Kong Polytechnic University, Hong Kong

    David D. Zhang  & George Baciu  & 

  2. University of Waterloo, Canada

    Mohamed Kamel

Rights and permissions

Reprints and permissions

Copyright information

© 2004 Kluwer Academic Publishers

About this chapter

Cite this chapter

Zhang, B., Liu, Z., Guo, B. (2004). Photo-Realistic Conversation Agent. In: Zhang, D.D., Kamel, M., Baciu, G. (eds) Integrated Image and Graphics Technologies. The International Series in Engineering and Computer Science, vol 762. Springer, Boston, MA. https://doi.org/10.1007/1-4020-7775-0_12

Download citation

  • DOI: https://doi.org/10.1007/1-4020-7775-0_12

  • Publisher Name: Springer, Boston, MA

  • Print ISBN: 978-1-4020-7774-6

  • Online ISBN: 978-1-4020-7775-3

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation