A Study on 2D Photo-Realistic Facial Animation Generation Using 3D Facial Feature Points and Deep Neural Networks

  • Kazuki SatoEmail author
  • Takashi Nose
  • Akira Ito
  • Yuya Chiba
  • Akinori Ito
  • Takahiro Shinozaki
Conference paper
Part of the Smart Innovation, Systems and Technologies book series (SIST, volume 82)


This paper proposes a technique for generating a 2D photo-realistic facial animation from an input text. The technique is based on the mapping from 3D facial feature points with deep neural networks (DNNs). Our previous approach was based only on a 2D space using hidden Markov models (HMMs) and DNNs. However, this approach has a disadvantage that generated 2D facial pixels are sensitive to the rotation of the face in the training data. In this study, we alleviate the problem using 3D facial feature points obtained by Kinect. The information of the face shape and color is parameterized by the 3D facial feature points. The relation between the labels from texts and face-model parameters are modeled by DNNs in the model training. As a preliminary experiment, we show that the proposed technique can generate the 2D facial animation from arbitrary input texts.


Photo-realistic facial animation Face image synthesis Deep neural network Kinect 



Part of this work was supported by JSPS KAKENHI Grant Number JP15H02720 and JP26280055.


  1. 1.
    Anderson, R., Stenger, B., Wan, V., Cipolla, R.: Expressive visual text-to-speech using active appearance models. In: Proceedings of the Computer Vision and Pattern Recognition (CVPR), pp. 3382–3389 (2013)Google Scholar
  2. 2.
    Besl, P.J., McKay, N.D.: Method for registration of 3-D shapes. In: Robotics-DL tentative, pp. 586–606. International Society for Optics and Photonics (1992)Google Scholar
  3. 3.
    Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001)CrossRefGoogle Scholar
  4. 4.
    Gales, M.J.: Cluster adaptive training of hidden Markov models. IEEE Trans. Speech Audio Process. 8(4), 417–428 (2000)CrossRefGoogle Scholar
  5. 5.
    Kinect for Windows SDK 2.0 Programming Guide: High definition face tracking.
  6. 6.
    Kurematsu, A., Takeda, K., Sagisaka, Y., Katagiri, S., Kuwabara, H., Shikano, K.: ATR Japanese speech database as a tool of speech recognition and synthesis. Speech Commun. 9(4), 357–363 (1990)CrossRefGoogle Scholar
  7. 7.
    Nose, T., Tachibana, M., Kobayashi, T.: HMM-based style control for expressive speech synthesis with arbitrary speaker’s voice using model adaptation. IEICE Trans. Inf. Syst. E92–D(3), 489–497 (2009)CrossRefGoogle Scholar
  8. 8.
    Nose, T., Yamagishi, J., Masuko, T., Kobayashi, T.: A style control technique for HMM-based expressive speech synthesis. IEICE Trans. Inf. Syst. E90–D(9), 1406–1413 (2007)CrossRefGoogle Scholar
  9. 9.
    Nose, T.: Efficient implementation of global variance compensation for parametric speech synthesis. IEEE/ACM Trans. Audio Speech Lang. Process. 24(10), 1694–1704 (2016)CrossRefGoogle Scholar
  10. 10.
    Sako, S., Tokuda, K., Masuko, T., Kobayashi, T., Kitamura, T.: HMM-based text-to-audio-visual speech synthesis. In: Proceedings of the INTERSPEECH, pp. 25–28 (2000)Google Scholar
  11. 11.
    Sato, K., Nose, T., Ito, A.: Synthesis of photo-realistic facial animation from text based on HMM and DNN with animation unit. In: Proceeding of the Twelfth International Conference on Intelligent Information Hiding and Multimedia Signal Processing (IIH-MSP), pp. 29–36 (2017)Google Scholar
  12. 12.
    Zen, H., Senior, A., Schuster, M.: Statistical parametric speech synthesis using deep neural networks. In: Proceedings of the ICASSP, pp. 7962–7966 (2013)Google Scholar
  13. 13.
    Zhang, Z.: Microsoft kinect sensor and its effect. IEEE Multimedia 19(2), 4–10 (2012)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • Kazuki Sato
    • 1
    Email author
  • Takashi Nose
    • 1
  • Akira Ito
    • 1
  • Yuya Chiba
    • 1
  • Akinori Ito
    • 1
  • Takahiro Shinozaki
    • 2
  1. 1.Graduate School of EngineeringTohoku UniversitySendaiJapan
  2. 2.Interdisciplinary Graduate School of Science and EngineeringTokyo Institute of TechnologyYokohamaJapan

Personalised recommendations