Facial Landmark Localization and Feature Extraction for Therapeutic Face Exercise Classification

  • Cornelia LanzEmail author
  • Birant Sibel Olgay
  • Joachim Denzler
  • Horst-Michael Gross
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 458)


In this work, we examine landmark localization and feature extraction approaches for the unexplored topic of therapeutic facial exercise recognition. Our goal is to automatically discriminate nine therapeutic exercises that have been determined in cooperation with speech therapists. We use colour, 2.5D and 3D image data that was recorded using Microsoft’s Kinect. Our features comprise statistical descriptors of the face surface curvature as well as characteristic profiles that are derived from face landmarks. For the nine facial exercises, we yield an average recognition accuracy of about \(91\,\%\) in conjunction with manually labeled landmarks. Additionally, we introduce a combined method for automatic landmark localization and compare the results to landmark positions obtained from Active Appearance Model fitting as well as manual labeling. The combined localization method exhibits increased robustness in comparison to AAMs.


Facial expressions Curvature analysis Point signatures Line profiles Therapeutic exercises 



We would like to thank the m&i Fachklinik Bad Liebenstein (in particular Prof. Dr. med. Gustav Pfeiffer, Eva Schillikowski) and Logopädische Praxis Irina Stangenberger, who supported our work by giving valuable insights into rehabilitation and speech-language therapy requirements and praxis. This work is partially funded by the TMBWK ProExzellenz initiative, Graduate School on Image Processing and Image Interpretation.


  1. 1.
    Lanz, C., Denzler, J., Gross, H.M.: Facial movement dysfunctions: conceptual design of a therapy-accompanying training system. In: Wichert, R., Klausing, H. (eds.) Ambient Assisted Living - Advanced Technologies and Societal Change. Springer, Heidelberg (2013)Google Scholar
  2. 2.
    Grosse, M., Schaffer, M., Harendt, B., Kowarschik, R.: Fast data acquisition for three-dimensional shape measurement using fixed-pattern projection and temporal coding. Opt. Eng. 50, 100503 (2011)CrossRefGoogle Scholar
  3. 3.
    Cootes, T., Edwards, G., Taylor, C.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23, 681–685 (2001)CrossRefGoogle Scholar
  4. 4.
    Nakamura, K., Toda, N., Sakamaki, K., Kashima, K., Takeda, N.: Biofeedback rehabilitation for prevention of synkinesis after facial palsy. Otolaryngol. Head Neck Surg. 128, 539–543 (2003)CrossRefGoogle Scholar
  5. 5.
    Gebhard, A., Paulus, D., Suchy, B., Wolf, S.: A system for diagnosis support of patients with facialis paresis. Kl 3/2000, 40–42 (2000)Google Scholar
  6. 6.
    Colombo, A., Cusano, C., Schettini, R.: 3d face detection using curvature analysis. Pattern Recogn. 39, 444–455 (2006)CrossRefzbMATHGoogle Scholar
  7. 7.
    Wang, J., Yin, L., Wei, X., Sun, Y.: 3d facial expression recognition based on primitive surface feature distribution. In: International Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 1399–1406 (2006)Google Scholar
  8. 8.
    Chua, C.S., Jarvis, R.: Point signature: a new representation for 3d object recognition. Int. J. Comput. Vis. 25, 63–85 (1997)CrossRefGoogle Scholar
  9. 9.
    Chua, C.S., Han, F., Ho, Y.K.: 3d human face recognition using point signature. In: Proceedings of the 4th International Automatic Face and Gesture Recognition Conference, pp. 233–238 (2000)Google Scholar
  10. 10.
    Wang, Y., Chua, C.S., Ho, Y.K.: Facial feature detection and face recognition from 2d and 3d images. Pattern Recogn. Lett. 23, 1191–1202 (2002)CrossRefzbMATHGoogle Scholar
  11. 11.
    Besl, P., Jain, R.: Invariant surface characteristics for 3d object recognition in range images. Comput. Vis. Graph. Image Process. 33, 33–80 (1986)CrossRefzbMATHGoogle Scholar
  12. 12.
    Salomon, D.: Data Compression: The Complete Reference. Springer, New York (2004)Google Scholar
  13. 13.
    Martin, C., Werner, U., Gross, H.M.: A real-time facial expression recognition system based on active appearance models using gray images and edge images. In: International Conference on Automatic Face and Gesture Recognition (2008)Google Scholar
  14. 14.
    Viola, P., Jones, M.: Robust real-time face detection. Int. J. Comput. Vis. 57, 137–154 (2004)CrossRefGoogle Scholar
  15. 15.
    Haase, D., Denzler, J.: Anatomical landmark tracking for the analysis of animal locomotion in x-ray videos using active appearance models. In: Heyden, A., Kahl, F. (eds.) SCIA 2011. LNCS, vol. 6688, pp. 604–615. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  16. 16.
    Zhu, X., Ramanan, D.: Face detection, pose estimation and landmark localization in the wild. In: International Conference for Computer Vision and Pattern Recognition, pp. 2879–2886 (2012)Google Scholar
  17. 17.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000)zbMATHGoogle Scholar
  18. 18.
    Khoshelham, K.: Accuracy analysis of kinect depth data. In: ISPRS Workshop Laser Scanning, vol. 38 (2011)Google Scholar
  19. 19.
    Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2, 27:1–27:27 (2011).
  20. 20.
    Hsu, C., Chang, C., Lin, C.: A practical guide to support vector classification. TR available at (2009)
  21. 21.
    Webb, A., Copsey, K., Cawley, G.: Statistical Pattern Recognition. Wiley, New York (2011)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Cornelia Lanz
    • 1
    Email author
  • Birant Sibel Olgay
    • 1
  • Joachim Denzler
    • 2
  • Horst-Michael Gross
    • 1
  1. 1.Neuroinformatics and Cognitive Robotics LabIlmenau University of TechnologyIlmenauGermany
  2. 2.Computer Vision GroupFriedrich Schiller University JenaJenaGermany

Personalised recommendations