Real–Time Robot Manipulation Using Mouth Gestures in Facial Video Sequences

  • Juan B. Gómez
  • Jorge E. Hernández
  • Flavio Prieto
  • Tanneguy Redarce
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4729)


In this paper a novel method for the automatic command of three degrees of freedom of a robot using mouth gestures is presented. The method uses a set of different pixel–based segmentation algorithms and morphological restrictions in order to extract the mouth area from the frames. A fuzzy inference system is then used in order to produce a small subset of discriminante features for gesture classification. A state machine was designed in order to stabilize the robot command task by using a temporal sliding mean on the detected gestures. Experimental results show that the method is both robust and reliable when operated by different people, and fast enough to keep the detection’s rate in real–time.


Human-machine interface gesture-driven systems lip segmentation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Sackier, J.M., Wang, Y.: Robotically assisted laparoscopic surgery from concept to development. Surgical Endoscopy 8(1), 63–66 (1994)CrossRefGoogle Scholar
  2. 2.
    Murioz, V.F., Thorbeck, C.V., DeGabriel, J.G., Lozano, J.F., Sanchez-Badajoz, E., Garcia-Cerezoand, A., Toscano, R., Jimenez-Garrido, A.: A medical robotic assistant for minimally invasive surgery. In: IEEE Int. Conf. Robotics and Automation, San Francisco, CA, USA, pp. 2901–2906. IEEE Computer Society Press, Los Alamitos (2000)Google Scholar
  3. 3.
    Hong, X., Yao, H., Wan, Y., Chen, R.: A pca based visual dct feature extraction method for lip-reading. In: IIH-MSP, pp. 321–326 (2006)Google Scholar
  4. 4.
    Arsic, I., Vilagut, R., Thiran, J.P.: Automatic extraction of geometric lip features with application to multi-modal speaker identification. In: ICME. IEEE International Conference on Multimedia and Expo, Toronto, ON, Canada, pp. 161–164. IEEE Computer Society Press, Los Alamitos (2006)Google Scholar
  5. 5.
  6. 6.
    Nishikawa, A., Hosoi, T., Koara, K., Negoro, D., Hikita, A., Asano, S., Kakutani, H., Miyazaki, F., Sekimoto, M., Yasui, M., Miyake, Y., Takiguchi, S., Monden, M.: Face mouse: A novel human-machine interface for controlling the position of a laparoscope. IEEE Trans. on Robotics and Automation 19(5), 825–841 (2003)CrossRefGoogle Scholar
  7. 7.
    Hsu, R., Abdel-Mottaleb, M., Jain, A.: Face detection in color images. IEEE Trans. on Pattern Analysis and Machine Intelligence 24(5), 696–706 (2002)CrossRefGoogle Scholar
  8. 8.
    Eveno, N., Caplier, A., Coulon, P.: A new color transformation for lips segmentation. In: IEEE Fourth Workshop on Multimedia Signal Processing, Cannes, France, pp. 3–8. IEEE Computer Society Press, Los Alamitos (2001)CrossRefGoogle Scholar
  9. 9.
    Liévin, M., Delmas, P., Coulon, P., Luthon, F., Fristot, V.: Automatic lip tracking: Bayesian segmentation and active contours in a cooperative scheme. In: ICMCS (1999)Google Scholar
  10. 10.
    Wang, S., Lau, W., Leung, S., Liew, A.: Lip segmentation with the presence of beards. In: IEEE International Conference on Acoustics, Speech and Signal Processing, vol. 3, pp. 529–532. IEEE Computer Society Press, Los Alamitos (2004)Google Scholar
  11. 11.
    Wang, S., Lau, W., Leung, S.: Automatic lip contour extraction from color images. Pattern Recongnition 37(12), 2375–2387 (2004)zbMATHCrossRefGoogle Scholar
  12. 12.
    Hasanuzzaman, M., Zhang, T., Ampornaramveth, V., Ueno, H.: Gesture-based human-robot interaction using a knowledge-based software platform. Industrial Robot: An International Journal 33(1), 37–49 (2006)CrossRefGoogle Scholar
  13. 13.
    Zelinsky, A., Heinzmann, J.: Human-robot interaction using facial gesture recognition. In: Proceedings of the International Workshop on Robot and Human Communication, pp. 256–261 (November 1996)Google Scholar
  14. 14.
    Heinzmann, J.: Real-time human face tracking and gesture recognition. Master’s thesis, Universität Karlsruhe, Fakultät für Informatik (1996)Google Scholar
  15. 15.
    Lewis, L., Powers, D.: Lip feature extraction using red exclusion. In: Trent, W. (ed.) Pan-Sydney Workshop on Visual Information Processing (2001)Google Scholar
  16. 16.
    Gómez, J., Prieto, F., Redarce, T.: Lips movement segmentation and features extraction in real time. In: CISSE 2006. International Joint Conferences on Computer, Information and Systems Sciences and Engineering (2006)Google Scholar
  17. 17.
    Eckert, M.: Compensación de movimiento avanzada para codificación de vídeo. PhD thesis, Universidad Politécnica de Madrid (2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Juan B. Gómez
    • 1
  • Jorge E. Hernández
    • 1
  • Flavio Prieto
    • 1
  • Tanneguy Redarce
    • 2
  1. 1.Universidad Nacional de Colombia Sede Manizales, ManizalesColombia
  2. 2.Laboratoire Ampère, INSA de Lyon, LyonFrance

Personalised recommendations