Skip to main content
SpringerLink
Log in
Book cover

Asian Conference on Computer Vision

ACCV 2014: Computer Vision - ACCV 2014 Workshops pp 157–173Cite as

Gesture Recognition Performance Score: A New Metric to Evaluate Gesture Recognition Systems

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 9008))

Abstract

In spite of many choices available for gesture recognition algorithms, the selection of a proper algorithm for a specific application remains a difficult task. The available algorithms have different strengths and weaknesses making the matching between algorithms and applications complex. Accurate evaluation of the performance of a gesture recognition algorithm is a cumbersome task. Performance evaluation by recognition accuracy alone is not sufficient to predict its successful real-world implementation. We developed a novel Gesture Recognition Performance Score (\(GRPS\)) for ranking gesture recognition algorithms, and to predict the success of these algorithms in real-world scenarios. The \(GRPS\) is calculated by considering different attributes of the algorithm, the evaluation methodology adopted, and the quality of dataset used for testing. The \(GRPS\) calculation is illustrated and applied on a set of vision based hand/ arm gesture recognition algorithms reported in the last 15 years. Based on \(GRPS\) a ranking of hand gesture recognition algorithms is provided. The paper also presents an evaluation metric namely Gesture Dataset Score (\(GDS\)) to quantify the quality of gesture databases. The \(GRPS\) calculator and results are made publicly available (http://software.ihpc.a-star.edu.sg/grps/).

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    We are in the process of publishing a detailed survey on the topic of gesture recognition.

  2. 2.

    Considering larger number of classes will not increase the score much. This is reasonable as the number of gestures used in interaction applications is limited.

  3. 3.

    The complexity due to the presence of other objects is considered in background index. The complexity due to the presence of other human (which is more challenging due to skin colored backgrounds) is considered in noise index.

  4. 4.

    The list will be maintained and updated regularly. The portal provides authors of research papers a provision to submit their GRPS score and paper details to be included in the ranking list.

References

  1. Pisharady, P.K., Vadakkepat, P., Loh, A.P.: Attention based detection and recognition of hand postures against complex backgrounds. Int. J. Comput. Vision 101, 403–419 (2013)

    Article  Google Scholar 

  2. Dipietro, L., Sabatini, A.M., Dario, P.: A survey of glove-based systems and their applications. IEEE Trans. Syst. Man Cybern. Part C: Appl. Rev. 38, 461–482 (2008)

    Article  Google Scholar 

  3. Mitra, S., Acharya, T.: Gesture recognition: A survey. IEEE Trans. Syst. Man Cybern. Part C: Appl. Rev. 37, 311–324 (2007)

    Article  Google Scholar 

  4. Erol, A., Bebis, G., Nicolescu, M., Boyle, R.D., Twombly, X.: Vision-based hand pose estimation: A review. Comput. Vis. Image Underst. 108, 52–73 (2007)

    Article  Google Scholar 

  5. Ong, S.C.W., Ranganath, S.: Automatic sign language analysis: A survey and the future beyond lexical meaning. IEEE Trans. Pattern Anal. Mach. Intell. 27, 873–891 (2005)

    Article  Google Scholar 

  6. Thacker, N.A., Clark, A.F., Barron, J.L., Beveridge, J.R., Courtney, P., Crum, W.R., Ramesh, V., Clark, C.: Performance characterization in computer vision: A guide to best practices. Comput. Vis. Image Underst. 109, 305–334 (2008)

    Article  Google Scholar 

  7. Ward, J.A., Lukowicz, P., Gellersen, H.W.: Performance metrics for activity recognition. ACM Trans. Intell. Syst. Technol. 02, 6:01–6:23 (2011)

    Google Scholar 

  8. Lichtenauer, J.F., Hendriks, E.A., Reinders, M.J.T.: Sign language recognition by combining statistical dtw and independent classification. IEEE Trans. Pattern Anal. Mach. Intell. 30 (2008)

    Google Scholar 

  9. Teng, X., Wu, B., Yu, W., Liu, C.: A hand gesture recognition system based on local linear embedding. J. Vis. Lang. Comput. 16, 442–454 (2005)

    Article  Google Scholar 

  10. Lui, Y.M.: A least squares regression framework on manifolds and its application to gesture recognition. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 13–18 (2012)

    Google Scholar 

  11. Keskin, C., Kirac, F., Kara, Y., Akarun, L.: Randomized decision forests for static and dynamic hand shape classification. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 31–46 (2012)

    Google Scholar 

  12. Yang, M.H., Ahuja, N., Tabb, M.: Extraction of 2d motion trajectories and its application to hand gesture recognition. IEEE Trans. Pattern Anal. Mach. Intell. 24, 1061–1074 (2002)

    Article  Google Scholar 

  13. Triesch, J., Malsburg, C.: A system for person-independent hand posture recognition against complex backgrounds. IEEE Trans. Pattern Anal. Mach. Intell. 23, 1449–1453 (2001)

    Article  Google Scholar 

  14. Yoon, H.S., Soh, J., Bae, Y.J., Yang, H.S.: Hand gesture recognition using combined features of location, angle, and velocity. Pattern Recogn. 34, 1491–1501 (2001)

    Article  MATH  Google Scholar 

  15. Huang, D.Y., Hu, W.C., Chang, S.H.: Gabor filter-based hand-pose angle estimation for hand gesture recognition under varying illumination. Expert Syst. Appl. 38, 6031–6042 (2011)

    Article  Google Scholar 

  16. Chen, F.S., Fu, C.M., Huang, C.L.: Hand gesture recognition using a real-time tracking method and hidden markov models. Image Vis. Comput. 21, 745–758 (2003)

    Article  Google Scholar 

  17. Zhou, R., Junsong, Y., Zhengyou, Z.: Robust hand gesture recognition based on finger-earth movers distance with a commodity depth camera. In: Proceedings of ACM Multimeida (2011)

    Google Scholar 

  18. Ramamoorthy, A., Vaswani, N., Chaudhury, S., Banerjee, S.: Recognition of dynamic hand gestures. Pattern Recogn. 36, 2069–2081 (2003)

    Article  MATH  Google Scholar 

  19. Licsar, A., Sziranyi, T.: User-adaptive hand gesture recognition system with interactive training. Image Vis. Comput. 23, 1102–1114 (2005)

    Article  Google Scholar 

  20. Lai, K., Konrad, J., Ishwar, P.: A gesture-driven computer interface using kinect. In: IEEE Southwest Symposium on Image Analysis and Interpretation (SSIAI), pp. 185–188 (2012)

    Google Scholar 

  21. Van den Bergh, M., Carton, D., De Nijs, R., Mitsou, N., Landsiedel, C., Kuehnlenz, K., Wollherr, D., Van Gool, L., Buss, M.: Real-time 3d hand gesture interaction with a robot for understanding directions from humans. In: IEEE International Symposium on Robot and Human Interactive Communication (IEEE RO-MAN) (2011)

    Google Scholar 

  22. Just, A., Marcel, S.: A comparative study of two state-of-the-art sequence processing techniques for hand gesture recognition. Comput. Vis. Image Underst. 113, 532–543 (2009)

    Article  Google Scholar 

  23. Frolova, D., Stern, H., Berman, S.: Most probable longest common subsequence for recognition of gesture character input. IEEE Trans. Cybern. 43, 871–880 (2013)

    Article  Google Scholar 

  24. Ge, S.S., Yang, Y., Lee, T.H.: Hand gesture recognition and tracking based on distributed locally linear embedding. Image Vis. Comput. 26, 1607–1620 (2008)

    Article  Google Scholar 

  25. Shin, M.C., Tsap, L.V., Goldgof, D.B.: Gesture recognition using bezier curves for visualization navigation from registered 3-d data. Pattern Recogn. 37, 1011–1024 (2004)

    Article  Google Scholar 

  26. Zhao, M., Quek, F.K.H., Wu, X.: Rievl: Recursive induction learning in hand gesture recognition. IEEE Trans. Pattern Anal. Mach. Intell. 20, 1174–1185 (1998)

    Article  Google Scholar 

  27. Guyon, I., Athitsos, V., Jangyodsuk, P., Hamner, B., Escalante, H.: Chalearn gesture challenge: Design and first results. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1–6 (2012)

    Google Scholar 

  28. Malgireddy, M.R., Inwogu, I., Govindaraju, V.: A temporal bayesian model for classifying, detecting and localizing activities in video sequences. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 43–48 (2012)

    Google Scholar 

  29. Di, W., Fan, Z., Ling, S.: One shot learning gesture recognition from rgbd images. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2012)

    Google Scholar 

  30. Mahbub, U., Imtiaz, H., Roy, T., Rahman, M., Ahad, M.: A template matching approach of one-shot-learning gesture recognition. Pattern Recogn. Lett. (2012). http://dx.doi.org/10.1016/j.bbr.2011.03.031

  31. Simon, F., Helena, M.M., Pushmeet, K., Sebastian, N.: Instructing people for training gestural interactive systems. In: International Conference on Human Factors in Computing Systems, CHI, pp. 1737–1746. ACM (2012)

    Google Scholar 

  32. Escalera, S., Gonzlez, J., Bar, X., Reyes, M., Lopes, O., Guyon, I., Athistos, V., Escalante, H.: Multi-modal gesture recognition challenge 2013: Dataset and results. In: Proceedings of the 15th ACM International Conference on Multimodal Interaction (ICMI), Sydney, Australia (2013)

    Google Scholar 

  33. Chen, M., AlRegib, G., Juang, B.H.: 6dmg: A new 6d motion gesture database. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2011)

    Google Scholar 

  34. Just, A., Bernier, O., Marcel, S.: Hmm and iohmm for the recognition of mono- and bi-manual 3d hand gestures. In: Proceedings of the British Machine Vision Conference (BMVC) (2004)

    Google Scholar 

  35. Song, Y., Demirdjian, D., Davis, R.: Tracking body and hands for gesture recognition: Natops aircraft handling signals database. In: Proceedings of the 9th IEEE Conference on Automatic Face and Gesture Recognition (FG 2011), Santa Barbara, CA, pp. 500–506 (2011)

    Google Scholar 

  36. Triesch, J., Malsburg, C.: Robust classification of hand postures against complex backgrounds. In: Proceedings of the Second International Conference on Automatic Face and Gesture Recognition, Killington, VT, USA, pp. 170–175 (1996)

    Google Scholar 

  37. Triesch, J., Malsburg, C.: A gesture interface for human-robot-interaction. In: Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, Nara, Japan, pp. 546–551 (1998)

    Google Scholar 

  38. Marcel, S.: Hand posture recognition in a body-face centered space. In: Proceedings of the Conference on Human Factors in Computer Systems (CHI) (1999)

    Google Scholar 

  39. Shen, X.H., Hua, G., Williams, L., Wu, Y.: Dynamic hand gesture recognition: An exemplar-based approach from motion divergence fields. Image Vis. Comput. 30, 227–235 (2012)

    Article  Google Scholar 

  40. Ruffieux, S., Lalanne, D., Mugellini, E.: Chairgest: A challenge for multimodal mid-air gesture recognition for close hci. In: Proceedings of the 15th ACM on International Conference on Multimodal Interaction (ICMI) (2013)

    Google Scholar 

  41. Liu, L., Shao, L.: Learning discriminative representations from rgb-d video data. In: Proceedings of International Joint Conference on Artificial Intelligence (IJCAI) (2013)

    Google Scholar 

  42. Zhuolin, J., Davis, L.S.: Recognizing actions by shape-motion prototype trees. In: IEEE International Conference on Computer Vision (ICCV), pp. 444–451 (2009)

    Google Scholar 

  43. Pisharady, P.K., Vadakkepat, P., Loh, A.P.: Hand posture and face recognition using a fuzzy-rough approach. Int. J. Humanoid Rob. 07, 331–356 (2010)

    Article  Google Scholar 

  44. Kim, T.K., Wong, S.F., Cipolla, R.: Tensor canonical correlation analysis for action classification. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–8 (2007)

    Google Scholar 

  45. Patwardhan, K.S., Roy, S.D.: Hand gesture modelling and recognition involving changing shapes and trajectories, using a predictive eigentracker. Pattern Recogn. Lett. 28, 329–334 (2007)

    Article  Google Scholar 

Download references

Acknowledgement

The authors would like to thank Mr. Joshua Tan Tang Sheng for helping in the implementation of online web-portal for the calculation of \(GRPS\).

Author information

Authors and Affiliations

  1. Institute of High Performance Computing (IHPC), A*STAR, Level 16-16 Connexis, 1 Fusionopolis Way, Singapore, 138632, Singapore

    Pramod Kumar Pisharady & Martin Saerbeck

Authors
  1. Pramod Kumar Pisharady
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Martin Saerbeck
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pramod Kumar Pisharady .

Editor information

Editors and Affiliations

  1. Center for Visual Information Technology, International Institute of Information Technology, Hyderabad, India

    C.V. Jawahar

  2. Institue of Computing Technology, Chinese Academy of Sciences, Beijing, China

    Shiguang Shan

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Pisharady, P.K., Saerbeck, M. (2015). Gesture Recognition Performance Score: A New Metric to Evaluate Gesture Recognition Systems. In: Jawahar, C., Shan, S. (eds) Computer Vision - ACCV 2014 Workshops. ACCV 2014. Lecture Notes in Computer Science(), vol 9008. Springer, Cham. https://doi.org/10.1007/978-3-319-16628-5_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-16628-5_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-16627-8

  • Online ISBN: 978-3-319-16628-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation