Abstract
In spite of many choices available for gesture recognition algorithms, the selection of a proper algorithm for a specific application remains a difficult task. The available algorithms have different strengths and weaknesses making the matching between algorithms and applications complex. Accurate evaluation of the performance of a gesture recognition algorithm is a cumbersome task. Performance evaluation by recognition accuracy alone is not sufficient to predict its successful real-world implementation. We developed a novel Gesture Recognition Performance Score (\(GRPS\)) for ranking gesture recognition algorithms, and to predict the success of these algorithms in real-world scenarios. The \(GRPS\) is calculated by considering different attributes of the algorithm, the evaluation methodology adopted, and the quality of dataset used for testing. The \(GRPS\) calculation is illustrated and applied on a set of vision based hand/ arm gesture recognition algorithms reported in the last 15 years. Based on \(GRPS\) a ranking of hand gesture recognition algorithms is provided. The paper also presents an evaluation metric namely Gesture Dataset Score (\(GDS\)) to quantify the quality of gesture databases. The \(GRPS\) calculator and results are made publicly available (http://software.ihpc.a-star.edu.sg/grps/).
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
We are in the process of publishing a detailed survey on the topic of gesture recognition.
- 2.
Considering larger number of classes will not increase the score much. This is reasonable as the number of gestures used in interaction applications is limited.
- 3.
The complexity due to the presence of other objects is considered in background index. The complexity due to the presence of other human (which is more challenging due to skin colored backgrounds) is considered in noise index.
- 4.
The list will be maintained and updated regularly. The portal provides authors of research papers a provision to submit their GRPS score and paper details to be included in the ranking list.
References
Pisharady, P.K., Vadakkepat, P., Loh, A.P.: Attention based detection and recognition of hand postures against complex backgrounds. Int. J. Comput. Vision 101, 403–419 (2013)
Dipietro, L., Sabatini, A.M., Dario, P.: A survey of glove-based systems and their applications. IEEE Trans. Syst. Man Cybern. Part C: Appl. Rev. 38, 461–482 (2008)
Mitra, S., Acharya, T.: Gesture recognition: A survey. IEEE Trans. Syst. Man Cybern. Part C: Appl. Rev. 37, 311–324 (2007)
Erol, A., Bebis, G., Nicolescu, M., Boyle, R.D., Twombly, X.: Vision-based hand pose estimation: A review. Comput. Vis. Image Underst. 108, 52–73 (2007)
Ong, S.C.W., Ranganath, S.: Automatic sign language analysis: A survey and the future beyond lexical meaning. IEEE Trans. Pattern Anal. Mach. Intell. 27, 873–891 (2005)
Thacker, N.A., Clark, A.F., Barron, J.L., Beveridge, J.R., Courtney, P., Crum, W.R., Ramesh, V., Clark, C.: Performance characterization in computer vision: A guide to best practices. Comput. Vis. Image Underst. 109, 305–334 (2008)
Ward, J.A., Lukowicz, P., Gellersen, H.W.: Performance metrics for activity recognition. ACM Trans. Intell. Syst. Technol. 02, 6:01–6:23 (2011)
Lichtenauer, J.F., Hendriks, E.A., Reinders, M.J.T.: Sign language recognition by combining statistical dtw and independent classification. IEEE Trans. Pattern Anal. Mach. Intell. 30 (2008)
Teng, X., Wu, B., Yu, W., Liu, C.: A hand gesture recognition system based on local linear embedding. J. Vis. Lang. Comput. 16, 442–454 (2005)
Lui, Y.M.: A least squares regression framework on manifolds and its application to gesture recognition. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 13–18 (2012)
Keskin, C., Kirac, F., Kara, Y., Akarun, L.: Randomized decision forests for static and dynamic hand shape classification. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 31–46 (2012)
Yang, M.H., Ahuja, N., Tabb, M.: Extraction of 2d motion trajectories and its application to hand gesture recognition. IEEE Trans. Pattern Anal. Mach. Intell. 24, 1061–1074 (2002)
Triesch, J., Malsburg, C.: A system for person-independent hand posture recognition against complex backgrounds. IEEE Trans. Pattern Anal. Mach. Intell. 23, 1449–1453 (2001)
Yoon, H.S., Soh, J., Bae, Y.J., Yang, H.S.: Hand gesture recognition using combined features of location, angle, and velocity. Pattern Recogn. 34, 1491–1501 (2001)
Huang, D.Y., Hu, W.C., Chang, S.H.: Gabor filter-based hand-pose angle estimation for hand gesture recognition under varying illumination. Expert Syst. Appl. 38, 6031–6042 (2011)
Chen, F.S., Fu, C.M., Huang, C.L.: Hand gesture recognition using a real-time tracking method and hidden markov models. Image Vis. Comput. 21, 745–758 (2003)
Zhou, R., Junsong, Y., Zhengyou, Z.: Robust hand gesture recognition based on finger-earth movers distance with a commodity depth camera. In: Proceedings of ACM Multimeida (2011)
Ramamoorthy, A., Vaswani, N., Chaudhury, S., Banerjee, S.: Recognition of dynamic hand gestures. Pattern Recogn. 36, 2069–2081 (2003)
Licsar, A., Sziranyi, T.: User-adaptive hand gesture recognition system with interactive training. Image Vis. Comput. 23, 1102–1114 (2005)
Lai, K., Konrad, J., Ishwar, P.: A gesture-driven computer interface using kinect. In: IEEE Southwest Symposium on Image Analysis and Interpretation (SSIAI), pp. 185–188 (2012)
Van den Bergh, M., Carton, D., De Nijs, R., Mitsou, N., Landsiedel, C., Kuehnlenz, K., Wollherr, D., Van Gool, L., Buss, M.: Real-time 3d hand gesture interaction with a robot for understanding directions from humans. In: IEEE International Symposium on Robot and Human Interactive Communication (IEEE RO-MAN) (2011)
Just, A., Marcel, S.: A comparative study of two state-of-the-art sequence processing techniques for hand gesture recognition. Comput. Vis. Image Underst. 113, 532–543 (2009)
Frolova, D., Stern, H., Berman, S.: Most probable longest common subsequence for recognition of gesture character input. IEEE Trans. Cybern. 43, 871–880 (2013)
Ge, S.S., Yang, Y., Lee, T.H.: Hand gesture recognition and tracking based on distributed locally linear embedding. Image Vis. Comput. 26, 1607–1620 (2008)
Shin, M.C., Tsap, L.V., Goldgof, D.B.: Gesture recognition using bezier curves for visualization navigation from registered 3-d data. Pattern Recogn. 37, 1011–1024 (2004)
Zhao, M., Quek, F.K.H., Wu, X.: Rievl: Recursive induction learning in hand gesture recognition. IEEE Trans. Pattern Anal. Mach. Intell. 20, 1174–1185 (1998)
Guyon, I., Athitsos, V., Jangyodsuk, P., Hamner, B., Escalante, H.: Chalearn gesture challenge: Design and first results. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1–6 (2012)
Malgireddy, M.R., Inwogu, I., Govindaraju, V.: A temporal bayesian model for classifying, detecting and localizing activities in video sequences. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 43–48 (2012)
Di, W., Fan, Z., Ling, S.: One shot learning gesture recognition from rgbd images. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2012)
Mahbub, U., Imtiaz, H., Roy, T., Rahman, M., Ahad, M.: A template matching approach of one-shot-learning gesture recognition. Pattern Recogn. Lett. (2012). http://dx.doi.org/10.1016/j.bbr.2011.03.031
Simon, F., Helena, M.M., Pushmeet, K., Sebastian, N.: Instructing people for training gestural interactive systems. In: International Conference on Human Factors in Computing Systems, CHI, pp. 1737–1746. ACM (2012)
Escalera, S., Gonzlez, J., Bar, X., Reyes, M., Lopes, O., Guyon, I., Athistos, V., Escalante, H.: Multi-modal gesture recognition challenge 2013: Dataset and results. In: Proceedings of the 15th ACM International Conference on Multimodal Interaction (ICMI), Sydney, Australia (2013)
Chen, M., AlRegib, G., Juang, B.H.: 6dmg: A new 6d motion gesture database. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2011)
Just, A., Bernier, O., Marcel, S.: Hmm and iohmm for the recognition of mono- and bi-manual 3d hand gestures. In: Proceedings of the British Machine Vision Conference (BMVC) (2004)
Song, Y., Demirdjian, D., Davis, R.: Tracking body and hands for gesture recognition: Natops aircraft handling signals database. In: Proceedings of the 9th IEEE Conference on Automatic Face and Gesture Recognition (FG 2011), Santa Barbara, CA, pp. 500–506 (2011)
Triesch, J., Malsburg, C.: Robust classification of hand postures against complex backgrounds. In: Proceedings of the Second International Conference on Automatic Face and Gesture Recognition, Killington, VT, USA, pp. 170–175 (1996)
Triesch, J., Malsburg, C.: A gesture interface for human-robot-interaction. In: Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, Nara, Japan, pp. 546–551 (1998)
Marcel, S.: Hand posture recognition in a body-face centered space. In: Proceedings of the Conference on Human Factors in Computer Systems (CHI) (1999)
Shen, X.H., Hua, G., Williams, L., Wu, Y.: Dynamic hand gesture recognition: An exemplar-based approach from motion divergence fields. Image Vis. Comput. 30, 227–235 (2012)
Ruffieux, S., Lalanne, D., Mugellini, E.: Chairgest: A challenge for multimodal mid-air gesture recognition for close hci. In: Proceedings of the 15th ACM on International Conference on Multimodal Interaction (ICMI) (2013)
Liu, L., Shao, L.: Learning discriminative representations from rgb-d video data. In: Proceedings of International Joint Conference on Artificial Intelligence (IJCAI) (2013)
Zhuolin, J., Davis, L.S.: Recognizing actions by shape-motion prototype trees. In: IEEE International Conference on Computer Vision (ICCV), pp. 444–451 (2009)
Pisharady, P.K., Vadakkepat, P., Loh, A.P.: Hand posture and face recognition using a fuzzy-rough approach. Int. J. Humanoid Rob. 07, 331–356 (2010)
Kim, T.K., Wong, S.F., Cipolla, R.: Tensor canonical correlation analysis for action classification. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–8 (2007)
Patwardhan, K.S., Roy, S.D.: Hand gesture modelling and recognition involving changing shapes and trajectories, using a predictive eigentracker. Pattern Recogn. Lett. 28, 329–334 (2007)
Acknowledgement
The authors would like to thank Mr. Joshua Tan Tang Sheng for helping in the implementation of online web-portal for the calculation of \(GRPS\).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Pisharady, P.K., Saerbeck, M. (2015). Gesture Recognition Performance Score: A New Metric to Evaluate Gesture Recognition Systems. In: Jawahar, C., Shan, S. (eds) Computer Vision - ACCV 2014 Workshops. ACCV 2014. Lecture Notes in Computer Science(), vol 9008. Springer, Cham. https://doi.org/10.1007/978-3-319-16628-5_12
Download citation
DOI: https://doi.org/10.1007/978-3-319-16628-5_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-16627-8
Online ISBN: 978-3-319-16628-5
eBook Packages: Computer ScienceComputer Science (R0)