A Review of Computational Approaches for Human Behavior Detection

Original Paper


Computer vision techniques capable of detecting human behavior are gaining interest. Several researchers have provided their review on behavior detection, however most of the reviews are focused on activity recognition only, and reviews on gesture and facial expression recognition are very few. Therefore, all of them lack to cover complete human behavior analysis. In this study, we provide a comprehensive review of human behavior detection approaches. The framework of this review is based on activity, gesture and facial expression recognition since these are the most important cues for behavior detection. These three areas are further classified in existing computational approaches. One can easily recognize from this review that hidden Markov model is widely exploited for activity recognition while motion history image is still a developing area. Haar-like features can be a valid alternative for gesture recognition. For facial expression recognition, local binary patterns feature is a very popular choice. We have reviewed behavior detection techniques, mostly developed after year 2009. The explicit advantages of this review are: (1) it provides a deep analysis of computational approaches for activity, gesture and facial expression recognition. (2) It includes both types of techniques that include single human as well as multiple human activities. (3) It considers techniques developed in the last decade only pertaining to information about the most recent techniques. (4) It provides a brief description of popular datasets used for activity, gesture and facial expression recognition. (5) It discusses open issues to provide an insight for future also.



This work is supported by Science and Engineering Research Board, Department of Science and Technology, Government of India under Grant Number PDF/2016/003644.


  1. 1.
    Abdulrahman M, Gwadabe TR, Abdu FJ, Eleyan A (2014, April) Gabor wavelet transform based facial expression recognition using PCA and LBP. In: IEEE signal processing and communications applications conference (SIU), 2014 22nd, pp 2265–2268Google Scholar
  2. 2.
    Ahad MAR (2011) In: Khalil I (ed) Computer vision and action recognition: a guide for image processing and computer vision community for action understanding, vol 5. Springer Science & Business MediaGoogle Scholar
  3. 3.
    Ahad MAR, Tan JK, Kim HS, Ishikawa S (2010) Motion history image: its variants and applications. Mach Vis Appl.  https://doi.org/10.1007/s00138-010-0298-4 Google Scholar
  4. 4.
    Ahad MAR, Tan JK, Kim HS, Ishikawa S (2011) Action dataset—a survey. In: SICE annual conference, pp 1650–1655Google Scholar
  5. 5.
    Ahad MAR, Tan JK, Kim H, Ishikawa S (2017) Activity representation by SURF-based templates. Comput Methods Biomech Biomed Eng Imaging Vis.  https://doi.org/10.1080/21681163.2017.1298472.
  6. 6.
    Ahmed AA, Zaman NAK (2017) Attack intention recognition: a review. Int J Netw Secur 19(2):244–250Google Scholar
  7. 7.
    Ahmed W, Chanda K, Mitra S (2016, August). Vision based hand gesture recognition using dynamic time warping for Indian sign language. In: International conference on information science (ICIS). IEEE, pp 120–125Google Scholar
  8. 8.
    Ahouandjinou A, Ezin E, Assogba K, Motamed C, Mousse M, Atohoun B (2017). Robust facial expression recognition using evidential hidden markov model. https://hal.archives-ouvertes.fr/hal-01448729
  9. 9.
    Ahsan T, Jabid T, Chong UP (2013) Facial expression recognition using local transitional pattern on Gabor filtered facial images. IETE Tech Rev 30(1):47–52Google Scholar
  10. 10.
    Akl A, Valaee S (2010) Accelerometer-based gesture recognition via dynamic–time warping, affinity propagation, & compressive sensing. In: IEEE international conference on acoustics speech and signal processing (ICASSP), pp 2270–2273Google Scholar
  11. 11.
    Al-Shabi M, Cheah WP, Connie T (2016) Facial expression recognition using a hybrid CNN-SIFT aggregator. arXiv preprint arXiv:1608.02833
  12. 12.
    Alemdar H, Ersoy C (2017) Multi-resident activity tracking and recognition in smart environments. J Ambient Intell Humaniz Comput.  https://doi.org/10.1007/s12652-016-0440-x Google Scholar
  13. 13.
    Alizadeh S, Fazel A (2017) Convolutional neural networks for facial expression recognition. arXiv preprint arXiv:1704.06756
  14. 14.
    Almaev TR, Valstar MF (2013, September) Local gabor binary patterns from three orthogonal planes for automatic facial expression recognition. In: 2013 Humaine association conference on Affective computing and intelligent interaction (ACII). IEEE, pp 356–361Google Scholar
  15. 15.
    Alp EC, Keles HY (2017, July) Action recognition using MHI based Hu moments with HMMs. In: IEEE EUROCON 2017-17th international conference on smart technologies, pp 212–216Google Scholar
  16. 16.
    Amor BB, Drira H, Berretti S, Daoudi M, Srivastava A (2014) 4-D facial expression recognition by learning geometric deformations. IEEE Trans Cybern 44(12):2443–2457Google Scholar
  17. 17.
    Arshid S, Hussain A, Munir A, Nawaz A, Aziz S (2017) Multi-stage binary patterns for facial expression recognition in real world. Cluster Comput.  https://doi.org/10.1007/s10586-017-0832-5 Google Scholar
  18. 18.
    Azeem A, Sharif M, Shah JH, Raza M (2015) Hexagonal scale invariant feature transform (H-SIFT) for facial feature extraction. J Appl Res Technol 13(3):402–408Google Scholar
  19. 19.
    Bakheet S, Al-Hamadi A (2016) A discriminative framework for action recognition using f-HOL Features. Information 7(4):68Google Scholar
  20. 20.
    Balakrishna D, Sailaja P, Rao RVVP, Indurkhya B (2010) A novel human robot interaction using the Wiimote. In: IEEE international conference on robotics and bioinformatics (ROBIO), pp 645–650Google Scholar
  21. 21.
    Belgacem S, Chatelain C, Paquet T (2017) Gesture sequence recognition with one shot learned CRF/HMM hybrid model. Image Vis Comput 61:12–21Google Scholar
  22. 22.
    Berretti S, Ben Amor B, Daoudi M, Del Bimbo A (2011) 3D facial expression recognition using SIFT escriptors of automatic detected keypoints. Vis Comput 27(11):1021–1036Google Scholar
  23. 23.
    Berretti S, Del Bimbo A, Pala P, Amor BB, Daoudi M (2010, August) A set of selected SIFT featuresfor 3D facial expression recognition. In: 2010 20th international conference on pattern recognition (ICPR). IEEE, pp. 4125–4128Google Scholar
  24. 24.
    Binh NT, Nigam S, Khare A (2013, November) Towards classification based human activity recognition in video sequences. In: International conference on context-aware systems and applications. Springer, New York, pp 209–218Google Scholar
  25. 25.
    Borghi G, Vezzani R, Cucchiara R (2017) Fast gesture recognition with multiple stream discrete HMMs on 3D skeletons. arXiv preprint arXiv:1703.02931
  26. 26.
    Breazeal C, Faridi F (2016). U.S. Patent No. D761,895. Washington, DC: U.S. Patent and Trademark OfficeGoogle Scholar
  27. 27.
    Bux A, Angelov P, Habib Z (2017) Vision based human activity recognition: a review. In: Angelov P, Gegov A, Jayne C, Shen Q (eds) Advances in computational intelligence systems. Advances in intelligent systems and computing, vol 513. Springer, BerlinGoogle Scholar
  28. 28.
    Cao J, Li W, Ma C, Tao Z (2018) Optimizing multi-sensor deployment via ensemble pruning for wearable activity recognition. Inf Fusion 41:68–79Google Scholar
  29. 29.
    Cao L, Liu Z, Huang TS (2010, June) Cross-dataset action detection. In: 2010 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 1998–2005Google Scholar
  30. 30.
    Caridakis G, Karpouzis K, Drosopoulos A, Kollias S (2010) SOMM: self organizing markov map for gesture recognition. Pattern Recognit Lett 31:52–59Google Scholar
  31. 31.
    Chaquet JM, Carmona EJ, Fernández-Caballero A (2013) A survey of video datasets for human action and activity recognition. Comput Vis Image Underst 117(6):633–659Google Scholar
  32. 32.
    Chen J, Chen Z, Chi Z, Fu H (2014, August) Facial expression recognition based on facial components detection and hog features. In: International workshops on electrical and computer engineering subfields, pp 884–888Google Scholar
  33. 33.
    Chen Q, Georganas ND, Petriu EM (2008) Hand gesture recognition using haar-like features and a stochastic context-free grammar. IEEE Trans Instrum Meas 57(8):1562–1571Google Scholar
  34. 34.
    Cheng H, Dai Z, Liu Z, Zhao Y (2016) An image-to-class dynamic time warping approach for both 3D static and trajectory hand gesture recognition. Pattern Recogn 55:137–147Google Scholar
  35. 35.
    Cheng H, Yang L, Liu Z (2016) Survey on 3D hand gesture recognition. IEEE Trans Circuits Syst Video Technol 26(9):1659–1673Google Scholar
  36. 36.
    Choi HR, Kim T (2017) Combined dynamic time warping with multiple sensors for 3D gesture recognition. Sensors 17(8):1893Google Scholar
  37. 37.
    Choi W, Shahid K, Savarese S (2009) What are they doing? Collective activity classification using spatio-temporal relationship among people. In: 12th IEEE international conference on computer vision workshops (ICCV workshops), pp 1282–1289Google Scholar
  38. 38.
    Chong YS, Tay YH (2017, June) Abnormal event detection in videos using spatiotemporal autoencoder. In: International symposium on neural networks. Springer, Cham, pp 189–196Google Scholar
  39. 39.
    Choujaa D, Dulay N (2008) TRAcME: temporal activity recognition using mobile phone data. In: IEEE/IFIP international conference on embedded and ubiquitous computing, vol 1, pp 119–126Google Scholar
  40. 40.
    Cornacchia M, Ozcan K, Zheng Y, Velipasalar S (2017) A survey on activity detection and classification using wearable sensors. IEEE Sens J 17(2):386–403Google Scholar
  41. 41.
    Corneanu CA, Simón MO, Cohn JF, Guerrero SE (2016) Survey on RGB, 3D, thermal, and multimodal approaches for facial expression recognition: history, trends, and affect-related applications. IEEE Trans Pattern Anal Mach Intell 38(8):1548–1568Google Scholar
  42. 42.
    Dahmane M, Meunier J (2011) Emotion recognition using dynamic grid-based HOG features. In: IEEE international conference on automatic face and gesture recognition, pp 884–888Google Scholar
  43. 43.
    Dailey MN, Joyce C, Lyons MJ, Kamachi M, Ishi H, Gyoba J, Cottrell GW (2010) Evidence and a computational explanation of cultural differences in facial expression recognition. Fac Expr 10(6):874–893Google Scholar
  44. 44.
    Deshmukh S, Patwardhan M, Mahajan A (2016) Survey on real-time facial expression recognition techniques. IET Biometrics 5(3):155–163Google Scholar
  45. 45.
    Devanne M, Berretti S, Pala P, Wannous H, Daoudi M, Del Bimbo A (2017) Motion segment decomposition of RGB-D sequences for human behavior understanding. Pattern Recogn 61:222–233Google Scholar
  46. 46.
    Ding YD, Pang HB (2012) An improved algorithm of hand-gesture recognition based on haar-like features and Adaboost. Adv Mater Res 588–589:1238–1241.  https://doi.org/10.4028/www.scientific.net/AMR.588-589.1238 Google Scholar
  47. 47.
    Dixit V, Agrawal A (2015) Real time hand detection & tracking for dynamic gesture recognition. Int J Intell Syst Appl 7(8):38Google Scholar
  48. 48.
    Eleyan A (2017) Comparative study on facial expression recognition using gabor and dual-tree complex wavelet transforms. Int J Eng Appl Sci (IJEAS) 9(1):1–13Google Scholar
  49. 49.
    Emambakhsh M, Evans A (2017) Nasal patches and curves for expression-robust 3D face recognition. IEEE Trans Pattern Anal Mach Intell 39(5):995–1007Google Scholar
  50. 50.
    Escalante HJ, Morales EF, Sucar LE (2016) A Naive Bayes baseline for early gesture recognition. Pattern Recogn Lett 73:91–99Google Scholar
  51. 51.
    Eum H, Yoon C, Lee H, Park M (2015) Continuous human action recognition using depth-MHI-HOG and a spotter model. Sensors 15(3):5197–5227Google Scholar
  52. 52.
    Fan X, Tjahjadi T (2015) A spatial–temporal framework based on histogram of gradients and optical flow for facial expression recognition in video sequences. Pattern Recogn 48(11):3407–3416Google Scholar
  53. 53.
    Garcia-Ceja E, Galván-Tejada CE, Brena R (2018) Multi-view stacking for activity recognition with sound and accelerometer data. Inf Fusion 40:45–56Google Scholar
  54. 54.
    Ghafouri S, Seyedarabi H (2013, May) Hybrid method for hand gesture recognition based on combination of Haar-like and HOG features. In: 2013 21st Iranian conference on electrical engineering (ICEE). IEEE, pp 1–4Google Scholar
  55. 55.
    Gharasuie MM, Seyedarabi H (2013, September) Real-time dynamic hand gesture recognition using hidden Markov models. In: 2013 8th Iranian conference on machine vision and image processing (MVIP). IEEE, pp 194–199Google Scholar
  56. 56.
    Ghotkar A, Vidap P, Deo K (2016) Dynamic hand gesture recognition using hidden Markov model by microsoft kinect sensor. Int J Comput Appl 150(5):5–9Google Scholar
  57. 57.
    Gong W, Zhang X, Gonzàlez J, Sobral A, Bouwmans T, Tu C, Zahzah EH (2016) Human pose estimation from monocular images: a comprehensive survey. Sensors 16(12):1966Google Scholar
  58. 58.
    Gorelick L, Blank M, Shechtman E, Irani M, Basri R (2007) Actions as space-time shapes. IEEE Trans Pattern Anal Mach Intell 29(12):2247–2253Google Scholar
  59. 59.
    Gori I, Aggarwal JK, Matthies L, Ryoo MS (2016) Multitype activity recognition in robot-centric scenarios. IEEE Robot Autom Lett 1(1):593–600Google Scholar
  60. 60.
    Guo M, Hou X, Ma Y, Wu X (2017) Facial expression recognition using ELBP based on covariance matrix transform in KLT. Multimed Tools Appl 76(2):2995–3010Google Scholar
  61. 61.
    Gurav RM, Kadbe PK (2015) Vision based hand gesture recognition with haar classifier and AdaBoost algorithm. Int J Latest Trends Eng Technol (IJLTET) 5(2):155–160Google Scholar
  62. 62.
    Hilsenbeck B, Münch D, Grosselfinger AK, Hübner W, Arens M (2016, December) Action recognition in the longwave infrared and the visible spectrum using Hough forests. In: 2016 IEEE international symposium on multimedia (ISM). IEEE, pp 329–332Google Scholar
  63. 63.
    Hsieh CC, Liou DH (2015) Novel Haar features for real-time hand gesture recognition using SVM. J Real Time Image Proc 10(2):357–370Google Scholar
  64. 64.
    Huang X, Wang SJ, Liu X, Zhao G, Feng X, Pietikainen M (2017) Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Trans Affect Comput 14(8):1–15Google Scholar
  65. 65.
    Islam MM, Siddiqua S, Afnan J (2017, February) Real time hand gesture recognition using different algorithms based on American sign language. In: 2017 IEEE international conference on imaging, vision & pattern recognition (icIVPR). IEEE, pp 1–6Google Scholar
  66. 66.
    Jalal A, Kim YH, Kim YJ, Kamal S, Kim D (2017) Robust human activity recognition from depth video using spatiotemporal multi-fused features. Pattern Recogn 61:295–308Google Scholar
  67. 67.
    Jampour M, Lepetit V, Mauthner T, Bischof H (2017) Pose-specific non-linear mappings in feature space towards multiview facial expression recognition. Image Vis Comput 58:38–46Google Scholar
  68. 68.
    Jeni LA, Cohn JF, Kanade T (2017) Dense 3D face alignment from 2D video for real-time use. Image Vis Comput 58:13–24Google Scholar
  69. 69.
    Ji X, Cheng J, Tao D, Wu X, Feng W (2017) The spatial Laplacian and temporal energy pyramid representation for human action recognition using depth sequences. Knowl Based Syst 122:64–74Google Scholar
  70. 70.
    Joshi A, Ghosh S, Betke M, Sclaroff S, Pfister H (2017) Personalizing gesture recognition using hierarchical bayesian neural networks. In: Proceedings of IEEE conference on computer vision and pattern recognition (CVPR)Google Scholar
  71. 71.
    Kabir MH, Salekin MS, Uddin MZ, Abdullah-Al-Wadud M (2017) Facial expression recognition from depth video with patterns of oriented motion flow. IEEE Access 5:8880–8889Google Scholar
  72. 72.
    Kamal S, Jalal A (2016) A hybrid feature extraction approach for human detection, tracking and activity recognition using depth sensors. Arab J Sci Eng 41(3):1043–1051Google Scholar
  73. 73.
    Kamal S, Jalal A, Kim D (2016) Depth images-based human detection, tracking and activity recognition using spatiotemporal features and modified HMM. J Electr Eng Technol 11(3):1921–1926Google Scholar
  74. 74.
    Kamarol SKA, Jaward MH, Kälviäinen H, Parkkinen J, Parthiban R (2017) Joint facial expression recognition and intensity estimation based on weighted votes of image sequences. Pattern Recogn Lett 92:25–32Google Scholar
  75. 75.
    Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings. Fourth IEEE international conference on automatic face and gesture recognition, 2000. IEEE, pp 46–53Google Scholar
  76. 76.
    Kataoka H, Miyashita Y, Hayashi M, Iwata K, Satoh Y (2016) Recognition of transitional action for short-term action prediction using discriminative temporal CNN feature. In: British machine vision conference (BMVC)Google Scholar
  77. 77.
    Kerola T, Inoue N, Shinoda K (2017) Cross-view human action recognition from depth maps using spectral graph sequences. Comput Vis Image Underst 154:108–126Google Scholar
  78. 78.
    Kim TK, Cipolla R (2009) Canonical correlation analysis of video volume tensors for action categorization and detection. IEEE Trans Pattern Anal Mach Intell 31(8):1415–1428Google Scholar
  79. 79.
    Kolekar MH, Dash DP (2016, November) Hidden Markov model based human activity recognition using shape and optical flow based features. In: Region 10 conference (TENCON), 2016. IEEE, pp 393–397Google Scholar
  80. 80.
    Krishna A, Strack F (2017) Reflection and impulse as determinants of human behavior. In: Meusburger P, Werlen B, Suarsana L (eds) Knowledge and action, vol 9. Springer, Cham, pp 145–167Google Scholar
  81. 81.
    Kumar P, Happy SL, Routray A (2016, December) A real-time robust facial expression recognition system using HOG features. In: International Conference on computing, analytics and security trends (CAST). IEEE, pp 289–293Google Scholar
  82. 82.
    Kumari P, Mathew L, Syal P (2017) Increasing trend of wearables and multimodal interface for human activity monitoring: a review. Biosens Bioelectron 90:298–307Google Scholar
  83. 83.
    Kung SH, Zohdy MA, Bouchaffra D (2016) 3D HMM-based facial expression recognition using histogram of oriented optical flow. Trans Mach Learn Artif Intell 3(6):42Google Scholar
  84. 84.
    Kurylosky P, Giani A, Giannantonio R, Gilani K, Gravina R, Seppa VP, Seto E, Shia V, Wang C, Yan P, Yang AY, Hyttinen J, Sastry S, Wicker S, Bajcsy R (2009) DexterNet: an open platform for heterogeneous body sensor networks and its applications. In: Proceedings of the sixth international workshop on wearable and implantable body sensor networks, pp 92–97Google Scholar
  85. 85.
    Laptev I, Marszalek M, Schmid C, Rozenfeld B (2008, June) Learning realistic human actions from movies. In: CVPR 2008. IEEE conference on computer vision and pattern recognition, 2008. IEEE, pp 1–8Google Scholar
  86. 86.
    Li J, Zhang B (2016, November) Facial expression recognition based on Gabor and conditional random fields. In: 2016 IEEE 13th international conference on signal processing (ICSP). IEEE, pp 752–756Google Scholar
  87. 87.
    Li Q, Qiu Z, Yao T, Mei T, Rui Y, Luo J (2017) Learning hierarchical video representation for action recognition. Int J Multimed Inf Retr 6(1):85–98Google Scholar
  88. 88.
    Limonchik B, Amdur G (2017) 3D model-based data augmentation for hand gesture recognition. Available at: http://cs231n.stanford.edu/reports/2017/pdfs/218.pdf
  89. 89.
    Liang J, Xu C, Feng Z, Ma X (2015) Hidden Markov model decision forest for dynamic facial expression recognition. Int J Pattern Recognit Artif Intell 29(07):1556010MathSciNetGoogle Scholar
  90. 90.
    Lin SJ, Chao MH, Lee CY, Yang CS (2016) Human action recognition using motion history image based temporal segmentation. Int J Pattern Recognit Artif Intell 30(06):1655017Google Scholar
  91. 91.
    Littlewort GC, Bartlett MS, Lee K (2009) Automatic coding of facial expressions displayed during posed and genuine pain. Image Vis Comput 27:1797–1803Google Scholar
  92. 92.
    Liu C, Chen YY, Fu LC (2016a) Robust dynamic hand gesture recognition system with sparse steric haar-like feature for human robot interaction. In: 55th annual conference of the society of instrument and control engineers of Japan (SICE), 2016. IEEE, pp 148–153Google Scholar
  93. 93.
    Liu J, Wang Z, Zhong L, Wickramasuriya J, Vasudevan V (2009) uWave: accelerometer-based personalized gesture recognition and its applications. In: IEEE international conference on pervasive computing and communications, pp 1–9Google Scholar
  94. 94.
    Liu X, Zhang M, Richardson A, Lucas T, Van Der Spiegel J (2016) The virtual Trackpad: an electromyography-based, wireless, real-time, low-power, embedded hand gesture recognition system using an event-driven artificial neural network. IEEE Trans Circuits Syst II Express Briefs.  https://doi.org/10.1109/tcsii.(2016).2635674 Google Scholar
  95. 95.
    Liu Y, Li Y, Ma X, Song R (2017) Facial expression recognition with fusion features extracted fromsalient facial areas. Sensors 17(4):712Google Scholar
  96. 96.
    Liu Z, Zhang C, Tian Y (2016) 3d-based deep convolutional neural network for action recognition with depth sequences. Image Vis Comput 55(2):93–100Google Scholar
  97. 97.
    Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010, June) The extended Cohn–Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW). IEEE, pp 94–101Google Scholar
  98. 98.
    Luo RC, Chou YT, Liao CT, Lai CC, Tsai AC (2007) NCCU security warrior: an intelligent security robot system. In: 33rd annual conference of the IEEE industrial electronics society, pp 2960–2965Google Scholar
  99. 99.
    Luo Y, Wu CM, Zhang Y (2013) Facial expression recognition based on fusion feature of PCA and LBP with SVM. Opt Int J Light Electron Opt 124(17):2767–2770Google Scholar
  100. 100.
    Luvizon DC, Tabia H, Picard D (2017) Learning features combination for human action recognition from skeleton sequences. Pattern Recogn Lett. 99:13–20.  https://doi.org/10.1016/j.patrec.(2017).02.001 Google Scholar
  101. 101.
    Ma S, Zhang J, Sclaroff S, Ikizler-Cinbis N, Sigal L (2017) Space-time tree ensemble for action recognition and localization. Int J Comput Vision.  https://doi.org/10.1007/s11263-016-0980-8 Google Scholar
  102. 102.
    Marcel S, Bernier O (1999, March) Hand posture recognition in a body-face centered space. In: International gesture workshop. Springer, Berlin, pp 97–100Google Scholar
  103. 103.
    Marcel S, Bernier O, Viallet JE, Collobert D (2000) Hand gesture recognition using input–output hidden markov models. In: Proceedings. Fourth IEEE international conference on automatic face and gesture recognition, 2000. IEEE, pp 456–461Google Scholar
  104. 104.
    Marszalek M, Laptev I, Schmid C (2009, June) Actions in context. In: CVPR 2009 IEEE conference on computer vision and pattern recognition, 2009. IEEE, pp 2929–2936Google Scholar
  105. 105.
    Martinez B, Valstar MF, Jiang B, Pantic M (2017) Automatic analysis of facial actions: a survey. IEEE Trans Affect Comput 99:1–11.  https://doi.org/10.1109/TAFFC.2017.2731763 Google Scholar
  106. 106.
    Maruvada S (2017) 3-D hand gesture recognition with different temporal behaviors using HMM and Kinect. Master Thesis, University of Magdeburg, Germany. http://wwwisg.cs.ovgu.de/sim/files/theses/maruvada.pdf
  107. 107.
    Medina-Catzin JL, Martin-Gonzalez A, Brito-Loeza C, Uc-Cetina V (2017) Body gestures recognition system to control a service robot. Int J Inf Tech Comput Sci 9:69–76.  https://doi.org/10.5815/ijitcs.2017.09.07 Google Scholar
  108. 108.
    Molchanov P, Yang X, Gupta S, Kim K, Tyree S, Kautz, J (2016) Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4207–4215Google Scholar
  109. 109.
    Mueid RM, Ahmed C, Ahad MAR (2016) Pedestrian activity classification using patterns of motion and histogram of oriented gradient. J Multimodal User Interfaces 10(4):299–305Google Scholar
  110. 110.
    Nakamura Y, Kimura Y, Ye Y, Ohta Y (1998) MMID: multimodal multi-view integrated database for human behavior understanding. In: Proceedings of third IEEE international conference on automatic face and gesture recognition, pp 540–545Google Scholar
  111. 111.
    Nascimento TH, Soares FAA, Irani PP, de Oliveira LLG, da Silva Soares A (2017, July) Method for text entry in smartwatches using continuous gesture recognition. In: 2017 IEEE 41st annual Computer software and applications conference (COMPSAC), vol 2. IEEE, pp 549–554Google Scholar
  112. 112.
    Nigam S, Singh R, Misra AK (2018) Efficient facial expression recognition using histogram of oriented gradients in wavelet domain, Multimed Tools Appl.  https://doi.org/10.1007/s11042-018-6040-3
  113. 113.
    Nigam S, Khare A (2016) Integration of moment invariants and uniform local binary patterns for human activity recognition in video sequences. Multimed Tools Appl 75(24):17303–17332Google Scholar
  114. 114.
    Nigam S, Khare A (2015) Multiscale local binary patterns for facial expression-based human emotion recognition. In: International conference on computational vision and robotics (ICCVR). Springer, India, pp 71–77Google Scholar
  115. 115.
    Neeru N, Kaur L (2016) Modified SIFT descriptors for face recognition under different emotions. J Eng 2016 Article ID 9387545Google Scholar
  116. 116.
    Owusu E, Zhan Y, Mao QR (2014) A neural-AdaBoost based facial expression recognition system. Expert Syst Appl 41(7):3383–3390Google Scholar
  117. 117.
    Pantic M, Valstar M, Rademaker R, Maat L (2005, July) Web-based database for facial expression analysis. In: IEEE international conference on multimedia and expo, 2005. ICME 2005. IEEEGoogle Scholar
  118. 118.
    Parisi GI, Tani J, Weber C, Wermter S (2016) Emergence of multimodal action representations from neural network self-organization. Cogn Syst Res.  https://doi.org/10.1016/j.cogsys.2016.08.002 Google Scholar
  119. 119.
    Parisi GI, Weber C, Wermter S (2015) Self-organizing neural integration of pose-motion features for human action recognition. Front Neurorobot.  https://doi.org/10.3389/fnbot.2015.00003 Google Scholar
  120. 120.
    Parisi GI, Tani J, Weber C, Wermter S (2017) Lifelong learning of human actions with deep neural network self-organization. Neural Netw 96:137–149Google Scholar
  121. 121.
    Petridis V, Deb B, Syrris V (2009) Detection and identification of human actions using predictive modular neural networks. In: 17th mediterranean conference on control and automation, pp 406–411Google Scholar
  122. 122.
    Plouffe G, Cretu AM (2016) Static and dynamic hand gesture recognition in depth data using dynamic time warping. IEEE Trans Instrum Meas 65(2):305–316Google Scholar
  123. 123.
    Poria S, Cambria E, Bajpai R, Hussain A (2017) A review of affective computing: from unimodal analysis to multimodal fusion. Inf Fusion 37:98–125Google Scholar
  124. 124.
    Qian H, Zhou J, Mao Y, Yuan Y (2017) Recognizing human actions from silhouettes described with weighted distance metric and kinematics. Multimed Tools Appl 76(21):21889–21910Google Scholar
  125. 125.
    Ragheb H, Velastin S, Remagnino P, Ellis T (2008) ViHASi: virtual human action silhouette data for the performance evaluation of silhouette-based action recognition methods. In: Second ACM/IEEE international conference on distributed smart cameras, pp 1–10Google Scholar
  126. 126.
    Raju MIH, Ananna SS, Meraz SSI, Azam MZ, Serikawa S, Ahad MAR (2017) Human action recognition: a template matching-based approach. J Inst Ind Appl Eng 5(1):15–23Google Scholar
  127. 127.
    Raman N, Maybank SJ (2016) Activity recognition using a supervised non-parametric hierarchical HMM. Neurocomputing 199:163–177Google Scholar
  128. 128.
    Rautaray SS, Agrawal A (2012) Real time gesture recognition system for interaction in dynamic environment. Procedia Technol 4:595–599Google Scholar
  129. 129.
    Ren F, Huang Z (2015) Facial expression recognition based on AAM–SIFT and adaptive regional weighting. IEE J Trans Electr Electron Eng 10(6):713–722Google Scholar
  130. 130.
    Reyes M, Dominguez G, Escalera S (2011) Feature weighting in dynamic time warping for gesture recognition in depth data. In: IEEE international conference on computer vision, pp 1182–1188Google Scholar
  131. 131.
    Richard A, Gall J (2016) A bag-of-words equivalent recurrent neural network for action recognition. Comput Vis Image Underst 156:79–91Google Scholar
  132. 132.
    Rodriguez MD, Ahmed J, Shah M (2008, June) Action mach a spatio-temporal maximum average correlation height filter for action recognition. In: CVPR 2008. IEEE conference on computer vision and pattern recognition, 2008. IEEE, pp 1–8Google Scholar
  133. 133.
    Rodriguez M, Orrite C, Medrano C, Makris D (2016) One-shot learning of human activity with an MAP adapted GMM and simplex-HMM. IEEE Trans Cybern.  https://doi.org/10.1109/tcyb.(2016).2558447 Google Scholar
  134. 134.
    Rodriguez M, Orrite C, Medrano C, Makris D (2017, July) Fast simplex-HMM for one-shot learning activity recognition. In: 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 1259–1266Google Scholar
  135. 135.
    Ruan J, Yin J, Chen Q, Chen G (2014) Facial expression recognition based on gabor wavelet transform and relevance vector machine. J Inf Comput Sci 11(1):295–302Google Scholar
  136. 136.
    Ryu B, Rivera AR, Kim J, Chae O (2017) Local directional ternary pattern for facial expression recognition. IEEE Trans Image Process 26(12):6006–6018MathSciNetGoogle Scholar
  137. 137.
    Saha A, Wu QMJ (2010) Facial expression recognition using curvelet based local binary patterns. In: IEEE international conference on acoustics speech and signal processing (ICASSP), pp 2470–2473Google Scholar
  138. 138.
    Saha S, Lahiri R, Konar A, Banerjee B, Nagar AK (2017, May) HMM-based gesture recognition system using kinect sensor for improvised human–computer interaction. In: 2017 international joint conference on neural networks (IJCNN). IEEE, pp 2776–2783Google Scholar
  139. 139.
    Sandbach G, Zafeiriou S, Pantic M, Rueckert D (2012) Recognition of 3D facial expression dynamics. Image Vis Comput 30(10):762–773Google Scholar
  140. 140.
    Sariyanidi E, Gunes H, Cavallaro A (2017) Learning bases of activity for facial expression recognition. IEEE Trans Image Process 26(4):1965–1978MathSciNetGoogle Scholar
  141. 141.
    Schuldt C, Laptev I, Caputo B (2004) Recognizing human actions: a local SVM approach. In: 17th international conference on pattern recognition, vol 3, pp 32–36Google Scholar
  142. 142.
    Selvam S, Gnanadurai D (2016) Shape-based features for reliable action recognition using spectral regression discriminant analysis. Int J Signal Imaging Syst Eng 9(6):379Google Scholar
  143. 143.
    Shah M, Jain R (eds) (2013) Motion-based recognition, vol 9. Springer, BerlinGoogle Scholar
  144. 144.
    Shan C, Gong S, McOwan P (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27:803–816Google Scholar
  145. 145.
    Sharma CM, Kushwaha AKS, Nigam S, Khare A (2011a) Automatic human activity recognition in video using background modeling and spatio-temporal template matching based technique. In: ACM international conference on advances in computing and artificial intelligence, pp 97–101Google Scholar
  146. 146.
    Sharma CM, Kushwaha AKS, Nigam S, Khare A (2011b) On human activity recognition in video sequences. In: IEEE international conference on computer and communication technology, pp 152–158Google Scholar
  147. 147.
    Sheng N, Cai Y, Zhan C, Qiu C, Cui Y, Gao X (2016, October) 3D facial expression recognition using distance features and LBP features based on automatically detected keypoints. In: International congress on image and signal processing, biomedical engineering and informatics (CISP-BMEI). IEEE, pp 396–401Google Scholar
  148. 148.
    Shi Y, Tian Y, Wang Y, Huang T (2017) Sequential deep trajectory descriptor for action recognition with three-stream CNN. IEEE Trans Multimed.  https://doi.org/10.1109/tmm.(2017).2666540 Google Scholar
  149. 149.
    Siddiqi MH, Ali R, Idris M, Khan AM, Kim ES, Whang MC, Lee S (2016) Human facial expression recognition using curvelet feature extraction and normalized mutual information feature selection. Multimed Tools Appl 75(2):935–959Google Scholar
  150. 150.
    Siddiqi MH, Ali R, Khan AM, Park YT, Lee S (2015) Human facial expression recognition using stepwise linear discriminant analysis and hidden conditional random fields. IEEE Trans Image Process 24(4):1386–1398MathSciNetGoogle Scholar
  151. 151.
    Singh B, Marks TK, Jones M, Tuzel O, Shao M (2016) A multi-stream bi-directional recurrent neural network for fine-grained action detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1961–1970Google Scholar
  152. 152.
    Singh D, Mohan CK (2017) Graph formulation of video activities for abnormal activity recognition. Pattern Recogn 65:265–272Google Scholar
  153. 153.
    Singh S, Arora C, Jawahar CV (2017) Trajectory aligned features for first person action recognition. Pattern Recogn 62:45–55Google Scholar
  154. 154.
    Singh S, Velastin SA, Ragheb H (2010, August) Muhavi: A multicamera human action video dataset for the evaluation of action recognition methods. In: 2010 seventh IEEE international conference on advanced video and signal based surveillance (AVSS), pp 48–55Google Scholar
  155. 155.
    Singha J, Laskar RH (2017) Hand gesture recognition using two-level speed normalization, feature selection and classifier fusion. Multimed Syst 23(4):499–514Google Scholar
  156. 156.
    Singha J, Laskar RH (2016) Recognition of global hand gestures using self co-articulation information and classifier fusion. J Multimodal User Interfaces 10(1):77–93Google Scholar
  157. 157.
    Smeets D, Keustermans J, Vandermeulen D, Suetens P (2013) meshSIFT: local surface features for 3D face recognition under expression variations and partial data. Comput Vis Image Underst 117(2):158–169Google Scholar
  158. 158.
    Soares ADS, Apolinário Jr AL (2017) Real-time 3D gesture recognition using dynamic time warping and simplification methods. J WSCG 25:59–66Google Scholar
  159. 159.
    Sohn MK, Lee SH, Kim DJ, Kim B, Kim H (2012, November) A comparison of 3D hand gesture recognition using dynamic time warping. In: Proceedings of the 27th conference on image and vision computing New Zealand. ACM, pp. 418–422Google Scholar
  160. 160.
    Sreekanth NS, Narayanan NK, Bangalore C (2017) Static hand gesture recognition using mon-vision based techniques. Int J Innov Comput Sci Eng 4(2):33–41Google Scholar
  161. 161.
    Sridevi K, Sundarambal M, Muralidharan K, Josephine RL (2017, January) FPGA implementation of hand gesture recognition system using neural networks. In: 2017 11th international conference on intelligent systems and control (ISCO). IEEE, pp 34–39Google Scholar
  162. 162.
    Stein S, McKenna SJ (2017) Recognising complex activities with histograms of relative tracklets. Comput Vis Image Underst 154:82–93Google Scholar
  163. 163.
    Stergiou N (ed) (2016) Nonlinear analysis for human movement variability. CRC Press, Boca RatonGoogle Scholar
  164. 164.
    Sung J, Ponce C, Selman B, Saxena A (2012) Unstructured human activity detection from RGBD images. In: 2012, IEEE international conference on robotics and automation (ICRA).  https://doi.org/10.1109/icra.(2012).6224591
  165. 165.
    Takeo K, Collins RT, Lipton AJ, Fujiyoshi H, Duggins D (2000) A system for video surveillance and monitoring: VSAM final report. CMU-RI-TR-00-12, Technical Report, Carnegie UniversityGoogle Scholar
  166. 166.
    Tong Y, Shen Y, Gao B, Sun F, Chen R, Xu Y (2017) A Noisy–Robust approach for facial expression recognition. KSII Trans Internet Inf Syst (TIIS) 11(4):2124–2148Google Scholar
  167. 167.
    Triesch J, Von Der Malsburg C (1996 October) Robust classification of hand postures against complex backgrounds. In: Proceedings of the second IEEE international conference on automatic face and gesture recognition, pp 170–175Google Scholar
  168. 168.
    Triesch J, Von Der Malsburg C (2001) A system for person-independent hand posture recognition against complex backgrounds. IEEE Trans Pattern Anal Mach Intell 23(12):1449–1453Google Scholar
  169. 169.
    Tripathi RK, Jalal AS, Agrawal SC (2017) Suspicious human activity recognition: a review. Artif Intell Rev.  https://doi.org/10.1007/s10462-017-9545-7 Google Scholar
  170. 170.
    Truong A, Zaharia T (2017) Laban movement analysis and hidden Markov models for dynamic 3D gesture recognition. EURASIP J Image Video Process 2017(1):52Google Scholar
  171. 171.
    Tsai HH, Chang YC (2017) Facial expression recognition using a combination of multiple facial features and support vector machine. Soft Comput.  https://doi.org/10.1007/s00500-017-2634-3
  172. 172.
    Uddin MZ, Hassan MM (2015) A depth video-based facial expression recognition system using radon transform, generalized discriminant analysis, and hidden Markov model. Multimed Tools Appl 74(11):3675–3690Google Scholar
  173. 173.
    Varatharajan R, Manogaran G, Priyan MK, Sundarasekar R (2017) Wearable sensor devices for early detection of Alzheimer disease using dynamic time warping algorithm. Cluster Comput.  https://doi.org/10.1007/s10586-017-0977-2
  174. 174.
    Wang H, Kläser A, Schmid C, Liu C (2013) Dense trajectories and motion boundary descriptors for action recognition. Int J Comput Vis 103(1):60–79MathSciNetGoogle Scholar
  175. 175.
    Wang H, Yang W, Yuan C, Ling H, Hu W (2017) Human activity prediction using temporally-weighted generalized time warping. Neurocomputing 225:139–147Google Scholar
  176. 176.
    Wang H, Yang Y, Yang E, Deng C (2017) Exploring hybrid spatio-temporal convolutional networks for human action recognition. Multimed Tools Appl.  https://doi.org/10.1007/s11042-017-4514-3 Google Scholar
  177. 177.
    Wang X, Jin C, Liu W, Hu M, Xu L, Ren F (2013, December) Feature fusion of hog and wld for facial expression recognition. In: 2013 IEEE/SICE international symposium on system integration (SII). IEEE, pp 227–232Google Scholar
  178. 178.
    Wang Y, Huang K, Tan T (2007, June) Human activity recognition based on R transform. In: 2007 IEEE computer society conference on computer vision and pattern recognition (CVPR’07), pp 1–8Google Scholar
  179. 179.
    Wang Z, Wu D, Gravina R, Fortino G, Jiang Y, Tang K (2017) Kernel fusion based extreme learning machine for cross-location activity recognition. Inf Fusion 37:1–9Google Scholar
  180. 180.
    Weinland D, Ronfard R, Boyer E (2006) Free viewpoint action recognition using motion history volumes. Comput Vis Image Underst 104(2):249–257Google Scholar
  181. 181.
    Wu D, Pigou L, Kindermans PJ, Le NDH, Shao L, Dambre J, Odobez JM (2016) Deep dynamic neural networks for multimodal gesture segmentation and recognition. IEEE Trans Pattern Anal Mach Intell 38(8):1583–1597Google Scholar
  182. 182.
    Xie C, Li C, Zhang B, Chen C, Han J (2017) Deep fisher discriminant learning for mobile hand gesture recognition. arXiv preprint arXiv:1707.03692
  183. 183.
    Xie R, Cao J (2016) Accelerometer-based hand gesture recognition by neural network and similarity matching. IEEE Sens J 16(11):4537–4545Google Scholar
  184. 184.
    Xu D, Wu X, Chen YL, Xu Y (2015) Online dynamic gesture recognition for human robot interaction. J Intell Rob Syst 77(3–4):583–596Google Scholar
  185. 185.
    Xu W, Miao Z, Zhang XP, Tian Y (2017) A hierarchical spatio-temporal model for human activity recognition. IEEE Trans Multimed.  https://doi.org/10.1109/tmm.(2017).2674622 Google Scholar
  186. 186.
    Xu X, Quan C, Ren F (2015b, August) Facial expression recognition based on Gabor Wavelet transform and Histogram of Oriented Gradients. In: 2015 IEEE international conference on mechatronics and automation (ICMA). IEEE, pp 2117–2122Google Scholar
  187. 187.
    Xu Y, Dai Y (2017) Review of hand gesture recognition study and application. Contemp Eng Sci 10(8):375–384Google Scholar
  188. 188.
    Yu Y, Bi S, Mo Y, Qiu W (2016, June) Real-time gesture recognition system based on Camshift algorithm and Haar-like feature. In: 2016, IEEE international conference on cyber technology in automation, control, and intelligent systems (CYBER). IEEE, pp 337–342Google Scholar
  189. 189.
    Zajdel W, Krijnders D, Andringa T, Gavrila DM (2007) CASSANDRA: audio-video sensor fusion for aggression detection. In: IEEE conference on advanced video and signal based surveillance, pp 200–205Google Scholar
  190. 190.
    Zhang J, Li W, Ogunbona PO, Wang P, Tang C (2016) RGB-D-based action recognition datasets: a survey. Pattern Recogn 60:86–105Google Scholar
  191. 191.
    Zhang XH, Wang JJ, Wang X, Ma XL (2016, June). Improvement of dynamic hand gesture recognition based on HMM algorithm. In: 2016 international conference on information system and artificial intelligence (ISAI). IEEE, pp 401–406Google Scholar
  192. 192.
    Zhang L, Wang Z, Yao T, Mei T, Feng DD (2017) Exploiting spatial–temporal context for trajectory based action video retrieval. Multimed Tools Appl.  https://doi.org/10.1007/s11042-017-4353-2 Google Scholar
  193. 193.
    Zhao K, Chu WS, De la Torre F, Cohn JF, Zhang H (2016) Joint patch and multi-label learning for facial action unit and holistic expression recognition. IEEE Trans Image Process 25(8):3931–3946MathSciNetGoogle Scholar
  194. 194.
    Zhao L, Wang Z, Zhang G (2017) Facial expression recognition from video sequences based on spatial–temporal motion local binary pattern and gabor multiorientation fusion histogram. In: Mathematical problems in engineering.  https://doi.org/10.1155/(2017)/7206041
  195. 195.
    Zheng W, Tang H, Lin Z, Huang TS (2009) A novel approach to expression recognition from non-frontal face images. In: Proceedings of the IEEE international conference on computer vision, pp 1901–1908Google Scholar
  196. 196.
    Zhu G, Zhang L, Shen P, Song J (2017) Multimodal gesture recognition using 3D convolution and convolutional LSTM. IEEE Access.  https://doi.org/10.1109/access.(2017).2684186 Google Scholar
  197. 197.
    Ziaie P, Müller T, Foster ME, Knoll A (2008) A Naïve Bayes classifier with distance weighting for hand-gesture recognition. In: Advances in computer science and engineering. Springer, Berlin, Heidelberg, pp 308–315. Available online at: http://mediatum.ub.tum.de/doc/1289362/269163.pdf
  198. 198.
    Ziaie P, Müller T, Foster ME, Knoll A (2009) Using a Naïve Bayes classifier based on K-nearest neighbours with distance weighting for static hand-gesture recognition in a human–robot dialog system. Adv Comput Sci Eng Commun Comput Inf Sci 6(1):308–315Google Scholar
  199. 199.
    Zong Z (2017) Efficient human face recognition method under subtle SIFT features using optimized K-means. Int J Signal Process Image Process Pattern Recogn 10(7):195–204Google Scholar

Copyright information

© CIMNE, Barcelona, Spain 2018

Authors and Affiliations

  1. 1.Computer Science and Engineering DepartmentSP Memorial Institute of TechnologyKaushambiIndia
  2. 2.Department of Computer ScienceBanasthali VidyapithBanasthaliIndia

Personalised recommendations