Skip to main content
SpringerLink
Log in

A Review of Computational Approaches for Human Behavior Detection

  • Original Paper
  • Published:
Archives of Computational Methods in Engineering Aims and scope Submit manuscript

Abstract

Computer vision techniques capable of detecting human behavior are gaining interest. Several researchers have provided their review on behavior detection, however most of the reviews are focused on activity recognition only, and reviews on gesture and facial expression recognition are very few. Therefore, all of them lack to cover complete human behavior analysis. In this study, we provide a comprehensive review of human behavior detection approaches. The framework of this review is based on activity, gesture and facial expression recognition since these are the most important cues for behavior detection. These three areas are further classified in existing computational approaches. One can easily recognize from this review that hidden Markov model is widely exploited for activity recognition while motion history image is still a developing area. Haar-like features can be a valid alternative for gesture recognition. For facial expression recognition, local binary patterns feature is a very popular choice. We have reviewed behavior detection techniques, mostly developed after year 2009. The explicit advantages of this review are: (1) it provides a deep analysis of computational approaches for activity, gesture and facial expression recognition. (2) It includes both types of techniques that include single human as well as multiple human activities. (3) It considers techniques developed in the last decade only pertaining to information about the most recent techniques. (4) It provides a brief description of popular datasets used for activity, gesture and facial expression recognition. (5) It discusses open issues to provide an insight for future also.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Abdulrahman M, Gwadabe TR, Abdu FJ, Eleyan A (2014, April) Gabor wavelet transform based facial expression recognition using PCA and LBP. In: IEEE signal processing and communications applications conference (SIU), 2014 22nd, pp 2265–2268

  2. Ahad MAR (2011) In: Khalil I (ed) Computer vision and action recognition: a guide for image processing and computer vision community for action understanding, vol 5. Springer Science & Business Media

  3. Ahad MAR, Tan JK, Kim HS, Ishikawa S (2010) Motion history image: its variants and applications. Mach Vis Appl. https://doi.org/10.1007/s00138-010-0298-4

    Article  Google Scholar 

  4. Ahad MAR, Tan JK, Kim HS, Ishikawa S (2011) Action dataset—a survey. In: SICE annual conference, pp 1650–1655

  5. Ahad MAR, Tan JK, Kim H, Ishikawa S (2017) Activity representation by SURF-based templates. Comput Methods Biomech Biomed Eng Imaging Vis. https://doi.org/10.1080/21681163.2017.1298472.

    Google Scholar 

  6. Ahmed AA, Zaman NAK (2017) Attack intention recognition: a review. Int J Netw Secur 19(2):244–250

    Google Scholar 

  7. Ahmed W, Chanda K, Mitra S (2016, August). Vision based hand gesture recognition using dynamic time warping for Indian sign language. In: International conference on information science (ICIS). IEEE, pp 120–125

  8. Ahouandjinou A, Ezin E, Assogba K, Motamed C, Mousse M, Atohoun B (2017). Robust facial expression recognition using evidential hidden markov model. https://hal.archives-ouvertes.fr/hal-01448729

  9. Ahsan T, Jabid T, Chong UP (2013) Facial expression recognition using local transitional pattern on Gabor filtered facial images. IETE Tech Rev 30(1):47–52

    Google Scholar 

  10. Akl A, Valaee S (2010) Accelerometer-based gesture recognition via dynamic–time warping, affinity propagation, & compressive sensing. In: IEEE international conference on acoustics speech and signal processing (ICASSP), pp 2270–2273

  11. Al-Shabi M, Cheah WP, Connie T (2016) Facial expression recognition using a hybrid CNN-SIFT aggregator. arXiv preprint arXiv:1608.02833

  12. Alemdar H, Ersoy C (2017) Multi-resident activity tracking and recognition in smart environments. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-016-0440-x

    Article  Google Scholar 

  13. Alizadeh S, Fazel A (2017) Convolutional neural networks for facial expression recognition. arXiv preprint arXiv:1704.06756

  14. Almaev TR, Valstar MF (2013, September) Local gabor binary patterns from three orthogonal planes for automatic facial expression recognition. In: 2013 Humaine association conference on Affective computing and intelligent interaction (ACII). IEEE, pp 356–361

  15. Alp EC, Keles HY (2017, July) Action recognition using MHI based Hu moments with HMMs. In: IEEE EUROCON 2017-17th international conference on smart technologies, pp 212–216

  16. Amor BB, Drira H, Berretti S, Daoudi M, Srivastava A (2014) 4-D facial expression recognition by learning geometric deformations. IEEE Trans Cybern 44(12):2443–2457

    Google Scholar 

  17. Arshid S, Hussain A, Munir A, Nawaz A, Aziz S (2017) Multi-stage binary patterns for facial expression recognition in real world. Cluster Comput. https://doi.org/10.1007/s10586-017-0832-5

    Article  Google Scholar 

  18. Azeem A, Sharif M, Shah JH, Raza M (2015) Hexagonal scale invariant feature transform (H-SIFT) for facial feature extraction. J Appl Res Technol 13(3):402–408

    Google Scholar 

  19. Bakheet S, Al-Hamadi A (2016) A discriminative framework for action recognition using f-HOL Features. Information 7(4):68

    Google Scholar 

  20. Balakrishna D, Sailaja P, Rao RVVP, Indurkhya B (2010) A novel human robot interaction using the Wiimote. In: IEEE international conference on robotics and bioinformatics (ROBIO), pp 645–650

  21. Belgacem S, Chatelain C, Paquet T (2017) Gesture sequence recognition with one shot learned CRF/HMM hybrid model. Image Vis Comput 61:12–21

    Google Scholar 

  22. Berretti S, Ben Amor B, Daoudi M, Del Bimbo A (2011) 3D facial expression recognition using SIFT escriptors of automatic detected keypoints. Vis Comput 27(11):1021–1036

    Google Scholar 

  23. Berretti S, Del Bimbo A, Pala P, Amor BB, Daoudi M (2010, August) A set of selected SIFT featuresfor 3D facial expression recognition. In: 2010 20th international conference on pattern recognition (ICPR). IEEE, pp. 4125–4128

  24. Binh NT, Nigam S, Khare A (2013, November) Towards classification based human activity recognition in video sequences. In: International conference on context-aware systems and applications. Springer, New York, pp 209–218

    Google Scholar 

  25. Borghi G, Vezzani R, Cucchiara R (2017) Fast gesture recognition with multiple stream discrete HMMs on 3D skeletons. arXiv preprint arXiv:1703.02931

  26. Breazeal C, Faridi F (2016). U.S. Patent No. D761,895. Washington, DC: U.S. Patent and Trademark Office

  27. Bux A, Angelov P, Habib Z (2017) Vision based human activity recognition: a review. In: Angelov P, Gegov A, Jayne C, Shen Q (eds) Advances in computational intelligence systems. Advances in intelligent systems and computing, vol 513. Springer, Berlin

    Google Scholar 

  28. Cao J, Li W, Ma C, Tao Z (2018) Optimizing multi-sensor deployment via ensemble pruning for wearable activity recognition. Inf Fusion 41:68–79

    Google Scholar 

  29. Cao L, Liu Z, Huang TS (2010, June) Cross-dataset action detection. In: 2010 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 1998–2005

  30. Caridakis G, Karpouzis K, Drosopoulos A, Kollias S (2010) SOMM: self organizing markov map for gesture recognition. Pattern Recognit Lett 31:52–59

    Google Scholar 

  31. Chaquet JM, Carmona EJ, Fernández-Caballero A (2013) A survey of video datasets for human action and activity recognition. Comput Vis Image Underst 117(6):633–659

    Google Scholar 

  32. Chen J, Chen Z, Chi Z, Fu H (2014, August) Facial expression recognition based on facial components detection and hog features. In: International workshops on electrical and computer engineering subfields, pp 884–888

  33. Chen Q, Georganas ND, Petriu EM (2008) Hand gesture recognition using haar-like features and a stochastic context-free grammar. IEEE Trans Instrum Meas 57(8):1562–1571

    Google Scholar 

  34. Cheng H, Dai Z, Liu Z, Zhao Y (2016) An image-to-class dynamic time warping approach for both 3D static and trajectory hand gesture recognition. Pattern Recogn 55:137–147

    Google Scholar 

  35. Cheng H, Yang L, Liu Z (2016) Survey on 3D hand gesture recognition. IEEE Trans Circuits Syst Video Technol 26(9):1659–1673

    Google Scholar 

  36. Choi HR, Kim T (2017) Combined dynamic time warping with multiple sensors for 3D gesture recognition. Sensors 17(8):1893

    Google Scholar 

  37. Choi W, Shahid K, Savarese S (2009) What are they doing? Collective activity classification using spatio-temporal relationship among people. In: 12th IEEE international conference on computer vision workshops (ICCV workshops), pp 1282–1289

  38. Chong YS, Tay YH (2017, June) Abnormal event detection in videos using spatiotemporal autoencoder. In: International symposium on neural networks. Springer, Cham, pp 189–196

    Google Scholar 

  39. Choujaa D, Dulay N (2008) TRAcME: temporal activity recognition using mobile phone data. In: IEEE/IFIP international conference on embedded and ubiquitous computing, vol 1, pp 119–126

  40. Cornacchia M, Ozcan K, Zheng Y, Velipasalar S (2017) A survey on activity detection and classification using wearable sensors. IEEE Sens J 17(2):386–403

    Google Scholar 

  41. Corneanu CA, Simón MO, Cohn JF, Guerrero SE (2016) Survey on RGB, 3D, thermal, and multimodal approaches for facial expression recognition: history, trends, and affect-related applications. IEEE Trans Pattern Anal Mach Intell 38(8):1548–1568

    Google Scholar 

  42. Dahmane M, Meunier J (2011) Emotion recognition using dynamic grid-based HOG features. In: IEEE international conference on automatic face and gesture recognition, pp 884–888

  43. Dailey MN, Joyce C, Lyons MJ, Kamachi M, Ishi H, Gyoba J, Cottrell GW (2010) Evidence and a computational explanation of cultural differences in facial expression recognition. Fac Expr 10(6):874–893

    Google Scholar 

  44. Deshmukh S, Patwardhan M, Mahajan A (2016) Survey on real-time facial expression recognition techniques. IET Biometrics 5(3):155–163

    Google Scholar 

  45. Devanne M, Berretti S, Pala P, Wannous H, Daoudi M, Del Bimbo A (2017) Motion segment decomposition of RGB-D sequences for human behavior understanding. Pattern Recogn 61:222–233

    Google Scholar 

  46. Ding YD, Pang HB (2012) An improved algorithm of hand-gesture recognition based on haar-like features and Adaboost. Adv Mater Res 588–589:1238–1241. https://doi.org/10.4028/www.scientific.net/AMR.588-589.1238

    Article  Google Scholar 

  47. Dixit V, Agrawal A (2015) Real time hand detection & tracking for dynamic gesture recognition. Int J Intell Syst Appl 7(8):38

    Google Scholar 

  48. Eleyan A (2017) Comparative study on facial expression recognition using gabor and dual-tree complex wavelet transforms. Int J Eng Appl Sci (IJEAS) 9(1):1–13

    Google Scholar 

  49. Emambakhsh M, Evans A (2017) Nasal patches and curves for expression-robust 3D face recognition. IEEE Trans Pattern Anal Mach Intell 39(5):995–1007

    Google Scholar 

  50. Escalante HJ, Morales EF, Sucar LE (2016) A Naive Bayes baseline for early gesture recognition. Pattern Recogn Lett 73:91–99

    Google Scholar 

  51. Eum H, Yoon C, Lee H, Park M (2015) Continuous human action recognition using depth-MHI-HOG and a spotter model. Sensors 15(3):5197–5227

    Google Scholar 

  52. Fan X, Tjahjadi T (2015) A spatial–temporal framework based on histogram of gradients and optical flow for facial expression recognition in video sequences. Pattern Recogn 48(11):3407–3416

    Google Scholar 

  53. Garcia-Ceja E, Galván-Tejada CE, Brena R (2018) Multi-view stacking for activity recognition with sound and accelerometer data. Inf Fusion 40:45–56

    Google Scholar 

  54. Ghafouri S, Seyedarabi H (2013, May) Hybrid method for hand gesture recognition based on combination of Haar-like and HOG features. In: 2013 21st Iranian conference on electrical engineering (ICEE). IEEE, pp 1–4

  55. Gharasuie MM, Seyedarabi H (2013, September) Real-time dynamic hand gesture recognition using hidden Markov models. In: 2013 8th Iranian conference on machine vision and image processing (MVIP). IEEE, pp 194–199

  56. Ghotkar A, Vidap P, Deo K (2016) Dynamic hand gesture recognition using hidden Markov model by microsoft kinect sensor. Int J Comput Appl 150(5):5–9

    Google Scholar 

  57. Gong W, Zhang X, Gonzàlez J, Sobral A, Bouwmans T, Tu C, Zahzah EH (2016) Human pose estimation from monocular images: a comprehensive survey. Sensors 16(12):1966

    Google Scholar 

  58. Gorelick L, Blank M, Shechtman E, Irani M, Basri R (2007) Actions as space-time shapes. IEEE Trans Pattern Anal Mach Intell 29(12):2247–2253

    Google Scholar 

  59. Gori I, Aggarwal JK, Matthies L, Ryoo MS (2016) Multitype activity recognition in robot-centric scenarios. IEEE Robot Autom Lett 1(1):593–600

    Google Scholar 

  60. Guo M, Hou X, Ma Y, Wu X (2017) Facial expression recognition using ELBP based on covariance matrix transform in KLT. Multimed Tools Appl 76(2):2995–3010

    Google Scholar 

  61. Gurav RM, Kadbe PK (2015) Vision based hand gesture recognition with haar classifier and AdaBoost algorithm. Int J Latest Trends Eng Technol (IJLTET) 5(2):155–160

    Google Scholar 

  62. Hilsenbeck B, Münch D, Grosselfinger AK, Hübner W, Arens M (2016, December) Action recognition in the longwave infrared and the visible spectrum using Hough forests. In: 2016 IEEE international symposium on multimedia (ISM). IEEE, pp 329–332

  63. Hsieh CC, Liou DH (2015) Novel Haar features for real-time hand gesture recognition using SVM. J Real Time Image Proc 10(2):357–370

    Google Scholar 

  64. Huang X, Wang SJ, Liu X, Zhao G, Feng X, Pietikainen M (2017) Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Trans Affect Comput 14(8):1–15

    Google Scholar 

  65. Islam MM, Siddiqua S, Afnan J (2017, February) Real time hand gesture recognition using different algorithms based on American sign language. In: 2017 IEEE international conference on imaging, vision & pattern recognition (icIVPR). IEEE, pp 1–6

  66. Jalal A, Kim YH, Kim YJ, Kamal S, Kim D (2017) Robust human activity recognition from depth video using spatiotemporal multi-fused features. Pattern Recogn 61:295–308

    Google Scholar 

  67. Jampour M, Lepetit V, Mauthner T, Bischof H (2017) Pose-specific non-linear mappings in feature space towards multiview facial expression recognition. Image Vis Comput 58:38–46

    Google Scholar 

  68. Jeni LA, Cohn JF, Kanade T (2017) Dense 3D face alignment from 2D video for real-time use. Image Vis Comput 58:13–24

    Google Scholar 

  69. Ji X, Cheng J, Tao D, Wu X, Feng W (2017) The spatial Laplacian and temporal energy pyramid representation for human action recognition using depth sequences. Knowl Based Syst 122:64–74

    Google Scholar 

  70. Joshi A, Ghosh S, Betke M, Sclaroff S, Pfister H (2017) Personalizing gesture recognition using hierarchical bayesian neural networks. In: Proceedings of IEEE conference on computer vision and pattern recognition (CVPR)

  71. Kabir MH, Salekin MS, Uddin MZ, Abdullah-Al-Wadud M (2017) Facial expression recognition from depth video with patterns of oriented motion flow. IEEE Access 5:8880–8889

    Google Scholar 

  72. Kamal S, Jalal A (2016) A hybrid feature extraction approach for human detection, tracking and activity recognition using depth sensors. Arab J Sci Eng 41(3):1043–1051

    Google Scholar 

  73. Kamal S, Jalal A, Kim D (2016) Depth images-based human detection, tracking and activity recognition using spatiotemporal features and modified HMM. J Electr Eng Technol 11(3):1921–1926

    Google Scholar 

  74. Kamarol SKA, Jaward MH, Kälviäinen H, Parkkinen J, Parthiban R (2017) Joint facial expression recognition and intensity estimation based on weighted votes of image sequences. Pattern Recogn Lett 92:25–32

    Google Scholar 

  75. Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings. Fourth IEEE international conference on automatic face and gesture recognition, 2000. IEEE, pp 46–53

  76. Kataoka H, Miyashita Y, Hayashi M, Iwata K, Satoh Y (2016) Recognition of transitional action for short-term action prediction using discriminative temporal CNN feature. In: British machine vision conference (BMVC)

  77. Kerola T, Inoue N, Shinoda K (2017) Cross-view human action recognition from depth maps using spectral graph sequences. Comput Vis Image Underst 154:108–126

    Google Scholar 

  78. Kim TK, Cipolla R (2009) Canonical correlation analysis of video volume tensors for action categorization and detection. IEEE Trans Pattern Anal Mach Intell 31(8):1415–1428

    Google Scholar 

  79. Kolekar MH, Dash DP (2016, November) Hidden Markov model based human activity recognition using shape and optical flow based features. In: Region 10 conference (TENCON), 2016. IEEE, pp 393–397

  80. Krishna A, Strack F (2017) Reflection and impulse as determinants of human behavior. In: Meusburger P, Werlen B, Suarsana L (eds) Knowledge and action, vol 9. Springer, Cham, pp 145–167

    Google Scholar 

  81. Kumar P, Happy SL, Routray A (2016, December) A real-time robust facial expression recognition system using HOG features. In: International Conference on computing, analytics and security trends (CAST). IEEE, pp 289–293

  82. Kumari P, Mathew L, Syal P (2017) Increasing trend of wearables and multimodal interface for human activity monitoring: a review. Biosens Bioelectron 90:298–307

    Google Scholar 

  83. Kung SH, Zohdy MA, Bouchaffra D (2016) 3D HMM-based facial expression recognition using histogram of oriented optical flow. Trans Mach Learn Artif Intell 3(6):42

    Google Scholar 

  84. Kurylosky P, Giani A, Giannantonio R, Gilani K, Gravina R, Seppa VP, Seto E, Shia V, Wang C, Yan P, Yang AY, Hyttinen J, Sastry S, Wicker S, Bajcsy R (2009) DexterNet: an open platform for heterogeneous body sensor networks and its applications. In: Proceedings of the sixth international workshop on wearable and implantable body sensor networks, pp 92–97

  85. Laptev I, Marszalek M, Schmid C, Rozenfeld B (2008, June) Learning realistic human actions from movies. In: CVPR 2008. IEEE conference on computer vision and pattern recognition, 2008. IEEE, pp 1–8

  86. Li J, Zhang B (2016, November) Facial expression recognition based on Gabor and conditional random fields. In: 2016 IEEE 13th international conference on signal processing (ICSP). IEEE, pp 752–756

  87. Li Q, Qiu Z, Yao T, Mei T, Rui Y, Luo J (2017) Learning hierarchical video representation for action recognition. Int J Multimed Inf Retr 6(1):85–98

    Google Scholar 

  88. Limonchik B, Amdur G (2017) 3D model-based data augmentation for hand gesture recognition. Available at: http://cs231n.stanford.edu/reports/2017/pdfs/218.pdf

  89. Liang J, Xu C, Feng Z, Ma X (2015) Hidden Markov model decision forest for dynamic facial expression recognition. Int J Pattern Recognit Artif Intell 29(07):1556010

    MathSciNet  Google Scholar 

  90. Lin SJ, Chao MH, Lee CY, Yang CS (2016) Human action recognition using motion history image based temporal segmentation. Int J Pattern Recognit Artif Intell 30(06):1655017

    Google Scholar 

  91. Littlewort GC, Bartlett MS, Lee K (2009) Automatic coding of facial expressions displayed during posed and genuine pain. Image Vis Comput 27:1797–1803

    Google Scholar 

  92. Liu C, Chen YY, Fu LC (2016a) Robust dynamic hand gesture recognition system with sparse steric haar-like feature for human robot interaction. In: 55th annual conference of the society of instrument and control engineers of Japan (SICE), 2016. IEEE, pp 148–153

  93. Liu J, Wang Z, Zhong L, Wickramasuriya J, Vasudevan V (2009) uWave: accelerometer-based personalized gesture recognition and its applications. In: IEEE international conference on pervasive computing and communications, pp 1–9

  94. Liu X, Zhang M, Richardson A, Lucas T, Van Der Spiegel J (2016) The virtual Trackpad: an electromyography-based, wireless, real-time, low-power, embedded hand gesture recognition system using an event-driven artificial neural network. IEEE Trans Circuits Syst II Express Briefs. https://doi.org/10.1109/tcsii.(2016).2635674

    Article  Google Scholar 

  95. Liu Y, Li Y, Ma X, Song R (2017) Facial expression recognition with fusion features extracted fromsalient facial areas. Sensors 17(4):712

    Google Scholar 

  96. Liu Z, Zhang C, Tian Y (2016) 3d-based deep convolutional neural network for action recognition with depth sequences. Image Vis Comput 55(2):93–100

    Google Scholar 

  97. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010, June) The extended Cohn–Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW). IEEE, pp 94–101

  98. Luo RC, Chou YT, Liao CT, Lai CC, Tsai AC (2007) NCCU security warrior: an intelligent security robot system. In: 33rd annual conference of the IEEE industrial electronics society, pp 2960–2965

  99. Luo Y, Wu CM, Zhang Y (2013) Facial expression recognition based on fusion feature of PCA and LBP with SVM. Opt Int J Light Electron Opt 124(17):2767–2770

    Google Scholar 

  100. Luvizon DC, Tabia H, Picard D (2017) Learning features combination for human action recognition from skeleton sequences. Pattern Recogn Lett. 99:13–20. https://doi.org/10.1016/j.patrec.(2017).02.001

    Article  Google Scholar 

  101. Ma S, Zhang J, Sclaroff S, Ikizler-Cinbis N, Sigal L (2017) Space-time tree ensemble for action recognition and localization. Int J Comput Vision. https://doi.org/10.1007/s11263-016-0980-8

    Article  Google Scholar 

  102. Marcel S, Bernier O (1999, March) Hand posture recognition in a body-face centered space. In: International gesture workshop. Springer, Berlin, pp 97–100

    Google Scholar 

  103. Marcel S, Bernier O, Viallet JE, Collobert D (2000) Hand gesture recognition using input–output hidden markov models. In: Proceedings. Fourth IEEE international conference on automatic face and gesture recognition, 2000. IEEE, pp 456–461

  104. Marszalek M, Laptev I, Schmid C (2009, June) Actions in context. In: CVPR 2009 IEEE conference on computer vision and pattern recognition, 2009. IEEE, pp 2929–2936

  105. Martinez B, Valstar MF, Jiang B, Pantic M (2017) Automatic analysis of facial actions: a survey. IEEE Trans Affect Comput 99:1–11. https://doi.org/10.1109/TAFFC.2017.2731763

    Article  Google Scholar 

  106. Maruvada S (2017) 3-D hand gesture recognition with different temporal behaviors using HMM and Kinect. Master Thesis, University of Magdeburg, Germany. http://wwwisg.cs.ovgu.de/sim/files/theses/maruvada.pdf

  107. Medina-Catzin JL, Martin-Gonzalez A, Brito-Loeza C, Uc-Cetina V (2017) Body gestures recognition system to control a service robot. Int J Inf Tech Comput Sci 9:69–76. https://doi.org/10.5815/ijitcs.2017.09.07

    Article  Google Scholar 

  108. Molchanov P, Yang X, Gupta S, Kim K, Tyree S, Kautz, J (2016) Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4207–4215

  109. Mueid RM, Ahmed C, Ahad MAR (2016) Pedestrian activity classification using patterns of motion and histogram of oriented gradient. J Multimodal User Interfaces 10(4):299–305

    Google Scholar 

  110. Nakamura Y, Kimura Y, Ye Y, Ohta Y (1998) MMID: multimodal multi-view integrated database for human behavior understanding. In: Proceedings of third IEEE international conference on automatic face and gesture recognition, pp 540–545

  111. Nascimento TH, Soares FAA, Irani PP, de Oliveira LLG, da Silva Soares A (2017, July) Method for text entry in smartwatches using continuous gesture recognition. In: 2017 IEEE 41st annual Computer software and applications conference (COMPSAC), vol 2. IEEE, pp 549–554

  112. Nigam S, Singh R, Misra AK (2018) Efficient facial expression recognition using histogram of oriented gradients in wavelet domain, Multimed Tools Appl. https://doi.org/10.1007/s11042-018-6040-3

    Google Scholar 

  113. Nigam S, Khare A (2016) Integration of moment invariants and uniform local binary patterns for human activity recognition in video sequences. Multimed Tools Appl 75(24):17303–17332

    Google Scholar 

  114. Nigam S, Khare A (2015) Multiscale local binary patterns for facial expression-based human emotion recognition. In: International conference on computational vision and robotics (ICCVR). Springer, India, pp 71–77

    Google Scholar 

  115. Neeru N, Kaur L (2016) Modified SIFT descriptors for face recognition under different emotions. J Eng 2016 Article ID 9387545

  116. Owusu E, Zhan Y, Mao QR (2014) A neural-AdaBoost based facial expression recognition system. Expert Syst Appl 41(7):3383–3390

    Google Scholar 

  117. Pantic M, Valstar M, Rademaker R, Maat L (2005, July) Web-based database for facial expression analysis. In: IEEE international conference on multimedia and expo, 2005. ICME 2005. IEEE

  118. Parisi GI, Tani J, Weber C, Wermter S (2016) Emergence of multimodal action representations from neural network self-organization. Cogn Syst Res. https://doi.org/10.1016/j.cogsys.2016.08.002

    Article  Google Scholar 

  119. Parisi GI, Weber C, Wermter S (2015) Self-organizing neural integration of pose-motion features for human action recognition. Front Neurorobot. https://doi.org/10.3389/fnbot.2015.00003

    Article  Google Scholar 

  120. Parisi GI, Tani J, Weber C, Wermter S (2017) Lifelong learning of human actions with deep neural network self-organization. Neural Netw 96:137–149

    Google Scholar 

  121. Petridis V, Deb B, Syrris V (2009) Detection and identification of human actions using predictive modular neural networks. In: 17th mediterranean conference on control and automation, pp 406–411

  122. Plouffe G, Cretu AM (2016) Static and dynamic hand gesture recognition in depth data using dynamic time warping. IEEE Trans Instrum Meas 65(2):305–316

    Google Scholar 

  123. Poria S, Cambria E, Bajpai R, Hussain A (2017) A review of affective computing: from unimodal analysis to multimodal fusion. Inf Fusion 37:98–125

    Google Scholar 

  124. Qian H, Zhou J, Mao Y, Yuan Y (2017) Recognizing human actions from silhouettes described with weighted distance metric and kinematics. Multimed Tools Appl 76(21):21889–21910

    Google Scholar 

  125. Ragheb H, Velastin S, Remagnino P, Ellis T (2008) ViHASi: virtual human action silhouette data for the performance evaluation of silhouette-based action recognition methods. In: Second ACM/IEEE international conference on distributed smart cameras, pp 1–10

  126. Raju MIH, Ananna SS, Meraz SSI, Azam MZ, Serikawa S, Ahad MAR (2017) Human action recognition: a template matching-based approach. J Inst Ind Appl Eng 5(1):15–23

    Google Scholar 

  127. Raman N, Maybank SJ (2016) Activity recognition using a supervised non-parametric hierarchical HMM. Neurocomputing 199:163–177

    Google Scholar 

  128. Rautaray SS, Agrawal A (2012) Real time gesture recognition system for interaction in dynamic environment. Procedia Technol 4:595–599

    Google Scholar 

  129. Ren F, Huang Z (2015) Facial expression recognition based on AAM–SIFT and adaptive regional weighting. IEE J Trans Electr Electron Eng 10(6):713–722

    Google Scholar 

  130. Reyes M, Dominguez G, Escalera S (2011) Feature weighting in dynamic time warping for gesture recognition in depth data. In: IEEE international conference on computer vision, pp 1182–1188

  131. Richard A, Gall J (2016) A bag-of-words equivalent recurrent neural network for action recognition. Comput Vis Image Underst 156:79–91

    Google Scholar 

  132. Rodriguez MD, Ahmed J, Shah M (2008, June) Action mach a spatio-temporal maximum average correlation height filter for action recognition. In: CVPR 2008. IEEE conference on computer vision and pattern recognition, 2008. IEEE, pp 1–8

  133. Rodriguez M, Orrite C, Medrano C, Makris D (2016) One-shot learning of human activity with an MAP adapted GMM and simplex-HMM. IEEE Trans Cybern. https://doi.org/10.1109/tcyb.(2016).2558447

    Article  Google Scholar 

  134. Rodriguez M, Orrite C, Medrano C, Makris D (2017, July) Fast simplex-HMM for one-shot learning activity recognition. In: 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 1259–1266

  135. Ruan J, Yin J, Chen Q, Chen G (2014) Facial expression recognition based on gabor wavelet transform and relevance vector machine. J Inf Comput Sci 11(1):295–302

    Google Scholar 

  136. Ryu B, Rivera AR, Kim J, Chae O (2017) Local directional ternary pattern for facial expression recognition. IEEE Trans Image Process 26(12):6006–6018

    MathSciNet  Google Scholar 

  137. Saha A, Wu QMJ (2010) Facial expression recognition using curvelet based local binary patterns. In: IEEE international conference on acoustics speech and signal processing (ICASSP), pp 2470–2473

  138. Saha S, Lahiri R, Konar A, Banerjee B, Nagar AK (2017, May) HMM-based gesture recognition system using kinect sensor for improvised human–computer interaction. In: 2017 international joint conference on neural networks (IJCNN). IEEE, pp 2776–2783

  139. Sandbach G, Zafeiriou S, Pantic M, Rueckert D (2012) Recognition of 3D facial expression dynamics. Image Vis Comput 30(10):762–773

    Google Scholar 

  140. Sariyanidi E, Gunes H, Cavallaro A (2017) Learning bases of activity for facial expression recognition. IEEE Trans Image Process 26(4):1965–1978

    MathSciNet  MATH  Google Scholar 

  141. Schuldt C, Laptev I, Caputo B (2004) Recognizing human actions: a local SVM approach. In: 17th international conference on pattern recognition, vol 3, pp 32–36

  142. Selvam S, Gnanadurai D (2016) Shape-based features for reliable action recognition using spectral regression discriminant analysis. Int J Signal Imaging Syst Eng 9(6):379

    Google Scholar 

  143. Shah M, Jain R (eds) (2013) Motion-based recognition, vol 9. Springer, Berlin

    Google Scholar 

  144. Shan C, Gong S, McOwan P (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27:803–816

    Google Scholar 

  145. Sharma CM, Kushwaha AKS, Nigam S, Khare A (2011a) Automatic human activity recognition in video using background modeling and spatio-temporal template matching based technique. In: ACM international conference on advances in computing and artificial intelligence, pp 97–101

  146. Sharma CM, Kushwaha AKS, Nigam S, Khare A (2011b) On human activity recognition in video sequences. In: IEEE international conference on computer and communication technology, pp 152–158

  147. Sheng N, Cai Y, Zhan C, Qiu C, Cui Y, Gao X (2016, October) 3D facial expression recognition using distance features and LBP features based on automatically detected keypoints. In: International congress on image and signal processing, biomedical engineering and informatics (CISP-BMEI). IEEE, pp 396–401

  148. Shi Y, Tian Y, Wang Y, Huang T (2017) Sequential deep trajectory descriptor for action recognition with three-stream CNN. IEEE Trans Multimed. https://doi.org/10.1109/tmm.(2017).2666540

    Article  Google Scholar 

  149. Siddiqi MH, Ali R, Idris M, Khan AM, Kim ES, Whang MC, Lee S (2016) Human facial expression recognition using curvelet feature extraction and normalized mutual information feature selection. Multimed Tools Appl 75(2):935–959

    Google Scholar 

  150. Siddiqi MH, Ali R, Khan AM, Park YT, Lee S (2015) Human facial expression recognition using stepwise linear discriminant analysis and hidden conditional random fields. IEEE Trans Image Process 24(4):1386–1398

    MathSciNet  MATH  Google Scholar 

  151. Singh B, Marks TK, Jones M, Tuzel O, Shao M (2016) A multi-stream bi-directional recurrent neural network for fine-grained action detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1961–1970

  152. Singh D, Mohan CK (2017) Graph formulation of video activities for abnormal activity recognition. Pattern Recogn 65:265–272

    Google Scholar 

  153. Singh S, Arora C, Jawahar CV (2017) Trajectory aligned features for first person action recognition. Pattern Recogn 62:45–55

    Google Scholar 

  154. Singh S, Velastin SA, Ragheb H (2010, August) Muhavi: A multicamera human action video dataset for the evaluation of action recognition methods. In: 2010 seventh IEEE international conference on advanced video and signal based surveillance (AVSS), pp 48–55

  155. Singha J, Laskar RH (2017) Hand gesture recognition using two-level speed normalization, feature selection and classifier fusion. Multimed Syst 23(4):499–514

    Google Scholar 

  156. Singha J, Laskar RH (2016) Recognition of global hand gestures using self co-articulation information and classifier fusion. J Multimodal User Interfaces 10(1):77–93

    Google Scholar 

  157. Smeets D, Keustermans J, Vandermeulen D, Suetens P (2013) meshSIFT: local surface features for 3D face recognition under expression variations and partial data. Comput Vis Image Underst 117(2):158–169

    Google Scholar 

  158. Soares ADS, Apolinário Jr AL (2017) Real-time 3D gesture recognition using dynamic time warping and simplification methods. J WSCG 25:59–66

    Google Scholar 

  159. Sohn MK, Lee SH, Kim DJ, Kim B, Kim H (2012, November) A comparison of 3D hand gesture recognition using dynamic time warping. In: Proceedings of the 27th conference on image and vision computing New Zealand. ACM, pp. 418–422

  160. Sreekanth NS, Narayanan NK, Bangalore C (2017) Static hand gesture recognition using mon-vision based techniques. Int J Innov Comput Sci Eng 4(2):33–41

    Google Scholar 

  161. Sridevi K, Sundarambal M, Muralidharan K, Josephine RL (2017, January) FPGA implementation of hand gesture recognition system using neural networks. In: 2017 11th international conference on intelligent systems and control (ISCO). IEEE, pp 34–39

  162. Stein S, McKenna SJ (2017) Recognising complex activities with histograms of relative tracklets. Comput Vis Image Underst 154:82–93

    Google Scholar 

  163. Stergiou N (ed) (2016) Nonlinear analysis for human movement variability. CRC Press, Boca Raton

    Google Scholar 

  164. Sung J, Ponce C, Selman B, Saxena A (2012) Unstructured human activity detection from RGBD images. In: 2012, IEEE international conference on robotics and automation (ICRA). https://doi.org/10.1109/icra.(2012).6224591

  165. Takeo K, Collins RT, Lipton AJ, Fujiyoshi H, Duggins D (2000) A system for video surveillance and monitoring: VSAM final report. CMU-RI-TR-00-12, Technical Report, Carnegie University

  166. Tong Y, Shen Y, Gao B, Sun F, Chen R, Xu Y (2017) A Noisy–Robust approach for facial expression recognition. KSII Trans Internet Inf Syst (TIIS) 11(4):2124–2148

    Google Scholar 

  167. Triesch J, Von Der Malsburg C (1996 October) Robust classification of hand postures against complex backgrounds. In: Proceedings of the second IEEE international conference on automatic face and gesture recognition, pp 170–175

  168. Triesch J, Von Der Malsburg C (2001) A system for person-independent hand posture recognition against complex backgrounds. IEEE Trans Pattern Anal Mach Intell 23(12):1449–1453

    Google Scholar 

  169. Tripathi RK, Jalal AS, Agrawal SC (2017) Suspicious human activity recognition: a review. Artif Intell Rev. https://doi.org/10.1007/s10462-017-9545-7

    Article  Google Scholar 

  170. Truong A, Zaharia T (2017) Laban movement analysis and hidden Markov models for dynamic 3D gesture recognition. EURASIP J Image Video Process 2017(1):52

    Google Scholar 

  171. Tsai HH, Chang YC (2017) Facial expression recognition using a combination of multiple facial features and support vector machine. Soft Comput. https://doi.org/10.1007/s00500-017-2634-3

    Google Scholar 

  172. Uddin MZ, Hassan MM (2015) A depth video-based facial expression recognition system using radon transform, generalized discriminant analysis, and hidden Markov model. Multimed Tools Appl 74(11):3675–3690

    Google Scholar 

  173. Varatharajan R, Manogaran G, Priyan MK, Sundarasekar R (2017) Wearable sensor devices for early detection of Alzheimer disease using dynamic time warping algorithm. Cluster Comput. https://doi.org/10.1007/s10586-017-0977-2

    Google Scholar 

  174. Wang H, Kläser A, Schmid C, Liu C (2013) Dense trajectories and motion boundary descriptors for action recognition. Int J Comput Vis 103(1):60–79

    MathSciNet  Google Scholar 

  175. Wang H, Yang W, Yuan C, Ling H, Hu W (2017) Human activity prediction using temporally-weighted generalized time warping. Neurocomputing 225:139–147

    Google Scholar 

  176. Wang H, Yang Y, Yang E, Deng C (2017) Exploring hybrid spatio-temporal convolutional networks for human action recognition. Multimed Tools Appl. https://doi.org/10.1007/s11042-017-4514-3

    Article  Google Scholar 

  177. Wang X, Jin C, Liu W, Hu M, Xu L, Ren F (2013, December) Feature fusion of hog and wld for facial expression recognition. In: 2013 IEEE/SICE international symposium on system integration (SII). IEEE, pp 227–232

  178. Wang Y, Huang K, Tan T (2007, June) Human activity recognition based on R transform. In: 2007 IEEE computer society conference on computer vision and pattern recognition (CVPR’07), pp 1–8

  179. Wang Z, Wu D, Gravina R, Fortino G, Jiang Y, Tang K (2017) Kernel fusion based extreme learning machine for cross-location activity recognition. Inf Fusion 37:1–9

    Google Scholar 

  180. Weinland D, Ronfard R, Boyer E (2006) Free viewpoint action recognition using motion history volumes. Comput Vis Image Underst 104(2):249–257

    Google Scholar 

  181. Wu D, Pigou L, Kindermans PJ, Le NDH, Shao L, Dambre J, Odobez JM (2016) Deep dynamic neural networks for multimodal gesture segmentation and recognition. IEEE Trans Pattern Anal Mach Intell 38(8):1583–1597

    Google Scholar 

  182. Xie C, Li C, Zhang B, Chen C, Han J (2017) Deep fisher discriminant learning for mobile hand gesture recognition. arXiv preprint arXiv:1707.03692

  183. Xie R, Cao J (2016) Accelerometer-based hand gesture recognition by neural network and similarity matching. IEEE Sens J 16(11):4537–4545

    Google Scholar 

  184. Xu D, Wu X, Chen YL, Xu Y (2015) Online dynamic gesture recognition for human robot interaction. J Intell Rob Syst 77(3–4):583–596

    Google Scholar 

  185. Xu W, Miao Z, Zhang XP, Tian Y (2017) A hierarchical spatio-temporal model for human activity recognition. IEEE Trans Multimed. https://doi.org/10.1109/tmm.(2017).2674622

    Article  Google Scholar 

  186. Xu X, Quan C, Ren F (2015b, August) Facial expression recognition based on Gabor Wavelet transform and Histogram of Oriented Gradients. In: 2015 IEEE international conference on mechatronics and automation (ICMA). IEEE, pp 2117–2122

  187. Xu Y, Dai Y (2017) Review of hand gesture recognition study and application. Contemp Eng Sci 10(8):375–384

    Google Scholar 

  188. Yu Y, Bi S, Mo Y, Qiu W (2016, June) Real-time gesture recognition system based on Camshift algorithm and Haar-like feature. In: 2016, IEEE international conference on cyber technology in automation, control, and intelligent systems (CYBER). IEEE, pp 337–342

  189. Zajdel W, Krijnders D, Andringa T, Gavrila DM (2007) CASSANDRA: audio-video sensor fusion for aggression detection. In: IEEE conference on advanced video and signal based surveillance, pp 200–205

  190. Zhang J, Li W, Ogunbona PO, Wang P, Tang C (2016) RGB-D-based action recognition datasets: a survey. Pattern Recogn 60:86–105

    Google Scholar 

  191. Zhang XH, Wang JJ, Wang X, Ma XL (2016, June). Improvement of dynamic hand gesture recognition based on HMM algorithm. In: 2016 international conference on information system and artificial intelligence (ISAI). IEEE, pp 401–406

  192. Zhang L, Wang Z, Yao T, Mei T, Feng DD (2017) Exploiting spatial–temporal context for trajectory based action video retrieval. Multimed Tools Appl. https://doi.org/10.1007/s11042-017-4353-2

    Article  Google Scholar 

  193. Zhao K, Chu WS, De la Torre F, Cohn JF, Zhang H (2016) Joint patch and multi-label learning for facial action unit and holistic expression recognition. IEEE Trans Image Process 25(8):3931–3946

    MathSciNet  MATH  Google Scholar 

  194. Zhao L, Wang Z, Zhang G (2017) Facial expression recognition from video sequences based on spatial–temporal motion local binary pattern and gabor multiorientation fusion histogram. In: Mathematical problems in engineering. https://doi.org/10.1155/(2017)/7206041

  195. Zheng W, Tang H, Lin Z, Huang TS (2009) A novel approach to expression recognition from non-frontal face images. In: Proceedings of the IEEE international conference on computer vision, pp 1901–1908

  196. Zhu G, Zhang L, Shen P, Song J (2017) Multimodal gesture recognition using 3D convolution and convolutional LSTM. IEEE Access. https://doi.org/10.1109/access.(2017).2684186

    Article  Google Scholar 

  197. Ziaie P, Müller T, Foster ME, Knoll A (2008) A Naïve Bayes classifier with distance weighting for hand-gesture recognition. In: Advances in computer science and engineering. Springer, Berlin, Heidelberg, pp 308–315. Available online at: http://mediatum.ub.tum.de/doc/1289362/269163.pdf

    Google Scholar 

  198. Ziaie P, Müller T, Foster ME, Knoll A (2009) Using a Naïve Bayes classifier based on K-nearest neighbours with distance weighting for static hand-gesture recognition in a human–robot dialog system. Adv Comput Sci Eng Commun Comput Inf Sci 6(1):308–315

    Google Scholar 

  199. Zong Z (2017) Efficient human face recognition method under subtle SIFT features using optimized K-means. Int J Signal Process Image Process Pattern Recogn 10(7):195–204

    Google Scholar 

Download references

Acknowledgements

This work is supported by Science and Engineering Research Board, Department of Science and Technology, Government of India under Grant Number PDF/2016/003644.

Author information

Authors and Affiliations

  1. Computer Science and Engineering Department, SP Memorial Institute of Technology, Kaushambi, Uttar Pradesh, India

    Swati Nigam & A. K. Misra

  2. Department of Computer Science, Banasthali Vidyapith, Banasthali, Rajasthan, India

    Rajiv Singh

Authors
  1. Swati Nigam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rajiv Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. A. K. Misra
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rajiv Singh.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nigam, S., Singh, R. & Misra, A.K. A Review of Computational Approaches for Human Behavior Detection. Arch Computat Methods Eng 26, 831–863 (2019). https://doi.org/10.1007/s11831-018-9270-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11831-018-9270-7

Search

Navigation