Advertisement

A Review of Computational Approaches for Human Behavior Detection

  • Swati Nigam
  • Rajiv Singh
  • A. K. Misra
Original Paper
  • 274 Downloads

Abstract

Computer vision techniques capable of detecting human behavior are gaining interest. Several researchers have provided their review on behavior detection, however most of the reviews are focused on activity recognition only, and reviews on gesture and facial expression recognition are very few. Therefore, all of them lack to cover complete human behavior analysis. In this study, we provide a comprehensive review of human behavior detection approaches. The framework of this review is based on activity, gesture and facial expression recognition since these are the most important cues for behavior detection. These three areas are further classified in existing computational approaches. One can easily recognize from this review that hidden Markov model is widely exploited for activity recognition while motion history image is still a developing area. Haar-like features can be a valid alternative for gesture recognition. For facial expression recognition, local binary patterns feature is a very popular choice. We have reviewed behavior detection techniques, mostly developed after year 2009. The explicit advantages of this review are: (1) it provides a deep analysis of computational approaches for activity, gesture and facial expression recognition. (2) It includes both types of techniques that include single human as well as multiple human activities. (3) It considers techniques developed in the last decade only pertaining to information about the most recent techniques. (4) It provides a brief description of popular datasets used for activity, gesture and facial expression recognition. (5) It discusses open issues to provide an insight for future also.

Notes

Acknowledgements

This work is supported by Science and Engineering Research Board, Department of Science and Technology, Government of India under Grant Number PDF/2016/003644.

References

  1. 1.
    Abdulrahman M, Gwadabe TR, Abdu FJ, Eleyan A (2014, April) Gabor wavelet transform based facial expression recognition using PCA and LBP. In: IEEE signal processing and communications applications conference (SIU), 2014 22nd, pp 2265–2268Google Scholar
  2. 2.
    Ahad MAR (2011) In: Khalil I (ed) Computer vision and action recognition: a guide for image processing and computer vision community for action understanding, vol 5. Springer Science & Business MediaGoogle Scholar
  3. 3.
    Ahad MAR, Tan JK, Kim HS, Ishikawa S (2010) Motion history image: its variants and applications. Mach Vis Appl.  https://doi.org/10.1007/s00138-010-0298-4 CrossRefGoogle Scholar
  4. 4.
    Ahad MAR, Tan JK, Kim HS, Ishikawa S (2011) Action dataset—a survey. In: SICE annual conference, pp 1650–1655Google Scholar
  5. 5.
    Ahad MAR, Tan JK, Kim H, Ishikawa S (2017) Activity representation by SURF-based templates. Comput Methods Biomech Biomed Eng Imaging Vis.  https://doi.org/10.1080/21681163.2017.1298472.
  6. 6.
    Ahmed AA, Zaman NAK (2017) Attack intention recognition: a review. Int J Netw Secur 19(2):244–250Google Scholar
  7. 7.
    Ahmed W, Chanda K, Mitra S (2016, August). Vision based hand gesture recognition using dynamic time warping for Indian sign language. In: International conference on information science (ICIS). IEEE, pp 120–125Google Scholar
  8. 8.
    Ahouandjinou A, Ezin E, Assogba K, Motamed C, Mousse M, Atohoun B (2017). Robust facial expression recognition using evidential hidden markov model. https://hal.archives-ouvertes.fr/hal-01448729
  9. 9.
    Ahsan T, Jabid T, Chong UP (2013) Facial expression recognition using local transitional pattern on Gabor filtered facial images. IETE Tech Rev 30(1):47–52CrossRefGoogle Scholar
  10. 10.
    Akl A, Valaee S (2010) Accelerometer-based gesture recognition via dynamic–time warping, affinity propagation, & compressive sensing. In: IEEE international conference on acoustics speech and signal processing (ICASSP), pp 2270–2273Google Scholar
  11. 11.
    Al-Shabi M, Cheah WP, Connie T (2016) Facial expression recognition using a hybrid CNN-SIFT aggregator. arXiv preprint arXiv:1608.02833
  12. 12.
    Alemdar H, Ersoy C (2017) Multi-resident activity tracking and recognition in smart environments. J Ambient Intell Humaniz Comput.  https://doi.org/10.1007/s12652-016-0440-x CrossRefGoogle Scholar
  13. 13.
    Alizadeh S, Fazel A (2017) Convolutional neural networks for facial expression recognition. arXiv preprint arXiv:1704.06756
  14. 14.
    Almaev TR, Valstar MF (2013, September) Local gabor binary patterns from three orthogonal planes for automatic facial expression recognition. In: 2013 Humaine association conference on Affective computing and intelligent interaction (ACII). IEEE, pp 356–361Google Scholar
  15. 15.
    Alp EC, Keles HY (2017, July) Action recognition using MHI based Hu moments with HMMs. In: IEEE EUROCON 2017-17th international conference on smart technologies, pp 212–216Google Scholar
  16. 16.
    Amor BB, Drira H, Berretti S, Daoudi M, Srivastava A (2014) 4-D facial expression recognition by learning geometric deformations. IEEE Trans Cybern 44(12):2443–2457CrossRefGoogle Scholar
  17. 17.
    Arshid S, Hussain A, Munir A, Nawaz A, Aziz S (2017) Multi-stage binary patterns for facial expression recognition in real world. Cluster Comput.  https://doi.org/10.1007/s10586-017-0832-5 CrossRefGoogle Scholar
  18. 18.
    Azeem A, Sharif M, Shah JH, Raza M (2015) Hexagonal scale invariant feature transform (H-SIFT) for facial feature extraction. J Appl Res Technol 13(3):402–408CrossRefGoogle Scholar
  19. 19.
    Bakheet S, Al-Hamadi A (2016) A discriminative framework for action recognition using f-HOL Features. Information 7(4):68CrossRefGoogle Scholar
  20. 20.
    Balakrishna D, Sailaja P, Rao RVVP, Indurkhya B (2010) A novel human robot interaction using the Wiimote. In: IEEE international conference on robotics and bioinformatics (ROBIO), pp 645–650Google Scholar
  21. 21.
    Belgacem S, Chatelain C, Paquet T (2017) Gesture sequence recognition with one shot learned CRF/HMM hybrid model. Image Vis Comput 61:12–21CrossRefGoogle Scholar
  22. 22.
    Berretti S, Ben Amor B, Daoudi M, Del Bimbo A (2011) 3D facial expression recognition using SIFT escriptors of automatic detected keypoints. Vis Comput 27(11):1021–1036CrossRefGoogle Scholar
  23. 23.
    Berretti S, Del Bimbo A, Pala P, Amor BB, Daoudi M (2010, August) A set of selected SIFT featuresfor 3D facial expression recognition. In: 2010 20th international conference on pattern recognition (ICPR). IEEE, pp. 4125–4128Google Scholar
  24. 24.
    Binh NT, Nigam S, Khare A (2013, November) Towards classification based human activity recognition in video sequences. In: International conference on context-aware systems and applications. Springer, New York, pp 209–218Google Scholar
  25. 25.
    Borghi G, Vezzani R, Cucchiara R (2017) Fast gesture recognition with multiple stream discrete HMMs on 3D skeletons. arXiv preprint arXiv:1703.02931
  26. 26.
    Breazeal C, Faridi F (2016). U.S. Patent No. D761,895. Washington, DC: U.S. Patent and Trademark OfficeGoogle Scholar
  27. 27.
    Bux A, Angelov P, Habib Z (2017) Vision based human activity recognition: a review. In: Angelov P, Gegov A, Jayne C, Shen Q (eds) Advances in computational intelligence systems. Advances in intelligent systems and computing, vol 513. Springer, BerlinGoogle Scholar
  28. 28.
    Cao J, Li W, Ma C, Tao Z (2018) Optimizing multi-sensor deployment via ensemble pruning for wearable activity recognition. Inf Fusion 41:68–79CrossRefGoogle Scholar
  29. 29.
    Cao L, Liu Z, Huang TS (2010, June) Cross-dataset action detection. In: 2010 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 1998–2005Google Scholar
  30. 30.
    Caridakis G, Karpouzis K, Drosopoulos A, Kollias S (2010) SOMM: self organizing markov map for gesture recognition. Pattern Recognit Lett 31:52–59CrossRefGoogle Scholar
  31. 31.
    Chaquet JM, Carmona EJ, Fernández-Caballero A (2013) A survey of video datasets for human action and activity recognition. Comput Vis Image Underst 117(6):633–659CrossRefGoogle Scholar
  32. 32.
    Chen J, Chen Z, Chi Z, Fu H (2014, August) Facial expression recognition based on facial components detection and hog features. In: International workshops on electrical and computer engineering subfields, pp 884–888Google Scholar
  33. 33.
    Chen Q, Georganas ND, Petriu EM (2008) Hand gesture recognition using haar-like features and a stochastic context-free grammar. IEEE Trans Instrum Meas 57(8):1562–1571CrossRefGoogle Scholar
  34. 34.
    Cheng H, Dai Z, Liu Z, Zhao Y (2016) An image-to-class dynamic time warping approach for both 3D static and trajectory hand gesture recognition. Pattern Recogn 55:137–147CrossRefGoogle Scholar
  35. 35.
    Cheng H, Yang L, Liu Z (2016) Survey on 3D hand gesture recognition. IEEE Trans Circuits Syst Video Technol 26(9):1659–1673CrossRefGoogle Scholar
  36. 36.
    Choi HR, Kim T (2017) Combined dynamic time warping with multiple sensors for 3D gesture recognition. Sensors 17(8):1893CrossRefGoogle Scholar
  37. 37.
    Choi W, Shahid K, Savarese S (2009) What are they doing? Collective activity classification using spatio-temporal relationship among people. In: 12th IEEE international conference on computer vision workshops (ICCV workshops), pp 1282–1289Google Scholar
  38. 38.
    Chong YS, Tay YH (2017, June) Abnormal event detection in videos using spatiotemporal autoencoder. In: International symposium on neural networks. Springer, Cham, pp 189–196Google Scholar
  39. 39.
    Choujaa D, Dulay N (2008) TRAcME: temporal activity recognition using mobile phone data. In: IEEE/IFIP international conference on embedded and ubiquitous computing, vol 1, pp 119–126Google Scholar
  40. 40.
    Cornacchia M, Ozcan K, Zheng Y, Velipasalar S (2017) A survey on activity detection and classification using wearable sensors. IEEE Sens J 17(2):386–403CrossRefGoogle Scholar
  41. 41.
    Corneanu CA, Simón MO, Cohn JF, Guerrero SE (2016) Survey on RGB, 3D, thermal, and multimodal approaches for facial expression recognition: history, trends, and affect-related applications. IEEE Trans Pattern Anal Mach Intell 38(8):1548–1568CrossRefGoogle Scholar
  42. 42.
    Dahmane M, Meunier J (2011) Emotion recognition using dynamic grid-based HOG features. In: IEEE international conference on automatic face and gesture recognition, pp 884–888Google Scholar
  43. 43.
    Dailey MN, Joyce C, Lyons MJ, Kamachi M, Ishi H, Gyoba J, Cottrell GW (2010) Evidence and a computational explanation of cultural differences in facial expression recognition. Fac Expr 10(6):874–893Google Scholar
  44. 44.
    Deshmukh S, Patwardhan M, Mahajan A (2016) Survey on real-time facial expression recognition techniques. IET Biometrics 5(3):155–163Google Scholar
  45. 45.
    Devanne M, Berretti S, Pala P, Wannous H, Daoudi M, Del Bimbo A (2017) Motion segment decomposition of RGB-D sequences for human behavior understanding. Pattern Recogn 61:222–233CrossRefGoogle Scholar
  46. 46.
    Ding YD, Pang HB (2012) An improved algorithm of hand-gesture recognition based on haar-like features and Adaboost. Adv Mater Res 588–589:1238–1241.  https://doi.org/10.4028/www.scientific.net/AMR.588-589.1238 CrossRefGoogle Scholar
  47. 47.
    Dixit V, Agrawal A (2015) Real time hand detection & tracking for dynamic gesture recognition. Int J Intell Syst Appl 7(8):38Google Scholar
  48. 48.
    Eleyan A (2017) Comparative study on facial expression recognition using gabor and dual-tree complex wavelet transforms. Int J Eng Appl Sci (IJEAS) 9(1):1–13Google Scholar
  49. 49.
    Emambakhsh M, Evans A (2017) Nasal patches and curves for expression-robust 3D face recognition. IEEE Trans Pattern Anal Mach Intell 39(5):995–1007CrossRefGoogle Scholar
  50. 50.
    Escalante HJ, Morales EF, Sucar LE (2016) A Naive Bayes baseline for early gesture recognition. Pattern Recogn Lett 73:91–99CrossRefGoogle Scholar
  51. 51.
    Eum H, Yoon C, Lee H, Park M (2015) Continuous human action recognition using depth-MHI-HOG and a spotter model. Sensors 15(3):5197–5227CrossRefGoogle Scholar
  52. 52.
    Fan X, Tjahjadi T (2015) A spatial–temporal framework based on histogram of gradients and optical flow for facial expression recognition in video sequences. Pattern Recogn 48(11):3407–3416CrossRefGoogle Scholar
  53. 53.
    Garcia-Ceja E, Galván-Tejada CE, Brena R (2018) Multi-view stacking for activity recognition with sound and accelerometer data. Inf Fusion 40:45–56CrossRefGoogle Scholar
  54. 54.
    Ghafouri S, Seyedarabi H (2013, May) Hybrid method for hand gesture recognition based on combination of Haar-like and HOG features. In: 2013 21st Iranian conference on electrical engineering (ICEE). IEEE, pp 1–4Google Scholar
  55. 55.
    Gharasuie MM, Seyedarabi H (2013, September) Real-time dynamic hand gesture recognition using hidden Markov models. In: 2013 8th Iranian conference on machine vision and image processing (MVIP). IEEE, pp 194–199Google Scholar
  56. 56.
    Ghotkar A, Vidap P, Deo K (2016) Dynamic hand gesture recognition using hidden Markov model by microsoft kinect sensor. Int J Comput Appl 150(5):5–9Google Scholar
  57. 57.
    Gong W, Zhang X, Gonzàlez J, Sobral A, Bouwmans T, Tu C, Zahzah EH (2016) Human pose estimation from monocular images: a comprehensive survey. Sensors 16(12):1966CrossRefGoogle Scholar
  58. 58.
    Gorelick L, Blank M, Shechtman E, Irani M, Basri R (2007) Actions as space-time shapes. IEEE Trans Pattern Anal Mach Intell 29(12):2247–2253CrossRefGoogle Scholar
  59. 59.
    Gori I, Aggarwal JK, Matthies L, Ryoo MS (2016) Multitype activity recognition in robot-centric scenarios. IEEE Robot Autom Lett 1(1):593–600CrossRefGoogle Scholar
  60. 60.
    Guo M, Hou X, Ma Y, Wu X (2017) Facial expression recognition using ELBP based on covariance matrix transform in KLT. Multimed Tools Appl 76(2):2995–3010CrossRefGoogle Scholar
  61. 61.
    Gurav RM, Kadbe PK (2015) Vision based hand gesture recognition with haar classifier and AdaBoost algorithm. Int J Latest Trends Eng Technol (IJLTET) 5(2):155–160Google Scholar
  62. 62.
    Hilsenbeck B, Münch D, Grosselfinger AK, Hübner W, Arens M (2016, December) Action recognition in the longwave infrared and the visible spectrum using Hough forests. In: 2016 IEEE international symposium on multimedia (ISM). IEEE, pp 329–332Google Scholar
  63. 63.
    Hsieh CC, Liou DH (2015) Novel Haar features for real-time hand gesture recognition using SVM. J Real Time Image Proc 10(2):357–370CrossRefGoogle Scholar
  64. 64.
    Huang X, Wang SJ, Liu X, Zhao G, Feng X, Pietikainen M (2017) Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Trans Affect Comput 14(8):1–15Google Scholar
  65. 65.
    Islam MM, Siddiqua S, Afnan J (2017, February) Real time hand gesture recognition using different algorithms based on American sign language. In: 2017 IEEE international conference on imaging, vision & pattern recognition (icIVPR). IEEE, pp 1–6Google Scholar
  66. 66.
    Jalal A, Kim YH, Kim YJ, Kamal S, Kim D (2017) Robust human activity recognition from depth video using spatiotemporal multi-fused features. Pattern Recogn 61:295–308CrossRefGoogle Scholar
  67. 67.
    Jampour M, Lepetit V, Mauthner T, Bischof H (2017) Pose-specific non-linear mappings in feature space towards multiview facial expression recognition. Image Vis Comput 58:38–46CrossRefGoogle Scholar
  68. 68.
    Jeni LA, Cohn JF, Kanade T (2017) Dense 3D face alignment from 2D video for real-time use. Image Vis Comput 58:13–24CrossRefGoogle Scholar
  69. 69.
    Ji X, Cheng J, Tao D, Wu X, Feng W (2017) The spatial Laplacian and temporal energy pyramid representation for human action recognition using depth sequences. Knowl Based Syst 122:64–74CrossRefGoogle Scholar
  70. 70.
    Joshi A, Ghosh S, Betke M, Sclaroff S, Pfister H (2017) Personalizing gesture recognition using hierarchical bayesian neural networks. In: Proceedings of IEEE conference on computer vision and pattern recognition (CVPR)Google Scholar
  71. 71.
    Kabir MH, Salekin MS, Uddin MZ, Abdullah-Al-Wadud M (2017) Facial expression recognition from depth video with patterns of oriented motion flow. IEEE Access 5:8880–8889CrossRefGoogle Scholar
  72. 72.
    Kamal S, Jalal A (2016) A hybrid feature extraction approach for human detection, tracking and activity recognition using depth sensors. Arab J Sci Eng 41(3):1043–1051CrossRefGoogle Scholar
  73. 73.
    Kamal S, Jalal A, Kim D (2016) Depth images-based human detection, tracking and activity recognition using spatiotemporal features and modified HMM. J Electr Eng Technol 11(3):1921–1926Google Scholar
  74. 74.
    Kamarol SKA, Jaward MH, Kälviäinen H, Parkkinen J, Parthiban R (2017) Joint facial expression recognition and intensity estimation based on weighted votes of image sequences. Pattern Recogn Lett 92:25–32CrossRefGoogle Scholar
  75. 75.
    Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings. Fourth IEEE international conference on automatic face and gesture recognition, 2000. IEEE, pp 46–53Google Scholar
  76. 76.
    Kataoka H, Miyashita Y, Hayashi M, Iwata K, Satoh Y (2016) Recognition of transitional action for short-term action prediction using discriminative temporal CNN feature. In: British machine vision conference (BMVC)Google Scholar
  77. 77.
    Kerola T, Inoue N, Shinoda K (2017) Cross-view human action recognition from depth maps using spectral graph sequences. Comput Vis Image Underst 154:108–126CrossRefGoogle Scholar
  78. 78.
    Kim TK, Cipolla R (2009) Canonical correlation analysis of video volume tensors for action categorization and detection. IEEE Trans Pattern Anal Mach Intell 31(8):1415–1428CrossRefGoogle Scholar
  79. 79.
    Kolekar MH, Dash DP (2016, November) Hidden Markov model based human activity recognition using shape and optical flow based features. In: Region 10 conference (TENCON), 2016. IEEE, pp 393–397Google Scholar
  80. 80.
    Krishna A, Strack F (2017) Reflection and impulse as determinants of human behavior. In: Meusburger P, Werlen B, Suarsana L (eds) Knowledge and action, vol 9. Springer, Cham, pp 145–167CrossRefGoogle Scholar
  81. 81.
    Kumar P, Happy SL, Routray A (2016, December) A real-time robust facial expression recognition system using HOG features. In: International Conference on computing, analytics and security trends (CAST). IEEE, pp 289–293Google Scholar
  82. 82.
    Kumari P, Mathew L, Syal P (2017) Increasing trend of wearables and multimodal interface for human activity monitoring: a review. Biosens Bioelectron 90:298–307CrossRefGoogle Scholar
  83. 83.
    Kung SH, Zohdy MA, Bouchaffra D (2016) 3D HMM-based facial expression recognition using histogram of oriented optical flow. Trans Mach Learn Artif Intell 3(6):42Google Scholar
  84. 84.
    Kurylosky P, Giani A, Giannantonio R, Gilani K, Gravina R, Seppa VP, Seto E, Shia V, Wang C, Yan P, Yang AY, Hyttinen J, Sastry S, Wicker S, Bajcsy R (2009) DexterNet: an open platform for heterogeneous body sensor networks and its applications. In: Proceedings of the sixth international workshop on wearable and implantable body sensor networks, pp 92–97Google Scholar
  85. 85.
    Laptev I, Marszalek M, Schmid C, Rozenfeld B (2008, June) Learning realistic human actions from movies. In: CVPR 2008. IEEE conference on computer vision and pattern recognition, 2008. IEEE, pp 1–8Google Scholar
  86. 86.
    Li J, Zhang B (2016, November) Facial expression recognition based on Gabor and conditional random fields. In: 2016 IEEE 13th international conference on signal processing (ICSP). IEEE, pp 752–756Google Scholar
  87. 87.
    Li Q, Qiu Z, Yao T, Mei T, Rui Y, Luo J (2017) Learning hierarchical video representation for action recognition. Int J Multimed Inf Retr 6(1):85–98CrossRefGoogle Scholar
  88. 88.
    Limonchik B, Amdur G (2017) 3D model-based data augmentation for hand gesture recognition. Available at: http://cs231n.stanford.edu/reports/2017/pdfs/218.pdf
  89. 89.
    Liang J, Xu C, Feng Z, Ma X (2015) Hidden Markov model decision forest for dynamic facial expression recognition. Int J Pattern Recognit Artif Intell 29(07):1556010MathSciNetCrossRefGoogle Scholar
  90. 90.
    Lin SJ, Chao MH, Lee CY, Yang CS (2016) Human action recognition using motion history image based temporal segmentation. Int J Pattern Recognit Artif Intell 30(06):1655017CrossRefGoogle Scholar
  91. 91.
    Littlewort GC, Bartlett MS, Lee K (2009) Automatic coding of facial expressions displayed during posed and genuine pain. Image Vis Comput 27:1797–1803CrossRefGoogle Scholar
  92. 92.
    Liu C, Chen YY, Fu LC (2016a) Robust dynamic hand gesture recognition system with sparse steric haar-like feature for human robot interaction. In: 55th annual conference of the society of instrument and control engineers of Japan (SICE), 2016. IEEE, pp 148–153Google Scholar
  93. 93.
    Liu J, Wang Z, Zhong L, Wickramasuriya J, Vasudevan V (2009) uWave: accelerometer-based personalized gesture recognition and its applications. In: IEEE international conference on pervasive computing and communications, pp 1–9Google Scholar
  94. 94.
    Liu X, Zhang M, Richardson A, Lucas T, Van Der Spiegel J (2016) The virtual Trackpad: an electromyography-based, wireless, real-time, low-power, embedded hand gesture recognition system using an event-driven artificial neural network. IEEE Trans Circuits Syst II Express Briefs.  https://doi.org/10.1109/tcsii.(2016).2635674 CrossRefGoogle Scholar
  95. 95.
    Liu Y, Li Y, Ma X, Song R (2017) Facial expression recognition with fusion features extracted fromsalient facial areas. Sensors 17(4):712CrossRefGoogle Scholar
  96. 96.
    Liu Z, Zhang C, Tian Y (2016) 3d-based deep convolutional neural network for action recognition with depth sequences. Image Vis Comput 55(2):93–100CrossRefGoogle Scholar
  97. 97.
    Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010, June) The extended Cohn–Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW). IEEE, pp 94–101Google Scholar
  98. 98.
    Luo RC, Chou YT, Liao CT, Lai CC, Tsai AC (2007) NCCU security warrior: an intelligent security robot system. In: 33rd annual conference of the IEEE industrial electronics society, pp 2960–2965Google Scholar
  99. 99.
    Luo Y, Wu CM, Zhang Y (2013) Facial expression recognition based on fusion feature of PCA and LBP with SVM. Opt Int J Light Electron Opt 124(17):2767–2770CrossRefGoogle Scholar
  100. 100.
    Luvizon DC, Tabia H, Picard D (2017) Learning features combination for human action recognition from skeleton sequences. Pattern Recogn Lett. 99:13–20.  https://doi.org/10.1016/j.patrec.(2017).02.001 CrossRefGoogle Scholar
  101. 101.
    Ma S, Zhang J, Sclaroff S, Ikizler-Cinbis N, Sigal L (2017) Space-time tree ensemble for action recognition and localization. Int J Comput Vision.  https://doi.org/10.1007/s11263-016-0980-8 CrossRefGoogle Scholar
  102. 102.
    Marcel S, Bernier O (1999, March) Hand posture recognition in a body-face centered space. In: International gesture workshop. Springer, Berlin, pp 97–100Google Scholar
  103. 103.
    Marcel S, Bernier O, Viallet JE, Collobert D (2000) Hand gesture recognition using input–output hidden markov models. In: Proceedings. Fourth IEEE international conference on automatic face and gesture recognition, 2000. IEEE, pp 456–461Google Scholar
  104. 104.
    Marszalek M, Laptev I, Schmid C (2009, June) Actions in context. In: CVPR 2009 IEEE conference on computer vision and pattern recognition, 2009. IEEE, pp 2929–2936Google Scholar
  105. 105.
    Martinez B, Valstar MF, Jiang B, Pantic M (2017) Automatic analysis of facial actions: a survey. IEEE Trans Affect Comput 99:1–11.  https://doi.org/10.1109/TAFFC.2017.2731763 CrossRefGoogle Scholar
  106. 106.
    Maruvada S (2017) 3-D hand gesture recognition with different temporal behaviors using HMM and Kinect. Master Thesis, University of Magdeburg, Germany. http://wwwisg.cs.ovgu.de/sim/files/theses/maruvada.pdf
  107. 107.
    Medina-Catzin JL, Martin-Gonzalez A, Brito-Loeza C, Uc-Cetina V (2017) Body gestures recognition system to control a service robot. Int J Inf Tech Comput Sci 9:69–76.  https://doi.org/10.5815/ijitcs.2017.09.07 CrossRefGoogle Scholar
  108. 108.
    Molchanov P, Yang X, Gupta S, Kim K, Tyree S, Kautz, J (2016) Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4207–4215Google Scholar
  109. 109.
    Mueid RM, Ahmed C, Ahad MAR (2016) Pedestrian activity classification using patterns of motion and histogram of oriented gradient. J Multimodal User Interfaces 10(4):299–305CrossRefGoogle Scholar
  110. 110.
    Nakamura Y, Kimura Y, Ye Y, Ohta Y (1998) MMID: multimodal multi-view integrated database for human behavior understanding. In: Proceedings of third IEEE international conference on automatic face and gesture recognition, pp 540–545Google Scholar
  111. 111.
    Nascimento TH, Soares FAA, Irani PP, de Oliveira LLG, da Silva Soares A (2017, July) Method for text entry in smartwatches using continuous gesture recognition. In: 2017 IEEE 41st annual Computer software and applications conference (COMPSAC), vol 2. IEEE, pp 549–554Google Scholar
  112. 112.
    Nigam S, Singh R, Misra AK (2018) Efficient facial expression recognition using histogram of oriented gradients in wavelet domain, Multimed Tools Appl.  https://doi.org/10.1007/s11042-018-6040-3
  113. 113.
    Nigam S, Khare A (2016) Integration of moment invariants and uniform local binary patterns for human activity recognition in video sequences. Multimed Tools Appl 75(24):17303–17332CrossRefGoogle Scholar
  114. 114.
    Nigam S, Khare A (2015) Multiscale local binary patterns for facial expression-based human emotion recognition. In: International conference on computational vision and robotics (ICCVR). Springer, India, pp 71–77Google Scholar
  115. 115.
    Neeru N, Kaur L (2016) Modified SIFT descriptors for face recognition under different emotions. J Eng 2016 Article ID 9387545Google Scholar
  116. 116.
    Owusu E, Zhan Y, Mao QR (2014) A neural-AdaBoost based facial expression recognition system. Expert Syst Appl 41(7):3383–3390CrossRefGoogle Scholar
  117. 117.
    Pantic M, Valstar M, Rademaker R, Maat L (2005, July) Web-based database for facial expression analysis. In: IEEE international conference on multimedia and expo, 2005. ICME 2005. IEEEGoogle Scholar
  118. 118.
    Parisi GI, Tani J, Weber C, Wermter S (2016) Emergence of multimodal action representations from neural network self-organization. Cogn Syst Res.  https://doi.org/10.1016/j.cogsys.2016.08.002 CrossRefGoogle Scholar
  119. 119.
    Parisi GI, Weber C, Wermter S (2015) Self-organizing neural integration of pose-motion features for human action recognition. Front Neurorobot.  https://doi.org/10.3389/fnbot.2015.00003 CrossRefGoogle Scholar
  120. 120.
    Parisi GI, Tani J, Weber C, Wermter S (2017) Lifelong learning of human actions with deep neural network self-organization. Neural Netw 96:137–149CrossRefGoogle Scholar
  121. 121.
    Petridis V, Deb B, Syrris V (2009) Detection and identification of human actions using predictive modular neural networks. In: 17th mediterranean conference on control and automation, pp 406–411Google Scholar
  122. 122.
    Plouffe G, Cretu AM (2016) Static and dynamic hand gesture recognition in depth data using dynamic time warping. IEEE Trans Instrum Meas 65(2):305–316CrossRefGoogle Scholar
  123. 123.
    Poria S, Cambria E, Bajpai R, Hussain A (2017) A review of affective computing: from unimodal analysis to multimodal fusion. Inf Fusion 37:98–125CrossRefGoogle Scholar
  124. 124.
    Qian H, Zhou J, Mao Y, Yuan Y (2017) Recognizing human actions from silhouettes described with weighted distance metric and kinematics. Multimed Tools Appl 76(21):21889–21910CrossRefGoogle Scholar
  125. 125.
    Ragheb H, Velastin S, Remagnino P, Ellis T (2008) ViHASi: virtual human action silhouette data for the performance evaluation of silhouette-based action recognition methods. In: Second ACM/IEEE international conference on distributed smart cameras, pp 1–10Google Scholar
  126. 126.
    Raju MIH, Ananna SS, Meraz SSI, Azam MZ, Serikawa S, Ahad MAR (2017) Human action recognition: a template matching-based approach. J Inst Ind Appl Eng 5(1):15–23Google Scholar
  127. 127.
    Raman N, Maybank SJ (2016) Activity recognition using a supervised non-parametric hierarchical HMM. Neurocomputing 199:163–177CrossRefGoogle Scholar
  128. 128.
    Rautaray SS, Agrawal A (2012) Real time gesture recognition system for interaction in dynamic environment. Procedia Technol 4:595–599CrossRefGoogle Scholar
  129. 129.
    Ren F, Huang Z (2015) Facial expression recognition based on AAM–SIFT and adaptive regional weighting. IEE J Trans Electr Electron Eng 10(6):713–722CrossRefGoogle Scholar
  130. 130.
    Reyes M, Dominguez G, Escalera S (2011) Feature weighting in dynamic time warping for gesture recognition in depth data. In: IEEE international conference on computer vision, pp 1182–1188Google Scholar
  131. 131.
    Richard A, Gall J (2016) A bag-of-words equivalent recurrent neural network for action recognition. Comput Vis Image Underst 156:79–91CrossRefGoogle Scholar
  132. 132.
    Rodriguez MD, Ahmed J, Shah M (2008, June) Action mach a spatio-temporal maximum average correlation height filter for action recognition. In: CVPR 2008. IEEE conference on computer vision and pattern recognition, 2008. IEEE, pp 1–8Google Scholar
  133. 133.
    Rodriguez M, Orrite C, Medrano C, Makris D (2016) One-shot learning of human activity with an MAP adapted GMM and simplex-HMM. IEEE Trans Cybern.  https://doi.org/10.1109/tcyb.(2016).2558447 CrossRefGoogle Scholar
  134. 134.
    Rodriguez M, Orrite C, Medrano C, Makris D (2017, July) Fast simplex-HMM for one-shot learning activity recognition. In: 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 1259–1266Google Scholar
  135. 135.
    Ruan J, Yin J, Chen Q, Chen G (2014) Facial expression recognition based on gabor wavelet transform and relevance vector machine. J Inf Comput Sci 11(1):295–302CrossRefGoogle Scholar
  136. 136.
    Ryu B, Rivera AR, Kim J, Chae O (2017) Local directional ternary pattern for facial expression recognition. IEEE Trans Image Process 26(12):6006–6018MathSciNetCrossRefGoogle Scholar
  137. 137.
    Saha A, Wu QMJ (2010) Facial expression recognition using curvelet based local binary patterns. In: IEEE international conference on acoustics speech and signal processing (ICASSP), pp 2470–2473Google Scholar
  138. 138.
    Saha S, Lahiri R, Konar A, Banerjee B, Nagar AK (2017, May) HMM-based gesture recognition system using kinect sensor for improvised human–computer interaction. In: 2017 international joint conference on neural networks (IJCNN). IEEE, pp 2776–2783Google Scholar
  139. 139.
    Sandbach G, Zafeiriou S, Pantic M, Rueckert D (2012) Recognition of 3D facial expression dynamics. Image Vis Comput 30(10):762–773CrossRefGoogle Scholar
  140. 140.
    Sariyanidi E, Gunes H, Cavallaro A (2017) Learning bases of activity for facial expression recognition. IEEE Trans Image Process 26(4):1965–1978MathSciNetCrossRefGoogle Scholar
  141. 141.
    Schuldt C, Laptev I, Caputo B (2004) Recognizing human actions: a local SVM approach. In: 17th international conference on pattern recognition, vol 3, pp 32–36Google Scholar
  142. 142.
    Selvam S, Gnanadurai D (2016) Shape-based features for reliable action recognition using spectral regression discriminant analysis. Int J Signal Imaging Syst Eng 9(6):379CrossRefGoogle Scholar
  143. 143.
    Shah M, Jain R (eds) (2013) Motion-based recognition, vol 9. Springer, BerlinGoogle Scholar
  144. 144.
    Shan C, Gong S, McOwan P (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27:803–816CrossRefGoogle Scholar
  145. 145.
    Sharma CM, Kushwaha AKS, Nigam S, Khare A (2011a) Automatic human activity recognition in video using background modeling and spatio-temporal template matching based technique. In: ACM international conference on advances in computing and artificial intelligence, pp 97–101Google Scholar
  146. 146.
    Sharma CM, Kushwaha AKS, Nigam S, Khare A (2011b) On human activity recognition in video sequences. In: IEEE international conference on computer and communication technology, pp 152–158Google Scholar
  147. 147.
    Sheng N, Cai Y, Zhan C, Qiu C, Cui Y, Gao X (2016, October) 3D facial expression recognition using distance features and LBP features based on automatically detected keypoints. In: International congress on image and signal processing, biomedical engineering and informatics (CISP-BMEI). IEEE, pp 396–401Google Scholar
  148. 148.
    Shi Y, Tian Y, Wang Y, Huang T (2017) Sequential deep trajectory descriptor for action recognition with three-stream CNN. IEEE Trans Multimed.  https://doi.org/10.1109/tmm.(2017).2666540 CrossRefGoogle Scholar
  149. 149.
    Siddiqi MH, Ali R, Idris M, Khan AM, Kim ES, Whang MC, Lee S (2016) Human facial expression recognition using curvelet feature extraction and normalized mutual information feature selection. Multimed Tools Appl 75(2):935–959CrossRefGoogle Scholar
  150. 150.
    Siddiqi MH, Ali R, Khan AM, Park YT, Lee S (2015) Human facial expression recognition using stepwise linear discriminant analysis and hidden conditional random fields. IEEE Trans Image Process 24(4):1386–1398MathSciNetCrossRefGoogle Scholar
  151. 151.
    Singh B, Marks TK, Jones M, Tuzel O, Shao M (2016) A multi-stream bi-directional recurrent neural network for fine-grained action detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1961–1970Google Scholar
  152. 152.
    Singh D, Mohan CK (2017) Graph formulation of video activities for abnormal activity recognition. Pattern Recogn 65:265–272CrossRefGoogle Scholar
  153. 153.
    Singh S, Arora C, Jawahar CV (2017) Trajectory aligned features for first person action recognition. Pattern Recogn 62:45–55CrossRefGoogle Scholar
  154. 154.
    Singh S, Velastin SA, Ragheb H (2010, August) Muhavi: A multicamera human action video dataset for the evaluation of action recognition methods. In: 2010 seventh IEEE international conference on advanced video and signal based surveillance (AVSS), pp 48–55Google Scholar
  155. 155.
    Singha J, Laskar RH (2017) Hand gesture recognition using two-level speed normalization, feature selection and classifier fusion. Multimed Syst 23(4):499–514CrossRefGoogle Scholar
  156. 156.
    Singha J, Laskar RH (2016) Recognition of global hand gestures using self co-articulation information and classifier fusion. J Multimodal User Interfaces 10(1):77–93CrossRefGoogle Scholar
  157. 157.
    Smeets D, Keustermans J, Vandermeulen D, Suetens P (2013) meshSIFT: local surface features for 3D face recognition under expression variations and partial data. Comput Vis Image Underst 117(2):158–169CrossRefGoogle Scholar
  158. 158.
    Soares ADS, Apolinário Jr AL (2017) Real-time 3D gesture recognition using dynamic time warping and simplification methods. J WSCG 25:59–66Google Scholar
  159. 159.
    Sohn MK, Lee SH, Kim DJ, Kim B, Kim H (2012, November) A comparison of 3D hand gesture recognition using dynamic time warping. In: Proceedings of the 27th conference on image and vision computing New Zealand. ACM, pp. 418–422Google Scholar
  160. 160.
    Sreekanth NS, Narayanan NK, Bangalore C (2017) Static hand gesture recognition using mon-vision based techniques. Int J Innov Comput Sci Eng 4(2):33–41Google Scholar
  161. 161.
    Sridevi K, Sundarambal M, Muralidharan K, Josephine RL (2017, January) FPGA implementation of hand gesture recognition system using neural networks. In: 2017 11th international conference on intelligent systems and control (ISCO). IEEE, pp 34–39Google Scholar
  162. 162.
    Stein S, McKenna SJ (2017) Recognising complex activities with histograms of relative tracklets. Comput Vis Image Underst 154:82–93CrossRefGoogle Scholar
  163. 163.
    Stergiou N (ed) (2016) Nonlinear analysis for human movement variability. CRC Press, Boca RatonGoogle Scholar
  164. 164.
    Sung J, Ponce C, Selman B, Saxena A (2012) Unstructured human activity detection from RGBD images. In: 2012, IEEE international conference on robotics and automation (ICRA).  https://doi.org/10.1109/icra.(2012).6224591
  165. 165.
    Takeo K, Collins RT, Lipton AJ, Fujiyoshi H, Duggins D (2000) A system for video surveillance and monitoring: VSAM final report. CMU-RI-TR-00-12, Technical Report, Carnegie UniversityGoogle Scholar
  166. 166.
    Tong Y, Shen Y, Gao B, Sun F, Chen R, Xu Y (2017) A Noisy–Robust approach for facial expression recognition. KSII Trans Internet Inf Syst (TIIS) 11(4):2124–2148Google Scholar
  167. 167.
    Triesch J, Von Der Malsburg C (1996 October) Robust classification of hand postures against complex backgrounds. In: Proceedings of the second IEEE international conference on automatic face and gesture recognition, pp 170–175Google Scholar
  168. 168.
    Triesch J, Von Der Malsburg C (2001) A system for person-independent hand posture recognition against complex backgrounds. IEEE Trans Pattern Anal Mach Intell 23(12):1449–1453CrossRefGoogle Scholar
  169. 169.
    Tripathi RK, Jalal AS, Agrawal SC (2017) Suspicious human activity recognition: a review. Artif Intell Rev.  https://doi.org/10.1007/s10462-017-9545-7 CrossRefGoogle Scholar
  170. 170.
    Truong A, Zaharia T (2017) Laban movement analysis and hidden Markov models for dynamic 3D gesture recognition. EURASIP J Image Video Process 2017(1):52CrossRefGoogle Scholar
  171. 171.
    Tsai HH, Chang YC (2017) Facial expression recognition using a combination of multiple facial features and support vector machine. Soft Comput.  https://doi.org/10.1007/s00500-017-2634-3
  172. 172.
    Uddin MZ, Hassan MM (2015) A depth video-based facial expression recognition system using radon transform, generalized discriminant analysis, and hidden Markov model. Multimed Tools Appl 74(11):3675–3690CrossRefGoogle Scholar
  173. 173.
    Varatharajan R, Manogaran G, Priyan MK, Sundarasekar R (2017) Wearable sensor devices for early detection of Alzheimer disease using dynamic time warping algorithm. Cluster Comput.  https://doi.org/10.1007/s10586-017-0977-2
  174. 174.
    Wang H, Kläser A, Schmid C, Liu C (2013) Dense trajectories and motion boundary descriptors for action recognition. Int J Comput Vis 103(1):60–79MathSciNetCrossRefGoogle Scholar
  175. 175.
    Wang H, Yang W, Yuan C, Ling H, Hu W (2017) Human activity prediction using temporally-weighted generalized time warping. Neurocomputing 225:139–147CrossRefGoogle Scholar
  176. 176.
    Wang H, Yang Y, Yang E, Deng C (2017) Exploring hybrid spatio-temporal convolutional networks for human action recognition. Multimed Tools Appl.  https://doi.org/10.1007/s11042-017-4514-3 CrossRefGoogle Scholar
  177. 177.
    Wang X, Jin C, Liu W, Hu M, Xu L, Ren F (2013, December) Feature fusion of hog and wld for facial expression recognition. In: 2013 IEEE/SICE international symposium on system integration (SII). IEEE, pp 227–232Google Scholar
  178. 178.
    Wang Y, Huang K, Tan T (2007, June) Human activity recognition based on R transform. In: 2007 IEEE computer society conference on computer vision and pattern recognition (CVPR’07), pp 1–8Google Scholar
  179. 179.
    Wang Z, Wu D, Gravina R, Fortino G, Jiang Y, Tang K (2017) Kernel fusion based extreme learning machine for cross-location activity recognition. Inf Fusion 37:1–9CrossRefGoogle Scholar
  180. 180.
    Weinland D, Ronfard R, Boyer E (2006) Free viewpoint action recognition using motion history volumes. Comput Vis Image Underst 104(2):249–257CrossRefGoogle Scholar
  181. 181.
    Wu D, Pigou L, Kindermans PJ, Le NDH, Shao L, Dambre J, Odobez JM (2016) Deep dynamic neural networks for multimodal gesture segmentation and recognition. IEEE Trans Pattern Anal Mach Intell 38(8):1583–1597CrossRefGoogle Scholar
  182. 182.
    Xie C, Li C, Zhang B, Chen C, Han J (2017) Deep fisher discriminant learning for mobile hand gesture recognition. arXiv preprint arXiv:1707.03692
  183. 183.
    Xie R, Cao J (2016) Accelerometer-based hand gesture recognition by neural network and similarity matching. IEEE Sens J 16(11):4537–4545CrossRefGoogle Scholar
  184. 184.
    Xu D, Wu X, Chen YL, Xu Y (2015) Online dynamic gesture recognition for human robot interaction. J Intell Rob Syst 77(3–4):583–596CrossRefGoogle Scholar
  185. 185.
    Xu W, Miao Z, Zhang XP, Tian Y (2017) A hierarchical spatio-temporal model for human activity recognition. IEEE Trans Multimed.  https://doi.org/10.1109/tmm.(2017).2674622 CrossRefGoogle Scholar
  186. 186.
    Xu X, Quan C, Ren F (2015b, August) Facial expression recognition based on Gabor Wavelet transform and Histogram of Oriented Gradients. In: 2015 IEEE international conference on mechatronics and automation (ICMA). IEEE, pp 2117–2122Google Scholar
  187. 187.
    Xu Y, Dai Y (2017) Review of hand gesture recognition study and application. Contemp Eng Sci 10(8):375–384CrossRefGoogle Scholar
  188. 188.
    Yu Y, Bi S, Mo Y, Qiu W (2016, June) Real-time gesture recognition system based on Camshift algorithm and Haar-like feature. In: 2016, IEEE international conference on cyber technology in automation, control, and intelligent systems (CYBER). IEEE, pp 337–342Google Scholar
  189. 189.
    Zajdel W, Krijnders D, Andringa T, Gavrila DM (2007) CASSANDRA: audio-video sensor fusion for aggression detection. In: IEEE conference on advanced video and signal based surveillance, pp 200–205Google Scholar
  190. 190.
    Zhang J, Li W, Ogunbona PO, Wang P, Tang C (2016) RGB-D-based action recognition datasets: a survey. Pattern Recogn 60:86–105CrossRefGoogle Scholar
  191. 191.
    Zhang XH, Wang JJ, Wang X, Ma XL (2016, June). Improvement of dynamic hand gesture recognition based on HMM algorithm. In: 2016 international conference on information system and artificial intelligence (ISAI). IEEE, pp 401–406Google Scholar
  192. 192.
    Zhang L, Wang Z, Yao T, Mei T, Feng DD (2017) Exploiting spatial–temporal context for trajectory based action video retrieval. Multimed Tools Appl.  https://doi.org/10.1007/s11042-017-4353-2 CrossRefGoogle Scholar
  193. 193.
    Zhao K, Chu WS, De la Torre F, Cohn JF, Zhang H (2016) Joint patch and multi-label learning for facial action unit and holistic expression recognition. IEEE Trans Image Process 25(8):3931–3946MathSciNetCrossRefGoogle Scholar
  194. 194.
    Zhao L, Wang Z, Zhang G (2017) Facial expression recognition from video sequences based on spatial–temporal motion local binary pattern and gabor multiorientation fusion histogram. In: Mathematical problems in engineering.  https://doi.org/10.1155/(2017)/7206041
  195. 195.
    Zheng W, Tang H, Lin Z, Huang TS (2009) A novel approach to expression recognition from non-frontal face images. In: Proceedings of the IEEE international conference on computer vision, pp 1901–1908Google Scholar
  196. 196.
    Zhu G, Zhang L, Shen P, Song J (2017) Multimodal gesture recognition using 3D convolution and convolutional LSTM. IEEE Access.  https://doi.org/10.1109/access.(2017).2684186 CrossRefGoogle Scholar
  197. 197.
    Ziaie P, Müller T, Foster ME, Knoll A (2008) A Naïve Bayes classifier with distance weighting for hand-gesture recognition. In: Advances in computer science and engineering. Springer, Berlin, Heidelberg, pp 308–315. Available online at: http://mediatum.ub.tum.de/doc/1289362/269163.pdf
  198. 198.
    Ziaie P, Müller T, Foster ME, Knoll A (2009) Using a Naïve Bayes classifier based on K-nearest neighbours with distance weighting for static hand-gesture recognition in a human–robot dialog system. Adv Comput Sci Eng Commun Comput Inf Sci 6(1):308–315Google Scholar
  199. 199.
    Zong Z (2017) Efficient human face recognition method under subtle SIFT features using optimized K-means. Int J Signal Process Image Process Pattern Recogn 10(7):195–204Google Scholar

Copyright information

© CIMNE, Barcelona, Spain 2018

Authors and Affiliations

  1. 1.Computer Science and Engineering DepartmentSP Memorial Institute of TechnologyKaushambiIndia
  2. 2.Department of Computer ScienceBanasthali VidyapithBanasthaliIndia

Personalised recommendations