Abstract
The psychological research found that human eyes could serve as a sensitive indicator of emotional response. Pupillary response has been used to analyze the affective video content in previous studies, but the performance is not good enough. In this paper, we propose a novel method for implicit affective video tagging using pupillary response. The issue of pupil size difference between subjects has not been effectively solved, which seriously affected the performance of the implicit affective video tagging. In our method, we first define the pupil diameter baseline of each subject to diminish individual difference on pupil size. Besides, the probabilistic support vector machine (SVM) and long short term memory (LSTM) network are used to extract valuable information and output the probability estimates based on the proposed global features and sequence features obtained from the pupil dilation ratio time-series data, respectively. The final decision is made by combining the probability estimates from these two models based on the sum rule. In empirical validation, we evaluate the proposed method on a standard dataset MAHNOB-HCI. The experimental results show that the proposed method achieves better classification accuracy compared with the existing method.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Keras: an open source neural network library. http://keras.io
Acar, E., Hopfgartner, F., Albayrak, S.: Understanding affective content of music videos through learned representations. In: Gurrin, C., Hopfgartner, F., Hurst, W., Johansen, H., Lee, H., O’Connor, N. (eds.) MMM 2014. LNCS, vol. 8325, pp. 303–314. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-04114-8_26
Baxter, M.: Notes on cinemetric data analysis (2014)
Belhumeur, P.N., Kriegman, D.J.: What is the set of images of an object under all possible illumination conditions? Int. J. Comput. Vis. 28(3), 245–260 (1998)
Bradley, M.M., Miccoli, L., Escrig, M.A., Lang, P.J.: The pupil as a measure of emotional arousal and autonomic activation. Psychophysiology 45(4), 602–607 (2008)
Chanel, G., Kierkels, J.J., Soleymani, M., Pun, T.: Short-term emotion assessment in a recall paradigm. Int. J. Hum.-Comput. Stud. 67(8), 607–627 (2009)
Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. (TIST) 2(3), 1–27 (2011)
Cleveland, W.S.: Robust locally weighted regression and smoothing scatterplots. J. Am. Stat. Assoc. 74(368), 829–836 (1979)
Fang, Y., Lin, W., Chen, Z., Tsai, C., Lin, C.: A video saliency detection model in compressed domain. IEEE Trans. Circ. Syst. Video Technol. 24(1), 27–38 (2014)
Fontaine, J.R., Scherer, K.R., Roesch, E.B., Ellsworth, P.C.: The world of emotions is not two-dimensional. Psychol. Sci. 18(12), 1050–1057 (2007)
Gajraj, R., Doi, M., Mantzaridis, H., Kenny, G.: Analysis of the EEG bispectrum, auditory evoked potentials and the EEG power spectrum during repeated transitions from consciousness to unconsciousness. Br. J. Anaesth. 80(1), 46–52 (1998)
Guggisberg, A.G., Hess, C.W., Mathis, J.: The significance of the sympathetic nervous system in the pathophysiology of periodic leg movements in sleep. Sleep 30(6), 755–766 (2007)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Huang, T., Weng, R.C., Lin, C.J.: Generalized Bradley-Terry models and multi-class probability estimates. J. Mach. Learn. Res. 7(Jan), 85–115 (2006)
Iwasaki, M., Kellinghaus, C., Alexopoulos, A.V., Burgess, R.C., Kumar, A.N., Han, Y.H., Lüders, H.O., Leigh, R.J.: Effects of eyelid closure, blinks, and eye movements on the electroencephalogram. Clin. Neurophysiol. 116(4), 878–885 (2005)
Katti, H., Yadati, K., Kankanhalli, M., Tat-Seng, C.: Affective video summarization and story board generation using pupillary dilation and eye gaze. In: Proceedings of the International Symposium on Multimedia, pp. 319–326. IEEE (2011)
Kreibig, S.D.: Autonomic nervous system activity in emotion: a review. Biol. Psychol. 84(3), 394–421 (2010)
Lins, O.G., Picton, T.W., Berg, P., Scherg, M.: Ocular artifacts in EEG and event-related potentials I: scalp topography. Brain Topogr. 6(1), 51–63 (1993)
Ong, K., Kameyama, W.: Classification of video shots based on human affect. J. Inst. Image Inf. Telev. Eng. 63(6), 847–856 (2009)
Poursaberi, A., Araabi, B.N.: Iris recognition for partially occluded images: methodology and sensitivity analysis. EURASIP J. Appl. Sig. Process. 2007(1), 20 (2007)
Rainville, P., Bechara, A., Naqvi, N., Damasio, A.R.: Basic emotions are associated with distinct patterns of cardiorespiratory activity. Int. J. Psychophysiol. 61(1), 5–18 (2006)
Robinson, B.F., Epstein, S.E., Beiser, G.D., Braunwald, E.: Control of heart rate by the autonomic nervous system. Circ. Res. 19(2), 400–411 (1966)
Shao, L., Zhen, X., Tao, D., Li, X.: Spatio-temporal Laplacian pyramid coding for action recognition. IEEE Trans. Cybern. 44(6), 817–827 (2014)
Soleymani, M., Lichtenauer, J., Pun, T., Pantic, M.: A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 3(1), 42–55 (2012)
Tang, Y.Y., Ma, Y., Fan, Y., Feng, H., Wang, J., Feng, S., Lu, Q., Hu, B., Lin, Y., Li, J., et al.: Central and autonomic nervous system interaction is altered by short-term meditation. Proc. Natl. Acad. Sci. 106(22), 8865–8870 (2009)
Tsukahara, J.S., Harrison, T.L., Engle, R.W.: The relationship between baseline pupil size and intelligence. Cogn. Psychol. 91, 109–123 (2016)
Wang, S., Ji, Q.: Video affective content analysis: a survey of state-of-the-art methods. IEEE Trans. Affect. Comput. 6(4), 410–430 (2015)
Wu, J., Zhong, S.H., Jiang, J., Yang, Y.: A novel clustering method for static video summarization. Multimedia Tools Appl., 1–17 (2016)
Yeasin, M., Bullot, B., Sharma, R.: Recognition of facial expressions and measurement of levels of interest from video. IEEE Trans. Multimedia 8(3), 500–508 (2006)
Zhao, S., Yao, H., Sun, X., Xu, P., Liu, X., Ji, R.: Video indexing and recommendation based on affective analysis of viewers. In: Proceedings of the 19th ACM international conference on Multimedia, pp. 1473–1476. ACM (2011)
Zhu, Y., Huang, X., Huang, Q., Tian, Q.: Large-scale video copy retrieval with temporal-concentration sift. Neurocomputing 187, 83–91 (2016)
Zhu, Y., Jiang, Z., Peng, J., Zhong, S.: Video affective content analysis based on protagonist via convolutional neural network. In: Chen, E., Gong, Y., Tie, Y. (eds.) PCM 2016. LNCS, vol. 9916, pp. 170–180. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-48890-5_17
Acknowledgements
This work was supported by the National Natural Science Foundation of China (No. 61502311, No. 61672358), the (Key) Project of Department of Education of Guangdong Province (No. 2014GKCG031, No. 12JGXM-MS29, No. 2015SQXX0), the Natural Science Foundation of Guangdong Province (No. 2016A030310053), the Special Program for Applied Research on Super Computation of the NSFC-Guangdong Joint Fund (the second phase) (No. U1501501), the Shenzhen high-level overseas talents program, and the Tencent “Rhinoceros Birds” Scientific Research Foundation for Young Teachers of Shenzhen University (2015, 2016).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this paper
Cite this paper
Gui, D., Zhong, Sh., Ming, Z. (2018). Implicit Affective Video Tagging Using Pupillary Response. In: Schoeffmann, K., et al. MultiMedia Modeling. MMM 2018. Lecture Notes in Computer Science(), vol 10705. Springer, Cham. https://doi.org/10.1007/978-3-319-73600-6_15
Download citation
DOI: https://doi.org/10.1007/978-3-319-73600-6_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-73599-3
Online ISBN: 978-3-319-73600-6
eBook Packages: Computer ScienceComputer Science (R0)