Introduction to Sequence Analysis for Human Behavior Understanding


This chapter introduces the sequence analysis problem in machine learning. The problem is formulated in terms of two major issues: The first is the classification (assignment of a label to an entire sequence of observations), and the second is the labeling (assignment of a label to each observation in a sequence). The chapter applies the framework of probabilistic graphical models to introduce two of the most important sequence analysis models, namely Hidden Markov Models and Conditional Random Fields, with particular attention to their factorization and their underlying independence assumptions. The introduction is completed with some details about inference and training as well as some pointers to the literature.


Hide Markov Model Bayesian Network Independence Assumption Conditional Random Field Markov Network 


  1. 1.
    Abbott, A.: Sequence analysis: New methods for old ideas. Annu. Rev. Sociol. 21, 93–113 (1995) CrossRefGoogle Scholar
  2. 2.
    Bakeman, R., Gottman, J.M.: Observing Interaction: An Introduction to Sequential Analysis. Cambridge University Press, Cambridge (1986) Google Scholar
  3. 3.
    Baldi, P., Brunak, S.: Bioinformatics: the machine learning approach. MIT Press, Cambridge (2001) MATHGoogle Scholar
  4. 4.
    Bilmes, J.: The concept of preference in conversation analysis. Lang. Soc. 17(2), 161–181 (1988) CrossRefGoogle Scholar
  5. 5.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006) MATHGoogle Scholar
  6. 6.
    Camastra, F., Vinciarelli, A.: Machine Learning for Audio, Image and Video Analysis: Theory and Applications. Springer, Berlin (2008) CrossRefMATHGoogle Scholar
  7. 7.
    Dietterich, T.: Machine learning for sequential data: A review. In: Caelli, T., Amin, A., Duin, R., de Ridder, D., Kamel, M. (eds.) Structural, Syntactic, and Statistical Pattern Recognition. Lecture Notes in Computer Science, vol. 2396, pp. 227–246. Springer, Berlin (2002) Google Scholar
  8. 8.
    Friedland, G., Vinyals, O., Huang, Y., Muller, C.: Prosodic and other long-term features for speaker diarization. IEEE Trans. Audio Speech Lang. Process. 17(5), 985–993 (2009) CrossRefGoogle Scholar
  9. 9.
    Heckerman, D.: A tutorial on learning with bayesian networks. In: Holmes, D., Jain, L. (eds.) Innovations in Bayesian Networks, pp. 33–82. Springer, Berlin (2008) CrossRefGoogle Scholar
  10. 10.
    Jelinek, F.: Statistical Methods for Speech Recognition. MIT Press, Cambridge (1997) Google Scholar
  11. 11.
    Jensen, F.V.: An Introduction to Bayesian Networks. UCL Press, London (1996) Google Scholar
  12. 12.
    Jensen, F.V., Nielsen, T.D.: Bayesian Networks and Decision Graphs. Springer, Berlin (2007) CrossRefMATHGoogle Scholar
  13. 13.
    Jordan, M.I.: Learning in Graphical Models. Kluwer Academic, Dordrecht (1998) MATHGoogle Scholar
  14. 14.
    Koller, D., Friedman, N.: Probabilistic Graphical Models: Principles and Techniques. MIT Press, Cambridge (2009) Google Scholar
  15. 15.
    Lafferty, J., McCallum, A., Pereira, F.: Conditional Random Fields: Probabilistic models for segmenting and labeling sequence data. In: Proceedings of the International Conference on Machine Learning, pp. 282–289 (2001) Google Scholar
  16. 16.
    Liu, D.C., Nocedal, J.: On the limited memory BFGS method for large scale optimization. Math. Program. 45, 503–528 (1989) MathSciNetCrossRefMATHGoogle Scholar
  17. 17.
    Mermelstein, P.: Distance measures for speech recognition, psychological and instrumental. Pattern Recognition and Artificial Intelligence 116 (1976) Google Scholar
  18. 18.
    Morency, L.P., Quattoni, A., Darrell, T.: Latent-dynamic discriminative models for continuous gesture recognition. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE Press, New York (2007) CrossRefGoogle Scholar
  19. 19.
    Murphy, K.: An introduction to graphical models. Technical Report, University of British Columbia (2001) Google Scholar
  20. 20.
    Pearl, J.: Bayesian networks: A model of self-activated memory for evidential reasoning. Computer Science Department, University of California (1985) Google Scholar
  21. 21.
    Poggi, I., D’Errico, F.: Cognitive modelling of human social signals. In: Proceedings of the 2nd International Workshop on Social Signal Processing, pp. 21–26 (2010) CrossRefGoogle Scholar
  22. 22.
    Quattoni, A., Wang, S., Morency, L.P., Collins, M., Darrell, T.: Hidden conditional random fields. IEEE Trans. Pattern Anal. Mach. Intell. 29(10), 1848–1852 (2007) CrossRefGoogle Scholar
  23. 23.
    Rabiner, L.R.: A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77(2), 257–286 (1989) CrossRefGoogle Scholar
  24. 24.
    Salamin, H., Vinciarelli, A., Truong, K., Mohammadi, G.: Automatic role recognition based on conversational and prosodic behaviour. In: Proceedings of the ACM International Conference on Multimedia, pp. 847–850. ACM, New York (2010) Google Scholar
  25. 25.
    Sansom, J., Thomson, P.: Fitting hidden semi-Markov models to breakpoint rainfall data. J. Appl. Probab. 38, 142–157 (2001) MathSciNetCrossRefGoogle Scholar
  26. 26.
    Sutton, C., McCallum, A.: An introduction to conditional random fields for relational learning. In: Getoor, L., Taskar, B. (eds.) Introduction to Statistical Relational Learning. MIT Press, Cambridge (2007) Google Scholar
  27. 27.
    Vinciarelli, A., Pantic, M., Bourlard, H.: Social signal processing: survey of an emerging domain. Image Vis. Comput. 27(12), 1743–1759 (2009) CrossRefGoogle Scholar
  28. 28.
    Wallach, H.M.: Conditional random fields: an introduction. Technical Report MS-CIS-04-21, Department of Computer and Information Science, University of Pennsylvania (2004) Google Scholar
  29. 29.
    Wu, Y., Huang, T.: Vision-based gesture recognition: A review. In: Braffort, A., Gherbi, R., Gibet, S., Teil, D., Richardson, J. (eds.) Gesture-Based Communication in Human-Computer Interaction. Lecture Notes in Computer Science, vol. 1739, pp. 103–115. Springer, Berlin (1999) CrossRefGoogle Scholar
  30. 30.
    Yedidia, J.S., Freeman, W.T., Weiss, Y.: Understanding belief propagation and its generalizations. In: Lakemeyer, G., Nebel, B. (eds.) Exploring Artificial Intelligence in the New Millennium, pp. 239–270. Morgan Kaufman, San Mateo (2003) Google Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  1. 1.School of Computing ScienceUniversity of GlasgowGlasgowScotland
  2. 2.Idiap Research InstituteMartignySwitzerland

Personalised recommendations