Unsupervised parsing of gaze data with a beta-process vector auto-regressive hidden Markov model

  • Joseph W. Houpt
  • Mary E. Frame
  • Leslie M. Blaha


The first stage of analyzing eye-tracking data is commonly to code the data into sequences of fixations and saccades. This process is usually automated using simple, predetermined rules for classifying ranges of the time series into events, such as “if the dispersion of gaze samples is lower than a particular threshold, then code as a fixation; otherwise code as a saccade.” More recent approaches incorporate additional eye-movement categories in automated parsing algorithms by using time-varying, data-driven thresholds. We describe an alternative approach using the beta-process vector auto-regressive hidden Markov model (BP-AR-HMM). The BP-AR-HMM offers two main advantages over existing frameworks. First, it provides a statistical model for eye-movement classification rather than a single estimate. Second, the BP-AR-HMM uses a latent process to model the number and nature of the types of eye movements and hence is not constrained to predetermined categories. We applied the BP-AR-HMM both to high-sampling rate gaze data from Andersson et al. (Behavior Research Methods 49(2), 1–22 2016) and to low-sampling rate data from the DIEM project (Mital et al., Cognitive Computation 3(1), 5–24 2011). Driven by the data properties, the BP-AR-HMM identified over five categories of movements, some which clearly mapped on to fixations and saccades, and others potentially captured post-saccadic oscillations, smooth pursuit, and various recording errors. The BP-AR-HMM serves as an effective algorithm for data-driven event parsing alone or as an initial step in exploring the characteristics of gaze data sets.


Eye tracking Fixation Saccade Smooth pursuit Indian buffet Vector auto-regressive Hidden Markov model 


Supplementary material (345 kb)
(ZIP 344 KB)


  1. Agtzidis, I., Startsev, M., & Dorr, M. (2016a). In the pursuit of (ground) truth: A hand-labelling tool for eye movements recorded during dynamic scene viewing. In Proceedings of the second workshop on visualizing eye tracking data (ETVIS).Google Scholar
  2. Agtzidis, I., Startsev, M., & Dorr, M. (2016b). Smooth pursuit detection based on multiple observers. In Proceedings of the ninth biennial acm symposium on eye tracking research & applications (pp. 303–306).Google Scholar
  3. Andersson, R., Larsson, L., Holmqvist, K., Stridh, M., & Nyström, M. (2016). One algorithm to rule them all? An evaluation and discussion of ten eye movement event-detection algorithms. Behavior Research Methods, 49(2), 1–22.Google Scholar
  4. Bahill, A. T., & Troost, B. T. (1979). Types of saccadic eye movements. Neurology, 29(8), 1150–1152.Google Scholar
  5. Baum, L. E., Petrie, T., Soules, G., & Weiss, N. (1970). A maximization technique occurring in the statistical analysis of probabilistic functions of Markov chains. The Annals of Mathematical Statistics, 41(1), 164–171.Google Scholar
  6. Berg, D. J., Boehnke, S. E., Marino, R. A., Munoz, D. P., & Itti, L. (2009). Free viewing of dynamic stimuli by humans and monkeys. Journal of Vision, 9(5), 19–19.Google Scholar
  7. Berg, D. J., Shen, J., & Itti, L. (2009). The eyeMarkup eye movement analysis tool. Retrieved from
  8. Duchowski, A. T. (2002). A breadth-first survey of eye-tracking applications. Behavior Research Methods, Instruments, & Computers, 34(4), 455–470.Google Scholar
  9. Engbert, R., & Kliegl, R. (2003). Microsaccades uncover the orientation of covert attention. Vision Research, 43(9), 1035–1045.Google Scholar
  10. Engbert, R., & Mergenthaler, K. (2006). Microsaccades are triggered by low retinal image slip. Proceedings of the National Academy of Sciences, 103(18), 7192–7197.Google Scholar
  11. Forney, G. D. (1973). The Viterbi algorithm. Proceedings of the IEEE, 61(3), 268–278.Google Scholar
  12. Fox, E. B. (2009). Bayesian nonparametric learning of complex dynamical phenomena (Ph.D. Thesis). Cambridge: MIT.Google Scholar
  13. Fox, E. B., Hughes, M., Sudderth, E., & Jordan, M. (2014). Joint modeling of multiple related time series via the beta process with application to motion capture segmentation. Annals of Applied Statistics, 8(3), 1281–1313.Google Scholar
  14. Fox, E. B., Sudderth, E. B., Jordan, M. I., & Willsky, A. S. (2011). A sticky HDP-HMM with application to speaker diarization. The Annals of Applied Statistics 5, 1020–1056.
  15. Gelman, A., & Loken, E. (2014). The statistical crisis in science data-dependent analysis - a garden of forking paths - explains why many statistically significant comparisons don’t hold up. American Scientist, 102(6), 460.Google Scholar
  16. Goldberg, J. H., & Schryver, J. C. (1995). Eye-gaze-contingent control of the computer interface: Methodology and example for zoom detection. Behavior Research Methods, Instruments, & Computers, 27(3), 338–350.Google Scholar
  17. Griffiths, T. L., & Ghahramani, Z. (2011). The Indian buffet process: An introduction and review. Journal of Machine Learning Research, 12(Apr), 1185–1224.Google Scholar
  18. Hayhoe, M., & Ballard, D. (2005). Eye movements in natural behavior. Trends in Cognitive Sciences, 9(4), 188–194.Google Scholar
  19. Henderson, J. M. (2016). The DIEM project. Retrieved from
  20. Hughes, M. C. (2016). Nonparametric Bayesian inference for sequential data. includes beta process hidden Markov model (BP-HMM). Implemented in Matlab. Retrieved from
  21. Hughes, M. C., Fox, E. B., & Sudderth, E. B. (2012). Effective split-merge Monte Carlo methods for nonparametric models of sequential data. In Advances in neural information processing systems (pp. 1295–1303).Google Scholar
  22. Kapoula, Z., Robinson, D., & Hain, T. (1986). Motion of the eye immediately after a saccade. Experimental Brain Research, 61(2), 386–394.Google Scholar
  23. Karpov, A., & Komogortsev, O. (2011). Automated classification and scoring of smooth pursuit eye movements in presence of fixations and saccades (Tech. Rep. No TR2011-11-23). HCI Lab, Texas State University.Google Scholar
  24. Komogortsev, O. V., Gobert, D. V., Jayarathna, S., Koh, D. H., & Gowda, S. M. (2010). Standardization of automated analyses of oculomotor fixation and saccadic behaviors. IEEE Transactions on Biomedical Engineering, 57(11), 2635–2645.Google Scholar
  25. Land, M. F., & Tatler, B. W. (2009). Looking and acting: Vision and eye movements in natural behaviour. Oxford: Oxford University Press.Google Scholar
  26. Larsson, L., Nyström, M., Andersson, R., & Stridh, M. (2015). Detection of fixations and smooth pursuit movements in high-speed eye-tracking data. Biomedical Signal Processing and Control, 18, 145–152.Google Scholar
  27. Larsson, L., Nyström, M., & Stridh, M. (2013). Detection of saccades and postsaccadic oscillations in the presence of smooth pursuit. IEEE Transactions on Biomedical Engineering, 60(9), 2484–2493.Google Scholar
  28. Mason, R. L. (1976). Digital computer estimation of eye fixations. Behavior Research Methods & Instrumentation, 8(2), 185–188.Google Scholar
  29. Mital, P. K., Smith, T. J., Hill, R., & Henderson, J. M. (2011). Clustering of gaze during dynamic scene viewing is predicted by motion. Cognitive Computation, 3(1), 5–24.Google Scholar
  30. Nakatani, C., & van Leeuwen, C. (2007). A pragmatic approach to multi-modality and non-normality in fixation duration studies of cognitive processes. Journal of Eye Movement Research, 1(2), 1–12.Google Scholar
  31. Nyamsuren, E., & Taatgen, N. A. (2013). Pre-attentive and attentive vision module. Cognitive Systems Research, 24, 62–71.Google Scholar
  32. Nyström, M., & Holmqvist, K. (2010). An adaptive algorithm for fixation, saccade, and glissade detection in eyetracking data. Behavior Research Methods, 42(1), 188–204.Google Scholar
  33. Otero-Millan, J., Castro, J. L. A., Macknik, S. L., & Martinez-Conde, S. (2014). Unsupervised clustering method to detect microsaccades. Journal of Vision, 14(2), 18, 1–17.Google Scholar
  34. Otero-Millan, J., Troncoso, X. G., Macknik, S. L., Serrano-Pedraza, I., & Martinez-Conde, S. (2008). Saccades and microsaccades during visual fixation, exploration, and search: Foundations for a common saccadic generator. Journal of Vision, 8(14), 21–21.Google Scholar
  35. R Development Core Team (2011). R: A language and environment for statistical computing [Computer software manual]. Vienna, Austria. Retrieved from (ISBN 3-900051-07-0).
  36. Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124(3), 372–422.Google Scholar
  37. Salvucci, D. D. (2000). An interactive model-based environment for eye-movement protocol analysis and visualization. In Proceedings of the 2000 symposium on eye tracking research & applications (pp. 57–63).Google Scholar
  38. Salvucci, D. D. (2001). An integrated model of eye movements and visual encoding. Cognitive Systems Research, 1(4), 201–220.Google Scholar
  39. Salvucci, D. D., & Anderson, J. R. (1998). Tracing eye movement protocols with cognitive process models. In Proceedings of the twentieth annual conference of the cognitive science society (pp. 923–928). Hillsdale: Lawrence Erlbaum Associates, Inc.Google Scholar
  40. Salvucci, D. D., & Goldberg, J. H. (2000). Identifying fixations and saccades in eye-tracking protocols. In Proceedings of the 2000 symposium on eye tracking research & applications (pp. 71–78).Google Scholar
  41. Sauter, D., Martin, B., Di Renzo, N., & Vomscheid, C. (1991). Analysis of eye tracking movements using innovations generated by a Kalman filter. Medical and Biological Engineering and Computing, 29(1), 63–69.Google Scholar
  42. Shic, F., Scassellati, B., & Chawarska, K. (2008). The incomplete fixation measure. In Proceedings of the 2008 symposium on eye tracking research and applications (pp. 111–114).Google Scholar
  43. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366.Google Scholar
  44. Smith, T. J., & Mital, P. K. (2013). Attentional synchrony and the influence of viewing task on gaze behavior in static and dynamic scenes. Journal of Vision, 13(8), 16–16.Google Scholar
  45. SR Research (2007). EyeLink user manual [Computer software manual]. Ottawa, ON.Google Scholar
  46. Stampe, D. M. (1993). Heuristic filtering and reliable calibration methods for video-based pupil-tracking systems. Behavior Research Methods, Instruments, & Computers, 25(2), 137–142.Google Scholar
  47. van der Lans, R., Wedel, M., & Pieters, R. (2011). Defining eye-fixation sequences across individuals and tasks: The Binocular-Individual Threshold (BIT) algorithm. Behavior Research Methods, 43(1), 239–257.Google Scholar
  48. Widdel, H. (1984). Operational problems in analysing eye movements. Advances in Psychology, 22, 21–29.Google Scholar
  49. Zhang, Y., & Hornof, A. J. (2014). Understanding multitasking through parallelized strategy exploration and individualized cognitive modeling. In Proceedings of the 32nd annual acm conference on human factors in computing systems (pp. 3885–3894).Google Scholar

Copyright information

© Psychonomic Society, Inc. 2017

Authors and Affiliations

  • Joseph W. Houpt
    • 1
  • Mary E. Frame
    • 2
  • Leslie M. Blaha
    • 3
  1. 1.Department of PsychologyWright State UniversityDaytonUSA
  2. 2.Department of PsychologyMiami UniversityOxfordUSA
  3. 3.Visual AnalyticsPacific Northwest National LaboratoryRichlandUSA

Personalised recommendations