Learning to Detect Event Sequences in Surveillance Streams at Very Low Frame Rate

Part of the Advances in Pattern Recognition book series (ACVPR)


Some camera surveillance systems are designed to be autonomous—both from the energy and storage points of view. Autonomy allows operation in environments where wiring cameras for power and data transmission is not feasible. In these contexts, for cameras to work unattended over long periods of time requires choosing a low frame rate to match the speed of the process to be supervised while minimizing energy and storage usage. The result of surveillance is a large stream of images acquired sparsely over time with limited visual continuity from one frame to the other. Reviewing these images to detect events of interest requires techniques that do not assume traceability of objects by visual similarity. When the process surveyed shows recurrent patterns of events, as it is often the case for industrial settings, other possibilities open up. Since images are time-stamped, techniques which use temporal data can help detecting events. This contribution presents an image review tool that combines a scene change detector (SCD) with a temporal filter. The temporal filter learns to recognize relevant SCD events by their time distribution on the image stream. Learning is supported by image annotations provided by end-users during past reviews. The concept is tested on a benchmark of real surveillance images stemming from a nuclear safeguards context. Experimental results show that the combined SCD-temporal filter significantly reduces the workload necessary to detect safeguards-relevant events in large image streams.


Markov Model Image Retrieval Sojourn Time Duration Model Review Report 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Ahmed Elgammal, A., Duraiswami, R., Harwood, D., Davis, L.S.: Background and foreground modeling using nonparametric kernel density estimation for visual surveillance. Proc. IEEE 90(7), 1151–1163 (2002) CrossRefGoogle Scholar
  2. 2.
    Akyildiz, I.F., Su, W., Sankarasubramaniam, Y., Cayirci, E.: Wireless sensor networks: a survey. Comput. Netw. 38(4), 393–422 (2002) CrossRefGoogle Scholar
  3. 3.
    Brdiczka, O., Langet, M., Maisonnasse, J., Crowley, J.L.: Detecting human behavior models from multimodal observation in a smart home. IEEE Trans. Autom. Sci. Eng. 5, 1 (2003) Google Scholar
  4. 4.
    Cover, T., Hart, P.: Nearest neighbor pattern classification. IEEE Trans. Inf. Theory 13, 21–27 (1967) MATHCrossRefGoogle Scholar
  5. 5.
  6. 6.
    Dorado, A., Calic, J., Izquierdo, E.: A rule-based video annotation system. IEEE Trans. Circuits Syst. Video Technol. 14(5), 622–633 (2004) CrossRefGoogle Scholar
  7. 7.
    Ewerth, R., Freisleben, B.: Semi-supervised learning for semantic video retrieval. In: Proc. 6th ACM International Conference on Image and Video Retrieval, pp. 154–161 (2007) Google Scholar
  8. 8.
    Fails, J.A., Olsen, D.R.: A design tool for camera-based interaction. In: Human Factors in Computing Systems, CHI ’03, pp. 449–456. ACM, New York (2003) CrossRefGoogle Scholar
  9. 9.
    Ferrando, S., Gera, G., Massa, M., Regazzoni, C.: A new method for real time abandoned object detection and owner tracking. In: IEEE International Conference on Image Processing, pp. 3329–3332 (2006) Google Scholar
  10. 10.
  11. 11.
    Guédon, Y.: Estimating hidden semi-Markov chains from discrete sequences. J. Comput. Graph. Stat. 12(3), 604–639 (2003) CrossRefGoogle Scholar
  12. 12.
    Hu, W., Tan, T., Wang, L., Maybank, S.: A survey of visual surveillance of object motion and behaviors. IEEE Trans. Syst. Man Cybern. C 34(3), 334–352 (2004) CrossRefGoogle Scholar
  13. 13.
    Iyengar, G., Nock, H.J.: Discriminative model fusion for semantic concept detection and annotation in video. In: Proc. 11th ACM International Conference on Multimedia, pp. 255–258 (2003) Google Scholar
  14. 14.
    Liu, Y., Yao, H., Gao, W., Chen, X., Zhao, D.: Nonparametric background generation. In: Proc. 18th International Conference on Pattern Recognition, vol. 4, pp. 916–919 (2006) Google Scholar
  15. 15.
    Lombardi, P., Versino, C.: Tracking nuclear material at low frame rate and numerous false detections. In: ICVS2007 Vision Systems in the Real World: Adaptation, Learning, Evaluation—Conference Proceedings, Bielefeld, Germany (2007) Google Scholar
  16. 16.
    Lombardi, P., Versino, C., Gonçalves, J.G.M., Heppleston, M., Tourin, L.: MM: the Markov model tool for image reviews. In: Proc. of the Institute of Nuclear Materials Management (2008) Google Scholar
  17. 17.
    Makris, D., Ellis, T.: Learning semantic scene models from observing activity in visual surveillance. IEEE Trans. Syst. Man Cybern. B 35(3), 397–408 (2005) CrossRefGoogle Scholar
  18. 18.
    Markou, M., Singh, S.: Novelty detection: a review—part 1: statistical approaches. Signal Process. 83, 2003 (2003) Google Scholar
  19. 19.
    Markou, M., Singh, S.: Novelty detection: a review—part 2: neural network based approaches. Signal Process. 83, 2499–2521 (2003) MATHCrossRefGoogle Scholar
  20. 20.
    Mcivor, A.M.: Background subtraction techniques. In: Proc. of Image and Vision Computing, pp. 147–153 (2000) Google Scholar
  21. 21.
    Messelodi, S., Modena, C.M., Zanin, M.: A computer vision system for the detection and classification of vehicles at urban road intersections. Pattern Anal. Appl. 8(1), 17–31 (2005) MathSciNetCrossRefGoogle Scholar
  22. 22.
    Mitra, S., Acharya, T.: Gesture recognition: a survey. IEEE Trans. Syst. Man Cybern., Part C, Appl. Rev. 37(3), 311–324 (2007) CrossRefGoogle Scholar
  23. 23.
    Morris, B.T., Trivedi, M.M.: A survey of vision-based trajectory learning and analysis for surveillance. IEEE Trans. Circuits Syst. Video Technol. 18(8), 1114–1127 (2008) CrossRefGoogle Scholar
  24. 24.
    Murphy, K.: Hidden semi-Markov models (HSMMs). Available online: http://people.cs.ubc.ca/~murphyk/Papers/segment.pdf (2002)
  25. 25.
    Qi, G.-J., Hua, X.-S., Rui, Y., Tang, J., Mei, T., Zhang, H.-J.: Correlative multi-label video annotation. In: Proc. 15th International Conference on Multimedia, pp. 17–26 (2007) Google Scholar
  26. 26.
    Quinlan, J.R.: C4.5: programs for machine learning. Morgan Kaufmann, San Mateo (1993) Google Scholar
  27. 27.
    Rabiner, L.R.: A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77(2), 257–286 (1989) CrossRefGoogle Scholar
  28. 28.
    Safeguards Techniques and Equipment IAEA—2003 Edition. http://www-pub.iaea.org/MTCD/publications/PDF/NVS1-2003_web.pdf
  29. 29.
    Smeulders, A.W.M., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell. 22(12), 1349–1380 (2000) CrossRefGoogle Scholar
  30. 30.
    Stauffer, C., Grimson, E.: Learning patterns of activity using real-time tracking. IEEE Trans. Pattern Recognit. Mach. Intell. 22(8), 747–757 (2000) CrossRefGoogle Scholar
  31. 31.
    Sullivan, G.J., Wiegand, T.: Video compression—from concepts to the H.264/AVC standard. Proc. IEEE 93(1), 18–31 (2005) CrossRefGoogle Scholar
  32. 32.
    Tao, D., Tang, X., Li, X., Wu, X.: Asymmetric bagging and random subspace for support vector machines-based relevance feedback in image retrieval. IEEE Trans. Pattern Recognit. Mach. Intell. 28(7), 1088–1099 (2006) CrossRefGoogle Scholar
  33. 33.
    Tong, S., Chang, E.: Support vector machine active learning for image retrieval. In: Proc. of ACM International Conference on Multimedia, pp. 107–118 (2001) Google Scholar
  34. 34.
    Tong, S., Koller, D.: Support vector machine active learning with applications to text classification. In: 17th International Conference on Machine Learning, pp. 401–412 (2000) Google Scholar
  35. 35.
    Toyama, K., Krumm, J., Brumitt, B., Meyers, B.: Wallflower: principles and practice of background maintenance. In: Proc. 7th IEEE International Conference on Computer Vision, pp. 255–261 (1999) Google Scholar
  36. 36.
    Trucco, E., Plakas, K.: Video tracking: a concise survey. IEEE J. Ocean. Eng. 31(2), 520–529 (2006) CrossRefGoogle Scholar
  37. 37.
    Versino, C., Stringa, E., Gonçalves, J.G.M., Heppleston, M., Tourin, L.: Towards the integrated review of safeguards surveillance data. In: 27th ESARDA Symposium on Safeguards and Nuclear Material Management (2005) Google Scholar
  38. 38.
    Wang, M., Hua, X.-S., Hong, R., Tang, J., Qi, G.-J., Song, Y.: Unified video annotation via multigraph learning. IEEE Trans. Circuits Syst. Video Technol. 19(5), 733–746 (2009) CrossRefGoogle Scholar
  39. 39.
    Yilmaz, A., Javed, O., Shah, M.: Object tracking: a survey. ACM J. Comput. Surv. 38(4), 13 (2006) CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

There are no affiliations available

Personalised recommendations