Video Based Live Tracking of Fishes in Tanks

  • José CasteloEmail author
  • H. Sofia Pinto
  • Alexandre Bernardino
  • Núria Baylina
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12131)


We explore video tracking and classification in the context of real time marine wildlife observation. Among other applications it can help biologists by automating the process of gathering data, which is often done manually. In this paper we present a system to tackle the challenge of tracking and classifying fish in real time. We apply Background Subtraction techniques to detect the fish, followed by Feature Matching methods to track their movements over time. To deal with the shortcomings of tracking by detection we use a Kalman Filter to predict fish positions and a local search recovery method to re-identify fish tracks that are temporarily lost due to occlusions or lack of contrast. The species of tracked fish is recognized through Image Classification methods, using environment dependent features. We developed and tested our system using a custom built dataset, with several labeled image sequences of the fish tanks in the Oceanário de Lisboa. The impact of the proposed tracking methods are quantified and discussed. The proposed system is able to track and classify fish in real time in two scenarios, main tank and coral reef, reflecting different challenges.


Object detection Video tracking Image classification Real time Fish 



This work was supported by Fundação para a Ciência e a Tecnologia, under project UIDB/50021/2020, Project E-ARK3 and partially supported by FCT with the LARSyS - FCT Plurianual funding 2020–2023.


  1. 1.
    Beauxis-Aussalet, E., Palazzo, S., Nadarajan, G., Arslanova, E., Spampinato, C., Hardman, L.: A video processing and data retrieval framework for fish population monitoring. In: Proceedings of the 2nd ACM International Workshop on Multimedia Analysis for Ecological Data (MAED), pp. 15–20 (2013)Google Scholar
  2. 2.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Proceeding of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 886–893 (2005)Google Scholar
  3. 3.
    De Maesschalck, R., Jouan-Rimbaud, D., Massart, D.L.: The mahalanobis distance. Chemometr. Intell. Lab. Syst. 50(1), 1–18 (2000)CrossRefGoogle Scholar
  4. 4.
    Friedman, N., Russell, S.: Image segmentation in video sequences: a probabilistic approach. In: Proceedings of the 13th Conference on Uncertainty in Artificial Intelligence (UAI), pp. 175–181 (1997)Google Scholar
  5. 5.
    Kavasidis, I., Palazzo, S.: Quantitative performance analysis of object detection algorithms on underwater video footage. In: Proceedings of the 1st ACM International Workshop on Multimedia Analysis for Ecological Data (MAED), pp. 57–60 (2012)Google Scholar
  6. 6.
    Kavasidis, I., Spampinato, C., Giordano, D.: Generation of ground truth for object detection while playing an online game: productive gaming or recreational working? In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 694–699 (2013)Google Scholar
  7. 7.
    Nadeem, U., Shah, S.A.A., Bennamoun, M., Togneri, R., Sohel, F.: Real time surveillance for low resolution and limited-data scenarios: an image set classification approach. arXiv:1803.09470 (2018)
  8. 8.
    Spampinato, C., Chen-Burger, Y.-H., Nadarajan, G., Fisher, R.B.: Detecting, tracking and counting fish in low quality unconstrained underwater videos. In: Proceedings of the 3rd International Conference on Computer Vision Theory and Applications (VISAPP), vol. 2, pp. 514–519 (2008)Google Scholar
  9. 9.
    Zivkovic, Z.: Improved adaptive gaussian mixture model for background subtraction. In Proceedings of the 17th International Conference on Pattern Recognition (ICPR), vol. 2, pp. 28–31 (2004)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • José Castelo
    • 1
    Email author
  • H. Sofia Pinto
    • 1
  • Alexandre Bernardino
    • 1
  • Núria Baylina
    • 1
  1. 1.Instituto Superior Técnico, Lisbon (INESC-ID & ISR), Oceanário de LisboaLisbonPortugal

Personalised recommendations