Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking

  • Martin DanelljanEmail author
  • Andreas Robinson
  • Fahad Shahbaz Khan
  • Michael Felsberg
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9909)


Discriminative Correlation Filters (DCF) have demonstrated excellent performance for visual object tracking. The key to their success is the ability to efficiently exploit available negative data by including all shifted versions of a training sample. However, the underlying DCF formulation is restricted to single-resolution feature maps, significantly limiting its potential. In this paper, we go beyond the conventional DCF framework and introduce a novel formulation for training continuous convolution filters. We employ an implicit interpolation model to pose the learning problem in the continuous spatial domain. Our proposed formulation enables efficient integration of multi-resolution deep feature maps, leading to superior results on three object tracking benchmarks: OTB-2015 (\(+5.1\,\%\) in mean OP), Temple-Color (\(+4.6\,\%\) in mean OP), and VOT2015 (\(20\,\%\) relative reduction in failure rate). Additionally, our approach is capable of sub-pixel localization, crucial for the task of accurate feature point tracking. We also demonstrate the effectiveness of our learning formulation in extensive feature point tracking experiments.


Training Sample Fourier Coefficient Object Tracking Convolution Operator Visual Tracking 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work has been supported by SSF (CUAS), VR (EMC\({}^2\)), CENTAURO, the Wallenberg Autonomous Systems Program, NSC and Nvidia.

Supplementary material

419978_1_En_29_MOESM1_ESM.pdf (563 kb)
Supplementary material 1 (pdf 562 KB)


  1. 1.
    Badino, H., Yamamoto, A., Kanade, T.: Visual odometry by multi-frame feature integration. In: ICCV Workshop (2013)Google Scholar
  2. 2.
    Baker, S., Matthews, I.A.: Lucas-kanade 20 years on: a unifying framework. IJCV 56(3), 221–255 (2004)CrossRefGoogle Scholar
  3. 3.
    Bertinetto, L., Valmadre, J., Golodetz, S., Miksik, O., Torr, P.H.S.: Staple: complementary learners for real-time tracking. In: CVPR (2016)Google Scholar
  4. 4.
    Boddeti, V.N., Kanade, T., Kumar, B.: Correlation filters for object alignment. In: CVPR (2013)Google Scholar
  5. 5.
    Bolme, D.S., Beveridge, J.R., Draper, B.A., Lui, Y.M.: Visual object tracking using adaptive correlation filters. In: CVPR (2010)Google Scholar
  6. 6.
    Bouguet, J.Y.: Pyramidal implementation of the lucas kanade feature tracker. Technical report Microprocessor Research Labs, Intel Corporation (2000)Google Scholar
  7. 7.
    Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part VI. LNCS, vol. 7577, pp. 611–625. Springer, Heidelberg (2012)Google Scholar
  8. 8.
    Cimpoi, M., Maji, S., Vedaldi, A.: Deep filter banks for texture recognition and segmentation. In: CVPR (2015)Google Scholar
  9. 9.
    Danelljan, M., Häger, G., Shahbaz Khan, F., Felsberg, M.: Accurate scale estimation for robust visual tracking. In: BMVC (2014)Google Scholar
  10. 10.
    Danelljan, M., Häger, G., Shahbaz Khan, F., Felsberg, M.: Convolutional features for correlation filter based visual tracking. In: ICCV Workshop (2015)Google Scholar
  11. 11.
    Danelljan, M., Häger, G., Shahbaz Khan, F., Felsberg, M.: Learning spatially regularized correlation filters for visual tracking. In: ICCV (2015)Google Scholar
  12. 12.
    Danelljan, M., Häger, G., Shahbaz Khan, F., Felsberg, M.: Adaptive decontamination of the training set: a unified formulation for discriminative visual tracking. In: CVPR (2016)Google Scholar
  13. 13.
    Danelljan, M., Shahbaz Khan, F., Felsberg, M., van de Weijer, J.: Adaptive color attributes for real-time visual tracking. In: CVPR (2014)Google Scholar
  14. 14.
    Felsberg, M.: Enhanced distribution field tracking using channel representations. In: ICCV Workshop (2013)Google Scholar
  15. 15.
    Fernandez, J.A., Boddeti, V.N., Rodriguez, A., Kumar, B.V.K.V.: Zero-aliasing correlation filters for object recognition. TPAMI 37(8), 1702–1715 (2015)CrossRefGoogle Scholar
  16. 16.
    Fusiello, A., Trucco, E., Tommasini, T., Roberto, V.: Improving feature tracking with robust statistics. Pattern Anal. Appl. 2(4), 312–320 (1999)CrossRefGoogle Scholar
  17. 17.
    Galoogahi, H.K., Sim, T., Lucey, S.: Multi-channel correlation filters. In: ICCV (2013)Google Scholar
  18. 18.
    Galoogahi, H.K., Sim, T., Lucey, S.: Correlation filters with limited boundaries. In: CVPR (2015)Google Scholar
  19. 19.
    Gao, J., Ling, H., Hu, W., Xing, J.: Transfer learning based visual tracking with gaussian processes regression. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8691, pp. 188–203. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-10578-9_13 Google Scholar
  20. 20.
    Gladh, S., Danelljan, M., Shahbaz Khan, F., Felsberg, M.: Deep motion features for visual tracking. In: ICPR (2016)Google Scholar
  21. 21.
    Hare, S., Saffari, A., Torr, P.: Struck: structured output tracking with kernels. In: ICCV (2011)Google Scholar
  22. 22.
    He, S., Yang, Q., Lau, R., Wang, J., Yang, M.H.: Visual tracking via locality sensitive histograms. In: CVPR (2013)Google Scholar
  23. 23.
    Henriques, J.F., Caseiro, R., Martins, P., Batista, J.: Exploiting the circulant structure of tracking-by-detection with kernels. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part IV. LNCS, vol. 7575, pp. 702–715. Springer, Heidelberg (2012)Google Scholar
  24. 24.
    Henriques, J.F., Caseiro, R., Martins, P., Batista, J.: High-speed tracking with kernelized correlation filters. TPAMI 37(3), 583–596 (2015)CrossRefGoogle Scholar
  25. 25.
    Jia, X., Lu, H., Yang, M.H.: Visual tracking via adaptive structural local sparse appearance model. In: CVPR (2012)Google Scholar
  26. 26.
    Kalal, Z., Matas, J., Mikolajczyk, K.: P-N learning: bootstrapping binary classifiers by structural constraints. In: CVPR (2010)Google Scholar
  27. 27.
    Klein, G., Murray, D.: Parallel tracking and mapping for small AR workspaces. In: ISMAR (2007)Google Scholar
  28. 28.
    Kristan, M., Leonardis, A., Matas, J., Felsberg, M., Pflugfelder, R., Čehovin, L., Vojír, T., Häger, G., Lukězič, A., Fernández, G.: The visual object tracking VOT 2016 challenge results. In: ECCV Workshop (2016)Google Scholar
  29. 29.
    Kristan, M., Matas, J., Leonardis, A., Felsberg, M., Čehovin, L., Fernández, G., Vojír, T., Nebehay, G., Pflugfelder, R., Häger, G.: The visual object tracking VOT 2015 challenge results. In: ICCV Workshop (2015)Google Scholar
  30. 30.
    Kristan, M., et al.: The visual object tracking VOT 2014 challenge results. In: Agapito, L., Bronstein, M.M., Rother, C. (eds.) ECCV 2014. LNCS, vol. 8926, pp. 191–217. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-16181-5_14 Google Scholar
  31. 31.
    Li, Y., Zhu, J.: A scale adaptive kernel correlation filter tracker with feature integration. In: Agapito, L., Bronstein, M.M., Rother, C. (eds.) ECCV 2014. LNCS, vol. 8926, pp. 254–265. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-16181-5_18 Google Scholar
  32. 32.
    Liang, P., Blasch, E., Ling, H.: Encoding color information for visual tracking: algorithms and benchmark. TIP 24(12), 5630–5644 (2015)MathSciNetGoogle Scholar
  33. 33.
    Liu, L., Shen, C., van den Hengel, A.: The treasure beneath convolutional layers: cross-convolutional-layer pooling for image classification. In: CVPR (2015)Google Scholar
  34. 34.
    Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: IJCAI (1981)Google Scholar
  35. 35.
    Ma, C., Huang, J.B., Yang, X., Yang, M.H.: Hierarchical convolutional features for visual tracking. In: ICCV (2015)Google Scholar
  36. 36.
    Ma, C., Yang, X., Zhang, C., Yang, M.H.: Long-term correlation tracking. In: CVPR (2015)Google Scholar
  37. 37.
    Nocedal, J., Wright, S.J.: Numerical Optimization, 2nd edn. Springer, New York (2006)zbMATHGoogle Scholar
  38. 38.
    Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Learning and transferring mid-level image representations using convolutional neural networks. In: CVPR (2014)Google Scholar
  39. 39.
    Ovren, H., Forssén, P.: Gyroscope-based video stabilisation with auto-calibration. In: ICRA (2015)Google Scholar
  40. 40.
    Possegger, H., Mauthner, T., Bischof, H.: In defense of color-based model-free tracking. In: CVPR (2015)Google Scholar
  41. 41.
    Sevilla-Lara, L., Learned-Miller, E.G.: Distribution fields for tracking. In: CVPR (2012)Google Scholar
  42. 42.
    Shi, J., Tomasi, C.: Good features to track. In: CVPR (1994)Google Scholar
  43. 43.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)Google Scholar
  44. 44.
    Tomasi, C., Kanade, T.: Detection and Tracking of Point Features. Technical report (1991)Google Scholar
  45. 45.
    Wu, Y., Lim, J., Yang, M.H.: Online object tracking: a benchmark. In: CVPR (2013)Google Scholar
  46. 46.
    Wu, Y., Lim, J., Yang, M.H.: Object tracking benchmark. TPAMI 37(9), 1834–1848 (2015)CrossRefGoogle Scholar
  47. 47.
    Zhang, J., Ma, S., Sclaroff, S.: MEEM: robust tracking via multiple experts using entropy minimization. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part VI. LNCS, vol. 8694, pp. 188–203. Springer, Heidelberg (2014)Google Scholar
  48. 48.
    Zografos, V., Lenz, R., Ringaby, E., Felsberg, M., Nordberg, K.: Fast segmentation of sparse 3D point trajectories using group theoretical invariants. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9006, pp. 675–691. Springer, Heidelberg (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Martin Danelljan
    • 1
    Email author
  • Andreas Robinson
    • 1
  • Fahad Shahbaz Khan
    • 1
  • Michael Felsberg
    • 1
  1. 1.CVL, Department of Electrical EngineeringLinköping UniversityLinköpingSweden

Personalised recommendations