HMPMR strategy for real-time tracking in aerial images, using direct methods


The vast majority of approaches make use of features to track objects. In this paper, we address the tracking problem with a tracking-by-registration strategy based on direct methods. We propose a hierarchical strategy in terms of image resolution and number of parameters estimated in each resolution, that allows direct methods to be applied in demanding real-time visual-tracking applications. We have called this strategy the Hierarchical Multi-Parametric and Multi-Resolution strategy (HMPMR). The Inverse Composition Image Alignment Algorithm (ICIA) is used as an image registration technique and is extended to an HMPMR-ICIA. The proposed strategy is tested with different datasets and also with image data from real flight tests using an Unmanned Aerial Vehicle, where the requirements of direct methods are easily unsatisfied (e.g. vehicle vibrations). Results show that using an HMPMR approach, it is possible to cope with the efficiency problem and with the small motion constraint of direct methods, conducting the tracking task at real-time frame rates and obtaining a performance that is comparable to, or even better than, the one obtained with the other algorithms that were analyzed.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Subscribe to journal

Immediate online access to all issues from 2019. Subscription will auto renew annually.

US$ 99

This is the net price. Taxes to be calculated in checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23


  1. 1.

    Anderson, C.H., Bergen, J.R., Burt, P.J., Ogden, J.M.: Pyramid methods in image processing. RCA Eng. 29(6), 33–41 (1984)

  2. 2.

    Baker, S., Datta, A., Kanade, T.: Parameterizing Homographies. Tech. Rep. CMU-RI-TR-06-11, Robotics Institute, Pittsburgh (2006)

  3. 3.

    Baker, S., Matthews, I.: Equivalence and efficiency of image alignment algorithms. In: Proceedings of the 2001 IEEE Conference on Computer Vision and Pattern Recognition 1, pp. 1090–1097 (2001)

  4. 4.

    Baker, S., Matthews, I.: Lucas-kanade 20 years on: a unifying framework. Int. J. Comput. Vis. 56(1), 221–255 (2004)

  5. 5.

    Bergen, J.R., Anandan, P., Hanna, K.J., Hingorani, R.: Hierarchical model-based motion estimation. In: ECCV ’92: Proceedings of the Second European Conference on Computer Vision, pp. 237–252 (1992)

  6. 6.

    Bouguet, J.Y.: Pyramidal implementation of the Lucas Kanade feature tracker: description of the algorithm. Technical report, OpenCV Document, Intel Microprocessor Research Labs (2002)

  7. 7.

    Bradski, G., Kaehler, A.: Learning OpenCV: Computer Vision with the OpenCV Library. O’Reilly (2008)

  8. 8.

    Burt, P., Adelson, E.: The laplacian pyramid as a compact image code. IEEE Trans. Commun. 31(4), 532–540 (1983). doi:10.1109/TCOM.1983.1095851

  9. 9.

    Can, A., Stewart, C., Roysam, B., Tanenbaum, H.: A feature-based, robust, hierarchical algorithm for registering pairs of images of the curved human retina. IEEE Transactions on Pattern Analysis and Machine Intelligence (2002)

  10. 10.

    Cao, X., Lan, J., Yan, P., Li, X.: Vehicle detection and tracking in airborne videos by multi-motion layer analysis. Mach. Vis. Appl. 23, 921–935 (2012)

  11. 11.

    Corral, E.M.: Efficient model-based 3d tracking by using direct image registration. Ph.D. thesis, Facultad de Informática. Universidad Politécnica de Madrid, Spain (2012)

  12. 12.

    Dufaux, F., Konrad, J.: Efficient, robust, and fast global motion estimation for video coding. IEEE Trans. Image Process. 9(3), 497–501 (2000)

  13. 13.

    Dupac, J., Matas, J., Naiser, F.: Ultra-fast tracking based on zero-shift points. Image Vis. Comput. 30(12), 1016–1031 (2012)

  14. 14.

    García Carrillo, L., Rondon, E., Sanchez, A., Dzul, A., Lozano, R.: Stabilization and trajectory tracking of a quad-rotor using vision. J. Intell. Robot. Syst. 61, 103–118 (2011)

  15. 15.

    Hager, G., Belhumeur, P.: Efficient region tracking with parametric models of geometry and illumination. IEEE Trans. Pattern Anal. Mach. Intell. 20(10), 1025–1039 (1998). doi:10.1109/34.722606

  16. 16.

    Hanna, K., Okamoto, N.: Combining stereo and motion analysis for direct estimation of scene structure. In: Proceedings of Fourth International Conference on Computer Vision, pp. 357–365 (1993). doi:10.1109/ICCV.1993.378192

  17. 17.

    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, New York (2003)

  18. 18.

    Hess, R.: SIFT feature detector implementation in C. (2007)

  19. 19.

    Hess, R.: An open-source SIFT library. In: Proceedings of the International Conference on Multimedia, pp. 1493–1496 (2010).

  20. 20.

    Holzer, S., Ilic, S., Navab, N.: Multilayer adaptive linear predictors for real-time tracking. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 105–117 (2013). doi:10.1109/TPAMI.2012.86

  21. 21.

    Hwangbo, M., Kim, J.S., Kanade, T.: Inertial-aided klt feature tracking for a moving camera. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, 2009. IROS 2009, pp. 1909–1916 (2009). doi:10.1109/IROS.2009.5354093

  22. 22.

    Irani, M., Anandan, P.: About direct methods. In: Vision Algorithms: Theory and Practice, Lecture Notes in Computer Science, vol. 1883, pp. 267–277. Springer, Berlin (2000)

  23. 23.

    Irani, M., Rousso, B., Peleg, S.: Computing occluding and transparent motions. Int. J. Comput. Vis. 12(1), 5–16 (1994). doi:10.1007/BF01420982

  24. 24.

    Jurie, F., Dhome, M.: Real time robust template matching. In: Rosin, P.L., Marshall, A.D. (eds.) British Machine Vision Conference, BMVC 2002, September, 2002, pp. 123–132. British Machine Vision Association, Cardiff (2002)

  25. 25.

    Kalal, Z., Mikolajczyk, K., Matas, J.: Tracking-learning-detection. IEEE Trans. Pattern Anal. Mach. Intell. 34(7), 1409–1422 (2012). doi:10.1109/TPAMI.2011.239

  26. 26.

    Kumar, R., Sawhney, H., Samarasekera, S., Hsu, S., Tao, H., Guo, Y., Hanna, K., Pope, A., Wildes, R., Hirvonen, D., Hansen, M., Burt, P.: Aerial video surveillance and exploitation. Proc. IEEE 89(10), 1518–1539 (2001)

  27. 27.

    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

  28. 28.

    Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proceedings of the International Joint Conference on Artificial Intelligence, pp. 674–679 (1981)

  29. 29.

    Malis, E., Benhimane, S.: A unified approach to visual tracking and servoing. Robot. Auton. Syst. 52(1), 39–52 (2005). doi:10.1016/j.robot.2005.03.014. (Advances in Robot Vision)

  30. 30.

    Martinez, C., Mejias, L., Campoy, P.: A multi-resolution image alignment technique based on direct methods for pose estimation of aerial vehicles. In: Proceedings of the International Conference on Digital Image Computing Techniques and Applications (DICTA), pp. 542–548 (2011). doi:10.1109/DICTA.2011.97

  31. 31.

    Martínez, C., Richardson, T., Thomas, P., du Bois, J.L., Campoy, P.: A vision-based strategy for autonomous aerial refueling tasks. Robot. Auton. Syst. 61(8), 876–895 (2013). doi:10.1016/j.robot.2013.02.006

  32. 32.

    Mejias, L., Saripalli, S., Campoy, P., Sukhatme, G.: Visual servoing approach for tracking features in urban areas using an autonomous helicopter. In: Proceedings of IEEE International Conference on Robotics and Automation, pp. 2503–2508, Orlando (2006)

  33. 33.

    Mondragon, I.F., Campoy, P., Correa, J., Mejias, L.: Visual model feature tracking for UAV control. In: IEEE International Symposium on Intelligent Signal Processing, 2007. WISP 2007, pp. 1–6 (2007). doi:10.1109/WISP.2007.4447629

  34. 34.

    Mondragón, I.F., Campoy, P., Martinez, C., Olivares-Mendez, M.: 3D pose estimation based on planar object tracking for UAVs control. In: Proceedings of IEEE International Conference on Robotics and Automation 2010 ICRA2010, Anchorage (2010)

  35. 35.

    Rao, C., Guo, Y., Sawhney, H., Kumar, R.: A heterogeneous feature-based image alignment method. In: 18th International Conference on Pattern Recognition, 2006. ICPR 2006 (2006)

  36. 36.

    Sawhney, H., Kumar, R.: True multi-image alignment and its application to mosaicing and lens distortion correction. IEEE Trans. Pattern Anal. Mach. Intell. 21(3), 235–243 (1999). doi:10.1109/34.754589

  37. 37.

    Sawhney, H.S., Hsu, S., Kumar, R.: Robust video mosaicing through topology inference and local to global alignment. In: Proc. European Conference on Computer Vision, pp. 103–119 (1998)

  38. 38.

    Sheikh, Y., Khan, S., Shah, M.: Feature-based georegistration of aerial images. In: International Conference on Geosensor Networks (2004)

  39. 39.

    Shi, J., Tomasi, C.: Good features to track. In: 1994 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’94), pp. 593–600 (1994)

  40. 40.

    Shum, H.Y., Szeliski, R.: Panoramic vision. chap. Construction of Panoramic Image Mosaics with Global and Local Alignment, pp. 227–268. Springer, Secaucus (2001).

  41. 41.

    Szeliski, R.: Image alignment and stitching: a tutorial. Found. Trends. Comput. Graph. Vis. 2(1), 1–104 (2006). doi:10.1561/0600000009

  42. 42.

    Szeliski, R., Shum, H.Y.: Creating full view panoramic image mosaics and environment maps. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’97, pp. 251–258. ACM Press/Addison-Wesley Publishing Co., New York (1997). doi:10.1145/258734.258861

  43. 43.

    Teuliere, C., Eck, L., Marchand, E.: Chasing a moving target from a flying uav. In: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4929–4934 (2011). doi:10.1109/IROS.2011.6094404

  44. 44.

    Torr, P.H.S., Zisserman, A.: Feature based methods for structure and motion estimation. In: Proceedings of the International Workshop on Vision Algorithms: Theory and Practice, ICCV ’99, pp. 278–294. Springer, London (2000)

  45. 45.

    Tsaig, Y., Averbuch, A.: Automatic segmentation of moving objects in video sequences: a region labeling approach. IEEE Trans. Circuits Syst. Video Techol. 12(7), 597–612 (2002)

  46. 46.

    Turcajova, R., Kautsky, J.: A hierarchical multiresolution technique for image registration. In: Proceedings of SPIE Mathematical Imaging: Wavelet Applications in Signal and Image Processing (1996)

  47. 47.

    Ye, G.: Image Registration and Super-resolution Mosaicing. Ph.D. thesis, The University of New South Wales (2005)

  48. 48.

    Zhang, H., Yuan, F.: Vehicle tracking based on image alignment in aerial videos. In: EMMCVPR’07: Proceedings of the 6th International Conference on Energy Minimization Methods in Computer Vision and Pattern Recognition, pp. 295–302. Springer, Berlin (2007)

  49. 49.

    Zimmermann, K., Matas, J., Svoboda, T.: Tracking by an optimal sequence of linear predictors. IEEE Trans. Pattern Anal. Mach. Intell. 31(4), 677–692 (2009)

  50. 50.

    Zitová, B., Flusser, J.: Image registration methods: a survey. Image Vis. Comput. 21(11), 977–1000 (2003)

Download references


The work described in this paper is the result of several research stages conducted at the Computer Vision Group of the Universidad Politécnica de Madrid. The authors would like to thank the Universidad Politécnica de Madrid, the Consejería de Educación de la Comunidad de Madrid, and the Fondo Social Europeo (FSE) for the Ph.D. Scholarships of some of the Authors. This work has been supported by the Spanish Ministry of Science under grant MICYT DPI2010-20751-C02-01.

Author information

Correspondence to Carol Martínez.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 48222 KB)

Supplementary material 1 (mp4 48222 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Martínez, C., Campoy, P., Mondragón, I.F. et al. HMPMR strategy for real-time tracking in aerial images, using direct methods. Machine Vision and Applications 25, 1283–1308 (2014).

Download citation


  • Visual tracking
  • Image registration
  • Hierarchical methods
  • UAVs
  • Parametric motion