Skip to main content
SpringerLink
Log in

Tracking of Ocean Surface Objects from Unmanned Aerial Vehicles with a Pan/Tilt Unit using a Thermal Camera

  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

This paper presents four vision-based tracking system architectures for marine surface objects using a fixed-wing unmanned aerial vehicle (UAV) with a thermal camera mounted in a pan/tilt gimbal. The tracking systems estimate the position and velocity of an object in the North-East (NE) plane, and differ in how the measurement models are defined. The first tracking system measures the position and velocity of the target with georeferencing and optical flow. The states are estimated in a Kalman filter. A Kalman filter is also utilized in the second architecture, but only the georeferenced position is used as a measurement. A bearing-only measurement model is the basis for the third tracking system, and because the measurement model is nonlinear, an extended Kalman filter is used for state estimation. The fourth tracking system extends the bearing-only tracking system to let navigation uncertainty in the UAV position affect the target estimates in a Schmidt-Kalman filter. All tracking architectures are evaluated on data gathered at a flight experiment near the Azores islands outside of Portugal. The results show that various marine vessels can be tracked quite accurately.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. COLREGs - convention on the international regulations for preventing collisions at sea, International Maritime Organization (IMO) (1972)

  2. Bar-Shalom, Y., Li, X.R., Kirubarajan, T.: Estimation with Applications to Tracking and Navigation: Theory Algorithms and Software. Wiley, Hoboken (2004)

    Google Scholar 

  3. Barber, D.B., Redding, J.D., McLain, T.W., Beard, R.W., Taylor, C.N.: Vision-based target geo-location using a fixed-wing miniature air vehicle. J. Intell. Robot. Syst. 47(4), 361–382 (2006)

    Article  Google Scholar 

  4. Horn, B.K.P., Schunk, B.G.: Determining optical flow. Artif. Intell. 17, 185–204 (1981)

    Article  Google Scholar 

  5. Bradski, G.: The opencv library. Dr Dobb’s Journal of Software Tools (2000)

  6. Bryson, M., Sukkarieh, S.: Bearing-Only Slam for an Airborne Vehicle. In: Proceedings of the Australasian Conference on Robotics and Automation, vol. 4 (2005)

  7. Bryson, M., Sukkarieh, S.: Building a robust implementation of bearing-only inertial slam for a uav. J. Field Rob. 24(1-2), 113–143 (2007)

    Article  Google Scholar 

  8. Doherty, P., Rudol, P.: A uav search and rescue scenario with human body detection and geolocalization. Lect. Notes Comput. Sci (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 4830, 1–13 (2007)

    Google Scholar 

  9. Dumble, S.J., Gibbens, P.W.: Efficient terrain-aided visual horizon based attitude estimation and localization. J. Intell. Robot. Syst. 78(2), 205–221 (2015)

    Article  Google Scholar 

  10. Egeland, O., Gravdahl, J.T.: Modeling and Simulation for Automatic Control. Marine Cybernetics Trondheim, Norway (2002)

    Google Scholar 

  11. Elkins, L., Sellers, D., Monach, W.R.: The autonomous maritime navigation (amn) project: Field tests, autonomous and cooperative behaviors, data fusion, sensors, and vehicles. J. Field Rob. 27(6), 790–818 (2010)

    Article  Google Scholar 

  12. Fossen, T.: Handbook of Marine Craft Hydrodynamics and Motion Control. Wiley, Hoboken (2011)

    Book  Google Scholar 

  13. Fusini, L., Hosen, J., Helgesen, H.H., Johansen, T.A., Fossen, T.I.: Experimental validation of a uniformly semi-globally exponentially stable non-linear observer for gnss- and camera-aided inertial navigation for fixed-wing Uavs. In: Proceedings of the International Conference on Unmanned Aircraft Systems, pp. 851–860 (2015)

  14. Gomez-Balderas, J.E., Flores, G., García Carrillo, L.R., Lozano, R.: Tracking a ground moving target with a quadrotor using switching control. J. Intell. Robot. Syst. 70(1), 65–78 (2013)

    Article  Google Scholar 

  15. Hansen, J., Fossen, T., Johansen, T.: Nonlinear observer for Ins aided by time-delayed Gnss measurements: implementation and Uav experiments. In: Proceedings of the International Conference on Unmanned Aircraft Systems, pp. 157–166 (2015)

  16. Helgesen, H.H., Leira, F.S., Johansen, T.A., Fossen, T.I.: Tracking of marine surface objects from unmanned aerial vehicles with a pan/tilt unit using a thermal camera and optical flow. In: Proceedings of the International Conference on Unmanned Aircraft Systems, pp. 107–117 (2016)

  17. Helgesen, H.H., Leira, F.S., Johansen, T.A., Fossen, T.I.: Detection and tracking of floating objects using a Uav with thermal camera. In: Fossen, T.I., Pettersen, K.Y., Nijmeijer, H. (eds.) Sensing and Control for Autonomous Vehicles: Applications to Land, Water and Air Vehicles, pp. 289–316. Springer International Publishing (2017)

  18. Hemerly, E.M.: Automatic georeferencing of images acquired by uav’s. Int. J. Autom. Comput. 11(4), 347–352 (2014)

    Article  Google Scholar 

  19. Hiba, A., Zsedrovits, T., Bauer, P., Zarandy, A.: Fast horizon detection for airborne visual systems. In: Proceedings of the International Conference on Unmanned Aircraft Systems, pp. 886–891 (2016). https://doi.org/10.1109/ICUAS.2016.7502661

  20. Hosen, J., Helgesen, H.H., Fusini, L., Fossen, T.I., Johansen, T.A.: A vision-aided nonlinear observer for fixed-wing Uav navigation. In: Proceedings of the AIAA Guidance, Navigation, and Control Conference (2016)

  21. Hosen, J., Helgesen, H.H., Fusini, L., Fossen, T.I., Johansen, T.A.: Vision-aided nonlinear observer for fixed-wing unmanned aerial vehicle navigation. J. Guid. Control. Dyn. 39(8), 1777–1789 (2016)

    Article  Google Scholar 

  22. Huntsberger, T., Aghazarian, H., Howard, A., Trotz, D.C.: Stereo vision–based navigation for autonomous surface vessels. J. Field Rob. 28(1), 3–18 (2011)

    Article  Google Scholar 

  23. Hutchinson, S., Hager, G., Corke, P.: A tutorial on visual servo control. IEEE Trans. Robot. Autom. 12(5), 651–670 (1996)

    Article  Google Scholar 

  24. Johansen, T., Fossen, T.: Nonlinear filtering with exogenous Kalman filter and double Kalman filter. In: Proceedings of the European Control Conference, Aalborg (2016)

  25. Johansen, T.A., Perez, T.: Unmanned aerial surveillance system for hazard collision avoidance in autonomous shipping. In: Proceedings of the International Conference on Unmanned Aircraft Systems (2016)

  26. Kadyrov, A., Yu, H., Liu, H.: Ship detection and segmentation using image correlation. In: Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, pp. 3119–3126 (2013)

  27. Kim, J.H., Sukkarieh, S.: Airborne simultaneous localisation and map building. In: Proceedings of the IEEE International Conference on Robotics and Automation, vol. 1 (2003)

  28. Leira, F.S., Johansen, T.A., Fossen, T.I.: Automatic detection, classification and tracking of objects in the ocean surface from Uavs using a thermal camera. In: Proceedings of the IEEE Aerospace Conference, Big Sky, US (2015)

  29. Leira, F.S., Trnka, K., Fossen, T.I., Johansen, T.A.: A ligth-weight thermal camera payload with georeferencing capabilities for small fixed-wing Uavs. In: Proceedings of the International Conference on Unmanned Aircraft Systems, pp. 485–494 (2015)

  30. Li, X.R., Jilkov, V.P.: Survey of maneuvering target tracking: dynamic models. In: Proceedings of SPIE, vol. 4048, pp. 212–235 (2000)

  31. Lowe, D.: Object recognition from local scale-invariant features. In: Proceedings of the International Conference on Computer Vision, pp. 1150–1157 (1999)

  32. Mammarella, M., Campa, G., Fravolini, M.L., Napolitano, M.R.: Comparing optical flow algorithms using 6-dof motion of real-world rigid objects. IEEE Trans. Syst. Man Cybern. 42(6), 1752–1762 (2012)

    Article  Google Scholar 

  33. Martínez, C., Mondragón, I.F., Campoy, P., Sánchez-López, J.L., Olivares-Méndez, M.A.: A hierarchical tracking strategy for vision-based applications on-board uavs. J. Intell. Robot. Syst. 72(3), 517–539 (2013)

    Article  Google Scholar 

  34. Muja, M., Lowe, D.G.: Flann, fast library for approximate nearest neighbors. In: Proceedings of the International Conference on Computer Vision Theory and Applications (2009)

  35. Novoselov, R.Y., Herman, S.M., Gadaleta, S.M., Poore, A.B.: Mitigating the effects of residual biases with Schmidt-Kalman filtering. In: Proceedings of the 8Th International Conference on Information Fusion, vol. 1 (2005)

  36. Ortiz, A., Bonnin-Pascual, F., Garcia-Fidalgo, E.: Vessel inspection: a micro-aerial vehicle-based approach. J. Intell. Robot. Syst. 76(1), 151–167 (2014)

    Article  Google Scholar 

  37. Rudol, P., Doherty, P.: Human body detection and geolocalization for Uav search and rescue missions using color and thermal imagery. In: Proceedings of the IEEE Aerospace Conference (2008)

  38. Wilthil, E.F., Brekke, E.F.: Compensation of navigation uncertainty for target tracking on a moving platform. In: Proceedings of the 19Th International Conference on Information Fusion, pp. 1616–1621 (2016)

  39. Wolf, M.T., Assad, C., Kuwata, Y., Howard, A., Aghazarian, H., Zhu, D., Lu, T., Trebi-Ollennu, A., Huntsberger, T.: 360-degree visual detection and target tracking on an autonomous surface vehicle. J. Field Rob. 27(6), 819–833 (2010)

    Article  Google Scholar 

  40. Yang, C., Blasch, E., Douville, P.: Design of Schmidt-Kalman filter for target tracking with navigation errors. In: Proceedings of the IEEE Aerospace Conference (2010)

  41. Yilmaz, A., Javed, O., Shah, M.: Object tracking: a survey. ACM Comput. Surv. 38(13), 1–45 (2006)

    Google Scholar 

  42. Yu, X., Zhang, Y.: Sense and avoid technologies with applications to unmanned aircraft systems: review and prospects. Prog. Aerosp. Sci. 74, 152–166 (2015)

    Article  Google Scholar 

  43. Zhao, S., Lin, F., Peng, K., Dong, X., Chen, B.M., Lee, T.H.: Vision-aided estimation of attitude, velocity, and inertial measurement bias for uav stabilization. J. Intell. Robot. Syst. 81(3), 531–549 (2016)

    Article  Google Scholar 

  44. Zhou, G., Li, C., Cheng, P.: Unmanned Aerial Vehicle (Uav) real-time video registration for forest fire monitoring. In: Proceedings of IEEE International Geoscience and Remote Sensing Symposium, vol. 3, pp. 1803–1806 (2005)

Download references

Acknowledgements

This work has been carried out at the NTNU Center for Autonomous Marine Operations and Systems. This work was supported by the Research Council of Norway through the Centers of Excellence funding scheme, Project number 223254. The authors are grateful for the support from the University of Porto, the University of the Azores, UAV operators Lars Semb and Krzysztof Cisek, REP15 coordinators João Tasso Sousa and Kanna Rajan and valuable feedback from Edmund Førland Brekke.

Author information

Authors and Affiliations

  1. NTNU Center for Autonomous Marine Operations and Systems, Department of Engineering Cybernetics, Norwegian University of Science and Technology, O. S. Bragstads plass 2D, 7034, Trondheim, Norway

    Håkon Hagen Helgesen, Frederik Stendahl Leira, Thor I. Fossen & Tor Arne Johansen

Authors
  1. Håkon Hagen Helgesen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Frederik Stendahl Leira
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thor I. Fossen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tor Arne Johansen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Håkon Hagen Helgesen.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Helgesen, H.H., Leira, F.S., Fossen, T.I. et al. Tracking of Ocean Surface Objects from Unmanned Aerial Vehicles with a Pan/Tilt Unit using a Thermal Camera. J Intell Robot Syst 91, 775–793 (2018). https://doi.org/10.1007/s10846-017-0722-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10846-017-0722-3

Keywords

Search

Navigation