Journal of Intelligent & Robotic Systems

, Volume 93, Issue 1–2, pp 261–276 | Cite as

Three Dimensional Intruder Closest Point of Approach Estimation Based-on Monocular Image Parameters in Aircraft Sense and Avoid

Motto: ’Almost Everything from Almost Nothing’
  • Peter BauerEmail author
  • Antal Hiba
  • Jozsef Bokor
  • Akos Zarandy


The paper deals with monocular image-based sense and avoid assuming constant aircraft velocities and straight flight paths. From very limited two dimensional image information it finally characterizes the whole three dimensional collision situation by estimating the time to closest point of approach, the horizontal relative distance and its direction and the vertical relative distance also. The distances are relative to the intruder aircraft horizontal and vertical sizes. The overall estimated relative distance is the closest between the two aircraft in three dimension. So finally, every important information can be extracted to be used in a collision decision. The applicability of the developed method is presented in software-in-the-loop simulation test runs. Several intruder size and speed values are considered together with trajectories covering the whole three dimensional space. The horizontal intruder flight directions relative to the own aircraft cover 360 and the intruder can come from below ar above also. Detailed evaluation and discussion of the results is also included. Finally, the missed detection rate results to be superior (below 3% in every test scenario) though the false alarm rate results a bit high between 7–14%.


Sense and avoid Monocular camera Closest point of approach Intruder direction 

Mathematics Subject Classification (2010)

93C41 93A30 93C85 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



The research leading to these results has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No. 690811 and the Japan New Energy and Industrial Technology Development Organization under grant agreement No. 062600 as a part of the EU/Japan joint research project entitled ’Validation of Integrated Safety-enchanced Intelligent flight cONtrol (VISION)’ This work was also supported by the Institute for Computer Science and Control (SZTAKI) Grant Number 008. The authors greatly appreciate the work of the reviewers which helped to improve the overall quality of the paper.


  1. 1. aircraft-data. (2015)
  2. 2.
    Bauer, P., Hiba, A.: Vision only collision detection with omnidirectional multi-camera system. In: Proc. of the 20th World Congress of the International Federation of Automatic Control, pp. 15,780–15,785. IFAC, Toulouse, France (2017)Google Scholar
  3. 3.
    Bauer, P., Hiba, A., Bokor, J.: Monocular image-based intruder direction estimation at closest point of approach. In: Proc. of the International Conference on Unmanned Aircraft Systems (ICUAS) 2017, pp. 1108–1117, ICUAS Association, Miami, FL, USA (2017)Google Scholar
  4. 4.
    Bauer, P., Hiba, A., Vanek, B., Zarandy, A., Bokor, J.: Monocular image-based time to collision and closest point of approach estimation. In: proceedings of 24th Mediterranean Conference on Control and Automation (MED’16). Athens, Greece (2016)Google Scholar
  5. 5.
    Bauer, P., Vanek, B., Peni, T., Futaki, A., Pencz, B., Zarandy, A., Bokor, J.: Monocular image parameter-based aircraft sense and avoid. In: proceedings of 23rd Mediterranean Conference on Control and Automation (MED’15). Torremolinos, Spain (2015)Google Scholar
  6. 6.
    Bauer, P., Vanek, B., Pni, T., Zsedrovits, T., Pencz, B., Zarndy, K., Bokor, J.: Aircraft trajectory tracking with large sideslip angles for sense and avoid intruder state estimation. In proceedings of 22nd Mediterranean Conference on Control and Automation (MED’14). Palermo, Italy (2014)Google Scholar
  7. 7.
    Byrne, J., Taylor, C.J.: Expansion segmentation for visual collision detection and estimation. In: Proc. of IEEE International Conference on Robotics and Automation (2009)Google Scholar
  8. 8.
    Choi, H., Kim, Y., Hwang, I.: Reactive collision avoidance of unmanned aerial vehicles using a single vision sensor. J. Guid. Control. Dyn. 36(4), 1234–1240 (2013). CrossRefGoogle Scholar
  9. 9.
    Degen, S.: Reactive image-based collision avoidance system for unmanned aircraft systems. Master’s thesis, Australian Research Centre for Aerospace Automation (2011)Google Scholar
  10. 10.
    Dempsey, M.: U.s. army unmanned aircraft systems roadmap 2010-2035. Tech. rep., U.S. Army UAS Center of Excellence (2010)Google Scholar
  11. 11.
    EU: Roadmap for the integration of civil Remotely-Piloted Aircraft Systems into the European Aviation System. Tech. rep., European RPAS Steering Group (2013)Google Scholar
  12. 12.
    FAA: Federal Aviation Regulation 14 CFR Part 91. Federal Aviation Administration (FAA) (2016)Google Scholar
  13. 13.
    Fasano, G., Accardo, D., Forlenza, L., Moccia, A., Rispoli, A.: A multi-sensor obstacle detection and tracking system for autonomousuav sense and avoid. In: XX Congresso Nazionale AIDAA, Milano (2009)Google Scholar
  14. 14.
    Fasano, G., Accardo, D., Tirri, A.E., Moccia, A.: Feature article: Experimental analysis of onboard non-cooperative sense and avoid solutions based on radar, optical sensors, and data fusion. IEEE Aerosp. Electron. Syst. Mag. 31(7), 6–14 (2016). CrossRefGoogle Scholar
  15. 15.
    Forlenza, L.: Vision based strategies for implementing Sense and Avoid capabilities onboard Unmanned Aerial Systems. Ph.D. thesis, UNIVERSITÁ DEGLI STUDI DI NAPOLI FEDERICO II (2012)Google Scholar
  16. 16.
    Frew, E.W.: Observer trajectory generation for target-motion estimation using monocular vision. Ph.D. thesis, Stanford University (2003)Google Scholar
  17. 17.
    Hutchings, T., Jeffryes, S., Farmer, S.J.: Architecting uav sense & avoid systems. In: Proc. Institution of Engineering and Technology Conf. Autonomous Systems, pp. 1–8 (2007)Google Scholar
  18. 18.
    Jamoom, M.B., Joerger, M., Pervan, B.: Unmanned aircraft system sense-and-avoid integrity and continuity risk. J. Guid. Control. Dyn. 39(3), 498–509 (2015). CrossRefGoogle Scholar
  19. 19.
    Lyu, Y., Pan, Q., Zhao, C., Zhang, Y., Hu, J.: Feature article: Vision-based UAV collision avoidance with 2D dynamic safety envelope. IEEE Aerosp. Electron. Syst. Mag. 31(7), 16–26 (2016). CrossRefGoogle Scholar
  20. 20.
    Mejias, L., McFadyen, A., Ford, J.J.: Feature article: Sense and avoid technology developments at Queensland university of technology. IEEE Aerosp. Electron. Syst. Mag. 31(7), 28–37 (2016). CrossRefGoogle Scholar
  21. 21.
    Melnyk, R., Schrage, D., Volovoi, V., Jimenez, H.: Sense and avoid requirements for unmanned aircraft systems using a target level of safety approach. Risk Anal. 34(10), 1894–1906 (2014)CrossRefGoogle Scholar
  22. 22.
    Meyer, F., Bouthemy, P.: Estimation of time-to-collision maps from first order motion models and normal flows. In: Proc. of 11th IAPR International Conference on Pattern Recognition (1992)Google Scholar
  23. 23.
    Mori, T., Scherer, S.: First results in detecting and avoiding frontal obstacles from a monocular camera for micro unmanned aerial vehicles. In: International Conference on Robotics and Automation (2013)Google Scholar
  24. 24.
    Nussberger, A., Grabner, H., Gool, L.V.: Feature article: Robust Aerial Object Tracking from an Airborne platform. IEEE Aerosp. Electron. Syst. Mag. 31(7), 38–46 (2016). CrossRefGoogle Scholar
  25. 25.
    Shakernia, O., Chen, W.-Z., Raska, V.: Passive ranging for uav sense and avoid applications. In: fotech Aerospace (2005)Google Scholar
  26. 26.
    Salazar, L.R., Sabatini, R., Ramasamy, S., Gardi, A.: A novel system for non-cooperative uav sense-and-avoid. In: Proceedings of european navigation conference 2013 (ENC 2013) (2013)Google Scholar
  27. 27.
    Schaub, A., Burschka, D.: Spatio-temporal prediction of collision candidates for static and dynamic objects in monocular image sequences. In Proc. of IEEE Intelligent Vehicles Symposium (IV 2013) (2013)Google Scholar
  28. 28.
    Speijker, L., Verstraeten, J., Kranenburg, C., van der Geest, P.: Scoping improvements to ’See And Avoid’ for general aviation (SISA). Tech. rep., European Aviation Safety Agency (EASA (2012)Google Scholar
  29. 29.
    Vanek, B., Peni, T., Zarandy, A., Bokor, J., Zsedrovits, T., Roska, T.: Performance characteristics of a complete vision only sense and avoid system. In: Proceedings of AIAA GNC 2012 (Guidance, Navigation and Control Conference), AIAA 2012-4703, pp. 1–15, Minneapolis, Minnesota (2012)Google Scholar
  30. 30.
    Watanabe, Y.: Stochastically optimized monocular vision-based navigation and guidance. Ph.D. thesis, Georgia Institute of Technology (2008)Google Scholar
  31. 31.
    Zarandy, A., Nagy, Z., Vanek, B., Zsedrovits, T., Kiss, A., Nemeth, M.: A five-camera vision system for uav visual attitude calculation and collision warning. In: Computer Vision Systems, Lecture Notes in Computer Science, pp. 11–20. Saint Petersburg, Russia (2013)Google Scholar
  32. 32.
    Zsedrovits, T., Bauer, P., Pencz, B.J.M., Hiba, A., Gőzse, I., Kisantal, M., Németh, M., Nagy, Z., Vanek, B., Zarándy, A., Bokor, J.: Onboard visual sense and avoid system for small aircraft. IEEE A&E Syst. Mag. 31, 18–27 (2016)CrossRefGoogle Scholar
  33. 33.
    Zsedrovits, T., Zarandy, A., Vanek, B., Peni, T., Bokor, J., Roska, T.: Collision avoidance for UAV using visual detection. In: Proceedings of IEEE ISCAS 2011 (International Symposium on Circuits and Systems), pp. 2173–2176. IEEE, Rio de Janeiro (2011)CrossRefGoogle Scholar

Copyright information

© Springer Nature B.V. 2018

Authors and Affiliations

  1. 1.Systems and Control Laboratory, Institute for Computer Science and ControlHungarian Academy of Sciences (MTA SZTAKI)BudapestHungary
  2. 2.Computational Optical Sensing and Processing Laboratory, Institute for Computer Science and ControlHungarian Academy of Sciences (MTA SZTAKI)BudapestHungary

Personalised recommendations