Augmenting Flight Imagery from Aerial Refueling

  • James D. Anderson
  • Scott NyklEmail author
  • Thomas Wischgoll
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11845)


When collecting real-world imagery, objects in the scene may be occluded by other objects from the perspective of the camera. However, in some circumstances an occluding object is absent from the scene either for practical reasons or the situation renders it infeasible. Utilizing augmented reality techniques, those images can be altered to examine the affect of the object’s occlusion. This project details a novel method for augmenting real images with virtual objects in a virtual environment. Specifically, images from automated aerial refueling (AAR) test flights are augmented with a virtual refueling boom arm, which occludes the receiving aircraft. The occlusion effects of the boom are quantified in order to determine which pixels are not viable for stereo image processing to reduce noise and increase efficiency of estimating aircraft pose from stereo images.


Augmented reality Virtual reality simulation Vision occlusion 


  1. 1.
    Bobick, A.F., Intille, S.S.: Large occlusion stereo. Int. J. Comput. Vision 33, 181–200 (1999). Scholar
  2. 2.
    Campa, G., Mammarella, M., Napolitano, M.R., Fravolini, M.L., Pollini, L., Stolarik, B.: A comparison of pose estimation algorithms for machine vision based aerial refueling for UAVs. In: 14th Mediterranean Conference on Control and Automation, MED 2006 (2006).
  3. 3.
    Campa, G., Napolitano, M.R., Fravolini, M.L.: Simulation environment for machine vision based aerial refueling for UAVs. IEEE Trans. Aerosp. Electron. Syst. 45, 138–151 (2009). Scholar
  4. 4.
    Crow, F.C.: Shadow algorithms for computer graphics. ACM SIGGRAPH Comput. Graphics 11, 242–248 (2005). Scholar
  5. 5.
    Dallmann, W.E.: Infrared and electro-optical stereo vision for automated aerial refueling. Master’s thesis, Air Force Institute of Technology (2019)Google Scholar
  6. 6.
    Duan, H., Zhang, Q.: Visual measurement in simulation environment for vision-based UAV autonomous aerial refueling. IEEE Trans. Instrum. Meas. 64, 2468–2480 (2015). Scholar
  7. 7.
    Fravolini, M.L., Brunori, V., Ficola, A., La Cava, M., Campa, G.: Feature matching algorithms for machine vision based autonomous aerial refueling. In: 14th Mediterranean Conference on Control and Automation, MED 2006 (2006).
  8. 8.
    Fravolini, M.L., Campa, G., Napolitano, M.R.: Evaluation of machine vision algorithms for autonomous aerial refueling for unmanned aerial vehicles. J. Aerosp. Comput. Inf. Commun. 4, 968–985 (2008). Scholar
  9. 9.
    Huang, X., Walker, I., Birchfield, S.: Occlusion-aware reconstruction and manipulation of 3D articulated objects. In: Proceedings - IEEE International Conference on Robotics and Automation (2012).
  10. 10.
    Johnson, D.T., Nykl, S.L., Raquet, J.F.: Combining stereo vision and inertial navigation for automated aerial refueling. J. Guidance Control Dyn. 40, 2250–2259 (2017). Scholar
  11. 11.
    Kimmett, J., Valasek, J., Junkins, J.: Autonomous aerial refueling utilizing a vision based navigation system. In: AIAA Guidance, Navigation, and Control Conference and Exhibit. American Institute of Aeronautics and Astronautics (2002).
  12. 12.
    Lepetit, V.: On computer vision for augmented reality. In: Proceedings - International Symposium on Ubiquitous Virtual Reality, ISUVR 2008 (2008).
  13. 13.
    Mammarella, M., Campa, G., Napolitano, M.R., Fravolini, M.L.: Comparison of point matching algorithms for the UAV aerial refueling problem. Mach. Vis. Appl. 21, 241–251 (2010). Scholar
  14. 14.
    Nykl, S., Mourning, C., Leitch, M., Chelberg, D., Franklin, T., Liu, C.: An overview of the STEAMiE educational game engine. In: Proceedings - Frontiers in Education Conference, FIE (2008).
  15. 15.
    Parsons, C.A.: Improving automated aerial refueling stereo vision pose estimation using a shelled reference model. Master’s thesis, Air Force Institute of Technology (2017)Google Scholar
  16. 16.
    Parsons, C., Nykl, S.: Real-time automated aerial refueling using stereo vision. In: Bebis, G., et al. (eds.) ISVC 2016. LNCS, vol. 10073, pp. 605–615. Springer, Cham (2016). Scholar
  17. 17.
    Parsons, C., Paulson, Z., Nykl, S., Dallman, W., Woolley, B.G., Pecarina, J.: Analysis of simulated imagery for real-time vision-based automated aerial refueling. J. Aerosp. Inf. Syst. 16(3), 77–93 (2019). Scholar
  18. 18.
    Paulson, Z., Nykl, S., Pecarina, J., Woolley, B.: Mitigating the effects of boom occlusion on automated aerial refueling through shadow volumes. J. Defense Model. Simul. 16, 175–189 (2019). Scholar
  19. 19.
    Pollini, L., Campa, G., Giulietti, F., Innocenti, M.: Virtual simulation set-up for UAVs aerial refuelling. In: AIAA Modeling and Simulation Technologies Conference and Exhibit. American Institute of Aeronautics and Astronautics (2012).
  20. 20.
    Shah, M.M., Arshad, H., Sulaiman, R.: Occlusion in augmented reality. In: 8th International Conference on Information Science and Digital Content Technology (ICIDT 2012) (2012)Google Scholar
  21. 21.
    Zanfir, A., Sminchisescu, C.: Large displacement 3D scene flow with occlusion reasoning. In: Proceedings of the IEEE International Conference on Computer Vision (2015).

Copyright information

© This is a U.S. government work and not under copyright protection in the U.S.; foreign copyright protection may apply 2019

Authors and Affiliations

  1. 1.Wright State UniversityDaytonUSA
  2. 2.Air Force Institute of TechnologyDaytonUSA

Personalised recommendations