Augmented Reality in a Remote Tower Environment Based on VS/IR Fusion and Optical Tracking

  • Maria HaglEmail author
  • Maik Friedrich
  • Anne Papenfuss
  • Norbert Scherer-Negenborn
  • Jörn Jakobi
  • Tim Rambau
  • Markus Schmidt
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10906)


Over the past years, several augmented reality features have been developed to make Remote Tower Operations more cost-efficient and user-friendly. In the context of a national research project (The paper reports results gained in the project “INVIDEON” (FKZ 20 V1505A) sponsored by the Luftfahrtforschungsprogramm (LuFo) of the Federal Ministry of Transport and Digital Infrastructure Germany.), augmented reality based on visual spectrum (VS) and infrared (IR) fusion and as well as on optical tracking is a study objective. Having both VS and IR information available at any time is expected to enable more efficient air traffic control, even at restricted visibility conditions. Integrating VS and IR in one video panorama should also decrease head-down times and therefore increase situation awareness and reduce workload. The integration of two different sensors will be realized by overlaying VS/IR combined with adapted input devices and optical tracking methods. Developing a good concept for the integration of VS/IR and testing it in an exploratory manner can only be achieved with the help of system experts and rapid prototyping methods in simulation environments. During three workshops, human factors specialists, project partners and seven air traffic controllers worked out a prototype that was gradually improved over time and helped to generate a first concept. Firstly, this paper addresses the challenges of VS/IR fusion, manual PTZ following (as a precursor for optical tracking) and adapted input devices. Secondly, it presents the construction process of a prototype in an explorative manner, based on a user-centered approach and implemented in a simulation environment. Finally, it summarizes and presents the results from the workshops and throughout the construction process.


Augmented reality Remote Tower VS/IR fusion PTZ control 


  1. 1.
    Fürstenau, N.: Introduction and overview. In: Fürstenau, N. (ed.) Virtual and Remote Control Tower, pp. 5–12. Springer International Publishing Switzerland, Cham (ZG) (2016). Scholar
  2. 2.
  3. 3.
    Manske, P.G., Schier, S.L.: Visual scanning in an air traffic control tower–a simulation study. Procedia Manuf. 3, 3274–3279 (2015)CrossRefGoogle Scholar
  4. 4.
    Ellis, S.R., Adelstein, B.D., Reisman, R.J., Schmidt-Ott, J.R., Gips, J., Krozel, J., Cohen, M.: Augmented reality in a simulated tower environment: effect of field of view on aircraft detection (2002)Google Scholar
  5. 5.
    Ellis, S.R.: Towards determination of visual requirements for augmented reality displays and virtual environments for the airport tower. NATO R&T Organization (2006)Google Scholar
  6. 6.
    Pinska, E.: An investigation of the head-up time at tower and ground control positions. In: Proceedings of 5th Eurocontrol Innovative Research Workshop, pp. 81–86 (2006)Google Scholar
  7. 7.
    Schmidt, M., Rudolph, M., Papenfuss, A., Friedrich, M., Möhlenbrink, C., Kaltenhäuser, S., Fürstenau, N.: Remote airport traffic control center with augmented vision video panorama. In: IEEE/AIAA 28th Digital Avionics Systems Conference, DASC 2009, p. 4-E. IEEE, October 2009Google Scholar
  8. 8.
    Reisman, R., Brown, D.: Design of augmented reality tools for air traffic control towers. In: 6th AIAA Aviation Technology, Integration and Operations Conference (ATIO), p. 7713, September 2006Google Scholar
  9. 9.
    Roberts, D., Menozzi, A., Cook, J., Sherrill, T., Snarski, S., Russler, P., Clipp, B., Karl, R., Wenger, E., Bennett, M., Mauger, J.: Testing and evaluation of a wearable augmented reality system for natural outdoor environments. In: Head-and Helmet-Mounted Displays XVIII: Design and Applications, vol. 8735, p. 87350A. International Society for Optics and Photonics, May 2013Google Scholar
  10. 10.
    Hofmann, T., König, C., Bruder, R., Bergner, J.: How to reduce workload–augmented reality to ease the work of air traffic controllers. Work 41(Suppl. 1), 1168–1173 (2012)Google Scholar
  11. 11.
    Fürstenau, N., Schmidt, M., Rudolph, M., Möhlenbrink, C., Halle, W.: Augmented vision videopanorama system for remote airport tower operation. In: Grant, I. (ed.) Proceedings of 26 International Congress of the Aeronautical Sciences (ICAS). Optimage Ltd., Edinburgh (2008)Google Scholar
  12. 12.
    Fürstenau, N., Rudolph, M., Schmidt, M., Lorenz, B., Albrecht, T.: On the use of transparent rear projection screens to reduce head-down time in the air-traffic control tower. In: Vincenzi, D.A., Mouloua, M., Hancock, P.A. (eds.) Human Performance, Situation Awareness and Automation Technology: Current Research and Trends, pp. 195–200. Lawrence Erlbaum, Mahwa (2004)Google Scholar
  13. 13.
    Papenfuss, A., Friedrich, M.: Head up only—a design concept to enable multiple remote tower operations. In: 2016 IEEE/AIAA 35th Digital Avionics Systems Conference (DASC), pp. 1–10. IEEE, September 2016Google Scholar
  14. 14.
    Tanaka, J.W., Presnell, L.M.: Color diagnosticity in object recognition. Percept. Psychophys. 61(6), 1140–1153 (1999)CrossRefGoogle Scholar
  15. 15.
    Alleysson, D.: Le traitement du signal chromatique dans la rétine: Un modèle de base pour la perception humaine des couleurs. Doctoral dissertation, Université Joseph-Fourier-Grenoble I (1999)Google Scholar
  16. 16.
    Bajcsy, R., Lieberman, L.: Texture gradient as a depth cue. Comput. Graph. Image Process. 5(1), 52–67 (1976)CrossRefGoogle Scholar
  17. 17.
    Hudson, R.D., Hudson, J.W.: The military applications of remote sensing by infrared. Proc. IEEE 63(1), 104–128 (1975)CrossRefGoogle Scholar
  18. 18.
    Bernard, E., Rivière, N., Renaudat, M., Péalat, M., Zenou, E.: Active and thermal imaging performance under bad weather conditions (2014)Google Scholar
  19. 19.
    Rogalski, A.: Infrared detectors: an overview. Infrared Phys. Technol. 43(3–5), 187–210 (2002)CrossRefGoogle Scholar
  20. 20.
    Brooke, J.: SUS-a quick and dirty usability scale. Usabil. Eval. Ind. 189(194), 4–7 (1996)Google Scholar
  21. 21.
    Bangor, A., Kortum, P., Miller, J.: Determining what individual SUS scores mean: adding an adjective rating scale. J. Usabil. Stud. 4(3), 114–123 (2009)Google Scholar
  22. 22.
    Dehn, D.M.: Assessing the impact of automation on the air traffic controller: the shape questionnaires. Air Traffic Control Q. 16(2), 127–146 (2008)CrossRefGoogle Scholar
  23. 23.
    Coch, L., French Jr., J.R.: Overcoming resistance to change. Hum. Relat. 1(4), 512–532 (1948)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Maria Hagl
    • 1
    Email author
  • Maik Friedrich
    • 1
  • Anne Papenfuss
    • 1
  • Norbert Scherer-Negenborn
    • 2
  • Jörn Jakobi
    • 1
  • Tim Rambau
    • 1
  • Markus Schmidt
    • 1
  1. 1.DLR German Aerospace CenterBrunswickGermany
  2. 2.Fraunhofer IOSBEttlingenGermany

Personalised recommendations