Hardware Implementation of KLT Tracker for Real-Time Intruder Detection and Tracking Using On-board Camera

  • Taehyeong Nam
  • Seho Kim
  • Dongwon JungEmail author
Original Paper


This paper presents the hardware implementation of a new hybrid KLT (Kanade–Lucas–Tomasi) algorithm for the sake of real-time intruder detection and tracking. The proposed algorithm takes advantage of BRISK sampling scheme to improve the feature detection and tracking performance. The RTL level hardware logics for feature detection and tracking are designed to implement the hybrid KLT tracker using on-board SoC FPGA. In particular, the hybrid KLT tracker is realized via a five-stage pipelining architecture to increase the throughput rate but also the availability of continuous output. The RTL hardware logics are written in VHDL codes and are executed on an Intel’s Cyclone V SoC development board while the video frames of VGA resolution are obtained from the on-board camera at 60 fps. The performance of the hybrid KLT tracker algorithm has been tested and validated via the software simulation based on the reference codes written in C/C++. The speed acceleration due to the hardware implementation has been verified so that the computational throughput increases by 24.46 times faster than the software implementation, while the object tracking performance remains the same. Consequently, it has been shown that the hardware implementation of KLT tracker for intruder detection and tracking can operate in real-time while providing the supplement surveillance information using on-board camera for mid-air collision avoidance of UAV.


FPGA KLT tracker BRISK sampling Parallel processing System-on-chip 



This research was supported by the Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT (No. NRF-2017M1A3A3A02016572) for algorithm development, and also supported by the Program through the Korea Evaluation Institute of Industrial Technology (KEIT) funded by the Ministry of Trade, Industry and Energy. (Project No. 10059035) for hardware implementation.


  1. 1.
    McCallie D, Butts J, Mills R (2011) Security analysis of the ADS-B implementation in the next generation air transportation system. Int J Crit Infrastruct Prot 4(2):78–87. CrossRefGoogle Scholar
  2. 2.
    Baek KY, Bang HC (2012) ADS-B based trajectory prediction and conflict detection for air traffic management. Int J Aeronaut Space Sci 13(3):377–385. CrossRefGoogle Scholar
  3. 3.
    Stark B, Stevenson B, Chen YQ (2013) ADS-B for small unmanned aerial systems: case study and regulatory practices. International Conference on Unmanned Aircraft Systems (IUCAS), Atlanta. CrossRefGoogle Scholar
  4. 4.
    Strohmeier M, Schafer M, Lenders V, Martinovic I (2014) Realities and challenges of Nextgen air traffic management: the case of ADS-B. IEEE Commun Mag 52(5):111–118. CrossRefGoogle Scholar
  5. 5.
    Wilson M, Ryan D, Bratanov D, Wainwright A, Ford J, Cork L, Brouckaert M (2016) Flight test and evaluation of a prototype sense and avoid system onboard a ScanEagle unmanned aircraft. IEEE Aerosp Electron Syst Mag 31(9):6–15. CrossRefGoogle Scholar
  6. 6.
    Park JW, Oh HY, Takh MJ (2009) UAV conflict detection and resolution based on geometric approach. Int J Aeronaut Space Sci 10(1):37–45. CrossRefGoogle Scholar
  7. 7.
    Abbas SHS, Lee JR, Kim Z (2016) FPGA-based design and implementation of data acquisition and real-time processing for laser ultrasound propagation. Int J Aeronaut Space Sci 17(4):467–475CrossRefGoogle Scholar
  8. 8.
    Kim YS, Kim JH, Han DI, Lee MH, Park JH, Lee DW (2016) FPGA based HW/SW co-design for vision based real-time position measurement of an UAV. Int J Aeronaut Space Sci 17(2):232–239CrossRefGoogle Scholar
  9. 9.
    Pauwels K, Tomasi M, Alonso JD, Ros E, Van Hulle MM (2012) A Comparison of FPGA and GPU for real-time phase-based optical flow, stereo, and local image features. IEEE Trans Comput 61(7):999–1012. MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Chase J, Nelson B, Bodily J, Wei Z and Lee DJ (2008) Real-time optical flow calculations on FPGA on GPU architectures: a comparison study. Int Symposium on Field-Programmable Custom Computing Machines, Palo Alto, CA.
  11. 11.
    Berten Digital Signal Processing (2016) GPU vs FPGA performance comparison. White Paper, Accessed 12 Nov 2018
  12. 12.
    Kalarot R and Morris J (2010) Comparison of FPGA and GPU implementations of real-time stereo vision. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshop, San FranciscoGoogle Scholar
  13. 13.
    Fowers J, Brown G, Coock P, and Stitt G (2012) A performance and energy comparison of FPGAs, GPUs, and multicores for sliding window applications. Int Symposium on Field-Programmable Gate Arrays, Monterey.
  14. 14.
    Jang W, Oh S, and Kim G (2009) A hardware implementation of pyramidal KLT feature tracker for driving assistance system. IEEE Conference on Intelligent Transportation Systems, St. Louis.
  15. 15.
    Schlessman J, Chen CY, Wolf W, Ozer B, Fujono K, and Itoh K (2006) Hardware/software co-design of an FPGA-based embedded tracking system. IEEE Conference on Computer Vision and Pattern Recognition Workshop, New York.
  16. 16.
    Chai Z and Shi J (2011) Improving KLT in embedded systems by processing oversampling video sequence in real-time. IEEE Int Conference on Reconfigurable Computing and FPGAs, Cancun, Quintana Roo.
  17. 17.
    Ali K, Khan SA, and Akram U (2013) Aircraft tracking based on KLT feature tracker and image modeling. IEEE Jordan Conference on Applied Electrical Engineering and Computing Technologies, Amman.
  18. 18.
    Yoo DW, Won DY, Tahk MJ (2011) Optical flow based collision avoidance of multi-rotor UAVs in urban environments. Int J Aeronaut Space Sci 12(3):252–259. CrossRefGoogle Scholar
  19. 19.
    Yilmaz A, Javed O, Shah M (2006) Object tracking: a survey. ACM Comput Surv (CSUR) 38(4):13. CrossRefGoogle Scholar
  20. 20.
    El-Shafie AA and Habib SED (2017) A survey on hardware implementation of visual object trackers. Image and Video Processing, Electronic Engineering and Systems Science, Cornell University Library.
  21. 21.
    Won DH, Sung SK, Lee YJ (2012) Particle filter based feature points tracking for vision based navigation system. J Korean Soc Aeronaut Space Sci 40(1):35–42. CrossRefGoogle Scholar
  22. 22.
    Shi J and Tomasi C (1994) Good features to track. IEEE Conference on Computer Vision and Pattern Recognition, Seattle.
  23. 23.
    Harris C and Stephens M (1988) A combined corner and edge detector. Proceedings of Fourth Alvey Vision Conference, ManchesterGoogle Scholar
  24. 24.
    Bailey DG (2011) Design for embedded image processing on FPGAs. Wiley, HobokenCrossRefGoogle Scholar
  25. 25.
    Leutenegger S, Chli M, and Siegward RY (2011) BRISK: binary robust invariant scalable keypoints. IEEE International Conference on Computer Vision, Barcelona.
  26. 26.
    Birchfield S (2007) KLT: an implementation of the Kanade–Lucas–Tomasi feature tracker (v.1.3.4). Accessed 12 Nov 2018
  27. 27.
    Laminar Research (2018) X plane 11. Accessed 12 Nov 2018

Copyright information

© The Korean Society for Aeronautical & Space Sciences 2019

Authors and Affiliations

  1. 1.Korea Aerospace UniversityGoyang-SiRepublic of Korea

Personalised recommendations