1 Introduction

Foreign Object Debris (FOD) is objects which are located in an inappropriate location in the airport environment that has the capacity to injure airport or air carrier personnel and damage aircraft [1]. For example, screws, wrenches, metal bars, rubber pieces of tires and so on. These foreign objects seriously endanger the safety of aircraft flight and likely cause serious economic losses to the airline.

Research on the FOD detection system is very necessary. Because there are many accidents that FOD injure the aircraft. For example, one of the most serious events is that a long metal strip caused the crash of a concord flight operated by Air France in July 25, 2000. One hundred and thirteen people died in this disaster [2]. Many counties began to develop FOD detection systems since that disaster of the Air France Flight 4590.

The typical FOD detection systems are Tarsier system developed by the British [3], FOD Finder system by United States [4], iFerret system by Singapore [5] and FODetect system by Israel [6]. These systems utilize different sensors and methods to effectively and reliably achieve the goal of FOD detection. However, the primary sensor of Tarsier system is a 94-gigahertz (GHz) coherent radar. The Tarsier system can detect FOD, but can’t classify the FOD. The FOD Finder system is based on a millimeter-band radar mounted on a vehicle. So the cost of the FOD Finder system is expensive. The iFerret system is based on optical or video sensors located on a tower as much as 175 m from the surface to be scanned [5]. The accurate of iFeret system is greatly affected by light conditions, which is obviously low at night. The FOD detect system is composed of a 76-to-77 GHz radar and a video camera located at edge light infrastructures. The deployment way of The FOD detect system is prohibited in China, because it is too dangerous for aircraft. A novel framework for FOD detection system based on Gabor wavelets and support vector machine was proposed by Niu et al. [7]. This system achieves goals of FOD detection and FOD classification. However, the system is mounted on a vehicle and cannot provide continuous surveillance of scanned surfaces.

In this paper, a novel FOD classification system based on visual feature was proposed. The system not only provides continuous surveillance of scanned surfaces and achieves the goal of FOD detection, but also performs FOD classification. FOD classification helps airport management to obtain the extent of the threat of FOD for airplane, making them discard the removal of less harmful FOD, and raise the utilization rate of the runway. This system reduces of FOD to aircrafts, as well as improves the utilization of runways.

2 FOD Detection System

We design a novel FOD detection system based on visual feature. Our system is a Foreign Object Debris detection system consisting of a radar and an electro-optical device and it is greatly different from the FODetect system. Our system effectively achieves the goals of FOD detection and FOD classification. It is mounted on a tower and it provides continuous surveillance of scanned surfaces. Our system mainly contains three parts: radar detection subsystem, electro-optical subsystem and the control center. Figure 1 depicts the FOD detection system.

Fig. 1.
figure 1

FOD detection system

Full size image

The function of the radar detection subsystem is to scan the runway of airports to detect FOD. Radar detection subsystem contains a radar scanning antenna, radar control elements and a servo turntable. The radar detection subsystem scans the entire runway in real-time through turntable servo. The radar detection subsystem then reports the location of FOD and transmits it to the control center when radar detected FOD. The function of the electro-optical subsystem is to take photos of FOD according the FOD location transmitted by the control center. The FOD picture is shown in Fig. 2. The electro-optical subsystem contains a high-definition camera, a rotary device and a subsystem controller. The electro-optical subsystem takes photos of FOD after obtaining the location of FOD, and then transmits the FOD image to the control center. The function of the control center provides a user interface, such as image display on monitor. Furthermore, the control center receives the location of FOD from the radar detection subsystem firstly, then transmits it to the electro-optical subsystem and receives the image of FOD from this later. Then, the control center accomplishes FOD image segmentation and FOD feature extraction, thus achieving the goal of FOD classification eventually.

Fig. 2.
figure 2

FOD image

Full size image

The Foreign Object Debris detection system work as follows. First, the radar detection subsystem scans the runway in real-time. Once found FOD, the radar detection subsystem reports the FOD to the control center, and transmits the location of FOD to the control center. Second, the control center receives the position information of FOD from the radar subsystem, and gives out the alarm or display it in a monitor. Then, the control center transmits the location information to the electro-optical subsystem. Third, The electro-optical subsystem takes photos of FOD, and transmits them to the control center. Fourth, The control center accepts and saves FOD images. Then the FOD images and the classes of FOD are displayed on a monitor after going through the processes of FOD images segmentation, FOD feature extraction and FOD classification. Figure 3, depicts the processes of FOD detection and FOD classification.

Fig. 3.
figure 3

The processes of FOD detection and FOD classification

Full size image

3 FOD Classification

Before FOD feature extraction and classification, we should preprocess the image because the resolution of FOD image obtained by the FOD detection system is 1024 by 1024 pixels, whereas the FOD object occupies a small area on the overall image. If we directly classify the FOD images, the accuracy will be very low. First, an edge detection technique is used to find the edge of FOD in order to segment the FOD object. Then, some algorithms, such as Principal Component Analysis (PCA), Histogram of Oriented Gradient (HOG), Local Binary Patterns (LBP), are utilized to extract FOD feature. Finally, we use support Vector Machine (SVM) to classify FOD image.

3.1 Feature Extraction

3.1.1 Feature Extraction Based on Low-Level Feature

Histogram of Oriented Gradient (HOG) feature was commonly used in computer vision and pattern recognition as a description of the image local texture feature. The method constructs feature by evaluating local histograms of image gradient orientations [8]. The hypothesis is that local object appearance and shape can often be characterized rather well by the distribution of local intensity gradient or edge directions, even without precise knowledge of the corresponding gradient or edge positions [9]. In order to extract HOG feature of FOD, first, we divide the overall image into small connector regions (cell) and evaluate the gradient orientation histogram of each block, then we express the detected target by combining these histograms. In practice, the procedures of HOG feature extraction [9] shown as Fig. 4.

Fig. 4.
figure 4

An overview of static HOG feature extraction.

Full size image

The Local Binary Patterns (LBP) operator was designed for local texture description. It has significant advantages in rotation invariance and gray-invariance. The original LBP operator assigns a label to every pixel of an image by thresholding the 3 × 3-neighborhood of each pixel with the center pixel value and considering the result as a binary number; then, the histogram of the labels can be used as a texture descriptor [10]. See Fig. 5 for an illustration of the basic LBP operator. The extended LBP operator that uses neighborhoods of different sizes was later proposed to deal with textures at different scales [10, 11].

3.1.2 Feature Extraction Based Subspace Feature

Principal Component Analysis (PCA) is a data analysis methods provided by K. Pearson [12]. In practice, many features depend on each other or on an underlying relationship. The core of PCA calculates the new features from a set of features and sorts them in descending order according to their significance. In pattern recognition, the process of PCA is: first, it computes covariance matrix of samples. Then, the characteristic equation is calculated to obtain eigenvalues and eigenvectors. Third, it selects the principal component, and maps the samples to feature subspace [13]. We use certain classifiers to classify feature subspace.

3.2 Multi-class Classification

In the field of machine learning, support vector machine (SVM) is a supervised learning model, typically used for pattern recognition, classification, and regression analysis. The foundation of support vector machine has been developed by Vapnik [14]. Support vector machines are techniques for the classification of both linear and nonlinear data [15]. It uses a nonlinear mapping to transform the original training data into a higher dimension. It makes nonlinear problem in the original sample space becoming linearly separable in the feature space.

There are two main approaches for tackling the multi-label classification problem: problem transformation methods and algorithm adaptation methods [16]. The formers transform the multi-label problem into a set of binary classification problems. For example, one-against-one classifiers and one-against-all classifiers [17]. Those classifiers can handle multi-class problem by using lots of single-class classifiers. The later adapt the algorithms to directly perform multi-label classification.

Fig. 5.
figure 5

The basic LBP operator

Full size image

4 Experiments

We gather a total of 277 FOD images from four classes. As mentioned before, the resolution of the original image obtained by the FOD detection system is large, whereas the FOD object within the overall image is small. So, image background has an effect on the result of the image classification. Hence, we preprocess FOD images as follows. First, we segment FOD images. Then, FOD images are normalized to the same size and transformed into gray pictures. After preprocessing FOD images, the resolution of all FOD images is 64 by 64. An example of preprocessed images from four classes is shown in Fig. 6. The top left is the process of an image from A class (wrench), the top right is for B class (plastic pip), the bottom left is C class (tire debris), and the bottom right is D class (fuel-tank cap).

Fig. 6.
figure 6

Examples of preprocessed FOD images

Full size image

We randomly select 93 FOD images to construct test data sets, the image numbers of A class, B class, C class and D class are 28, 25, 21 and 19 respectively. The rest of the images will constitute the training data sets. The procedure of FOD classification is shown in Fig. 7.

Fig. 7.
figure 7

The procedure of FOD classification

Full size image

We use the algorithm of HOG to extract the feature of FOD, and used multi-class SVM is utilized to classify FOD. Table 1 depicts the confusion matrix of HOG+SVM.

Table 1. Confusion matrix of HOG+SVM.
Full size table

In this work, the HOG method presented in the previous section is used for FOD image description. The FOD objects within the image are very small, and have rich shape and varied posture. This is a fairly difficult database for testing classification accuracy because of its large intra-class variability and small interclass dissimilarity, and furthermore, there are occlusions, scale variation and background clutter on many images. HOG extracts local features of images; hence it has good performance in maintaining geometric distortion and optical deformation.

In order to evaluate the performance of the proposed method in FOD classification, we also use other methods, such as SVM, PCA+SVM and LBP+SVM. From Table 2, it is easy to observe that the proposed system performs well in FOD classification..

Table 2. Accuracy rate of FOD.
Full size table

From Table 2, we can see that the performance of low level features is better than subspace features. The shape of FOD of intra-class is rich. Low level features based local of images maintain good performance in geometric distortion and optical deformation. However, PCA treats all samples as a whole, and ignores the attribute of the class. The projection direction ignored by PCA may contain important the information relevant (significant) information for FOD classification.

5 Conclusions

In this paper, we propose a novel framework of FOD detection system. This system not only achieves the goal of FOD detection and provides continuous surveillance of scanned surfaces, but also provides a performance in FOD classification. The experimental results show good performance of the FOD classification system when low level features to extract FOD are used.