Hierarchical Denoising Method of Crop 3D Point Cloud Based on Multi-view Image Reconstruction

Abstract

Since the advantages of low cost and high efficiency, the three dimensional point cloud reconstruction based on multi-view image sequence and stereo matching has been widely used in agriculture. However, the reconstructed three dimensional point cloud often contains a lot of noise data because of the complex morphology of crop. In order to improve the precision of three dimensional point cloud reconstruction, the paper proposed a hierarchical denoising method which first adopts the density clustering to deal with the large scale outliers, combined with crop morphology analysis, and then smooths the small scale noise with fast bilateral filtering. Two crops of rice and cucumber were taken to validate the method in the experiments. The results demonstrated that the proposed method can achieve better denoising results while preserving the integrity of the boundary of crop 3D model.

1 Introduction

As an application of virtual reality technology in agriculture, crop three dimensional (3D) modeling is an important subject in the study of crop phenotyping and digitization [1]. The method can exploring the natural law of crop growth and development more intuitively by means of the visualization method. At present, the common methods of 3D reconstruction in the field of agriculture always use some professional equipment, such as three-dimensional digitize [2, 3], high precision laser scanner [4, 5], depth imaging equipment [6, 7], etc. The 3D models constructed by these methods can obtain high accuracy. Unfortunately, the professional equipment of these methods are expensive. And these methods usually have high requirements for environmental conditions, e.g. no wind, uniform lighting, and crop morphology, e.g. simple structure, no occlusion between branches and leaves [8]. Comparatively, since the advantages of low cost and high efficiency, the 3D model reconstruction based on multi-view image sequence and stereo matching has been widely used in the field of agriculture [9, 10]. However, since the complex morphology and structure of crops, the 3D point cloud data reconstructed by this method always contains a lot of noise. In order to solve this problem, many works have been proposed. Fleishman et al. [11], Zheng et al. [12] and Dutta et al. [13] proposed the anisotropic mesh denoising algorithms based on the idea of image bilateral filtering which were effective and robust to irregular surface sampling. Lange et al. [14] presented a method for anisotropic fairing of a point sampled surface which removed noise from a point set while it detects and enhances geometric features of the surface such as edges and corners. The method can maintain the morphological characteristics of the model while removing the noise. Jalba and Roerdink [15] proposed a physically motivated method for surface reconstruction which can recover smooth surfaces from noisy and sparse data sets. Wang et al. [16] presented a point clouds denoising method based on moving least-squares which found the adaptive weighting parameter to extend the denoising method to 3D. Yu et al. [17] proposed a two-stage feature-preserving mesh denoising framework which can track and minimize updating errors and hence well preserve the overall shape as well as detailed features of a mesh. Zaman et al. [18] presented a density-based point cloud denoising method to remove outliers and noisy points, which used particle-swam optimization technique and obtained the good effect for isolated points. He et al. [7] proposed a denoising algorithm based on density analysis and depth data bilateral filtering to process the outlier noise and internal high-frequency noise.

Although the above methods have achieved good results in some aspects, they still need to be further improved while applying in the field of agriculture. More concretely, the efficiency of the method based on curvature is low since it needs to calculate the curvature of each point. The denoising effect of traditional bilateral Filtering is not good enough for the outliers. Some denoising algorithms can cause the edge distortion.

In order to solve these problems, combining with the crop morphology analysis, the paper proposed a denoising method of crop 3D point cloud data reconstructed based on multi-view images. In this method, different denoising methods are used for different scales of noise. For large scale outliers, since crop morphology is irregular, it requires that the selected method should preserve the integrity of the crop model edge while removing large-scale outliers. Therefore, the density clustering method called DBSCAN (Density-based Spatial Clustering of Applications with Noise) [19] is used. With this algorithm, the number of point cloud data clusters of arbitrary shape can be obtained automatically without interference from the noises, since the classification of this algorithm is not based on the distance. And for smoothing small-scale noises, inspiring by the work of Paris and Durand [20], we extend the fast bilateral filtering algorithm to the higher-dimensional space, namely the 3D point cloud data. Through the above combination of different scale denoising algorithms, the proposed method can effectively achieve the removal of outliers and internal smoothing while preserving the integrity of crop 3D model.

2 Materials and Methods

2.1 Crop Point Cloud Acquisition

This paper involves two crops, that is, cucumber and rice. The multi-view image sequences of cucumber and rice were captured around the plant indoor, using Canon digital SLR camera EOS 6D, with EF 17-40 mm f/4L USM lens. Each image sequence includes 50 to 80 images. The original image resolution is 5472 * 3648 pixels. In the reconstruction of crop 3D point cloud, first the method called Structure from Motion (SFM) [21] is adopted to calibrate the camera parameters and obtain the sparse point cloud. The SFM method uses SIFT algorithm [22] to extract feature points from the images and incorporates a series of matching points of each two images into the trajectory, which is used to recover the camera parameters and the 3D position information. And then in order to improve the efficiency of dense matching, clustering views for multi-view stereo (CMVS) is used to cluster images and optimize the input of SFM. Finally, three main steps of matching, expansion and filtering are conducted by using patch-based multi-view stereo (PMVS) [23] to obtain the dense point cloud.

In addition, the crop 3D point cloud data obtained by the above multi-view stereo matching usually contains a large amount of background data, which is not conducive to the later denoising of the point cloud data. Therefore, the background data should be removed in preprocessing. First we observe the initial point cloud data to label the 3D space bounding box of the crop. So that the value range of the point cloud data coordinate can be obtained. And then the efficient and simple Gauss filter is adopted to filter the background data. After the preprocessing of removing the background data, since the complex crop morphology and the influences of many environment factors, the crop 3D point cloud data still contains a lot of noises, which can be divided into two categories, that is, the large scale outlier noises and the small scale internal high-frequency noises. Because of the different spatial distributions, it is difficult to use some conventional method to remove the two kinds of noise at one time. Therefore, the paper proposes a method which deals with the large scale noises and the small scale noises separately according to their spatial distributions. The large scale noises are removed based on the density clustering method DBSCAN and the small scale noises are smoothed by the extended fast bilateral filtering algorithm.

2.2 Density Clustering for Removing Outlier Noises

Since the classification of the density clustering method DBSCAN is not based on the distance, the number of point cloud data clusters of arbitrary shape can be obtained automatically without interference from the noises, which is suitable for dealing with the crop 3D point cloud of complex morphology.

Two parameters $Eps$ and $Int$ are very important in the density clustering method DBSCAN. So before clustering, a statistical analysis of the density information of crop 3D point cloud data is required to select the appropriate parameter values. First, the distance distribution matrix $Dist_{k} = \{ dist_{k} (p)|p \in D\}$ is calculated by searching the nearest kth object from the given object $p \in D$. And then the kNN diagram can be obtained by sorting the matrix $Dist_{k}$, where the value of $k$ ranges from 1 to 80 in Fig. 1(a). As shown in Fig. 1(b), the diagram of $k = 4$ can represent the ascending and descending trend of other diagrams. It means that the parameter $Int$ is valued at 4. And then the parameter value of $Eps$ can be determined by locating the rising point of the diagram of $k = 4$.

Fig. 1.

The determination of parameters in density clustering for removing outlier noises

The procedure of density clustering for removing outlier noises is given as follows:

• Calculating the $k$ distance value for each point to form the distance distribution matrix $Dist_{k}$.

• Drawing the kNN diagram by sorting the distance matrix for all points in ascending order.

• Selecting the appropriate $Eps$ parameter according to the kNN diagram.

• Constructing the space index by adopting the data structure of $R^{ * }$ in clustering to search the data point.

• Observing the clustering effect: The procedure is finished if the noise point can be effectively removed; Otherwise, going back to select a more appropriate parameter $Eps$ again.

2.3 Fast Bilateral Filtering for Smoothing Internal Noises

After removing the large scale outlier noises, the crop 3D point cloud still contains a lot of small scale non-smooth noises, which will have a bad effect on the later 3D surface reconstruction. Since the traditional bilateral filtering is nonlinear, it requires linear convolution of the image, which makes the method very time-consuming. In order to improve the algorithm efficiency, Paris and Durand [20] proposed a bilateral grid filtering method, which combines the 2D coordinates of pixels with their gray values into 3D space and adopts the linear filtering method to accelerate the algorithm after performing sampling in both spatial and gray domains. Inspiring by this work, we extend the fast bilateral filtering algorithm to the higher-dimensional space, namely the 3D point cloud data, to smooth the small scale noises.

Since the point cloud is scattered and three-dimensional, the dimension increasing sampling is not required in the filtering. The method directly performs the Gauss linear convolution operation and calculates the coordinate value by the bilinear interpolation function to smooth the point cloud data. The fast filtering algorithm for 3D point cloud is shown as the following formula.

$$ I_{b} (x,y) = \frac{{WI_{b} (x,y)}}{{W_{b} (x,y)}} = \frac{in(G \otimes WI,x,y,z)}{in(G \otimes W,x,y,z)} $$

(1)

where $in$ is an interpolating function, which interpolates $G \otimes WI$ and $G \otimes W$ in 3D space to calculate the corresponding values on the coordinate $(x,y,z)$. $WI$ and $W$ are shown in the following formula.

$$ WI(x,y,z) = \sum\limits_{(u,v) \in R(x,y,z)} {d_{i} (u,v)} $$

(2)

$$ W(x,y,z) = \sum\limits_{(u,v) \in R(x,y,z)} 1 $$

(3)

where $d_{i} (u,v)$ is the depth value of the neighborhood of the point $p_{i}$, $R$ is the neighborhood of the point $p_{i}$, $G$ is the product of spatial proximity factor $G_{s}$ and depth similarity factor $G_{d}$, defined by the following formulas:

$$ G_{s} = \exp ( - \frac{{|x - u|^{2} + |y - v|^{2} }}{{2\sigma_{s}^{2} }}) $$

(4)

$$ G_{d} = \exp ( - \frac{{|d_{i} (x,y) - d_{i} (u,v)|^{2} }}{{2\sigma_{d}^{2} }}) $$

(5)

where $\sigma_{s}$ and $\sigma_{d}$ are standard deviation of spatial distance and depth based on Gauss’s function respectively.

Compared to traditional bilateral filtering, the above algorithm can achieve significant improvement in time efficiency.

2.4 Integration of Density Clustering and Fast Bilateral Filtering

By integrating the above two algorithms, the denoising of large scale outlier noises and the smoothing of small scale internal noises can be achieved. The procedure is given as follows:

• (1) Input point cloud data and calculate the data quantity of the point cloud model.

• (2) By using octree nearest neighbor search, calculate the neighborhood points $k_{ij} ,j = 1, \ldots ,m$ for each data point $p_{i}$.

• (3) Compute spatial proximity factor $G_{s}$, depth similarity factor $G_{d}$ and their product $G$.

• (4) Calculate the sum of depths $WI$ and 3D weight $W$ of the neighborhood set of the data point $p_{i}$.

• (5) Make the convolution operation between $WI$, $W$ and $G$ respectively in 3D space.

• (6) Perform the interpolation operation on the data point $p_{i}$ to obtain the corresponding coordinate value.

• (7) Calculate $I_{b} (x,y)$, according to the formula (1), to get the smoothed 3D point cloud data.

3 Experiments and Discussion

3.1 Experimental Setup

The experiments consist of two parts:

• Validating the effectiveness of our method on crop 3D point cloud data.

• Evaluating the generality of our method in standard test point cloud data.

Other two similar algorithms were used for comparison. Concretely, the work of Yang and Xiao [24] were based on K-means clustering. And the work of Li et al. [25] also divided the noises into different categories, and then adopted different methods for denoising.

The experimental hardware and software environment are as follows: Intel Core i7-4790@3.66 GHz CPU, 4 GB RAM, Microsoft Windows 10 professional edition 64-bit, Microsoft Visual Studio 2013, Open Source Computer Vision Library (OpenCV) 2.4.8, and Matlab R2014b.

3.2 Experiments on Crop 3D Point Cloud Data

As mentioned above, the background of 3D point cloud data of rice and cucumber need to be filtered in the preprocessing, where the selected coordinate segmentation thresholds are shown in Table 1.

Table 1. Coordinate segmentation threshold for filtering the background of 3D point cloud

After preprocessing, the crop 3D point cloud data of rice and cucumber contain 57676 and 105176 points respectively. According to the kNN diagram, the parameters $Eps$ and $Int$ of rice and cucumber are set to (0.023, 5) and (0.37, 7) respectively in the density clustering for removing outlier noises. The experimental results are given in Table 2, where N₀ is the number of points in the original point cloud data after preprocessing, N₁, N₂, N₃ are the denoising results of Yang and Xiao [24], Li et al. [25] and the proposed method respectively.

Table 2. The experimental results of removing outlier noises

From Table 2 we can see that compared with the other two similar methods, the density clustering method adopted in this paper can remove the outlier noises of 3D point cloud more effectively. More intuitive examples of denoising effect are shown in Figs. 2 and 3.

Fig. 2.

An example of removing outlier noises for rice 3D point cloud

Fig. 3.

An example of removing outlier noises for cucumber 3D point cloud

When using fast bilateral filtering to smooth internal noises, how to choose the appropriate parameters $\sigma_{s}$ and $\sigma_{d}$ is important. Since the spatial texture of rice and cucumber is simple, the parameter $\sigma_{s}$ has little influence on the smoothing effect. For the parameter $\sigma_{d}$, it will cause excessive smoothing when $\sigma_{d}$ is too large and the smoothing effect is not good when $\sigma_{d}$ is too small. Therefore, in order to select the appropriate parameters, this paper conducted the comparative experiments on different parameter combinations. Figure 4 shows the experimental results of different parameters for rice 3D point cloud. We can see that the smooth effect is best for the given rice point cloud when $\sigma_{s} = 5$ and $\sigma_{d} = 0.2$.

Fig. 4.

The effect of 4 groups parameters ($\sigma_{s}$, $\sigma_{d}$) for the input rice 3D point cloud

Figure 5 gives the experimental results of different parameters for cucumber 3D point cloud, which shows that the parameter combination of $\sigma_{s} = 5$ and $\sigma_{d} = 0. 1$ has the best smoothing effect for cucumber 3D point cloud.

Fig. 5.

The effect of 4 groups parameters ($\sigma_{s}$, $\sigma_{d}$) for the input cucumber 3D point cloud

After selecting the appropriate parameters $\sigma_{s}$ and $\sigma_{d}$, we conducted the experiments for comparing fast bilateral filtering with traditional bilateral filtering. Both rice and cucumber contain 5 point cloud models. The evaluation criteria include maximum offset error, which is the maximum distance that the coordinate point moves after smoothing, denoted by maxE, and average error, which indicates the noise level of the crop point cloud after smoothing, denoted by aveE. Besides, the time spent by the algorithm is also used as one of the evaluation criteria. Table 3 gives the results, where TBF and FBF denotes traditional bilateral filtering and fast bilateral filtering respectively. We can see that the smoothing effect of these two methods is almost the same. However, our method is more efficient. The time spent is obviously less than the traditional bilateral filtering, 48% of rice and 46% of cucumber.

Table 3. The comparison between traditional bilateral filtering and fast bilateral filtering

3.3 Experiments on Standard 3D Point Cloud Data

The in order to further evaluate the generality of density clustering for removing outlier noises, we also conducted the experiments on the standard 3D point cloud data, including Bunny and Horse. First we manually added outlier noises into the standard test point cloud data. And then as mentioned above, the works of Yang and Xiao [24] and Li et al. [25] were used for comparison in the denoising task. The experimental results of iterating 20 times are given in Table 4, where N₀ is the number of points in the original point cloud data, N_n is the number of points after adding noises, N₁, N₂, N₃ are the denoising results of Yang and Xiao [24], Li et al. [25] and the proposed method respectively, together with the denoising efficiency, denoted by E₁, E₂, E₃. From the experimental results, we can see that the method of this paper can achieve the best denoising effect.

Table 4. Density clustering for removing outlier noises for standard 3D point cloud data

The more intuitive comparison of the experimental results is shown in Fig. 6. Our method can better preserve the boundary of the standard point cloud model while achieving better effect than other contrastive algorithms in the removal of outlier noises.

Fig. 6.

The denoising results of the standard 3D point cloud Bunny and Horse: (a) The original point cloud; (b) The point cloud with manually adding noises; (c) The result of Yang and Xiao [24]; (d) The result of Li et al. [25]; (e) The result of the proposed method.

4 Concluding Remarks

According to the characteristics of noise distribution of crop 3D point cloud model reconstructed based on multi-view stereo matching, the paper proposed a hierarchical denoising method. The 3D point cloud models of rice and cucumber were used in experiments to validate this method. And the experimental results show that:

• (1) Comparing with the similar algorithms, the density clustering method can achieve better effect in the removal of large scale outlier noises.

• (2) Comparing with the traditional bilateral filtering algorithm, the fast bilateral filtering can significantly improve the smoothing efficiency for small scale internal noises.

• (3) By combining the two different scale denoising algorithms, the proposed method can effectively achieve the removal of outliers and internal smoothing while preserving the integrity of crop 3D model.

• (4) The proposed method is universal and can be applied to the denoising task of other 3D point cloud models.