Keywords

1 Introduction

The quality of images of outdoor scenes are usually degraded by adverse weather conditions like fog and haze. The degraded images often have low contrast and glow effects, which affect the normal performance of many computer vision systems since most of them assume that the input image is the haze-free scene radiance. Therefore, the effective haze removal of image is a work with great significance.

Many existing image dehazing algorithms are designed for daytime haze removal [1,2,3,4]. Almost all of these methods rely on the atmospheric scattering model [5] and the estimation of the parameters in the model. Based on this model, Tan [1] proposed an image dehazing method by maximizing local image contrast. Fattal [2] estimated medium transmission using statistical independence between shading and albedo. He [3] restored a hazy image using the Dark Channel Prior which was derived from statistics of daytime haze-free images. Even though these methods are effective for daytime haze images, they show great defects under the nighttime hazy scenes. This is mainly due to the spatially non-uniform environmental illumination are present at nighttime scenes that can generate glow.

Recently, there have been some methods for nighttime haze removal [6,7,8,9]. Pei et al. [6] propose a color transfer method as a preprocessing step to transform the brightness and color of a nighttime haze image into a daytime haze image. Then the Dark Channel Prior method is used to estimate the transmission as well as bilateral filtering is applied as a post-processing step. Zhang et al. [7] proposed a new model to account for various non-uniform environmental illumination and removed the haze after light compensation and color correction. Ancuti et al. [8] proposed a nighttime haze removal algorithm based on image fusion. They estimated the atmospheric light component on image patch rather on the entire image by generating two inputs using multiple patch sizes and the Laplacian of the original image is defined to be the third input to reduce the glowing effect.

The above methods are effective for haze removal and reduce the effects of glow, but they do not really remove the glow since the models they adopted do not account for glow effects. Li et al. [9] modelled the glow effects by adding the atmospheric point spread function into the daytime haze imaging model. He decomposed the glow image from the original input image and restored the scene radiance using Dark Channel Prior. Although Li’s method removed haze and glow effects, the resultant images may contain noise and blocking artifacts. In our work, we propose a novel method to solve these problems. First, we adopt Li’s method to decompose the glow image. Then we estimate the atmospheric light using image fusion approach. Finally, transmission is estimated by maximizing the improved entropy.

The rest of paper is organized as follows. Section 2 introduces the nighttime imaging model. Section 3 describes the proposed approach. Section 4 provides the experimental results and comparisons with conventional works and the work is concluded in Sect. 5.

2 Nighttime Haze Imaging Model

In computer vision and computer graphics, the daytime haze imaging model which widely used in many image de-haze approaches is called the atmospheric scattering model [5]. Figure 1(a) shows a diagram of the daytime haze imaging model. The pixel intensity in the captured image is a linear combination of two parts: the direct transmission part and the airlight part. Mathematically, it can be expressed as:

$$\begin{aligned} I\left( x \right) = J\left( x \right) t\left( x \right) + A(1 - t(x)), \end{aligned}$$
(1)

where I(x) is the observed intensity at pixel x, J(x) is the scene radiance when there is no haze or fog particles. A is the global atmospheric light, and t(x) is the medium transmission describing the portion of the light that is not scattered and reaches the camera. J(x)t(x) is called direct transmission, and A(1 − t(x)) is called atmospheric light. For a haze image I(x), the goal of haze removal is to recover J(x) from I(x).

Fig. 1.
figure 1

Nighttime haze imaging model.

Full size image

In the daytime imaging model, it assumes that the atmospheric light intensity is global constant. Nevertheless, nighttime scenes generally have active light sources that can generate glow when the presence of particles in the atmosphere is substantial. To express the active light sources and glow under the nighttime scenes, Li et al. [9] proposed a novel model for nighttime haze imaging by adding a glow model with an atmospheric point spread function (APSF) [10] into the daytime haze imaging model as follows [11]:

$$\begin{aligned} I\left( x \right) = J\left( x \right) t\left( x \right) + A\left( x \right) \left( {1 - t\left( x \right) } \right) + {A_a}\left( x \right) * APSF, \end{aligned}$$
(2)

where A(x) is the atmospheric light which is no longer globally uniform, but space varying. A \(_a\) is the active light sources, that the intensity is convolved with APSF. J(x)t(x) + A(x)(1 − t(x)) is called the nighttime haze image, \(A_a*APSF\) is the glow image. Figure 1(b) shows diagram of the nighttime haze model. The active light sources is scattered as it travels through the suspended particles (haze and fog), resulting the glowing effects in the captured image. Given a nighttime haze image I(x), the goal is to restore the scene radiance J(x) by decomposing glow from I(x) and estimating both the transmission t(x) and the varying atmospheric light A(x).

3 Nighttime Haze Removal

Our basic pipeline is illustrated in Fig. 2. It starts by decomposing the original input image into two images: nighttime haze image and glow image, as shown in Fig. 2(b), (c). Then, we estimate the atmospheric light by combining multiple patch sizes local atmospheric light using multiscale fusion algorithm as shown in Fig. 2(j). Transmission is estimated using the nighttime haze image and atmospheric light that has been estimated as shown in Fig. 2(k). Finally, the two estimated parameters are used to restore the scene radiance as shown in Fig. 2(l). In the following, the details of each step are discussed.

Fig. 2.
figure 2

The framework of the proposed method. (a) Original input image. (b) Nighttime haze image. (c) Glow image. (d) Local atmospheric light with size 10 \(\times \) 10. (e) Local atmospheric light with size 30 \(\times \) 30. (f), (g) Refined results of (d), (e). (h), (i) Normalized weight maps of (f), (g). (j) Fusion atmospheric light. (k) Estimated transmission. (l) Output image.

Full size image

3.1 Glow Decomposition

In the nighttime scenes, the brightness of the glow decreases gradually and smoothly away from the light sources due to the multiple scattering, it causes that the glow image has a “short tail” distribution in the gradient histogram [11]. Based on this fact, Li et al. proposed the glow decomposition method [11] by layer separation using relative smoothness. In this paper, we adopt Li’s approach [9] to decompose the glow image from original input. The objective function for layer separation is defined as:

$$\begin{aligned} \begin{array}{l} E\left( R \right) = \sum \limits _x {\left( {\rho \left( {R\left( x \right) * {f_{1,2}}} \right) + \lambda {{\left( {\left( {I\left( x \right) - R\left( x \right) } \right) * {f_3}} \right) }^2}} \right) } \\ \quad s{.}t.\;{0 \le R\left( x \right) } \le I\left( x \right) ,\\ \quad \sum \nolimits _x {{R_r}\left( x \right) } = \sum \nolimits _x {{R_g}\left( x \right) } = \sum \nolimits _x {{R_b}\left( x \right) } . \end{array} \end{aligned}$$
(3)

where R(x) = J(x)t(x) + A(x)(1−t(x)) is the nighttime haze image, f \(_{1,2}\) is the two direction first order derivative filters, f \(_3\) is the second order Laplacian filter and the operator \(*\) denotes convolution. \(\rho \)(s) = min(s \(^2\),\(\tau \)) is a robust function which preserve the large gradients of input image I in the remaining nighttime haze layer R(x). \(\lambda \) is the parameter that controls the smoothness of the glow layer. The first inequality constraint is to ensure the solution is in a proper range, and the second constraint is to force the range of the intensity values for difference color channels to be balanced. The objective function in Eq. (3) can be solved efficiently using the half-quadratic splitting technique [11].

3.2 Atmospheric Light Estimation

After glow decomposition, we get the nighttime haze image. To obtain the haze-free scene radiance, we still need to estimate the atmospheric light and transmission. In a previous work [9], Li assumed that atmospheric light is locally constant and the brightest intensity in a local area is the atmospheric light of that area. He split the original input image into a grid of small patches (15 \(\times \) 15) and found the brightest pixel in each area as atmospheric light. However, Li’s estimation of the varying atmospheric light is admittedly a rough approximation that suffers from two weaknesses. First, there may be of the boosting noise and blocking artifacts in the de-hazed results as shown by the sky region of Fig. 3(b). Second, it is very hard to select the size of the patches, since small patches are desirable to achieve fine spatial adaptation to the atmospheric light, but might also induce poor light estimates and reduced chance of capturing hazy pixels. By choosing a too large patch size, the haze is better removed, but the color might be shifted, the influence of the airlight might not be entirely removed and some details may remain poorly restored [8].

Fig. 3.
figure 3

Dehazed results using different atmospheric light estimation methods.

Full size image

To circumvent this problem, we employ a multiscale fusion approach to estimate the atmospheric light [8]. We assume that the atmospheric light is locally constant and define the local atmospheric light intensity to be:

$$\begin{aligned} {A^c}\left( x \right) = {\max _{y \in \psi (x)}}\left[ {{{\min }_{z \in \varOmega (y)}}({I^c}(z))} \right] = {\max _{y \in \psi (x)}}\left[ {I_{MIN}^c(z)} \right] , \end{aligned}$$
(4)

where I is the observed intensity, A is the local atmospheric, \(c\in \{r,g,b\}\) is the color channel index, and \(\psi \), \(\varOmega \) are local patches. In this paper, all results have been generated using patches \(\psi \) twice the size of \(\varOmega \).

Our fusion process approach is implemented in three main steps. First, based on the formula for estimating atmospheric light, we derive the two inputs of the fusion approach by using different sizes of the patches to estimate atmospheric light. As shown in Fig. 2(d), (e), for an original input image of size 500 \(\times \) 300, we take the patch size 10 \(\times \) 10 and 30 \(\times \) 30, respectively. We then apply the guided image filter [12] as a post-process, the atmospheric light after refined is shown in Fig. 2(f), (g). In the second step, we calculate the corresponding normalized weight maps according to the contrast and saturation of the two inputs as shown in Fig. 2(h), (i). Finally, the derived inputs and the normalized weight maps are blended by means of multi-resolution fusion algorithm using a Laplacian pyramid decomposition of the inputs and a Gaussian pyramid of the normalized weights. By using these maps, we can get the fused atmospheric light as shown in Fig. 2(j). Figure 3 compares the final results of using the Li’s method [9] and our proposed method, our result have a good visual effect without blocking artifacts at the sky region.

3.3 Transmission Estimation

As written in (2), once we estimate the Atmospheric light, the scene radiance depends on the transmission. In previous methods, transmission was estimated by using the Dark Channel Prior [3] even though they used different imaging models. As we all know, the Dark Channel Prior is a statistical result based on the observation on a great deal of daytime outdoor haze-free images. When it is applied to nighttime scenes, this assumption may fail and lead to inaccurate estimation of transmission.

Note that the entropy of haze-free image is bigger than that of hazy image at same scene, Park et al. [13] proposed a novel method to estimate transmission by maximizing the image entropy at non-overlapped sub-block regions without using any prior knowledge. The objective function he proposed is comprised of two functions. The first one is the image entropy f \(_{entropy}\) as a contrast measure, which has been proven can be used to characterize texture of an image [14, 15]. An image entropy can be expressed by the function of transmission t as follows:

$$\begin{aligned} {f_{entropy}}\left( t \right) = - \sum \limits _{i = 0}^{255} {\frac{{{h_i}\left( t \right) }}{N}} \log \frac{{h{}_i\left( t \right) }}{N}, \end{aligned}$$
(5)

where N is the number of pixels in the image, h \(_i\)(t) is the number of pixels that have intensity i in the gray-scaled image of scene radiance J calculated from (2), when the transmission is set to t. However, the de-hazed image may take values smaller than 0 or larger than 255 when transmission t is too small. Therefore, the second objective functions is designed to restrict excessive overflow and underflow as follows:

$$\begin{aligned} {f_{fidelity}}\left( t \right) = \mathop {\min }\limits _{c \in \{ r,g,b\} } {s^c}\left( t \right) , \end{aligned}$$
(6)
$$\begin{aligned} {s^c}\left( t \right) = \frac{1}{N}\sum \limits _{p = 1}^N {\delta \left( p \right) } ,\begin{array}{*{20}{c}} {}&{\delta \left( p \right) } \end{array} = \left\{ \begin{array}{l} 1,\begin{array}{*{20}{c}} {} \end{array}0 \le {J^c}\left( p \right) \le 255\\ 0,\begin{array}{*{20}{c}} {}&{}{otherwise} \end{array} \end{array} \right. \end{aligned}$$
(7)

where, s \(^c\)(t) expresses the ratio of pixels between 0 and 255 at each color channel of the de-hazed image J when the transmission is set to t. Note that as the number of overflow and underflow pixels becomes fewer, f \(_{fidelity}\)(t) becomes larger. The final objective function is defined as:

$$\begin{aligned} {f_{objective}}\left( t \right) = {f_{entropy}}\left( t \right) \cdot {f_{fidelity}}\left( t \right) , \end{aligned}$$
(8)

by maximizing the objective function in (8), the transmission t can be estimated from the daytime haze image. However, as shown in Fig. 4(b), maximizing this objective function of nighttime haze image may cause color distortion since the contrast has been over enhanced. In order to solve this problem, we introduce the third objective functions f \(_{hue}\)(t) to indicate the color distortion before and after haze removal. It can be defined as:

$$\begin{aligned} {f_{hue}}\left( t \right) = \frac{{{h_J}\left( t \right) - {h_R}\left( t \right) }}{N}, \end{aligned}$$
(9)
Fig. 4.
figure 4

Dehazed results using transmission estimated by different objective functions.

Full size image
$$\begin{aligned} h\left( t \right) = \arctan \left( {\frac{{{O_{2x}}\left( t \right) }}{{{O_{1x}}\left( t \right) }}} \right) , \end{aligned}$$
(10)
$$\begin{aligned} \left\{ \begin{array}{l} {O_{1x}}\left( t \right) = \frac{{{R_x}\left( t \right) - {G_x}\left( t \right) }}{{\sqrt{2} }}\\ {O_{2x}}\left( t \right) = \frac{{{R_x}\left( t \right) + {G_x}\left( t \right) - 2{B_x}\left( x \right) }}{{\sqrt{6} }} \end{array} \right. , \end{aligned}$$
(11)

where N is the number of pixels in the image, O \(_{1x }\)(t) and O \(_{2x }\)(t) are the two component of the opponent color space, h(t) is corresponding to the hue of an image, h \(_{\scriptscriptstyle J}\)(t) and h \(_{\scriptscriptstyle R}\)(t) are the hue of scene radiance and nighttime haze image, respectively, when the transmission is set to t. In theory, h \(_{\scriptscriptstyle J}\)(t) and h \(_{\scriptscriptstyle R}\)(t) are equal [16]. However, the existing haze removal algorithms are difficult to achieve the desired effect, resulting the value of h \(_{\scriptscriptstyle J}\)(t) and h \(_{\scriptscriptstyle R}\)(t) are usually biased. Therefore, the color distortion after haze removal can be measured by calculating the degree of variation of f \(_{hue}\)(t). Note that as the image has less color distortion after haze removal, f \(_{hue}\)(t) becomes smaller. In other words, we can reduce the color distortion effectively by minimizing the proposed color distortion measure. Consequently, the improved objective function we propose is defined as:

$$\begin{aligned} {f_{objective}}\left( t \right) = \frac{{{f_{entropy}}\left( t \right) \cdot {f_{fidelity}}\left( t \right) }}{{\max ({f_{hue}}\left( t \right) ,\begin{array}{*{20}{c}} {} \end{array}0.01)}}, \end{aligned}$$
(12)

the minimum f \(_{hue}\)(t) value is set by 0.01 to prevent division by zero. The objective function we proposed can well satisfy two quotas of contrast and color distortion. By maximizing the objective function in (12), we can estimate the transmission which provides good contrast and faithful dehazing results without distortions as shown in Fig. 4(c).

However, since the values of the transmission in outdoor hazy image is non-homogeneous and space-varying [17], local optimal transmission is estimated at each of the non-overlapped sub-block regions as follows:

$$\begin{aligned} t_k^{block} = \arg \begin{array}{*{20}{c}} {} \end{array}\mathop {\max }\limits _{t \in \left\{ {0.01 \le t \le 1} \right\} } {f_{objective}}\left( t \right) , \end{aligned}$$
(13)

where \(t_k^{block}\) is the k-th sub-block which is divided by pre-specified block size from a hazy image, the minimum t value is set by 0.01 to prevent division by zero. The size of the block has an important impact on the estimation of the t. If we choose a larger block, transmission may have different values in this region. Conversely, if the block we choose is too small, the estimated transmission may be inaccurate due to the insufficient number of pixels. We set a medium block size to \(15 \times 15\) in this paper. We find the optimal solution by exhaustive searching of t with 0.01 step size within the range of 0.01 to 1.0, which achieves high computational efficiency than traditional optimization algorithms.

Since the transmission from (13) is estimated by non-overlapped local region, halo effect and block artifact may occur. We chose the guided filter [12] to alleviate those artifacts. Figure 2(k) shows the refined transmission map.

Fig. 5.
figure 5

The comparison of the proposed method with conventional methods.

Full size image

4 Experimental Results

To demonstrate the effectiveness of the proposed algorithm, we evaluated our method compared our method with conventional methods [6, 7, 9]. Figure 5 shows the comparison of the proposed method with Pei’s work [6], Zhang’s work [7] and Li’s work [9]. As can be seen from Fig. 5, Pei’s method and Zhang’s method can remove haze effectively, but there are color distortion, and glow effects are still visible since their model did not account for glow effects. Although Li’s method removes the glow effects effectively, there are blocking artifacts at the sky area as well as areas with low brightness values look unnatural. The method we proposed can remove the glow effects effectively while provide better visual effect overall as shown in Fig. 5(e).

Fig. 6.
figure 6

The comparison of the proposed method with other conventional methods.

Full size image

We also compared performance of the proposed method with other conventional methods [3, 6, 9] whose results are shown in Fig. 6 in terms of image entropy [15], structural similarity index (SSIM) [18] and peak signal to noise ratio (PSNR) [19]. Image entropy is a measure of image contrast that can be used to characterize texture of an image. High entropy means that the image can present more detail and texture features. SSIM index can measure the structural similarity of the dehazed image to the original image. Table 1 shows the quantitative results in Fig. 6. From the statistical data in Table 1, we can see that the entropy and PSNR of our result improves obviously with the SSIM is very close to the others. The comparison results show that our result can provide more detailed texture features while maintaining high structural similarity.

Table 1. Quantitative measurements of results in Fig. 6.
Full size table

5 Conclusion

In this paper, we propose an efficient algorithm for nighttime dehazing. We exploit a new nighttime haze imaging model which takes into account both the non-uniform illumination from artificial light sources and the scattering and attenuation effects of haze. Based on the new model, we first decompose the glow image from the nighttime haze image. Then, we estimate two parameters, the atmospheric light and transmission. Finally, scene radiance is restored using the two estimated parameters. Experimental results on nighttime hazy images show that the proposed algorithm can successfully enhance image visibility while provides a good visual effect.