Efficient Retinex-Based Low-Light Image Enhancement Through Adaptive Reflectance Estimation and LIPS Postprocessing

Abstract

In this paper, a novel Retinex-based low-light image enhancement method is proposed, in which it has two parts: reflectance component estimation and logarithmic image processing subtraction (LIPS) enhancement. The enhancement processing is performed in the V channel of the color HSV space. First, adaptive parameter bilateral filters are used to get more accurate illumination layer data, instead of Gaussian filter. Moreover, the weighting estimation method is used to calculate the adaptive parameter to adjust the removal of the illumination and obtain the reflectance by just-noticeable-distortion (JND) factor. In this way, it can effectively prevent the over-enhancement in high-brightness regions. Then, the logarithmic image processing subtraction (LIPS) method based on maximum standard deviation of the histogram is applied to enhance reflectance component part, where the interval of the parameter is according to the cumulative distribution function (CDF). Experimental results demonstrate that the proposed method outperforms other competitive methods in terms of subjective and objective assessment.

The work was supported by the National Nature Science Foundation P.R. China No. 61471201.

1 Introduction

Image enhancement is highly required for various application, such as video surveillance, medical image processing. However, images captured in low-light conditions often have low dynamic range and seriously degrade by noise. In this case, in order to obtain images with good contrast and details, various low-light image enhancement techniques are needed. In recent years, low-light image enhancement approaches have received stacks of studies, commonly used methods including histogram equalization [1], Retinex-based methods [2], dehaze-based methods [3], and logarithmic image processing (LIP) models [4]. In these approaches, the Retinex theory is first proposed by Land [5] to model the image process of human visual system. This theory assumes that the scene in human’s eyes is the product of reflectance and illumination. In recent years, many Retinex-based image enhancement algorithms have been proposed as follows: Single Scale Retinex (SSR) [6], Multi-scale Retinex (MSR) [7], and MSR with color restoration (MSRCR) [8]. In few years later, Kimmel et al. [9] proposed an image enhancement method based on variational Retinex. The effects have greatly improved over the previous methods based on Retinex. He converted the previously estimated approximate illumination problem into a quadratic programming optimal solution problem, calculated the illumination through the gradient descent method, and enhanced the observation image with gamma correction. Ng et al. [10] used the idea of total variation to describe the nature of reflectance under the variational framework, and brought the reflection into the solution model to obtain an ideal reflectance image. Fu et al. [11] proposed a weighted variational model considering both illumination and reflection on the Kimmel and Ng’s methods, in which the resulting reflection images can retain high-frequency details. What’s more, the methods based on logarithmic image processing (LIP) models have been widely used in recent years. Jourlin et al. [12] developed a logarithmic image processing (LIP) model which is a mathematical framework based on abstract linear mathematics. The LIP models contains several specific algebraic and functional operations which can be used to manipulate image intensity values in a bounded range. HDR (high dynamic range) image generation is one of most important low light image enhancement method [13]. At present, the acquisition of high dynamic range image is mainly based on the software method. The most widely used software method is to obtain HDR image by multi-exposure, which is the mainstream HDR imaging technology. Multi-exposure HDR imaging technology includes two categories: one is the method based on the inverse camera response function recovery; the other is direct fusion method. The resulting HDR images exhibit fewer artifacts and encode a wider dynamic range, but the former one often cause color shift when used in RGB color space; the fusion method only can slightly expand the dynamic range, it’s not enough for generating authentic image.

In this paper, we developed a weighted just-noticeable-distortion (JND) based MSR to adjust reflectance and utilize the logarithmic image processing model to enhance the contrast. Compare with existing techniques, the proposed method can’t expand the dynamic range well than HDR methods, however, it can effectively prevent over enhancement in bright regions and adaptively select appropriate enhanced result. The main contribution of this work are as follows is include two part: Firstly, we obtain the illumination layer using adaptive bilateral filter instead of Gaussian filter. Secondly, we calculate the JND-based factor that is adjusted by adding a weighted factor based on illumination intensity to remove the illumination. Finally, we set the interval of the parameter according to the cumulative distribution function (CDF) of the reflectance and then we apply the logarithmic image processing subtraction (LIPS) based on maximum standard deviation of histogram to it. Experimental results demonstrate that the proposed method can effectively preserve the details in bright regions.

2 Related Works

2.1 Retinex Based Low Light Image Enhancement

The multi-scale Retinex (MSR) algorithm was raised by Jobson et al. [7]. This algorithm was developed to attain lightness and color constancy for machine vision. It is based on single scale Retinex (SSR) and could balance the dynamic compression and color constancy. The single scale Retinex is given by:

$$ R_{i} \left( {x,y} \right) = logI_{i} \left( {x,y} \right) - { \log }\left[ {F\left( {x,y} \right) *I_{i} \left( {x,y} \right)} \right] $$

(1)

where $R_{i} \left( {x,y} \right)$ is the Retinex output, $I_{i} \left( {x,y} \right)$ is the image distribution in the $i$th spectral band, “*” denotes the convolution operation, and $F\left( {x,y} \right)$ is a Gaussian kennel. The MSR output is simply a weighted sum of several different SSR outputs, and MSR is produced as follows:

$$ R_{MSRi} = \sum\nolimits_{n = 1}^{N} {w_{n} R_{ni} } $$

(2)

where $N$ is the number of scales, $R_{ni}$ is the $i$th component of the MSR output, $w_{n}$ is a collection of weights. In general, the weights $w_{n}$ is a chosen to be equal.

2.2 Logarithmic Image Processing (LIP) Models

LIP model generally makes use of the logarithm, as transmitted images combine by logarithmic laws and the human visual system processes light logarithmically. The LIP model has been shown to satisfy Weber’s Law and the saturation characteristics of the human visual system. From a physical point of the view, this LIP model is physically justified in a number of aspects. For example, the addition operation is consistent with the transmittance image formation model and the saturation characteristic of the human’s eye, the contrast definition based on subtraction is consistent with Weber’s law, and the zero gray-tone function corresponds to the highest intensity of an image, and the gray-tone function is the inverted images of the original images. The relative formula for the model [14] is:

$$ f_{1} \oplus f_{2} = f_{1} + f_{2} - \frac{{f_{1} f_{2} }}{M} $$

(3)

$$ f_{1} { \ominus }f_{2} = M\frac{{f_{1} - f_{2} }}{{M - f_{2} }} $$

(4)

where $\oplus \left( { \ominus } \right)$ is LIP addition(subtraction), $f_{1} = 255 - f_{1}^{'}$, $f_{2} = 255 - f_{2}^{'}$, namely, $f_{1}$, $f_{2}$ are inverted images of the initial images $f_{1}^{{\prime }}$, $f_{2}^{{\prime }}$, and M is 256 by default. If we set $f_{2}$ as a constant $C$, the image $f_{1}$ will be darker or brighter when we use LIP addition or subtraction.

The LIP model has been adopted for various applications such as medical image enhancement [15] and edge detection [16]. In [15], an un-sharp masking framework for medical image enhancement is proposed, which combines a generalized un-sharp masking algorithm with operations of LIP and get the good effects.

3 Proposed Method

We propose a new approach to enhance the low-light images to prevent over enhancement in highlight region. The main contribution of proposed method is include two part: the first stage is to obtain the reflectance layer by weighted just-noticeable-distortion (JND) based MSR, In this part, a weighting factor is used to control the removal of the illuminance, in order to prevent the gloomy phenomenon in bright regions, we set a fixed range by normalized background brightness. The second part is applying adaptive logarithmic image processing subtraction (LIPS) on reflectance layer to enhance the contrast. The parameters is selected by maximum standard deviation of the enhanced images, in order to obtain best enhanced images, we adaptive fix the interval of the parameter by cumulative density function of the reflectance component in first part. As shown in Fig. 1, the framework of the proposed algorithm is presented.

Fig. 1.

Framework of the proposed algorithm: the first part is JND-based MSR process and the second part is LIPS-based contrast enhancement

3.1 Weighted JND-Based MSR

Unlike classical MSR methods, we perform the proposed methods in V channel of the color space HSV without considering color adjustment. As we all know, the most influential effect on the low light image is the luminance component. If we perform similar processing on the color channels, it will lead to color distortion. For the sake of preserving appropriate illumination and compressing the dynamic range of the image, we set a control factor to adaptively remove the illumination. According to the theory of MSR, we obtain the reflectance R as follows:

$$ r\left( {x,y} \right) = \sum\nolimits_{n = 1}^{N} {w_{n} } \cdot \left\{ {\lg \left[ {V\left( {x,y} \right)} \right] - \beta \cdot lg\left[ {L\left( {x,y} \right)} \right]} \right\} $$

(5)

where $r$ is reflectance intensity after illumination adaptation, V is the channel of HSV color space, and $\beta$ is the control factor based on JND thresholds.

Most classical MSR-based enhancement algorithms performs the convolution between Gaussian smoothing function and the original image to get the illumination layer and the halo artifacts and details loss appear frequently. In [17], adaptive filter was used to prevent halo artifacts by adapting the shape of filter to the high-contrast edges. In [17, 18], it used a canny edge detector to detect high-contrast edges. Then the factor $\upsigma$ of Gaussian smoothing function is defined as follows:

$$ \sigma = \left\{ {\begin{array}{*{20}l} {\sigma_{1} } \hfill & {a\;high\;contrast\;edge\;was\;acrossed} \hfill \\ {\sigma_{0} } \hfill & {no\;high\;contrast\;edge\;was\;acrossed} \hfill \\ \end{array} } \right. $$

(6)

The bilateral filter is proposed in [19], and it has been proved to be good at edge-preserving. The bilateral filter performs better near the edge than the Gaussian filter by adding a coefficient defined by intensity value. In order to estimate approximate illumination accurately and prevent halo artifacts efficiency, we use an adaptive bilateral filtering instead of Gaussian filtering.

$$ L_{n} \left( {x,y} \right) = \frac{{\mathop \sum \nolimits_{x,y} V\left( {x,y} \right)W_{n} \left( {i,j,x,y} \right)}}{{\mathop \sum \nolimits_{x,y} W_{n} \left( {k,l,x,y} \right)}} $$

(7)

$$ W_{n} \left( {i,j,x,y} \right) = e^{{ - \frac{{\left( {i - x} \right)^{2} + \left( {j - y} \right)^{2} }}{{2\sigma _{d}^{2} }} - \frac{{\left\| {V\left( {i,j} \right) - V\left( {x,y} \right)^{2} } \right\|}}{{2\sigma _{r}^{2} }}}} $$

(8)

where $W_{n} \left( {i,j,x,y} \right)$ measures the geometric closeness between the neighborhood center $\left( {x,y} \right)$ and a nearby point $\left( {i,j} \right)$. If the difference between the two pixels is more than the threshold value, we set the range domain factor $\sigma_{r}$ to $\sigma_{1}$; otherwise, we use $\sigma_{0}$. We set $\sigma_{1}$ to $0.6\sigma_{0}$ in this paper. Figure 2. has shown the differences between the Gaussian smoothing and adaptive bilateral smoothing. In Fig. 2(b), it causes halo artifacts along the strong edges, however, as shown in Fig. 2(c), the edges is preserved and artifacts is deduced.

Fig. 2.

Visual results for Gaussian filtering and adaptive bilateral filtering. (a) Initial image. (b) Mask with Gaussian smoothing. (c) Mask with adaptive bilateral smoothing.

According to the theory of MSR, we could compress the dynamic range by Eq. (5). The smaller $\beta$ is, the more similar to the original images and the smaller dynamic range compression is. Also, the greater $\beta$ is, the more obvious the details are and the more dynamic range compression is.

In [20], Barten et al. discovered the relationship between the actual luminance and the brightness by human eye perception through experiments. Then, Jayant et al. [21] addressed a key concept of perceptual coding called just-noticeable-distortion (JND). Namely, if the difference between two luminance values in an image is below the JND value, the difference is imperceptible. In [19], it proposed the luminance adaptation Retinex-based contrast enhancement algorithms, which adopted the JND into the luminance adaptation and got the good effects.

According to the previous work [21], the relationship between the visibility and background luminance is obtained as follows:

$$ T_{l} \left( {x,y} \right) = \left\{ {\begin{array}{*{20}l} {17(1 - \sqrt {\frac{{L\left( {x,y} \right)}}{127})} + 3} \hfill & {L\left( {x,y} \right) \le 127 } \hfill \\ {\frac{3}{128}\left( {L\left( {x,y} \right) - 127} \right) + 3} \hfill & {otherwise} \hfill \\ \end{array} } \right. $$

(9)

where $L\left( {x,y} \right)$ is background luminance of the input low-light image, in this paper, $L\left( {x,y} \right)$ is the mean of $L_{n} \left( {x,y} \right)$, (n = 1, 2, 3); and ${\text{T}}_{l}$ is the visibility threshold, namely JND value. The visibility thresholds are high in dark region while low in bright regions, thus, human’s eyes are more sensitive to bright region than dark region, We proposed that human’s eyes sensitively to background luminance is contrary to visibility threshold as follows:

$$ w_{1} = 1 - \frac{{T_{l} \left( {x,y} \right) - min\left( {T_{l} \left( {x,y} \right)} \right)}}{{max \left( {T_{l} \left( {x,y} \right)} \right) - min\left( {T_{l} \left( {x,y} \right)} \right)}} $$

(10)

Then we add a weighted factor based on illumination intensity to control $w_{1}$.

$$ w_{2} = k \cdot e^{{\frac{{l^{2} (x,y)}}{{\sigma_{b}^{2} }}}} $$

$$ \beta = w_{1} \cdot w_{2} $$

(12)

where $w_{2}$ is an adaptive factor to control the value of the $w_{1}$, $l\left( {x,y} \right)$ is the normalized background luminance, $\sigma_{b}$ is the constant to control the $w_{2}$ and ${\text{k}}$ is to control the maximum value.

3.2 LIPS-Based Contrast Enhancement

In this paper, we utilize efficient image enhancement method to enhance its contrast. The algorithm is based on [14], in which an enhanced image is modeled as:

$$ E^{\prime} = R{ \ominus }C = \frac{{R^{\prime} - C}}{{1 - \frac{C}{M}}} $$

(13)

where $C$ is a constant, $M$ is constant as 256, $R^{\prime}$ and ${\text{E}}^{\prime }$ are inverted image by reflectance $R$ and enhanced image $E$, $R^{\prime} = 255 - R$, $E^{\prime} = 255 - E$.

LIPS can enhance low light image effectively as we found the best parameter $C$, but it always causes over-enhancement if we choose a large one. Because in LIP subtraction models, there are some bright points existing in the initial low-light images. The bright region of $R$ is expanded by subtraction of a constant $C \in \left[ {0,M} \right]$, which can generate negative values by $R{ \ominus }C$. In order to prevent the negative effects, we adaptively choose the constant number $C$.

Firstly, as the dynamic range of the low-light image is compressed, the probability density function (PDF) of reflectance image $R$ can be used to adaptively choose the proper threshold value $C$. The PDF can be approximated by

$$ {\text{PDF}}\left( l \right) = \frac{{n_{l} }}{M \cdot N} $$

(14)

where $n_{l}$ is the number of pixels that have intensity ${\text{l}}$ and ${\text{M}} \cdot {\text{N}}$ is the total number of pixels in the image. According to the PDF, we can calculate the cumulative distribution (CDF). The equation is formulated as:

$$ CDF\left( l \right) = \sum\nolimits_{k = 0}^{l} {PDF\left( k \right)} $$

(15)

Then, we set an error ${\text{e}}$. We do not take into account the pixels of CDF greater than $1 - {\text{e}}$, and we use the maximum pixel value that CDF value is equal to or approximately $1 - e$ as the threshold $T$, and the interval of parameter $C$ is in $\left[ {0,255 - T} \right]$.

Finally, we adaptively selected the best parameter $C_{op}$ in the interval by maximization of standard deviation of the histogram of $R{ \ominus }C$ [16]. The specific steps is as follows:

• (1) We compute the logarithmic subtraction on reflectance components by $R{ \ominus }C$.

• (2) Create the histogram ${\text{h}}\left( {R{ \ominus }C} \right)$.

• (3) Compute the standard deviation $\upsigma\left[ {{\text{h}}\left( {R{ \ominus }C} \right)} \right].$

• (4) Compute the best parameter C such that:

$$ \upsigma\left[ {{\text{h}}\left( {R{ \ominus }C_{op} } \right)} \right] = Max_{{C \in \left[ {0,255 - T} \right]}} \left\{ {\upsigma\left[ {{\text{h}}\left( {R{ \ominus }C_{op} } \right)} \right]} \right\} $$

(16)

The proposed method is briefly described in Algorithm 1.

4 Experimental Results

This section shows the qualitative comparison results of our method with five state-of-the-art methods, including classical multi-scale Retinex (MSR) [7], low-light image enhancement via Illumination Map Estimation (LIME) [22], joint intrinsic-extrinsic prior model for Retinex (JIEPMR) [23], and Retinex-based perceptual contrast enhancement (RPCE) [18], along with our proposed method.

All the methods were tested on 45 images with different degree of darkness. All the 45 low-light images and the enhanced results by the proposed method are briefly shown in Figs. 3 and 4 and the test images and results are compressed into 200 * 200 displays. Due to the space limitation, we just present four representative low-light images, as shown in Figs. 5, 6, 7 and 8.

Fig. 3.

45 tested images with various degree of darkness, most of image with high contrast

Fig. 4.

Enhanced results of 45 low light image

Fig. 5.

Enhanced results for block (a) Original image (b) MSR [7] (c) LIME [22] (d) JIEPMR [23] (e) RPCE [18] (f) Ours

Fig. 6.

Enhanced results for block (a) Original image (b) MSR [7] (c) LIME [22] (d) JIEPMR [23] (e) RPCE [18] (f) Ours

Fig. 7.

Enhanced results for block (a) Original image (b) MSR [7] (c) LIME [22] (d) JIEPMR [23] (e) RPCE [18] (f) Ours

Fig. 8.

Enhanced results for block (a) Original image (b) MSR [7] (c) LIME [22] (d) JIEPMR [23] (e) RPCE [18] (f) Ours

4.1 Subjective Assessment

Compared with these state-of-art methods, as show in Figs. 5, 6, 7 and 8, the proposed methods can adaptively control the contrast enhancement degree for different areas to prevent over-enhancement on high contrast image. Figures 5, 6, 7 and 8 show their zoomed results in highlight regions, As shown in Figs. 5(a)–(d), the region on the sky is over-enhancement, in Fig. 5(e), the clouds on the sky is close to the original image, but the wall of the house becomes darker. As shown in Fig. 5(f), our proposed method is outperform both in two regions. Compared with other four methods, details and edges in their zoomed result are enhanced best in our proposed method in Figs. 5, 6, 7 and 8.

4.2 Objective Assessment

Objective assessment is always used to explain some important characteristics of an image. According to the [24], a blind image quality assessment called natural image quality evaluator (NIQE) is used to evaluate the enhanced results. The lower NIQE value represents the higher image quality. As shown is Table 1, it demonstrates the average NIQE of all the 45 images enhanced by the mentioned five methods. The results obviously shown our method has a lower value compared with other methods.

Table 1. Quantitation performance comparison on 45 images with NIQE

5 Conclusion

In this paper, an effective Retinex-based low-light image enhancement method was presented. By utilizing JND-based illumination adaptation, the over-enhancement in bright areas and the loss of details and textures are all eliminated. Additionally, we add the adaptive LIPS based on maximum standard deviation of histogram to the reflectance images, which can effective preserve the details in highlight regions. Experimental results show that the proposed algorithm can achieve better image quality and succeed in keeping textures in highlight regions. In future work, we will study an effective low light video enhancement methods, and improve the performance of the algorithm.