1 Introduction

The majority of the current challenges related to AI and deep learning for image restoration and enhancement [3, 4, 6, 28, 32, 35] are primarily targeting only one goal—high quantitative results measured by mean square error (MSE), peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), mean opinion score (MOS) and other similar metrics. As a result, the general recipe for achieving top results in these competitions is quite similar: more layers/filters, deeper architectures and longer training on dozens of GPUs. However, one question that might arise here is whether often marginal improvements in these scores are actually worth the tremendous computational complexity increase. Maybe it is possible to achieve very similar perceptual results by using much smaller and resource-efficient networks that can run on common portable hardware like smartphones or tablets. This question becomes of special interest due to the uprise of many machine learning and computer vision problems directly related to these devices, such as image classification [10, 31], image enhancement [13, 14], image super-resolution [8, 34], object tracking [11, 38], visual scene understanding [7, 21], face detection and recognition [20, 26], etc. A detailed description of the smartphones’ hardware acceleration resources that can be potentially used for deep learning and mobile machine learning frameworks are given in [15].

The PIRM 2018 challenge on perceptual image enhancement on smartphones is the first step towards benchmarking resource-efficient architectures for computer vision and deep learning problems targeted at high perceptual results and deployment on mobile devices. It considers two classical computer vision problems—image super-resolution and enhancement, and introduces specific target performance metrics that are taking into account both networks’ runtime, their quantitative and qualitative visual results. In the next sections we describe the challenge and the corresponding datasets, present and discuss the results and describe the proposed methods.

2 PIRM 2018 Challenge

The PIRM 2018 challenge on perceptual image enhancement on smartphones has the following phases:

  1. i

    development: the participants get access to the data;

  2. ii

    validation: the participants have the opportunity to validate their solutions on the server and compare the results on the validation leaderboard;

  3. iii

    test: the participants submit their final results, models, and factsheets.

Fig. 1.
figure 1

A low-res image (left) and the same image super-resolved by SRGAN (right).

Full size image

The PIRM 2018 challenge on perceptual image enhancement on smartphones consists of two different tracks described below.

Fig. 2.
figure 2

The original iPhone 3GS photo (left) and the same image enhanced by the DPED network [13] (right).

Full size image

2.1 Track A: Image Super-Resolution

The first track is targeting a conventional super-resolution problem, where the goal is to reconstruct the original image based on its bicubically downscaled version. To make the task more practical, we consider a downscaling factor of 4, some sample results for which obtained with SRGAN network [19] are shown in the Fig. 1. To train deep learning models, the participants used DIV2K dataset [1] with 800 diverse high-resolution train images crawled from the Internet.

2.2 Track B: Image Enhancement

The goal of the second track is to automatically improve the quality of photos captured with smartphones. In this task, we used DPED [13] dataset consisting of several thousands of images captured simultaneously with three smartphones and one high-end DSLR camera. Here we consider only a subtask of mapping photos from a very old iPhone 3GS device into the photos from Canon 70D DSLR. An example of the original and enhanced DPED test images are shown in the Fig. 2.

3 Scoring and Validation

The participants were required to submit their models as TensorFlow .pb files that were later run on the test images and validated based on three metrics:

  • Their speed on HD-resolution (1280 \(\times \) 720 pixels) images measured compared to the baseline SRCNN [8] network,

  • PSNR metric measuring their fidelity score,

  • MS-SSIM [37] metric measuring their perceptual score.

Though MS-SSIM scores are known to correlate better with human image quality perception than PSNR, they are still often not reflecting many aspects of real image quality. Therefore, during the final test phase we conducted a user study involving more than 2000 participants (using MTurk platformFootnote 1) that were asked to rate the visual results of all submitted solutions, and the resulting Mean Opinion Scores (MOS) then replaced MS-SSIM results. For Track B methods, the participants in the user study were invited to select one of four quality levels (probably worse, probably better, definitely better, excellent) for each method result in comparison with the original input image. The expressed preferences were averaged per each test image and then per each method to obtain the final MOS.

The final score of each submission was calculated as a weighted sum of the previous scores:

$$\begin{aligned} \begin{aligned} \mathbf {Total \, Score} = \alpha \,\cdot \, (\mathrm {{PSNR_{solution}} \, - \, {PSNR_{\tiny {baseline}}}}) \, + \\ \, \beta \, \cdot \, (\mathrm {{MS\text {-}SSIM_{solution}} \, - \, {\, MS\text {-}SSIM_{\tiny {baseline}}}}) \, + \\ \gamma \, \cdot \, \mathrm {min(4,\, {Time_{baseline}} \, / \, \, {Time_{solution}} )}. \end{aligned} \end{aligned}$$
(1)

To cover a broader range of possible targets, we have additionally introduced three validation tracks with different weight coefficients: the first one (score A) was favoring solutions with high quantitative results, the second one (score B)—with high perceptual results, and the third one (score C) was aimed at the best balance between the speed, visual and quantitative scores. Below are the exact coefficients for all tracks:

Image Super-Resolution:

  • \(\mathrm {PSNR_{\tiny {baseline}}}\) = 26.5, \(\mathrm {SSIM_{\tiny {baseline}}}\) = 0.94,

  • \((\alpha , \beta , \gamma )\): score A - (4, 100, 1), score B - (1, 400, 1), score C - (2, 200, 1.5).

Image Enhancement:

  • \(\mathrm {PSNR_{\tiny {baseline}}}\) = 21.0, \(\mathrm {SSIM_{\tiny {baseline}}}\) = 0.90,

  • \((\alpha , \beta , \gamma )\): score A - (4, 100, 2), score B - (1, 400, 2), score C - (2, 200, 2.9).

The implementation of the scoring scripts, pre-trained baseline models and submission requirements are also available in the challenge github repositoryFootnote 2.

4 Results

During the validation phase, we have obtained more than 100 submissions from more than 20 different teams. 12 teams entered in the final test phase and submitted their models, codes and factsheets; Tables 1 and 2 summarize their results.

Table 1. Track A (Image super-resolution), final challenge results.
Full size table

4.1 Image Super-Resolution

First of all, we would like to note that all submitted solutions demonstrated high efficiency: they were generally three to eight times faster than SRCNN, and at the same time were providing radically better visual and quantitative results. Another interesting aspect is that according to the results of the user study, its participants were not able to distinguish between the visual results produced by different solutions, and MOS scores in all cases except for the baseline SRCNN model were almost identical. The reason for this is that neither of the submitted models were trained with a strong adversarial loss component: they were mainly optimizing Euclidean, MS-SSIM and VGG-based losses. In this track, however, we still have two winners: the first one is the solution proposed by TEAM_ALEX that achieved the best scores in all three validation tracks, while the second winning solution from KAIST-VICLAB has demonstrated the best runtime on all platforms, including two Android smartphones (Razer Phone and Huawei P20) on which it was able to process HD-resolution images under 1 s.

4.2 Image Enhancement

Similarly to the previous task, all submissions here were able to significantly improve the runtime and PSNR scores of the baseline SRCNN [8, 13] and DPED [13] approaches. Regarding the perceptual quality, in this case there is no clear story, mainly high PSNR scores did not guarantee the best visual results, and vice versa. Also, MS-SSIM does not predict well the perceptual quality captured by MOS. The winner of this track is Mt.Phoenix team that achieved top MOS scores, as well as the best A, B and C scores and the fastest runtime on CPU and GPU. On smartphones, this solution required around 1.5 and 2 s for enhancing one HD-resolution photo on the Razer Phone and Huawei P20, respectively.

4.3 Discussion

The PIRM 2018 challenge on perceptual image enhancement on smartphones promotes the efficiency in terms of runtime and memory as a critical measure for successful deployment of solutions on real applications and mobile devices. For both considered tasks (super resolution and enhancement) a diversity of proposed solutions surpassed the provided baseline methods and demonstrated a greatly improved efficiency compared to many conventional techniques [15]. We conclude that the challenge through the proposed solutions define the state-of-the-art for image enhancement on smartphones.

Table 2. Track B (Image enhancement), final results. The results are sorted according to the MOS scores. CNN model from Rainbow team was using tf.image.adjust_contrast operation not yet available in TensorFlow Mobile and was not able to run on Android.
Full size table

5 Proposed Methods

This section describes solutions submitted by all teams participating in the final stage of the PIRM 2018 challenge on perceptual image enhancement on smartphones.

5.1 TEAM_ALEX

For track A, TEAM_ALEX proposed a residual neural network with 20 residual blocks [36], though all computations in this CNN were mainly done on the images downscaled by a factor of 4 with two desubpixel blocks; in the last two layers they were upscaled back to their original resolution with two subpixel modules. The main idea of desubpixel downsampling is shown on the Fig. 3—this is a reversible downsampling done via rearranging the spatial features into several channels to reduce spatial dimensions without losing information. The whole network was trained with a combination of MSE and VGG-based loses on patches of size 196 \(\times \) 196px (image super-resolution) and 100 \(\times \) 100px (image enhancement) for 2 \(\times \) 10\(^5\) and 2 \(\times \,\)10\(^6\) iterations, respectively. The authors used Adam optimizer with \(\beta \_1\) set to 0.9 and a batch size of 8; training data was additionally augmented with random flips and rotations. The learning rate was initialized at \(1e-4\) and halved when the network was 60% trained.

Fig. 3.
figure 3

Desubpixel block and the CNN architecture proposed by TEAM_ALEX.

Full size image

5.2 KAIST-VICLAB

In track A, KAIST-VICLAB proposed a similar approach of using 4\(\times \) image downscaling and residual learning, however their CNN (Fig. 4) consisted of only 8 convolutional layers. High visual and quantitative results were still obtained by using a slightly different training scheme: the authors applied a small amount of Gaussian blur to degrade the downscaled low-resolution training patches, while they improved construct and sharpness of the target high-resolution images. Furthermore, residual units, pixel shuffle [27], error feedback scheme [9] and xUnit [17] were integrated into network for faster learning and higher performance. The authors used 2,800 additional images from the BSDS300, Flickr500 and Flickr2K datasets for training, and augmented data with random flips and rotations. The network was trained for 2000 epochs on 128 \(\times \) 128px patches with L1 loss only; the batch size was set to 4, the learning rate was \(1e-4\).

Fig. 4.
figure 4

Solutions proposed by KAIST-VICLAB for tracks A (left) and B (right).

Full size image

For track B, KAIST-VICLAB presented an encode-decoder based architecture (Fig. 4), where spatial sizes are reduced with a space-to-batch technique: instead of using stride-2 convolutions, the feature maps obtained after each layer are divided into 4 smaller feature maps that are then concatenated along the batch dimension. The authors used an additional adversarial component, and for the discriminator they proposed relativistic RGAN [16] with twice as many parameters as in the generator. The network was trained similarly to track A, but with a combination of color and adversarial losses defined in [13].

5.3 Mt.Phoenix

For image super-resolution, the Mt.Phoenix authors used a deep residual CNN with two downsampling blocks performing image downscaling and two deconvolution blocks for its upscaling to the original size. Besides the standard residual blocks, additional skip connections between the input and middle layers were added to improve the performance of the network. CNN was trained on 500 \(\times \) 500px patches using Adam optimizer with an initial learning rate of \(5e-4\) and a decay of \(5e-5\). The network was trained with L1 loss, no data augmentation was used.

Fig. 5.
figure 5

U-net architecture for image enhancement proposed by Mt.Phoenix.

Full size image

In the second track, Mt.Phoenix proposed a U-net style architecture [25] (Fig. 5) and augmented it with global features calculated by applying average pooling to features from its bottleneck layer. Additionally, a global transform layer performing element-wise multiplication of the outputs from the second and last convolutional layers was proposed. The network was trained with a combination of L1, MS-SSIM, VGG, total variation and GAN losses using Adam optimizer with a constant learning rate of \(5e-4\).

5.4 CARN_CVL

For image super-resolution, CARN_CVL proposed the convolutional anchored regression network (CARN) [22] (see Fig. 6) which has the capability to efficiently trade-off between speed and accuracy. Inspired by A+ [33, 34] and ARN [2], CARN is formulated as a regression problem. The features are extracted from input raw images by convolutional layers. The regressors map features from low dimension to high dimension. Every regressor is uniquely associated with an anchor point so that by taking into account the similarity between the anchors and the extracted features, CARN can assemble the different regression results to form output features or the original image. In order to overcome the limitations of patch-based SR, all of the regressions and similarity comparisons between anchors and features are implemented by convolutional layers and encapsulated by a regression block. Furthermore, by stacking the regression block, the performance of the network increases steadily. CARN_CVL starts with the basic assumption of locally linear regression, derives the insights from it, and points out how to convert the architecture to convolutional layers in the proposed CARN.

Fig. 6.
figure 6

CARN architecture and CARN Regression Block presented by CARN_CVL.

Full size image

The challenge entry uses CARN with 5 regression blocks, 16 anchors/regressors per block, and a number of feature layers reduced to 2. In the two feature layers, the stride of the convolution operation is set to 2 because the bicubic interpolated image contains no high frequency information compared to the LR image but slows down the executation of the network. The number of inner channels is set as 8 for the upscaling factor 4.

5.5 EdS

EdS proposed a modification [30] of the original DPED ResNet architecture used for image enhancement (Fig. 7). The main difference in their network was the use of two 4 \(\times \) 4 convolutional layers with stride 2 for going into lower dimensional space, and additional skip connections for faster training. The network was trained for 33K iterations using the same losses and setup as in [13].

Fig. 7.
figure 7

A variation of the original DPED architecture proposed by EdS team.

Full size image

5.6 IV SR+

The authors proposed a Fast Clique Convolutional Network (FCCN), which architecture was inspired by CliuqueNet [39] and MobileNet [10]. The proposed FCCN consists of feature extraction, fast clique block (FCB) and two deconvolution layers (Fig. 8). For feature extraction, two convolutional layers with 32 and 20 kernels are utilized. Then, to accelerate the FCCN architecture, these features are fed to FCB layers for extracting more informative convolutional features. The FCB layer consists of one input convolutional layer and four bidirectional densely connected convolutional layers with both depthwise and pointwise convolution. The network was trained using Adam optimizer and a batch size of 16 for 3M iterations with an initial learning rate of \(1e-4\) halved after 2 million iterations.

Fig. 8.
figure 8

FCCN and the corresponding Fast Clique Block (FCB) proposed by IV SR+.

Full size image

5.7 BOE-SBG

The architecture of the network used for image super-resolution is presented in the Fig. 9 and is based on the Laplacian pyramid framework with a denseblock inspired by [18]. The parameters of denseblocks, strided and transposed convolutional layers are shared among different network levels to improve the performance. For image enhancement problem, the authors proposed a different architecture [23] (Fig. 9). First of all, it featured several Mux and Demux layers performing image up- and downscaling without information loss and that are basically a variant of (de)subpixel layers used in other approaches. This network was additionally trained with an extensive combination of various losses, including L1 loss for each image color channel, contextual, VGG, color, total variation and adversarial losses.

Fig. 9.
figure 9

Neural networks for image super-resolution (top), image enhancement (bottom) and the corresponding Denseblock (right) proposed by BOE-SBG team.

Full size image

5.8 Rainbow

The CNN architecture used in the first track is shown in the Fig. 10. The network consists of two convolutional layers with stride 2, three convolutional layers with stride 1, cascaded residual blocks and a subpixel layer. The network was trained to minimize L1 and SSIM losses on 384 \(\times \) 384px patches augmented with random flips and rotations. The learning rate was set to \(5e-4\) and decreased by a factor of 5 every 1000 epochs.

Fig. 10.
figure 10

CNN architectures proposed by Rainbow for tracks A (left) and B (right).

Full size image

A different approach [12] was used for image enhancement: the authors first trained a larger teacher generator and then used it to guide the training of the smaller student network (see Fig. 10). The latter was done by imposing additional knowledge distillation loss calculated as Euclidian distance between the corresponding normalized student’s and teacher’s feature maps. Besides this loss, the networks were trained with a combination of SSIM, VGG, L1, context, color and total variation losses using Adam optimizer with an initial learning rate of \(5e-4\) decreased by a factor 10 for every \(10^4\) iterations.

5.9 MENet

MENet team proposed a \(\theta \)-inception Network depicted in the Fig. 11 for image enhancement problem. This CNN has a \(\theta \)-inception block where the image is processed in parallel by convolutional and deconvolutional layers with strides 2 and 4 for multi-scale learning. Besides that, the size of the convolutional filters is different too: 3 and 5 in the first and the second case, respectively. At the end of this block, the corresponding two outputs are concatenated together with the output from the first convolutional layer and are passed to the last CNN layer. The network is trained using the same setup as in [13] with the following two differences: (1) two additional texture loss functions (local contrast normalization and gradient) are used and (2) after pre-training the network is additionally fine-tuned on the same dataset with Adam minimizer and a learning rate of \(1e-4\).

Fig. 11.
figure 11

\(\theta \)-inception Network (generator and discriminator) presented by MENet team.

Full size image

5.10 SuperSR

Figure 12 presents the CNN architecture used for image super-resolution problem. The network consists of one space-to-depth 4\(\times \) downsampling layer followed by convolutional and residual layers with PReLU activation functions and one deconvolutional layer for image upscaling. The model was trained on 192 \(\times \) 192px patches augmented with flips and rotations. Adam optimizer with a mini-batch size of 32 and a learning rate of \(1e-3\) decayed by 10 every 1000 epochs was used for CNN training. After the initial pre-training with L2 loss, the training process was restarted with the same settings, while the loss function was replaced by a mixture of Charbonnier [5] loss and MS-SSIM losses.

Fig. 12.
figure 12

Deep residual network proposed by SuperSR team.

Full size image

5.11 SNPR

For image enhancement, SNPR derives three network architectures corresponding to different operating points. The generator networks (G1, G2, and G3) corresponding to the three different approaches and the common discriminator network D are shown in Fig. 13. Conv(f, k, s) refers to a convolution layer with f \(k\times k\) filters performing convolution by a stride factor of s, ReLU is a Rectified Linear Unit, BN refers to batch-normalization, and Pixel-Shuffler X2 refers to the pixel shuffler layer [27] which increases resolution by a factor of 2. The first three layers are meant to extract the features that are relevant for image enhancement. Feature extraction at low-image-dimension has the advantages of larger receptive field and much lower computational complexity [29]. To compensate for detrimental effects of spatial dimension reduction in features, the input image (which have full-resolution spatial features) is concatenated with the features extracted at low-dimensional space and then combined by the succeeding convolutional layers. Overall G3 achieves the best speed-up-ratio but with a lower performance as compared to DPED baseline [13], whereas G1 achieves the lowest speed-up-ratio while having comparable quality to that of DPED.

Fig. 13.
figure 13

Neural networks proposed by SNPR (left) and Geometry (right) teams.

Full size image

5.12 Geometry

The overall structure of the network [24] presented by Geometry team is shown in the Fig. 13. Each convolutional layer has 16 filters, and the network itself produces two outputs: one based on the features from the middle CNN layer, and one from the last layer. The intermediate output (Output OC) is used to compute SSIM loss, while the final one (Output OE) is used to compute the loss function consisting of adversarial, smooth, and style losses. During the training all losses are summed, and the network is trained as a whole using Adam optimizer with a learning rate of \(5e-4\) decreased by a factor of 10 every 8000 iterations.