Generative Adversarial Network Applications in Industry 4.0: A Review

Abstract

The breakthrough brought by generative adversarial networks (GANs) in computer vision (CV) applications has gained a lot of attention in different fields due to their ability to capture the distribution of a dataset and generate high-quality similar images. From one side, this technology has been rapidly adopted as an alternative to traditional applications and introduced novel perspectives in data augmentation, domain transfer, image expansion, image restoration, image segmentation, and super-resolution. From another side, we found that due to the lack of industrial datasets and the limitation for acquiring and accurately annotating new images, GANs form an exciting solution to generate new industrial image datasets or to restore and augment existing ones. Therefore, we introduce a review of the latest trend in GANs applications and project them in industrial use cases. We conducted our experiments with synthetic images and analyzed most of GAN’s failures and image artifacts to provide training’s best practices.

Appendices

Appendix A: Experimentation Datasets

In this section, we describe our 4 in-house rendered synthetic datasets that we used in our experimentation.

Industrial Assets: This dataset consists of simple synthetic images ($512 \times 512$ px) for different industrial assets with domain randomization. We consider the following assets combinations: smart transport robot (STR) (16,000), trolly (16,000), STR and trolly (11,457), pallet (11,271), jack (9,425), electrical jack (8,944), stillage (8,933), forklift (8,840), tugger train (8,794), small load carrier (KLT) box (9,976), and random combinations of grouped assets (4,029). This dataset was rendered using Unity Engine (Fig. 30).

Fig. 30

Sample of the Industrial Assets dataset including STR, trolly, pallet, jack, electrical jack, stillage, forklift, tugger train, KLT, and random combinations of assets

All three remaining datasets are rendered using NVIDIA Omniverse.

KLT & Pallet: The dataset includes 4 combinations of small load carrier (KLT) box and pallet single images: (1) Low variation of KLT boxes (9,948) (2) Low variations of Pallet (10,893) (3) Higher variations of Pallet (5,000) (4) Higher variations of KLT box (2,500) (Fig. 31).

Fig. 31

Sample of the KLT & Pallet dataset. From top to bottom: low variation of KLT and Pallet, higher variation of Pallet and KLT

Stillage Modalities: The dataset consists of 2,006 synthetic images (1,280 $\times$ 720 px) for different stillages, sided next to each other. In addition, paired semantic and instance segmentations, and depth images are included (In total, 8,024 images) (Fig. 32).

Fig. 32

Sample of the Stillage Modalities dataset. From top to bottom: RGB plain color, semantic segmentation, instance segmentation, depth images

Klt2Cardboard: The dataset consists of around 20K synthetic images ($3{,}206 \times 1{,}440$ px) in total for randomly stacked boxes placed on a euro pallet in two different room environments. The first 10,138 consists of small load carrier (KLT) boxes surrounded by logistic assets, while the second 10,222 images consist of cardboard boxes surrounded by office assets. Both contain a maximum of 2-sided, 5-stacked boxes (Fig.33).

Fig. 33

Sample of the Klt2Cardboard dataset. Top: KLT boxes, bottom: cardboard boxes

Appendix B: Conditional GAN (cGAN)

It is the conditional version of GAN (Mirza & Osindero, 2014). In cGAN, conditional settings are applied to both the generator and the discriminator. The conditional settings can be any type of auxiliary information such as class labels (Karras et al., 2020a), instance images (Casanova et al., 2021), data pairing (Isola et al., 2017), etc. Alongside the latent space, the generator network inputs the class information condition and produces images. Fundamentally, a generator is free to generate whatever output as long as it satisfies the discriminator, which explains the necessity of applying conditionality on both GAN networks (Boulahbal et al., 2021). Additionally, a cGAN converges faster than a classical GAN, and its generated image random distribution follows a certain pattern.

As previously mentioned, a default simple GAN consists of 2 networks: a generative and a discriminator network. Although, training GAN’s networks does not differ from training any other network. Therefore, we refer to the universal taxonomy in Machine Learning (ML) to consider four major GAN learning methods based on the different ways of handling the available training datasets: supervised, unsupervised, semi-supervised, and few-shot as stated below in Appendix C.

Appendix C: GAN Training Taxonomy

Supverised learning (SL): It uses labeled datasets to train GANs (Cunningham et al., 2008). Due to the various GAN applications, we consider different shapes of labels. For instance, GAN’s conditional generation requires additional category labels for each training image as in Karras et al. (2020a), Karras et al. (2021), Casanova et al. (2021). Supervised I2I applications require paired image dataset to obtain accurate domain translations as in Wang et al. (2018a), Zhu et al. (2017b). However, despite the high efficiency of supervised learning over other learning methods, ensuring labeled data needs a certain level of expertise to structure that data, is intensively time-consuming, and has a likelihood of human error (IBM Cloud Education, 2020). In addition, it is sometimes impossible to acquire, especially when it comes to paired datasets, since it is “double the trouble” as capturing and processing a single domain dataset. In this case, synchronized hardware or an entire area control are additionally required.

Unsupverised learning (UL): Unlike SL, the model is not provided with labels or domain pairings. Therefore, the model works on its own to discover patterns and information (Chu et al., 2017; Park et al., 2020a). Yet, for optimal performance, unsupervised learning requires a more extensive dataset. In fact, unlabeled datasets are much more easily accessible. Therefore, researchers have focused more on enhancing unsupervised learning architecture over supervised learning methods (Chen & Jia, 2021).

Semi-supervised learning (SSL): It lies between supervised and unsupervised learning. Typically, a semi-supervised learning operates a dataset with few labels. Therefore, it aims to label all remaining images based on the small number of existing labeled images (e.g., it is possible to reconstruct the whole video by labeling 10–20% of its frames) (Mustafa & Mantiuk, 2020). Moreover, SSL is efficient for good disentanglement learning using only 0.25–2.5% of labeled data (Nie et al., 2020). However, semi-supervised learning generally has lower accuracy since all future predictions rely on predicted GT labels.

Few-shot learning (FSL): FSL models are based on a very few (Benaim & Wolf, 2018; Cohen & Wolf, 2019) to single image, i.e. one-shot learning (OSL), training dataset (Park et al., 2020a; Lin et al., 2020; Shaham et al., 2019). By definition, “FSL is a type of machine learning problems (specified by E, T and P), where E contains only a limited number of examples with supervised information for the target T.” Afterward, FSL training is evaluated based on its performance P (Wang et al., 2020b). Despite the fast FSL training generalization on a new small dataset, FSL models outperform in their field of experience. They may not be optimal when inferring slightly different use cases (check Sect. 6.3). Although, FSL is a promising field to relieve the burden of collecting huge datasets for the methods above, especially since limited hardware is enough for training. Thus, many researchers are focusing on fulfilling that aim.

Additionally, transfer learning (TL) is becoming the de facto solution for CV training (Jayram et al., 2019). The main idea is transferring knowledge from an auxiliary model into the main one. In other terms, an auxiliary model is a model that is trained on huge datasets to satisfy a source task. Afterward, the training “is resumed” with a smaller dataset to solve the interesting target task. Therefore, we divide TL methods into two categories: transductive and inductive. A transductive TL maintains the same task and labels as in the source task, e.g., Domain Adaptation (Guo et al., 2020; Li et al., 2020b; Cao et al., 2018; Murez et al., 2018). However, in inductive TL, the task, and therefore the labels, are changed and defined in the target task (e.g., sequential transfer learning, which is the most popular TL method) (Voita, 2022). Yet, some limitations may occur when training the model too long (check Sect. 4.1.2).

Appendix D: StyleGAN Retrospective

1.1 StyleGAN

In 2019, NVIDIA published StyleGAN (Karras et al., 2019; NVLabs, 2021a), an extension of the traditional ProGAN architecture (Karras et al., 2017). The generator network has been modified to include progressive resolution blocks, starting from $2^2\times 2^2$ to $2^{10}\times 2^{10}$ pixels. At each block, and after each convolution layer (Gatys et al., 2015b), a different sample of Gaussian noise is added to the feature map (Nielsen, 2019). Inspired by the style transfer literature (Huang & Belongie, 2017; Jing et al., 2019), an AdaIN layer (Huang & Belongie, 2017; Dumoulin et al., 2016, 2018; Ghiasi et al., 2017) controls the style transfer process. While the style only affects global effects, such as shape, identity, pose, lighting, and background, the noise injection at each block directly controls the image features and guarantees stochastic variations at different scales. This process separates the high-level attributes from the stochastic variation of local effects, such as beard, freckles, and hair. As a result, StyleGAN generates high-quality and high-resolution images (up to $1024\times 1024$ pixels) with detailed style-level stochastic variations. However, all images above 64$\times$64 resolution show water droplet artifacts in the feature map, often visible in the generated output image (Karras et al., 2020b). Additionally, the progressive growing technique used in all versions of StyleGAN produces phase artifacts, where some details are stuck to the same location regardless of slight movements of the parent object, as shown in Fig. 1.

1.2 StyleGAN2

StyleGAN2 (Karras et al., 2020b; NVLabs, 2021b), published in 2020, is a revised version of StyleGAN proposed to improve the image quality and remove all artifacts. First, Karras et al. replaced in the generator all AdaIN normalization (Huang & Belongie, 2017; Dumoulin et al., 2016, 2018; Ghiasi et al., 2017)—causing the droplet artifacts—with estimated statistics (Glorot & Bengio, 2010; He et al., 2015). Second, artifacts related to progressive growing (Karras et al., 2017) are reduced by using a modified version¹⁷ of a hierarchical (Denton et al., 2015; Zhang et al., 2017a, 2018a) generator: Multi-Scale Gradients for GAN (MSG-GAN) (Karnewar & Wang, 2020) with skip connections (Ronneberger et al., 2015). Skip connections are used to connect matching resolutions between both networks. In parallel, residual networks (Gulrajani et al., 2017; He et al., 2016; Miyato et al., 2018) have shown benefits in the discriminator. Both alternatives replace StyleGAN’s generator (synthesis network), and discriminator networks’ feedforward design (Huang et al., 2020) respectively. Compared to StyleGAN, the new revision improves the training performance¹⁸ by 40%, equivalent to 61 img/s.

However, despite the image quality improvements, tens of thousands of images with obvious variations are still required for the GAN training. Otherwise, it leads to discriminator overfitting and a training divergence (Karras et al., 2020a; Arjovsky & Bottou, 2017). Thus, acquiring this amount of varying dataset is sometimes unfeasible, as previously explained in Sect. 1.

Appendix E: Fine-Grained Image Generation

1.1 InfoGAN

Information Maximization GAN (InfoGAN) (Chen et al., 2016) is a GAN that utilizes unsupervised training to disentangle common visual concepts between small subsets of the latent variables, such as the presence of objects, lighting, object azimuth, pose, elevation, etc. By doing so, InfoGAN maximizes the mutual information between the latent variables and the generated images, thereby increasing the variation in the generated dataset.

1.2 FineGAN

FineGAN, proposed by Singh et al. (2019), is an architecture that disentangles the background, object shape, and object appearance hierarchically without the use of masks or fine-grained annotations. FineGAN iteratively executes in three stages: Background stage, parent stage, and child stage, where the object of interest’s features, such as appearance and shape (parent stage), are combined with the previously extracted background (background stage) and then colorized with a texture (child stage) to perform FineGAN generation.

${{{\textbf {Limitations:}}}}$ However, FineGAN does not support conditioning on real images and only supports sampling from latent codes. Therefore, before using FineGAN, additional work to extract the background, object pose, and appearance’s latent code is required to support image-conditioned generation (Li et al., 2020c).

1.3 MixNMatch

Li et al. developed MixNMatch (Li et al., 2020c; Yuheng-Li, 2020) which is built on top of FineGAN (Singh et al., 2019) and does not only allow sampled latent codes, but also real images to be used for image generation. Unlike previous fine-grained GAN architectures such as MUNIT (Huang et al., 2018), FusionGAN (Joo et al., 2018), and other disentangling techniques (Lee et al., 2018; Lorenz et al., 2019; Xiao et al., 2019), which focus on only two features: appearance and pose, MixNMatch simultaneously disentangles four factors: background, object pose, object shape, and object texture with minimal supervision. Unlike other approaches that require strong supervision annotations, such as key points, pose, masks, etc. Peng et al. (2017), Balakrishnan et al. (2018), Ma et al. (2018b), Esser et al. (2018), MixNMatch uses only bounding box annotations to model the backgrounds, as all training images have an object of interest. Once the background generator model is trained, no bounding boxes are needed for image generation.

The MixNMatch training process consists of two stages: (1) In the first stage, the “code mode,” MixNMatch takes up to 4 images and encodes them into four latent codes to generate realistic images with high accuracy. On the one hand, the shape’s latent code space capacity is too small to handle unique 3D shape variations such as boxes, STRs, trolleys, etc. On the other hand, the small capacity of the shape code limits the generation of pixel-level shape and pose details, which is handled in the second stage. (2) In the second stage, the “feature mode,” MixnMatch maps the image to a higher-dimensional feature space to preserve the shape and pose spatially-aligned details. Then, the FineGAN 3-stages pipeline is executed for conditional MixNMatch generation. In Fig. 15, we can see the combination of the KLT box texture, ground texture, and ground color features in the generated output.

Appendix F: GAN SOTA Datasets

In this section, we review the mentioned datasets in Tables 1, 2, and 3 with brief description and their downloadable links (Tables 6, 7, 8, 9).

Table 6 Public datasets used in GAN SOTA for Table 1

Table 7 Public datasets used in GAN SOTA for Table 2

Table 8 Public datasets used in GAN SOTA for Table 3 - supervised + cross-view translations

Table 9 Public datasets used in GAN SOTA for Table 3 - unsupervised + semi-supervised

Appendix G: Style Transfer Motivation

Style transfer (Gatys et al., 2015b; Jing et al., 2019) is one of the first and most known I2I translations to automate pastiche (Dumoulin et al., 2016). When talking about style transfer, we consider three types of images: two inputs and one output (Chen & Jia, 2021; Dumoulin et al., 2016):

• 1. Content image: Style transfer preserves the high-level semantic features of the content input, which are noted as invariant features, e.g., contrast, brightness, shape, etc.

• 2. Style image: Style transfer extracts style features from the second input image such as texture, contrast (Ulyanov et al., 2016b), color, etc.

• 3. Generated stylized image: Style transfer combines extracted style and content features into one single output.

Applying GANs as a backbone for I2I models is the most effective strategy (Chen & Jia, 2021). This section presents various techniques for GAN-based domain transfer and adaptive industrial applications that are essential and common to different fields. In I2I translation applications (Pang et al., 2021) e.g. semantic image synthesis (Park et al., 2019; Zhu et al., 2020; Tang et al., 2020), image segmentation (Guo et al., 2020; Li et al., 2020b), style transfer (Yi et al., 2017; Alami Mejjati et al., 2018), image inpainting (Pathak et al., 2016; Song et al., 2018; Zhao et al., 2020a), image deblurring (Zhang et al., 2020a, 2019a; Kupyn et al., 2019), image denoising (Kim et al., 2019a; Tian et al., 2020; Tran et al., 2020), image colorization (Zhang et al., 2017c; Suárez et al., 2017; He et al., 2018), super-resolution (Ledig et al., 2017; Zhang et al., 2019c; Yuan et al., 2018), domain adaptation (Cao et al., 2018; Murez et al., 2018), etc. the generative model aims to generate images that look like the target domain distribution (Chen & Jia, 2021; Pang et al., 2021). We executed arbitrary style transfer methods from Nakano (2018) to apply styles of different patterns, drawings, and modalities and the original image itself on an image of stillages. As a result, the style is hardly applied all over the input image: it overrides original colors and paints the whole image with a single to minimal amount of textures, as shown in Fig. 34 below.

Fig. 34

Style transfer experiments based on Nakano (2018): a Style image b Stylized output

Table 10 GAN-based Super Resolution Brief

Researchers focused on increasing the number of supported styles per a single network, combining arbitrary styles with interpolations (Dumoulin et al., 2016; Ghiasi et al., 2017; Nakano, 2018), conserving original images’ colors and luminance beyond the style and textures (Gatys et al., 2016a), replace the per-pixel loss function into a perceptual loss depending on extracted high-level features (Johnson et al., 2016), maintain photo-realistic style transfer to preserve the generated image realism similarly to the content image (Li et al., 2019; Park et al., 2020b), apply instance image style transfer (Castillo et al., 2017) because of the natural image complexity which contains a variety of distinct textures (Luan et al., 2017), etc. Yet, despite its successful contribution to artistic and painter’s style transfer (Pang et al., 2021), current approaches are inefficient for stylizing into industrial image modalities without any information loss. For instance, when applying a depth image as a style, the generated output consists of a sharpened grayscale image with depth information absence. Additionally, a segmentation-based style transfers the segmentation’s colors into the whole content image.

Appendix H: SR Briefing

In Table 10, we compared the latest different GAN-base SR technics. Recently, the latest Real-ESRGAN+ has shown superiority over SATO’s SR.

Appendix I: Additional Results

In this section, we present additional experimentation and extended results for some of the GAN architectures mentioned in the paper, such as: conditional StyleGAN3, IC-GAN, ccIC-GAN, Instance2Color, Color2Depth, image de-filtering, and image expansion in Figs. 35, 36, 37, 38, 39, 40, and 41 respectively.

Fig. 35

StyleGAN3 conditional image generation for 8 logistic assets (from top to bottom): load carrier box aka. KLT Box, Trolly, Forklift, Jack, Pallet, Stillage, STR and Electrical Jack

Fig. 36

IC-GAN model trained from scratch: a Conditional instance b Generated output

Fig. 37

ccIC-GAN model trained from scratch: a Conditional input b Output

Fig. 38

Comparing domain transfer from instance segmentation image into a plain color image using Pix2PixHD and CUT (200 epochs and 1500 paired images: 3000 total images)

Fig. 39

Comparing domain transfer from plain color image into a depth image using Pix2PixHD and CUT (200 epochs and 1500 paired images: 3000 total images)

Fig. 40

Additional results for image deblurring, saturating, dehazing and denoising

Fig. 41

Image expansion using Boundless