1 Introduction

How to restore a painting when physical and chemical alteration of the painted layer has not yet reached a final stage and is partly caused by unknown ageing processes? Can we make a painting look restored without any physical intervention on it? We encountered this problem in the course of conservation treatment of paintings of the famous Serbian Informalist painter Lazar Vozarević (Merenik 2010; Trifunovic 1998; Stanojević 2013). In the period of 2015–2017, the “Lazar Vozarević” Gallery, based in Sremska Mitrovica, carried out a large project of conservation and restoration of Vozarević’s paintings from different periods. The conservation and restoration treatment was successfully carried out on all paintings but one, a painting from his Informal period, entitled “Untitled” (1961) (Fig. 1).

Fig. 1.
figure 1

Lazar Vozarević “Untitled” 1961, Gallery “Lazar Vozarević”, Sremska Mitrovica

Full size image

An integral part of the paint layer consists of two copper plates that were tinted with dark thin coating to fit the tonality of the painting. The plates were firmly attached (using nails) but in some places a bit disconnected from the surface to which they are bound. The material that was used for the “application” of copper plates in the surface layer of the painting, a kind of filler, was cracked and unstable. The color of the paint layer was degraded, discolored and missing in some places.

The building where the painting was located within, the Lazar Vozarević Gallery, was built in the first half of the twentieth century. It was poorly maintained and showed visible signs of damage to the structure and traces of humidity influx. The building has never had acclimatization; it has large window surfaces and lacks insulation. Typically, this type of exhibition spaces are exposed to sudden, significant changes in temperature and relative humidity, which lead to mechanical damage to the paintings such as cracking. In addition, during the reconstruction of the Gallery, the collection was stored in inadequate conditions; paintings were kept in piles and exposed to daily rise in temperature due to the presence of two windows on the eastern side of the storage.

However, the existing damage to the surface of the painting is evidently not only the consequence of effects of microclimate conditions, but also of the techniques and materials used. Analysis of the combination of adhesives, pigments and dyes used on the painting, confirmed the existence of changeable pigments, in particular lithol red, which is not only highly light-sensitive but also chemically unstable and sensitive to heat. This made the painting additionally susceptible to environmental conditions and mechanical damage, at the level of the paint layer (Standeven 2008). These underlying causes and the extent of the deterioration would have demanded exhaustive treatment, the use of toxic compounds (Artigas 2009) and a near-complete restoration with similar materials and techniques. This was considered invasive and hazardous. The alternative was to restore the pristine state of the painting virtually, concealing the damage of the painted layer without physical intervention on the painting itself (Stenger et al. 2016; Janke and Riedl 2010).

The question posed was how can we make the painting look restored? At present, there is a multitude of tools and specialist knowledge, to create a near-identical digital copy of the original painting and to perform a virtual restoration. To achieve this, the painting could either be 3D scanned or acquired by photogrammetric technique, obtaining a highly precise 3D mesh with realistic texture. Lacunas and damage to the paint layer could be remodeled in the generated model’s geometry, with colors corrected on the texture. The result would be a realistic digital counterpart of original painting’s appearance. The created model can be presented in a tailor-made software application or in a web browser, allowing it to be observed and valuated. Most importantly, the painters’ original idea and painting’s artistic significance can be preserved and conveyed. This virtually restored replica cannot be exhibited within the museum as a single undivided work of art, as the original and its digital representation will always be separate items in confrontation with each other, whether the digital representation is presented in graphical (printed) form or as an on-screen video. One could argue that this is nonetheless a legitimate solution. Its greatest benefit is that it would not disrupt or compromise the paintings integrity and authenticity. But, the question was raised, if we could go one step further, can we improve the painting’s appearance by using its digitally restored replica? If we were to create an augmented reality application for mobile devices, we could present digitally restored version of the painting to the visitors. After observing the original painting, one could download the AR application and through mobile device observe the painting restored. This solution demands that spectators have powerful devices and that the light conditions in the exhibition room can be precisely measured and reproduced in the AR application, for the best quality experience. Other issues with this solution are the targeting of the painting, as it is a two-dimensional object. The technology needs to be user friendly enough for regular museum visitors to use it without the need for additional training.

2 Resources and Comparison

The initial idea was further developed by studying the conservation work performed on Mark Rothko’s Harvard Murals, where the painting’s appearance was successfully altered by means of a projected compensation image as an integral part of the restoration process. Rothko’s murals had suffered significant color fading and alterations that were virtually restored by software controlled light conditions and a fine-tuned color compensation image that was projected onto it. In fact, there turned out to be similarities between Rothko’s mural and the Lazar Vozarević’s painting, in terms of the pigments and techniques used.

Comparing the colors used for the painting of Lazar Vozarević with those of Rothko’s murals, it was concluded that both used lithol red. Representative Raman spectra were collected on the selected samples from the painting “Untitled”. Due to high spatial resolution, micro Raman spectroscopy was employed to identify lithol red synthetic organic pigment (Vandenabeele et al. 2000). In the literature there is clear evidence of lithol red fading due to sunlight (Standeven 2008). Having this in mind, careful attention was paid to Raman spectra collection on the paint that contains lithol red, especially to the used power of illumination.

Firstly, physical conservation treatment was carried out to stabilize the paint layer on Vozarević’s painting. Based on the test results of the cleaning exercise, dirt from the paintings surface was removed with distilled water, while the copper plates were cleaned with artificial saliva. The new ground was applied on the damaged areas, which was identically structured to the relief of the original one. Retouching was performed with watercolor paints and completed with MAIMERI® Restaurocolours - pigments with ketone-based resin and selected hydrocarbon solvents, with retouching varnish (J.G. VIBERT retouching varnish; LEFRANC & BOURGEOIS: with acrylic and ketone resin, quick drying petroleum; contains the most resin with 25% dry extract after drying) between them (Jovanović 2012). Both retouching techniques were done in the tratteggio, using vertical lines in different colors. Degraded areas on the surface of the paint layer were not retouched. Although the conservation and restoration activities were carried out successfully, it was evident that the color fading was still highly visible.

Secondly, the virtual restoration treatment was initiated. It was to be conducted in a mixed reality environment, where virtual retouch would be projected onto the painting’s surface, in controlled light conditions. The difference to Rothko’s murals is that Vozarević’s painting has a more three dimensional geometry, which means that the image projection could not have just one direction, since it would not cover all surfaces and would create highly contrasted shadow areas around bump feathers. Also, the decision was made not to use software algorithms for the real-time calculation of the compensation image, but rather to create an exact virtual retouch to be used as a projection under strictly controlled light conditions in the exhibition space, so that the result would be an installation that both emphasizes and unifies the original painting and its virtual counterpart.

3 Proposal for the Virtual Restoration in Improved Reality Process

The virtual restoration project was divided into two phases:

  1. 1.

    Real to virtual – A process of obtaining the exact spatial information of the painting, as well as information on the paint layer’s color and characteristics, i.e. full digitalization of the painting to enable the restoration work in the virtual environment.

  2. 2.

    Virtual to real - Extracting the virtual retouch and implementing it in the real world environment by projecting it onto the painting’s surface. The aim was to create a visual experience that made the painting look fully restored and without physical damage.

3.1 Real to Virtual

To implement this idea, an exact digital copy of the painting needs to be created. This means collecting all the spatial and color information that can be recovered, with the highest possible precision and in a visually adequate sense. Geometrical and spatial information of the painting is needed because the paint layer is not flat, two copper plates are attached in the paint layer and are an integral part of the composition. In addition, the paint layer doesn’t have the same thickness throughout the surface, since the artistic strokes are applied freely and unevenly. Therefore, the visual data collection process has to collect the maximum level of output quality to achieve the desired results. Therefore, the visual data collection process has to meet minimal requirements needed to achieve the desired results.

The initial facts and requirements were:

  • The painting’s dimensions are 160 cm (width) × 58 cm (height) (area of interest - 9280 cm2);

  • The painting has to be an integral part of exhibition;

  • Light conditions in the painting’s environment need to be strictly controlled, to match the virtual compensation retouch.

In order to keep the same light conditions within the exhibition space physical barrier has to be constructed to obstruct light sources that cannot not be controlled (sunlight from windows, distant light sources, etc.). The barrier has to leave enough space for the visitors to observe the painting without getting too close. Within the gallery exhibition space a suitable niche was found in a short corridor that connects the collections from painters cubistic and informal period. The space is adequate for the set-up of controlled light conditions and the size of the space is suitable for visitors to see the painting, without separating the painting from the rest of the collection (Fig. 2).

Fig. 2.
figure 2

Exhibition area and spatial set-up

Full size image

The next step was choosing the projection resolution. We considered the Full HD projectors (1920 × 1080 px) and Ultra HD projectors (3840 × 2160 px). Ultra HD projectors offer up to 0.4 px/mm of line resolution of the projected image, while Full HD 1 px/mm. To attain the best projection quality, UHD projectors were the first choice as they provide a higher resolution image. However, after some initial tests, it was concluded that Full HD projector offers satisfying results at the minimum viewing distance of around 2 m, taking into account the painting’s details which are not very sharp and applied with thick strokes. Even at this lower resolution the projector will have an adequate amount of over 2 million pixels projected onto the painting’s surface. The projection image size resolution was set to 1920 × 696 px. Because of the paint layer’s uneven surface it was decided to project the image from two separate sources, the combination of which would create a united projection, with even lighting on the entire surface of the painting. The projectors will be positioned behind the spectators and above the painting’s top level at an optimal angle. A very small area of the painting cannot be lit this way and will be lit with compensating white LED light, acting as ambient light inside the installation (Fig. 3).

Fig. 3.
figure 3

Schematic view of virtual retouch installation

Full size image

The final set-up of the virtual retouch will use 3 main additive light sources: 2 from projectors and 1 from the general ambient light, in uneven amounts. In other words, the illumination of the painting’s surface will consist of around 20% from white ambient light and 40% from each of the projectors. Some areas (such as the sides of the bump feathers) will be lit from one projector only, which means the projectors have to be strong enough to emit enough light.

A crucial question was how much radiation power would actually be emitted onto the painting. For a conclusive answer to this question, a preventive conservation specialist would have to be consulted. Before a final decision on the project’s approval is made, the exact value of emitted power has to be measured, its influence on the paint layer and the chemical processes inside the pigments analyzed, in order to determine the exact amount of power, light intensity, and safe time duration of single continuous projection per day, month and year. With the potential detrimental effects of the projection in mind, it was proposed to turn the projection automatically on only when visitors approached, while setting strict time limits on the maximum allowed duration (example, just 8 h per day).

The relatively small projection image size of 1920 × 696 px makes the virtual retouch technology widely applicable. The 3D digital model of the painting that will be created will have a higher resolution, making it suitable for other digital-only applications. It will be an exact 3D document of the present state of the painting. To acquire the data, close-range photogrammetry is a chosen method, because of its high-density models and high quality texture maps, and because the process of spatial and color information is simultaneous. The targeted 3D model mesh size is 10 points on 1 mm2, with the same texel density. With the production of high density 3D models with regular mesh grid and rectangular UV map, the first phase of the project will be finished.

3.2 Virtual to Real

The next phase moves the work into the digital realm. The UV map of the 3D model of the painting will be fully flattened, with some little distortions on the bump area, which is inevitable in all 3D models with uneven surfaces. By working with very large-scale texture maps, the appearance of these errors can be minimised to the point of invisibility. The virtual restoration is the next step in the process, creating a layer of retouch in the same manner and colors as it would be created in a regular restoration treatment. This can be carried out in rasterized image editing software such as Adobe Photoshop or Gimp. To ensure the quality of the work, it must be carried out by skilled conservation professionals (Fig. 4).

Fig. 4.
figure 4

Virtual retouch test area and example of projection image

Full size image

While the work on digital retouch is being performed, testing and measurement of the distortions of the projected images should start. Like all devices with optical lenses, projectors have some distortion of image output. This is especially significant in this case, as the image is being projected at an angle. It is crucial for the projected image to perfectly match the painting’s surface in order to prevent blurry and inadequate results. To optimise the projector’s positions, curvature of the image projection (curvilinear projection) needs to be determined, as well as the perspective distortion. The resulting image of the distortion patterns need to be applied on a final virtual retouch image to compensate for all the perspective bending, so that projection geometrically matches the painting’s surface.

The resulting images have to be fine-tuned with chromatic additive matching of the virtual retouch and lightness. This fine-tuning can be embedded directly into the color channels (RGB). The lightness of each projection image will be determined by mutual adjustment and with the consideration of ambient light. The last adjustments will be done on site when the installation is put into place.

4 Conclusion

The aim of this research was to show that the causes and processes of deterioration can only be properly understood when an appropriate conservation treatment is combined with a long-term preservation strategy that focuses on a historical-critical analysis of objects involved, while performing a detailed scientific study of the materials used.

In the case of Lazar Vozarević’s painting, entitled “Untitled”, extensive mechanical damage to the paint layer and lithol red fading had developed, due to inappropriate storage and the continuous exposure to fluctuations in temperature, relative humidity and especially light intensity. The application of traditional conservation techniques was limited. The restoration project focused on a promising new alternative to conventional preservation methods: virtual restoration. It raises important questions on the future of this hybrid digital approach as an alternative to conventional restoration treatments.

We would like to express our gratitude to Ognjen Kovačević, conservator at the Central Institute for Conservation in Belgrade, for his contribution to this research and for his help in writing the article.