Digital Cultural Heritage Imaging via Osmosis Filtering

Abstract

In Cultural Heritage (CH) imaging, data acquired within different spectral regions are often used to inspect surface and sub-surface features. Due to the experimental setup, these images may suffer from intensity inhomogeneities, which may prevent conservators from distinguishing the physical properties of the object under restoration. Furthermore, in multi-modal imaging, the transfer of information between one modality to another is often used to integrate image contents.

In this paper, we apply the image osmosis model proposed in [4, 10, 12] to solve correct these problems arising when diagnostic CH imaging techniques based on reflectance, emission and fluorescence mode in the optical and thermal range are used. For an efficient computation, we use stable operator splitting techniques to solve the discretised model. We test our methods on real artwork datasets: the thermal measurements of the mural painting “Monocromo” by Leonardo Da Vinci, the UV-VIS-IR imaging of an ancient Russian icon and the Archimedes Palimpsest dataset.

1 Introduction

Non-destructive imaging techniques are a well established tool in the diagnostics of artworks. Different spectral bands may unveil hidden information about the surface and the sub-surface layers. Most notable methods within the optical range include UV-based imaging in an emission mode, which exploits the fluorescence of materials to map pigments and past restorations, and IR-based imaging in reflectance mode (reflectography), which allows the detection of underlying features (e.g. the artist preparatory drawing and repaintings) thanks to the optical transparency of pictorial matter in the near-IR (from $0.8\,\upmu \text {m}\, \text {to}\, 2.5\, \upmu \text {m}$). Recently, it has been shown that reflectography techniques can be performed also in the mid-IR thermal region (from $3 \,\upmu \text {m}\, \text {to} \, 5 \upmu \text {m}$) and that such method, called Thermal-Quasi Reflectography (TQR), can complement optical techniques in the analysis of frescoes [5, 7].

Artwork analysis demands the above procedures to have sub-millimetric spatial resolution and full-field capability, which often (large paintings, small sensor array) leads to the practice of mosaicking images acquired in multiple acquisition sessions, each one being the result of its specific setup. Irradiance inhomogeneities, for instance, can be induced by a number of experimental factors related to the kind of acquisition modality, source-camera positioning as well as environmental light. The intensity values in the final mosaic are thus affected by errors and must be post-processed by suitable radiometric procedures in order to obtain the correct reflectance or fluorescence map.

In this work, we discuss the use of the linear image osmosis model proposed in [4, 10, 12] as an unsupervised mathematical tool to correct the inter-frame light differences and to transfer image information in multi-modal imaging [2]. Our numerical implementation relies on the use of mathematical operator splitting techniques considered, e.g., in [1, 3, 4, 8], which allows significant computational speed-ups for large collages. Numerical methods are validated on real artwork datasets acquired via different imaging techniques: a TQR mosaic of the “Monocromo” by Leonardo Da Vinci (Castello Sforzesco, Milan, Italy) [6], a stack of UV-VIS-IR images of an ancient Russian icon and multi-modal data of the Archimedes palimpsest.

Structure of the Paper. In Sect. 2 we present some of the challenges frequently encountered in CH imaging which we are going to address, with a brief review of the image modalities used to acquire the datasets used for our tests. In Sect. 3 the mathematical model of image osmosis [10, 12] is revised and for its numerical realisation operator splitting techniques are briefly described. In Sect. 4 we report the results obtained using the osmosis model to solve the challenges described.

2 Intensity Balance in CH Imaging Modalities

Optical Ultraviolet (UV) - Visible (VIS) - Infrared (IR) Imaging. CH imaging in the optical region from $0.35\, \upmu \text {m}\,\text {to}\, 1.1\,\upmu \text {m}$ is performed with CCD or CMOS sensors with state-of-the-art arrays up to 100 Mp. Multimodal UV-VIS-IR images are acquired in fix geometry, changing lamps and camera setting, and combined in a stack for their subsequent analysis. Problems of large-area uniform irradiance are typical due to the geometry and the size of the available sources, e.g. wood tubes, bulbs, flash, led arrays.

IR reflectography records the near-IR radiation reflected by the object after scattering through the painting layers: image intensity values are here related to the presence of an absorbing/reflective medium. Therefore, the intensity map allows the description of sub-surface features by contrast. In an ideally correct UV fluorescence image, the intensities represent the signal emitted in the VIS range in response to UV-light stimulus and are related to the kind and quantity of the materials in the outer surface.

In multimodal imaging the raw signal is collected using linear detectors for all the different modalities considered, but the acquisition procedure is different (reflectance, thermal quasi-reflectance, luminescence) as well as the employed radiation (optical and thermal).

In CH diagnostics, the fusion of multimodal optical images in pseudo-colour schemes is typically used to enhance differences in the spectral reflectance. A notable example is the IR False Colour technique (IR-R-G), which allows discriminating pigments similar in the VIS but with different IR behaviour.

Thermal-Quasi Reflectography (TQR). TQR is a recent imaging technique for analysing frescoes [5] based on the acquisition of the thermal mid-IR (from $3\, \upmu \text {m}\,\text {to}\, 5\,\upmu \text {m}$) signal in reflectance mode. The rational behind quasi-reflectography is that an object at room temperature emits only $1.1\%$ of its thermal energy in the mid-IR: thus, a signal dominated by the reflected thermal radiation can be collected by sending a non-heating mid-IR stimulus.

In TQR imaging, the intensities in the final image represent the TQR reflectance r, defined as the ratio of the incident and reflected radiant flux in the mid-IR, and obtained point-wise from the acquired radiometric image u using an in-scene calibration target of known reflectance $r_\mathrm {ref}$, i.e. $u_{ij} = r_{ij} u_{\mathrm {ref}} r_{\mathrm {ref}}^{-1},$ where $u_{\mathrm {ref}}$ is the averaged radiometric image of the target.

Thermal camera are equipped with small array sensors (1 Mp). The uniform irradiance of the object is often hard to achieve for thermal sources due to reflections from the environment. Furthermore, an increase of the object surface temperature may occur during the time of a multiple-frames measuring session, causing a variation of the emitted radiance and a shift of the spectral distribution towards the mid-IR [7], which results in inter-frame dis-homogeneity. As a consequence, the mosaic of the TQR frames is often affected by piecewise non-uniform illumination, even after the calibration with the reference target.

Quantitative Material Mapping. In all the imaging modalities described above, it is therefore quite common observing inhomogeneities in the acquired intensity image depending on the experimental setup and measurement errors. A careful processing of the resulting intensity image is therefore desirable to quantify and map the composition of the material under consideration. To solve this problem we consider in the following a mathematical drift-diffusion PDE model.

3 Image Osmosis Filtering

For a regular image domain $\varOmega \subset \mathbb {R}^2$ with boundary $\partial \varOmega$ and given a vector field $\varvec{d}:\varOmega \rightarrow \mathbb {R}^2$, the linear image osmosis model considered in [4, 10, 12] is a drift-diffusion PDE which computes for every $t\in (0,T],~ T>0$, a family $\left\{ u(x,t)\right\} _{t>0}$ of regularised images of a positive initial gray-scale image $f:\varOmega \rightarrow \mathbb {R}^+$ by solving:

$$\begin{aligned} {\left\{ \begin{array}{ll} \partial _t u = \varDelta u - \text {div}(\varvec{d}u) &{} \text { on } \varOmega \times (0,T] \\ u(x,0)=f(x) &{} \text { on } \varOmega \\ \langle \nabla u - \varvec{d}u, \varvec{n} \rangle = 0 &{} \text { on } \partial \varOmega \times (0, T], \end{array}\right. } \end{aligned}$$

(1)

where $\langle \cdot , \cdot \rangle$ is the Euclidean scalar product and $\varvec{n}$ the outer normal vector on $\partial \varOmega$. Any solution of (1) preserves the average gray value (AVG) of f and it is non-negative at any time $t>0$, see [12, Proposition 1]. Moreover, by setting $\varvec{d}:=\varvec{\nabla } \ln v$ for a given reference image $v>0$, the steady state of (1) is a rescaled version of v, i.e. $w(x)=\frac{\mu _f}{\mu _v}~v(x)$, where $\mu _f$ and $\mu _v$ are the average of f and v over $\varOmega$, respectively. To avoid the dependence on the reference image v (not available in practice), the vector field $\varvec{d}$ can be modified to balance intensity information in the image or to integrate on one image the information contained in another image (seamless image cloning).

3.1 Numerical Realisation

In [10], a fully discrete theory for the model (1) is studied. For a given $\varvec{f}\in \mathbb {R}^N_+$, a spatial finite-difference discretisation of the differential operators in (1) reads

$$\begin{aligned} \varvec{u}(0) = \varvec{f}, \qquad \varvec{u}'(t) =\varvec{A} \varvec{u}(t),\qquad t>0. \end{aligned}$$

(2)

For the time discretisation of (2), in [10, 12] the authors consider standard forward and backward Euler schemes and prove conservation and convergence properties analogous to the continuous model (1), [10, Proposition 1]. In particular, unconditional stability is observed for the fully implicit case. Computationally, iterative solvers (e.g., BiCGStab) are used to solve the resulting penta-diagonal and non-symmetric linear systems.

Operator Splitting Schemes. The computation of the numerical solution of (2) may be extremely costly for large images. To overcome this issue, we consider in the following a method similar to the one proposed in [4] and based on a dimensional splitting idea.

Given a domain $\varOmega \in \mathbb {R}^s$ for $s\ge 1$, dimensional splitting methods are based on the decomposition of the operator $\varvec{A}$ of the initial boundary value problem (2) into the sum:

$$\begin{aligned} \varvec{A}=\varvec{A_0}+\varvec{A_1}+\ldots +\varvec{A_s} \end{aligned}$$

(3)

where the terms $\varvec{A_j}$, $j=1,\ldots , s,$ encode the linear action of $\varvec{A}$ along the space direction j while $\varvec{A_0}$ may contain mixed-derivative, non-stiff and non-linear terms.

Note that when $\varvec{A_0}=0$ additive (AOS) and multiplicative (MOS) operator splitting techniques can be used to solve the problem in a fully implicit way, thus ensuring unconditional stability [8, 11].

Using standard finite difference space discretisation, such methods requires the numerical inversion of tridiagonal matrices, which can be rendered efficiently via standard matrix factorization techniques.

In our case $s=2$, $\varvec{A}_0=0$ and the tridiagonal matrices $\varvec{A_1}$, $\varvec{A_2}$ read

$$\begin{aligned} \varvec{A_1}\varvec{u}&= u_{i,j} \bigg ( -\frac{2}{h^2}+\frac{d_{1,i-\frac{1}{2},j}}{h}-\frac{d_{1,i+\frac{1}{2}}}{h}+\frac{d_{2,i,j-\frac{1}{2}}}{h}-\frac{d_{2,i,j+\frac{1}{2}}}{h} \bigg ) \nonumber \\&+ u_{i+1,j} \bigg ( \frac{1}{h^2}-\frac{d_{1,i+\frac{1}{2},j}}{2h} \bigg )+ u_{i-1,j} \bigg ( \frac{1}{h^2}+\frac{d_{1,i-\frac{1}{2},j}}{2h} \bigg );\\ \varvec{A_2}\varvec{u}&= u_{i,j} \bigg ( -\frac{2}{h^2}+\frac{d_{1,i-\frac{1}{2},j}}{h}-\frac{d_{1,i+\frac{1}{2}}}{h}+\frac{d_{2,i,j-\frac{1}{2}}}{h}-\frac{d_{2,i,j+\frac{1}{2}}}{h} \bigg ) \nonumber \\&+ u_{i,j+1} \bigg ( \dfrac{1}{h^2}-\dfrac{d_{2,i,j+\frac{1}{2}}}{2h} \bigg )+ u_{i,j-1} \bigg ( \dfrac{1}{h^2}+\dfrac{d_{2,i,j-\frac{1}{2}}}{2h} \bigg ). \nonumber \end{aligned}$$

(4)

We make use of the Additive Operator Splitting (AOS) numerical scheme [8] where for every iteration $k\ge 1$ and any time-step $\tau >0$ the update reads

$$\begin{aligned} \mathbf {u}^{k+1}&= \frac{1}{2}~\sum _{n=1}^2~\left( \mathbf {I} -2\tau \mathbf {A}_n\right) ^{-1}\mathbf {u}^{k}. \end{aligned}$$

(AOS)

The AOS method is unconditionally stable and first-order accurate in time. In [9] the scale-space properties of such splitting scheme applied to the osmosis problem are shown.

4 Applications

In the following we adapt the osmosis model (1) for the CH imaging problems described in Sect. 2.

Datasets. The TQR mosaic in Fig. 1b ($4717\times 7066$ pixels) is made by 33 overlapped TQR tiles shot by FLIR X6540sc camera used as a radiometer on the “Monocromo” wall fresco (L. Da Vinci, Castello Sforzesco, Milan, Italy) in Fig. 1a, see [6, 7] for more details on the workflow. The UV and IR data acquired on a Russian icon in Fig. 1a compose a mosaic of 5 overlapped tiles ($3671\times 4551$ pixels) shot by a Nikon D810A camera. The Archimedes Palimpsest in Fig. 4a ($1200\times 1600$ pixels) reveals a 10th-century copy of works by Archimedes of Syracuse in Fig. 4b, a pseudo-colour UV-RGB separation.

4.1 Light and Colour Correction

We start testing the osmosis model on the light balance problems described in Sect. 2 for TQR and IR imaging. Let $\varOmega _{b}\subset \varOmega$ be the boundary between the collaged images, for this application the vector field $\varvec{d}$ is defined as: $\varvec{d} = \chi _{\varOmega \setminus \varOmega _b} \nabla \ln f$. In Fig. 1 we consider a TQR mosaic of images of the “Monocromo” fresco showing light inhomogeneities. We compute the solution by (AOS) scheme with nested LU factorisation of the tridiagonal matrices. We fix the parameters as $T=1\mathrm {e}5$, with time-step $\tau =1000$. The elapsed time is reported in the caption.

Fig. 1.

Light-balance on TQR data: $T=1\mathrm {e}5$, $\tau =1\mathrm {e}3$.

A similar approach is applied in Fig. 2 for UV fluorescence imaging data sampled from the Russian icon in Fig. 2a. Once again the inhomogeneities due to the collage of multiple samples (inter-frame error) are minimized.

Fig. 2.

Light-balance on UV fluorescence data (detail): $T=1\mathrm {e}5$, $\tau =1\mathrm {e}3$.

4.2 False Colour Adjustment

In Fig. 3, we observe how the measurement errors affect the false colour rendering of the Russian icon in Fig. 2a and lead to incorrect material mapping. In particular, note in Fig. 3a the false colour IR-R-G image where the IR channel is affected by measurement errors. The use of the osmosis model for IR light balance makes the different regions comparable, see Fig. 3b.

Fig. 3.

Light-balance for (zoomed) false-colour images: $T=1\mathrm {e}5$, $\tau =1\mathrm {e}3$.

Fig. 4.

Multi-modal data integration on Archimedes Palimpsest. Top row: full experiment; bottom row: zoom. Parameters: $T=1\mathrm {e}5$, $\tau =1\mathrm {e}3$.

4.3 Multi-modal Data Integration

The osmosis model can also be tailored to fuse image information with data coming from other image sources, see, e.g., in [2] an application to art restoration. In Fig. 4, we report the digital rendering of a CH image integrating the visible image with multi-modal pseudo colour data.

Namely, having set $v_1$ to be the image in Fig. 4a and $v_2$ to be the image in Fig. 4b revealing the text underlying the Fig. 4a, we post-process $v_2$ with a local Otsu thresholding method to segment the calligraphic data, while averaging the remaining information. Then we decompose the image domain $\varOmega$ into $\varOmega =\varOmega _1\cup \varOmega _2\cup \varOmega _3$ where $\varOmega _3$ is a tiny overlapping frame between an exterior rectangular frame $\varOmega _1$ (the parchment) and an interior rectangular $\varOmega _2$ (the text bounding box). Thus, we define the driving vector field $\varvec{d}:=\varvec{\nabla } \ln v$ with $v= \chi _{\varOmega _1} v_1+ \chi _{\varOmega _2} v_2+ \chi _{\varOmega _3} (v_1+v_2)/2$ and run the osmosis model to get the composite result in Fig. 4b integrating the pseudo-colour data visible on the input image, thus unveiling the hidden Archimedes’ writing.

5 Conclusions

In this paper we consider a variety of problems arising in CH imaging where the image osmosis model (1) [4, 10, 12] is used to correct and integrate multi-modal image acquisitions in different bands (from optical to thermal) in both reflectance and emission modes. To compute the numerical solution of the model efficiently for large image data, a dimensional reduction splitting approach is used.

A free release of the MATLAB code used to compute the results reported in this work will be made available in the next future.