1 Introduction

Important historical and cultural assets are under threat of destruction for degradation and destruction due to aging, natural disasters such as fire and earthquakes, wars and social conflicts. Nowadays it is a common understanding that high quality documentation of these important assets using the state-of-the-art digital technology is an urgent social need [1, 2]. During the past decades, especially during the world war two which destroyed enormous amount of cultural assets in Europe and in Asia, local wars and invasions such as the events in Afghanistan, Iraq, and Syria destroyed some extremely important human cultural assets. In this paper we focus on developing an ultra high resolution digital imaging system with high definition color (including monochromatic tone) reproduction of large objects, glass plates as the main items. In an exceptional documentation (photography) project, wall paintings of golden hall in Horyu-ji Temple were photographed on a one-to-one scale on glass dry plates with multi-band filter 80 years ago. The glass plates were 45 by 60 cm and were photographed by using a special made camera, on 362 pieces of glass plates, using 4 colors and IR (5 spectral bands). Because of the importance of these wall paintings of about seven century, almost 1300 years old, and the invaluable cultural importance of this temple, the project was financed and supervised by Japanese Ministry of Culture (of that period) and a team of experts from Benrido Inc., a printing company in Kyoto, was assigned to implement the project. This amazing project was successfully carried out and the result, 431 large glass plates, was produced, 70 of them in 5 band spectral images [3]. A few years after the photography, the temple was on fire and the major part of the temple’s wall paintings were heavily burnt (1949). In 1993, Horyu-ji together with Hokki-ji was designated as a UNESCO World Heritage Site under the name Buddhist Monuments in the Horyu-ji Area. What exist now after the fire are some burnt wall paintings, part of the building, many mosha (replica reproduction of the wall images before fire) and a collection of photographic 5-band spectral images on large glass plates. Because of the importance of these historical remains, the glass plates are designated as Japanese National Heritage in recent years.

On the other hand, glass dry plates deteriorate over time and also affected by preservation condition. Therefore, there is a need to preserve them in digital format. Because of the size of the glass plates, it is difficult to find a proper image grabbing systems to scan them with high resolution. There were no color reference charts when the wall paintings of Horyu-ji Temple were photographed. Although the team carried out a tremendous technical job, the level of the technology of those days were very low. Creating the original data without having standard reference data is a very difficult job. Using a color chart has not been systematized. Therefore, this paper gives a brief record of our project to develop a scanning system that is capable of digitizing these large glass plates with ultra high resolution (2500–5000 dpi) and a method to reproduce the colours with high color definition from available information. The imaging mechanism of traditional scanning systems is that the light reflected from an object was detected by the camera sensor then collected light would be changed into electrical signal [4]. And usually the traditional scanner has limited size requirement for an object and the glass dry plate of Horyu-ji Temple is large. Therefore, in a scanning of the glass dry plate, they cannot be used. In this study, we develop a scanning system that can be used for large permeable objects with practically no limit in size. This study can make it possible to increase the number of objects for high-resolution digital archiving and to enable everyone to access the wall painting of Horyu-ji Temple by preserving it in digital format.

2 Image Acquisition Experiment

2.1 Mechanism of Scanning System

Line sensor camera and area sensor camera are widely used to acquire images. Line sensor camera can get higher resolution images compared with area sensor camera [5]. Therefore, line sensor camera is more suitable for this study. In this paper, main scanning direction means the direction in which sensors are lined up and sub scanning direction means vertical direction to the main scanning direction.

There are several points need to be noticed. The glass dry plate is a transparent object, and it is difficult to finish it in one scan. In addition, this significant cultural heritage would be digitized as monochromatic image in this experiment. Therefore, there is a need to construct a non-contact-type scanner that can scan transparent object in monochromatic image without size limitation.

Imaging system is roughly divided into a camera unit, a subject base unit, and a light source unit, and arranged in this order so that light from the light source passes through the subject and enters the camera. The scanner that has this structure is named as transmission type scanner. Figure 1 shows the mechanism of my system. One object is acquired by moving the subject base portion in the sub scanning direction in a state where the camera and the light source are fixed. Also, since the subject in this study is a cultural property, it is not permissible for the subject to be damaged. Light and heat are known to degrade color, and ultraviolet rays and heat are factors that deteriorate cultural properties, so they cannot be used. For this reason, we use a LED light source with less heat and ultraviolet emission compared with halogen lamps and HMI (Hydrargyrum Medium-arc Iodide). Since the whole image cannot be acquired with the line sensor camera at high-resolution, it is necessary to move the subject in the sub scanning direction with a fixed width, and the subject is moved in the main scanning direction, and by repeating this, several images are acquired. High-transmission glass is used for the subject base part in order to image an object that transmits light.

Fig. 1.
figure 1

The mechanism of the transmission type scanner.

Full size image

2.2 Device Details and Imaging Conditions

In the transmission type line scanner developed in this research, it is possible to freely set the resolution of the image and the size of the object. For each part of the scanner, we used monochrome line sensor camera, lens, IR-UV cut filter for camera part, LED light for light source, and high-transparency glass for object base part.

The imaging resolution is set to 2400 dpi. In this setting, it is possible to acquire a monochrome image of 8000 pixels in the main scanning direction, and imaging with a width of 8.5 cm can be performed in one scan. It is known that the image acquired by the line sensor camera is uneven in the main scanning direction due to unevenness of the light source or the sensitivity of the sensor, and the edge of the image tends to be dark. Since the width of the subject is 45 cm, one glass dry plate is scanned in nine times of imaging with an offset interval of 5 cm and joined by image processing.

2.3 Imaging Result and Image Alignment with Affine Transformation

Figure 2 shows acquired images of four color decomposed glass dry plate of wall painting of golden hall of Horyu-ji Temple. The four glass plates were taken through filters with different extraction wavelength ranges, respectively. In this study, glass plates were captured by filters of cyan, magenta, yellow and black are defined as C, M, Y and K, respectively. We must superimpose the images with a perfect pixel to pixel registration. However this can only be done if the acquisition system is with high resolution and high special accuracy. Therefore positional deviation can be seen. Since luminance values are used for color correction, it is necessary to correct positional deviation. Figure 3 shows a diagram in which four images are superimposed on the same portion as a reference only with parallel shift. As shown in Fig. 3, the color blurring phenomenon is caused by positional deviation. In order to correct this, there is a need to perform image alignment with more linear transformations such as rotation and scaling. For this reason, we used affine transformation for image alignment.

Fig. 2.
figure 2

Photographed images of four color decomposed glass plate.

Full size image
Fig. 3.
figure 3

The whole image aligned by affine transformation and enlarged image of blue frame. (a) represents the whole image. (b) represents the enlarged image of the blue frame in (a). (Color figure online)

Full size image

Affine transformation is a combination of some linear transformations (rotation, scaling, shearing). Since a combination of several transformations is obtained as one linear transformation, if X and Y are affine spaces, then every affine transformation f: X → Y is the form x ↦ Ax + b, where A is a linear transformation on X and b is a vector in Y. Affine transformation is generally described as follows:

$$ \left( {\begin{array}{*{20}c} y \\ 1 \\ \end{array} } \right) = \left( {\begin{array}{*{20}c} \varvec{A} & \varvec{b} \\ {0, \cdots ,0} & 1 \\ \end{array} } \right)\left( {\begin{array}{*{20}c} x \\ 1 \\ \end{array} } \right) $$
(1)

This is equivalent to y = Ax + b.

Figure 4 shows the result comparison of the alignment manually and using affine transformation. According to Fig. 4, the color blur with manual alignment has disappeared. Then, the detail of color correction would be discussed in next part.

Fig. 4.
figure 4

Enlarged images aligned by hand and by affine transformation. (a) represents the image with manual alignment. (b) represents the image with affine transformation. (Color figure online)

Full size image

3 Color Correction Based on Pigment Information

3.1 Method for Color Correction

In general, for color correction, it is necessary to convert RGB values, that is a luminance obtained from imaging, into XYZ values and then L*a*b* values [6]. In conversion from RGB values into XYZ values, linear multiple regression analysis is often used. Setting the matrix F as a transformation matrix from RGB values to XYZ values and using the image luminance value matrix S obtained from the raw data and the matrix X’ of XYZ values of the reference data, the linear multiple regression analysis can be represented as X’ = FS. By defining S and X’, the coefficient matrix F of the multiple regression analysis model is determined by following equation using Moore–Penrose generalization inverse matrix: F = X’S T(SS T)−1. Here, symbol T expresses transposition of a matrix. By applying the coefficient matrix F to each pixel value of the target image, the color information of the image can be converted into XYZ space. Then, the pixel values are converted into L*a*b* values based on the definition by CIE. Calculation formulas are shown below:

When \( Y/Y_{\text{n}} > 0.008856 \),

$$ L^{*} = 116\left( {Y/Y_{n} } \right)^{1/3} - 16 $$
(2)

or when \( Y/Y_{\text{n}} \le 0.008856 \),

$$ L^{*} = 903.29\left( {Y/Y_{n} } \right) $$
(3)

where \( Y_{\text{n}} \) is the value of Y due to the standard illuminant of the fully diffused reflector and the auxiliary standard illuminant.

$$ a^{*} = 500\left[ {\left( {X/0.9505} \right)^{1/3} - Y^{1/3} } \right] $$
(4)
$$ b^{*} = 200\left[ {Y^{1/3} - \left( {Z/1.089} \right)^{1/3} } \right] $$
(5)

Then, minimize the color difference in the L*a*b* space to make it look more faithful to the real image when viewing the image with human eyes. Setting the matrix G as a transformation matrix from L*a*b* values to L*a*b* values and using the matrix L obtained from Eqs. (3, 4, 5) and the matrix L’ of L*a*b* values of the reference data, the linear multiple regression analysis is described as L’ = GL. As with the matrix F, by determining L and L’, G is obtained from the following equation: G = L’L T (LL T)−1. By applying the coefficient matrix G to each pixel value of the target image, a color closer to the reference data can be obtained. In order to output as an image, the obtained L*a*b* values are translated into XYZ values and then into RGB values.

3.2 Color Correction Without Reference Color Chart

There was no color chart when the wall paintings of golden hall of Horyu-ji Temple were photographed, so it is not possible to create a reference data using a color chart. The method of creating the reference data without using a color chart has not been developed yet. Therefore, some existing pigment information documents or database are employed for color correction in this study. In this method, pigments are specified by reference documents, and a reference data base is prepared from them. As a result of literature survey, we found the pigments that were used in each part of the mural painting by artists visual observations and a rough classification of the pigment color [7,8,9], although there is no scientific analytical proof. Table 1 shows the pigment used in the mural painting of Horyu-ji Temple. However, it is the rough classification of the pigment and to make a reference data, it is necessary to specify detail classification. Therefore, we referred to our own data base, mainly after the book [10] with more than 1000 kinds of pigment such as classical natural mineral pigments and synthetic mineral pigments, etc. and detailed explanations of each color. Then, most of the pigment colors of the mural painting of Horyu-ji Temple were specified based on whether it had existed during that time or on the basis of the main component and the size of the grain. The color information of the pigment was obtained from the list which obtained XYZ values and L*a*b* values from spectral reflectance of enormous number of colors in our laboratory. For colors not listed, XYZ values and L*a*b* values were identified using the pigment estimation software developed in my laboratory [11,12,13]. From these results, reference data could be created.

Table 1. Pigment table of the wall painting of golden hall of Horyu-ji Temple.
Full size table

Regarding the method of acquiring a sample data of each color, it is found that the color correction result obviously became better when the sample data was not acquired from the position too closed to each other but different positions or non-adjacent positions. Figure 5 shows the model of acquiring a sample data in the case of acquiring 81 pixels and Fig. 6 shows the result comparison of color correction by different sample data acquisition methods. According to Fig. 6, it can be seen that there is color collapse in the image of (a), but the image of (b) is reproduced without color collapse. Therefore, the luminance value acquisition area is selected from as many locations as possible, and the luminance values of all the pixels in the selected area are adopted as input data.

Fig. 5.
figure 5

Acquisition models of sample data when obtaining 81 pixels. (a) represents the case of obtaining a cluster of points from one place. (b) represents the case of obtaining a cluster of points from various places.

Full size image
Fig. 6.
figure 6

Images that show difference between two cases to obtain 405 pieces of data around the lips of the figure. (a) represents the case of obtaining a cluster of points from one place. (b) represents the case of obtaining a cluster of points from various places. (Color figure online)

Full size image

4 Results and Discussion

Figure 7 shows the image as a result of green-based color correction. From the image, it can be seen that color restoration is performed with high resolution. On the other hand, there are some problems with color reproduction. Figure 8 shows the result of color correction when different green-based colors are used as reference data. First, according to Fig. 8, the result varies depending on minor difference in reference data. Although the related documents about pigment and replica of wall painting are employed as reference in this study, it is still difficult to confirm the appearance of real pigments used on that wall painting. Moreover, some pigment color changed over the years. There is a need to consider pigment discoloration and fading. However, it is difficult to quantify and estimate the discoloration, so it is not included in this study. Therefore, it is a future task to study how much they can affect the final result of color reproduction. Second, color correction using color chart is performed with 228 types of color information with different hue and lightness, whereas five types of pigments ware used in this study. The color reproducibility is inferior to that of general digital archiving. In fact, the color difference of the image reproduced in this research is 9.82, 14.7, 10.9 of Fig. 8(a), (b) and (c), respectively, much higher than that of the conventional digital archiving using the color chart that is 1.6 to 3.2. However this work is a primary stage and when information on pigments used for the paintings of golden hall of Horyu-ji Temple is increased and reference data can be added, color difference can be reduced.

Fig. 7.
figure 7

Green type color correction result with referring to burned patina color and a replica. (a) represents the result of color correction (2400 dpi). (b) represents the replica (96 dpi). (Color figure online)

Full size image
Fig. 8.
figure 8

Comparison of results of color correction using some green colors as reference data. (a) represents the result image of ancient patina color. (b) represents the result image of pine needle patina color. (c) represents the result image of burned patina color. (Color figure online)

Full size image

Although there are many more factors to consider, a transmission type scanner has been successfully developed, and a method for color reproduction has been proposed. And the color reproduction method can reproduce high-fidelity-color according to the reference data.

5 Conclusion

In this study, we constructed a transmission type scanning system with ultra high resolution (2500–5000 dpi) and a method to reproduce colors in high-fidelity from color information by using digital image processing. Our objects were numerous old glass plates having the images of the wall paintings of the golden hall of Horyu-ji Temple that is designated as a UNESCO World Heritage Site, and these glass plates are also designated as Japanese National Heritage in recent years. Because of the importance of these wall paintings, there is a need to construct a non-contact-type scanner that can scan transparent object in monochromatic image without size limitation. Usually traditional scanners have limited size requirement for an object and the glass dry plate of Horyu-ji Temple is large. Therefore, in a scanning of the glass dry plate, they cannot be used. In this study, we developed a scanning system that can be used for large permeable objects with practically no limit in size and succeeded in imaging monochrome dry glass plate decomposed into four colors. However, when four images are superimposed, positional deviation can be seen and the color blur phenomenon was caused by position deviation. In order to correct this, we adopted affine transformation to alignment and then the color blur with manual alignment has disappeared.

In this study, we established a color reproduction method based on pigment information and realized color reproduction at high resolution. There was no color chart when the wall paintings of Horyu-ji Temple were photographed, so it is not possible to create a reference data using a color chart. The method of creating the reference data without using color chart has not been systematized. Therefore, some existed pigment information documents or database are employed for color correction in this study. In this method, pigments are specified by reference documents, and a reference data is prepared from them.

From the result of green-based color correction, it can be seen that color correction is performed with high resolution. However, the result varies depending on delicate difference in reference data and some pigment color changed over the years. It is difficult to quantify the discoloration, so it is not considered in this study. And color correction using color chart is performed with 228 types of color information with different hue, lightness, and saturation, whereas five types of pigments can be adopted in this study. Therefore, the color difference of the image reproduced in this research is inferior to that of the conventional digital archiving using color chart. Therefore, it is a future task to study how much the discoloration can affect the final result of color reproduction and to investigate information on pigments used for the paintings of golden hall of Horyu-ji Temple and to increase reference data. Although there are some things to consider, we developed a transmission type scanner and a method for color reproduction for high-fidelity-color without color chart.