Kyushu Decorative Tumuli Project: From e-Heritage to Cyber-Archaeology

Abstract

Digitization of cultural assets has become an important sub-area of computer vision (CV). Thus far, the value of digitization has been emphasized in terms of asset preservation and exhibition. The third aspect of digitization value is that the obtained digital data can be used to perform archaeological analysis based on physics and optics theories and simulations. This position paper emphasizes the importance of this third aspect, using our Kyushu decorative tumuli project as an illustrative example. In particular, we focus on the photometric approaches in the third aspect and explain the equipment and methods developed there as well as archaeological findings. This paper, then, proposes to establish this area as “cyber-archaeology” through categorizing and organizing those methodologies.

1 Introduction

Digitization of cultural assets for preservation and promotion, which is referred to as e-Heritage (Ikeuchi 2013), is an interesting and promising sub-area of computer vision (CV). This sub-area, which has spread over the last two decades, promotes the advances in data-collection hardware and data-processing software in the CV area as well (Levoy et al. 2000; Fontana et al. 2002; Allen et al. 2003; Ikeuchi and Miyazaki 2008; Bok et al. 2011; Gomes et al. 2014). These early projects focused on accurately collecting and recording geometric digital data for preservation of cultural assets.

Digital data provide important cues for restoration when the original assets are damaged or destroyed. For example, recently, when Notre Dame de Paris was destroyed by fire, 3D data acquired before the fire provided valuable clues for restoration. When some tumuli in Kyushu were damaged by Kumamoto earthquake, our data obtained in Kyushu project served as references for restoration.

Digital data are also used for exhibition by creating VR displays and MR tours. IBM’s Eathernal Egypt project(Rushmeier 2005) aimed to create a digital guide based on the collected 3D data. The original 3D data of Kalabsha temple, relocated for avoiding being submerged in water, was used to create how it looked in its previous location and how the unevenness of the hieroglyphics in the temple would have looked under a sesame oil lamp (Sundstedt et al. 2004). U Tokyo’s Virtual Asuka project (Sato et al. 2014; Kakuta et al. 2008; Fukiage et al. 2014) designed a system that allows visitors to experience ancient landscapes by installing multiple head-mount displays on an electric bus and superimposing images of ancient buildings on the actual outdoor landscape. IIT’s digital Hampii project allows visitors to virtually tour the temple from a remote location using a mobile terminal (Mallik et al. 2017).

This position paper argues that digital data are valuable for archaeological analysis. In fact, if digitization is to be used only for content creation, then, detailed 3D geometric and photometric data are not necessary, and the content can be produced by stitching together 2D images of the assets of interest. This paper emphasizes that the detailed data collected can be of significant value as basis for conducting archaeological studies. This paper names this field of digital data applications as “cyber-archaeology”.

Cyber-archaeology can be categorized into geometry-based and photometry-based approaches. Early e-Heritage projects also included some elements of geometry-based cyber-archaeology. The geometry-based approach aims to compare cultural assets on the basis of quantitative measures derived from 3D geometric data, which cannot be obtained from subjective comparison of appearances. For example, in Digital Michelangelo project, an attempt was made to identify the type of the chisel used based on the 3D data of line engravings (Levoy et al. 2000; Pietroni et al. 2011). In order to evaluate the credibility of the attribute hypothesis of a Renaissance artifact, comparison of the contour generated from its 3D data with a Michelangelo’s silver-point drawing was conducted (Dellepiane et al. 2007). In our Bayon project, the similarity of the 173 Bayon deity faces was evaluated quantitatively using those 3D data, and from the comparison, we concluded that at least four independent groups of sculptors worked in parallel (Kamakura et al. 2005). In our Roman project, we identified the previously unknown sculptor of an Amazon bust from comparisons of its 3D data with the bust data of which sculptors are known in the literature (Sengoku-Haga et al. 2017).

Fig. 1

Kyushu decorative tumuli. The 3D geometric and spectral data of these tumuli were collected, and based on these data, permanent exhibitions at Kyushu National Museum were created. Each photo depicts one scene from each of 10 VR exhibits

This paper overviews photometry-based approaches of cyber-archaeology using our Kyushu tumuli project as an illustrative example. Compared to the geometry-based approach, the photometry-based approach has not received much attention. Indeed some photometry-based approaches were used in the analysis of medieval paintings (Cucci et al. 2016), but not so much in archaeological sites. The reason for this is that colors are rarely preserved in archaeological sites, making analysis difficult. Fortunately, many of the tumuli in Kyushu island retain their original colors, and various archaeological findings were gained through digitizing and analyzing them. By explaining the equipment and methods developed in Kyushu project, this paper will organize and systematize the various types of photometry-based approaches and emphasize its importance as a field. Further, it clarifies the position that two approaches, geometry and photometry, form the two wheels of cyber-archaeology. Since Kyushu project itself has not been introduced well in the computer vision community, and since the perspective of the entire project is the key to understanding this position, the paper also describes Kyushu project in some detail. As such, the paper takes on the color of a project report as well as a position paper.

Section 2 overviews Kyushu project and introduces the sensors developed to obtain a wide range of spectral data for the project. This section also describes the flow of how the design of the sensors evolved in the course of Kyushu project. Sections 3 and 4 describe the simulation-based forward synthesis and model-based backward analysis, respectively. These two sections describe representative methods of processing tumulus data for photometry-based cyber-archaeology. In other words, it explains what kind of methods are used for cyber-archaeology and what kind of results can be achieved in collaborating with archaeologists. There is no logical flow between these chapters, but rather a dictionary-like arrangement of the descriptions. In Sect. 5, based on the discussions in Sects. 3 and 4, we will organize the methods used in Kyushu project of the photometry-based approaches and clarify the position of the photometry-based cyber-archaeology using them as examples. Section 6, as an epilogue, summarizes how our cyber-archaeology was perceived in archaeology community, the remaining issues and the future directions of cyber-archaeology.

2 Kyushu Project and Developed Sensors

This section overviews Kyushu project and sensors developed in the project, for providing the basis to explain the position of cyber-archaeology.

Kyushu project started with the aim of providing digital data for creating high-definition VR contents at Kyushu National Museum. In Kyushu island, Japan, many decorative tumuli with mural paintings constructed around the 6th century still remain. Most of these tumuli are not open to the public in order to protect these paintings from deterioration caused by carbon dioxide and external contamination of harmful organisms such as mold by visitors. Thus, very few people in Japan or even in Kyushu island know about the existence of these tumuli. Due to the low profile of the tumuli, Kyushu National Museum decided to create VR contents of these tumuli and their mural paintings for increasing public awareness. From among the existing Kyushu tumuli, the museum committee selected the following ten tumuli as representative ones based on the beauty of their mural paintings or the uniqueness of the pigments used: Ozuka, Benkei, Hinooka, Segonko, Noriba, Sakurakyo, Tashiro-Ohta, Mezurashizuka, Kodadani, and Sekijinsan. See Fig. 1. The committee invited us to collect 3D geometric and spectral data for the production of VR contents. Those contents have been being displayed in the permanent collection of the museum. See some examples of pamphlets in  (Kyushu National Museum 2014, 2015).

In this Kyushu project, we obtained spectral data in addition to RGB data. Usually, RGB data obtained by a standard color camera is used to record color information of cultural assets. However, this simple method is not accurate enough because the observed spectral distribution is the distribution of the products of the light source spectrum and reflectance characteristics of each wavelength. Further, the continuous spectral-distribution observed is compressed into RGB values through three RGB filters specific to the color camera used. Thus, even for the same object, different RGB values are obtained under different light sources using different color cameras. This characteristic is not suitable for recording and, in particular, analyzing accurate color information of archaeological sites.

More precisely, the observed spectral value, $E(\lambda )$, at each wavelength, $\lambda$, is described as the product of the incoming light, $L(\lambda )$, and the surface albedo, $R(\lambda )$:

$$\begin{aligned} E(\lambda )=L(\lambda )R(\lambda ), \end{aligned}$$

(1)

The digital color camera obtains RGB values at each pixel using RGB filters, $F_c$, as:

$$\begin{aligned} E_c = \int L(\lambda ) R(\lambda ) F_c(\lambda ) d\lambda , \end{aligned}$$

(2)

where $c = (r, g, b)$, and $E_c$ denotes the c channel value.

Equation (2) reveals two facts: first, the observed RGB values $E_c$ depend on the illumination spectral distribution $L(\lambda )$. Second, because the camera response curve $F_c(\lambda )$ has a function of wavelength, $\lambda$, it is not constant but varies over the sensitivity range; further, it is not a delta function of a particular wavelength, and the original reflectance R cannot be obtained by simple division of the observed RGB values using the illumination RGB values.

To remedy this situation, it is necessary to acquire spectral data where the input light is sampled at a very narrow interval, $\delta$. The integration of Eq. (2) at the wavelength $\lambda _i$, i.e., the ith sampling band, can be rewritten as:

$$\begin{aligned} E_i = \int _{\lambda _i}^{\lambda _i + \delta } L(\lambda ) R(\lambda ) F(\lambda ) d\lambda . \end{aligned}$$

(3)

Over this small interval, we can assume that $L(\lambda ), R(\lambda )$ and $F(\lambda )$ are constant. Under this narrow-band assumption,

$$\begin{aligned} E_i= & {} k \, L(\lambda _i) R(\lambda _i) F(\lambda _i) \end{aligned}$$

(4)

$$\begin{aligned}= & {} k \, L_i \, R_i\, F_i, \end{aligned}$$

(5)

where k is a constant proportional to the interval.

From this equation, if we know $L_i$ from the measurement, and $F_i$ and k from the calibration, we can obtain the spectral reflectance ratio that is independent of the characteristics of the light source as:

$$\begin{aligned} R_i = \frac{E_i}{k \, L_i \, F_i} \end{aligned}$$

(6)

Another common requirement of Kyushu project was the need to measure a large area at high speed. Spectrometers are well-known devices for acquiring spectral data; however, these general measurement systems can only measure a very narrow area of $0.1^\circ$ to $1^\circ$ around the optical axis. In archaeological sites, painted patterns are often distributed over the entire wall, and it is difficult to estimate in advance where these important patterns are located. Furthermore, the available measurement time is very limited in order to minimize the burden on the tumuli. These three conditions require the development of high-speed wide-range spectral sensors.

2.1 Prism-Based System

We mounted a spectral line filter, a line-spectroscope, in front of a monochrome camera as shown in the left image of Fig. 2a (Ikari et al. 2005). Under the configuration depicted in the figure, the filter has a prism mounted in the vertical direction, and the input vertical light is expanded in the horizontal direction according to their wavelengths as shown in Fig. 2b, which shows an example of the output of the Macbeth color chart under incandescent light along the one line depicted in Fig. 2a. The x-axis shows wavelengths from 400 to 700 nm. The y-axis is aligned with pixels along the vertical line. Namely, the sampling band in the frequency domain and spatial accuracy along the y-axis has resolutions of the x and y axes of the TV camera, respectively. A black and white value indicates the intensity value of the wavelength at that y position in the 8 bit resolution.

Because the device can only measure the spectral data along the y-axis, it is necessary to rotate it for scanning the horizontal direction. Thus, we mount this system on a rotating table to allow us to scan the entire scene. This system was used to obtain spectral data of Ozuka tumulus.

Fig. 2

Prism-based system. a Image acquisition scene. A line spectroscope is mounted in front of a monochrome camera. In this calibration scene, the scanning line crosses the sky-blue, orange, red, and gray boxes of the Macbeth color chart. b Scanning result of the Macbeth color chart under sunlight along one line depicted in a. The x-axis shows wavelengths from 400–700 nm. The y-axis is aligned with pixels along the vertical direction

2.2 LCTF-Based System

The LCTF (liquid crystal tunable filter) system was developed to avoid the issue of viewpoint shift of the prism system. The prism-based system can measure the spectral data of an area with high-definition bands. However, because the system physically rotates and its rotation center does not coincide the optical center of the camera, non-uniformity of data occurred when measuring uneven surfaces due to the effects of occlusion and differences in depth of focus. To solve this issue, we developed a device that can measure the spectral data of the entire image while keeping the viewpoint fixed.

The newly developed system comprises a monochrome camera with a LCTF as shown in Fig. 3a. The LCTF changes the transmittable wavelength according to the applied voltage. The system samples the entire visible light at 81 wavelengths automatically by changing the voltage, which results in a sequence of 81 images from the same viewpoint. Because the filter covers the entire field of view of the camera, the system provides a uniform sampling density in the horizontal and vertical directions. The spatio-resolution of one spectral data was $1280 \times 920$, and each pixel had a spectral data of 81 bands at intervals of $\delta = 4$nm in 400–720 nm with 8 bit intensity resolution. This system automatically adjusts the exposure time according to the sensitivity $F_i$ in Eq. (6) for each wavelength $\lambda _i$.

This device is mounted on a two-axis automatic tripod head to expand the measurement areas. Further, we mount a high-color-rendering light source of which spectral data are obtained in advance. By controlling the head, we can rotate the filter, the camera and the light source simultaneously to acquire an omni-directional spectral image. Figure 3b shows an omni-directional RGB image reconstructed from 81 omni-directional spectral images obtained by the system. This LCFT system was used as the main sensor to acquire spectral data for Kyushu project.

Fig. 3

LCTF-based system. a A LCTF is mounted in front of a monochrome camera. The system can sample the entire visible light at 81 wavelengths according to the applied voltage to the LCTF. The automatic tripod head rotates the system omni-directionally by $360^{\circ }$. b Omni-directional RGB image reconstructed from 81 omni-directional spectral images

2.3 Photometric-Wing System

The main purpose of this position paper is to describe the usefulness of photometric information; geometric information is a secondary topic. However, sometimes, it is necessary to compare detailed geometric shapes with photometric patterns. For example, in Sakura-kyo tumulus, both engraved lines and painting patterns co-exist, and engraved lines have been considered to be draft sketches for the paintings. For this reason, line engravings have not received much attention, and no detailed data was available. However, some parts of them have significant gaps from the painting boundaries, and our team members suspected that these line engravings were not the drafts of the paintings. The detailed comparison requires to measure precise geometric shapes of those engrave lines.

We developed the photometric-wing system that can obtain detailed geometric shapes using relatively weak light sources (Miyazaki et al. 2010; Miyazaki and Ikeuchi 2010). This Kyushu project employed multiple active Lidar sensors to measure the approximate shapes of the chambers. On the other hand, in order to use these sensors to measure the detailed shapes of line engravings with width and depth of about 0.5 mm, it is necessary to irradiate a powerful beam over a narrow area, and this powerful beam is required to scan in sequence over an entire wall, which could have negative impacts on the paintings. After consulting with tumulus administrators and archaeologists, we decided to develop the photometric-wing system to avoid such adverse effects.

The photometric-wing system obtains precise 3D range information over a relatively wide range using shading information under different illumination conditions from the same viewing direction. This system belongs to the family of photometric stereo and  photometric sampler  (Woodham 1979; Nayar et al. 1990; Rushmeier and Bernardini 1999). Because a monochrome camera is used as the receiving system, the spatial resolution can be easily increased. Our current photometric-wing system can measure the distribution of narrow line engraves, approximately 0.5 mm width and 0.5 mm deep, over a relatively wide area of 1 m by 1 m in about 30 s. Also, the light sources for measurement do not need to be very bright, and currently we are using regular 30-W bulbs, and the damage to the paintings is considered to be minor compared to the high-powered Lidar sensors. See Fig. 4.

Fig. 4

Photometric-wing system. a RGB camera with six light sources. b Measurement scene using the photometric-wing system

A photometric-wing system employs six light sources to avoid specular components. Observable angles of specular components are relatively limited. Among the six images under six different light sources, some images contain only Lambertian components. Using the graph-cut method for out-layer rejection, the photometric wing system selects only those images with Lambertian components and determines surface orientations.

3 Forward Synthesis

Usage of obtained photometric information can be roughly divided into two categories: forward synthesis, which generates new appearances under new conditions for archaeological analysis, and backward analysis, which uses several models based on prior knowledge to obtain some unknown parameters for archaeological analysis. Both categories provide interesting findings through the use of archaeological knowledge.

This section describes the use of photometric information for the forward synthesis. A representative example is the case of Ozuka tumulus, which reproduces the appearances of the paintings under different light sources, and provides an example of archaeological interpretations on the construction method of the tumulus based on the reproduction results. First, since the original purpose of the project was to create VR content, an outline of the geometric as well as spectral data acquisition was provided. Then, we will explain the appearance simulation from the spectral data and the archaeological interpretations on the results. Although methodologically it is a simple appearance simulation, it is a good example of cyber-archaeology because the addition of the archaeological knowledge makes the interpretation of the simulation results very interesting. See more examples of the forward synthesis category in Kyushu project in Table 1.

3.1 Data Acquisition for VR Contents

Ozuka tumulus has a large keyhole-shaped mound built in the middle of the 6th century, located in Keisen-cho, Fukuoka Prefecture, as shown in Fig. 5. Mural paintings including concrete patterns such as horses and abstract patterns such as triangular and bi-legged ring-shaped patterns almost entirely cover the interior walls of the main burial chamber. Ozuka tumulus is a historical treasure because it is the only tumulus in Japan known to have six pigments simultaneously: red, yellow, white, black, green, and gray.

Fig. 5

Ozuka tumulus. Ozuka tumulus has a large keyhole-shaped mound build in the middle of the 6th century, located in Keisen-cho, Fukuoka, Japan. Ozuka tumulus is a historical treasure in Japan because six different pigments were used simultaneously on the mural paintings in the stone chambers

Fig. 6

Measurement of Ozuka tumulus. a Measuring the tumulus mound using a laser range sensor. b Obtained point cloud data of the round mound and the stone chambers. c Texture-mapped result of the burial chamber. d Texture-mapped result of the front chamber

We obtained 3D geometric data in a point-cloud format using active-lighting Lidar sensors: the Z + F imager and Minolta Vivid. The Z+F imager is a commercially available active-lighting sensor that projects amplitude-modulated (AM) laser light onto objects. It is used to determine the distance to an object by analyzing the differences between the phases of the outgoing and returned light. The sensor applies laser light to a rotating polygon mirror to sweep one vertical plane. Then, the axis of rotation is changed to sweep the horizontal direction for enabling omni-directional measurement. This sensor was used to measure the overview of the tumulus mound and the rough shape of the burial chamber. The Minolta Vivid sensor is another commercially available active-lighting sensor that is used to measure the detailed wall geometry. This sensor follows the triangulation method to obtain the distance by measuring how a laser beam on a flat surface thrown from one side is observed from a different direction and provides range information.

Figure 6a shows the scene of the data acquisition of the mound and the chamber. After we scanned the entire mound and the inside of the chambers, we combined these partial data into the entire 3D geometric data of the tumulus as shown in Fig. 6b. For this process, we used the simultaneous alignment algorithm developed in the Great Buddha project (Ikeuchi et al. 2007) and a parallel merging algorithm developed in the Bayon project (Ikeuchi and Miyazaki 2008).

Mapping color images to the entire 3D geometric data requires us to capture pictures without leaving gaps. Taking pictures randomly and manually stitching them together yields unsatisfactory results because of differences in resolution and vignetting. Thus, we mounted a prism-based spectral system and a high-color-rendering light source to an omni-directional automatic tripod head. The light source was mounted as close as possible to the viewpoint of the system for avoiding self-shadow regions. This system allowed us to obtain omni-directional high-resolution spectral images quickly.

Figure 6c, d show the texture-mapped result of the 3D geometric data with the RGB values generated from the spectral data. These data were used to create Ozuka VR content, a permanent exhibit, at Kyushu national museum.

3.2 Painting Condition

There are two archaeological theories in Japan regarding the conditions under which Ozuka’s paintings were created. The majority theory is that the artist entered the stone chamber with a torch after the completion of the chamber. However, the minority scholars, including one of the authors, question this theory because there are no traces of soot at the lower part of the walls and the floor of the stone chamber, usually generated by the use of torches, and there is no evidence of soot contamination of the pigments during the painting process. The minority theory, therefore, is that the painting was performed under sunlight when only the walls were completed in the middle of the construction of the stone chamber. To shed some light on these theories, we simulated how the mural paintings on the entire walls would look like under torchlight and sunlight, using the geometric and spectral data obtained for the content creation.

We also measured the spectral distributions of torchlight and sunlight, as shown in Fig. 7, beside Ozuka spectral data. For the spectral data of torchlight, we burned wood that we imagined would have been used at that time, and we measured the spectral data of a white paper illuminated by the flame at night outdoors. For this measurement, we used a typical spot-type spectrometer.

The narrow-band assumption holds for the spectral data. Thus, in each spectral band, the product of the spectral value of the illuminating light and the spectral value of the painting is the observed spectral value. A filter of the human RGB sensitivity function was applied to this distribution of the observed spectral values to generate the appearance of the wall to human. Figure 8a shows the mural painting under sunlight, while Fig. 8b shows one under torchlight (Masuda et al. 2008).

We found the part of the painting shown in the figure, and on the basis of this we concluded that it is more likely that the painting was done under sunlight. Region A is the part painted with the green pigment (green clay), and Region B is the part painted with the black pigment (Manganese clay). The green and black regions form a repeating triangle pattern. Under sunlight, the two regions are clearly distinguished, whereas under torchlight, they are quite difficult to distinguish. The finding indicates that ancient artists were most likely working in the sun when painting the patterns.

This finding has led to further conjecture on the construction method of the tumulus. We can conjecture that they first made walls, drew pictures, then closed the ceiling, and finally made a mound on it. Of course, it is possible that the constituent rocks were painted elsewhere in the sun before being assembled, but it seems that such a possibility is unlikely, if not impossible, because of the continuity of the repeating patterns among multiple rocks surrounding the chamber.

This case is one of the representative examples of forward synthesis aspect of cyber-archaeology in which simulations based on e-Heritage data provide a new interpretation of how to construct a tumulus. For other examples of the forward synthesis category, see Table 1.

Fig. 7

Spectral distribution of torchlight and sunlight. a Torchlight. b Sunlight

Fig. 8

Simulation results. Some regions cannot be distinguished under torchlight, which indicates the patterns are likely to have been drawn in the middle of construction. a Appearance under torchlight. b Appearance under sun light

4 Backward Analysis

This section will discuss representative examples of the backward analysis of cyber-archaeology, to backward track the image formation process from appearances to the parameters such as clear boundaries or thickness of the pigment layer. The methods in this section is based on the assumption that the number of colors that would be observed is known in advance from the chemical analysis of the remaining pigments. Although spectral data is a rich representation, its high dimensionality makes it problematic to handle. The methods in this section tackles this issue by either selecting specific spectral bands or efficiently compressing high-dimensional data into low-dimensional ones.

4.1 Infrared Band Analysis

The simplest method is to generate images with only specific spectral bands for archaeological analysis. It is a relatively simple method from a computer vision perspective, but it is still a great method to provide new discoveries in archaeology. In Kyushu project, Tashiro-ohta’s case is a representative example of this type.

Tashiro-Ohta tumulus has a circular burial mound belonging to the genealogy of the chief tumulus. as shown in Fig. 9a. The chamber is divided into three parts: front, middle, and burial. Three types of pigments remain on the back wall of the granite boulder in the burial chamber. Adding the color of the granite boulder, it can be interpreted that four colors are used for the mural paintings.

Unfortunately, however, when the tumulus was discovered, the lower half of the mural paintings on the back wall were buried in soil. Fig. 9b shows the RGB image of the back wall captured by a standard RGB camera. In the image, some patterns are difficult to recognize because of adhered soil and dirt.

Fig. 9

Tashiro-ohta tumulus. a Overview. b RGB image of the wall painting. c Image generated using only near-infrared bands of spectral data. d Interpretation by an artist. Patterns are depicted as concentric circles and two separate polygons. e Infrared-bands image. The top part is depicted as a polygonal shape with two connected leg-like regions. This is most likely a bipedal ring pattern

It is known that long-wavelength light, especially near-infrared light, which has little scattering on the surface layer, is effective in reconstructing paintings drawn on such a dirty surface. Therefore, the LCTF system was used to make an image consisting only of the near infrared light, as shown in Fig. 9c. The effect of adhering red clay on the surface was reduced, and the patterns drawn in black pigments that remain underneath become clearly visible.

The analysis of the generated image provides the possibility of different interpretations from the current ones. Figure 9d shows the current common interpretation of a pattern on the wall, depicted by an artist, while Fig. 9e is an reconstructed image of the same area in the near-infrared bands image. These two patterns are slightly different in detail. In the infrared-bands image, the top region—drawn as a concentric circle by the artist—is not a circle but a polygon with faint projections at the corners. The top region and two leg-like regions are connected continuously in the infrared-bands image. From this observation, the pattern can be interpreted as a bi-legged ring-shaped pattern (Sokyaku-rinzyou-mone) with two legs sticking out of the polyhedron; this type of patterns was believed not to exist in this tumulus.

The operation is relatively simple for computer vision scientists, as they only need to select the appropriate band to generate the image, although the generation requires a new equipment, i.e., the LCTF system, but the dense cooperation of scientists in the two fields provides new findings in cyber-archaeology.

4.2 Dimension-Reduction Analysis

In the previous section, we avoided high dimensionality in spectral data by focusing on specific bands to be processed. More generally, when it is not clear which bands are effective, we can apply dimension-reduction methods to efficiently map such high-dimensional data onto ones in a low-dimensional space. This subsection introduces the analysis of Noriba tumulus as an example to illustrate the dimensional reduction method. For other examples of dimensional-reduction in Kyushu project, see in Table. 1.

Noriba tumulus has a keyhole-shaped mound built in the second half of the 6th century located in Yame City, Fukuoka Prefecture. Patterns of concentric circles, triangles, swords, and quivers exist on the inner walls of the burial chamber. The mural paintings at Noriba tumulus were severely deteriorated, and the patterns were difficult to recognize.

Mural paintings are considered to have a layered structure that comprise a pigment layer and an underlying rock layer. The surface pigment layer becomes thinner as it deteriorates and becomes mixed with the dirt and soil over time, and this makes it difficult to distinguish in RGB images due to isochromatic conditions. To determine the boundaries of the regions where those pigment layers exist, we used spectral data obtained by the LCTF-based system and applied a dimension-reduction method.

Spectral data of the paintings are considered to form low-dimensional manifolds existing in high dimensional space; the spectrum data of each region in the painting are represented as points on this low-dimensional manifold surface, parameterized by the thickness of a layer. This manifold surface is considered to be locally linearly approximable. Based on this assumption, we developed a two-step region-extraction algorithm, principle component analysis (PCA)-based local and normalized cut (NC)-based global method (Morimoto et al. 2008; Belkin and Niyogi 2003).

The first step locally groups N pixels and generates M super-pixels, where $M (M \ll N )$ using the PCA-based dimensional-reduction method. At each super-pixel, we calculate the average spectral data over the corresponding region and store these P bands spectral data to those super-pixels.

For the second step, a similarity matrix W among the super-pixels using

$$\begin{aligned} W_{ij}= & {} \left\{ \begin{array}{lll} &{}&{}exp\left( \frac{ -\Vert I(i)-I(j)\Vert ^2 }{\sigma ^{2}_{I}}\right) *exp\left( \frac{-\Vert X(i)-X(j)\Vert ^{2}}{\sigma ^{2}_{X}}\right) \\ &{} &{}\mathrm{~~~ if} \; \; \Vert X(i)-X(j)\Vert ^2<r\\ &{}0 &{} \mathrm{~~~ otherwise}, \end{array} \right. \end{aligned}$$

(7)

where I(i) and I(j) respectively represent the spectral data at super-pixels i and j, and X(i) and X(j) represent the spatial positions of the super-pixels.

Region-extraction among super-pixels can be performed by solving the general eigenvalue problem of the NC method proposed by (Shi and Malik 1997) as

$$\begin{aligned} (D-W) \; Y = \lambda D \; Y, \end{aligned}$$

(8)

where D denotes a $M\times M$ diagonal matrix, $D_j=diag(W(1,j),W(2,j)\ldots ), \;j=1,2,\cdots , M$, and W represents an $M\times M$ symmetric matrix $W(i,j)=w_{i,j}$.

By converting the generalized eigenvalue problem to the standardized eigenvalue problem, Eq. (8) can be transformed into a matrix containing the normalized Laplacian as

$$\begin{aligned} D^{-1/2}WD^{-1/2}\;Z = (1 - \lambda ) Z. \end{aligned}$$

(9)

We can obtain the feature matrix U of $L \times M$ $\left( L\ll M\right)$.

$$\begin{aligned} u_{ij} = \frac{z_{i+1,j}}{\sqrt{D_{jj}}},\quad (i=1,...,L,j=1,...,M,). \end{aligned}$$

(10)

Here, the element $u_{ij}$ is given by the j pixel value of the second and subsequent ith eigen vectors, which corresponds to the eigenvalues in the ascending order.

Finally, M super-pixels are classified into an E class based on the feature matrix U using the K-means clustering method. Generally speaking, in the analysis of mural paintings, the chemical analysis of those remaining pigments reveals in advance how many colors were used in the site. In other words, the number of clusters, E in the k-means clustering method is known; only the boundaries of the pigment regions are unclear. Note that here, clustering is done in the spectral space, and this clustering result is used to extract regions in the spatial space.

Before applying the proposed method to mural paintings, we evaluated the method by analyzing watercolor paintings. Unlike oil paintings, watercolor and mural paintings can be modeled as semi-transparent layers in which incoming light penetrates into the pigment layer and is reflected back from the bottom layer under the effect of scattering on the way. Figure 10 show the region-extraction results of watercolor paintings for the evaluation. The input image comprises four regions of watercolor pigments on a white paper. Thus, we set the number of clusters are five; four pigments and background. First, we utilize three methods: PCA and K-mean clustering, Kernel PCA, and K-mean clustering and the proposed method. The PCA-based method cannot extract the yellow region, and the kernel PCA-based method cannot separate the yellow and the green regions. Only the proposed method can separate the input image into four regions.

Fig. 10

Region-extraction results of layered surfaces. a Input image. b PCA-based feature vectors and k-means clustering. c Kernel PCA-based feature vectors and k-means. (d) Proposed method

Fig. 11

Noriba mural painting. a The front and back chambers viewed from the the front chamber. b Extracted pattern. c Target area in the left wall of the front chamber image. d Extracted pattern. e Interpretation as a bi-legged ring-shaped pattern

We applied the proposed method to spectral images of the walls of the front and back chambers (Fig. 11a). From the image on the left wall of the back chamber, we confirmed the existence of triangular patterns that existence has been believed from the common opinion. See Fig. 11b.

The image of the left wall of the front chamber, Fig. 11c, was grouped in Fig. 11d using the method. We can interpret this region-extraction result as shown in Fig.11e, which suggests the existence of a bi-legged ring-shaped pattern. This is a very surprising discovery because this pattern is very rare, and it has been confirmed in only a small number of tumuli.

4.3 Physics-Based Analysis

In this section, we discuss a region-extraction method that utilizes the light attenuation model in pigment layers (Morimoto et al. 2010b). We named this method the Spider method because the shapes of the curves (manifolds) in a low-dimensional space resemble the legs of a spider. The NC-cut method in the previous section is a general solution to handle low-dimensional manifolds in a high dimensional space and is not based on the underlying physics model of those layers. The Spider method models the physical behavior of the layers and extracts regions by augmenting the data to some extent based on the physical model.

Generally speaking, Western oil paintings consist of thick layers, coated heavily with paints and colored by the paints themselves. The filling ratios of the particles in the paint layers are close to 100 %. Light transmission is almost non-extent and incoming light is reflected back by internal scattering without reaching the bottom layer. On the other hand, those tumulus paintings are similar to watercolor paintings of the East as mentioned previously, with a low particle percentage, and, as the result, the incoming light penetrates the layer and reflected from the bottom rocks with scattering on the way. Namely, the layers are semi-transparent and the appeared colors are mixtures of those of the rocks behind and the paintings. We employed the Lambert–Beer model (Beer 1852) to model this light attenuation effect.

Fig. 12

Layered surface decomposition method. a Spider model in RGB space. b Original image. c Initial bottom and top layers. d Estimated top pigment colors. e Estimated layer thickness

The current version of the Spider method processes three bands corresponding to RGB channels, which we refer to as an RGB-band image for convenience. This operation is necessary to avoid modeling complexity in high dimensions. This RGB-band image is represented in the RGB space and looks like an RGB image, but it is different from a standard RGB image because the standard RGB image is the result of the convolution while the RGB-band image only consists of three short bands, holding the narrow-band assumption.

The apparent color in a mural painting changes gradually as the thickness of the layer changes from an underlying rock color to the pigment color. This color change can be modeled as shown in Fig. 12a based on the Lambert-Beer model. The finger shows three “legs,” which correspond to the three pigment colors in Fig. 12b. Here, it was assumed that the number of the spider legs is known in advance since the chemical analysis of the remaining pigments reveals how many colors were used. Note that the number of the spider legs is the number of colors, not the number of regions. In this example, three colors, three pigments, are accidentally used to paint three regions.

The model is expressed as spectral data as:

$$\begin{aligned} I_{\lambda }(x) = B_{\lambda }(x) e^{-\mu _{\lambda }(x) d(x)} + F_{\lambda }(x) ( 1 - e^{-\mu _{\lambda }(x)d(x)} ), \end{aligned}$$

(11)

where $\lambda$ represents the wavelength and x denotes the pixel position in the image. $B_{\lambda }$ denotes the spectral data of the wavelength, $\lambda$ reflected from the lower layer and $F_{\lambda }$ denotes the reflected data when the thickness of the upper layer is infinite, i.e., when the upper layer is sufficiently thick to eliminate all influences of the lower layer.

Let us assume that we have an RGB-band image consisting of, not a standard RGB components from a standard color camera, but the RGB bands data with a narrow band by the LCTF system. Then, the values of the mixed pixel of the lower and upper layer colors can be derived using the Lambert–Beer model as:

$$\begin{aligned} I_{r}= & {} B_r e^{-\mu _r d} + F_r ( 1 - e^{-\mu _r d}) \nonumber \\ I_{g}= & {} B_g e^{-\mu _g d} + F_g ( 1 - e^{-\mu _g d}) \nonumber \\ I_{b}= & {} B_b e^{-\mu _b d} + F_b ( 1 - e^{-\mu _b d}). \end{aligned}$$

(12)

Here, these three equations are independent of all parameters except the layer thickness d. With a small calculation to eliminate the parameter d, each $(I_r, I_b, I_g)$ can be represented as a curve in the RGB space.

$$\begin{aligned} I_r= & {} F_r + \psi _r(I_g-F_g)^{\gamma _r} \nonumber \\ I_b= & {} F_b + \psi _b(I_g-F_g)^{\gamma _b}, \end{aligned}$$

(13)

where

$$\begin{aligned} \gamma _r= & {} \mu _r/\mu _g \\ \gamma _b= & {} \mu _b/\mu _g \\ \psi _r= & {} (B_r -F_r)/(B_g - F_g)^{\gamma _r} \\ \psi _b= & {} (B_b -F_b)/(B_g - F_g)^{\gamma _b}. \\ \end{aligned}$$

Figure 12a shows an example of a spider model depicted in the RGB space. The three curves in space correspond to the three pigment colors shown in Fig. 12b. Starting from the common bottom rock color in the space, and following the equations, the three curves end up to the three different pigment colors. The shapes of the curves depend on the absorption parameters of the pigment layers. We refer to these curves as spider curves.

The decomposition method can be realized by solving the labeling problem of each pixel to determine the pigment layer to which it belongs. First, the upper and lower layers of the image are marked, as illustrated in Fig. 12c; then, the parameters of the spider model, $\psi _r$, $\gamma _r$, $\psi _b$, and $\gamma _b$ are estimated from the input data using the Levenberg–Marquart method.

(14)

Here $\Omega$ denotes the set of points on the marked lines in the figure. The bottom layer color $B_c$ is obtained as the intersection of several spider legs of the point set, and each upper layer color $F_c$ is obtained as the farthest point on the leg from the bottom point.

Fig. 13

Restoration of the horse pattern in Benkei tumulus. a The original faint patterns. b Extracted pigment layer by the spider method. c Absorption ratio

The similarity values between the spider legs and the remaining pixels are calculated as

$$\begin{aligned} D(Z(x) = z) = \left\{ \begin{array}{ll} 0 &{} \; \; \mathrm{if}\; d(I(x),B)< \theta \\ 1 - e^{-d(I(x),w_z)} &{} \; \; \mathrm{otherwise} . \end{array} \right. \end{aligned}$$

(15)

Z(x) denotes the label of the pixel, x. $w_z$ represents the nearest point of I(x) on the spider’s foot that corresponds to the label. The value, d denotes the Euclidean distance between two points.

We use the multi-label graph cut method (Szeliski et al. 2006) to minimize this cost function.

$$\begin{aligned} E= & {} \sum _x D + \sum _{x,y} S . \end{aligned}$$

(16)

Here, S denotes the smoothness term.

Figure 12d shows the labels obtained using this method. When this method is applied to a mural painting, this labeling image provides distinct shapes in the original painting. Next, the thickness parameters $\mu _c d$ at each pixel can be obtained using Eq. (12) with the obtained parameters and I(x), as shown in Fig. 12e.

We applied the spider method to restore patterns in Benkei tumulus. Benkei tumulus is also a 6th century burial mound with red pigments on the walls depicting horses on boats in the hope that the dead would be able to use such horses and other boats in the afterlife. However, there are traces of human habitation in later times, and the condition of the paintings is not good due to the soot from the fires and the deterioration of the pigments from the heat.

In Fig. 13a, the RGB image of the original patterns, it is quite difficult to recognize a horse and a boat. Figure 13b shows the region-extraction results. Here, we can clearly see a horse on the boat, of which tail is depicted to be extended. This can be interpreted as an indication of the horse running. Figure 13c represents the absorption ratio $\mu d$, proportional to the thickness, d of the pigment layer.

5 Summary and Discussion

This paper has described the overview of our Kyushu project wherein we collected the 3D geometric and spectral/RGB photometric data of the ten tumuli in Kyushu island for creation of VR contents at Kyushu National Museum. It also explained several examples of cyber-archaeology, in which the results of analysis of the processed data provided new findings of archaeology. We argue that the value of e-Heritage data, digital data of cultural assets, for archaeological research, along with conservation, restoration, and exhibition, should be emphasized more. Since the main body of this paper only describes some of the representative cases, the remainder of this section overviews all the cases, systematizes them and summarizes the discussions as shown in Table 1.

As mentioned in the introduction, cyber-archaeology can be divided into geometry-based and photometry-based approaches. Roughly speaking, the geometry-based approaches objectively compare shapes and the photometry-based approaches objectively compare colors, and then, the both approaches attempt to obtain new interpretations from these metric comparisons. The Kyushu project focused on developing various photometry-based approaches for archaeological analysis of mural paintings.

Photometry-based approaches can be further divided into three categories: forward-synthesis, backward-analysis and fusion. The forward-synthesis category, close to the Computer Graphics approach, is to forwardly track the image formation process, starting from model parameters and to generating appearances under various conditions, for archaeological interpretations. The backward-analysis category, close to the computer vision approach, is to backwardly track the image formation process, starting from the appearances and to determine model parameters such as clear boundaries and layer thickness for archaeological interpretations. The approaches in the fusion category analyze the photometric and geometric data in parallel to provide archaeological interpretations.

5.1 Forward-Synthesis Category

The forward-synthesis category leverages the narrow band nature of the spectral data to generate appearances. Assuming narrow bands, the reflectance ratios for each band can be obtained by dividing the spectral data of the object by the spectral data of the light source at the time of data acquisition in each frequency band. Based on these spectral reflectance ratios, the appearance of the target under different light sources can be generated for archaeological interpretations.

In Ozuka tumulus, we simulated the appearance of the paintings under sunlight and torchlight using the obtained spectral data to show the possibility that the painting was performed under sunlight. What is important here is that close collaboration between computer scientists and archaeologists provides interesting research results; simple simulations for computer science may have important consequences for archaeologists.

In Tashiro–Ohta tumulus, the issue was raised in the Japan Diet that despite construction of the preservation facility, the mural paintings turned whitish, which may indicate the facility may not be functioning properly. However, the tumulus administrator noticed that the appearance of the mural painting changes with the seasons and it happened to look whitish at the time of the Diet visit. We received a request from the administer to clarify this issue, when we visited the tumulus to obtain the data for creating the VR content.

We hypothesized that this change in appearance might be caused by changes in the amount of condensation due to changes in temperature and rainfall. In order to examine this hypothesis, we used the spectral data to synthesize appearances under different condensations and found that it does indeed look whitish in dry winter and vivid in wet summer. From this, we confirmed the color changes are due to the seasons, not damage to the paintings, and thus, the facility is indeed working properly; there is no fault in the administrator. This is not an example of archaeological discoveries, but it remains the social significance of the photometric-based analysis.

So far, we have mainly explained examples of appearance generations using the spectral distribution, but from now on, we will explain some examples of appearance generations under different illumination directions and different viewing directions. Note that these cases are not based solely on photometric information, but take into account geometric information. However, since they are basically methods of generating appearances by simulation, they are included in this forward-simulation category instead of fusion category.

Hugoppe tumulus, located in Hokkaido island, is an example based on appearance generations by different light source directions. The line engravings of Hugoppe tumulus are unevenly distributed over the walls of the chamber; they are located only some part of the walls. In order to solve this mystery, we first measured the shape of the chamber. Then, by examining the past sunshine conditions and vegetation, we simulated which areas would receive sunlight. The result indicated that the line engraving area overlaps with the sunlit area. This supports the hypothesis that the line-engraving was done only under sunlight.

A similar analysis, illumination synthesis, was performed in Sekizinsan tumulus, although the purpose was different. In Sekizinsan tumulus, the administrator stated that the moss on the sarcophagus has increased since the surrounding forests were logged in 2004. Using the 3D shape data of the tumulus and the surrounding topography, we show that the moss growth area and the sunlit area coincide, which support that the 2004 logging indeed promoted the growth of moss.

Archaeologically interesting results may be obtained by examining appearances from various viewing directions. In Segon-kou tumulus, concentric circles are carved on the upper part of the back wall and on the low front wall, which surround the place where the body was placed. However, the lower part of the back wall, which is a blind spot from the entrance of the chamber, does not have any line engraving. Using the shape data of the chamber and the line-engravings, views were generated from various locations. The results show that these concentric circles appeared to be evenly distributed when viewed from a certain location in the entrance of the chamber; it can be seen as evidence of ancient shoddy work.

5.2 Backward-Analysis Category

The main issue with the backward analysis is the high dimensionality of the spectral data. Spectral data is obtained by decomposing the input light using a number of filters. In the case of our LCTF, it employs 81 filters; each pixel value is represented as 81 dimensional data.

One approach to handle high-dimensional data is to select a specific dimension, a specific wavelength band, based on prior knowledge and analyze the data only in that band. Infrared bands are known to be less susceptible to surface contamination because of their ability to pass through obstacles. Images comprising only these bands often reveal new patterns hidden under contaminated and deteriorated surfaces, as in the case at Tashiro-Ohta. Although simple in terms of computer vision, infrared-band analysis is an effective method for archaeological analysis; this method is listed in the table as the single-band-selection approach.

The PCA (Principle component analysis) is a well known approach of dimensional reduction. The analysis of pigments in Kokadani tumulus employed this method. The current mural paintings in Kokadani tumulus look to have been painted in two colors: base rock color and pigment color. By analyzing the near-infrared bands, triangular repeated patterns can be detected from the wall; however, the repetition of the colors of the triangles did not make sense, if we interpret that the repetition consists of only two colors. Therefore, the spectral data were analyzed using PCA to determine the number of basis functions, which can be considered to correspond the number of colors used. Then, the PCA results were compared with the microscopic particle examinations. This analysis concluded that it comprises of three colors: the color of the base rock, the color of the red pigment, and the color of a mixture of the red pigment and mud.

The NC method compresses the space using non-linear compression and then employs the effective features for region extraction. A two-step, local PCA-based and global NC-based, method was developed for the analysis of mural paintings in Noriba tumulus analysis for finding new patterns.

The spider method employs a physics based model for the analysis. The PCA and the NC methods so far use general dimensional compression without considering any underlying physics model of pigment layers. Instead, the spider method uses the Lambert–Beer model for light attenuation along the layer. Spectral data naturally have different absorption and reflection coefficients for each band. By modeling these non-linear characteristics along the thickness of the pigment, we can estimate the thickness of the layer and then reconstruct the original patterns, as was the case for Benkei and Mezurashizuka tumuli.

In Benkei tumulus, we can clarify the pattern of a horse on a boat which were considered to help the person’s prosperous afterlife. It is also interesting to note that the extracted horse’s tail is flowing backwards, depicting a galloping, energetic horse. In Mezurasizuka tumulus, we focus on extracting the frog pattern, which is considered as a symbol of the moon in ancient China and the existence of the frog pattern in Benkei tumulus indicates that some Chinese culture had already spread to Kyushu area in AD 6.

5.3 Fusion Method

Fusion of photometric data with geometric data sometime provides another powerful means of analysis. In Sakura-kyo tumulus, there seemed to be gaps between the under-paintings by engraving and the actual over-paintings by the pigment. In order to accurately compare these two patterns, the geometry-based data need to be superimposed on the photometry-based data.

It was necessary to develop a new sensor to collect geometry data of line engravings without damaging tumulus paintings. Line engravings are very shallow, only approximately 0.5 mm deep and 0.5 mm width, and exist in a wide area. We developed a new sensor, referred to as a photometric wing, to meet the requirements of high resolution and high speed, while using weak light to avoid damaging the tumulus.

Using the RGB data and geometric data obtained from the photometric wing, we showed that there are inconceivable gaps between the two types of patters, which cannot be considered as the draft and the main paintings. However, it is unclear why such large gaps occurred everywhere. Further archaeological research is needed to unravel this mystery.

Table 1 Cyber-archaeology in Kyushu decorative tumuli project

6 Epilogue

Cyber-archaeology is an interdisciplinary and promising field. For example, as you see the researcher names in the author list, computer scientists, conservation scientists, and archaeologists participated this Kyushu project. This is common to our Rome Project (Sengoku-Haga et al. 2017), Somma-Vesuviana Project (Matsuda 2019), Bayon Project (Kamakura et al. 2005) and other’s large projects (Mallik et al. 2017; Dessales et al. 2020) as well. Although the concepts used in each field differ from each other, the main idea that cyber-archaeology is a new and promising direction was shared among the researchers and the coordination of concepts among the fields was achieved in a relatively short time of discussion. This direction also caused a lot of interest in archaeology community; for example, some collaborators were often invited to present our results at archaeological conferences such as Internationaler Museumstag am Leibniz-Rechenzentrum held in Munich Germany in 2014 (Kader et al. 2015), New Approaches to the Temple of Zeus at Olympia held in Budapest Hungary in 2015 (Sengoku-Haga et al. 2015) and XIX International Congress on Ancient Bronzes held in Los Angels USA in 2015 (Sengoku-Haga et al. 2017). Our chapter in the proceedings of the Budapest conference was highly reviewed in a book review (Ausonius 2016).

On the other hand, the biggest challenge to us was the theme setting. In order to conduct archaeologically meaningful research, the setting of the theme needs to be done under the leadership of archaeologists. On the other hand, the constraint for computer vision researchers is the necessity of “something new” to be developed toward the theme for justify their efforts; even if the theme is archaeologically meaningful, to use existing techniques cannot be justified as thesis topics in computer vision. Sometime, this makes it difficult to find an appropriate problem that make sense both to archaeology and computer vision. As a fundamental solution, perhaps we could consider an academic program that straddles both fields and trains archaeologists who also have basic knowledge of computer vision.

Related to this, in some cases even existing computer vision techniques may lead to new findings in archaeology by applying them. Thus, for effortless or minimum efforts of applying existing CV techniques, it is necessary to store the developed cyber-archaeology methods in the form of reusable libraries and APIs on GitHub and other cloud sites so as for archaeologists to be able to use such methods without the help of computer scientists. In fact, this is not only a cyber-archaeology issue, but will become a common issue when computer science cooperates with other fields. In that sense, we should think broadly about the field of cyber-humanities beyond cyber-archaeology, and consider methods and data storage systems for such interdisciplinary usages.

More broadly, in the middle ages, the sciences and the arts were developed in the same place, universities. In fact, Michelangelo was both a scientist and an artist. In the 20th century, these fields have been subdivided and developed in parallel in effective ways under the reductionist divide-and-conquer paradigm. We admit that this reductionism worked well for a while and have certainly achieved great success. However, on the other hand, these fragmented fields are incomprehensible to any one person, and no one has a holistic perspective overlooking all fields, and we scientists seem to be gradually losing our holistic humanity. In order to restore this situation, these disciplines may need to be reintegrated, reorganized and reunited under the holism to develop new art and science disciplines.

Third aspect is regarding the utilization of measurement data. At present, the ownership of e-Heritage data is unclear under the copyright laws in various countries; in the case of 2D photographs, it is clear that the photographer has the copyright, but in the case of e-Heritage data, in order to get measurement permission, we are often asked for the joint ownership of the copyright of the data obtained between the owner and us; it is often difficult to release data to the public at the request of the owner to prevent commercial and other malicious use. In particular, this problem becomes more prominent when measuring an object of faith such as a Buddhist statue. It is necessary to establish common practices and methods for making data widely available while satisfying the requests from the owners.

Heritage sites often exist in very harsh environments. For example, among the cultural assets we measured, Bayon temple was under the scorching sun (Ikeuchi et al. 2004; Banno et al. 2008), while Kyushu tumuli were underground in high humidity. Preah-Vihear temple was confronted by the military in a territorial dispute between two countries (Kamakura et al. 2019). Furthermore, Kyushu project required the wearing of protective clothing to prevent the entry of germs, as you see in Fig. 14. In addition, the measurement time needs to be shortened in order to reduce damage to the tumuli, and in order to keep the permitted measurement period, the measurement work may have to be continued even under adverse conditions; in fact, a typhoon hit Kyushu island, while we were measuring Segon-kou tumulus, caused a power outrage and all traffic stopped. The number of people to enter was limited due to the small size of the sites. It was sometime a little scary to be trapped in the underground tomb alone for a long time in the middle of a remote mountain with heavy rain. This placed heavy psychological and physical burdens on computer vision researchers. Automation and remote measurements are strongly required, and this will be a challenge in the future.

Fig. 14

Measurement under severe conditions inside of a tumulus

Finally, Kyushu decorative tumuli project, which took more than ten years, was made possible by the efforts of many people. In particular, besides the authors, the following students and staff members participated: Yoshifumi Ikari, Katsumi Ikegami, Kenji Inose, Rei Kawakami, Yoshie Kobayashi, Tomoaki Masuda, Daisuke Miyazaki, Kiminori Hasegawa, Masataka Kagesawa, Kiminori Hasegawa, Hiroki Unten, Akina Wada and Keiichi Watanabe of the University of Tokyo and Makoto Ando, Makio Honda, Ryuichi Kamo, Tetsuya Komuro, Toru Mihashi, Kaichiro Nakayama, Keiji Ogawa, Tomoaki Saito, Taro Terashi and Tsutomu Yamashoji of Toppan Printing.

We also would like to thank to the board of education of Fuggope town, Hinooka town, Keisen town, Kumamoto city, Munakata city, Tosu city, Ukiha city, Yame city, Yamaga city as well as Kyushu National Museum and Kumamoto Prefectual Burial Mound Museum for their permissions, arrangements and corporation of our digitization efforts.