Digital Archiving of Takigi Noh Based on Reflectance Analysis

Abstract

We propose a real-time bidirectional texture function (BTF) and image-based lighting (IBL) rendering of the Takigi Noh based on reflectance analysis. Firstly, we measured a sample of the Noh costume by omnidirectional anisotropic reflectance measurement system called Optical Gyro Measuring Machine (OGM), and we modeled the BTF of the Noh costume based on multi-illuminated High Dynamic Range (HDR) image analysis and modeled Noh stage in 3D based on archival records. Secondly, we captured motion data of Noh player, and modeled Noh player wearing a costume. To achieve the real-time rendering, we modeled the Noh costume by mass spring damper model. Finally, we modeled animated ambient map based on the Improving Noise to achieve the real-time dynamic lighting by fire of the Takigi, and we calculated the optical reflection by the IBL and deformation of the Noh costume.

Keywords

1 Introduction

Recently, research on the digital archive [1, 2] by various digital technologies have been attracted attention. In Japan, research on the digital museum [3] have been conducted. In this work, we attempt to archive the Takigi Noh digitally. The Noh (traditional masked dance-drama) is one of the traditional arts in Japan. Among this, there is one called Takigi (firewood) Noh which is performed by burning firewood around the stage after sunset. In Takigi Noh, Noh costumes of gold brocades shines beautifully with torches set around the Noh stage that provide the illumination (see Fig. 1).

Fig. 1.

Takigi Noh [4]

The Noh costume is interlaced with gold threads and silk, and has fine 3D structure. Therefore, it has very complex anisotropic reflecting property. The hue of the Noh costume is changing according to the incident direction of light and the view direction. Therefore, we model the Noh costume under the fabric structure and anisotropic reflectance, then we set the torches around Noh stage, and represent the anisotropic reflectance of the Noh costume fabric under dynamic lighting with torches.

In previous work, we modeled and represented the anisotropic reflection of the Noh costume based on the reflectance analysis [5], and we proposed a real-time anisotropic rendering method of the Noh costume which is lit by the firewood around the Noh stage [6]. In this work, we propose a real-time bidirectional texture function (BTF) and image-based lighting (IBL) rendering of the Takigi Noh based on reflectance analysis. Firstly, we measured a sample of the Noh costume by omnidirectional anisotropic reflectance measurement system called Optical Gyro Measuring Machine (OGM), and we modeled the BTF of the Noh costume based on multi-illuminated High Dynamic Range (HDR) image analysis and modeled Noh stage in 3D based on archival records. Secondly, we captured motion data of Noh player, and modeled Noh player wearing a costume. To achieve the real-time rendering, we modeled the Noh costume by mass spring damper model. Finally, we modeled animated ambient map based on the Improving Noise to achieve the real-time dynamic lighting by fire of the Takigi, and we calculated the optical reflection by the IBL and deformation of the Noh costume.

2 Related Work

The anisotropic reflectance of object surface can be characterized from the bidirectional reflectance distribution function (BRDF) which is the ratio of the irradiance from an arbitrary lighting direction to the radiance from any viewing direction at any point. Also, the permeable object surface can be characterized from the bidirectional transmittance distribution function (BTDF) which is the ratio of the transmitted light from an arbitrary lighting direction to the radiance from any viewing direction at any point [7]. Recently, some approaches which applied the BTDF [8], the bidirectional scattering distribution function (BSDF) [9], and the radiative transfer equation (RTE) [10] for the fabric are proposed. To achieve the realistic rendering of Noh costume, it is necessary to consider these properties because Noh costume is interlaced with permeable threads and gold brocades. In this work, we consider the BTF at the present stage.

About the lighting effect, for the rendering under real-world multiple light sources, image based lighting (IBL) rendering [11], which uses actual captured images (of the environmental map) as environment illumination information, is considered. IBL rendering considers the incidence light from all directions. However, it uses a large amount of calculation thus rendering in real-time is difficult. To reduce the computational cost, environment map sampling method [12–16], environment map and BRDF product method [17–20], and the direct calculation method from environment map [21–26] are proposed. In these methods, ambient light is constant and the environment will not change. Moreover, in the easy-to-use approach of BRDF [26], focused sampling is obtained from the incident direction. With fewer necessary pre-calculation, real-time rendering of BRDF is possible. However, this method is not applicable to BTF real-time rendering. On the other hand, to represent the bonfire, there are two methods: physically-based method [27] and a method by procedural texture [28]. However, the computational cost for lighting is a problem. Furthermore, it is difficult to use the procedural texture directly, which is easy-to-use for the dynamic light. Therefore, we represent the flame by procedural texture, which is easy-to-use, as the dynamic ambient map. It is expected to represent dynamic flame and ambient light in the same calculation as direct calculation method from environment map.

3 Reflectance Analysis of Noh Costume

Silk is primarily used in the Takigi Noh costume fabrics. The costume is interlaced with gold threads as shown in Fig. 2. Gold brocades, and colorful patterns such as flowers, leaves and branches, are woven into vermillion or red base.

Fig. 2.

Noh costume with gold brocades

The fabrics are generally woven using warp and weft patterns. The gold brocade can be interwoven using the warp-weft pattern with the red base. In Noh costumes, the aforementioned pattern is done throughout the whole fabric. However, arrangements that do not resemble the pattern of the silk threads are also present. The Noh costume fabric is made up of warp-weft patterns in an up-and-down arrangement. The weave of the underlying fabric is primarily composed of plain fabric, twill, and satin. These three types of weave are the most commonly produced. Figure 3 shows the three types of weave and their corresponding pattern representation. White area denotes warp, while gray denotes weft. The minimal recurring pattern in each weave is bounded by the red frame, which will be referred to as weave structure.

Fig. 3.

Weave structure chart

In this work, we modeled the BTF of the Noh costume based on multi-illuminated High Dynamic Range (HDR) image analysis. We don’t consider the optical transmission for the costume in current work and use the surface model. Noh costume with gold brocade has 3D structures. Therefore, it is necessary to define the BRDF at each points of surface to portray the anisotropic reflectance faithfully.

The reflection characteristic at any point on the surface of the object, that is, the ratio of the reflected light from the incident light with respect to the incident direction in any viewing point can be described using BTF. Figure 4 shows the geometric representation of BTF (Fig. 5).

Fig. 4.

Geometric representation of BTF

Fig. 5.

Anisotropic BTF

BTF $f_{r} \left( {\theta_{i} , \varphi_{i} ,\theta_{r} , \varphi_{r} , u,v} \right)$ shown in Eq. 1 is defined in the spherical coordinate system as the ratio of the radiance $L_{r} \left( {\theta_{r} , \varphi_{r} ,u,v} \right)$ from the incident direction $L\left( {\theta_{i} , \varphi_{i} } \right)$ taken from the viewing direction $\left( {\theta_{r} , \varphi_{r} } \right)$ for texel $\left( {u,v} \right)$, which is any point on the object surface, to the irradiance $L_{i} \left( {\theta_{i} , \varphi_{i} ,u,v} \right)$ taken from the incident direction. The minimum recurring pattern, shown in Fig. 3, in a Noh costume fabric is considered as the observed weave structure. We measured Noh costume with the OGM and estimated the diffuse reflection component, specular reflection component, normal vector, tangent vector, and standard deviations $\sigma_{x}$ and $\sigma_{y}$ which are based on the Ashikhmin model [29].

4 Modeling of the Takigi Noh

4.1 Takigi Noh Stage

We modeled Noh stage in 3D based on archival records [30]. Figure 6 shows a restored 3D model of the Noh stage.

Fig. 6.

3D model of Noh stage

To portray the Takigi Noh, we disposed Takigi objects to around the Noh stage (see Fig. 7). We used the Perlin Fire [31] to generate fire animation.

Fig. 7.

Takigi model based on the Perlin Fire

Moreover, we calculate the IBL from animated ambient map of generated fire images directly. Figure 8 shows ambient map of the Noh stage.

Fig. 8.

Animated ambient map

4.2 Animation of the Noh Player

We captured a motion of the Noh player with the infrared tracking system, and measured 6 DOF data (3 directions of movement and 3 directions of rotation) of 19 markers. Figure 9 shows a part of motion capture data.

Fig. 9.

Motion capture data of a Noh player.

4.3 Noh Costume

From reflectance analysis, parameters of the BTF texture such as the diffuse reflection component $\rho_{d}$, specular reflection component $\rho_{s}$ normal vector $\varvec{n}$, tangent vector $\varvec{t}$, and standard deviations $\sigma_{x}$ and $\sigma_{y}$ are estimated. Figure 10 shows a part of the BTF textures of the Noh costume.

Fig. 10.

Part of the BTF textures of Noh costume. From left, diffuse map, normal map, specular map, $\sigma_{x}$ map, and $\sigma_{y}$ map.

Diffuse map represents the color, normal map represents the surface gradient, specular map represents the power of specular reflection, and $\sigma_{x}$ map and $\sigma_{y}$ map are represents the spread of anisotropic reflection. Then we created a 3D model of Noh costume and clothed player (see Fig. 11).

Fig. 11.

3D model of Noh costume and clothed player

5 Rendering

We calculate the anisotropic reflectance of the Noh costume with IBL at per-fragment. To do the per-fragment lighting, it is necessary to program with the shader language. There are several shader languages such as the High Level Shader Language (HLSL), OpenGL Shading Language (GLSL), and C for graphics (Cg). In this work, we used GLSL that is compatible with OpenGL.

Final color of each pixel on the display is determined with sum of the diffuse reflectance color $\varvec{ D}$ and the specular reflectance color $\varvec{ S}$.

5.1 Diffuse Lighting

The power of diffuse reflectance $P_{d}$ is determined with the dot product of the gradient of the object (normal vector) $\varvec{C}_{\varvec{n}}$ and the direction of the light $\varvec{L}$.

$$ P_{d} = \varvec{C}_{\varvec{n}} \cdot \varvec{L}, $$

(1)

where, $\varvec{C}_{\varvec{n}}$ is the 3D vector that converted from the color of the object that mapped the normal map, $\varvec{L}$ is direction to the 4 Takigi objects that placed around the Noh stage. Then, diffuse reflectance color $\varvec{D}$ is determined with product of the object color $C_{d}$ that mapped the diffuse map and the color of fire $C_{l}$.

$$ \varvec{D} = P_{d} C_{l} , $$

(2)

where, $C_{l}$ is average color of the center of the fire, and it is inversely proportional to the square of distance from fire.

5.2 Specular Lighting

We used the Ashikhmin model to the Noh costume and used the Blinn-Phong model to other object. In Ashikhmin model, the power of specular reflectance $P_{s}$ is calculated according to the following equation.

$$ P_{s} = \frac{{D\left( {\theta_{h} \varphi_{h} } \right)}}{{\cos \theta_{i} \cos \theta_{r} }}, $$

(3)

$$ D\left( {\theta_{h} \varphi_{h} } \right) = \rho_{s} exp\left( { - { \tan }^{2} \theta_{h} \left( {\frac{{{ \cos }^{2} \varphi_{h} }}{{\sigma_{x}^{2} }} + \frac{{{ \sin }^{2} \varphi_{h} }}{{\sigma_{y}^{2} }}} \right)} \right), $$

(4)

where, $D\left( {\theta_{h} ,\varphi_{h} } \right)$ is the small surface distribution function at each texel $\left( {u, v} \right)$. The binormal vector $\varvec{b} = \varvec{n} \times \varvec{t}$ in which $\left( {\theta_{n} ,\varphi_{n} } \right)$ is the direction of the normal vector $\varvec{n}$ of the thread, and $\left( {\theta_{t} ,\varphi_{t} } \right)$ i is the direction of the tangent vector $\varvec{t}$. The half vector $\varvec{H}$ taken from the local Cartesian coordinate system is $\varvec{h}$, and its direction $\left( {\theta_{h} ,\varphi_{h} } \right)$ is determined. $\theta_{h}$ is the angle between $\varvec{n}$ and $\varvec{h}$, $\varphi_{h}$ is the angle between the projected vector $\varvec{h}$ in the XY-plane and $\varvec{t}$. $\rho_{s}$ is the specular reflection component, $\sigma_{x}$ is the standard deviation in the X-direction, and $\sigma_{y}$ is the standard deviation in Y-direction.

$\rho_{s}$ is obtained from the maximum value of specular reflection component $\rho_{bd, s} \left( {\theta_{i} , \varphi_{i} ,0,0} \right)$ observed from BRDF data. $\varvec{t}$ is the direction of the maximum dispersion of points of $\rho_{bd, s} \left( {\theta_{i} , \varphi_{i} ,0,0} \right)$ projected on the XY-plane. Furthermore, $\sigma_{x}$ and $\sigma_{y}$ in Eqs. 3 and 4 are obtained.

In Blinn-Phong model, the power of specular reflectance $P_{s}$ is calculated according to the following equation.

$$ P_{s} = \left( {\varvec{H} \cdot \varvec{C}_{\varvec{n}} } \right)^{{\rho_{s} }} $$

(5)

Then, specular reflectance color $\varvec{S}$ is determined with product of the $P_{s}$ and the color of ambient map $\varvec{C}_{\varvec{a}}$ that is reflected from view vector to the object.

$$ \varvec{S} = P_{s} \varvec{C}_{\varvec{a}} $$

(6)

6 Results

Figure 12 shows rendering results of the Takigi Noh stage. It can see flickers of the bonfire and attenuation of the bonfire light.

Fig. 12.

Rendering results of the Takigi Noh stage

Figure 13 shows rendering results of the Blinn-Phong model and Ashikhmin model. Ashikhmin model can see hexagonal pattern clearly because we considered the specular lobe.

Fig. 13.

Comparison of shading model, (a) Blinn-Phong model, (b) Ashikhmin model

Figure 14 shows rendering results of Noh animation This frame rate is 60 Hz.

Fig. 14.

Results of Noh animation

7 Conclusion

We proposed a real-time BTF and IBL rendering of the Takigi Noh based on reflectance analysis. Firstly, we measured a sample of the Noh costume by omnidirectional anisotropic reflectance measurement system, and we modeled the BTF of the Noh costume based on multi-illuminated HDR image analysis and modeled Noh stage in 3D based on archival records. Secondly, we captured motion data of Noh player, and modeled Noh player wearing a costume. To achieve the real-time rendering, we modeled the Noh costume by mass spring damper model. Finally, we modeled animated ambient map based on the Improving Noise to achieve the real-time dynamic lighting by fire of the Takigi, and we calculated the optical reflection by the IBL and deformation of the Noh costume. As a result, we achieved a real-time BTF and IBL rendering in the dynamic lighting.

In future work, we consider the physically-based IBL lighting of Takigi and deformation of Noh costume, and the BSDF.