Keywords

1 Introduction

Sign language is a visual language for which there are no general methods of providing descriptions in text [1, 2]. When translating sign language, the system needs to understand its morphological structure. Using the NVSG element model that we propose, a system can express the morphological structure of sign language with ease [3]. We are currently developing a dictionary of the morphological elements of sign language using the NVSG element model. The NVSG element model defines the elements that comprise sign language using N for a hand shape, V for a hand movement, S for the sightline, and G for all non-manual markers other than the sightline. For this report, we discussed the details of how to express the non-manual S and G elements that were not yet determined. As a result, we are now able to express the detailed morphological structures of sign language at the individual word level. With this result, and by using the NVSG element model to express the structure of sign language words, we are now able to offer a new methodology of translating sign language.

2 NVSG Element Model

We defined the four elements of the NVSG element model: hand shape, movement, sight line and non-manual markers. Sign combines these four elements morphologically to form more than one word in some cases. In composing a sign, we need to analyze what kind of movement each element describes. The NVSG element model thus expresses each of these four elements independently. The N element describes items concerning hand shape, while the V element describes those concerning movement. Non-manual markers related to the sight line are written as S elements, and all other non-manual markers are written as G elements. The NVSG element model that we propose is able to express morphological elements in a combination of these four elements. This model is also able to form two or more morphological elements at the same time, thereby expressing multiple morphological structures simultaneously.

Below, we first offer a simple explanation of how we express the N and V elements. We then report on how we would write the non-manual S and G elements.

2.1 N Element and V Element

Of all the elements that comprise manual movements, those that concern hand shape (other than their general movements) are expressed as N elements. The dominant hand is expressed as Ns to mean the “strong” hand, and the non-dominant hand is expressed as Nw, or the “weak” hand. Internal values would describe the shape, position and direction of the hand. The description of the hand shape would consist of changes in the finger joint motion or the movement of a certain finger joint. We therefore assign each finger a number from 1 to 5.

Changes in finger joint motion largely break down to how a finger bends and how the fingers relate to one another. How a finger bends is expressed based on defining the closed hand as the original form and expressing the shape of bend with a specific code together with the finger number. We would express the relationship of the fingers by describing it for the fingers subject to the bend/change.

The general movements of the arms (rather than the local movements of the joints beyond the wrist) are expressed as V elements. We would write internal values using either Vs to define the strong arm or Vw to define the weak arm. Arm motion is complex, as sign language uses three-dimensional space. The parameters that serve as internal values are simplified classifications of sign movements to which we assigned codes. We classified the sign movements into four categories: Movement, Instruction, Presentation and Air Writing.

2.2 S Element

The S element would consist of non-manual markers descriptions that concern the eye line. The line of sight of the signer (speaker) may serve the role of pointing to something. Particularly in a sign language sentence, the eye line frequently replaces finger pointing (PT: Pointing) to refer to a person or pronoun using spatial expression.

The direction of sight may be the hand itself or where the hand moves, which are both N elements. The parameter in this case would take the value of N, Ns (strong hand) or Nw (weak hand). Take, for example, the sign for “Purpose”, which is made by hitting the presented weak hand (Nw) with the strong hand (Ns). Since the eyes look at the target weak hand (Nw), the sign is described as below.

$$ {\text{Example:}}\left\{ {mokuteki\left( {\text{target}} \right)} \right\} = \left[ {\text{Nw}} {\left( {{\text{g}}_{{\left( {\text{UM}} \right)}} }@_{\text{C2}} \right)} \left[ {{\text{Ns}}_{{\left( {\text{H1}} \right)}} {\text{Vs}}_{{\left( {{\text{MV}}\left( {\text{contact}} \right) \gg {\text{WH}}} \right))}} } \right] \right]{\text{S}}_{{\left( {\text{Ns}} \right)}} $$

There are cases where the eyes follow a hand in motion. In such cases, the parameter would take a V element target value. When the signer is “thinking of” something, his eyes will normally look up in most cases. In such cases, we assign an “r”. The sign for “Think” as described in dictionary form would not include an S element indication to describe the sightline. But to indicate the sentence “Think of < something/someone>”, we would need an S element description.

$$ {\text{Example:}}\left\{ {wo \, omou\left( {{\text{think of}} < {\text{something}}/{\text{someone}}>} \right)} \right\} = \left[ {{\text{N}}_{{\left( {\text{h1}} \right)}} {\text{V}}_{{\left( {\text{PT}}\gg{\text{temple}} \right)}} } \right]{\text{S}}\left( {\text{r}} \right) $$

A signer may refer to a person in a sentence by using his line of sight. When a sign refers to a particular person, the information on that person remains in effect until the next sign is presented. In such a case, the signer’s eyes spatially express the person rather than looking at either one of the hands. The parameter for such a case would be per1, per2 or per3.

2.3 G Element

Considering the conversational nature of sign language, where its words connect and form a chain, we define non-manual markers (NMM) such as the facial expressions, nodding and mouth shapes necessary to expressing context and meaning as G elements, or the grammar elements of sign language. We substitute values for G elements using sIGNDEX V.2 [4]. sIGNDEX V.2 defines 91 types of NMS codes in 14 categories. Since we define finger pointing (PT) as a V element, we would not use it to describe G elements. The sightline, as mentioned in the preceding section, would be written as an independent S element, so it would also be excluded from our parameters for G elements. Thus, the NVSG element model that we propose would use 12 categories of codes as defined in sIGNDEX V.2. Table 1 lists the sIGNDEX V.2 codes that our NVSG element model would use.

Table 1. sIGNDEX V.2 codes
Full size table

3 Sign Description Using the NVSG Element Model

The NVSG element model indicates the hands separately as either the strong or weak hand. However, to better describe their motional relationship with one another, their motional relationship would determine whether the description would be in the form of an N element or a V element. An example is given below.

$$ \begin{aligned} & {\text{Example:``I}}\,{\text{visit}}\,{\text{his}}\,{\text{house"}} = \left\{ {{\text{his}}\,{\text{house}}} \right\} + \left\{ {\text{visit}} \right\} \\ & \quad \quad \quad \quad \quad \quad \quad \quad \quad \,\,\,\,\, = \left[ {{\text{NN}}_{{\left( {{\text{CL:many}}\left( {{\text{H}}0} \right)} \right)}} \,{\text{V}}_{{\left( {{\text{PR}}@{\text{PT}}3} \right)}} } \right]\,{\text{S}}_{{\left( {{\text{PT}}3} \right)}} \\ & \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad + {\text{Ns}}_{{\left( {{\text{CL:one}}\left( {{\text{H}}1} \right)} \right)}} \,{\text{V}}_{{\left( {{\text{MV}}\left( {{\text{per}}1 > {\text{per}}3} \right)} \right)}} \rm{]} {\text{S}}_{{\left( {\text{Ns}} \right)}} \\ \end{aligned} $$

4 Conclusion

This paper has suggested methods of writing parameters for S and G element non-manual markers in the NVSG element model that describes sign language. By adding S and G element descriptions, we are now capable of expressing the detailed morphological structure of sign words. And with the capability to describe people and pronouns as expressed spatially using the sightline, the model can also describe sentence structures. We are currently using the sign language stored in our database to confirm the descriptive indications, and we will then refine and finalize the parameters. We will also increase our NVSG element model data and aim for efficient collaboration with the three-dimensional motion database.