Keywords

1 Introduction

One example of a setting in which sign language interpreting is needed is the medical setting. In the medical setting, important exchanges that are a matter of life and death sometimes take place, requiring accuracy of expression. However, in the medical setting, specialist terms are often used, making translation difficult. One reason for this is that there are no fixed sign language expressions for certain medical terms such as the names of diseases and tests. While books introducing medical sign language expressions do exist, they are not very satisfactory, for example, there is no unity of expression even within the same book and expressions that are not widely used are introduced.

We therefore created a medical sign language dictionary with the aim of standardizing and spreading medical sign language expressions. We began by gathering the medical terms necessary for treatment and examined sign language expressions corresponding to these terms. For difficult terms, we prepared explanations in sign language. Then using an optical motion capture system, we filmed the sign language for the terms and the explanations, we created 3D animations of the sign language. The sign language expressions for all the terms and explanations in the dictionary can be verified using the 3D animations. The use of moving images gives people who are learning sign language expressions a more accurate understanding of the sign language expressions than if they learned them using static images only. This dictionary has been designed so that it is easy for anyone to use, offering not only a keyword search function but also functions that enable users to search by category or from a list of terms.

2 Medical Terms and Explanations

To determine the terms included in the dictionary, we first gathered the terms considered necessary for treatment in hospital and used frequently in hospital. We then examined sign language expressions for the gathered terms, aiming to ensure that the sign language expressions would be easily understandable to anyone. We also added explanations in sign language for terms that might be difficult to understand.

2.1 Medical Terms

Table 1 shows an example of the Japanese terms gathered for inclusion in the dictionary. We gathered mainly terms necessary for treatment and in addition to key “parts of the body,” “names of organs and bones,” “names of diseases,” “symptoms,” “names of departments,” “names of tests and equipment” and “drug names,” we also provided terms believed to be frequently used in hospitals such as “doctor” and “reception.” We did not, however, include dentistry-related terms.

Table 1. Examples of gathered medical terms
Full size table

We gathered 1,113 Japanese terms. For these Japanese terms, we examined sign language expressions. We examined sign language expressions in cooperation with sign language interpreters, including medical providers, and persons who use sign language as their primary language. While being careful to ensure that the sign language expressions were easily understandable to anyone, we paid attention to the unity and consistency of expressions and determined sign language expressions through verification. Consequently, the number of repetitions and preservation of hand shape are also unified in the dictionary. As a rule, we provided one sign language expression per Japanese term, but, in view of factors such as individual differences in expression and ease of expression, there are some terms for which we provided more than one expression, enabling adaptation to the future popularity of the sign language expressions in the future. This resulted in 1,272 sign language terms in total, compared to 1,113 Japanese terms.

2.2 Medical Explanations

Since medical terms are not routinely used and are often specialist terms, there are many terms that lay people do not understand. Therefore, we also prepared explanations in sign language to explain the meanings of terms considered difficult to understand through sign language expressions alone and terms that generally tend to be misunderstood. This also removes the need to look up the meanings of terms in another dictionary. Furthermore, among those who have difficulty hearing, there are many people who are not good at Japanese. We, therefore, prepared the explanations in sign language rather than Japanese text to make it easily understandable to persons without good Japanese.

The terms we explained are terms that are difficult to understand because they are rarely heard, for example, the names of diseases such as “Crohn’s disease” and “collagen disease” and words like remission, and terms that are difficult to distinguish from each other such as “virus” and “bacteria.” There are 122 explanations compared to 141 terms because we explained related terms together. Again, we prepared the sign language for the explanations in cooperation with sign language interpreters, including medical providers, and persons who use sign language as their primary language.

3 Creation of 3D Animations of Sign Language

Since sign language is a language expressed through movement, it is preferable to check expression in moving images rather than static images such as pictures or photographs. We, therefore, prepared 3D animations of the sign language. Figure 1 shows the created sign language animation. Use of 3D animations enables the image rights of sign language signers to be taken into consideration.

Fig. 1.
figure 1

The screen of the animation

Full size image

To reproduce sign language movements in 3D animations, it is necessary to acquire 3D movements corresponding to the bone structure of the models. We, therefore, made recordings of the 3D movements via an optical motion capture system. Table 2 shows details during recordings. The sign language signer wore a total of 112 retro-reflective markers all over her body, including her face and hands. This enabled us to create highly accurate animations. In sign language, facial expressions are also important. Through the attachment of markers to the face, animations with facial expression were created. It is also possible to view the sign language from all directions (360-degree view) to acquire the 3D movements. The right side of Fig. 1 shows the screen of the animation when the sign language signer is viewed from the left. Even if it is difficult to understand the sign language based on an image taken from the front alone, it is still possible to view the sign language expression from another direction, allowing users to properly verify the movement in detail .

Table 2. Motion capture information
Full size table

We asked a person who uses sign language as her primary language to be the sign language data model to acquire the Japanese sign language. We have already confirmed that this resulted in expression in more natural movements.

4 Functions of the Dictionary

The dictionary features a “keyword search” function, a “category-based search” function, a “list of terms” and “sign language expression-based search” function. Besides searching for terms, it is also possible to search for explanations using the “medical term explanation” function. The top screen of the dictionary has search buttons for each of these functions. Clicking on the relevant button will cause the corresponding search screen to appear.

4.1 Keyword Search

In terms of the GUI, the “keyword search” function has a larger button than the other search methods on the basis that it is the most frequently used search method. Users can use the keyword search function to quickly find the term they are looking for. When users input any character they choose into the search box, the relevant term will be displayed. A “yomigana” search (search in kana indicating the pronunciation of kanji) and a partial match search are also possible. The keyword search screen also has panel with the 50 characters of the Japanese syllabary. Characters can be entered into the search box by means of mouse operation only, without using the keyboard buttons. This makes the dictionary easy to use even for people who are no good at operating computers.

4.2 Other Search Methods

The category-based search function enables users to search for applicable terms based on each category. The terms are put into categories which are more specific than the categories shown in Table 1 and are divided into categories besides “names of departments” or “drug names” such as “facility” or “reception” according to the situation in which they are used. The “term list” function displays all the terms recorded in the order of the Japanese syllabary. This makes it possible to search for terms simply by clicking, without the use of a keyboard.

The “Medical term explanation” function allows users to search for the meaning of sign language terms. The terms that have explanations are displayed in a list and users can search from the list.

4.3 Search by Symbols Describing Sign Language Expression

The words contained in the dictionary are described using the “NVSG element model [1, 2]” that was proposed by the authors. This description method was proposed to describe the complex morphological structure of sign language. It describes the morphological structure of each element of sign language such as hand shape and movement. Using this description method to describe the expression of each sign language term will make it possible to conduct searches based on an element such as the hand shape or movement and will make it possible to search for sign language based on sign language expression. Therefore, someone enable to look the term up in this dictionary even when users do not know how sign language translates into Japanese. This feature is currently being development.

5 Conclusion

We created a medical dictionary that allows users to view sign language in the form of 3D animation, thus enabling sign language interpreting to be used in a medical setting where its use is important. To record the dictionary, we examined sign language expressions of medical terms in cooperation with persons who use sign language as their primary language. To ensure that the examined sign language expressions will become widely used in the future, we also paid attention to the creation of terms. We also added explanations in sign language for terms considered difficult to understand. We created sign language animations for all the sign language expressions created and created 3D animations of the sign language. The use of 3D animation enables users to view the sign language from all directions (360-degree view).

To make the dictionary easy for anyone to use, we considered the GUI in the dictionary design and ensured that word searches can be performed without any keyboard operation. Besides keyword search, it is also possible to search based on category and search from a term list.

In the future, we plan to add a function for searching based on sign language expression. Sign language can only be looked up in a dictionary based on a Japanese translation. However, we are currently building a “reverse lookup” type function that enables anyone who sees sign language which they do not know the Japanese translation for to look up the meaning in the dictionary and to search based on the hand shape and movement of the sign language.