JP3346799B2 - Sign language interpreter - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a sign language interpreting apparatus and a sign language interpreting system for recognizing sign language, converting the sign language into an expression such as text, voice, or another type of sign language, and transmitting the information. Things.

[0002]

2. Description of the Related Art As conventional related arts, (Publication Example 1) Japanese Patent Application Laid-Open No. 2-144675, "Hand Motion Recognition Device and Sign Language Conversion Device", and (Publication Example 2) Japanese Patent Application Laid-Open Publication No. 3-186979, "Neurocomputer" A hand posture recognition method that uses an image "has been proposed. In (known example 1), the positional relationship of the finger is obtained by color recognition using colored gloves. Then, the input finger character is recognized by comparing the finger position relationship of several finger characters registered in advance with the finger position relationship. In (known example 2), the correspondence between the static shape of the finger and the meaning of the finger character is learned using a known data glove using a neural network.
At the time of recognition, the static shape data of the finger from the data glove is input to the neural network, and the output is the meaning of the finger character.

[0003]

The above prior art is basically for recognizing a static finger character. Therefore, it is necessary to recognize a finger or hand movement (sign language pattern) for the following reasons. There is a problem that certain natural language sign language conversation cannot be recognized. In the following, a transmission method using a static one-handed finger shape will be referred to as a finger character, and a method using dynamic two-handed fingers or hand movements will be referred to as sign language, and each will be described separately. (1) In (known example 1), the elongation, bending, indefiniteness, and the like of a finger as a finger character are symbolized, but the temporal movement of the finger or hand in sign language has a large degree of freedom, so that the temporal movement is symbolized. Is generally difficult and in principle difficult to apply to sign language recognition. In (2) (known example 2), neuro learning is used.
When trying to apply it to dynamic sign language, it faces a difficult problem of sign language word cutout and temporal fluctuation of the sign language word, and cannot be applied to sign language recognition in principle. Also, (known example 1)
There is also a description for the purpose of collating words in sign language, but as described there, it is necessary to insert a pause between words, which results in a burden on the user and results in a natural sign language. It hinders conversation.

[0004] Further, other issues of the sign language interpreter itself include the following. (3) In sign language, there are individual differences in the finger shape, hand position, and movement. Specific speaker recognition is simpler than indefinite speaker recognition as knowledge from speech recognition. In the case of finger characters, since the number of types is as small as less than 50 words, it is conceivable that the user registers personal finger characters and learns weight coefficients of personal neurons before use. However, in the case of sign language, such registration and learning for personal application are temporally impossible because the number of basic words is as large as at least 1000 words. (4) Generally, it is often desired to refer to or record past conversations performed in sign language. These functions have not been realized yet. (5) Sign language has few words that convey emotions. Therefore, it is expressed by the size of facial expressions and gestures.
However, in general, a healthy person is often unable to read the facial expression and the size of the gesture because nerves are concentrated in sign language. Therefore, in order to realize natural conversation, a sign language interpreting device is indispensable for a function of transmitting emotions. An object of the present invention is to solve the above-described problems and other problems of the prior art, recognize a dynamic sign language having a large number of words and having individual differences, and communicate with a partner using various forms of expression. It is an object of the present invention to provide a sign language interpreting apparatus and a sign language interpreting system capable of transmitting information.

[0005]

In order to solve the above-mentioned problems and problems, in the present invention, means for obtaining finger and hand movements as time series data, and inputting the finger and hand movement time series data A conversion unit for calibrating the input sign language time series data as sign language time series data; a sign language word dictionary for storing the sign language time series data of each sign language word as sign language word dictionary data; an output of the conversion unit and sign language word dictionary data And a sign language recognition unit having a matching unit for recognizing and outputting a sign language word corresponding to the input sign language time-series data, and adding a particle or the like based on the rule norm to the sign language word output from the sign language recognition unit to generate a natural sentence Is output. Further, the user inputs the sign language time-series data of the specific sign language word and the sign language word dictionary data corresponding to the sign language time-series data, obtains the correspondence between the input data at each time, and outputs the correspondence. A second matching unit, and a selecting unit that selects one of the input sign language time-series data and the output of the second matching unit and outputs the selected sign language time-series data to the conversion unit; Means for learning the recognition parameters for the calibration based on the output of the collating unit is provided. Further, a neural network is used as means for learning the recognition parameters for the calibration. Further, a facial expression recognition unit and a processing device for inputting a facial image of a sign language user, recognizing facial expressions from the facial image, and obtaining an emotion type such as "joy" or "sadness" and an emotion level (intensity) are provided. The processing device outputs a natural sentence to which an emotional adjective is added by using a natural sentence output from the natural sentence conversion unit and an emotion type and an emotion degree (strength) output from the expression recognition unit as inputs. I have to. In addition, a sign language interpreter is installed at each of a plurality of stations in the local area network so that information can be exchanged between the plurality of sign language interpreters.

[0006]

In the present invention, time-series sign language time-series data input from a data glove or the like and each sign language word dictionary data are scanned and collated. Then, the sign language word dictionary data can be collated at the part having the highest degree of coincidence (the dotted line part in FIG. 4). It is also possible to learn the difference between some limited number of input sign language word data and the corresponding sign language word dictionary data, that is, the general habit of user's sign language as a data conversion rule (calibration rule). it can. By using this conversion rule, the sign language of the user can be converted into data closer to sign language word dictionary data, so that recognition accuracy can be improved. Furthermore, the provision of the facial expression recognition section enables the interpretation of emotional expressions, thereby realizing a more natural conversation.

[0007]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present invention will be described below in detail with reference to FIGS. First, in FIG.
The overall configuration of 00, the function of each element, and the transition of data will be described. This device is roughly divided into eight parts,
Sign language recognition part (1, 2, 3), facial expression recognition part (4,
5), data input / output part (6, 7), voice recognition part (9, 10), keyboard part (8), display part (1
3, 14, 15), a voice generating portion (11, 12), and a processing device (computer) (16). First, the operation of the sign language interpreter will be described through an example of a conversation between a sign language user and another user. In order to translate the sign language performed by the sign language user, time-series information d1 on the finger shape, hand direction and position of the sign language user is sent from the data glove 1 to the sign language recognition unit 2. The sign language recognizing unit 2 dynamically matches the time series information d1 with the sign language word dictionary data to symbolize the words included in the time series information d1 (d
2) Do it. The natural sentence conversion unit 3 creates and outputs a natural sentence (d3) by supplementing an appropriate particle or the like between the sequentially input symbolized words d2. On the other hand, the face image (d4) of the sign language user is simultaneously captured by the TV camera 4,
The facial expression recognition unit recognizes his facial expression (smile, sadness, etc.) and outputs the degree (d5) (smile, sadness, etc.).

[0008] When using voice as an interpreter output medium,
These data d3 and d5 are sent to the computer 16, converted into natural sentences to which a simple emotional adjective is added or a parameter for speech synthesis (d12), and then sent to the speech synthesis unit 12. In the voice synthesizing unit 12, the data d1
2, a voice with an emotion (d5) corresponding to the natural sentence (d3) is synthesized. When text is used as an interpreter output medium, the data d3 and d5 are sent to the computer 16 and a natural sentence (d13) to which a simple emotional adjective is added.
After the conversion into 2), the text is sent to the monitor 13 and displayed as text. When another type of sign language is used as the interpreting output medium, the computer 16 extracts the word part (d14) from the natural sentence (d3) to which a simple emotional adjective is added, and then outputs the sign language CG generating unit. 14 and the corresponding sign language CG in word units is output to the monitor 13.

[0009] The other user who understands the sign language user's conversation through these interpreting output media can use the keyboard 8, the microphone 9, or the data glove 1 (another sign language recognition unit may be prepared). Conversation using In the case of a text conversation, a conversation sentence (d
8) and display it on the monitor 13. Alternatively, by using the keyboard 8, a simple sentence (d
8), the sign language CG generating unit 14 may convert the sign language CG into the corresponding sign language CG and output it to the monitor 13. In the case of a voice conversation using the microphone 9, voice data uttered for each word is recognized by the voice recognition unit 10 and converted into a symbol (d9).
I do. This is displayed on the monitor 13 as a text, or the sign language CG generation unit 14 outputs the corresponding sign language CG for each word to the monitor 13. In the case of sign language conversation using the data glove 1, the sign language conversation is output as voice, text, and sign language CG in the same procedure as described above.

The computer 16 controls the entire processing unit and performs simple data conversion. Note that the speech recognition unit 10 and the speech synthesis unit 12 can be easily achieved by using a technology that has already been developed (for example, “Hearing and Speech”, supervised by IEICE, Toshitaka Miura 1980). Hereinafter, the sign language recognizing units (1, 2, 3), the facial expression recognizing units (4, 5), and the data input / output units (6, 7) directly related to the present invention in FIG. 1 will be sequentially described in detail.

FIG. 2 shows a specific configuration of the sign language recognition section 2. There are the following two functions of the recognition unit. Function (1): Acquire user's sign language word data corresponding to some predetermined sign language words from the data globe, compare the relationship between input sign language word data and sign language word dictionary data for the same word, and Perform the conversion. This is intended to calibrate individual differences in sign language, that is, to convert user sign language data into data more similar to sign language word dictionary data. This calibration is executed only at the start of use of the apparatus. Function (2): dynamically collates the time series data of the user's sign language conversation (after calibration) with the sign language word dictionary data, and detects words included in the time series data.

First, the implementation of the function (1) will be described. The sign language user inputs the predetermined sign language words d1 to the collation unit 1 one by one. Then, the sign language word dictionary data d22 corresponding thereto is read from the sign language word dictionary storage unit 22 and input to the matching unit 1, and the matching unit 1 performs end point fixed dynamic matching. Generally, a sign language word is different from a static finger character in that a word is represented by movement of a finger or a hand. Therefore, the data obtained from the data glove is represented by f11 to f52 (for example, f11 is the joint angle of the first joint of the thumb, f12 is the joint angle of the second joint of the thumb) as shown in FIG. Hereafter, f21 to f52 represent the joint angles of the first and second joints of the other fingers), hd1 to hd3 representing the direction of the palm, and h representing the position of the hand.
time series multidimensional function P (t) from px to hpz (many 1
Dimensional function). The sign language word dictionary data expresses one word, and can be similarly expressed as a time-series multidimensional function p (t). Although the figure shows time-series data for one hand for simplicity, it is actually a function having twice as many dimensions as using two-hand data.

As shown in FIG. 4, naturally,
Since there is a difference between the input sign language word data and the sign language word dictionary data due to individual differences, it is necessary to convert the input sign language word data into the sign language word dictionary data in order to absorb the difference. For this reason, first, an end point fixed dynamic collation is performed, and an optimal corresponding portion between them is obtained. In the end-point fixed dynamic collation, the start point of the input data and the start point of the dictionary data are fixed, and the end point of the input data and the end point of the dictionary data are fixed. This is a kind of dynamic template matching that allows variation, and uses a known method (“Pattern Recognition and Learning Algorithm” Bunichi Sogo Publishing Yoshinori Uesaka / Kazuhiko Ozeki, p91).

This collation is performed for every several words, and the information of the corresponding locations is output as d24 in FIG. d24 is
As shown in FIG. 4, the data at the corresponding locations of the input sign language word data and the sign language word dictionary data, that is, P (A), P
(B), P (C), P (D), P (E), P (F), P
(G) ... and p (a), p (b), p (c), p
(D), p (e), p (f), p (g),. Reference numeral 25 in FIG. 2 denotes a selection unit. When the function 1 is executed, the selection unit 25 selects the corresponding part data d24 without selecting the data d1, and inputs the data d24 to the conversion unit 21.
25. The conversion unit 21 includes a layered neuron 211 and selection units 212 and 213, as shown in FIG. In the conversion unit 21, the selection unit 213 selects P (A), P (B), P (C), P
(D), P (E), P (F), P (G)... Are selected and input to the layered neuro 211, and the selection unit 212 is selected.
P (a), p (b), p (c), p in d25
Learning of the layered neuro 211 is performed by using (d), p (e), p (f), p (g), etc. as the teacher data d212 of the layered neuro 211. The layered neuron 211 is obtained by using some sample data d25.
, The conversion rule from the time series data P (t) of the sign language user to the word dictionary data p (t) becomes
The learning is performed in the form of the weight coefficient of the layered neuron 211.

Next, the implementation of the function (2) will be described. When executing the function (2), the selection unit 25 shown in FIG.
In FIG. 3, the data d1 is selected and becomes d25. The selection unit 213 in FIG. 3 selects d25 and uses it as the input of the layered neuron 211. The selection unit 212 selects d212 and d2.
It becomes 1. The input sign language time series data d1 of the sign language user is
Unlike the case of the calibration described above, several sign language words intended for sign language conversation are arranged in chronological order. The input sign language time series data d1 is converted by the learned layered neuro, and becomes converted input sign language time series data d21 excluding the individual differences of users. That is, the converted input sign language time series data d21 is converted into time series data similar to some corresponding sign language word dictionary data by the function (1) described above.

The collating unit 2 (23) of FIG. 2 collates the converted input sign language time series data d21 with each sign language word dictionary data to detect a series of sign language words in the input sign language time series data. Things. The operation of the matching unit 2 (23) will be described with reference to FIG. In FIG. 5, d21 is input sign language time series data, and d22 is sign language word dictionary data. The input sign language time-series data is a function of the multidimensional time t, as described above. The sign language word dictionary data d22 includes, for example, A, B, C,... As shown in the figure, and each is dynamically matched with the input sign language time-series data d21. For this dynamic matching, a method called continuous DP matching (Oka, “Consecutive Word Recognition Using Continuous DP”, Acoustic Society of Japan, S78-20, pp. 145-1)
52, 1978). That is, this dynamic collation is basically performed by scanning each sign language word dictionary data on the input sign language time-series data, performing matching at each time, and searching for a position (time) having the best matching degree. It recognizes that a sign language word exists. The matching can be performed while allowing a slight temporal variation. The matching unit 2 (23) outputs the recognized words in time series as d2. The natural conversion unit 3 adds particles and the like according to the rule norm based on these data, and symbolizes and outputs a sentence closer to a natural sentence.

Next, the expression recognition section 5 will be described in detail with reference to FIGS. FIG. 6 shows a specific configuration of the facial expression recognition unit 5. The input d4 is a face image of a sign language user, and the output d5 is an emotion type obtained from the expression and an emotion degree obtained by quantifying the emotion. (60% joy, 10% sadness, etc.). At 41 in the figure, “eyes” and “mouth” of the face image
“Nose” and “eyebrows” position detection (d410) and its “eye” partial image (d414), “mouth” partial image (d413),
"Nose" partial image (d412), "brow" partial image (d41)
Cut out in 1). Expression matching section (42)
4, 423, 422, 421), while referring to the basic expression pattern dictionary (434, 433, 432, 431), the degree of emotion (d424, d423, d422, d422,
d421) is detected. The comprehensive judgment unit 44 detects these emotion degrees and the positions of “eyes”, “mouth”, “nose”, and “eyebrows” (d41).
0) is comprehensively determined, and the final emotion type and the emotion level are output.

FIG. 7 shows a specific structure for detecting and cutting out the position of each part. First, a face image d411 without a background is extracted by a face extraction unit 411 based on a difference.
Get 0. To realize this, a background image 4 registered in advance is used.
This can be easily achieved by obtaining the difference between the input image 12 and the input image d4 and obtaining a portion having a large absolute difference value. Extraction of each part (4
In 13), the whole position and the whole size of the face image d4110 are detected by using an image processing method such as a projection distribution.
Standard image patterns and basic position information (414) are prepared for each part of "eyes", "mouth", "nose", and "eyebrows".
Template matching is performed on the face image d4110 having no background using the standard image pattern (414) of each part as a template, and the position of each part is detected. At this time, the range in which template matching is performed can be limited by normalizing the basic position information (414) of each unit using the entire position and the entire size of the obtained face image, thereby improving accuracy and efficiency. Matching can be realized. Once
If the position of each part can be detected, it becomes easy to cut out each partial image. The output d410 is the position coordinates of each part, d
414, d413, d412, and d411 are the “eye” partial image, the “mouth” partial image, the “nose” partial image, and the “eyebrow” partial image.

FIG. 8 shows a face expression collation 42 of the mouth of the expression recognition section 5.
3 shows a specific configuration example. First, "mouth"
The partial image d413 is a feature extracting unit,
Shape features such as area, aspect ratio, and xy projection distribution shape are extracted. These topographical features are compared with the pre-registered ones of the happy “mouth” and the sadness of the “mouth” (comparison by distance in the feature space). It is determined. Then, an emotion degree inversely proportional to the closer emotion type and distance is output (d423). Expression matching of the other parts ("eyes", "nose", "eyebrows") can be realized by the completely same configuration. The overall determination unit 44 of the expression matching unit 5 uses the emotion type and emotion degree obtained from the shape characteristics of each part of the input image and the emotion type and emotion degree obtained from the positional coordinate relationship of each part to determine the final emotion. The type is decided by majority vote. Then, among the obtained emotion degrees, the average of the emotion degrees of the same emotion type as the final emotion type is set as the final emotion degree (d5). In addition, the determination of the emotion type and the emotion degree obtained from the positional coordinate relationship of each part is performed by storing the position coordinates of each part of the standard image of each expression in advance, and summing the positional deviation error of each part of the input image and the standard image. Is determined, and a method of determining the magnitude of the error is used. That is, the emotion type and emotion degree of the input image that are inversely proportional to the emotion type and emotion degree of the standard image having the positional relationship with the least error are set as the emotion type and emotion degree of each input image.

Next, the data input / output unit (6,
7) will be described. This part includes (1) storing the conversational raw data from the sign language user's data glove or the sentence after natural sentence conversion in the floppy desk 6, or
The stored data is read out from the floppy disk 6. (2) The layered neuro parameter (weight coefficient) of the conversion unit 21 of the sign language recognition unit 2 is stored in the floppy disk 6, or the stored data is stored. This is for reading from the floppy desk 6. Read out (1)
The data of is a speech synthesizer, monitor, or sign language CG
It is sent to the generator and its contents are displayed. Further, the read data (2) is set as a parameter (weight coefficient) of the layered neuron of the conversion unit 21.

FIG. 9 shows the structure of the sign language CG generator 14. 9 is word information, and the address (d1) of the word CG pattern corresponding to the word information is indicated by d14.
42) is generated by the address generator (142), and the designated word CG pattern (d131) is sent to the monitor 13,
indicate. The parts of the sign language interpreter have been described above in detail. With such a configuration, it is possible to realize a sign language interpreter that can solve all of the problems described above. In the above-described embodiment of the sign language interpreting apparatus 100, the description has been made on the assumption that the data exchange is performed by wire. However, all or some of the data exchanges of each unit are performed wirelessly, and the operability is improved. May be improved.

FIG. 10 shows the sign language interpreter 10 described above.
1 illustrates a sign language interpreting system in which a plurality of sign language interpreters 0 are connected via a LAN (local area network) interface 101 via a LAN or a wireless LAN. By adopting such a configuration, it is possible to access information communication and databases in sign facilities and gestures in public facilities such as hospitals, police stations, and government offices, or between facilities.

[0023]

According to the present invention, dynamic sign language having individual differences can be recognized, and natural sentence conversion can be performed based on the recognition result. In addition, facial expressions can be recognized, and natural sentences with emotions can be generated. Furthermore, conversation between a sign language user and a healthy person can be facilitated.

[Brief description of the drawings]

FIG. 1 is a diagram showing an entire configuration of a sign language interpreting apparatus.

FIG. 2 is a diagram illustrating a configuration of a sign language recognition unit.

FIG. 3 is a diagram showing a configuration of a conversion unit in the sign language recognition unit.

FIG. 4 is a diagram illustrating an operation of a collation unit 1 in a sign language recognition unit.

FIG. 5 is a diagram illustrating an operation of a collation unit 2 in the sign language recognition unit.

FIG. 6 is a diagram illustrating a configuration of a facial expression recognition unit.

FIG. 7 is a diagram illustrating the position detection of each part in the facial expression recognition unit and the configuration of a cutout unit;

FIG. 8 is a diagram showing a configuration of a mouth expression matching unit in the expression recognition unit.

FIG. 9 is a diagram illustrating a configuration of a sign language CG generation unit.

FIG. 10 is a diagram showing a configuration example of a sign language interpreting system.

[Explanation of symbols]

 Reference Signs List 1 data glove 2 sign language recognition unit 3 natural sentence conversion unit 4 TV camera 5 expression recognition unit 6 floppy disk 7 I / O device unit 8 keyboard 9 microphone 10 voice recognition unit 11 speaker 12 voice synthesis unit 13 monitor 14 sign language CG generation unit 15 Selection unit 16 Calculator

──────────────────────────────────────────────────続 き Continuing from the front page (72) Inventor Akira Ichikawa 1-280 Higashi Koikekubo, Kokubunji-shi, Tokyo Hitachi Central Research Laboratory, Inc. (72) Inventor Kiyoshi Inoue 1-280 Higashi Koikekubo, Kokubunji-shi, Tokyo Hitachi, Ltd. Central Research Laboratory (72) Inventor Kiyoshi Arai 1-280 Higashi Koikekubo, Kokubunji-shi, Tokyo Hitachi, Ltd.Central Research Laboratories (72) Inventor Takanori Shimura 1-1280 Higashi-Koikekubo, Kokubunji-shi, Tokyo Hitachi Central Co., Ltd. ( 72) Inventor Yuji Toda 1-280 Higashi-Koigakubo, Kokubunji-shi, Tokyo Inside the Central Research Laboratory, Hitachi, Ltd. (56) References JP-A-2-144675 (JP, A) JP-A-3-186979 (JP, A) JP-A-4-134515 (JP, A) JP-A-63-172297 (JP, A) JP-A 59-132079 (JP, A) (58 ) investigated the field (Int.Cl. ^7, DB name) G09B 21/00 G06F 17/28 G06T 1/00

Claims (4)

(57) [Claims] 1. A means for obtaining finger and hand movements as time series data, and a conversion unit for setting the finger and hand movements time series data as input sign language time series data and calibrating the input sign language time series data. A sign language word dictionary that stores the sign language time series data of each sign language word as sign language word dictionary data, and compares the output of the conversion unit with the sign language word dictionary data to recognize a sign language word corresponding to the input sign language time series data. A sign language recognition unit having a collating unit for outputting, a natural sentence conversion unit for adding a particle or the like based on a rule norm to the sign language word output from the sign language recognition unit and outputting a natural sentence, and a face image of the sign language user. A facial expression recognition unit for inputting and recognizing a facial expression from the face image to obtain an emotion type such as "joy" or "sadness" and a degree of emotion (intensity); a natural sentence output from the natural sentence conversion unit and the facial expression recognition Emotion types are output and its emotion degree (intensity) sign language interpretation system, characterized in that it comprises a processing device for outputting a natural sentence added with emotional adjectives as input. 2. The sign language interpreting apparatus according to claim 1, wherein a speech output device having a speech synthesizing unit is connected to the processing device so as to output a speech corresponding to an emotion type and an emotion level (intensity). A sign language interpreter. 3. The sign language interpreter according to claim 1, wherein a text output device for outputting a text is connected to the processing device, and the text is output in accordance with an emotion type and an emotion level (intensity). A sign language interpreter. 4. The sign language interpreting apparatus according to claim 1, wherein a sign language graphics output device having a sign language CG generating unit is connected to the processing device, and the emotion type and the emotion level (intensity) are connected.
A sign language interpreter, wherein the sign language graphics are output according to the language.