JP6730636B2 - Information processing apparatus, control program, and control method - Google Patents

Description

The present invention relates to an information processing device, a control program, and a control method.

Sometimes, a spoken word is judged from the imaging of the mouth movements of a deaf person, and the voice of this word is output to a computer.

JP 60-3793 A JP 2000-308198 A JP, 2003-18278, A JP, 2002-197465, A JP, 2006-227425, A

Tsuyoshi Miyazaki, 1 person, "Proposal for detecting the first mouth shape in Japanese speech video", [online], January 13, 2012, Information Processing Society of Japan, [Search November 30, 2017], Internet <URL: http://ci.nii.ac.jp/naid/110009423605>

In image recognition, a word having a higher degree of accuracy may be used as a recognition result from a plurality of utterance word candidates obtained by recognition.

Some of the spoken words have little difference on the image, such as the lip shape, when spoken with other words, and among the candidates obtained by image recognition, the accuracy of the correct word is It may be less accurate than the word. In such a case, the computer outputs not the correct word but the sound of another word that is determined to have higher accuracy than the accuracy of the correct word, so that the person who hears the sound may not be able to speak.

In one aspect, the present invention aims to reliably convey to the listener the sound of the word corresponding to the lips in unvoiced speech.

Therefore, this information processing apparatus is similar to an image acquisition control unit that acquires a processing target image including the lip of the recognition target person, the acquired processing target image, and a plurality of reference images corresponding to a plurality of words. A similarity calculation unit that calculates a degree, a first determination unit that determines a pronunciation candidate word related to the processing target image based on the similarity, and a plurality of pronunciation candidate words of a plurality of pronunciation candidate words when there are a plurality of pronunciation candidate words. From the inside, a similar sound priority word that is pre-defined as a pseudo-appropriate word that can be heard as a correct word by the listener even if it is different from the word that should actually be uttered when it is pronounced is determined as the pronunciation word. 2 determination unit, and an output control unit that causes the output device to output the determined similar-sound priority word by voice.

According to one embodiment, the sound of the word corresponding to the lip in the unvoiced speech can be reliably transmitted to the listener.

It is a figure which illustrates the hardware constitutions of the unvoiced word authentication utterance device as an example of an embodiment. FIG. 1 is a diagram illustrating a functional configuration of a silent word authentication utterance device as an example of an embodiment. It is a figure which illustrates the similar sound group information in the unvoiced word authentication utterance device as an example of an embodiment. 4 is a flowchart for explaining a speech determination method in the unvoiced word authentication speech device as an example of the embodiment.

Hereinafter, embodiments of the information processing apparatus, the control program, and the control method will be described with reference to the drawings. However, the embodiments described below are merely examples, and there is no intention to exclude application of various modifications and techniques not explicitly described in the embodiments. That is, the present embodiment can be variously modified and implemented without departing from the spirit thereof. Further, each drawing is not intended to include only the constituent elements shown in the drawing, but may include other functions and the like.

(A) Configuration FIG. 1 is a diagram illustrating a hardware configuration of a silent word authentication utterance device 1 as an example of an embodiment, and FIG. 2 is a diagram illustrating a functional configuration thereof.

The silent speech authentication utterance device 1 illustrated in FIG. 1 is a computer (information processing device) configured as a wristwatch-type wearable device.

The silent word authentication utterance device 1 includes a main body 10 and a belt 20, and is configured to be worn by the user by winding the belt 20 around the user's arm and fixing it. The unvoiced word authentication utterance device 1 captures an image of the face (especially the lips) of the user who is the wearer, determines the word (word) uttered from this imaging, and outputs it as a voice (utterance, utterance).

The main body 10 includes a processor 11, a RAM (Random Access Memory) 12, a speaker 13, a nonvolatile memory 14, a communication interface (IF) 15 and a camera 16.

The camera 16 is arranged on the surface of the main body 10 of the silent-voice authentication utterance device 1, and is worn on the user's face (especially the mouth) while the belt 20 of the silent-voice authentication utterance device 1 is wound around the user's arm. An image capturing device that captures an image. The image (captured image) captured by the camera 16 is stored in a specific area of the RAM 12.

The speaker 13 outputs (speaks) a voice (synthetic voice, synthetic voice) instructed to be output by the processor 11 (voice synthesis output control unit 114) described later.

The communication IF 15 is a wireless communication interface for performing wireless communication with a server computer (not shown), and is, for example, a wireless LAN (Local Area Network) interface or a Bluetooth (registered trademark) interface.

The nonvolatile memory 14 stores an OS (Operating System) program, application programs, and various data.

The RAM 12 is used as a main storage device. The RAM 12 temporarily stores at least part of an OS program and an application program executed by the processor 11. The RAM 12 also stores various data necessary for the processing by the processor 11. The application program may include a control program executed by the processor 11 so as to realize the silent word authentication utterance function of the present embodiment by the silent word authentication utterance device 1.

Further, in the RAM 12, a pronunciation template image 122 and similar sound group information 121 (see FIG. 3), which will be described later, are stored (developed).

The processor 11 controls the entire silent speech authentication utterance device (computer) 1. The processor 11 may be a multiprocessor. The processor 11 is any one of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array). It may be one. Further, the processor 11 may be a combination of two or more types of elements among a CPU, MPU, DSP, ASIC, PLD, and FPGA.

Then, as the processor 11 executes the control program, as shown in FIG. 2, the functions of the image acquisition control unit 111, the similarity calculation unit 112, the pronunciation determination unit 113, and the voice synthesis output control unit 114 are realized. It

The silent word authentication utterance device (computer) 1 executes the silent word authentication utterance function of the present embodiment by executing a program (control program or the like) recorded in a computer-readable non-temporary recording medium, for example. Realize. The program describing the processing content to be executed by the computer 1 can be recorded in various recording media. For example, the program executed by the computer 1 may be stored in the non-volatile memory 14. The processor 11 loads at least a part of the program in the non-volatile memory 14 into the RAM 12 and executes the loaded program.

Further, the program to be executed by the computer 1 (processor 11) may be recorded in a non-transitory portable recording medium such as an optical disc, a memory device, a memory card. The program stored in the portable recording medium becomes executable after being installed in the non-volatile memory 14 under the control of the processor 11, for example. Further, the processor 11 can directly read and execute the program from the portable recording medium.

Next, the functional configuration of the silent word authentication utterance device 1 according to the present embodiment will be described with reference to FIG.

The image acquisition control unit 111 acquires an image of the face of the user. For example, the image acquisition control unit 111 acquires an image of the user's face by capturing an image of the user's face with the camera 16. At this time, it is desirable that the image acquisition control unit 111 captures an image of the mouth (lips) of the user who is speaking. Hereinafter, the image of the user's lip taken by the camera 16 may be referred to as a taken image or an imaged image.

The image acquisition control unit 111 may acquire the image of the user's face by reading the image of the user's face stored in the storage device such as the RAM 12, and the image capturing control unit 111 may store the image of the user's face in a camera other than the camera 16. An image of the user's face may be acquired by capturing the face. Furthermore, the image acquisition control unit 111 may acquire the image of the user's face by receiving the image data transmitted from the external device, and can be modified in various ways.

The unvoiced word authentication utterance device 1 outputs a voice corresponding to the word the user is trying to utter based on the image of the user's lips taken by the camera 16. That is, the image of the user's lips taken by the camera 16 is an image to be processed by the silent output device 1 for voice output. Hereinafter, the image of the user's lip acquired by the image acquisition control unit 111 may be referred to as a processing target image. The image acquisition control unit 111 performs control to acquire a processing target image. The camera 16 also captures an image to be processed.

The image acquisition control unit 111 stores the image captured by the camera 16 in a predetermined area of the RAM 12.

The similarity calculation unit 112 compares the processing target image with the pronunciation template image (reference image) 122.

The pronunciation template image 122 is, for example, an image of a person's lip when pronouncing each word (sound, single tone) analyzed as a different sound in Japanese, and is prepared in advance. For example, in addition to the Japanese syllabary in Japanese, by shooting the lips when pronouncing each word (sound) of voiced sound, semi-voiced sound, long sound, consonant sound, repellant sound, and Japanese sound, each Japanese sound (single sound) A pronunciation template image 122 is created. One pronunciation template image 122 corresponds to one word, that is, one sound (single sound).

A plurality of (100 or more) pronunciation template images 122, which are the same as the number of single notes in Japanese created in this way, are created in advance and stored in the non-volatile memory 14. The pronunciation template image 122 may be created by causing the user of the unvoiced word authentication utterance device 1 to pronounce each word in advance and shoot it.

The similarity calculation unit 112 calculates the similarity between the processing target image captured by the camera 16 and each of the plurality of pronunciation template images (reference images) 122 read from the nonvolatile memory 14.

Specifically, the similarity calculation unit 112 sequentially reads the pronunciation template image 122 from the non-volatile memory 14 and stores it in a specific area of the RAM 12.

Then, the similarity calculation unit 112 extracts the image of the lip portion (lip region) that is the processing target image from the captured image captured by the camera 16 stored in the RAM 12, reads it from the non-volatile memory 14, and stores it in the RAM 12. The generated pronunciation template image 122 is compared.

Note that extraction of the lip portion from the captured image can be realized using various known methods, and detailed description thereof will be omitted.

The similarity calculation unit 112 calculates the similarity by expressing the similarity between the shape of the lip in the processing target image and the shape of the lip in the pronunciation template image 122 with a numerical value (similarity). The degree of similarity represents, for example, the degree of matching.

The calculation of the degree of similarity is performed, for example, by comparing the shapes of the lips in the images. The similarity calculation unit 112 digitizes the similarity between the lip shape of the processing target image and the lip shape of the pronunciation template image 122 by comparing the images of the lips using a known image matching method, for example. As the image matching method, for example, a feature-based method or a region-based method may be used.

The calculation of the similarity between the shape of the lips in the processing target image and the shape of the lips in the pronunciation template image 122 can be realized using various known methods (for example, see Non-Patent Document 1).

The calculation of the similarity regarding the shape of the lip in the processing target image is performed for all the pronunciation template images 122. That is, the similarity calculation unit 112 calculates the similarity for each word of Japanese for the shape of the lip in the captured image.

The similarity calculation unit 112 outputs (calculates) the similarity as a percentage (% value), for example, as a result of the comparison between the processing target image and the pronunciation template image 122.

The pronunciation determining unit 113 determines a pronunciation (word) according to the shape of the lip in the processing target image, based on the similarity calculated by the similarity calculating unit 112. The word (sound) determined by the pronunciation determining unit 113 is output from the speaker 13 by the voice synthesis output control unit 114 described later.

The pronunciation determination unit 113 confirms the number of pronunciation template images (reference images) 122 whose similarity to the processing target image is equal to or higher than a predetermined threshold value (for example, 90%).

As a result of the confirmation, when only one pronunciation template image 122 whose similarity is equal to or higher than the threshold value is detected, the pronunciation determining unit 113 determines that the lip of the processing target image is the pronunciation template image 122 in which only one is detected. Judge that it corresponds to the corresponding sound (word). That is, the pronunciation determining unit 113 determines that the lips of the processing target image have the shape corresponding to the pronunciation template image 122 in the pronunciation, and determines the word as the pronunciation word.

On the other hand, as a result of the comparison, when a plurality of pronunciation template images 122 whose similarity is equal to or higher than the threshold value are detected, the pronunciation determination unit 113 refers to the similar sound group information 121 and performs similar sound conversion processing. The pronunciation word is determined according to the shape of the lips.

Here, when a plurality of pronunciation template images 122 whose similarity is equal to or more than a threshold value are detected, the lip shape of the processing target image is similar to any of these plurality of pronunciation template images 122 and it is difficult to distinguish them. It can be regarded as a state.

In such a case, the pronunciation determination unit 113 causes the speech synthesis output control unit 114, which will be described later, to output the word corresponding to the processing target image according to the similar sound processing rule defined as the similar sound group information 121. Determine the word.

Hereinafter, a case in which a plurality of pronunciation template images 122 having a degree of similarity equal to or greater than a threshold value are detected may be referred to as a case of detecting a plurality of candidates. Further, each word corresponding to the plurality of pronunciation template images 122 whose detected similarity is equal to or higher than the threshold value may be referred to as a pronunciation candidate word.

In addition, hereinafter, a sound that has a similar shape of the lip at the time of pronunciation and is difficult to identify by the appearance of the lip may be referred to as a similar sound.

In general, vowels can be easily identified from captured images of lips, but consonants are difficult to identify. The reason why it is difficult to identify consonants from a captured image lies in the mechanism of human speech. That is, the voice pushes air through the lungs, applies vibrations (frequency) to the air through the vocal cords, and makes sounds in the mouth (position of the tongue, how to open the teeth, how to open the mouth) and the shape of the lips. Therefore, it is considered difficult to distinguish all consonants only by the shape of the lips.

As described above, it is difficult to identify all consonants from a captured image, but there are also consonants that can be identified from the appearance such as vowels, mouth closing actions, and cheek and throat movements. In addition, even if all sounds included in a word are not correctly uttered, the listener may be able to correctly recognize the situation based on the situation of the person listening to the word, the context before and after the word, and the like.

If the applicant speaks a specific (specific side) word for a combination of words that has little difference in the appearance of the lips during pronunciation, the word may be different from the actual word to be spoken. However, he found that the listener could hear the words as if they were correct words.

If there are both combinations of words as candidates (pronunciation candidate words) recognized from the image, regardless of the accuracy of both, if you select the word on the predetermined side as the utterance, the result will be Whether it is true or false, it will be easier for the listener to understand.

For example, the case where “ha” and “a” are described as words having similar lip shapes when uttered.

To the listener, even if "a" is spoken in the place of "ha", it sounds like "ha". On the other hand, if you say "ha" at the "a" part, you will hear "ha" as it is. Therefore, if the pronunciation candidate word of the word recognized from the image is "ha" or "a", irrespective of the accuracy of the recognition, by uttering "a", it is possible to check whether the recognition result is correct or not. No, it sounds right to the listener.

In the following, in a combination of words that has little difference in the appearance of the lip shape during pronunciation, when the words are pronounced, even if they differ from the words that should actually be spoken, the listener should hear the words as if they were correct. A specific word that can do is sometimes referred to as a sound-like sound word. In the above example, in the case where the pronunciation candidate words are "a" and "ha", "a" is a similar sound word. Then, this similar sound word is one of a plurality of pronunciation candidate words included in the similar sound group, and is defined as a pronunciation word with priority over other pronunciation candidate words. It can be said that it is a (similar sound priority word).

A similar sound word is one of a combination of words with little difference in the appearance of the lip shape at the time of pronunciation, and for each of various combinations of words with little difference in the appearance of the lip shape at the time of pronunciation, It may be decided by conducting a pronunciation test in advance and confirming that the correct listening is possible.

In the unvoiced word authentication utterance device 1, by outputting (speaking) a similar-sound appropriate word for a similar sound that has a lip shape that is similar to a plurality of sounds at the time of pronunciation and is difficult to identify by the appearance of the lips. (Similar sound processing), to make the listener correctly recognize the words.

FIG. 3 is a diagram illustrating the similar sound group information 121 in the silent word authentication utterance device 1 as an example of the embodiment.

The similar sound group information 121 illustrated in FIG. 3 is configured by associating similar sound rules with a combination of words (sounds) having a high degree of similarity.

A combination of words having a high degree of similarity is a combination of words (sounds) that have similar lip shapes at the time of pronunciation and are difficult to identify. Hereinafter, such a combination of words having a high degree of similarity may be referred to as a similar sound group. The similar sound group includes a plurality of pronunciation candidate words.

A plurality of similar sound groups are registered in the similar sound group information 121, and 29 sets of similar sound groups are registered in the similar sound group information 121 illustrated in FIG. In the similar sound group information 121 illustrated in FIG. 3, the identification numbers (1 to 29) are set for each similar sound group.

For example, the similar sound group indicated by the identification number 1 includes the pronunciation candidate words “A”, “HA”, and “BA”. In the similar sound group indicated by this identification number 1, the shape of the lips when "a" is pronounced is the shape of the lips when "ha" is pronounced, and the shape of the lips when "ba" is pronounced. Similar, indicating that identification is difficult.

The pronunciation candidate words that form the same similar sound group have similar actual pronunciations. For example, the sounds of the pronunciation candidate words “a”, “ha”, and “ba” that form the same similar sound group are similar.

In the similar-sound rule, an appropriate similar-sound word is set as a word to be output from the speaker 13 by the voice synthesis output control unit 114 described later. The similar sound appropriate word set in this similar sound rule is a word (similar sound priority word) defined in advance from each pronunciation candidate word forming the corresponding similar group.

In the unvoiced word authentication utterance device 1, when a plurality of pronunciation candidate words are detected, a combination of these pronunciation candidate words is registered as a similar sound group in the similar sound group information 121, that is, a pronunciation candidate. When the word is a specific combination, the pronunciation determining unit 113 determines the pronunciation according to the similar rule. Specifically, the pronunciation determining unit 113 defines the similar-sound-appropriate word specified to be preferentially selected in advance as a pronunciation word among the pronunciation candidate words constituting the similar-sound group, which are defined in the similar-sound rule. (Similar sound priority word) is determined as an output word.

For example, when a vowel is included in a similar sound group, this vowel may be used as a similar sound priority word and set in a similar sound rule. When a similar sound group includes a dull sound and a clean sound, the clean sound may be used as a similar sound priority word and set in a similar sound rule. Furthermore, when a similar sound group includes a dakuon and a semi-voiced sound, the semi-voiced sound may be used as a similar sound priority word and set in the similar sound rule.

For example, in the similar sound group information 121 illustrated in FIG. 3, the similar sound group having the identification number “1” includes “A”, “ha”, and “ba”, and the similar sound groups include the similar sounds. As a rule, it is set to output the vowel "a" which is a similar sound priority word.

The pronunciation determination unit 113, based on the words (pronunciation candidate words) corresponding to the two pronunciation template images 122 that the similarity is determined to be higher than a predetermined threshold value (for example, 90%), the similar sounds of the similar sound group information 121. Look up a group and get the corresponding similar rule.

For example, as the pronunciation template image (reference image) 122 whose similarity to the image to be processed is a predetermined threshold value (for example, 90%) or more, the pronunciation template image 122 corresponding to “a” and the pronunciation template image corresponding to “ha” When the image 122 is detected, the similar sound group with the identification number “1” is selected in the similar sound group information 121. The pronunciation determining unit 113 determines to pronounce “a” based on the similar sound rule associated with the similar sound group having the identification number “1”.

That is, in the similar sound group indicated by the identification number 1, it is determined that the shape of the lips of the processing target image is similar to the shape of the lips at the time of pronunciation of any of "a", "ha", and "ba". In this case, the pronunciation determining unit 113 determines to pronounce “a”.

For example, when the user of the unvoiced word authentication utterance device 1 says the word “Good morning”, the pronunciation determination unit 113 causes the words “ha”, “yo”, “go”, It is assumed that a plurality of candidates are detected for each "za".

The pronunciation determining unit 113 refers to the similar sound group information 121 for each of the words “ha”, “yo”, “go”, and “za” in which these plural candidates are detected, and defines them as similar sound rules. The similar sounding priority word is determined as the pronunciation word.

Specifically, the pronunciation determining unit 113 refers to the similar sound group information 121 based on the word “ha” and determines to output “a” (in the similar sound group information 121 illustrated in FIG. (See identification number "1"). Similarly, the pronunciation determination unit 113 refers to the similar sound group information 121 based on the word "yo" and determines to output "Io" (identification of similar sound group information 121 illustrated in FIG. 3). See number "23"). Further, the pronunciation determining unit 113 refers to the similar sound group information 121 based on the word “go” and determines to output “ko” (the identification number “of the similar sound group information 121 illustrated in FIG. 10"). Further, the pronunciation determining unit 113 refers to the similar sound group information 121 based on the word "za" and determines to output "sa" (the identification number "of the similar sound group information 121 illustrated in FIG. 11").

As a result of these, the pronunciation determination unit 113 determines “Oioi Ukomasui” as the output word for the utterance “Good morning” of the user of the unvoiced speech authentication utterance device 1, and synthesizes the voice. The output control unit 114 is caused to output.

These "a", "io", "ko" and "sa" are like-sound proper words, and even if they are different from the words that should actually be spoken, the listener should hear them as if they were correct words. Is a word that can Therefore, the output word “Oi Ou Usashimasu” including these similar sound words is recognized by the listener as “Good morning” without any problem.

The voice synthesis output control unit 114 controls the speaker 13 to output a sound (voice) corresponding to the word determined by the pronunciation determination unit 113.

For example, the voice synthesis output control unit 114 selects the synthesized voice data corresponding to the word determined by the pronunciation determination unit 113 from the synthesized voice data prepared in advance and stored in the non-volatile memory 14 or the like. Then, the voice synthesis output control unit 114 outputs the synthesized voice data corresponding to the word determined by the pronunciation determination unit 113 from the speaker 13 by transmitting the selected synthesized voice data to a voice output circuit (not shown), for example. Let

(B) Operation A speech determination method in the unvoiced word authentication speech device 1 as an example of the embodiment configured as described above will be described according to the flowchart (steps S1 to S5) shown in FIG.

In step S1, the image acquisition control unit 111 causes the camera 16 to capture an image of the mouth of the user. The image (processing target image) captured by the camera 16 is stored in a predetermined area of the RAM 12.

In step S2, the similarity calculation unit 112 calculates the similarity between the processing target image and each pronunciation template image (reference image) 122.

In step S3, the pronunciation determining unit 113 compares the degree of similarity between the processing target image and each pronunciation template image (reference image) 122 with a predetermined threshold value. The pronunciation determination unit 113 confirms the number of pronunciation template images (reference images) 122 whose similarity to the processing target image is equal to or higher than a predetermined threshold value (for example, 90%).

If the result of the confirmation is that there is no similarity above the threshold (see “Route where similarity is above threshold” in step S3), the pronunciation determining unit 113 determines that the user is not speaking a word. Then, the process is terminated without outputting the voice (utterance) from the speaker 13.

In addition, as a result of the confirmation in step S3, when only one pronunciation template image 122 whose similarity is equal to or higher than the threshold value is detected (refer to the route “the similarity is 1 or higher than the threshold value” in step S3), the pronunciation determination unit. 113 determines that the lips of the processing target image correspond to the sound (word) corresponding to the pronunciation template image 122. That is, the pronunciation determining unit 113 determines that the lips of the processing target image have the shape corresponding to the pronunciation template image 122 in the pronunciation, and determines the word as the pronunciation word.

The voice synthesis output control unit 114 controls the speaker 13 to output a sound (voice) corresponding to the pronunciation word thus determined. As a result, the speaker 13 outputs a voice that matches the lip shape of the user wearing the unvoiced speech authentication utterance device 1.

As a result of the confirmation in step S3, when two or more pronunciation template images 122 whose similarity is equal to or greater than the threshold value are detected, that is, when a plurality of candidates are detected (the “similarity degree is equal to or more than threshold value is two or more” route in step S3). ), the process proceeds to step S4.

In step S4, the pronunciation determining unit 113 refers to the similar sound group information 121 based on each word (pronunciation candidate word) corresponding to the plurality of pronunciation template images 122 whose similarity is detected as a threshold value or more, and refers to these. Identify (determine) a group of similar sounds that includes pronunciation candidate words.

Note that it is desirable that the adopted similar sound group includes all of the plurality of pronunciation candidate words when the similar sound group is determined, but the present invention is not limited to this. For example, when there are three or more pronunciation candidate words, the similar sound group including the most pronunciation candidate words may be adopted.

In step S5, the pronunciation determination unit 113 refers to the similar sound group information 121, and according to the similar sound rule corresponding to the similar sound group determined in step S4, pronounces words (appropriate similar sounds, similar sound priority words). decide.

The pronunciation determining unit 113 notifies the voice synthesis output control unit 114 of the determined word, causes the speaker 13 to output the word, and ends the process.

(C) Effect As described above, in the silent word authentication utterance device 1 as an example of the embodiment, the image acquisition control unit 111 causes the camera 16 to capture an image of the mouth of the user, and the similarity calculation unit 112 causes the image capturing control unit 111 to capture the image. The degree of similarity between the processed image to be processed and the plurality of pronunciation template images 122 is calculated.

Then, the pronunciation determining unit 113 determines the word corresponding to the pronunciation template image 122 having the highest similarity as the word to be pronounced, based on the calculated similarity. As a result, the pronunciation determining unit 113 can easily determine the pronunciation word, and the load on the processor 11 can be reduced.

When a plurality of pronunciation candidate words are detected, the pronunciation determining unit 113 refers to the similar sound group information 121 and determines the utterance according to the similar sound rule, so that the pronunciation deciding unit 113 is close to the unvoiced utterance by the user. The pronunciation word can be easily determined, and the sound of the word corresponding to the lip in the unvoiced speech can be surely transmitted to the listener. Further, at this time, by reducing the number of determination processes, it is possible to reduce the load on the processor 11 and reduce erroneous recognition.

As a similar sound rule of the similar sound group information 121, even if each similar sound group is different from the word to be actually spoken, the listener can hear the sound like a correct word. Set the word. Then, when there are a plurality of pronunciation candidate words, the pronunciation determining unit 113 determines the utterance by referring to this similar-sound rule. As a result, a word corresponding to the user's lip cannot be specified as one, and even when there are a plurality of pronunciation candidate words, it is possible to output a natural utterance according to the shape of the user's lip. That is, smooth communication can be realized without causing a difference between the user's lip and speech.

As a similar rule, when a word is pronounced in a combination of words that has little difference in the appearance of the lip shape during pronunciation, even if the word is different from the word that should actually be spoken, it seems that the word is correct for the listener. Stipulates that the appropriate sound word that can be heard is output. Thereby, the content of the utterance can be transmitted to the listener regardless of whether the recognition result obtained from the processing target image is correct or incorrect.

In addition, since it is not necessary to input a long sentence such as a context to perform context interpretation and collation with a word database, the processing load can be reduced and the processing speed can be increased.

(D) Others The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit of the present invention.

For example, in the above-described embodiment, the similar sound group information 121 is not limited to that illustrated in FIG. 3 and can be implemented by appropriately changing it. For example, another combination of pronunciation candidate words may be registered as a similar sound group in the similar sound group information 121. Also, instead of preferentially selecting one word from the pronunciation candidate words that make up the corresponding similar group, as the similar sound appropriate word set in the similar sound rule, use another word that has a sound that is close to either one. May be.

Further, in the above-described embodiment, the functions as the similarity calculation unit 112 and the pronunciation determination unit 113 are realized by the processor 11 of the silent word authentication utterance device 1, but the present invention is not limited to this.

For example, it may be configured to be able to wirelessly communicate with a server computer (not shown) via the communication interface 15, and the server computer may be made to realize the functions of the similarity calculation unit 112 and the pronunciation determination unit 113. As a result, the load on the silent speech authentication utterance device 1 can be reduced.

Further, it is possible for those skilled in the art to implement and manufacture the present embodiment based on the above disclosure.

(E) Supplementary notes The following supplementary notes will be disclosed regarding the above-described embodiment.

(Appendix 1)
An image acquisition control unit that acquires a processing target image including the lip of the recognition target person,
A similarity calculation unit that calculates the similarity between the acquired target image and a plurality of reference images corresponding to a plurality of words,
A first determination unit that determines a pronunciation candidate word for the processing target image based on the similarity;
A second deciding unit which, when there are a plurality of pronunciation candidate words, decides a predefined similar-sound priority word as a pronunciation word from among the plurality of pronunciation candidate words;
An information processing apparatus, comprising: an output control unit that causes the output device to output the determined similar-sound priority word by voice.

(Appendix 2)
Reference in which the second determination unit presets one of the plurality of words as the similar-sound priority word with respect to a combination of a plurality of words whose similarity in lip shape at pronunciation is equal to or greater than a threshold value. The information processing apparatus according to appendix 1, wherein the similar-sound priority word is acquired with reference to information.

(Appendix 3)
In an information processing device including a processor,
Acquire the processing target image including the lip of the recognition target person,
Calculating the degree of similarity between the acquired image to be processed and a plurality of reference images corresponding to a plurality of words,
Determining a pronunciation candidate word for the image to be processed based on the similarity,
When there are a plurality of pronunciation candidate words, from among the plurality of pronunciation candidate words, a pre-defined similar sound priority word is determined as a pronunciation word,
A control program that causes the processor to execute a process of outputting the determined similar-sound priority word as an audio from an output device.

(Appendix 4)
For a combination of a plurality of words whose similarity of the shape of the lip at the time of pronunciation is equal to or more than a threshold value, one of the plurality of words is referred to by reference information preset as the similar sound priority word, A control program according to appendix 3, which causes the processor to execute a process of acquiring a similar-sound priority word.

(Appendix 5)
A process of acquiring a processing target image including the lip of the recognition target person,
A process of calculating the degree of similarity between the acquired target image and a plurality of reference images corresponding to a plurality of words,
A process of determining a pronunciation candidate word for the processing target image based on the similarity,
When there are a plurality of pronunciation candidate words, a process of determining a pre-defined similar sound priority word as a pronunciation word from among the plurality of pronunciation candidate words,
And a process of causing the output device to output the determined similar-sound priority word by voice.

(Appendix 6)
For a combination of a plurality of words whose similarity of the shape of the lip at the time of pronunciation is equal to or more than a threshold value, one of the plurality of words is referred to by reference information preset as the similar sound priority word, 6. The control method according to appendix 5, further comprising a process of acquiring a similar sound priority word.

1 Silent Word Authentication Utterance Device 10 Main Body 11 Processor 12 RAM
13 speaker 14 non-volatile memory 15 communication interface 16 camera 20 belt 111 image acquisition control unit 112 similarity calculation unit 113 pronunciation determination unit 114 voice synthesis output control unit 121 similar sound group information 122 pronunciation template image

Claims (4)

An image acquisition control unit that acquires a processing target image including the lip of the recognition target person,
The acquired image to be processed and a similarity calculation unit that respectively calculates the similarity between a plurality of reference images corresponding to a plurality of words,
A first determining unit that determines a pronunciation candidate word for the processing target image based on the similarity;
When there are multiple pronunciation candidate words, even if the pronunciation candidate word is different from the word that should actually be uttered when the pronunciation is pronounced, the listener can hear the word as a correct word. A second determination unit that determines a similar-sound priority word that is defined in advance as a word as a pronunciation word;
An information processing apparatus, comprising: an output control unit that causes the output device to output the determined similar-sound priority word by voice. Reference in which the second determination unit presets one of the plurality of words as the similar-sound priority word with respect to a combination of a plurality of words whose similarity in lip shape at pronunciation is equal to or greater than a threshold value. The information processing apparatus according to claim 1, wherein the similar-sound priority word is acquired by referring to information. In an information processing device including a processor,
Acquire the processing target image including the lip of the recognition target person,
Calculate the degree of similarity between the acquired image to be processed and a plurality of types of reference images,
Determining a pronunciation candidate word for the image to be processed based on the similarity,
When there are multiple pronunciation candidate words, even if the pronunciation candidate word is different from the word that should actually be uttered from the pronunciation candidate words, the listener can hear the word as a correct word. The phonetic word is defined as a similar sound priority word that has been defined in advance as a word.
A control program that causes the processor to execute a process of outputting the determined similar-sound priority word as an audio from an output device. A process of acquiring a processing target image including the lip of the recognition target person,
A process of calculating the degree of similarity between the acquired target image and a plurality of reference images corresponding to a plurality of words,
A process of determining a pronunciation candidate word for the processing target image based on the similarity,
When there are multiple pronunciation candidate words, even if the pronunciation candidate word is different from the word that should actually be uttered from the pronunciation candidate words, the listener can hear the word as a correct word. a process of determining a pronunciation words a predefined anion preferred terms as words,
And a process of causing the output device to output the determined similar-sound priority word by voice.

Images