JP7624707B2 - Facial synthesis lip reading device and facial synthesis lip reading method - Google Patents

Description

The present invention relates to a facial synthesis lip reading device and a facial synthesis lip reading method that can predict the speech content of an unspecified speaker to be recognized with high accuracy using machine learning.

Conventionally, speech recognition technology that converts speech information into text has a sufficient recognition rate in low-noise environments such as laboratories, and is gradually becoming more widespread, but it is difficult to use in noisy environments such as offices and outdoors where it is easily affected by surrounding noise, or in public places such as trains and hospitals where it is difficult to speak, and there is a problem that it lacks practicality. In addition, speech recognition technology cannot be used by people with speech disorders who have difficulty speaking, and there is also a problem that it lacks versatility.
In contrast, lip-reading technology can infer the content of speech from the movement of the speaker's lips, etc., and does not require the speaker to emit sound (no audio information is required), and can infer the content of speech from video alone, making it possible to use it not only in noisy environments and public places, but also by people with speech disorders. In particular, lip-reading technology using a computer does not require special training and can be easily used by anyone, so its widespread use is expected.
For example, Patent Document 1 discloses a word spotting lip reading device that includes an imaging means for acquiring a facial image including a lip area, an area extraction means for extracting the lip area from the acquired image, a feature measurement means for measuring shape features from the extracted lip area, a keyword DB for registering features of keyword speech scenes measured in a registration mode, a determination means for performing a recognition process in a recognition mode to recognize (infer) the speech content of the lips by comparing the features of the registered keywords with features measured for speech scenes in a sentence, thereby performing word spotting lip reading to recognize keywords from within a sentence, and a display means for displaying the recognition results performed by the determination means.

JP 2012-59017 A

In the conventional lip reading technology using a computer, such as that described in Patent Document 1, in a registration mode (learning mode, during learning), a model is learned by machine learning using the speech scenes of a speaker (learning target speaker) to be registered (learned), and in a recognition mode, the speech contents of a speaker (recognition target speaker) to be recognized are inferred. This recognition task generally includes two types: specific speaker recognition, in which the recognition target speaker is included in the learning target speakers, and unspecified speaker recognition, in which the recognition target speaker is not included in the learning target speakers. In specific speaker recognition, the learning target speaker and the recognition target speaker are specified (limited), so the recognition accuracy is usually higher than that of unspecified speaker recognition, but there is a problem that it is necessary to pre-register (learn) speech scenes in which the user (recognition target speaker) is the learning target speaker, and it is time-consuming to build a learning model. In contrast, speaker-independent recognition does not require prior registration as in speaker-specific recognition (any speaker can be the target speaker), and learning can be performed on speech scenes of unspecified speakers (using existing databases, etc.). This makes it easy to build a learning model, but it has the problem that recognition is more difficult than speaker-specific recognition.
The present invention has been made in consideration of the above circumstances, and aims to provide a highly practical face synthesis lip reading device and face synthesis lip reading method that can improve the accuracy of lip reading recognition by processing the speaker's facial image.

A face synthesis lip reading device according to a first invention for achieving the above object comprises an image acquisition unit which, during learning, reads a learning target image in which a speech scene of a learning target speaker is recorded, and, during recognition, reads a recognition target image in which a speech scene of the recognition target speaker is recorded; an image processing unit which performs image processing on the learning target image and the recognition target image read by the image acquisition unit to extract learning target data and recognition target data, respectively; a learning processing unit which, during learning, performs machine learning of lip reading based on the learning target data and constructs a learning model; a lip reading database which stores the learning model; and a recognition processing unit which, during recognition, infers the speech content of the recognition target speaker by machine learning from the recognition target data and the learning model stored in the lip reading database,
The image processing unit includes a face detection means for detecting a training-time facial image of the training target speaker from the training target image and a recognition-time facial image of the recognition target speaker from the recognition target image, a face synthesis means for converting the training-time facial image and the recognition-time facial image detected by the face detection means into a training-time synthetic facial image and a recognition-time synthetic facial image, respectively, using a facial image of a specific speaker, a lip area extraction means for extracting a training-time lip area and a recognition-time lip area from the training-time synthetic facial image and the recognition-time synthetic facial image created by the face synthesis means, respectively, and a feature extraction means for extracting training-time lip features from the training-time lip area as the training target data and extracting recognition-time lip features from the recognition-time lip area as the recognition target data.

In the face synthesis lip reading device of the first invention, the image processing unit may have a facial feature point detection means for detecting training-time facial feature points and recognition-time facial feature points from the training-time synthetic face image and the recognition-time synthetic face image, respectively, and the lip area extraction means may extract the training-time lip area and the recognition-time lip area from the training-time facial feature points and the recognition-time facial feature points, respectively.

In the face synthesis lip reading device of the first invention, the feature extraction means extracts, as the learning target data, in addition to the learning lip features, learning facial expression features from the learning facial feature points, and, as the recognition target data, in addition to the recognition lip features, recognition facial expression features from the recognition facial feature points.

The face synthesis lip reading device according to the first invention may include an imaging means for imaging the speech scenes of the learning target speaker and the recognition target speaker, and a recognition result output unit for outputting the speech content of the recognition target speaker estimated by the recognition processing unit.

In the face synthesis lip reading device of the first invention, it is preferable that the recognition result output unit includes a display that displays the speech content of the recognition target speaker estimated by the recognition processing unit in text and/or a speaker that outputs the speech content in audio.

A face synthetic lip reading method according to a second invention in accordance with the above object includes a first learning step of reading a learning target image in which a speech scene of a learning target speaker is recorded, a second learning step of detecting a learning face image of the learning target speaker from the learning target image, a third learning step of converting the learning face image into a learning synthetic face image using a face image of a specific speaker, a fourth learning step of extracting a learning lip area from the learning synthetic face image, a fifth learning step of extracting learning lip features from the learning lip area as learning target data, a sixth learning step of repeating the first learning step to the fifth learning step, performing machine learning of lip reading based on the learning target data, and constructing a learning model, and a seventh learning step of saving the learning model; a first recognition step of loading the saved learning model; a second recognition step of loading a recognition target image in which a speech scene of a recognition target speaker is recorded; a third recognition step of detecting a recognition target face image of the recognition target speaker from the recognition target image; a fourth recognition step of converting the recognition target face image into a recognition target synthetic face image using a face image of a specific speaker; a fifth recognition step of extracting a recognition target lip area from the recognition target synthetic face image; a sixth recognition step of extracting recognition target lip features from the recognition target lip area as recognition target data; and a seventh recognition step of inferring the speech content of the recognition target speaker by machine learning from the recognition target data and the learning model.

In the face synthesis lip reading method according to the second invention, in the fourth learning step, learning facial feature points are detected from the learning synthetic face image and the learning lip area is extracted from the learning facial feature points, and in the fifth recognition step, recognition facial feature points are detected from the recognition synthetic face image and the recognition lip area is extracted from the recognition facial feature points.

In the face synthesis lip reading method according to the second invention, in the fifth learning step, in addition to the learning lip features, learning facial expression features are extracted from the learning facial feature points as the learning target data, and in the sixth recognition step, in addition to the recognition lip features, recognition facial expression features can also be extracted from the recognition facial feature points as the recognition target data.

The facial synthesis lip reading device according to the first invention and the facial synthesis lip reading method according to the second invention convert the facial image of an unspecified speaker (the learning target speaker and the recognition target speaker are collectively referred to as speakers) into the facial image of a specific speaker and perform learning and recognition processes, thereby making it possible to estimate the spoken content with high accuracy during recognition, thereby improving recognition accuracy.

1 is a block diagram showing a configuration of a face synthesis lip reading device according to an embodiment of the present invention; FIG. 2 is a block diagram showing functions of an image processing unit of the face synthesis lip reading device. 5 is a flowchart showing an operation during learning of the face synthesis and lip reading method according to the embodiment of the present invention. 11 is a flowchart showing an operation during recognition of the same face synthesis lip reading method.

Next, specific embodiments of the present invention will be described for better understanding of the present invention.
The facial synthesis lip reading device 10 and facial synthesis lip reading method according to one embodiment of the present invention shown in FIG. 1 are lip reading technologies using a computer (machine learning), which convert the facial images of an unspecified learning target speaker and an unspecified recognition target speaker into facial images of a specific speaker and then perform learning and recognition processes to predict the spoken content of the unspecified recognition target speaker with high accuracy, thereby improving recognition accuracy (lip reading accuracy).
As shown in Fig. 1, the face synthesis lip reading device 10 includes an image capture unit 11 for capturing (recording) speech scenes of a learning target speaker and a recognition target speaker. The face synthesis lip reading device 10 also includes an image capture unit 13 for reading a learning target image in which a speech scene of the learning target speaker is recorded from the image capture unit 11 during learning, and reading a recognition target image in which a speech scene of the recognition target speaker is recorded from the image capture unit 11 during recognition. The face synthesis lip reading device 10 also includes an image processing unit 14 for processing the learning target image and the recognition target image loaded into the image capture unit 13, respectively, to extract learning target data and recognition target data required for machine learning. The face synthesis lip reading device 10 also includes a learning processing unit 15 for performing machine learning of lip reading based on the learning target data during learning, and constructing a learning model, and a lip reading database 16 for storing the learning model. The face synthesis lip reading device 10 is equipped with a recognition processing unit 17 that, during recognition, infers the spoken content of the speaker to be recognized through machine learning from the recognition target data and the learning model stored in the lip reading database 16.

Here, as shown in FIG. 1, the face synthesis lip reading device 10 is configured to include an image acquisition unit 13, an image processing unit 14, a learning processing unit 15, a lip reading database 16, and a recognition processing unit 17, but a program for executing the face synthesis lip reading method used in the face synthesis lip reading device 10 is installed in a computer 18, and the CPU of the computer 18 executes the program, thereby allowing the computer 18 to function as the image acquisition unit 13, the image processing unit 14, the learning processing unit 15, the lip reading database 16, and the recognition processing unit 17. As for the form of the computer, a desktop type or a notebook type is preferably used, but is not limited to these and can be selected as appropriate. Note that some or all of the image acquisition unit 13, the image processing unit 14, the learning processing unit 15, the lip reading database 16, and the recognition processing unit 17 can also be used through a network by cloud computing. In addition, while a video camera is preferably used as the imaging means, the face synthesis lip reading device does not need to be equipped with a dedicated imaging means, and various imaging means that capture speech scenes or various storage means (storage media) in which speech scenes are recorded can be connected to a computer (image acquisition unit) to read the learning target image or the recognition target image. Therefore, a smartphone or the like equipped with a video recording function can be used as the imaging means, and instead of connecting the imaging means to a computer (image acquisition unit) to read the image, a storage medium such as a memory card built into the imaging means can be inserted from the imaging means to the computer (image acquisition unit) to read the image.

The facial synthesis lip reading device 10 also includes a recognition result output unit 19 that outputs the speech content of the recognition target speaker estimated by the recognition processing unit 17. In this embodiment, the recognition result output unit 19 includes a display 20 that displays the speech content of the recognition target speaker estimated by the recognition processing unit 17 in text and a speaker 21 that outputs it in audio, but either or both of the display 20 and the speaker 21 can be appropriately selected and used depending on the location and environment of use of the facial synthesis lip reading device 10. The display and the speaker may be accessories or built-in components of the computer, or may be separately attached (externally) to the computer. The recognition result output unit may also be configured to include only one of the display and the speaker.

Next, the image processing unit 14 will be described in detail with reference to FIG.
The image processing unit 14 includes face detection means 22 for detecting a learning face image of the learning target speaker from the learning target image during learning, and detecting a recognition face image of the recognition target speaker from the recognition target image during recognition. The image processing unit 14 also includes face synthesis means 23 for converting the learning face image and the recognition face image detected by the face detection means 22 into a learning composite face image and a recognition composite face image using the face image of a specific speaker, respectively, and lip area extraction means 24 for extracting a learning lip area and a recognition lip area from the learning composite face image and the recognition composite face image created by the face synthesis means 23, respectively. The image processing unit 14 also includes feature extraction means 25 for extracting learning lip features from the learning lip area as learning target data, and for extracting recognition lip features from the recognition lip area as recognition target data.

In the face synthesis means 23, the face image (still image) of a specific person can be moved (animated) in accordance with the movement of the learning face image and the recognition face image of an unspecified person when speaking. In other words, the learning face image and the recognition face image of an unspecified speaker can be converted into a learning synthetic face image and a recognition synthetic face image using the face image of a specific speaker, so that the recognition accuracy of lip reading can be improved. As the face image of the specific speaker, for example, a face image of a person registered in an existing database or the like can be used, or a face image of another person can be used. In addition to a real person, the face image can be a face image of a person drawn by computer graphics or animation, or for example, a face image of a person other than a person, such as a robot, a car, an animal, or an insect. In addition, the specific speaker is not limited to one person, but can be a small number of specified people. The recognition accuracy is improved by consolidating the learning target data and the recognition target data from a large number of unspecified speakers to one or a small number of specific speakers. When a specific speaker is selected from the learning target speakers or the recognition target speakers, the facial image of the learning target speaker or the recognition target speaker selected as the specific speaker may or may not be converted to the facial image of the specific speaker, as with other learning target speakers or recognition target speakers. Note that the face synthesis means 23 preferably uses the First Order Motion Model (FOMM), but is not limited to this.

The image processing unit 14 extracts the learning-time lip area and the recognition-time lip area by image processing the learning-time synthetic face image and the recognition-time synthetic face image using the lip area extraction means 24, and the means and method for extracting the learning-time lip area and the recognition-time lip area are selected as appropriate. For example, the image processing unit 14 can detect the learning-time facial feature points and the recognition-time facial feature points from the learning-time synthetic face image and the recognition-time synthetic face image, respectively, using the facial feature point detection means, and then extract the learning-time lip area and the recognition-time lip area based on the learning-time facial feature points and the recognition-time facial feature points using the lip area extraction means. The learning-time facial feature points and the recognition-time facial feature points represent, for example, the facial contour of a specific speaker and the positions and shapes of the eyebrows, eyes, nose, and mouth, and the number of feature points is selected as appropriate. In addition, as in this embodiment, when the face synthesis means 23 first converts the learning-time face image of the learning target speaker and the recognition-time face image of the recognition target speaker into a learning-time composite face image and a recognition-time composite face image using the face image of a specific speaker, the number of learning-time facial feature points and recognition-time facial feature points can be reduced to speed up or simplify processing in the facial feature point detection means, or the facial feature point detection means can be omitted to simplify processing in the image processing unit.

In addition, the feature extraction means extracts learning-time lip features from the learning-time lip area as learning target data, and extracts recognition-time lip features from the recognition-time lip area as recognition target data, so that the learning processing unit and the recognition processing unit can perform machine learning based on the respective lip features (movement features of the area around the lips), construct a learning model, and infer the content of the speech. As described above, if the image processing unit has a facial feature point detection means and the learning facial feature points and the recognition facial feature points are detected from the learning synthetic face image and the recognition synthetic face image, respectively, the feature extraction means can extract learning facial expression features from the learning facial feature points in addition to the learning lip features as the learning target data, and can extract recognition facial expression features from the recognition facial feature points in addition to the recognition lip features as the recognition target data. It is possible to further improve the recognition accuracy (recognition rate) of the speech content by performing machine learning taking into account the facial expression features of the entire face (for example, changes in the position, shape, and angle of the eyebrows, eyes, mouth, etc.) in addition to the lip features, but sufficient recognition accuracy can be obtained with just the learning lip features and the recognition lip features.

Next, the operation during learning of the face synthesis and lip reading method according to the embodiment of the present invention will be described with reference to FIG.
First, in a first learning step, a learning target image in which a speech scene of a learning target speaker is recorded is read into the image acquisition unit 13 (S1). Next, in a second learning step, the face detection means 22 of the image processing unit 14 detects a learning face image of the learning target speaker from the learning target image (S2). Next, in a third learning step, the face synthesis means 23 of the image processing unit 14 converts the learning face image into a learning composite face image using the face image of a specific speaker (S3), and in a fourth learning step, the lip region extraction means 24 of the image processing unit 14 extracts a learning lip region from the learning composite face image (S4). Furthermore, in a fifth learning step, the feature extraction means 25 of the image processing unit 14 extracts learning lip features from the learning lip region as learning target data (S5). The above first to fifth learning steps are repeated as many times as the number of speech scenes to be learned. Then, in a sixth learning step, the learning processing unit 15 performs machine learning of lip reading based on the learning target data extracted from each speech scene, and constructs a learning model (S6). The learning model thus constructed is stored in the lip reading database 16 in a seventh learning step (S7).
As described above, the means and method for extracting the learning lip area are appropriately selected. For example, in the fourth learning step, the face feature point detection means of the image processing unit detects the learning face feature points from the learning composite face image, and the lip area extraction means extracts the learning lip area based on the learning face feature points. If the learning face feature points are detected, in the fifth learning step, the feature extraction means of the image processing unit extracts learning facial expression features from the learning face feature points in addition to the learning lip area as learning target data, and machine learning can be performed taking into account the facial expression features of the entire face in addition to the lip features.

Next, the operation during recognition in the face synthesis and lip reading method will be described with reference to FIG.
First, in a first step during recognition, a learning model (trained model) stored in the lip reading database 16 is read (S1). Then, in a second step during recognition, a recognition target image in which a speech scene of a recognition target speaker is recorded is read into the image acquisition unit 13 (S2). Next, in a third step during recognition, the face detection means 22 of the image processing unit 14 detects a recognition target face image of the recognition target speaker from the recognition target image (S3). Then, in a fourth step during recognition, the face synthesis means 23 of the image processing unit 14 converts the recognition target face image into a recognition target composite face image using the face image of the specific speaker (S4), and in a fifth step during recognition, the lip region extraction means 24 of the image processing unit 14 extracts a recognition target lip region from the recognition target composite face image (S5). Furthermore, in a sixth step during recognition, the feature extraction means 25 of the image processing unit 14 extracts recognition target lip features from the recognition target lip region as recognition target data (S6). Then, in the seventh step of the recognition process, the speech content of the speaker to be recognized is inferred from the recognition target data and the learning model by machine learning (lip reading processing) (S7). The inferred speech content (evaluation result) is converted into characters and/or voice, and is output from the display 20 and/or speaker 21 of the evaluation result output unit 19 (S8).
As described above, if the image processing unit has a facial feature point detection means, in the fifth recognition step, the recognition facial feature points are detected from the recognition composite face image, and the recognition lip area can be extracted based on the recognition facial feature points. In the sixth recognition step, the feature extraction means can extract recognition facial expression features from the recognition facial feature points in addition to the recognition lip features as recognition target data, and can perform machine learning taking into account the facial expression features of the entire face in addition to the lip features.

Next, the results of evaluations carried out to confirm the effects of the present invention will be described.
(Comparative Example 1)
A conventional lip-reading method was evaluated using the database CUAVE, which is open to the public for lip-reading. CUAVE contains scenes in which 36 registered speakers (19 males and 17 females) each speak 10 numbers from 0 to 9 in English. As the conventional lip-reading method, the third learning step and the fourth recognition step of the face synthesis lip-reading method of the present invention were omitted, and in the subsequent steps, the learning face image and the recognition face image were used as they were, instead of the learning synthetic face image and the recognition synthetic face image. As the evaluation method, the existing method of the leave-one-person method was used. In other words, one recognition target speaker was selected from the 36 registered speakers, and the remaining 35 registered speakers were used as the learning target speaker for learning and evaluation. The recognition target speaker was changed, and 36 recognition experiments (one set) were performed, and the average recognition accuracy was 73%.

Example 1
The same database CUAVE as in Comparative Example 1 was used, and the face synthesis lip reading method of the present invention was used as the lip reading method. First, in order to select the face images of the specific speakers to be used in the third learning step and the fourth recognition step, one frame image was taken out from each registered speaker's speech scene to prepare face images (still images) of 36 registered speakers, and one face image of the specific speaker was selected from the face images of each registered speaker, and 36 recognition experiments (one pattern) were performed for each specific speaker by the leave-one-out method similar to Comparative Example 1. Of the total 36 recognition experiments for the 36 specific speakers, 35 (97%) showed improved recognition accuracy compared to the recognition experiments in Comparative Example 1, with an average recognition accuracy of 83%. In addition, when the results of 36 (one) recognition experiments for one specific speaker with the highest recognition accuracy out of the 36 recognition experiments (36 specific speakers) were compared with the results of the recognition experiments in Comparative Example 1 for each target speaker, it was found that the recognition accuracy was improved compared to the recognition experiments in Comparative Example 1 in 27 of the 36 recognition experiments, that is, in the recognition experiments for 27 out of 36 target speakers (75%).

(Comparative Example 2)
OuluVS was used as the database instead of CUAVE. OuluVS contains scenes in which 20 registered speakers (17 men and 3 women) each utter 10 types of greetings in English. Evaluation was performed in the same manner as in Comparative Example 1, except that the number of registered speakers was different. In other words, one recognition target speaker was selected from the 20 registered speakers, and the remaining 19 registered speakers were used as learning targets for learning and evaluation. The recognition target speaker was changed and 20 (one pattern) recognition experiments were performed, and the average recognition accuracy was 81%.

Example 2
Except for using the same database OuluVS as in Comparative Example 2, the evaluation was performed in the same manner as in Example 1. That is, the facial image of a specific speaker was selected one by one from the facial images (still images) of the prepared 20 registered speakers, and 20 (one-way) recognition experiments were performed for each specific speaker by the leave-one-out method. Of the total 20 recognition experiments for the 20 specific speakers, the recognition accuracy was improved in 17 (85%) recognition experiments compared to the recognition experiment in Comparative Example 2, and the average recognition accuracy was 87%. In addition, when the results of the 20 (one-way) recognition experiments for the one specific speaker with the highest recognition accuracy among the 20 recognition experiments (20 specific speakers) were compared with the results of the recognition experiment in Comparative Example 2 for each recognition target speaker, the recognition accuracy was improved in 12 of the 20 recognition experiments, that is, in the recognition experiments for 12 of the 20 recognition experiments (60%) of the recognition target speakers, compared to the recognition experiment in Comparative Example 2.

From the above, it was found that by using the face synthesis lip reading method of the present invention, the lip reading recognition accuracy is improved by about 6 to 7% on average compared to conventional lip reading methods, and the effectiveness of the face synthesis lip reading method of the present invention was confirmed. Note that in Examples 1 and 2, the recognition experiment was conducted by converting all facial images of the learning target speakers and the recognition target speakers into the facial image of one specific speaker, but even if there are multiple specific speakers, it is believed that the recognition accuracy can be improved as long as the number of specific speakers is at least less than the number of learning target speakers.

The present invention has been described above with reference to an embodiment, but the present invention is not limited to the configurations described in the above embodiment, and also includes other embodiments and modifications that are possible within the scope of the matters described in the claims.
For example, in the present embodiment, a gated recurrent unit (GRU), which is a type of deep learning, is used for machine learning, but the machine learning algorithm may be selected as appropriate.

10: Face synthesis lip reading device, 11: Photographing means, 13: Image acquisition unit, 14: Image processing unit, 15: Learning processing unit, 16: Lip reading database, 17: Recognition processing unit, 18: Computer, 19: Recognition result output unit, 20: Display, 21: Speaker, 22: Face detection means, 23: Face synthesis means, 24: Lip area extraction means, 25: Feature extraction means

Claims (8)

an image acquisition unit that loads a learning target image in which a speech scene of a learning target speaker is recorded during learning, and loads a recognition target image in which a speech scene of the recognition target speaker is recorded during recognition, an image processing unit that performs image processing on the learning target image and the recognition target image loaded into the image acquisition unit to extract learning target data and recognition target data, respectively, a learning processing unit that performs machine learning of lip reading based on the learning target data during learning and constructs a learning model, a lip reading database that stores the learning model, and a recognition processing unit that infers the speech content of the recognition target speaker by machine learning from the recognition target data and the learning model stored in the lip reading database during recognition,
said image processing unit comprises face detection means for detecting a training-time facial image of the training target speaker from said training image and for detecting a recognition-time facial image of the recognition target speaker from said recognition target image; face synthesis means for converting the training-time facial image and the recognition-time facial image detected by said face detection means into a training-time synthetic facial image and a recognition-time synthetic facial image, respectively, using a facial image of a specific speaker; lip area extraction means for extracting a training-time lip area and a recognition-time lip area from the training-time synthetic facial image and the recognition-time synthetic facial image created by said face synthesis means, respectively; and feature extraction means for extracting training-time lip features from the training-time lip area as said training target data and extracting recognition-time lip features from the recognition-time lip area as said recognition target data. A facial synthesis lip reading device according to claim 1, characterized in that the image processing unit has a facial feature point detection means for detecting the learning-time facial feature points and the recognition-time facial feature points from the learning-time synthetic face image and the recognition-time synthetic face image, respectively, and the lip area extraction means extracts the learning-time lip area and the recognition-time lip area from the learning-time facial feature points and the recognition-time facial feature points, respectively. A facial synthesis lip reading device according to claim 2, wherein the feature extraction means extracts, as the learning target data, in addition to the learning lip features, learning facial expression features from the learning facial feature points, and, as the recognition target data, in addition to the recognition lip features, recognition facial expression features from the recognition facial feature points. A facial synthesis lip-reading device according to any one of claims 1 to 3, characterized in that it is provided with an imaging means for imaging the speech scenes of the learning target speaker and the recognition target speaker, and a recognition result output unit for outputting the speech content of the recognition target speaker estimated by the recognition processing unit. A facial synthesis lip-reading device according to claim 4, characterized in that the recognition result output unit is provided with a display that displays the speech content of the speaker to be recognized that is estimated by the recognition processing unit in text and/or a speaker that outputs the speech content in audio. A face synthesis and lip reading method using machine learning by a computer, comprising:
a learning step of reading a learning target image in which a speech scene of a learning target speaker is recorded; a learning step of detecting a learning face image of the learning target speaker from the learning target image; a learning step of converting the learning face image into a learning synthetic face image using a face image of a specific speaker; a learning step of extracting a learning lip area from the learning synthetic face image; a learning step of extracting learning lip features from the learning lip area as learning target data; a learning step of repeating the learning steps from the learning step to the learning step, and constructing a learning model based on the learning target data; a learning step of storing the learning model; a third step at recognition time of detecting a time-of-recognition face image of the recognition target speaker from the recognition target image; a fourth step at recognition time of converting the time-of-recognition face image into a time-of-recognition synthetic face image using a face image of a specific speaker; a fifth step at recognition time of extracting a time-of-recognition lip area from the time-of-recognition synthetic face image; a sixth step at recognition time of extracting a time-of-recognition lip feature from the time-of-recognition lip area as recognition target data; and a seventh step at recognition time of inferring the speech content of the recognition target speaker from the recognition target data and the learning model by machine learning. A face synthesis lip reading method according to claim 6, characterized in that in the fourth learning step, learning facial feature points are detected from the learning synthetic face image and the learning lip area is extracted from the learning facial feature points, and in the fifth recognition step, recognition facial feature points are detected from the recognition synthetic face image and the recognition lip area is extracted from the recognition facial feature points. A face synthesis lip reading method according to claim 7, characterized in that in the fifth learning step, in addition to the learning lip features, learning facial expression features are extracted from the learning facial feature points as the learning target data, and in the sixth recognition step, in addition to the recognition lip features, recognition facial expression features are extracted from the recognition facial feature points as the recognition target data.

Images