JP7640964B2 - Speech content recognition device, method, and program - Google Patents

Description

The present invention relates to an apparatus, method, and program for recognizing speech content.

Conventionally, there is known a speech content recognition device that recognizes the content of a speaker's speech. For example, a device is known that inputs image data of a speaker's lips and recognizes the content of the speaker's speech using a lip reading unit that has high lip reading accuracy for the lip image data captured from a corresponding direction.

Non-Patent Document 1 also discloses a method called "View2View" that uses an encoder-decoder model using a convolutional neural network. In this method, lip-reading results are output using a machine lip-reading model that has been trained in advance on image data of a frontal face (image data of lips captured from the front of the face). When non-frontal image data is input, it is converted to image data of a frontal face and then input to the machine lip-reading model, which then outputs the lip-reading results.

Non-Patent Document 2 discloses an end-to-end lip-reading method that uses a deep learning technique called bidirectional long-short-term memory. This non-patent document 2 describes how the lip-reading performance of a lip-reading model can be improved by learning by combining face image data captured from multiple imaging angles, such as front and side views.

In conventional speech recognition devices equipped with a lip reading unit, lip reading results can be obtained with high accuracy when using lip image data captured from a specific direction (corresponding direction), but there is an issue in that accuracy drops when using lip image data captured from a direction different from the corresponding direction.

In order to solve the above-mentioned problems, the present invention provides a speech content recognition device that recognizes the speech content of a speaker, comprising an input unit that inputs lip image data of the speaker, a plurality of lip reading units that have high lip reading accuracy for lip image data captured from corresponding directions, and an integration generation unit that integrates the lip reading processing results of the plurality of lip reading units for the lip image data input to the input unit and generates a recognition result of the speech content of the speaker based on the result of the integration, wherein at least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units , and the plurality of lip reading units includes a multi-directional lip reading unit having two or more corresponding directions .

According to the present invention, accurate lip reading results can be obtained with high accuracy even for lip image data captured from a direction that does not match the corresponding direction of the lip reading unit, so the speech content can be recognized with high accuracy for various types (various imaging directions) of lip image data that exceed the number of corresponding directions.

FIG. 1 is a block diagram showing a lip reading device according to a first embodiment. FIG. 1 is an explanatory diagram showing an example of a model of one neuron that constitutes a neural network. FIG. 1 is an explanatory diagram showing an example of a neural network having a multi-layer structure. FIG. 2 is an explanatory diagram showing an overview of a method (learning mode) for creating a machine lip reading model (trained model) in the first embodiment. FIG. 4 is a block diagram showing another example of the lip reading device according to the first embodiment. FIG. 4 is a block diagram showing yet another example of the lip reading device according to the first embodiment. FIG. 13 is a block diagram showing a lip reading device according to a first modified example. FIG. 11 is a block diagram showing a lip reading device according to a second modified example. FIG. 11 is a block diagram showing a multimodal speech recognition device according to a second embodiment. FIG. 11 is an explanatory diagram showing the configuration of a learning data collection system according to a third embodiment. FIG. 2 is an explanatory diagram showing the camera array of the learning data collection system viewed vertically from above.

[Embodiment 1]
Hereinafter, an embodiment in which the present invention is applied to a lip reading device as a speech content recognition device (hereinafter, this embodiment will be referred to as "embodiment 1") will be described.
The lip reading device of this embodiment 1 inputs facial image data obtained by capturing an image of a speaker's face as lip image data, analyzes the lip portion of the input facial image data, and outputs a recognition result (lip reading result) of the speech content uttered by the speaker.

FIG. 1 is a block diagram showing a lip reading device according to the first embodiment.
The lip reading device 100 of this embodiment 1 is mainly composed of an image input unit 111 as an input unit, two single-angle compatible lip reading units 131 and 132 as multiple lip reading units, and a lip reading result integration unit 141 as an integration generation unit.

The image input unit 111 accepts input of face image data (lip image data) of a speaker whose speech content is to be recognized. In this embodiment 1, the image input unit 111 is connected to a camera 1, which is an imaging device that captures the face of the speaker, and a storage medium 2 that stores the face image data, so that they can communicate with each other via wired or wireless communication. Real-time face image data of the speaker currently speaking is input from the camera 1 to the image input unit 111. The storage medium 2 stores face image data from when the speaker spoke in the past, and past face image data is input from the storage medium 2 to the image input unit 111.

The image input unit 111 processes the input face image data as necessary before inputting the data to the two single-angle compatible lip reading units 131, 132, and passes the data to each of the single-angle compatible lip reading units 131, 132. For example, the lip image portion of the input face image data is arranged in chronological order and extracted, and the data of the lip image portion is passed to each of the single-angle compatible lip reading units 131, 132.

There are no particular limitations on the imaging direction of the lip image data input to image input unit 111, so long as the image data is imaged so as to include the lips of the speaker.
Furthermore, the lip image data input to the image input unit 111 may be in an image data format, or may be in a non-image data format obtained by processing or calculating the lip image data.
In addition, lip image data is typically captured image data obtained by capturing an image of a real speaker using an imaging device, etc., but it may also be image data of a virtual speaker (e.g., created using computer graphics, etc.) viewed from a specified viewpoint.

The two single-angle compatible lip reading units 131 and 132 each perform lip reading processing with high lip reading accuracy on lip image data captured from a specific direction (corresponding direction) and generate the lip reading processing result. The two single-angle compatible lip reading units 131 and 132 are configured so that their corresponding directions include directions that are not included in the corresponding directions of the other single-angle compatible lip reading unit.

In this embodiment 1, the corresponding direction is expressed as an angle (hereinafter referred to as the "corresponding angle") around a vertical axis with the imaging direction when the speaker's face is imaged from the front direction as the reference (0°). In this case, the first single-angle corresponding lip reading unit 131 has high lip reading accuracy for face image data when the speaker's face is imaged from the front direction (exceeding the accuracy threshold that meets the user's required level), and its corresponding angle (angle with high lip reading accuracy) is 0°. On the other hand, the second single-angle corresponding lip reading unit 132 has high lip reading accuracy for face image data when the speaker's face is imaged from a direction shifted 30° to the side from the front direction, and its corresponding angle (angle with high lip reading accuracy) is 30°.

The single-angle compatible lip reading units 131, 132 of this embodiment 1 execute a predetermined lip reading processing program (estimation program) on a computer to perform lip reading processing on the face image data input to the image input unit 111 and generate a lip reading processing result. The lip reading processing program of this embodiment 1 uses a machine lip reading model (trained model) trained using training data including the speaker's face image data, but a lip reading processing program programmed by a programmer may also be used.

The machine lip reading model (trained model) in this embodiment 1 estimates the speaker's speech content from the input data (facial image data), and there are no particular limitations on the format of the estimation result (lip reading processing result) output from the machine lip reading model. As an example, this embodiment 1 describes a case where data including one or more speech content candidates (such as one character, one word or word sequence) and reliability information for each of the speech content candidates (hereinafter referred to as "reliability score") is output as the lip reading processing result for the input data (facial image data).

A machine lip-reading model specialized for a specified corresponding angle (a trained model with high lip-reading accuracy for face image data at the corresponding angle) can be created, for example, by performing machine learning or deep learning using a large amount of face image data captured from the corresponding angle as training data. For example, by using such training data and performing supervised learning on a specified model, it is possible to obtain a machine lip-reading model (trained model) that, when receiving input of unknown face image data, outputs data including one or more speech content candidates and a reliability score for each speech content candidate as an estimation result according to the features learned from the training data. Note that in this embodiment 1, a neural network model is adopted as the specified model, but other machine learning models can also be used.

In "supervised learning," a machine learning device is generally given a large number of pairs of input and result (label) data, which allows it to learn the features of those data sets and inductively acquire a model that estimates results from inputs, i.e., the relationships between them. This can be achieved using algorithms such as neural networks and SVMs (Support Vector Machines), which are described below.

A neural network is composed of a computing device and a memory that realizes a neural network that mimics a neuron model, for example, as shown in FIG. 2. As shown in FIG. 2, a neuron outputs an output y for multiple inputs x (inputs x1 to x3 are shown here as an example, but the number of inputs may be less or more). Each of the inputs x1 to x3 is multiplied by a weight W (W1 to W3) corresponding to each input x. As a result, the neuron outputs an output y expressed by the following equations (1) and (2). In equations (1) and (2), θ is a bias, and fk is an activation function.

y = fk(v)...(1)
v=Σ(W×x)-θ...(2)

The neural network operates in a learning mode and an evaluation mode. In the learning mode, the weights W are learned using training data, and in the evaluation mode, the weights W are used to obtain an output for the input of evaluation data (in this embodiment 1, utterance content candidates and their respective reliability scores). The weights W1 to W3 can be learned using backpropagation or the like. The backpropagation method is a technique for adjusting (learning) the weights for each neuron so as to reduce the difference between the output y when an input x is input and the correct output y (correct label).

As shown in Figure 3, neural networks can have a multi-layer structure known as deep learning. The example in Figure 3 shows a three-layer structure of intermediate (hidden) layers. Each layer is made up of multiple nodes (neurons), and the nodes between layers are connected with different weights W. Inputs x1 to x6 input to the input layer are combined while being weighted by the weight W as they pass through the nodes in the intermediate layer, which have different weights W, and then pass through the output layer to derive the output y.

In this embodiment 1, a neural network model consisting of a multi-layered neural network as shown in FIG. 3 is adopted, and a machine lip reading model (trained model) is created by performing supervised learning using training data including facial image data of a speaker who speaks known speech content and using the correct speech content as a correct answer label.

FIG. 4 is an explanatory diagram showing an overview of a method (learning mode) for creating a machine lip-reading model (trained model) in the first embodiment.
In the learning mode of the first embodiment, as shown in Fig. 4, a speaker speaks a specified utterance content, and this is captured by each of the recording cameras 31-1 and 31-2 from the corresponding angles (0° and 30° in the first embodiment). The facial image data captured in this manner is stored in the learning data storage medium 32 for each corresponding angle. The facial image data stored in the learning data storage medium 32 is stored in a format that allows a time series to be specified. Therefore, the facial image data stored in the learning data storage medium 32 can be associated with the speaker's utterance content by checking it against the time when the speaker spoke (the time when each utterance content was spoken), and can be used as learning data.

The learning data thus stored in the learning data storage medium 32 is used by the first learning unit 33-1 for face image data with a corresponding angle of 0°, and by the second learning unit 33-2 for face image data with a corresponding angle of 30°. In order to further improve accuracy, the learning data may include other information useful as a feature for identifying (estimating) the content of the speech. The first learning unit 33-1 generates a machine lip-reading model (trained model) with a corresponding angle of 0° from the input learning data, and the generated machine lip-reading model is installed in the first single-angle compatible lip-reading unit 131 of this embodiment 1. Similarly, the second learning unit 33-2 generates a machine lip-reading model (trained model) with a corresponding angle of 30° from the input learning data, and the generated machine lip-reading model is installed in the second single-angle compatible lip-reading unit 132 of this embodiment 1.

For the generated machine lip reading model (trained model), parameter tuning may be performed by repeatedly trying to create the trained model (training mode). The parameters to be adjusted (tuned) in parameter tuning refer to the setting values and limit values (hyperparameters) in the trained model. Parameter tuning is, for example, the task of scanning and setting parameters that allow the model to produce an optimal solution. Types of parameter tuning that can be used include the grid search method and the random search method.

Model evaluation may also be performed on the machine lip reading model (trained model). For this model evaluation, for example, cross-validation or the hold-out method can be used. Model evaluation can also be performed using a combination of the hold-out method and cross-validation.

Specifically, in the hold-out method, the original data is first split into training data to be used in training mode and test data to be used in evaluation mode, and an attempt is made to create a trained model using only the training data. After that, test data is input into the trained model that has been created, and the model is evaluated by comparing the output result with the correct label of the test data (error = estimated accuracy).

In cross-validation, the original data is divided into, for example, five groups, and in the first round, one of the groups is used as test data and the other groups are used as training data to create a trained model and evaluate the model. In the second round, a different group from the first round is used as test data, and in the third round, a different group from the first and second rounds is used as test data, and a trained model is created and evaluated in the same way. This is done for all five groups, and the model evaluations (estimation accuracy) evaluated each time are averaged.

In addition, the estimation program (trained model) of the present embodiment 1 can be distilled to create a new estimation program (distilled model) with similar functions. Specifically, face image data with known speech content is input as distillation input data to the estimation program (trained model) of the present embodiment 1, and its reliability score is output. Then, distillation learning data is created with the output reliability score as the correct answer label for the distillation input data, and a new estimation program (distilled model) with similar functions to the estimation program (trained model) of the present embodiment 1 is created by training a model using this distillation learning data. The new estimation program (distilled model) created in this way is generally lighter than the estimation program (trained model) of the present embodiment 1. In addition, it is also possible to improve the estimation accuracy compared to the estimation program (trained model) of the present embodiment 1 by devising the distillation input data.

Note that the machine lip reading model of this embodiment 1 is an example in which data including speech content candidates and the reliability scores of each speech content candidate is used as the lip reading processing result, but intermediate representation data that matches the data format used in the subsequent lip reading result integration unit 141 may also be used as the lip reading processing result. Specifically, vector data that records the internal state of the machine lip reading model when lip reading processing is performed using the machine lip reading model may also be used as the lip reading processing result.

In this embodiment 1, the two single-angle compatible lip reading units 131 and 132, which have high lip reading accuracy for lip image data captured from a specific direction (corresponding direction), are used to perform lip reading processing on lip image data. The two single-angle compatible lip reading units 131 and 132 used in this embodiment 1 include angles whose corresponding angles (angles at which high lip reading accuracy can be obtained) are not included in the corresponding angles of the other single-angle compatible lip reading unit. Therefore, if the lip image data is captured from an angle that matches one of the corresponding angles (0° and 30°) of these single-angle compatible lip reading units, these single-angle compatible lip reading units can obtain high lip reading accuracy, and the spoken content can be recognized with high accuracy from the lip reading processing results. Therefore, according to this embodiment 1, the spoken content can be recognized with high accuracy for lip image data captured from angles of 0° and 30°.

Here, for lip image data captured from an angle (e.g., 15° or 45°) that does not match the corresponding angle of either of the two single-angle compatible lip reading units 131, 132, sufficient lip reading accuracy cannot be obtained by the lip reading process of each of the single-angle compatible lip reading units 131, 132. Therefore, if only the lip reading process result of one of the single-angle compatible lip reading units 131, 132 is used, it is not possible to recognize the speech content of such lip image data with high accuracy.

Therefore, in this embodiment 1, a lip reading result integration unit 141 is provided to integrate the lip reading process results obtained by the two single-angle compatible lip reading units 131, 132, and generate a recognition result of the speaker's speech content as a final lip reading result based on the integration result. As a result, even if the lip reading process results of the individual single-angle compatible lip reading units 131, 132 result in the reliability (confidence score) of the correct utterance content being not significantly higher or conversely lower than the reliability of the other incorrect utterance content, by integrating these lip reading process results, it is possible to highlight the reliability (confidence score) of the correct utterance content and process it so that it is significantly higher than the reliability of the other incorrect utterance content.

There are no particular limitations on the integration process performed by the lip reading result integration unit 141, so long as it is a process of integrating the lip reading process results obtained by the two single-angle compatible lip reading units 131, 132 so as to obtain a highly accurate recognition result (in the case of this embodiment 1, so that the reliability score of the correct utterance content becomes relatively high).

As an example of the integration process performed by the lip reading result integration unit 141, for example, the integration result may be the utterance content candidate with the highest reliability score among the common utterance content candidates (such as words and word sequences) included in both of the lip reading process results obtained by the two single-angle compatible lip reading units 131 and 132. Alternatively, the integration result may be two or more utterance content candidates in descending order of reliability score.
Also, for example, the total value of the reliability score for each of the speech content candidates included in the lip reading processing results obtained by the two single-angle compatible lip reading units 131 and 132 may be calculated, and the speech content candidate with the highest total value may be used as the integrated result. Alternatively, two or more speech content candidates for which the total value of the reliability score has been calculated may be used as the integrated result.

As another example of the integration process performed by the lip-reading result integration unit 141, a trained model (integration model) that obtains recognition results from intermediate representations in the two single-angle compatible lip-reading units 131, 132 is trained in advance, for example, by a neural network model. Then, the intermediate representations (vectors, etc.) included in the lip-reading process results obtained by the two single-angle compatible lip-reading units 131, 132 are linked to generate one intermediate representation, and the trained integration model is executed by the computer of the lip-reading result integration unit 141 to obtain recognition results of one or more speech contents from the one intermediate representation, which is used as the integration result. Alternatively, the recognition results of one or more speech contents from the one intermediate representation may include their respective confidence scores.

According to the lip reading device 100 of the first embodiment, by integrating the lip reading process results of the two single-angle compatible lip reading units 131, 132 in the lip reading result integration unit 141, it becomes possible to recognize the speech content with high accuracy not only for the corresponding angles of 0° and 30° at which highly accurate lip reading results can be obtained by the lip reading process of each single-angle compatible lip reading unit 131, 132, but also for lip image data at angles outside these corresponding angles (e.g., 15° and 45°). As a result, it becomes possible to recognize the speech content with high accuracy for lip image data from various angles that exceed the number of corresponding angles (0° and 30°) of the two single-angle compatible lip reading units 131, 132.

The recognition results generated by the lip reading result integration unit 141 are output to a downstream information processing device that uses the recognition results of the speaker's speech content, or to an information storage device that accumulates the recognition results of the speaker's speech content. There are no particular limitations on the output mode, and for example, the recognition results may be displayed on a display unit provided in the lip reading device 100 of this embodiment 1, or output as audio from an audio output unit provided in the lip reading device 100.

In the above-mentioned embodiment 1, an example is given in which there are two lip reading units, but an example in which there are three or more lip reading units may also be used. For example, as shown in FIG. 5, a configuration may be used in which a third single-angle compatible lip reading unit 133 is added that has high lip reading accuracy for face image data captured from a direction shifted 60° to the side from the front direction of the speaker's face.

It is also possible to create a multiple-angle lip reading unit capable of obtaining highly accurate lip reading results at two or more corresponding angles using a single lip reading unit. Specifically, for example, as shown in FIG. 4, a learning data storage medium 32 is used that stores face image data captured from two corresponding angles (0° and 30°), and by inputting these as learning data into a single learning unit and learning them, it is possible to generate a machine lip reading model (trained model) capable of highly accurate lip reading processing at two corresponding angles, 0° and 30°.

Therefore, for example, as shown in FIG. 6, instead of the second single-angle lip reading unit 132 described above, a first multiple-angle lip reading unit 134 capable of highly accurate lip reading at two corresponding angles, 0° and 30°, may be provided. Furthermore, as shown in FIG. 6, instead of the third single-angle lip reading unit 133 described above, a second multiple-angle lip reading unit 135 capable of highly accurate lip reading at three corresponding angles, 0°, 30°, and 60°, may be provided.

However, while a multi-angle lip-reading unit usually has the advantage of being able to improve lip-reading accuracy for a wider range of angles compared to creating multiple single-angle lip-reading units with each corresponding angle corresponding to the multiple corresponding angles of the multi-angle lip-reading unit, it also has the disadvantage of increased learning costs. This disadvantage can be compared, for example, with the amount of training data required to build a machine lip-reading model that executes these lip-reading processes. To achieve a predetermined high level of accuracy (accuracy exceeding a predetermined accuracy threshold) for each of the multiple corresponding angles, the machine lip-reading model of the multi-angle lip-reading unit usually requires much more training data than the total amount of training data required to build each machine lip-reading model of the multiple single-angle lip-reading units. Furthermore, as the training data increases, the costs of parameter tuning and the like also increase.

On the other hand, considering that the multiple-angle lip reading unit has the above-mentioned advantages, it is preferable to have a configuration in which a single-angle lip reading unit and a multiple-angle lip reading unit are mixed, as in the example of Figure 6. In this way, for example, the most frequent imaging angle and nearby angles (corresponding angles that provide high lip reading accuracy) in the input face image data (lip image data) are handled by the single-angle lip reading unit, and other angles are handled by the multiple-angle lip reading unit, and the results of these lip reading processes are integrated by the lip reading result integration unit 141, making it possible to recognize the speech content of lip image data from a wider variety of angles with high accuracy.

In this regard, it is also theoretically possible to construct a single lip-reading unit that is capable of recognizing speech content with high accuracy from lip image data captured from various angles similar to that of the first embodiment. However, it is extremely difficult to realize such a single lip-reading unit using a lip-reading program programmed by a programmer. Furthermore, to realize such a single lip-reading unit using a machine lip-reading model (trained model), a huge amount of training data is required, and it is also practically difficult to build such a machine lip-reading model.

In contrast, the lip reading unit used in the lip reading device 100 of this embodiment 1 is a lip reading unit specialized for one corresponding angle (single-angle compatible lip reading unit) or a lip reading unit specialized for multiple (several) corresponding angles (multiple-angle compatible lip reading unit), and it is relatively easy to construct such lip reading units. Therefore, according to this embodiment 1, there is also the advantage that it is possible to more easily create a lip reading device that can recognize the speech content with high accuracy from lip image data from various angles.

The corresponding angles of the lip reading units whose lip reading process results are integrated by the lip reading result integration unit 141 may overlap partially. In other words, the corresponding angles of the multiple-angle compatible lip reading units may include some or all of the corresponding angles of other single-angle compatible lip reading units or other multiple-angle compatible lip reading units, as long as they are not exactly the same combination. For example, as shown in FIG. 6, 0° may be the corresponding angle of all lip reading units 131, 134, and 135, and 30° may be the corresponding angle of the two multiple-angle compatible lip reading units 134 and 135.

[Modification 1]
Next, a modification of the lip reading apparatus 100 in the above-mentioned first embodiment (hereinafter, this modification will be referred to as "Modification 1") will be described.
FIG. 7 is a block diagram showing a lip reading device 100 according to the first modification.
As shown in Figure 7, the lip reading device 100 in this variant example 1 has three lip reading units: a first single-angle compatible lip reading unit 131 capable of highly accurate lip reading processing at a corresponding angle of 0°, a second single-angle compatible lip reading unit 136 capable of highly accurate lip reading processing at a corresponding angle of 45°, and a multiple-angle compatible lip reading unit 137 capable of highly accurate lip reading processing at two corresponding angles of 0° and 45°.

The lip reading device 100 in this modified example 1 is equipped with angle conversion units 121, 122, and 123 as data conversion units that generate data converted based on the lip image data input to the image input unit 111 so that the imaging direction becomes the corresponding angle of at least one of the lip reading units 131, 136, and 137. In the example of Figure 7, angle conversion units 121, 122, and 123 are provided for all three lip reading units 131, 136, and 137 as processing units in the preceding stage, and each angle conversion unit performs conversion processing of the lip image data input to the image input unit 111 so that the imaging direction matches one of the corresponding angles of the respective lip reading units 131, 136, and 137. That is, each angle conversion unit 121, 122, and 123 performs a conversion process so that the lip image of the lip image data input to the image input unit 111 becomes pseudo-equivalent to the lip image captured from the corresponding angle of each lip reading unit 131, 136, and 137.

For example, when lip image data captured at an angle of 30° is input to the image input unit 111, the first angle conversion unit 121 performs conversion processing on the lip image data input to the image input unit 111 so that the imaging direction matches the corresponding angle of 0° of the first single-angle compatible lip reading unit 131. In this case, similarly, the second angle conversion unit 122 performs conversion processing on the lip image data input to the image input unit 111 so that the imaging direction matches the corresponding angle of 45° of the second single-angle compatible lip reading unit 136. Furthermore, the third angle conversion unit 123 performs conversion processing on the lip image data input to the image input unit 111 so that the imaging direction matches either the corresponding angle of 0° or 45° (here, 0°) of the multiple-angle compatible lip reading unit 137.

Each angle conversion unit 121, 122, 123 may perform the conversion using a linear mapping such as an affine transformation, or may perform the conversion using a conversion model based on machine learning or deep learning. Furthermore, the converted data converted by each angle conversion unit 121, 122, 123 only needs to correspond to the input data format of each lip reading unit 131, 136, 137, and may be in the format of image data or an intermediate representation of the conversion model, for example.

According to this first modification, the data (image data or intermediate representation) received by each of the angle conversion units 121, 122, 123 for each of the lip reading units 131, 136, 137 matches the corresponding angle of each of the lip reading units 131, 136, 137. Therefore, each of the lip reading units 131, 136, 137 can perform lip reading processing with high accuracy, and highly accurate lip reading processing results can be obtained from each of the lip reading units 131, 136, 137. As a result, the final lip reading result (recognition result of the speech content) obtained by integrating these lip reading processing results by the lip reading result integration unit 141 is also highly accurate.

[Modification 2]
Next, another modification of the lip reading apparatus 100 in the above-mentioned first embodiment (hereinafter, this modification will be referred to as "Modification 2") will be described.
FIG. 8 is a block diagram showing a lip reading device 100 according to the second modification.
As shown in Figure 8, the lip reading device 100 in this variant example 2 has three lip reading units: a first single-angle compatible lip reading unit 131 capable of highly accurate lip reading processing at a corresponding angle of 0°, a second single-angle compatible lip reading unit 136 capable of highly accurate lip reading processing at a corresponding angle of 45°, and a multiple-angle compatible lip reading unit 134 capable of highly accurate lip reading processing at two corresponding angles of 0° and 30°.

The lip reading device 100 in this second modification includes an angle estimation unit 112 as an imaging direction estimation unit that estimates the imaging direction of the lip image data input to the image input unit 111. For example, a large amount of face image data captured from various angles and stored in the learning data storage medium 32 described above is used to learn a model that estimates the imaging angle of the input face image data through machine learning or deep learning. The angle estimation model thus generated is then executed by the computer of the angle estimation unit 112 to estimate the imaging direction of the lip image data input to the image input unit 111.

When the angle estimation unit 112 of this modified example 2 receives face image data from the image input unit 111, it estimates the imaging angle of the face image data and outputs a certainty factor indicating the likelihood for each preset angle as an angle estimation result. For example, when the preset angles are 0°, 30°, 45°, and 60°, the angle estimation unit 112 outputs information such as a certainty factor of 0.3 for 0°, a certainty factor of 0.4 for 30°, a certainty factor of 0.2 for 45°, and a certainty factor of 0.1 for 60° as an estimation result of the imaging angle of the input face image data.

In this second modification, the angle estimation result of the angle estimation unit 112 is sent to the lip reading result integration unit 141. The lip reading result integration unit 141 in this second modification uses the angle estimation result received from the angle estimation unit 112 to integrate the lip reading process results obtained by the three lip reading units 131, 136, and 134, and generates a recognition result of the speaker's speech content as the final lip reading result.

As an example of the integration process in this modified example 2, for example, the reliability score of each lip reading process result obtained by the three lip reading units 131, 136, and 134 is multiplied by the certainty of the estimated angle that matches the corresponding angle of each lip reading unit included in the angle estimation result of the angle estimation unit 112. For example, in the above example, for the first single-angle compatible lip reading unit 131 whose corresponding angle is 0°, the reliability score is multiplied by 0.3, for the second single-angle compatible lip reading unit 136 whose corresponding angle is 45°, the reliability score is multiplied by 0.2, and for the multiple-angle compatible lip reading unit 134 whose corresponding angles are 0° and 30°, the reliability score is multiplied by 0.4 using the higher of the certainties of 0° and 30°.

After using the angle estimation result in this way, the lip reading result integration unit 141 may, as in the first embodiment described above, take as the integrated result the speech content candidate with the highest reliability score among the common speech content candidates included in all of the lip reading process results obtained by the three lip reading units 131, 136, and 134. Alternatively, two or more speech content candidates in descending order of reliability score may be taken as the integrated result. Also, for example, after using the angle estimation result, the lip reading result integration unit 141 may calculate the total reliability score for each speech content candidate included in the lip reading process results obtained by the three lip reading units 131, 136, and 134, and take as the integrated result the speech content candidate with the highest total value. Alternatively, two or more speech content candidates for which the total reliability score has been calculated may be taken as the integrated result.

According to this second modification, the lip reading results of each lip reading unit 131, 136, 134 can be weighted using the angle estimation result obtained by estimating the imaging angle of the face image data from the image input unit 111. In other words, the angle estimation result from the angle estimation unit 112 is used, and the lip reading results of a lip reading unit with a higher degree of certainty of an estimated angle that matches the corresponding angle are weighted more heavily, thereby increasing the influence of the lip reading results of the lip reading unit on the recognition result of the speech content. This makes it possible to achieve a higher accuracy in the final lip reading result (recognition result of the speech content) obtained by the lip reading result integration unit 141.

[Embodiment 2]
Next, an embodiment in which the present invention is applied to a multimodal speech recognition device as a speech content recognition device (hereinafter, this embodiment will be referred to as "embodiment 2") will be described.
The multimodal speech recognition device of the second embodiment uses two types of speech content recognition processing, namely lip reading processing and speech recognition processing, to output a recognition result of the content of an utterance spoken by a speaker.

FIG. 9 is a block diagram showing a multimodal speech recognition device according to the second embodiment.
The multimodal speech recognition device 300 of the second embodiment is composed of a lip-reading recognition processing unit 101 , a speech recognition processing unit 201 , and a recognition result integration unit 301 .

The configuration of the lip reading recognition processing unit 101 can be the same as that of the lip reading device 100 of the first embodiment described above. The example in FIG. 9 uses the configuration of the lip reading device 100 shown in FIG. 1.

The voice recognition processing unit 201 is mainly composed of a voice input unit 211 and a voice recognition unit 231.

The voice input unit 211 accepts input of voice data of a speaker who is speaking. In this embodiment 2, the voice input unit 211 is connected to a microphone 3 that collects the speaker's voice and a storage medium 2 that stores the speaker's voice data, so that they can communicate with each other via wired or wireless communication. Real-time voice data of the speaker currently speaking is input from the microphone 3 to the voice input unit 211. The storage medium 2 stores voice data from when the speaker spoke in the past, and past voice data is input from the storage medium 2 to the voice input unit 211.

The voice input unit 211 processes the input voice data as necessary to make it correspond to the input of the voice recognition unit 231, and passes it to the voice recognition unit 231. For example, it extracts a voice signal from which noise has been removed from the input voice data, and passes the data of the voice signal to the voice recognition unit 231.

The voice recognition unit 231 of the present embodiment 2 executes a predetermined voice recognition program on a computer, thereby executing voice recognition processing on the voice data input to the voice input unit 211 and generating a voice recognition result. The voice recognition program of the present embodiment 2 uses a voice recognition model (trained model) trained using training data including the speaker's voice data, but a voice recognition program programmed by a programmer may also be used.

The speech recognition model (trained model) in this embodiment 2 estimates the speaker's speech content from the input data (speech data), and there are no particular restrictions on the format of the estimation result (speech recognition result) output from the speech recognition model, as in the case of the machine lip reading model described above. As an example, in this embodiment 2, data including one or more speech content candidates (such as a single word or word sequence) and a reliability score for each of the speech content candidates is output as a speech recognition result for the input data (speech data) in accordance with the format of the lip reading recognition processing unit 101.

The recognition result integration unit 301 integrates the recognition result (lip reading result) output from the lip reading result integration unit 141 of the lip reading recognition processing unit 101 and the recognition result (voice recognition result) output from the voice recognition unit 231 of the voice recognition processing unit 201, and outputs the final recognition result of the spoken content.

There are no particular limitations on the integration process performed by the recognition result integration unit 301, so long as it is a process of integrating the recognition result (lip reading result) of the lip reading recognition processing unit 101 and the recognition result (voice recognition result) of the voice recognition processing unit 201 so as to obtain a highly accurate recognition result (for example, so that the reliability score of the correct speech content becomes relatively high).

As an example of the integration process performed by the recognition result integration unit 301, for example, the integration result may be an utterance content candidate with the highest reliability score among common utterance content candidates included in both of the recognition results of the two recognition processing units 101 and 201. Alternatively, the integration result may be two or more utterance content candidates in descending order of reliability score.
Also, for example, the total value of the reliability scores for each of the utterance content candidates included in the recognition results of the two recognition processing units 101 and 201 may be calculated, and the utterance content candidate with the highest total value may be used as the integrated result. Alternatively, two or more utterance content candidates for which the total value of the reliability scores has been calculated may be used as the integrated result.

As another example of the integration process performed by the recognition result integration unit 301, a trained model (integration model) that obtains recognition results from intermediate representations in the two recognition processing units 101, 201 is trained in advance, for example, by a neural network model. Then, the intermediate representations (vectors, etc.) included in the recognition results obtained by the two recognition processing units 101, 201 are concatenated to generate one intermediate representation, and the trained integration model is executed by the computer of the recognition result integration unit 301 to obtain recognition results of one or more utterance contents from the one intermediate representation, which is used as the integration result. Alternatively, the recognition results of one or more utterance contents from the one intermediate representation may include their respective confidence scores.

The multimodal speech recognition device 300 of this embodiment 2 has a lip reading processing unit 101 that adopts the configuration of the lip reading device 100 of the above-mentioned embodiment 1, so that it is possible to obtain lip reading results with high accuracy.

In addition, the multimodal speech recognition device 300 of the second embodiment obtains recognition results using two different types of speech content recognition methods, namely, a lip reading device and a voice recognition device, and integrates these recognition results to output the final recognition result of the speech content. Therefore, even in a situation where the speech recognition processing unit 201 has difficulty recognizing the speaker's speech content with high accuracy (e.g., a noisy environment, a conference environment where multiple speakers often speak at the same time, etc.), the lip reading recognition processing unit 101 can recognize the speech content with high accuracy. Also, even in a situation where the lip reading recognition processing unit 101 has difficulty recognizing the speaker's speech content with high accuracy (e.g., a dark environment with low lighting, an environment where it is difficult to capture the speaker's lips due to the speaker's movement, etc.), the speech recognition processing unit 201 can recognize the speech content with high accuracy.

In this way, the multimodal speech recognition device 300 of the second embodiment can realize a highly robust speech content recognition device in which the recognition accuracy of the speaker's speech content is less affected by the speaker's environment. Specifically, such a multimodal speech recognition device 300 can be suitably used as an automatic meeting minutes generation system in a conference room or online, and as a speech input interface for smartphones.

[Embodiment 3]
Next, we will explain one embodiment of a training data collection system (hereinafter, this embodiment will be referred to as "embodiment 3") that collects training data for constructing a machine lip reading model used in the lip reading recognition processing unit 101 and a speech recognition model used in the speech recognition processing unit 201 in the multimodal speech recognition device 300 of embodiment 2 described above.

FIG. 10 is an explanatory diagram showing the configuration of a learning data collection system in the third embodiment.
The learning data collection system of the present embodiment 3 includes a camera array 31 having a plurality of imaging devices, a recording microphone 21 as a voice acquisition device, a display 42 as an instruction device, and a control device 43. In addition, the learning data collection system of the present embodiment 3 includes a reporting unit 41 and a learning data storage medium 32 as a storage device.

The camera array 31 is composed of multiple cameras (imaging devices) 31-1 to 31-10 that capture images of a speaker at a predetermined position from multiple different imaging directions. In this embodiment 3, as shown in FIG. 11, 10 recording cameras 31-1 to 31-10 are arranged at equal intervals. Specifically, if the imaging angle is the angle around the vertical axis based on the imaging direction when the speaker's face is imaged from the front direction (0°), the recording cameras 31-1 to 31-10 are arranged at 10° intervals between 0° and 90°. This camera array 31 makes it possible for the recording cameras 31-1 to 31-10 to simultaneously capture images of the lips of the speaker from their respective imaging angles.

The camera array 31 is connected to a learning data storage medium 32, and the speaker's facial image data (lip image data) captured by each recording camera 31-1 to 31-10 is stored and accumulated in the learning data storage medium 32. The camera array 31 is also connected to a control device 43, and the image capture operation is controlled by the control device 43.

There is no particular limit to the number of cameras that make up the camera array 31. In addition, in the third embodiment, the arrangement of the cameras that make up the camera array 31 is such that the multiple recording cameras 31-1 to 31-10 are arranged along a horizontal plane so that the imaging angles around the vertical axis are different from each other, but this is not limited to this. For example, the multiple recording cameras 31-1 to 31-10 may be arranged so that the imaging angles around the horizontal axis or around an inclined axis inclined relative to the vertical axis are different from each other.

The recording microphone 21 acquires the voice of the speaker who is the subject of imaging by the camera array 31. The recording microphone 21 is connected to a learning data storage medium 32, and the voice data collected by the recording microphone 21 is stored and accumulated in the learning data storage medium 32. The recording microphone 21 is also connected to a control device 43, and its operation is controlled by the control device 43. The recording microphone 21 may be, for example, a lapel microphone installed on the speaker's collar, or a stand microphone may be installed near the speaker. In this case, it is desirable to install the recording microphone 21 so as not to get in the way when the speaker's lip image is captured by each of the recording cameras 31-1 to 31-10 of the camera array 31.

If the cameras that make up the camera array 31 have a function as an audio capture device, the camera's audio capture device may be used as the recording microphone 21.

The display 42 displays the spoken instructions to the speaker. The display contents of the display 42 are controlled by a control device 43 connected by wire or wirelessly, and under the control of the control device 43, the display 42 displays the spoken instructions to the speaker, instructs the speaker to repeat the utterance, etc.

The operation of the notification unit 41 is controlled by a control device 43 connected by wire or wirelessly, and under the control of the control device 43, the notification unit 41 notifies the speaker of the start and end timing of speech using light, sound, etc.

As described above, the learning data storage medium 32 stores the speaker's facial image data captured by each of the recording cameras 31-1 to 31-10 of the camera array 31 and the speaker's voice data collected by the recording microphone 21 in a format that allows a time series to be identified. Specifically, the speech start synchronization signal and speech start synchronization signal emitted by the reporting unit 41 are embedded in the facial image data captured by each of the recording cameras 31-1 to 31-10 and the voice data collected by the recording microphone 21. This allows the facial image data and voice data stored in the learning data storage medium 32 to be compared with the time when the speaker spoke the speech content instructed by the display 42. Therefore, the speech content instructed to the speaker via the display 42 is associated with the speaker's facial image data and voice data at the time when the speech content was spoken.

The learning data storage medium 32 may be a distributed storage device consisting of multiple storage media connected to each of the recording cameras 31-1 to 31-10 and the recording microphone 21 of the camera array 31, or may be composed of a single storage device (such as a file server).

The control device 43 is composed of an information processing device such as a personal computer, and controls the entire system. Specifically, the control device 43 starts recording with the camera array 31 and the recording microphone 21 in response to instructions from the operator, issues a speech start synchronization signal through the notification unit 41 and instructs the speaker when to start speaking, displays the speech content on the display 42, and causes the speaker to speak the speech content. It also issues a speech end synchronization signal through the notification unit 41 and instructs the speaker when to end speaking, and ends recording with the camera array 31 and the recording microphone 21. The control device 43 also instructs the speaker to repeat the speech through the display 42 in response to instructions from the operator.

According to the learning data collection system of the third embodiment, the lip images of the speaker who is speaking the utterance content instructed by the display 42 are simultaneously captured from different imaging angles by the recording cameras 31-1 to 31-10, and the speaker's voice at that time is acquired by the recording microphone 21, and these can be stored in the learning data storage medium 32. This makes it possible to quickly and rapidly collect a large amount of face image data (lip image data) captured from different imaging angles and the corresponding voice data. Therefore, it is possible to easily and quickly collect a large amount of learning data required to construct the machine lip reading model used in the lip reading recognition processing unit 101 and the voice recognition model used in the voice recognition processing unit 201 in the multimodal voice recognition device 300 of the above-mentioned second embodiment.

The learning data collection system of the third embodiment is also useful for easily and quickly collecting a large amount of learning data required to construct a machine lip reading model used in the lip reading device 100 of the first embodiment described above. In this case, recording using the recording microphone 21 is not necessarily required.

The above description is merely an example, and the present invention provides unique effects for each of the following aspects.
[First aspect]
The first aspect is a speech content recognition device (e.g., lip reading device 100, multimodal speech recognition device 300) that recognizes the speech content of a speaker, and includes an input unit (e.g., image input unit 111) that inputs lip image data (e.g., face image data) of the speaker, a plurality of lip reading units (e.g., single-angle compatible lip reading units 131, 132, 133, 136, multiple-angle compatible lip reading units 134, 135, 137) that have high lip reading accuracy for lip image data captured from corresponding directions (e.g., corresponding angles), and an integration generation unit (e.g., lip reading result integration unit 141) that integrates the lip reading processing results of the plurality of lip reading units for the lip image data input to the input unit and generates a recognition result of the speaker's speech content based on the result of the integration, and is characterized in that at least one of the plurality of lip reading units is configured to include a direction in the corresponding direction that is not included in the corresponding direction of any of the other lip reading units.
In general, a lip reading unit that performs lip reading processing of lip image data input to an input unit can perform lip reading processing with high accuracy (accuracy that meets the user's requirements) when the imaging direction of the input lip image data is a specific direction (corresponding direction), and the recognition accuracy of the spoken content is high. Specifically, for example, a lip reading unit whose corresponding direction is the front direction of the speaker's face has high lip reading accuracy when lip image data obtained when the speaker's face is imaged from the front is input. On the other hand, when lip image data obtained when the speaker's face is imaged from the side or diagonally is input to this lip reading unit, the lip reading accuracy drops, and in many cases high lip reading accuracy cannot be obtained.
In this embodiment, a lip reading process is performed on lip image data input to an input unit using a plurality of lip reading units having different corresponding directions. At this time, at least one of the plurality of lip reading units used in this embodiment includes a corresponding direction (a direction in which high lip reading accuracy can be obtained) that is not included in the corresponding direction of any of the other lip reading units. Therefore, if the lip image data is captured from a direction that matches one of the corresponding directions of these lip reading units, the lip reading unit can obtain high lip reading accuracy, and the speech content can be recognized with high accuracy from the lip reading process result. Therefore, according to this embodiment, the imaging directions of the lip image data that can recognize the speech content with high accuracy can be increased by the number of corresponding directions of these plurality of lip reading units.
Here, when lip image data captured from a direction that does not match any of the corresponding directions of the multiple lip readers is input to the input unit, the lip reading process of each lip reader cannot obtain sufficient lip reading accuracy. Therefore, if only the lip reading process result of one of the lip readers is used, the speech content of such lip image data cannot be recognized with high accuracy.
Therefore, in this embodiment, the integration generating unit integrates the results of the lip reading processes of the multiple lip reading units for the lip image data input to the input unit, and generates a recognition result of the speech content based on the integration result. As a result, even if the results of the lip reading processes of the individual lip reading units show that the reliability (confidence) of the correct speech content is not significantly higher than the reliability of the other incorrect speech content, or conversely, is lower, the integration of these lip reading results can highlight the reliability of the correct speech content and create a significant difference from the reliability of the other incorrect speech content. For example, by accumulating the reliability included in the results of the lip reading processes of the individual lip reading units for each speech content candidate, it is possible to derive a reliability of the correct speech content that is significantly different from the reliability of the other incorrect speech content. Therefore, by integrating the results of the lip reading processes of the multiple lip reading units described above and generating a recognition result of the speech content based on the integration result, it is possible to recognize the speech content with high accuracy for lip image data captured from a direction in which sufficient lip reading accuracy cannot be obtained by the lip reading processes of the individual lip reading units.
Therefore, according to this aspect, it is possible to recognize the speech content with high accuracy for various types (imaging directions) of lip image data, which exceeds the number of corresponding directions of the above-mentioned multiple lip reading units.

[Second aspect]
The second aspect is characterized in that in the first aspect, the multiple lip reading units include a unidirectional lip reading unit (e.g., single-angle compatible lip reading units 131, 132, 133, 136) that correspond to one direction.
A unidirectional lip reader that supports one direction can be constructed more easily than a multi-directional lip reader that supports two or more directions, and therefore a speech content recognition device can be easily realized.

[Third aspect]
The third aspect is characterized in that, in the first or second aspect, the multiple lip reading units include a multi-directional lip reading unit (e.g., multiple angle compatible lip reading units 134, 135, 137) having two or more corresponding directions.
This allows the number of lip readers to be reduced, making it easier to realize a simple speech content recognition device.

[Fourth aspect]
A fourth aspect is characterized in that, in any of the first to third aspects, the device has a data conversion unit (e.g., angle conversion units 121 to 123) that generates data converted based on lip image data input to the input unit so that the imaging direction corresponds to the corresponding direction of at least one of the multiple lip reading units, and the at least one lip reading unit performs lip reading processing using the data converted by the data conversion unit.
According to this, since the lip reading units are each input with lip image data of an imaging direction that matches the corresponding direction, each lip reading unit can obtain a lip reading process result with high accuracy. As a result, the recognition result of the speech content obtained by integrating these lip reading process results by the integration generation unit can also be highly accurate.

[Fifth aspect]
A fifth aspect is characterized in that, in any of the first to fourth aspects, it has an imaging direction estimation unit (e.g., an angle estimation unit 112) that estimates the imaging direction of the lip image data input to the input unit, and the integration generation unit generates a recognition result of the speech content of the speaker using the estimation result of the imaging direction estimation unit.
According to this, the lip reading process results of each lip reading unit can be weighted using the estimation result of the imaging direction estimation unit estimating the imaging angle of the face image data from the input unit. In other words, using the estimation result of the imaging direction estimation unit, the lip reading process results of the lip reading unit with a higher degree of certainty of the estimated angle matching the corresponding direction can be weighted more heavily, thereby increasing the influence of the lip reading process results of the lip reading unit on the recognition result of the utterance content. This makes it possible to increase the accuracy of the recognition result of the utterance content obtained by the integration generation unit.

[Sixth aspect]
A sixth aspect is characterized in that, in any of the first to fifth aspects, the multiple lip reading units generate lip reading processing results including one or more speech content candidates estimated by the lip reading process and reliability information (e.g., a reliability score) for each speech content candidate, and the integration generation unit integrates the reliability information included in each lip reading processing result of the multiple lip reading units for each speech content candidate.
This makes it possible to obtain a highly accurate recognition result of the speech content in the integration generation unit using a simple method.

[Seventh aspect]
A seventh aspect is characterized in that, in any of the first to fifth aspects, the multiple lip reading units generate intermediate information (e.g., intermediate expressions) for estimating candidate speech content as a lip reading processing result by lip reading processing, and the integration generation unit integrates the intermediate information contained in each lip reading processing result of the multiple lip reading units.
According to this, the lip reading processing results of multiple lip reading units can be integrated using a trained model (integrated model) trained using intermediate information learning data, and the integrated generation unit can obtain highly accurate recognition results of the speech content.

[Eighth aspect]
The eighth aspect is characterized in that, in any of the first to seventh aspects, it has a voice recognition unit (e.g., the voice recognition processing unit 201) that recognizes the content of the speaker's speech from the speaker's voice data, and the integration generation unit (e.g., the lip reading result integration unit 141 and the recognition result integration unit 301) generates a recognition result of the content of the speaker's speech using the recognition result of the voice recognition unit.
This makes it possible to realize a multimodal speech content recognition device (e.g., the multimodal speech recognition device 300) that outputs a recognition result of the speech content uttered by a speaker using two types of speech content recognition processing, namely lip reading processing and speech recognition processing. This makes it possible to realize a speech content recognition device with excellent robustness in which the recognition accuracy of the speaker's speech content is less affected by the speaker's environment.

[Ninth aspect]
A ninth aspect is characterized in that in any of the first to eighth aspects, the plurality of lip reading units include a lip reading unit that performs lip reading processing of lip image data input to the input unit by having a computer execute a machine lip reading model trained using training data including lip image data of a speaker.
According to this, lip reading processing is performed using a machine lip reading model, so that lip reading processing results with higher accuracy can be obtained.

[Tenth aspect]
The tenth aspect is a learning data collection system that collects learning data for constructing the machine lip reading model used in the speech content recognition device of the ninth aspect, characterized in that it has a plurality of imaging devices (e.g., recording cameras 31-1 to 31-10) that image a speaker at a predetermined position from a plurality of different imaging directions, a voice acquisition device (e.g., recording microphone 21) that acquires the voice of the speaker, an instruction device (e.g., display 42) that instructs the speaker on the content of the utterance, and a control device 43 that executes control to have the instruction device instruct the content of the utterance, simultaneously image images of the lips of the speaker uttering the instructed content by the plurality of imaging devices, acquire the voice of the speaker by the voice acquisition device, and store the obtained lip image data and voice data in a storage device (e.g., learning data storage medium 32).
According to this, the lip images of the speaker who speaks the utterance contents instructed by the instruction device can be simultaneously captured from different imaging angles by the multiple imaging devices, and the voice of the speaker at that time can be acquired by the voice acquisition device and stored in the storage device. This makes it possible to quickly and massively collect lip image data captured from different imaging angles and corresponding voice data. Therefore, it is possible to easily and quickly collect a large amount of learning data required to build a machine lip reading model used in the lip reading process and a voice recognition model used in the voice recognition process in the multimodal utterance content recognition device of the ninth aspect described above.

[Eleventh aspect]
An eleventh aspect is a method for recognizing the content of a speaker's utterance by a speech content recognition device, comprising: an input step of inputting lip image data of the speaker into the speech content recognition device; a lip reading step in which the speech content recognition device performs lip reading processing of the lip image data input in the input step using a plurality of lip reading units having high lip reading accuracy for lip image data captured from corresponding directions; and an integration generation step in which the speech content recognition device integrates the lip reading processing results of the plurality of lip reading units obtained by the lip reading step and generates a recognition result of the speaker's speech content based on the result of the integration, wherein at least one lip reading unit of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units.
According to this aspect, it is possible to recognize the speech content with high accuracy for various types (imaging directions) of lip image data, which exceeds the number of corresponding directions of the above-mentioned multiple lip reading units.

[Twelfth aspect]
A twelfth aspect is a program executed on a computer of a speech content recognition device that recognizes the content of a speaker's speech, which causes the computer to function as an integration generation means that integrates the results of lip reading processing performed by multiple lip reading means with high lip reading accuracy on lip image data captured from corresponding directions input to the speech content recognition device, and generates a recognition result for the content of the speaker's speech based on the result of the integration, and is characterized in that at least one lip reading unit among the multiple lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units.
According to this aspect, it is possible to recognize the speech content with high accuracy for various types (imaging directions) of lip image data, which exceeds the number of corresponding directions of the above-mentioned multiple lip reading units.

[Thirteenth aspect]
A thirteenth aspect is a method for collecting learning data for constructing the machine lip reading model used in the speech content recognition device by the learning data collection system of the tenth aspect, characterized in that the instruction device is caused to indicate the speech content, lip images of a speaker speaking the indicated speech content are simultaneously captured by the multiple imaging devices, and the speaker's voice is acquired by the voice acquisition device, and the obtained lip image data and voice data are stored in a storage device.
According to this aspect, it is possible to easily and quickly collect a large amount of training data required to construct a machine lip reading model used in the lip reading process and a speech recognition model used in the speech recognition process in the multimodal speech content recognition device of the ninth aspect described above.

[14th aspect]
A fourteenth aspect is a program executed by a computer of the control device in the learning data collection system of the tenth aspect, which causes the instructing device to instruct the speech content, simultaneously captures lip images of a speaker speaking the instructed speech content using the multiple imaging devices, acquires the speaker's voice using the voice acquisition device, and causes the obtained lip image data and voice data to function as a control means that executes control to store in a storage device.
According to this aspect, it is possible to easily and quickly collect a large amount of training data required to construct a machine lip reading model used in the lip reading process and a speech recognition model used in the speech recognition process in the multimodal speech content recognition device of the ninth aspect described above.

1: Camera 2: Storage medium 3: Microphone 21: Recording microphone 31: Camera array 31-1 to 31-10: Recording camera 32: Learning data storage medium 33-1: First learning unit 33-2: Second learning unit 41: Reporting unit 42: Display 43: Control device 100: Lip reading device 101: Lip reading recognition processing unit 111: Image input unit 112: Angle estimation unit 121 to 123: Angle conversion unit 131, 132, 133, 136: Single angle compatible lip reading unit 134, 135, 137: Multiple angle compatible lip reading unit 141: Lip reading result integration unit 201: Speech recognition processing unit 211: Speech input unit 231: Speech recognition unit 300: Multimodal speech recognition device 301: Recognition result integration unit

A. Koumparoulis et al., "Deep view2view mapping for view-invariant lipreading", IEEE SLT, 2018, p.588-594 S. Petridis et al., "End-to-end Multiview Lip Reading", IEEE ICASSP, 2018, p.6548-6552

Claims (16)

A speech content recognition device that recognizes the content of a speaker's speech,
an input unit for inputting lip image data of a speaker;
A plurality of lip reading units having high lip reading accuracy for lip image data captured from corresponding directions;
an integration unit that integrates the lip reading process results of the plurality of lip reading units for the lip image data input to the input unit and generates a recognition result of the speech content of the speaker based on the integration result;
At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units;
The speech content recognition device, wherein the plurality of lip reading units includes a multi-directional lip reading unit having two or more corresponding directions. A speech content recognition device that recognizes the content of a speaker's speech,
an input unit for inputting lip image data of a speaker;
A plurality of lip reading units having high lip reading accuracy for lip image data captured from corresponding directions;
an integration unit that integrates the lip reading process results of the plurality of lip reading units for the lip image data input to the input unit and generates a recognition result of the speech content of the speaker based on the integration result;
At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units;
an imaging direction estimation unit that estimates a plurality of imaging directions of the lip image data input to the input unit and certainty factor information for each imaging direction;
the integration generation unit integrates the lip reading process results using a plurality of estimation results from the imaging direction estimation unit, and generates a recognition result of the speaker's speech content based on the integration result . A speech content recognition device that recognizes the content of a speaker's speech,
an input unit for inputting lip image data of a speaker;
A plurality of lip reading units having high lip reading accuracy for lip image data captured from corresponding directions;
an integration unit that integrates the lip reading process results of the plurality of lip reading units for the lip image data input to the input unit and generates a recognition result of the speech content of the speaker based on the integration result;
At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units;
the plurality of lip reading units generate a lip reading process result including one or more speech content candidates estimated by the lip reading process and reliability information for each of the speech content candidates;
The speech content recognition device, wherein the integration/generation unit integrates reliability information included in each of the lip reading processing results of the plurality of lip reading units for each speech content candidate. A speech content recognition device that recognizes the content of a speaker's speech,
an input unit for inputting lip image data of a speaker;
A plurality of lip reading units having high lip reading accuracy for lip image data captured from corresponding directions;
an integration unit that integrates the lip reading process results of the plurality of lip reading units for the lip image data input to the input unit and generates a recognition result of the speech content of the speaker based on the integration result;
At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units;
the plurality of lip reading units generate a lip reading process result including one or more speech content candidates estimated by the lip reading process and reliability information for each of the speech content candidates;
a speech recognition unit that outputs, as a recognition result, speech content candidates of the speaker and reliability information for each of the speech content candidates from the speech data of the speaker;
The speech content recognition device is characterized in that the integration generation unit integrates the lip reading processing results of the multiple lip reading units and the recognition results of the voice recognition unit, and generates a recognition result of the speaker's speech content based on the result of the integration . 5. The speech recognition apparatus according to claim 1,
The speech content recognition device, wherein the plurality of lip reading units include a unidirectional lip reading unit corresponding to one direction. 6. The speech recognition apparatus according to claim 1,
a data conversion unit that generates data converted based on the lip image data input to the input unit so that the imaging direction corresponds to a corresponding direction of at least one of the plurality of lip reading units;
The speech content recognition device according to claim 1, wherein the at least one lip reading unit performs lip reading processing using data converted by the data conversion unit. 5. The speech recognition apparatus according to claim 1, 2 or 4,
the plurality of lip reading units generate intermediate information for estimating speech content candidates as a lip reading process result by lip reading processing;
The speech content recognition device, wherein the integration/generation unit integrates intermediate information included in each of the lip reading processing results of the plurality of lip reading units. 8. The speech recognition apparatus according to claim 1,
The speech content recognition device is characterized in that the multiple lip reading units include a lip reading unit that performs lip reading processing of the lip image data input to the input unit by having a computer execute a machine lip reading model trained using training data including the speaker's lip image data. A method for recognizing a speaker's utterance content by a speech recognition device, comprising:
an input step of inputting lip image data of a speaker into the speech recognition device;
a lip reading process in which the speech content recognition device performs lip reading processing on the lip image data input in the input process using a plurality of lip reading units having high lip reading accuracy for the lip image data captured from corresponding directions;
the speech content recognition device has an integration generating step of integrating the lip reading processing results of the plurality of lip reading units obtained by the lip reading step and generating a recognition result of the speech content of the speaker based on the integration result,
At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units;
The method according to claim 1, wherein the plurality of lip readers includes a multi-directional lip reader having two or more corresponding directions. A method for recognizing a speaker's utterance content by a speech recognition device, comprising:
an input step of inputting lip image data of a speaker into the speech recognition device;
a lip reading process in which the speech content recognition device performs lip reading processing on the lip image data input in the input process using a plurality of lip reading units having high lip reading accuracy for the lip image data captured from corresponding directions;
the speech content recognition device has an integration generating step of integrating the lip reading processing results of the plurality of lip reading units obtained by the lip reading step and generating a recognition result of the speech content of the speaker based on the integration result,
At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units;
an imaging direction estimating step of estimating a plurality of imaging directions of the lip image data inputted in the input step and pieces of certainty information for each imaging direction;
The method is characterized in that in the integration and generation step, the lip reading process results are integrated using multiple estimation results from the imaging direction estimation step, and a recognition result of the speaker's speech content is generated based on the integration result . A method for recognizing a speaker's utterance content by a speech recognition device, comprising:
an input step of inputting lip image data of a speaker into the speech recognition device;
a lip reading process in which the speech content recognition device performs lip reading processing on the lip image data input in the input process using a plurality of lip reading units having high lip reading accuracy for the lip image data captured from corresponding directions;
the speech content recognition device has an integration generating step of integrating the lip reading processing results of the plurality of lip reading units obtained by the lip reading step and generating a recognition result of the speech content of the speaker based on the integration result,
At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units;
the plurality of lip reading units generate a lip reading process result including one or more speech content candidates estimated by the lip reading process and reliability information for each of the speech content candidates;
The method according to the present invention, characterized in that in the integrating and generating step, reliability information included in each of the lip reading processing results of the plurality of lip reading units is integrated for each speech content candidate. A method for recognizing a speaker's utterance content by a speech recognition device, comprising:
an input step of inputting lip image data of a speaker into the speech recognition device;
a lip reading process in which the speech content recognition device performs lip reading processing on the lip image data input in the input process using a plurality of lip reading units having high lip reading accuracy for the lip image data captured from corresponding directions;
the speech content recognition device has an integration generating step of integrating the lip reading processing results of the plurality of lip reading units obtained by the lip reading step and generating a recognition result of the speech content of the speaker based on the integration result,
At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units;
the plurality of lip reading units generate a lip reading process result including one or more speech content candidates estimated by the lip reading process and reliability information for each of the speech content candidates;
a speech recognition step of outputting, as a recognition result, speech content candidates of the speaker and reliability information for each of the speech content candidates from the speech data of the speaker;
The method is characterized in that in the integration and generation process, the lip reading processing results of the multiple lip reading units and the recognition results of the voice recognition process are integrated, and a recognition result of the speaker's speech content is generated based on the result of the integration . A program executed on a computer of an utterance content recognition device for recognizing the content of a speaker's utterance,
The computer is caused to function as an integration generating means for integrating lip reading results of lip reading processes performed by a plurality of lip reading means having high lip reading accuracy on lip image data captured from corresponding directions, inputted into the speech content recognition device, and generating a recognition result of the speech content of the speaker based on the integration result,
At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units,
The program, wherein the plurality of lip reading means includes a multi-directional lip reading means having two or more corresponding directions. A program executed on a computer of an utterance content recognition device for recognizing the content of a speaker's utterance,
the computer is caused to function as an integration generating means for integrating lip reading results obtained by lip reading lip image data input to the speech content recognition device using a plurality of lip reading means each having a high lip reading accuracy for lip image data captured from corresponding directions, and generating a recognition result for the speech content of the speaker based on the integration result; and an imaging direction estimating means for estimating a plurality of imaging directions of the input lip image data and certainty information for each imaging direction,
At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units,
The program, wherein the integration generation means integrates the lip reading process results using multiple estimation results from the imaging direction estimation means, and generates a recognition result of the speaker's speech content based on the integration result . A program executed on a computer of an utterance content recognition device for recognizing the content of a speaker's utterance,
The computer is caused to function as an integration generating means for integrating lip reading results of lip reading processes performed by a plurality of lip reading means having high lip reading accuracy on lip image data captured from corresponding directions, inputted into the speech content recognition device, and generating a recognition result of the speech content of the speaker based on the integration result,
At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units,
the plurality of lip reading means generate a lip reading process result including one or more speech content candidates estimated by the lip reading process and reliability information for each of the speech content candidates;
The program, wherein the integrating and generating means integrates reliability information included in each lip reading process result of the plurality of lip reading means for each speech content candidate. A program executed on a computer of an utterance content recognition device for recognizing the content of a speaker's utterance,
The computer is caused to function as an integration generating means for integrating lip reading results obtained by lip reading lip image data input to the speech content recognition device using a plurality of lip reading means each having high lip reading accuracy for lip image data captured from corresponding directions, and generating a recognition result of the speaker's speech content based on the integration result, and a voice recognition means for outputting, as recognition results, speech content candidates of the speaker and reliability information for each of the speech content candidates from the voice data of the speaker,
At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units,
the plurality of lip reading means generate a lip reading process result including one or more speech content candidates estimated by the lip reading process and reliability information for each of the speech content candidates;
The program, wherein the integration and generation means integrates the lip reading processing results of the multiple lip reading means and the recognition results of the voice recognition means, and generates a recognition result of the speaker's speech content based on the result of the integration .

Images