JP7565041B2 - Speech section extraction method, speech section extraction program, and speech section extraction device - Google Patents

Description

Article 30, paragraph 2 of the Patent Act applies 1. Announcement in a publication Publisher: Executive Committee of the 2019 Tohoku-section Joint Convention of the Institutes of Electrical and Related Engineers Publication name: Proceedings of the 2019 Tohoku-section Joint Convention of the Institutes of Electrical and Related Engineers, 2F07 Publication date: August 22, 2019 2. Presentation at a meeting Name of the meeting: 2019 Tohoku-section Joint Convention of the Institutes of Electrical and Related Engineers Date held: August 23, 2019 3. Announcement on a website Posting address: https://sites.google.com/ com/view/miru2020/extended-abstracts The 23rd Image Recognition and Understanding Symposium MIRU2020 Extended Abstracts Abstracts Proceedings downloaded from the above address Date of publication: July 28, 2020 4. Presentation at the meeting Meeting name: 23rd Image Recognition and Understanding Symposium MIRU2020 Date held: August 4, 2020

The present invention relates to a speech section extraction method, a speech section extraction program, and a speech section extraction device.

In recent years, there has been a trend to improve work efficiency and the working environment in order to realize work style reform. One of the measures to improve work in the workplace is to improve the efficiency of work and meetings.
Meeting minutes are taken to record the contents and agreements discussed and to share decisions and their history, and the minutes created contribute to improving the quality of subsequent meetings and the efficiency of other work. An automatic minutes-creation system built using speech recognition technology can reduce human error in creating minutes and reduce the number of people and time required to create minutes. Furthermore, technology that automatically identifies the speaker for each statement and technology that improves speech recognition accuracy in such an automatic minutes-creation system contributes to reducing the manpower required to create minutes, and contributes to the efficiency of meetings and work.

Patent Document 1 discloses a minutes distribution system that can acquire image and audio information from a meeting and distribute minutes with an appropriate amount of information to the distribution destination. The system acquires video and audio from the meeting venue and has a database that records the recording start time and the filming start time, and can send automatically generated minutes by email to those involved in the meeting. However, although the system is equipped with a function to estimate speech sections based on audio, this function is not intended to identify the speaker, but rather is a function to determine the start and end times of speech. Furthermore, no consideration has been given to extracting speech sections using lip movement in a facial image, and the system is not equipped with a function for identifying the speaker.

In addition, when lip movement is used to extract speech periods, periods when the subject is not speaking may be included in the detected voice period, making it difficult to properly detect the voice period. Patent Document 2 discloses detection of a vocalization period and a breathing period immediately before speaking using lip movement and voice information. However, as this method assumes one-to-one dialogue between a subject and a robot, it has not been considered for environments such as meetings in which multiple people participate, and its purpose is to improve the accuracy of voice recognition.

Patent Document 3 discloses a voice recognition result and a voice recognition method that shortens the delay time until a voice recognition result is obtained and suppresses degradation of recognition processing performance, even when used on hardware with low processing performance. The shape of the lips is acquired by a camera, and whether or not the person is speaking is determined based on whether or not the change in the lip shape accompanying the speech matches a preset lip shape pattern for speech. However, since voice and image information is acquired by pressing a button when performing voice recognition, it is considered difficult to use such a voice acquisition method in a conference. In addition, since this method is considered to be a method for when the distance between the user of a tablet terminal or the like and the equipment is short, it is considered difficult to use in a conference where there are multiple people and the people are far away from the camera.

Patent Document 4 discloses a method for accurately recognizing the speech state of each participant with a simple configuration. Specifically, first, a distorted circular image including the face of each participant in the meeting is captured using a wide-angle imaging device with a fisheye lens. Next, the lip vicinity area of the captured image is set, and the speech state is estimated using a feature value indicating the brightness or color within the lip vicinity area. Finally, the speech time is displayed on a display device along with each person's image. However, since this is not used in conjunction with audio information, if a participant's lips move when no audio is being produced, it may be recognized that speech has occurred.

Patent Document 5 discloses a method for reducing noise in speech and improving speech recognition accuracy. Specifically, it provides a speech recognition device that can suppress erroneous recognition in non-speech segments by using features calculated based on MFCC and lip movement. However, because this method aims to improve speech recognition accuracy, it does not take into account cases where multiple speakers are present in the camera at the same time, such as in a conference environment.

JP 2020-27351 A JP 2020-3783 A International Publication No. 20160-98228 JP 2015-019162 A JP 2011-59186 A

In view of the above, the present invention aims to provide a speech section extraction method that requires simple equipment and can accurately extract the speech section of a speaker from multiple subjects. It also provides a program and device for this purpose.

The first invention is a method for extracting speech sections from video and audio data, comprising the steps of: obtaining a plurality of time-series vertical widths of the intra-mouth area of a subject from the obtained video; obtaining a plurality of time-series audio features based on the obtained audio data; and extracting speech sections. The step of extracting speech sections comprises the steps of obtaining a plurality of time-series judgment differences of the vertical width of the intra-mouth area from the vertical width of the intra-mouth area and extracting speech sections from the judgment differences of the vertical width of the intra-mouth area; and obtaining a plurality of time-series judgment differences of the audio features from the audio features and extracting speech sections from the judgment differences of the audio features. This speech section extraction method distinguishes a section that is an utterance section extracted from the vertical width of the intra-mouth area and includes an utterance section extracted from the audio features as a speech section.

In the process of extracting speech sections from the determination difference of the audio features in the first aspect of the present invention, it is preferable that the determination difference of the audio features is 0.00 or more.

In the process of extracting speech sections from the determined difference in the vertical width of the intra-mouth area in the first aspect of the present invention, it is preferable that the determined difference in the vertical width of the intra-mouth area is greater than a threshold A, and that the threshold A is expressed by the following formula (1) using the spatial resolution Rs of the image.

In the first aspect of the present invention, it is preferable to include a step of performing noise reduction processing on a plurality of speech features to obtain a plurality of determination differences of the speech features in a time series.

In the first aspect of the present invention, it is preferable to include a step of performing noise reduction processing on multiple vertical widths of the intraoral region to obtain multiple judgment differences of the vertical width of the intraoral region in a time series.

It is preferable to include a step of re-extracting frames of speech sections from frames classified as non-speech sections in the step of extracting speech sections from the determined difference in the vertical width of the intra-mouth area in the first invention.

The second invention is a program for extracting speech sections from video and audio data, comprising the steps of: obtaining a plurality of vertical widths of the intra-mouth area of a subject in a time series from the obtained video; obtaining a plurality of audio features in a time series based on the obtained audio data; and extracting speech sections, wherein the step of extracting speech sections comprises the steps of obtaining a plurality of judged differences in the vertical width of the intra-mouth area in a time series from the vertical width of the intra-mouth area and extracting speech sections from the judged differences in the vertical width of the intra-mouth area; and obtaining a plurality of judged differences in the audio features in a time series from the audio features and extracting speech sections from the judged differences in the audio features, and the speech section extraction program distinguishes sections that are extracted from the vertical width of the intra-mouth area and include speech sections extracted from the audio features as speech sections.

In the second speech section extraction program of the present invention, in the step of extracting a speech section from the judgment difference of the speech features, it is preferable that the judgment difference of the speech features is 0.00 or more.

In the step of extracting a speech section from the determined difference in the vertical width of the intra-mouth region of the second speech section extraction program of the present invention, it is preferable that the determined difference in the vertical width of the intra-mouth region is greater than a threshold value A, and that the threshold value A is expressed by formula (1) using the spatial resolution Rs of the image.

The speech segment extraction program of the second aspect of the present invention preferably includes a step of performing noise reduction processing on multiple speech features to obtain multiple decision differences of the speech features in a time series.

In the second speech segment extraction program of the present invention, it is preferable to include a step of performing noise reduction processing on multiple vertical widths of the intra-mouth area to obtain multiple determination differences in the vertical width of the intra-mouth area in a time series.

It is preferable that the second speech segment extraction program of the present invention further includes a step of re-extracting frames of speech segments that have been erroneously classified as non-speech segments in the step of extracting speech segments from the difference in the vertical width of the intra-mouth area.

A third aspect of the present invention is an apparatus for extracting speech sections from video and audio data, the apparatus comprising: a camera for acquiring video; a microphone for acquiring audio data; a storage means for storing the speech section extraction program; and a calculation means for performing calculations based on the speech section extraction program, the calculation means acquiring video captured by the camera and audio data captured by the microphone, and performing calculations according to the speech section extraction program using the acquired video and audio data, in a speech section extraction device.

The present invention allows for simple equipment and the accurate extraction of a speaker's speech section from multiple subjects.

FIG. 1 is a diagram showing the flow of a speaker output method S1 in a system. FIG. 2A is a diagram showing a schematic example of a portion of video, and FIG. 2B is a diagram showing an example of a portion of audio data. FIG. 3 is a diagram showing the flow of the speech section extraction step S20. FIG. 4 is a diagram for explaining the arrangement of feature points. FIG. 5 is an enlarged view of the lip portion of FIG. 4, illustrating the vertical width of the intraoral region. FIG. 6(a) is an example showing a time series change in the vertical width of the intraoral region before the first smoothing process, and FIG. 6(b) is an example showing a time series change in the vertical width of the intraoral region after the first smoothing process. FIG. 7A shows an example of the smoothed vertical width of the intra-mouth area, and FIG. 7B shows an example of the result of calculating the determined difference dn of the vertical width of the intra-mouth area from FIG. 7A. FIG. 8 is a diagram illustrating target frames in the first speech segment re-extraction processing step. 9(a) and 9(b) are diagrams for explaining audio data. FIG. 10 is a diagram for explaining how the MFCC is obtained from the voice data. FIG. 11A is an example showing a time series change in speech feature quantity before the second smoothing process, and FIG. 11B is an example showing a time series change in speech feature quantity after the second smoothing process. 12A is a diagram showing an example of speech features after the second smoothing process, FIG. 12B is a diagram showing an example of a calculation result of a first determination difference dm of speech features, and FIG. 12C is a diagram showing an example of a calculation result of a second determination difference Dm of speech features. FIG. 13A shows an example of a speech section extracted using the vertical width of the intra-mouth region, and FIG. 13B shows an example of a speech section extracted using audio features. FIG. 14 is a diagram illustrating the configuration of the speech segment extraction device. FIG. 15 is a diagram for explaining an utterance start frame and an utterance end frame. FIG. 16 is a diagram showing an example of a speech section extraction result. FIG. 17 is a diagram showing the calculation results of F _ave for combinations of parameters S and T related to speech features and the classification results based on the F _ave values. FIG. 18 is a diagram showing the calculation results of F _ave for combinations of parameters W and A related to spatial resolution, and the classification results according to the F _ave values.

{Speaker output method}
Fig. 1 is a diagram showing the flow of a speaker output method S1 in a system according to one embodiment. As shown in Fig. 1, the speaker output method S1 includes a video/audio data acquisition step S10, a speech section extraction step S20, a speaker count determination step S30, a speaker determination step S40, and a speaker output step S50. The speech section extraction step S20 of the present disclosure makes it possible to extract the speech sections of speakers from multiple targets with high accuracy.

[Video/audio data acquisition step S10]
In the video/audio data acquisition step S10, video and audio data of the speech section extraction target person are acquired. Video can be acquired by a so-called camera, and audio data can be acquired by a microphone. According to this embodiment, a camera (for example, an omnidirectional camera or a wide-angle camera) capable of simultaneously capturing the video of multiple speech section extraction target people and a microphone capable of capturing the audio of the speech section extraction target people are used to perform data processing in the steps described below, thereby making it possible to extract which person has a speech section. In addition, it is also possible to use image information of the narrowed-down person in a processing step such as speaker discrimination. When acquiring video and audio data, multiple video cameras and microphones may be used, but as long as information on all of the speech section extraction target people can be acquired, one video camera and one microphone may be used, and the equipment can be minimized.
In addition, the camera and the microphone may be separate devices or may be integrated together, so that the microphone provided in the camera can be used.

By the video/audio data acquisition process S10, it is possible to acquire an image of the face of the person from whom the speech section is to be extracted, as shown, for example, in FIG. 2(a). Also, it is possible to acquire audio data as a waveform with the horizontal axis representing time, as shown, for example, in FIG. 2(b).

[Speech section extraction step S20]
In the speech section extraction step S20, a speech section is extracted using the video and audio data acquired in the video and audio data acquisition step S10. A flow of the speech section extraction step S20 is shown in FIG.
3, the speech section extraction process S20 includes a face area detection process S21, an intraoral area vertical width extraction process S22, a first smoothing process S23, a first speech section extraction process S24, and a first speech section re-extraction process S25, which are video data processing processes, and a voice feature acquisition process S26, a second smoothing process S27, a second speech section extraction process S28, and a speech section discrimination process S29, which are voice data processing processes. Each process will be described below.

<Face area detection processing step S21>
In the face region detection process S21, feature points are arranged on the face of the person to be extracted from the speech section for the video data acquired in the video/audio data acquisition process S10. An example is shown in Fig. 4. In the example of Fig. 4, feature points A indicated by "●" are arranged on the face of the video shown in Fig. 2(a) (for ease of viewing, the symbol A is attached to only some of the feature points, and the others are omitted). In this embodiment, multiple feature points A are arranged along the contours of the lower half of the face (from cheeks to chin), eyebrows, eyes, nose (bridge of the nose, lower end), and lips (upper and lower lips).
Although there is no particular limitation on the method of arranging the feature points, it is possible to use the brightness difference between adjacent pixels and determine the positions having a brightness difference equal to or greater than a predetermined threshold as the contours of each part. In addition, commercially available or publicly available software may be used, such as Dlib.

In this embodiment, since the vertical width of the intraoral region is grasped in time series as the feature point, as will be described later, it is sufficient that the feature point A is arranged at least in the position and number necessary for grasping this. Therefore, in this embodiment, the feature point A is arranged at least on the lips.
However, feature points A may be arranged other than those for other reasons. For example, feature points A along the contour of the face may be used to obtain the position and size of the face of the person to be identified, or a process may be performed to delete information other than the face.

The feature point A is arranged for each image in the video. That is, the feature point A is arranged for each of a plurality of images that are successive in time series to compose the video. Fig. 4 shows an example of one image.
When processing multiple images (frames) that are consecutive in time series, for example, face detection processing may be performed on the image of the first frame, and the area around the face may be trimmed based on the face detection results, and from the second frame onwards, face detection may be performed on the area around the face in the previous frame, thereby improving the efficiency of processing.

<Vertical width extraction process S22 of intraoral region>
In the mouth area vertical width extraction process S22, the vertical width of the mouth area is extracted and calculated from the feature point A arranged in the face area detection process S21. Fig. 5 shows an enlarged view of the lip portion in Fig. 4.
Here, the "vertical width of the intra-mouth area" refers to the vertical distance B between feature point _A1 at the upper end of the intra-mouth area and feature point _A2 at the lower end of the intra-mouth area of the lips. The vertical width of the intra-mouth area is extracted for each of a number of images that are consecutive in time series, as long as it can be understood that it changes over time depending on the dictation of the person from whom the speech section is to be extracted. In the example shown in Fig. 5, the feature point closest to the midline of the intra-mouth area, where distance B is likely to be maximum, was selected. This makes it easier to clarify the change over time in the vertical width of the intra-mouth area.

The distance B extracted in this process may be expressed in any unit, such as coordinates, length, or number of pixels. In this embodiment, the distance B of the vertical width of the intraoral region is expressed in number of pixels, as shown in FIG. 6(a).

People tend to keep their mouths closed when they are not speaking. Therefore, the vertical width of the intra-mouth area when they are not speaking tends to be close to 0 pixels and to maintain a nearly constant value. Furthermore, the tendency to keep the mouth closed when they are not speaking is observed in all subjects for speech section extraction, regardless of the person. Furthermore, when the mouth is closed, the vertical width of the intra-mouth area is a value that is unlikely to change due to fluctuations in the distance between the camera and the subject for speech section extraction or the angle of the face of the subject for speech section extraction.
By focusing on the time-series changes in the vertical width of the intraoral area, the present invention reduces the effects of facial movement, differences in lip movement between different people, and changes in the distance between the camera and the person being extracted from, making it possible to extract speech sections, even in environments such as meetings.

<First smoothing treatment step S23>
In the first smoothing process step S23, a smoothing process is performed on the time series change in the vertical width of the intramouth area obtained in the intramouth area vertical width extraction process step S22. Fig. 6(a) is an example showing the time series change in the vertical width of the intramouth area before the first smoothing process step S23, and Fig. 6(b) is an example showing the time series change in the vertical width of the intramouth area after the first smoothing process step S23. The first smoothing process step S23 can reduce the effects of noise caused by minute changes in the lighting and other factors in the time series change in the vertical width of the intramouth area.
In this embodiment, a smoothing process is used in this process, but any known noise reduction process that can reduce the effects of noise caused by time-series changes in the vertical width of the intraoral area can be used in this process without any restrictions.

The smoothing procedure in the first smoothing process step S23 is not particularly limited, but an example is a method of calculating the average value of the vertical width of the intra-mouth area in P consecutive frames (frame n to frame n+P) starting from an arbitrary nth frame, and setting this average value as the vertical width of the intra-mouth area in the nth frame. Here, P may be an integer of 1 or more, and in this example, P is set to 10 frames, but as described below, in order to obtain high accuracy in extracting speech segments, it is preferable that P be 5 or more and 15 or less.

<First speech section extraction processing step S24>
In the first speech section extraction processing step S24, a speech section is extracted based on the time series change in the vertical width of the intra-mouth area after the first smoothing processing step S23. Fig. 7(a) shows an example of the smoothed vertical width of the intra-mouth area, and Fig. 7(b) shows an example of the result of calculating the judgment difference dn of the vertical width of the intra-mouth area from Fig. 7(a). The "judgment difference of the vertical width of the intra-mouth area" is the result of obtaining the maximum and minimum values of the vertical width of the intra-mouth area within a range of multiple chronologically consecutive frames and calculating the difference between the maximum and minimum values, and serves as an index for extracting whether or not a frame is within the speech section.

More specifically, the maximum and minimum values of the vertical width of the intra-mouth area are obtained from an arbitrary nth frame and the Q frames before and after it (a total of 2 x Q + 1 frames from n-Qth frame to n+Qth frame), and the judgment difference dn of the vertical width of the intra-mouth area, which is the difference between the maximum and minimum values, is calculated and set as the value of the nth frame. The judgment difference of the vertical width of the intra-mouth area is calculated for multiple frames that are consecutive in time series. Here, Q may be an integer of 1 or more, and in this example, the above Q is set to 15 frames, but as will be described later, in order to obtain high accuracy in extracting speech segments, it is preferable that Q be between 10 and 30.

For example, if there are consecutive frames of silent intervals in any nth frame and Q frames before and after it, the maximum and minimum values of the vertical width of the intramouth area will both be close to 0 pixels, and the determined difference dn of the vertical width of the intramouth area will be close to 0. Also, if there are frames of speech intervals in any nth frame and Q frames before and after it, the vertical width of the intramouth area will be larger in the speech interval compared to the silent interval, and the determined difference dn of the vertical width of the intramouth area will be larger than the silent interval.
If the judged difference dn of the vertical width of the intra-oral region is larger than a certain threshold A, the frame is classified as a speech section. Although it varies depending on the number of pixels of the video data, the ratio of the lips of the person to be identified in the entire image in the video, the distortion of the wide-angle camera, etc., for example, in the case of Fig. 7 where video data of 3840 x 2160 pixels and a spatial resolution of one pixel of about 1 mm (calculated from the fact that the vertical width of the lips is about 20 mm and the same region is displayed in about 20 pixels on the image) is obtained, the speech section can be extracted by setting the threshold A to 1.5.
The threshold value A is preferably a value that satisfies the formula (1) using the spatial resolution Rs of the image. If the determination difference dn of the vertical width of the intraoral region is larger than the threshold value A, the frame is classified as a speech section with high extraction accuracy.

<First speech section re-extraction processing step S25>
In the first speech section re-extraction processing step S25, frames of speech sections erroneously classified as non-speech sections in the first speech section extraction processing step S24 are re-extracted. Fig. 8 shows an example of a result of calculating the determined difference dn of the vertical width of the intra-mouth area, and frames surrounded by "o" in Fig. 8 represent frames of speech sections that have been erroneously classified as non-speech sections.

In this embodiment, in order to prevent frames in a speech section from being erroneously classified as a silent section, if there is a frame determined to be a speech section within R frames (frame n to frame n+R) after any nth frame determined to be a silent section, the nth frame may be treated as a frame in the speech section. Here, R may be an integer of 1 or more, and in this example, R is set to 30 frames, but as described later, in order to obtain high speech section extraction accuracy, R is preferably 5 to 50.
If there is no frame of an utterance period that has been erroneously classified as a non-utterance period, the first utterance period re-extraction processing step S25 does not need to be performed.

<Acquisition step S26 of voice features>
In the voice feature acquisition step S26, voice features are calculated and obtained from the voice data (e.g., FIG. 2(b)) obtained in the video/audio data acquisition step S10. This makes it possible to extract the voice data necessary for speech section extraction from the voice data containing a lot of complex information, and makes it easier to handle the data while maintaining accuracy.

The speech features are not particularly limited as long as they can extract the speech data necessary for extracting speech segments from speech data and make the data easy to handle while maintaining accuracy, but it is preferable to use Mel-Frequency Cepstrum Coefficients (MFCCs). Of these, it is even more preferable to use the zeroth dimension. This is because they have low-frequency component characteristics that are useful for speech recognition features, and the low-dimensional (zeroth dimension) components change due to the acoustic properties of the vocal tract and the shape of the oral cavity.

As a more specific example, speech features are obtained as follows: Figs. 9 and 10 are diagrams for explanation.
First, a portion of a predetermined time length D (portion _E1 ) is extracted from the audio data obtained in the video/audio data acquisition step S10 shown in Fig. 9(a) as shown in Fig. 9(b). The size of D is not particularly limited, but is 20 ms in this example.
Next, the MFCC is calculated for the speech data of this portion _E1 to obtain the MFCC data as shown in Fig. 10. The method of calculating the MFCC is well known, and for example, the method described in "Speech Recognition System, Revised 2nd Edition, edited by Tatsuya Kawahara, Ohmsha, 2016" can be mentioned.
For example, the calculation is performed as follows. First, the voice data (voice waveform) is Fourier transformed to obtain frequency components, and the power spectrum (the volume of sound at each frequency component) is calculated using these frequency components. Next, this power spectrum is multiplied by a Mel filter bank. Since human hearing has a characteristic that the resolution decreases as the frequency increases, by multiplying by the Mel filter bank, it is possible to extract features according to the human hearing characteristics. Then, from this, a cepstral feature is calculated, and the harmonic components of the ripple wave (features that change depending on the person) and the envelope components due to the vocal tract (features that change depending on the speech content) are separated. The low-dimensional components (0th to 14th dimensions) of the cepstral feature are mainly used for voice recognition, but as described above, it is preferable to use the 0th dimension in this embodiment.
The cepstral features extracted in this manner are called MFCCs and are used as speech features.

Then, as shown in Fig. 9(a), a speech feature is obtained similarly for part _E2 (length of time D) delayed by time d from part _E1 . By repeating this process, a plurality of speech features are obtained in a time series. The length of the delay time d is not particularly limited, and may be D>d, D=d, or D<d, but it is preferable that D>d from the viewpoint of improving accuracy. In this example, D is 20 ms, while d is 10 ms.
It is also possible to exclude parts where there is no speech and target only parts where there is speech.

By doing the above, we can obtain the time series changes in the speech features, which are the numerical values in the 0th dimension of the MFCC.

<Second smoothing process S27>
In the second smoothing process step S27, a smoothing process is performed on the time series changes in the speech features obtained in the speech feature acquisition process S26. The second smoothing process step S27 makes it possible to reduce noise and the like contained in the time series changes in the acquired speech features. Fig. 11(a) is an example showing the time series changes in the speech features before the second smoothing process step S27, and Fig. 11(b) is an example showing the time series changes in the speech features after the second smoothing process step S27.
In this embodiment, a smoothing process is used in this step, but any known noise reduction process capable of reducing noise, etc. contained in the time-series changes in the acquired speech features can be used in this step without any restrictions.

The smoothing procedure in the second smoothing process S27 is not particularly limited, but an example is a method of calculating the average value of the speech features for T consecutive frames (nth frame to n+Tth frame) starting from an arbitrary nth frame, and setting the average value as the speech feature for the nth frame. Here, T may be an integer of 1 or more, and in this example, the above T is set to 100 frames (approximately 1.0 second), but as described below, in order to obtain high speech section extraction accuracy, it is preferable that T is between 10 and 150.

<Second speech section extraction processing step S28>
In the second speech section extraction processing step S28, a speech section is extracted based on the time series change of the speech feature that has been subjected to the second smoothing processing step S27. Fig. 12(a) shows an example of the speech feature after the second smoothing processing step S27, Fig. 12(b) shows the result of calculating the provisional decision difference dm of the speech feature from Fig. 12(a), and Fig. 12(c) shows the result of calculating the decision difference Dm of the speech feature from Fig. 12(a) and Fig. 12(b). The "provisional decision difference of the speech feature" is the result of obtaining the maximum and minimum values of the speech feature in a range of multiple frames that are consecutive in time series and calculating the difference between the maximum and minimum values, and the "decision difference of the speech feature" is the result of calculating the time series change of the difference between the speech feature and the provisional decision difference dm of the speech feature, and serves as an index for extracting whether or not a frame is within an utterance section.

Specifically, for example, the maximum and minimum values of the speech feature are obtained from an arbitrary nth frame and the S frames before and after it (a total of 2×S+1 frames from the n-Sth frame to the n+Sth frame), and a provisional decision difference dm of the speech feature, which is the difference between the maximum and minimum values, is calculated and set as the nth frame. The provisional decision difference dm of the speech feature is calculated for a plurality of frames that are consecutive in time series. Here, S may be an integer of 1 or more, and in this example, the above S is set to 50 frames, but as described later, in order to obtain high speech section extraction accuracy, S is preferably 10 to 150. Furthermore, it is preferable that S≦T+30.
As can be seen from the frames of the "actual speech section" indicated by the arrow in Fig. 12, compared with the frames of the non-speech section, the frames of the speech section tend to have higher speech feature values and to maintain stable values. Furthermore, the provisional decision difference dm of the speech feature values tends to have a lower value.
Furthermore, when the decision difference Dm of the speech feature, which is the time series change of the difference between the speech feature and the provisional decision difference dm of the speech feature, is calculated, it is found that the value of the decision difference Dm of the speech feature tends to be positive in the speech section. Therefore, the area where the value of the decision difference Dm of the speech feature is 0.00 or more is extracted as the speech section.

<Speech section discrimination process S29>
The speech section discrimination processing step S29 discriminates the speech section from the speech section based on the vertical width of the intra-oral area obtained in the first speech section re-extraction processing step S25 and the speech section based on the audio features obtained in the second speech section extraction processing step S28.
The conditions for the speech section discrimination include that the speech section is a section extracted using the vertical width of the intra-mouth region and includes a speech section extracted using audio features. A series of frames that satisfy the above conditions for the speech section discrimination is discriminated as a speech section.

Fig. 13(a) shows an example of a speech section extracted using the vertical width of the intra-mouth region, and Fig. 13(b) shows an example of a speech section extracted using audio features. However, the frames of the vertical width of the intra-mouth region and the frames of the audio features are used in the same time series. For example, in Fig. 13, the determination difference dn of the vertical width of the intra-mouth region for three frames corresponds to the determination difference Dm of the audio features for ten frames.
If the speech section is estimated using only the vertical width of the intraoral area, it may be mistakenly determined to be a speech section even when no sound is being produced; however, the speech section discrimination processing step S29 can prevent errors, making it possible to discriminate only sections where sound is actually being produced as speech sections.

As shown in FIG. 1, after the speech section extraction process S20 of the present disclosure, a speaker number determination process S30, a speaker determination process S40, and a speaker output process S50 may be provided, making this a speaker output method that is effective for an automatic minutes creation system, etc.

[Number of speakers determination step S30]
In the number-of-speakers determination step S30, any known method can be used as long as it is possible to determine the number of people speaking at the same time. If there are multiple speakers whose speech sections are extracted simultaneously in the speech section extraction data obtained in the speech section extraction step S20, the process proceeds to a speaker determination step S40, and if there is only one speaker whose speech section is extracted simultaneously, the process proceeds to a speaker output step S50.

[Speaker identification step S40]
The speaker discrimination step S40 can use any known method without limitation, and examples thereof include the contents described in "Yoichi Kageyama, Etsuro Nakamura, and Muneshige Shirasu, 62nd Joint Conference on Automatic Control, Lecture Papers, 1J4-0" and Japanese Patent Application No. 2020-000673. For example, a neural network (NN) capable of estimating lip behavior features from voice features is trained. Next, lip behavior features and voice features are acquired for the person to be discriminated, and the voice features are input to the trained NN to estimate the lip behavior features. Finally, the difference between the estimated value of the lip behavior features and the actual lip behavior features is calculated, and the person to be discriminated with the smallest difference is determined to be the speaker.
Once the speaker is identified in the speaker identification step S40, the process proceeds to a speaker output step S50.

[Speaker output step S50]
The speaker output process S50 may use any known method as long as it can output the content of the speech section extracted in the speech section extraction process S20 for the speaker determined in the speaker determination process S40.

{Speech segment extraction program and speech segment extraction device}
14 is a conceptual diagram showing the configuration of a speech segment extraction device 50 according to one embodiment that specifically performs calculations in accordance with the speaker output method S1 including the above-mentioned speech segment extraction method S20. The speech segment extraction device 50 has an input device 57, a calculation device 51, and a display means 58. The calculation device 51 has a calculation means 52, a RAM 53, a storage means 54, a receiving means 55, and an output means 56.

The calculation means 52 is configured by a so-called CPU (central processor) and is connected to each of the above-mentioned components, and is capable of controlling them. The calculation means 52 also executes various programs stored in the storage means 54, which functions as a storage medium, and performs calculations to create data for each process of the above-mentioned speech section extraction method S20 based on these programs.

RAM 53 is a component that functions as a working area for the calculation means 52 and as a temporary data storage means. RAM 53 can be composed of SRAM, DRAM, flash memory, etc., and is similar to known RAM.

The storage means 54 is a member that functions as a storage medium in which programs and data that are the basis for various calculations are stored. The storage means 54 may also be capable of storing various intermediate and final results obtained by executing the programs. More specifically, the storage means 54 stores (stores) programs. Other information may also be stored therein.

The stored programs include programs that are the basis for computing each step of the speaker output method S1, including the speech section extraction method S20 described above. In other words, the speaker output method program, including the speech section extraction method program, includes steps in which each step is replaced with each step so as to correspond to each step of the speaker output method S1 shown in FIG. 1 (including each step of the speech section extraction method S20 shown in FIG. 3). The specific computation contents of the speech section extraction method program are as explained in the speech section extraction method S20 described above.

The receiving means 55 is a component that has the function of appropriately inputting information from the outside into the computing device 51, and is connected to an input device 57. This also includes so-called input ports, input connectors, etc.

The output means 56 is a component that has the function of appropriately outputting information that should be output from among the obtained results to the outside, and is connected to a display means 58 such as a monitor and various devices. This also includes so-called output ports, output connectors, etc.

The input device 57 may be a device that captures the image and voice of the speaker. Typical devices are a microphone, a camera, or a video camera with a microphone. However, the device is not limited to these, and may be other types of devices that capture the image and voice of the speaker. Information input from here is taken into the calculation device 51, and the above program is executed using this information.

In addition, information may be provided to the computing device 51 via the receiving means 55 via a network or communication. Similarly, information may be transmitted to an external device via the output means 56 via a network or communication.

With this type of speech segment extraction device 50, it is possible to efficiently and accurately perform the speaker output method S1, including the above-mentioned speech segment extraction method S20. As such a speech segment extraction device 50, for example, a computer can be used.

The inventors conducted tests to actually extract speech segments, examined the range of each parameter, and examined the relationship between resolution and threshold. The conditions, tests, and evaluation methods are shown below.

[conditions]
Camera: Omnidirectional camera, THITA V, manufactured by RICOH (30 fps, 3840 x 2160 pixels)
Microphone: TA-1, RICOH (unidirectional x 4 channels)
・Lighting: fluorescent lamp, illuminance 700lx to 900lx
・Subjects: 6 people (in their 20s, 3 men and 3 women)
Position of subject: 50 cm away from the camera, facing forward toward the camera

[Testing and Evaluation Methods]
(1) Six subjects (in their 20s, three men and three women) each read the same sentence aloud, and this was recorded using the camera and microphone mentioned above.
(2) The subjects read aloud 11 types of texts excerpted from news articles. Therefore, a total of 66 video and audio data were obtained.

(3) Speech segments were extracted from the obtained video and audio data by the speech segment extraction step of the present invention, and extraction results of speech frames and non-speech frames were obtained.
(4) Actual speech sections extracted by visual inspection were set. Specifically, from the video and audio data obtained by visual inspection, the frame in which the mouth is closed immediately before frame F _S +1 in which the mouth begins to open at the start of speech was set as the speech start frame F _S , as shown in Fig. 15(a), and the frame in which the mouth is closed at the end of speech was set as the speech end frame F _E , as shown in Fig. 15(b). Furthermore, the frames between the speech start frame F _S and the speech end frame F _E were set as speech frames, and the other frames were set as non-speech frames.

(5) An evaluation was performed by comparing the actual speech sections with the extraction results obtained by the speech section extraction process of the present invention, using the number of frames (unit: number of frames) for determining the extraction success rate shown in Table 1. As shown in Table 1, each index is defined as the number of frames in which speech frames were correctly determined as speech frames: TP, the number of frames in which speech frames were erroneously classified as non-speech frames: FN, the number of frames in which non-speech frames were erroneously classified as speech frames: FP, and the number of frames in which non-speech frames were correctly determined as non-speech frames: TN, and the number of frames corresponding to each index was calculated.

Using the number of frames corresponding to each index in Table 1, precision is calculated using formula (2) and recall is calculated using formula (3). Precision and recall take values between 0.0 and 1.0, with values closer to 1.0 indicating a higher extraction success rate.

{Speech segment extraction test}
The results of the speech segment extraction test conducted based on the above conditions and the test and evaluation methods are shown in Table 2.

As can be seen from Table 2, the average precision value was 0.90 or more for all subjects, and the average recall value was 1.00 for all subjects.
The above results suggest that the present invention can reduce the difference in lip movement between people and can extract speech segments using a simple threshold judgment method.

Fig. 16 shows the results of speech interval extraction for subject B in one of the 11 types of sentences. Fig. 16(a) shows the speech interval extracted using the vertical width of the intra-mouth region and audio features, and Fig. 16(b) shows the speech interval extracted using audio features. Fig. 16(a) shows that by using the vertical width of the intra-mouth region and audio features in combination, it is possible to appropriately exclude non-speech intervals and extract only the areas where speech intervals exist.
In other words, according to the speech section extraction method disclosed herein, even with a single omnidirectional camera and microphone, the speaker can be identified by processing the captured speech video, making it convenient and eliminating the need to increase the number of devices depending on the number of people.
In addition, the time series changes in the vertical width of the intraoral area, which is video data, are compared with the audio features, which are audio data, and used to extract speech sections, making it possible to exclude non-speech sections that have been mistakenly extracted as speech sections.

{Consideration of the range of each parameter}
Based on the above conditions and the test and evaluation methods, the range of each parameter was examined. The range of each parameter is examined by calculating F-measure from precision and recall calculated in (2) and (3) and using it as an evaluation index. F-measure is calculated by formula (4) and takes a value between 0.0 and 1.0, with the closer the value is to 1.0, the higher the extraction success rate. For each parameter, if F-measure is 0.9 or more, it is considered that the speech section extraction accuracy is good.

The parameters are the number of frames P for smoothing the vertical width of the intraoral area, the number of frames Q for calculating dn, the number of frames R for re-extracting the speech section, the number of frames S for calculating dm, and the number of frames T for smoothing the audio features.

<Parameter study on speech period extraction method using vertical width of intra-mouth area>
Using videos taken under the above conditions and based on the test and evaluation methods, the preferable ranges of the parameters P, Q, and R were examined. Specifically, the values of each parameter were varied within the ranges shown in Table 3, and the average values of F-measure (F _ave ) for 1000 parameter combinations were calculated and compared. Since the number of parameter examination patterns would be enormous, fixed values were used for the parameters S and T related to the speech feature.

Table 4 shows the ranges of parameters P, Q, and R and the results of F _ave within the ranges. Of the 1,000 patterns, 265 patterns had F _ave values exceeding 0.90. Among the patterns with F _ave values exceeding 0.90, the parameters were set to their maximum ranges, and it became clear that the ranges shown in Table 4 (150 patterns) were preferable.

<Parameter study on speech segment extraction method using speech features>
Using videos taken under the above conditions and under the test and evaluation methods, the preferable ranges of parameters S and T were investigated. Specifically, the values of each parameter were varied within the ranges shown in Table 5, and the average values of F-measure (F _ave ) for 225 parameter combinations were calculated and compared. Since the number of parameter investigation patterns would be enormous, fixed values were used for parameters P, Q, and R relating to the feature amounts of the lips.

Fig. 17(a) shows the calculation results of F _ave for combinations of parameters S and T, and Fig. 17(b) shows the results classified according to whether the F _ave value is 0.9 or more. As shown in Fig. 17(b), the range of parameters S and T for which the F _ave value is 0.9 or more can be expressed by a linear function and a constant. Table 6 shows the ranges of parameters S and T and the results of F _ave within the ranges. By setting the numerical values of parameters S and T to values within the ranges shown in Table 6, the F _ave value is 0.9 or more, making it possible to extract speech sections with high accuracy.

{Study on the relationship between spatial resolution and threshold}
In the speech section extraction process, if the judged difference dn of the vertical width of the intra-mouth area is greater than a certain threshold A, the frame is classified as a speech section. The relationship between the value of the threshold A and the spatial resolution Rs of the image was examined. Specifically, a weight W was added to the obtained vertical width of the intra-mouth area to change the spatial resolution Rs of the image in a pseudo manner. As described above, the spatial resolution in this embodiment is 1.0 mm, so that the spatial resolution Rs is 1.0 mm when the value of the weight W is 1.0. Therefore, the relationship between the weight W added to the vertical width of the intra-mouth area and the spatial resolution Rs is expressed by Equation (5).

本検討では、上記条件および試験および評価の方法に基づいて、表７に示すような条件の下撮影した動画を使用して、重みＷの値と閾値Ａを変化させ、各パターンにおけるＦ－ｍｅａｓｕｒｅの平均値Ｆ_ａｖｅを算出して比較した。

In this study, based on the above conditions and the test and evaluation methods, videos taken under the conditions shown in Table 7 were used, and the weight W and threshold A were changed, and the average value F _ave of the F-measure in each pattern was calculated and compared.

Fig. 18(a) shows the calculation results of F _ave for combinations of weight W and threshold A, and Fig. 18(b) shows the results of classifying each pattern according to whether the F _ave value is 0.9 or more. As shown in Fig. 18(b), the range of weight W and threshold A where the F _ave value is 0.9 or more can be expressed by two linear functions and a constant. Table 8 shows the range of weight W and threshold A and the results of F _ave within the range. By setting the numerical values of weight W and threshold A to values within the ranges shown in Table 8, the F _ave value is 0.9 or more, making it possible to extract speech sections with high accuracy.

If the value of weight W is replaced with the spatial resolution Rs (unit: mm) within the parameter range in Table 8, the above-mentioned formula (1) is derived. It is preferable that threshold A is a value that satisfies formula (1) using the spatial resolution Rs of the image, and if the judgment difference dn of the vertical width of the intraoral area is larger than threshold A, the frame is classified as a speech section with high extraction accuracy.

50 Speech period extraction device 51 Calculation device 52 Calculation means 53 RAM
54 Storage means 55 Receiving means 56 Output means 57 Input device 58 Display means

Claims (13)

A method for extracting speech segments from video and audio data, comprising the steps of:
Obtaining a plurality of time-series vertical widths of the subject's intraoral region from the acquired image;
obtaining a plurality of speech features in a time series based on the acquired speech data;
Extracting a speech section,
The step of extracting a speech section includes:
obtaining a plurality of determined differences in the vertical width of the intra-mouth region in a time series from the vertical width of the intra-mouth region, and extracting a speech section from the determined differences in the vertical width of the intra-mouth region;
obtaining a plurality of speech feature determination differences in a time series from the speech feature values, and extracting a speech section from the speech feature determination differences;
a speech section extracted from a vertical width of the intra-mouth region and including an utterance section extracted from the audio feature quantity, the speech section being determined as the speech section. The speech section extraction method according to claim 1, wherein in the step of extracting a speech section from the speech feature value determination difference, the speech feature value determination difference is 0.00 or more. In the step of extracting a speech section from a determined difference in the vertical width of the intra-mouth area, the determined difference in the vertical width of the intra-mouth area is greater than a threshold A,
The threshold value A is expressed by the following formula (1) using the spatial resolution Rs of the image:

The speech period extraction method according to claim 1 or 2. The speech section extraction method according to any one of claims 1 to 3, further comprising a step of performing noise reduction processing on the plurality of speech features to obtain a plurality of determination differences of the speech features in a time series. The speech section extraction method according to any one of claims 1 to 4, further comprising a step of performing noise reduction processing on the vertical widths of the multiple intra-mouth regions to obtain multiple determination differences of the vertical widths of the intra-mouth regions in a time series. The speech section extraction method according to any one of claims 1 to 5, further comprising a step of re-extracting frames of speech sections from frames classified as non-speech sections in the step of extracting speech sections from the determined difference in the vertical width of the intra-mouth area. A program for extracting speech segments from video and audio data, comprising:
Obtaining a plurality of time-series vertical widths of the subject's intraoral region from the acquired image;
obtaining a plurality of speech features in a time series based on the acquired speech data;
Extracting a speech section,
The step of extracting a speech section includes:
obtaining a plurality of determined differences in the vertical width of the intra-mouth region in a time series from the vertical width of the intra-mouth region, and extracting a speech section from the determined differences in the vertical width of the intra-mouth region;
obtaining a plurality of speech feature determination differences in a time series from the speech feature values, and extracting a speech section from the speech feature determination differences;
a speech section extraction program for determining, as a speech section, an utterance section that is extracted from a vertical width of the intra-mouth area and that includes an utterance section extracted from the audio feature amount. The speech section extraction program according to claim 7, wherein in the step of extracting a speech section from the determination difference of the speech features, the determination difference of the speech features is 0.00 or more. In the step of extracting a speech section from a determined difference in the vertical width of the intra-mouth area, the determined difference in the vertical width of the intra-mouth area is greater than a threshold A,
The threshold value A is expressed by the following formula (1) using the spatial resolution Rs of the image:

9. The speech segment extraction program according to claim 7 or 8. The speech segment extraction program according to any one of claims 7 to 9, further comprising a step of performing noise reduction processing on the plurality of speech features to obtain a plurality of determination differences of the speech features in a time series. The speech section extraction program according to any one of claims 7 to 10, further comprising a step of performing noise reduction processing on the vertical widths of the multiple intra-mouth regions to obtain multiple determination differences in the vertical widths of the intra-mouth regions in a time series. The speech section extraction program according to any one of claims 7 to 11, further comprising a step of re-extracting frames of speech sections that have been erroneously classified as non-speech sections in the step of extracting speech sections from the determined difference in the vertical width of the intra-mouth area. An apparatus for extracting speech segments from video and audio data, comprising:
A camera for acquiring the image;
A microphone for acquiring the voice data;
A speech segment extraction program according to any one of claims 7 to 12, comprising: a storage means for storing the speech segment extraction program; and a calculation means for performing calculations based on the speech segment extraction program,
The calculation means acquires video captured by the camera and audio data captured by the microphone, and performs calculations according to the speech segment extraction program using the acquired video and audio data.