JP7396590B2 - Speaker identification method, speaker identification program, and speaker identification device - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act 1. Presentation at the meeting Meeting name: 62nd Automatic Control Union Lecture Date: November 9, 2020 2. Published in a publication Publisher name: Japan Society of Mechanical Engineers Publication name: 62nd Automatic Control Association Conference Proceedings 1J4-04 Publication date: November 8, 2020

The present invention relates to identifying a speaker based on video and audio information.

In recent years, efforts have been made to improve work efficiency and review the working environment in order to realize work style reform. One of these measures is to improve the efficiency of work and meetings as a way to improve labor in the workplace.
Minutes are kept for the purpose of recording the content and agreements discussed and sharing decisions and background.The minutes created can be used to improve the quality of subsequent meetings and streamline other tasks. Contribute. According to an automatic minutes creation system constructed by applying voice recognition technology, it is possible to reduce human errors in creating minutes and reduce the number of people and time required to create minutes. Furthermore, in such an automatic minutes creation system, technology that automatically identifies the speaker for each utterance and technology that improves speech recognition accuracy contributes to reducing the man-hours required to create minutes and improves the efficiency of meetings and business operations.

Patent Document 1 discloses a web conference system that can improve the visibility of the lips of a participant who is speaking, and identifies the speaker using voice and lip movements. Specifically, voice information is used to identify the client terminal where the speaker is located, and the person whose lips are moving the most within the identified terminal is identified as the speaker. However, with this technique, if the lips of people present in the same terminal are moving at the same time, it becomes difficult to identify the speaker.

Patent Document 2 discloses a speaker estimation system capable of estimating at least one of the next speaker and the timing at which the next speaker speaks in a multi-person conversation involving a plurality of participants. However, although this document describes the use of lip movements to determine the next speaker, it does not describe a technique for determining the current speaker.

Patent Document 3 discloses a terminal device that can create minutes of a meeting without requiring a special device equipped with a large number of microphones. However, with this technology, it is necessary to register the user's voice information for each terminal in advance, so advance preparation is required before use.

Patent Document 4 discloses a system that recognizes speech states of conference participants. This technology sets the region near the lips of each participant in the meeting in an image obtained using a fisheye lens, and estimates the state of speech using features indicating the brightness or color in the region near the lips. . However, since this technique focuses only on the movement of the lips, it becomes difficult to identify the speaker if the lip areas of the conference participants are moving at the same time.

Patent Document 5 discloses a camera microphone device for video conferencing that displays an image above (overhead) the faces of conference attendees. However, since this technique identifies the speaker using the direction of arrival of the voice, discrimination becomes difficult when the distance between the persons is close.

Patent Document 6 discloses a technique that makes it possible to synthesize emotional speech while taking into account the unique settings of each speaker. This document describes that a process is implemented that uses voice feature data extracted for each frame from the speech uttered by a speaker to construct a network for generating facial feature points for the corresponding frame. However, there is a limit to the amount of features, and it is necessary to improve the accuracy.

JP2019-117997A Japanese Patent Application Publication No. 2018-077791 JP2016-029468A Japanese Patent Application Publication No. 2015-019162 Japanese Patent Application Publication No. 2012-147420 Patent No. 6582157

In view of these points, it is an object of the present invention to provide a method for identifying a speaker, which is a simple device and is capable of accurately determining a speaker from a plurality of subjects. We also provide programs and devices for this purpose.

One aspect of the present invention is a method for determining a speaker from video and audio data, wherein a first lip behavior feature is acquired in a time series based on the distance between the subject's upper lip and lower lip from an acquired video. a process of obtaining a plurality of voice features in chronological order based on the acquired voice data; and a process of identifying a speaker. a process of obtaining a plurality of second lip behavior features in time series from This is a method for identifying a speaker, including the step of determining a target person with the smallest discrimination difference as a speaker.

The first lip behavior feature amount may be obtained from the ratio between the distance between the upper lip and the lower lip and the distance between two points on the bridge of the subject's nose.

In the process of identifying a speaker, there is a process of creating a plurality of sections having different starting points and a predetermined time range, and a first lip behavior feature and a second lip behavior feature for each of the plurality of sections. It is also possible to calculate the interval difference between each interval and use the average of the interval differences for each interval as the discrimination difference.

In a plurality of sections, starting times may be determined so that adjacent sections overlap in part.

The plurality of first lip behavior feature amounts and the plurality of voice feature amounts may be normalized and expressed in a range of 0.0 or more and 1.0 or less.

Another aspect of the present invention is a program for determining a speaker from video and audio data, which calculates a first lip behavior feature value in time series based on the distance between the upper lip and lower lip of the subject from the acquired video. a step of obtaining a plurality of voice feature values in chronological order based on the acquired voice data; and a step of determining a speaker. a step of obtaining a plurality of second lip behavior features in time series from The present invention is a speaker discrimination program including the step of determining a subject with the smallest discrimination difference among the subjects as a speaker.

In the speaker discrimination program, the first lip behavior feature may be obtained from the ratio of the distance between the upper lip and the lower lip and the distance between two points on the bridge of the subject's nose.

In the above speaker identification program, the step of identifying the speaker includes the step of creating a plurality of sections having different starting times and predetermined time ranges, and the step of creating a plurality of sections having different starting times and predetermined time ranges, and a first lip behavior for each of the plurality of sections. It may be configured such that a section difference between the feature amount and the second lip behavior feature amount is obtained, and an average of the section differences of each section is obtained and this is used as the discrimination difference.

In the plurality of sections of the above-mentioned speaker discrimination program, starting times may be determined so that some of the times in adjacent sections overlap.

In the speaker identification program, the plurality of first lip behavior feature amounts and the plurality of voice feature amounts may be normalized within a range of 0.0 or more and 1.0 or less.

The apparatus also includes a device for determining a speaker from video and audio data, including a camera for acquiring video, a microphone for acquiring audio data, a storage means in which the speaker identification program is stored, and a speaker identification program. and a calculation means that performs calculations based on the above, and the calculation means acquires the video acquired by the camera and the audio data acquired by the microphone, and uses the acquired video and audio data to perform a calculation based on the speaker discrimination program. A speaker identification device is provided in which calculations are performed.

Advantageous Effects of Invention According to the present invention, it is possible to use simple equipment and to accurately identify a speaker from a plurality of subjects.

FIG. 1 is a diagram showing the flow of the speaker identification method S1. FIG. 2(a) is a diagram schematically illustrating a portion of video, and FIG. 2(b) is a diagram illustrating a portion of audio data. FIG. 3 is a diagram showing the flow of step S20 of calculating the first lip behavior feature amount. FIG. 4 is a diagram illustrating the arrangement of feature points. FIG. 5 is an enlarged view of the lip portion of FIG. 4. FIG. 6 is a diagram illustrating changes in the vertical feature amount of the lips. FIG. 7 is an enlarged view of the nose portion of FIG. 4. FIG. 8 is a diagram illustrating the first lip behavior feature amount. FIGS. 9(a) and 9(b) are diagrams illustrating audio data. FIG. 10 is a diagram illustrating how MFCC is obtained from audio data. FIG. 11 is a diagram illustrating the flow of the speaker discrimination process S40. FIG. 12 is a diagram illustrating the speaker discrimination process S40. FIG. 13 is a diagram illustrating the configuration of the speaker identification device. FIGS. 14(a) and 14(b) are diagrams explaining the test procedure.

{Speaker identification method}
FIG. 1 is a diagram showing the flow of a speaker discrimination method S1 according to one embodiment. As can be seen from FIG. 1, the speaker discrimination method S1 of the present embodiment includes a video/audio data acquisition process S10, a first lip behavior feature calculation process S20, an audio feature calculation process S30, and a speaker discrimination process S40. Contains. Each process will be explained below.

[Video/audio data acquisition process S10]
In the video/audio data acquisition process S10, video and audio data of the person to be identified is acquired. Acquisition of images can be carried out by a so-called camera, and acquisition of audio data can be carried out by a microphone, but according to this embodiment, a camera (for example, an omnidirectional camera, a wide-angle camera) that can simultaneously capture images of a plurality of persons to be discriminated; The speaker can be identified using a microphone that can capture the voice of the person to be identified. That is, one video camera and one microphone may be used as long as information on all persons to be determined can be acquired. Although multiple video cameras and microphones may be used, they can be kept to a minimum as long as video and audio data of the person to be identified can be acquired.
Further, the camera and microphone may be separate devices or may be integrated. Therefore, it is also possible to use the microphone provided in the camera.

Through this video/audio data acquisition step S10, it is possible to acquire an image of the face of the person to be determined, as schematically shown in FIG. 2(a), for example. Further, as schematically shown in FIG. 2(b), audio data can be acquired as a waveform with time as the horizontal axis.

[First lip behavior feature calculation process S20]
In the first lip behavior feature calculation step S20, a feature representing lip behavior (first lip behavior feature) is calculated based on the video acquired in the video/audio data acquisition step S10. FIG. 3 shows the flow of the first lip behavior feature calculation process S20.
As can be seen from FIG. 3, the first lip behavior feature calculation step S20 includes a feature point arrangement step S21, a lip vertical feature calculation step S22, a nose feature calculation step S23, and a first lip behavior feature calculation step S21. The calculation process S24 includes a calculation step S24 of the lip behavior feature amount. Each process will be explained below.

<Feature point arrangement process S21>
In the feature point placement process S21, feature points are placed on the facial part of the person to be discriminated with respect to the video acquired in the video/audio data acquisition process S10. An example is shown in FIG. In the example of Fig. 4, feature points A indicated by "●" are placed in the facial part of the image shown in Fig. (The rest have been omitted.) In this form, multiple feature points A are set along the contours of the lower half of the face (from cheeks to chin), eyebrows, eyes, nose (bridge of nose, lower end), and lips (upper and lower lips). It is located.
Although there is no particular limitation on the method of arranging the feature points, it is possible to use the brightness difference between adjacent pixels and determine that a position having a brightness difference greater than a predetermined threshold value is the outline of each part. In addition, commercially available or publicly available software may be used, such as Dlib.

In this embodiment, in order to grasp the vertical position of the lips and the distance between the root of the nose and the tip of the nose in chronological order, as will be described later as feature points, the feature points A are arranged at least in the positions and numbers necessary to grasp these points. It would be fine if it had been done. Therefore, in this embodiment, feature points A are arranged at least on the lips and nose.
However, the feature point A may be placed elsewhere for other reasons. For example, feature points A along the contour of the face may be used to obtain the position and size of the face of the person to be identified, or processing may be performed to delete information other than the face.

Note that such arrangement of feature points A is performed for each image in the video. That is, feature points A are arranged for each of a plurality of chronologically continuous images forming a video. FIG. 4 is an example explaining one image.

<Calculation process S22 of vertical feature amount of lips>
In the lip vertical feature calculation step S22, the vertical feature of the lips is calculated. FIG. 5 shows an enlarged view of FIG. 4 focusing on the lip portion.
Here, the "vertical feature amount of the lips" refers to the feature amount in the direction in which the upper lip and lower lip are lined up in the lip part, and specific examples include feature point _A1 belonging to the upper lip and feature point A1 belonging to the lower lip. ₂ in the relevant direction. The selection of the feature point A ₁ and the feature point A ₂ is not particularly limited, as long as it can be understood that the vertical feature amount of the lips changes over time depending on the dictation of the person to be discriminated. In the example shown in FIG. 5, among the feature points A that are closest to the midline of the face, the feature point that is located at the farthest distance between them is selected. This makes it easier to clarify time-series changes in the vertical feature amount of the lips.

Therefore, in this process, the distance B between the feature point A ₁ and the feature point A ₂ in the direction in which the upper lip and the lower lip are lined up is determined. This distance B may be expressed in any unit such as coordinates, length, number of pixels, etc. In this embodiment, distance is expressed by the number of pixels. In this embodiment, this distance B becomes the "vertical feature amount of the lips."

Such a vertical feature amount of the lips is calculated for each of a plurality of images (frames) that are continuous in time series. Thereby, for example, as shown in FIG. 6, it is possible to obtain time-series changes in the vertical feature amount of the lips over time (frame numbers).

<Nose feature calculation process S23>
In the nose feature calculation step S23, the vertical feature of the nose is calculated. FIG. 7 shows an enlarged view of FIG. 4 focusing on the nose.
Here, "nose feature amount" refers to the feature amount in the direction along the bridge of the nose in the nose, and a specific example is two feature points (A ₃ , A ₄ ) arranged along the bridge of the nose. The distance between The selection of the two feature points A ₃ and A ₄ is not particularly limited, but in the example shown in _FIG . The feature point A was set to ₄ . As a result, since the two are separated, time-series changes in the nose feature become clearer.

Therefore, in this process, the distance C between the nose feature point A ₃ and the feature point A ₄ in the direction in which the upper lip and lower lip are lined up is determined. This distance C may be expressed in any unit such as coordinates, length, number of pixels, etc., but it is assumed to be in the same unit as the distance B related to the lips described above. Therefore, in this embodiment, distance is expressed by the number of pixels.
In this embodiment, this distance C becomes the "nose feature amount."
Such nose feature amounts are calculated for each of a plurality of images (frames) that are continuous in time series. As a result, although not shown in the drawings, it is possible to obtain changes in the characteristic amount of the nose over time (frame numbers) in the same manner as in FIG. 6 described above.

<First lip behavior feature calculation process S24>
In the first lip behavior feature calculation step S24, the lip vertical feature obtained in the lip vertical feature calculation step S22 and the nose feature obtained in the nose feature calculation step S23. From this, a first lip behavior feature is calculated. Specifically, it can be determined using the following formula.
First lip behavior feature amount = Vertical feature amount of lips / Feature amount of nose In the case of examples B and C, the first lip behavior feature amount is determined by the following formula.
First lip behavior feature = B/C

This first lip behavior feature is a dimensionless amount that represents the ratio of the vertical feature of the lips to the feature of the nose, and can thereby reduce the influence of the distance between the person to be identified and the imaging means. .

Therefore, it is not necessary to take the feature amount of the nose into consideration, and the first lip behavior feature amount may be the vertical feature amount of the lips. The reason why the feature amount of the nose is used in this embodiment is as follows.
If the distance between the camera and the person to be identified changes during image acquisition, the distance in the vertical width of the lips changes, so even if the lips move the same, the vertical feature amount of the lips changes. On the other hand, by taking the ratio of the nasal feature amount that changes less with respect to the movement of the lips and using this as an index, it is possible to reduce the influence of changes in the distance between the camera and the person to be discriminated.
Therefore, it does not necessarily have to be a nose feature, but it can also be used as the first lip behavior feature by extracting a feature that satisfies the following two conditions and taking the ratio of this to the vertical feature of the lips. good.
The first condition is that the distance between feature points is difficult to change due to changes in speech movements and facial expressions. In this regard, the distance between the root of the nose and the tip of the nose used for the nasal feature amount is a distance between feature points that is not easily affected by changes in speech movements or facial expressions, and has little movement.
The second condition is that the distance between feature points exhibits a change similar to the movement of the vertical width of the lips when the angle of the face changes in the vertical and horizontal directions. When the face faces the camera, the ratio of the "distance between feature points" to the "distance between the camera and the person to be discriminated" is constant no matter which pair of feature points is used. However, when the angle of the face changes, this ratio changes. For example, even if the distance between the camera and the person to be classified does not change, if the person to be classified turns to the side, the width of the face (the length of the straight line connecting the feature points on the left and right ends of the face) varies. On the other hand, the vertical feature amount of the lips and the feature amount of the nose as in the present embodiment have a generally parallel relationship and are located in the center of the face. Therefore, when the angle of the face changes, the tendency of the distance between feature points to change is similar, so by using the nose feature amount, it is possible to reduce the influence of the change in the angle of the face.

In this embodiment, the first lip behavior feature is calculated using the vertical feature of the lips and the feature of the nose in images (frames) at the same time, and is calculated for each of multiple images (frames) that are consecutive in time. be done. Therefore, as shown in FIG. 8, it is possible to obtain changes in the first lip behavior feature amount over time (frame numbers).
Note that it is also possible to exclude the portions where no utterances are made and target only the portions where there are utterances.

Further, the first lip behavior feature amount may be normalized within a range of 0.0 or more and 1.0 or less in order to reduce individual differences in lip movements of speakers.

[Voice feature calculation process S30]
In the audio feature amount calculation step S30, the audio feature amount is calculated and obtained from the audio data (for example, FIG. 2(b)) obtained in the video/audio data acquisition step S10. This makes it possible to extract voice data necessary for speaker identification from voice data that includes a lot of complex information, and to make the data easier to handle while maintaining accuracy.

Voice features are not particularly limited as long as they can extract the voice data necessary for speaker identification from the voice data as described above and make the data easy to handle while maintaining accuracy. It is preferable to use a frequency cepstrum coefficient (Mel-Frequency Cepstrum Coefficient, MFCC). It is more preferable to use the 0th dimension. This is because it has low-frequency component characteristics that are useful as features for speech recognition, and that the low-dimensional (0th dimension) component changes due to the acoustic characteristics of the vocal tract and the shape of the oral cavity. by.

As a more specific example, the voice features are obtained as follows. Diagrams for explanation are shown in FIGS. 9 and 10.
First, a portion (portion E ₁ ) of a predetermined time length D is extracted from the audio data obtained in the video/audio data acquisition step S10 shown in FIG. 9(a), as shown in FIG. 9(b). Although the magnitude of D is not particularly limited, it is 20 ms in this example.
Next, the MFCC is calculated for the audio data of this portion _E1 , and MFCC data as shown in FIG. 10 is obtained. The method for determining MFCC is known, and for example, the content described in "Tatsuya Kawahara, editor, Speech Recognition System, Revised 2nd Edition, Ohmsha, 2016" can be mentioned.
For example, calculate as follows. First, audio data (sound waveform) is Fourier-transformed to obtain frequency components, and the frequency components are used to calculate the power spectrum (the loudness of the sound in each frequency component). Next, this power spectrum is multiplied by a Mel filter bank. Since human hearing has a characteristic that the resolution decreases as the frequency increases, by applying a Mel filter bank, it becomes possible to extract feature amounts according to the human auditory characteristics. Then, cepstral features are calculated from this, and the harmonic components of the voiceprint wave (features that change depending on the person) and envelope components due to the vocal tract (features that change depending on the content of the utterance) are separated. The low-dimensional components (0th dimension to 14th dimension) in the cepstral feature amount are mainly used for speech recognition, but as described above, it is preferable to use the 0th dimension in this embodiment.
The cepstrum feature extracted in this way is called MFCC and is used as a speech feature.

Thereafter, as shown in FIG. 9A, the audio feature amount is similarly obtained for the portion E ₂ (time length D) delayed by the time d with respect to the portion E ₁ . By sequentially repeating this process, a plurality of voice features are obtained in chronological order. Note that the size of this delay time d is not particularly limited, and may be any of D>d, D=d, and D<d, but from the viewpoint of improving accuracy, it is preferable that D>d. preferable. In this example, while the above D is 20 ms, d is set to 10 ms.
Furthermore, it is also possible to exclude portions where no utterances are made and target only portions where utterances are made.

As described above, it is possible to obtain the time-series change in the audio feature amount, which is the 0th dimension value of MFCC.
Note that this voice feature amount may be normalized within the range of 0.0 or more and 1.0 or less within the speech interval in order to reduce individual differences in the loudness of the speaker's voice.

[Speaker identification process S40]
In the speaker discrimination step S40, the speaker is identified from the first lip behavior feature obtained in the first lip behavior feature calculation step S20 and the voice feature obtained in the speech feature calculation step S30. Figure 11 shows the flow. The speaker discrimination process S40 of this embodiment includes a section setting process S41, a second lip behavior feature amount calculation process S42, a section difference calculation process S43, and a speaker discrimination process S44. Each process will be explained below.

<Section setting process S41>
In the section setting step S41, the video and audio data obtained in the video and audio data acquisition step S10 are divided into a plurality of "sections." FIG. 12 shows a diagram for explanation.
This "section" is the minimum unit when calculating the section difference between the first lip behavior feature amount and the second lip behavior feature amount that will be obtained thereafter. The interval can be set to 1000 ms, for example. A plurality of sections are set so that the starting points are slightly shifted in time, and sections 1 to M are considered as shown in FIG.
By using intervals in this manner, it is possible to identify the speaker in consideration of time-series changes in lip movements rather than on a so-called frame basis.

The difference in starting point time between adjacent sections is not particularly limited, and may be the same as the length of the section, shorter than the length of the section, or longer than the length of the section. However, it is preferable that the start point time shift is shorter than the length of the section as shown in FIG. 12, and can be, for example, about 0.1 times the length of the section (100 ms in this example). That is, in a plurality of sections, the starting times can be determined so that some of the times in adjacent sections overlap.
By setting sections such that adjacent sections partially overlap temporally in this way, more pattern feature amounts can be acquired than when they do not overlap.

The length of each section is set so that each section includes a plurality of first lip behavior feature amounts and a plurality of voice feature amounts. For example, when the length of one section is 1000ms and the voice feature is created every 10ms as in the above example (d in Figure 9(a) is 10ms), the voice feature data included in this section is The number is 100. On the other hand, regarding the first lip behavior feature, a normal camera captures 30 frames per second (1000ms), so one first lip behavior feature is obtained per frame, so the number of data is It is 30.

<Second lip behavior feature calculation process S42>
In the second lip behavior feature calculation step S42, for each section set in the section setting step S41, the speech feature obtained in the speech feature calculation step S30 is input into a pre-trained neural network. Calculate the second lip behavior feature amount. Therefore, through this process, it is possible to obtain lip behavior features based on voice data.

It is preferable that the number of data of the second lip behavior feature is the same as the number of data of the first lip behavior feature. In other words, if the number of voice feature data included in one section is 100 and the number of first lip behavior feature data is 30 as described above, 30 voice feature data are used based on the 100 voice feature data. A second lip behavior feature amount of the data is calculated. This makes it easier to obtain interval differences, which will be described later.

Then, such calculation of the second lip behavior feature amount is performed for each section.

Here, the conditions and method for pre-learning the neural network are not particularly limited as long as the voice feature can be associated with the lip behavior feature, but in this embodiment, it was performed as follows.

A neural network with a three-layer structure of an input layer, a middle layer, and an output layer is created under the conditions that match the length of the above-mentioned section, the number of voice feature data included therein, and the number of data of the first lip behavior feature. Use it to learn. For example, using the above example, the length of the interval is 1000 ms, the input layer is 100 dimensions according to the number of voice feature data, the middle layer is 50 dimensions, and the output layer is the data of the first lip behavior feature. The number of dimensions is set to 30 so that 30 data are output. Note that Adam can be used for the gradient method.
Learning proceeds by comparing the output from this output layer with teacher data.

<Section difference calculation process S43>
In the section difference calculation step S43, for each section, the first lip behavior feature obtained in the calculation step S20 of the first lip behavior feature belonging to the section and the second lip behavior feature belonging to the section The difference from the second lip behavior feature obtained in the amount calculation step S42 is calculated to obtain a section difference. More specifically, it is as follows.

As described above, in this embodiment, the number of data of the lip behavior feature amount based on the video (the first lip behavior feature amount) and the lip behavior feature amount based on the audio (the second lip behavior feature amount) is made to match. Therefore, the difference between the two can be calculated sequentially starting from the earliest time. Therefore, for example, 30 pieces of differential data can be obtained in one section.
In this process, each obtained difference data is expressed as an absolute value, and this is averaged to obtain the section difference δ in the section. Therefore, as shown in FIG. 12, interval differences δ ₁ , δ ₂ , δ ₃ , . . . , δ _M are obtained for each interval in this process.

<Speaker identification process S44>
In the speaker discrimination process S44, a discrimination difference δ _ave is calculated by averaging the plurality of interval differences obtained in the interval difference calculation process S43, and a person with the smallest discrimination difference δ _ave is selected from among the subjects to be discriminated. Be the speaker.
This allows the speaker to be identified.

According to the method described above, even with one omnidirectional camera and microphone, the speaker can be identified by processing the captured speech video, so the number of devices can be increased according to the number of people. It's not necessary and convenient.
In addition, the lip behavior features are calculated using the audio features, and the speaker is determined by comparing this with the lip behavior features based on the video and taking the difference, so multiple people's lips are moving at the same time. Even when lip movements occur simultaneously, it is possible to appropriately identify the speaker.

{Speaker identification program and speaker identification device}
FIG. 13 is a diagram conceptually showing the configuration of a speaker discrimination device 50 according to one embodiment that specifically performs calculations in accordance with the above-described speaker discrimination method S1. The speaker identification device 50 includes an input device 57, an arithmetic device 51, and a display means 58. The arithmetic device 51 includes a calculation means 52, a RAM 53, a storage means 54, a reception means 55, and an output means 56.

The calculation means 52 is constituted by a so-called CPU (central operator), and is connected to each of the above-mentioned constituent members and is capable of controlling them. Further, the calculation means 52 executes various programs stored in the storage means 54 etc. which function as a storage medium, and performs calculations for creating data for each process of the above-mentioned speaker identification method S1 based on the programs. be.

The RAM 53 is a component that functions as a work area for the calculation means 52 and a temporary data storage means. The RAM 53 can be configured with SRAM, DRAM, flash memory, etc., and is similar to known RAM.

The storage means 54 is a member that functions as a storage medium in which programs and data that serve as the basis for various calculations are stored. Further, the storage means 54 may be capable of storing various intermediate and final results obtained by executing the program. More specifically, a program is stored (saved) in the storage means 54. Further, other information may also be stored.

Here, the stored programs include a program that is the basis for calculating each process of the above-mentioned speaker discrimination method S1. That is, the speaker identification program replaces each process with each step so as to correspond to each process of the speaker identification method S1 shown in FIG. 1 (including each process shown in FIGS. 3 and 11). Contains steps. The specific calculation contents of this program are as explained in the above-mentioned speaker discrimination method S1.
The storage means 54 may also store a database based on the trained results of the neural network, which is the basis for calculating the second lip behavior feature from the voice feature. In this case, the program proceeds by sequentially referring to this database.

The receiving means 55 is a component having a function of appropriately taking in information from the outside into the arithmetic device 51, and is connected to an input device 57. This also includes so-called input ports, input connectors, etc.

The output means 56 is a component having a function of appropriately outputting information to be outputted to the outside from among the obtained results, and a display means 58 such as a monitor and various devices are connected here. This also includes so-called output ports, output connectors, etc.

Examples of the input device 57 include devices that acquire video and audio of the speaker. Typical equipment is a microphone, camera, or video camera with a microphone. However, the device is not limited to this, and may be a device that acquires video and audio of other types of speakers. Information input from here is taken into the arithmetic unit 51, and the above program is executed using this information.

In addition, information may be provided to the arithmetic device 51 via the receiving means 55 via a network or communication. Similarly, information may be transmitted to external equipment via the output means 56 via a network or communication.

According to such a speaker discrimination device 50, it becomes possible to perform the above-described speaker discrimination method S1 efficiently and accurately. For example, a computer can be used as such a speaker identification device 50.

{Speaker identification test}
The inventor actually conducted a test to identify speakers. The conditions and test methods are shown below.
・Camera: Omnidirectional camera, THITA V, manufactured by RICOH ・Microphone: TA-1, manufactured by RICOH ・Lighting: Fluorescent lamp, illuminance 700lx to 900lx
・Target people: 2 people (A, B)
・Placement of person to be identified: 50cm away from the camera, facing forward towards the camera

The test was conducted as follows based on the conditions described above.
- Two people (A, B) to be judged read the same sentence aloud separately, and this was recorded using the camera and microphone mentioned above.
・The sentences read aloud by the test subjects were 11 types extracted from news articles. Therefore, a total of 22 pieces of video and audio data were obtained.
・Out of these 22 data, 20 were used as teacher data for learning the neural network, and the remaining 2 data were divided into test data. The same process was performed using different division patterns by changing the combination of teacher data and test data, resulting in a total of 231 patterns.

After preparing as described above, the test was conducted as follows. FIG. 14 shows a diagram for explanation.
(1) Among 22 data (11 data for discrimination target A, 11 data for discrimination target B), select two arbitrary data as test data and the remaining 20 data as teacher data.
(2) Learning a neural network using the teacher data (20 data) and constructing a trained neural network.
(3) Using this trained neural network, identify the speaker on the test data. At this time, as shown in FIG. 14, the speaker is determined for each of the two prepared test data (test data 1 in FIG. 14(a) and test data 2 in FIG. 14(b)).
That is, for test data 1, as shown in FIG. 14(a), the second lip behavior feature amount is obtained using the audio data of test data 1. This is compared with the first lip behavior feature obtained from the video of test data 1 and the first lip behavior feature obtained from the video of test data 2, and δ _ave is calculated for each. In this case, the determination is successful if the δ _ave of the test data 1 is smaller.
Similarly, for test data 2, as shown in FIG. 14(b), the second lip behavior feature amount is obtained using the audio data of test data 2. This is compared with the first lip behavior feature obtained from the video of test data 1 and the first lip behavior feature obtained from the video of test data 2, and δ _ave is calculated for each. In this case, the determination is successful if the δ _ave of the test data 2 is smaller.
(4) The combination of test data and teacher data was changed and the above steps (1) to (3) were repeated 231 times, which is the total pattern.
(5) Using the 462 (231×2) discrimination results obtained in (1) to (4) above, the discrimination success rate and the average value of δ _ave were calculated.
In this example, tests were conducted for the case where the first lip behavior feature amount and the voice feature amount were normalized in the range of 0.0 or more and 1.0 or less, and the case where the normalization was not performed.

As a result of the above, as will be shown later, when we performed the above test by changing the number of learning times in the neural network, we were able to obtain a discrimination success rate of 79.2% to 83.8% when normalized. Ta. On the other hand, even when normalization was not performed, a discrimination success rate of 78.1% to 82.5% could be obtained. Therefore, a high discrimination success rate can be obtained regardless of the presence or absence of normalization. However, it is possible to increase the discrimination success rate by normalizing. The discrimination success rate is the ratio of the number of successful discriminations to the total number of discriminations expressed as a percentage.

As mentioned above, in this test, the inventor investigated the relationship between the number of learning times in the neural network and the discrimination success rate. That is, the number of repetitions of learning (the number of learning times) was changed, and the relationship with the success rate of speaker identification was investigated. The results are shown in Table 1. This is the result of changing the number of learning times from 500 times to 10,000 times in increments of 500 times. The test method is the same as above.
Further, "δ _ave " in Table 1 is the value of the discrimination difference.

As can be seen from Table 1, the speaker can be identified with high probability in either case.

50 Speaker identification device

Claims (11)

A method for determining a speaker from video and audio data, the method comprising:
a step of obtaining a plurality of first lip behavior features in time series based on the distance between the subject's upper lip and lower lip from the acquired video;
a step of obtaining a plurality of voice features in time series based on the acquired voice data;
a process of determining a speaker;
In the process of determining the speaker,
a step of obtaining a plurality of second lip behavior features in time series from the plurality of voice features;
a step of obtaining a discrimination difference that is a difference between the first lip behavior feature amount and the second lip behavior feature amount;
A method for identifying a speaker, comprising the step of determining, as a speaker, a target person with the smallest discrimination difference among the target persons included in the video. 2. The speaker discrimination method according to claim 1, wherein the first lip behavior feature is obtained by a ratio between a distance between the upper lip and the lower lip and a distance between two points on the bridge of the subject's nose. In the process of determining the speaker,
a process of creating a plurality of sections having different starting points and predetermined time ranges;
A section difference between the first lip behavior feature amount and the second lip behavior feature amount is determined for each of the plurality of sections, and an average of the section differences of each section is set as the discrimination difference. The speaker identification method described in 1 or 2. 4. The speaker identification method according to claim 3, wherein in the plurality of sections, the time serving as the starting point is determined so that a part of the time overlaps in adjacent sections. The speaker according to any one of claims 1 to 4, wherein the plurality of first lip behavior feature quantities and the plurality of voice feature quantities are normalized and expressed in a range of 0.0 to 1.0. Discrimination method. A program for determining a speaker from video and audio data, the program comprising:
obtaining a plurality of first lip behavior features in time series based on the distance between the subject's upper lip and lower lip from the acquired video;
obtaining a plurality of audio features in time series based on the acquired audio data;
a step of determining a speaker;
In the step of determining the speaker,
obtaining a plurality of second lip behavior features in time series from the plurality of voice features;
obtaining a discrimination difference that is a difference between the first lip behavior feature amount and the second lip behavior feature amount;
A program for identifying a speaker, including the step of determining, as a speaker, a target person with the smallest discrimination difference among the target persons included in the video. 7. The speaker identification program according to claim 6, wherein the first lip behavior feature is obtained by a ratio between a distance between the upper lip and the lower lip and a distance between two points on the bridge of the subject's nose. In the step of determining the speaker,
creating a plurality of sections having different starting times and predetermined time ranges;
For each of the plurality of sections, a section difference between the first lip behavior feature amount and the second lip behavior feature amount is determined, an average of the section differences for each section is calculated, and this is calculated as the discrimination difference. The speaker identification program according to claim 6 or 7. 9. The speaker identification program according to claim 8, wherein in the plurality of sections, the time serving as the starting point is determined so that a part of the time overlaps in the adjacent sections. 10. The speaker identification program according to claim 6, wherein the plurality of first lip behavior feature amounts and the plurality of voice feature amounts are normalized within a range of 0.0 or more and 1.0 or less. A device for determining a speaker from video and audio data, the device comprising:
a camera that acquires the image;
a microphone that acquires the audio data;
A storage device storing a speaker identification program according to any one of claims 6 to 10, and a calculation device that performs calculations based on the speaker identification program,
The calculation means is configured to perform speaker identification by obtaining an image obtained by the camera and audio data obtained by the microphone, and performing an operation by the speaker identification program using the obtained image and the audio data. Device.

Images