JP3070073B2 - Shape control method based on audio signal - Google Patents

Description

Description: BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a shape control method based on an audio signal that controls the shape of a chin and lips of a face of a video or a doll based on the audio signal.

[Summary of the Invention]

The present invention provides a linear and non-linear conversion of the formant center frequency of an input audio signal to obtain the degree of enlargement of the lower jaw and the lip, thereby enabling a more realistic control of the shape of the jaw and the lip. The present invention provides a shape control method based on the above.

[Conventional technology]

In a conventional animation, for example, in a so-called animation, the movement of the lips and chin when a person in the animation has a conversation or the like, the animation image creator compares the movement of the lips or the like in accordance with the conversation with, for example, conventional experience. It is decided by guessing together. The same applies to the case where the lips and jaws of a humanoid robot or a doll are moved according to the conversation.

[Problems to be solved by the invention]

By the way, in recent years, in the above-mentioned animation, robots, dolls and the like, it has been demanded that the lips and jaws can be moved more realistically in accordance with the conversation.

However, as described above, conventionally, the movement of the lips and the like in accordance with the conversation is estimated based on experience and the like, and therefore, it is hardly a realistic movement.
In addition, for example, a device that calculates the movement of the lips or the like using a computer or the like has been considered.
The reality is that it is not possible to easily obtain more realistic movements of the lips and the like. Further, conventionally, the lips and the like cannot be moved in real time.

Therefore, the present invention has been proposed in view of the above-described situation, and it is possible to more realistically control the shape of the chin and lips of a face of a video or a doll, and furthermore, it is possible to realize the real-time control. It is an object of the present invention to provide a shape control method based on an audio signal that can be controlled.

[Means for solving the problem]

A shape control method based on an audio signal according to the present invention has been proposed to achieve the above-described object.From an input audio signal, a center frequency of a formant frequency indicating a peak of a spectrum envelope of the input audio signal is determined. Asked,
By linearly and non-linearly transforming the formant center frequency, the degree of dilation of the lower jaw and the degree of dilation of the lips in the lateral direction are obtained, for example, for the face of a video, a doll (robot) or the like. In other words, the jaw movement and tongue movement are obtained by linearly converting the formant center frequency, and the lower jaw opening and the lateral opening of the lips are obtained by nonlinearly converting this tongue movement. I have.

[Action]

According to the present invention, a simple linear conversion operation is performed on the formant center frequency of the input audio signal based on the actual audio,
Further, since the degree of opening of the lower jaw and the lip is obtained by performing a simple nonlinear conversion operation, the movement of the lip and the lower jaw can be easily reproduced in real time.

〔Example〕

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 shows a face image of, for example, an animation to which a shape control method based on an audio signal according to an embodiment of the present invention is applied.

In the face image of the present embodiment shown in FIG. 1, a formant (for example, the first or second form) indicating the peak of the spectrum envelope of the input audio signal as shown in FIGS. 2 to 4 is obtained from the input audio signal. Formant H ₁ , H ₂ ) The center frequency of the frequency is obtained, and the formant center frequency is subjected to linear conversion and non-linear conversion to obtain the degree of dilation D of the lower jaw of the face.
(Cm) and the degree of opening L (cm) in the lateral direction of the lips. That is, in the present embodiment, the formant center frequency is linearly transformed by using the expression (1) described later, whereby the movement of the jaw, that is, the opening degree D of the lower jaw, and each movement of the tongue shown in FIG. obtains a motion of the tongue tip shape illustrated in motion (Figure 6) and / or Figure 7 in position P ₁ to P _5, to the non-linear converting the movement of the tongue with the below equation (2) To obtain the degree of lateral enlargement L of the lips.

Here, FIG. 2 shows a spectrum envelope of an audio signal when, for example, the sound of "A" is generated, FIG. 3 shows a spectrum envelope of an audio signal of "A", and FIG. The spectrum envelope of "U" is shown. The peak portions of the spectral envelopes shown in FIGS. 2 to 4 are usually called formants, and the formants of this audio signal are generally defined as attenuated sinusoidal components of the acoustic impulse response of the vocal tract.
This formant typically has 3-4 formants within 3 kHz for an average vocal tract about 17 cm long, and 5k
Within Hz there are 4-5 formants. For voiced sounds, the first three formants are the most important, and generally the peak appearing at the lowest frequency is the first formant (H ₁ )
And the peak appearing after this first formant is called the second peak.
Formants (H ₂ ), followed by third formants, fourth formants, and so on. These formants can be obtained, for example, based on so-called Cepstall analysis or linear prediction analysis. For example, by using the linear prediction analysis, they can be obtained with a small amount of calculation.

As described above, for example, each vowel "A", "I",
Find the formant frequencies of "U", "E", and "O".
A formant first formant H ₁ example of the frequency of each of vowels in FIG. 5 shows the positional relationship when plotting the position of the second center frequency of the formant H ₂ in a plane. In FIG. 5, the positional relationship between the vowels is represented by "o" between "a" and "u".
Is located, and the positional relationship is such that “E” is located between “A” and “A”.

By the way, it is known that the voice and the movement of the tongue have a relationship as shown in FIGS. 6 and 7, for example. In FIG. 6, the vowels "A", "I", "U", "E",
FIG. 8 shows a curvature function (tongue shape curvature) C (cm ^-1 ) at each of the operating positions P _{1 to} P ₅ of the tongue shown in FIG. 8 corresponding to “o”, and FIG. 7 shows a tongue corresponding to each vowel. Shows the tip shape. Here, in FIG. 8, each of the operation positions P _{1 to} P ₅ is a position on the center line on the front side of the tongue, and the operation position P ₁ is, for example, a position 10 mm from the tip of the tongue. ₂ is the above operating position
At a position 5 mm further from P ₁ , the operation positions P ₃ , P ₄ , and P ₅ are shown to be 5 mm deeper in this order in the following order. That is, based on the said 8 Figure, the Figure 6, at the time of utterance of each vowel in each operating position P ₁ to P _5, the tongue-shaped in the region of these 5mm on each operating position P ₁ to P ₅ FIG. 7 shows a curvature function (tongue-shaped curvature) C, and FIG. 7 shows the tongue-shaped curvature C as shown in FIG.

Further, a tongue-shaped curvature C in each operating position P ₁ to P ₅ in each vowel, the open Daito D of the lower jaw of the face, and has a relationship as shown in FIG. 9 to 13 FIG. That is, FIG. 9 shows the relationship between the tongue-shaped curvature C during each vowel production in the open SEV D and the operating position P ₁ of the lower jaw of the face at the time of each vowel utterance, in FIG. 10 each vowel production The relationship between the degree of opening D of the lower jaw of the face at the time and the tongue-shaped curvature C at the time of each vowel utterance at the above-mentioned operation position P ₂ is similarly described below.
11 shows the lower jaw opening D and the operating position P ₃ , FIG. 12 shows the lower jaw opening D and the operating position P ₄ , and FIG. 13 shows the lower jaw opening D and the operating position P _5.
In relation to the tongue-shaped curvature C of FIG. These figures 9 ~
From FIG. 13, the relationship between the tongue and jaw opening degree D and the tongue shape curvature C is
It can be seen that “O” is located between “A” and “U”, and “E” is located between “A” and “I”. It can be confirmed that the positional relationship among these vowels is common in all of FIGS. 9 to 13.

From FIG. 9 to FIG. 13 and FIG. 5, the positional relationship between the lower jaw opening degree D and the tongue-shaped curvature C and the vowels in the _first and second formants H ₁ and H ₂ is shown. , As described above,
It can be confirmed that “O” is located between “A” and “U”, and that “E” is located between “A” and “A”. That is, it can be confirmed that the formant frequency substantially matches the positional relationship between the tongue-shaped curvature C and the degree of mandibular enlargement D.

Because of this, the first and the second as shown in FIG.
From the formant frequencies of the second formants H ₁ and H ₂ , a function that approximately maps the tongue-shaped curvature C and the mandibular opening D as shown in FIGS. 9 to 13 can be relatively easily formed. Will be able to guide you.

In this embodiment, the function that is approximately mapped is linear. In this case, the linear transformation is Can be represented by However, in the equation (1), F ₁ is the first
A formant H ₁ formant frequency (Hz), F ₂ is the second formant of H ₂ formant frequency (Hz). Also, It is.

At this time, in the equation (1), A and B are points r _x , p _x , q _x indicating respective positions of the formant frequencies “A”, “A”, “U” as shown in FIG. At the tongue-shaped curvature C and the lower jaw opening D as shown in FIG.
The points are obtained by performing linear conversion to respective points r _y , p _y , and q _y of “U”.

Since the formant frequency has individual differences depending on the utterer, this individual difference is removed by normalization. This normalization is performed, for example, by converting a triangle having the vertices “A”, “A”, and “U” in FIG. 14 into an equilateral triangle as shown in FIG. This makes it possible to normalize each vowel. The conversion to the equilateral triangle will be described later.

Further, in the embodiment of the present invention, the conversion from the tongue-shaped curvature C to the degree of widening L of the lips in the lateral direction is performed. The conversion at this time is performed by using a non-linear conversion. As the non-linear conversion, for example, L = (C + d ₁ ) ^1/2 + d ₂ (2) is used. However, in the equation (2), d ₁ and d ₂ are constants.
This non-linear conversion can enhance the naturalness of mouth movement.

In the present embodiment, the tongue shape curvature C and the mandibular opening degree D are converted into the tongue shape curvature D by the linear conversion of the formant frequency as described above, and the lateral opening degree L of the lip is converted by the nonlinear conversion of the lower shape curvature C. By performing the conversion operation, the degree of enlargement D of the lower jaw and the degree of enlargement L of the lips in the lateral direction can be estimated from the audio signal. Therefore, by using the shape control method of the present embodiment, it is possible to more realistically control not only images such as animations but also shapes of chins and lips of a face of a doll or the like. Further, since individual differences in utterances are removed by normalization, more accurate shape control can be performed.

Here, a method of converting an arbitrary triangle used for the normalization into a regular triangle or inversely transforming the triangle will be described. That is, as shown in FIG. 17, the triangle pq in the XY plane
Considering r, it is converted into an equilateral triangle p ⁽⁶⁾ q ⁽⁶⁾ r ⁽⁶⁾ having one side length as shown in FIG. 23 by the following procedure. First, in the triangle pqr in FIG. 17, the point p (x ₁ , y ₁ ) is translated so as to move to the origin (FIG. 18). Triangle p in Fig. 18
⁽¹⁾ q ⁽¹⁾ r X coordinate of ⁽¹⁾ the point q ⁽¹⁾ (x ₂ ^(1)), and the point r ⁽¹⁾
The X and Y coordinates are scale-transformed so that the Y coordinate (y ₃ ⁽¹⁾ ) becomes 1 (FIG. 19). The triangle p ⁽²⁾ q ⁽²⁾ r by the angle θ so that the Y coordinate (y ₂ ⁽²⁾ ) of the point q ^{(2) in} FIG. 19 becomes zero.
Rotate ⁽²⁾ (Fig. 20). The triangle p ⁽³⁾ q in FIG. 20
⁽³⁾ X, Y such that the X coordinate (x ₂ ^{(3) )} of the point q ⁽³⁾ of r ^{(3) and} the Y coordinate (y ₃ ⁽³⁾ ) of the point r ⁽³⁾ become 1. The coordinates are scaled (FIG. 21). Point r ⁽⁴⁾ of triangle p ⁽⁴⁾ q ⁽⁴⁾ r ⁽⁴⁾ in FIG. 21
Is the difference between the X coordinate (x ₃ ⁽⁴⁾ ) of X and X = 0.5, and the straight line Y =
X / a is used to convert the x coordinate of point r ⁽⁴⁾ into a scale (2nd
2). Y coordinate of the triangle p of said Figure 22 ^{(5) q (5) r} (5) is 3 ^1/2
/ 2 scale conversion of Y coordinate
Figure). Through the above procedure, an arbitrary triangle pqr can be converted into an equilateral triangle. Inverse conversion is also possible by following this procedure in reverse.

〔The invention's effect〕

In the shape control method based on the audio signal of the present invention,
By converting the formant center frequency of the input audio signal into a linear transformation and a non-linear transformation to obtain the degree of opening of the lower jaw and the lips, for example, images such as animation, dolls,
The shape of the chin and lips of the face of a robot or the like can be controlled simply and more realistically, and furthermore, it is possible to control the shape in real time.

[Brief description of the drawings]

FIG. 1 is a diagram showing a face image according to an embodiment of the present invention, and FIGS.
The figure is a characteristic diagram showing the spectrum envelope of the audio signal,
FIG. 5 is a diagram for explaining the formant frequency of the audio signal, FIG. 6 is a diagram showing the tongue-shaped curvature, FIG. 7 is a diagram showing the shape of the tip of the tongue, FIG. FIG. 9 to FIG.
Fig. 13 is a diagram for explaining the tongue shape curvature and mandibular enlargement at each movement position of the tongue, and Figs. 14 to 16 explain the conversion from formant frequency to tongue shape curvature and mandibular enlargement 17 to 23 are diagrams for explaining a method of converting an arbitrary triangle into an equilateral triangle. D: The degree of dilation of the lower jaw L: The degree of dilation of the lips in the lateral direction

────────────────────────────────────────────────── ─── Continued on the front page (51) Int.Cl. ⁷ Identification code FI G10L 19/02 G10L 9/06 C 21/06 3/00 551H // A63F 13/00 S 9/04 G (56) References JP-A-4-359299 (JP, A) JP-A-2-83727 (JP, A) JP-A-57-126000 (JP, A) JP-A-47-3008 (JP, A) (JP, U) Patent 2667455 (JP, B2) Patent 2644789 (JP, B2) Patent 2518683 (JP, B2) (58) Fields investigated (Int. Cl. ⁷ , DB name) G10L 11/00-13/08 G10L 19/00-21/06 G10L 15/00-17/00 JICST file (JOIS) Practical file (PATOLIS) Patent file (PATOLIS)

Claims (1)

(57) [Claims] A center frequency of a formant frequency indicating a peak of a spectrum envelope of the input audio signal is obtained from the input audio signal, and the formant center frequency is linearly and non-linearly converted to open the lower jaw of the face shape. A shape control method based on an audio signal, wherein a degree of opening and a degree of opening of the lips in the lateral direction are obtained.