JP3403378B2 - Kansei expression system - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention expresses sensibility by changing multimedia expressions using video and music according to human sensibilities classified based on physical quantities of human body expressions. A new Kansei expression system.

[0002]

2. Description of the Related Art In this type of conventional emotional expression device, a pose or a specific movement of a person (subject) photographed by a camera is identified by image processing, and multimedia expression is performed based on the identification result.

[0003]

However, the conventional technique merely controls the multimedia expression from the physical movement of the body, and cannot express the human sensitivity.

Therefore, a main object of the present invention is to provide a kansei expression system capable of freely expressing kansei rather than control by physical movement of the body.

[0005]

SUMMARY OF THE INVENTION The present invention is an estimation means for estimating human sensitivity based on the physical quantity of human body expression.
And a sensitivity expression system that changes multimedia expressions according to the sensitivity estimated by the estimation means.
Then, the estimation means classifies the sensibility into a plurality of sensibility categories in advance.
And by processing the parameters of physical quantities
I tried to estimate one of multiple affective categories
It is a characteristic Kansei expression system.

[0006]

[ Function ] Sensitivity is, for example, "Natural", "Happy", "Dynam"
ic "," Flowing "," Sharp "," Lonely "," Solemn "
Pre-classified into the sensitivity category, the estimation means is physical
One of them by processing the quantity parameter
Estimate the sensitivity by selecting the sensitivity category of
It And the multimedia representation changes according to the estimated sensitivity.

The parameters of physical quantities are time, power and
Includes spatiality. And, the time and power are preferably related to the temporal change of a specific part of human being, and the specific part is preferably the center of gravity.

Spatiality is preferably related to the spatial extent of the body, and this spatial extent is obtained from the ratio of a human body image viewed from a predetermined direction to a rectangle circumscribing the body image.

Also, the multimedia representation is preferably a representation by video and music.

[0010]

According to the present invention, a plurality of sensibilities are previously set.
Process into physical quantity parameters by classifying them into sex categories
To estimate one of multiple affective categories by
Since the multimedia expression is changed in accordance with the estimated sensitivity, the human sensitivity can be freely expressed by the multimedia of video and music.

The above-mentioned objects, other objects, features and advantages of the present invention will become more apparent from the following detailed description of the embodiments with reference to the drawings.

[0012]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Referring to FIG. 1, the emotional expression system 10 of this embodiment includes a camera 12 for photographing a subject H. The camera 12 photographs the whole body of the subject H at a frame rate of, for example, 1/30 second, and the photographed image signal is taken by the computer 14.
Give to. The computer 14 estimates the posture of the subject H based on the captured image signal input from the camera 12,
A character (avatar) having the same posture is displayed on the monitor 18. The computer 14 also samples the captured image signal, for example, every 10 frames / second, specifically 0.1 seconds, and estimates the sensitivity of the subject H based on the sampled captured image signal. Then, the background image and music corresponding to the estimated sensitivity are output from the monitor 18 and the speaker 20.

When estimating the sensitivity of the subject H, the computer 14 first obtains three parameters of temporality, power, and space based on the physical quantity of the body expression of the subject H. Here, the temporality means a parameter related to the movement and rhythm of the body of the subject H, the force is a parameter related to the acceleration of the body movement, and the spatial property is a parameter related to the spatial spread of the body. is there. Referring to FIG. 2, the temporality is the barycentric coordinates (Xc, Y) of the silhouette image of the subject H.
The force is obtained by the first derivative of c), and the force is obtained by the second derivative of the barycentric coordinates (Xc, Yc) of the silhouette image. In addition, the spatial property is a rectangle (width Xs, height Ys where the silhouette image of the subject H circumscribes the silhouette image).
Rectangle).

When three parameters are obtained in this way,
The computer 14 performs a weighted average for each parameter. When the period corresponding to two measures elapses or when the subject H performs a specific action, the computer 14 performs statistical analysis on the weighted average value at that time and estimates the subject H's sensitivity. A neural network or multiple regression analysis is applied to the statistical analysis, and the sensitivity is based on the prepared "Na
tural ”,“ Happy ”,“ Dynamic ”,“ Flowing ”,“ Sh
The computer 14 reads out the background image data and the music data corresponding to the estimated sensitivity from the database 16, and selects the monitor 18 and the speaker 20 from the seven categories of “arp”, “Lonely”, and “Solemn”. Output from each.

In the statistical analysis by the neural network, teacher data which is a base for estimating the sensitivity is required. In this teacher data, the dance expressing each sensitivity is danced by the subject H in front of the camera 12 for a certain period of time, and the weighted average value of each parameter is calculated every two bars based on the captured image signal output from the camera 12. Then, the calculated weighted average value is classified for each parameter. That is, the set of the obtained weighted average values for each parameter becomes the teacher data of the corresponding sensitivity. By learning this teacher data in the neural network, it is possible to estimate seven types of sensibilities from the weighted average value obtained when the subject H freely dances.

Teacher data is also required for statistical analysis by multiple regression analysis, but the teacher data here is obtained as follows. 7 sets for each parameter of the weighted average value
The same as the above, the points for obtaining the sensibilities of the types are the same as the above. In addition to this, several tens of subjects are presented with dance images expressing the seven types of sensibilities and the sensitivities indicated by the respective dance images are answered, and Find the voting rate for the video. Seven numerical values showing the voting rate are calculated for each dance video. The set of weighted average values for each parameter and the seven voting rates serve as teacher data corresponding to each sensitivity. When teacher data is obtained, each weighted average value is used as an explanatory variable,
Using the voting rate as the objective variable, the coefficient value corresponding to each sensitivity is obtained. The obtained coefficient values of each sensitivity are the subject H.
Is subjected to weighted average and weighted linear sum obtained when dancing freely, and the estimated voting rate for each sensitivity is obtained. Of these, the estimated sensitivity is the sensitivity with the highest estimated voting rate.

The computer 14 specifically processes the flow chart shown in FIG. First, in step S1, the camera 12
The captured image signal is taken in from and the posture of the subject H is estimated based on the captured image signal in step S3. In step S5, the posture of the avatar displayed on the monitor 18 is controlled according to the estimated posture. As a result, the avatar in the virtual space moves in the same manner as the subject H.

The computer 14 then proceeds to step S7.
Check if 0.1 seconds has elapsed. If "NO" here, the process returns to step S1 to repeat the above process, but if "YES", the processes of step S9 and thereafter are performed. In step S9, Equation 1 is calculated, and three parameters of temporality, power, and spatiality are calculated based on the captured image signal.

[0019]

## EQU1 ## T = (dXc / dt, dYc / dt) P = (d ² Xc / dt ² , d ² Yc / dt ² ) A = Hs / (Xs × Ys) T; Temporal P; Power A The spatial temporality of the silhouette image of the subject H is determined by first-order differentiating the barycentric coordinates, that is, the difference between the barycentric coordinates 0.1 second before and the current barycentric coordinates is 0.
It is calculated by dividing by 1 second. The spatial property is obtained by second-order differentiating the barycentric coordinates, that is, by dividing the difference between the previous first-order differential value and the current first-order differential value by 0.1 seconds. The force is obtained by dividing the area of the silhouette image of the subject H by the area of the rectangle circumscribing the silhouette image.
The silhouette image is obtained during the process of step S3.

The computer 14 then proceeds to step S1.
Equation 2 is calculated by 1 to obtain the weighted average value for each parameter.

[0021]

[Formula 2] T _{AV (n)} = αT _(n) + (1-α) T _{AV (n-1)} P _{AV (n)} = αP _(n) + (1-α) P _{AV (n-1)} A _{AV (n)} = α A _(n) + (1-α) A _{AV (n-1)} T _{AV (n)} ； The weighted average value of the temporality calculated this time T _(n) ； The temporality calculated this time T _{AV (n-1)} ; The weighted average value P _{AV (n)} of the previous time obtained; The weighted average value P _{(n) of} the power obtained this time; The power P _{AV (n-} obtained this time ₁₎ ； Previously obtained weighted average value of power A _{AV (n)} ； Currently obtained weighted average value of spatial property A _(n) ； This time obtained spatial property A _{AV (n-1)} ； Previously obtained The weighted average value α of spatial properties; constant (0 <α <1) The computer 14 then proceeds to step S13, and a predetermined fixed period, for example, a period corresponding to two measures has elapsed, or the subject H is Determine if you have taken a particular posture. Then, if the period corresponding to two measures has not yet passed and the subject H does not take a specific posture, the process returns to step S1. On the other hand, when the period corresponding to two measures elapses or the subject H takes a specific posture, step S
Proceed to 15. In step S15, the weighted average values T _AV and P
The sensitivity of the subject H is estimated based on _AV and A _AV . At this time, the neural network processing or the multiple regression analysis is applied to each of the weighted average values T _AV , P _AV and A _AV to obtain “Natural”, “Happy”, “Dynamic”, “Flowin”.
Any one of "g", "Sharp", "Lonely", "Solemn"
Estimate one sensitivity. If sensitivity is estimated, step S1
7, the background image and music corresponding to the estimated sensitivity are read from the database 16. The background image is displayed on the monitor 18, and the music is output from the speaker 20.
The computer 14 then initializes the weighted average values T _AV , P _AV and A _AV in step S19, and returns to step S1.

According to this embodiment, the posture of the subject is estimated based on the photographed image signal output from the camera, and avatars having the same posture are displayed in the virtual space of the monitor. In addition, three characteristics of temporality, power, and spatiality are obtained every 0.1 seconds based on the physical quantity of the body expression of the subject, and the sensitivity of the subject is 2 with temporality, power, and spatiality as parameters.
Estimated for each measure.

Here, the temporality and power are related to the temporal change of a specific part (coordinate of the center of gravity) of a human, and the spatiality is related to the spatial spread of the body. Furthermore, the subject's sensitivity is selected from seven pre-classified items (categories). The avatar's background image and music change depending on the selected sensibility. That is, the avatar moves in front of the background image that changes according to the sensitivity, and the music that changes according to the sensitivity is output from the speaker.

A concrete multimedia expression will be described. For example, assuming that the estimated sensitivity is “Happy”, the monitor 18 displays an image, and in the case of the pattern illustrated in FIG. 1, the hexagonal pattern shines pink and grows rhythmically in sequence, and at the same time. Bright and lively music is generated from the speaker 20. When the sensitivity is estimated to be “Lonely”, the dark blue hexagonal pattern on the monitor 18 gradually becomes smaller, and at the same time, the speaker 20 plays lonely music in a minor tone. Of course, it is not limited to the hexagonal pattern, and a gorgeous flower pattern may be treated, or white and black vertical stripes may be gently swirled in the wind to express loneliness.

As described above, since the sensitivity is estimated using the time, power, and space as parameters and the multimedia expression is changed according to the estimated sensitivity, the sensitivity of the subject is freely expressed by the multimedia. can do.

Therefore, according to the present invention, an expression of the body of a person dancing or singing in a disco or karaoke room, that is, a background image matching the sensitivity is expressed.
It can be applied to training of dance and dance choreography. Further, in the above description, the seven sensibilities are estimated based on the movement of the body, but it is also possible to track the movement of the body and estimate, for example, a “bad” sensitivity, which is useful for crime prevention.

[Brief description of drawings]

FIG. 1 is an illustrative view showing one embodiment of the present invention.

FIG. 2 is an illustrative view showing a part of the operation of the embodiment in FIG.

FIG. 3 is a flowchart showing a part of the operation of FIG. 1 embodiment.

[Explanation of symbols]

10 ... Kansei expression system 12 ... Camera 14 ... Computer 16 ... Database 18 ... Monitor 20 ... speaker

Front page continued (72) Masayuki Inoue, Masayuki Inoue, 2-2 Kodai, Seika-cho, Soraku-gun, Kyoto Prefecture 2nd, ATR Intelligent Video Communications Laboratory, Inc. 2-chome, Taiwan 2 ATR Intelligent Video Communications Laboratories, Inc. (72) Inventor Bernard Shampoo, 2-2, Hikaridai, 2-chome, Seika-cho, Soraku-gun, Kyoto Prefecture (56) References Iwadate, Research on Image Representation, IEICE Technical Report (HCS99-40-53), October 21, 1999, Vol. 99, No. 384, pp. 87-94 (58) Fields investigated (Int.Cl. ⁷ , DB name) G06T ⁷ /00-7/60 G06T 1/00 A63B 69/00 JISST file (JOIS)

Claims (7)

(57) [Claims] 1. A comprising estimation means for estimating the human senses on the basis of the physical quantity of the human body expression in sensitive expression system to vary the multimedia presentation in accordance with the sensitivity estimated by said estimating means, said estimating Means classifies sensitivity in advance into multiple sensitivity categories
By processing the parameters of the physical quantity,
To estimate one of the multiple affective categories
A Kansei expression system that features 2. The parameters are time, power and space.
The emotional expression system according to claim 1 , which includes sexuality. 3. The kansei expression system according to claim 2 , wherein the temporality and the force are related to temporal changes of a specific part of the human. 4. The emotional expression system according to claim 3 , wherein the specific portion is a center of gravity. 5. The affective expression system according to claim 2, wherein the spatial property is related to a spatial extent of the human body. 6. The emotional expression system according to claim 5 , wherein the spatial spread is obtained from a ratio of the human body image viewed from a predetermined direction to a rectangle circumscribing the body image. Wherein said multimedia presentation is represented by the video and music, sensibility expression system according to any one of claims 1 to 6.