JP6059614B2 - Gesture generation device, gesture generation system, gesture generation method, and computer program - Google Patents

Description

  The present invention relates to a gesture generation device, a gesture generation system, a gesture generation method, and a computer program.

  In recent years, a voice interactive interface that operates on a mobile terminal has been realized. Furthermore, an anthropomorphic agent character is displayed on the display screen of the mobile terminal, and this character can speak and move according to information. It is being considered to let For example, the conventional technology described in Non-Patent Document 1 is used for a specific scenario such as news or weather forecast as a technology for automatically generating video content by TVML (TV program Making language) using APE (Automatic Production Engine). A computer graphics (CG) animation is automatically generated by preparing a template. In addition, a CG animation is automatically generated from text data of an e-mail or a blog (for example, see Patent Documents 1 and 2), or a sign language corresponding to text data input by a user is generated by a CG animation (for example, a patent). The technique of (refer literature 3) is known.

JP 2004-88335 A JP 2008-107904 A JP 2011-175598 A

Hayashi, M .; Douke, M .; Hamaguchi, N., 'Automatic TV program production with APEs,' Creating, Connecting and Collaborating through Computing, 2004. Proceedings.Second International Conference on, vol., No., Pp.20 , 25, 29-30 Jan. 2004 McNeill, D. 1992 Hand and Mind.Chicago: University of Chicago Press Hitoshi Isahara, Francis Bond, Kiyotaka Uchimoto, Masao Utiyama and Kyoko Kanzaki, Development of Japanese WordNet. In LREC-2008, Marrakech. Cormen, Thomas H .; Leiserson, Charles E., Rivest, Ronald L. (1990). Introduction to Algorithms (2st ed.). MIT Press and McGraw-Hill. ISBN 0-262-03141-8. Pp. 323- 69

  However, in the above-described conventional technology, it is difficult to automatically generate a natural CG animation in accordance with the speech of the speech when the speech text data and speech speech data are given. In the prior arts disclosed in Patent Documents 1 to 3, a CG animation that matches text data can be generated, but a CG animation that matches the voice timing of the text data cannot be generated. Further, since the conventional technique of Non-Patent Document 1 does not introduce a synchronization technique, the timing of the CG animation cannot be synchronized with the given voice.

  The present invention has been made in consideration of such circumstances, and includes a gesture generation device, a gesture generation system, a gesture generation method, and a computer program capable of generating a moving image of a natural gesture that matches a speech of a speech. The issue is to provide.

(1) The gesture generating apparatus according to the present invention is configured to generate a motion graph in which the plurality of gesture data is connected based on the serif data that is the text data of the serif, the speech data of the serif, and the connectivity of the plurality of gesture data. A gesture generation device that inputs and generates gesture data according to the speech of the line, selects a path with the lowest cost on the motion graph based on the duration of the line, and indicates the gesture data of the selected path Is provided with a gesture data generation unit that performs adjustment to match the speech of the speech.

(2) In the gesture generation device according to the present invention, in the gesture generation device according to (1), the gesture data generation unit is configured to generate a voice of the keyword in the speech corresponding to the stroke of the selected path of the gesture data. The data is adjusted to match the start timing and end timing of the stroke.

(3) In the gesture generation device according to the present invention, in the gesture generation device according to (2), the gesture data generation unit adjusts the length of the selected path's gesture data to the duration of the line. It is characterized by performing.

(4) In the gesture generation device according to the present invention, in the gesture generation device according to (3), the gesture data generation unit sets an end timing of a preparation period of the selected path of the gesture data as a start timing of the stroke. The duration of the preparation period is expanded or contracted so as to match the above.

(5) In the gesture generation device according to the present invention, in the gesture generation device according to (3) or (4), the gesture data generation unit sets a start timing for an end period of the gesture data of the selected path. The duration of the end period is expanded or contracted so as to match the end timing of the stroke and to match the end timing with the end timing of the speech data of the speech.

(6) In the gesture generation device according to the present invention, in the gesture generation device according to any one of (1) to (5), the gesture data generation unit is a first node of the gesture data included in the motion graph. Among them, the node having the best connectivity with the last pose of the gesture data is set as the start point node.

(7) In the gesture generation device according to the present invention, in the gesture generation device according to any one of (1) to (6), the motion graph includes a plurality of categories, and the gesture data generation unit includes Using the same motion graph as that of a keyword category, gesture data matching the speech of the speech is generated.

(8) In the gesture generation device according to the present invention, in any of the gesture generation devices according to (1) to (7), the gesture data generation unit is configured to prepare or end the gesture data of the selected path. The frame of the period is replaced with a similar frame from predetermined gesture data.

(9) In the gesture generating apparatus according to the present invention, in any of the gesture generating apparatuses according to (1) to (8), the gesture data generating unit is configured to perform a predetermined steady pose on the gesture data having only a stroke. Is used to add a preparation period and an end period of a certain time before and after the stroke.

(10) In the gesture generating apparatus according to the present invention, in the gesture generating apparatus according to any one of the above (1) to (8), the gesture data generating unit is configured to prepare for the gesture data having no end period. Using a pause, an end period of a certain time is added after the stroke.

(11) In the gesture generating device according to the present invention, in the gesture generating device according to any one of the above (1) to (8), the gesture data generating unit is configured to perform an end period for gesture data having no preparation period. It is characterized in that a fixed period of preparation is added before the stroke using a pose.

(12) In the gesture generation device according to the present invention, in the gesture generation device according to any one of (1) to (11), the gesture data generation unit is configured such that the stroke length of the motion graph is a duration of the dialogue. If it is a predetermined multiple or more, it is switched to a predetermined steady motion graph, or a silent section of a certain time is inserted immediately after the speech data line corresponding to the stroke of the motion graph. .

(13) The gesture generation system according to the present invention sets the phase boundary of the preparation period, the stroke, and the end period of the input gesture data in any one of the above (1) to (12) as nodes, In addition, a motion graph generation unit that generates a motion graph provided with edges based on connectivity between the nodes, and a motion graph database that stores the motion graph are provided.

(14) In the gesture generation method according to the present invention, a motion graph in which the plurality of gesture data is connected based on the serif data that is the text data of the serif, the speech data of the serif, and the connectivity of the plurality of gesture data. A gesture generation method of a gesture generation device that inputs and generates gesture data according to the speech of the speech, wherein the gesture generation device selects a path with the lowest cost on the motion graph based on the duration of the speech Then, adjustment is performed so that the speech data of the selected path is adjusted to the speech of the speech.

(15) A computer program according to the present invention inputs speech data that is text data of speech, speech data of the speech, and a motion graph in which the plurality of gesture data are connected based on connectivity of the plurality of gesture data. And selecting a path with the lowest cost on the motion graph based on the duration of the lines to a computer of a gesture generation apparatus that generates gesture data according to the speech of the lines, and the gesture data of the selected path Is a computer program for executing the step of adjusting the voice to the speech of the line.

  According to the present invention, an effect that a moving image of a natural gesture matched to the speech of a speech can be generated.

It is a block diagram which shows the structure of the gesture generation system 1 which concerns on one Embodiment of this invention. It is the schematic of the example of a definition of the gesture data which concerns on one Embodiment of this invention. It is a conceptual diagram which shows the flow of the motion graph production | generation method which concerns on one Embodiment of this invention. It is a conceptual diagram of the blending process which concerns on one Embodiment of this invention. It is a conceptual diagram for demonstrating the blending process which concerns on one Embodiment of this invention. It is a flowchart which shows the flow of the gesture data generation method which concerns on one Embodiment of this invention. It is explanatory drawing of the gesture data adjustment method which concerns on one Embodiment of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing a configuration of a gesture generation system 1 according to an embodiment of the present invention. In FIG. 1, the gesture generation system 1 includes a gesture generation device 10, a motion graph generation unit 20, and a motion graph database 30. The gesture generation device 10 uses the motion graph database 30 to generate gesture data that matches the speech of the input data (line data, voice data), and outputs the generated gesture data. The motion graph generation unit 20 generates a motion graph using the input gesture data. The motion graph database 30 stores the motion graph generated by the motion graph generation unit 20.

  Here, the gesture data according to the present embodiment will be described. The gesture data is movement data representing movement of a person or an animal. In particular, the gesture data is motion data representing a motion called a gesture. In general, a gesture is composed of a series of three-phase movements that begin in a preparation period, end in a run period (stroke), and end in a retraction period (for example, Non-Patent Document 2). reference). The preparation period is a phase from the first pose of the gesture (hereinafter referred to as a steady pose) to the start of the stroke. For example, the preparation period includes a movement from a position where a human hand is placed (steady pose) until a stroke starts. Stroke is the main phase of gestures. For example, as a stroke, a strong swing motion of a human hand can be cited. The end period is a phase that becomes the last pose of the gesture after the stroke. For example, as the end period, returning to a steady pose after a stroke may be mentioned. The preparation period and the end period are not essential and may be omitted.

  FIG. 2 is a schematic diagram of a definition example of gesture data according to the present embodiment. In the example of FIG. 2, skeleton type motion data of a human body is used as gesture data. Skeleton motion data of the human body is based on the human skeleton, using bone and bone connection points (joints), with one joint as the root (root), and the bone structure that is sequentially connected from the root via the joint. Defined as a (tree) structure. FIG. 2 shows only a part of the definition of the skeleton type motion data. In FIG. 2, a joint 100 is a waist part and is defined as a root. Joint 101 is the elbow portion of the left arm, Joint 102 is the wrist portion of the left arm, Joint 103 is the elbow portion of the right arm, Joint 104 is the wrist portion of the right arm, Joint 105 is the knee portion of the left foot, and Joint 106 is the left foot portion. Ankle part, joint 107 is right leg knee part, joint 108 is right leg ankle part, joint 109 is clavicle part, joints 110 and 111 are shoulder parts, joint 112 is head part, joints 113 and 114 are The hip joint part.

  The skeleton type motion data is data that records the movement of each joint of the skeleton type object, and a human body or an animal can be applied as the skeleton type object. The skeleton type motion data is generated based on, for example, motion capture data.

  In the present embodiment, skeleton type motion data of the human body exemplified in FIG. 2 is used as gesture data. The gesture data represents a series of movements of a person by a series of a plurality of postures (poses). One pose corresponds to one frame and records position information of all joints. One frame x (t) is expressed by Expression (1).

Here, p ^k (t) is the position of the k-th joint at time t and is represented by three-dimensional coordinates. Time t is the time of the frame. K is the number of joints. Therefore, x (t) is a 3K-dimensional vector.

  The gesture data X composed of T frames is expressed by Expression (2).

  X is a 3K × T matrix. In the present embodiment, time t is simply handled as a “frame index”. Thereby, the time t takes a value of “0, 1, 2,..., T−1”. T is the number of frames included in the gesture data.

Further, as another frame definition example, the amount of movement from the basic pose can be expressed for each joint. One frame specifies one pose in which the movement amount of each joint is added to the basic pose. Thereby, a series of movements of a person is specified by the continuation of each pose specified by each frame. In this case, angle information is used as the movement amount. Then, the joint position is calculated using the basic pose data and the frame data in the angle information data. The basic pose data includes information for specifying the basic pose, such as the position of the root and the position of each joint in the basic pose, and the length of each bone. The frame data has angle information representing the amount of movement from the basic pose for each joint. The position p ^k (t) of the k-th joint at time t is calculated by Expression (3) and Expression (4).

However, the 0th (i = 0) joint is the root. R _axis ^{i-1, i} (t) is a coordinate rotation matrix between the i-th joint and its parent joint ("i-1" -th joint), and is included in the basic pose data. A local coordinate system is defined for each joint, and the coordinate rotation matrix represents the correspondence of the local coordinate system between joints in a parent-child relationship. R ⁱ (t) is a rotation matrix of the i-th joint in the local coordinate system of the i-th joint, and is angle information included in the frame data. T ⁱ (t) is a transition matrix between the i-th joint and its parent joint, and is included in the basic pose data. The transition matrix represents the bone length between the i-th joint and its parent joint.

  This is the description of the gesture data. Returning to FIG.

[Motion graph generator]
The input gesture data is input to the motion graph generation unit 20. The input gesture data includes gesture data and metadata. The metadata includes a category identifier (category ID) indicating a category of gesture data, and a phase identifier (phase ID) for dividing the gesture data into three phases of the gesture. The phase ID is “P” indicating a preparation period, “S” indicating a stroke, and “R” indicating an end period. The preparation period, stroke, and end period of the gesture data are specified by the phase ID.

  An example of a method for classifying gesture data into categories will be described below. For a set of available words, keywords are extracted by morphological analysis of the text of the words. Each keyword is labeled with a category using a concept dictionary (semantic dictionary). For example, “WordNet” described in Non-Patent Document 3 can be used as the concept dictionary. For example, "Good morning", "good morning", "Hello", for a keyword such as "Good evening", put a category ID of the category of "greeting". This creates a category set for the serif set. Next, the category ID of the category included in the category set is attached to each gesture data included in the available gesture data set. The category ID is assigned to the gesture data manually. For example, a category ID of “greeting” is attached to the gesture data of “bow”.

  The gesture data is manually divided into three phases of the gesture (preparation period, stroke, and end period). According to this classification, phase IDs “P (preparation period)”, “S (stroke)”, and “R (end period)” are assigned to the gesture data. However, depending on the gesture data, there may be no preparation period or end period.

  The motion graph generation unit 20 generates a motion graph using the input gesture data. The motion graph is generated for each category. Therefore, only the input gesture data with the category ID of the category is used to generate the motion graph of a certain category.

  FIG. 3 is a conceptual diagram showing a flow of a motion graph generation method according to the present embodiment. Hereinafter, an operation in which the motion graph generation unit 20 generates a motion graph will be described with reference to FIG.

[Frame extraction step]
First, in the frame extraction step, all the frames corresponding to the phase boundary of the gesture data are extracted from all the input gesture data of the motion graph generation target category. The set of frames at the extracted phase boundary is represented as ^{Fi ALL} _B.

[Connectivity calculation step]
Next, in the connectivity calculation step, all the frames included in the set F ^iALL _B are set as the nodes of the motion graph. Therefore, the initial value of the number of nodes in the motion graph matches the number of frames included in the set F ^iALL _B. Next, distances are calculated for all pairs targeting all nodes by using Equation (5) or Equation (6). The distance between a certain node F ⁱ _{B and a} certain node F ^j _B is represented as d (F ⁱ _B , F ^j _B ).

Where q _{i, k} is the quaternion of the k-th joint of the node F ⁱ _B. w _k is a weight related to the k-th joint. The weight w _k is preset.

Here, p _{i, k} is a vector of relative positions with respect to the root of the k-th joint of the node F ⁱ _B. That is, p _{i, k} is a vector of the position of the k-th joint of the node F ⁱ _B calculated without considering the position and direction of the route.

  Note that the distance between nodes can be calculated as a weighted average of differences in physical quantities such as the position, velocity, acceleration, angle, angular velocity, and angular acceleration of each joint that constitutes a pose at the target node.

Next, the connectivity is calculated by the equation (7). The connectivity between a certain node F ⁱ _{B and a} certain node F ^j _B is represented as c (F ⁱ _B , F ^j _B ).

However, d (F ⁱ _B ) is a distance between the previous frame and the rear frame of the node F ⁱ _B (calculated by a calculation formula similar to the formula (5) or the formula (6)). TH is a preset threshold value.

When the connectivity c (F ⁱ _B , F ^j _B ) is 1, it can be determined that the pose of the node F ⁱ _{B and} the pose of the node F ^j _B are similar. When the connectivity c (F ⁱ _B , F ^j _B ) is 0, it cannot be determined that the pose of the node F ⁱ _{B and} the pose of the node F ^j _B are similar.

[Motion graph construction step]
Next, in the motion graph construction step, when the connectivity c (F ⁱ _B , F ^j _B ) is 1, a bidirectional edge is provided between the node F ⁱ _B and the node F ^j _B. When the connectivity c (F ⁱ _B , F ^j _B ) is 0, no bidirectional edge is provided between the node F ⁱ _B and the node F ^j _B.

  Next, a unidirectional edge is provided between adjacent nodes in the same gesture data. Unidirectional edges travel from the previous node to the next node in time.

  Next, blending processing is performed on the gesture data relating to the nodes at both ends of the bidirectional edge. The blending process is performed for each bidirectional edge direction. Therefore, two blending processes are performed on one bidirectional edge as shown in FIGS. 4 (1) and (2). FIG. 4 is a conceptual diagram of blending processing related to a bidirectional edge between the node i and the node j. FIG. 4 (1) represents the blending process in the direction from the node i to the node j, and FIG. 4 (2) represents the blending process in the direction from the node j to the node i.

  FIG. 5 is a conceptual diagram for explaining the blending process and corresponds to FIG. Here, with reference to FIG. 5, the blending process in the direction from node i to node j shown in FIG. 4A will be described as an example.

  In the blending process, interpolated data (blending data) in which the connection parts of the gesture data are mixed with respect to the gesture data 1 having the node i and the gesture data 2 having the node j so as not to unnaturally connect the movements. 1_2 is generated. In the present embodiment, the connected portions are interpolated using spherical linear interpolation by quaternions using frames for a fixed time. Specifically, blending data 1_2 of a connection section (section length m, where m is a predetermined value) connecting the gesture data 1 and the gesture data 2 is set to a section length m around the node i of the gesture data 1. The data 1_m and the gesture data 2 are generated using the data 2_m having the section length m around the node j.

  At this time, according to the ratio (u / m) of the distance u from the head of the connection section to the section length m of the connection section, the frame i corresponding to the distance u in the data 1_m corresponds to the distance u in the data 2_m. Mix frame j. Specifically, each frame constituting the blending data 1_2 is generated by Expression (8) and Expression (9). Equation (8) is an equation for a certain bone.

  Where m is the total number (predetermined value) of the frames (blending frames) constituting the blending motion data 1_2, u is the order from the top of the blending frame (1 ≦ u ≦ m), and q (k, u) is the uth The quaternion of the kth bone in the blending frame, q (k, i) is the quaternion of the kth bone in frame i, and q (j) is the quaternion of the kth bone in frame j. However, blending is not performed on the route. Equation (9) is a calculation formula of slerp (spherical linear interpolation).

  The blending data 1_2 is data of a connection portion between the gesture data 1 and the gesture data 2.

  Next, the dead end is removed from the motion graph. A dead end is a node whose degree is 1. In the motion graph, the number of edges connected to a node is called an order. The number of edges entering the node is referred to as the input order, and the number of edges exiting from the node is referred to as the output order. If the dead end is removed from the motion graph, a new dead end may occur. However, the dead end elimination is repeated until the dead end disappears.

Next, metadata is attached to each edge of the motion graph. A distance d (F ⁱ _B , F ^j _B ) between the node F ⁱ _B and the node F ^j _B related to the edge is given as a weight to the bidirectional edge. In addition, a phase ID and a duration related to the edge are given as labels to a unidirectional edge.

  The above is the description of the motion graph generation process. Thereby, a motion graph is generated for each category. The motion graph generation unit 20 generates a steady motion graph as a special motion graph. A steady motion graph is a motion graph that does not belong to a specific category. The steady motion graph is not limited to a specific category, but is generated in the same manner by the above-described motion graph generation processing using the gesture data for steady use.

[Motion Graph Database]
The motion graph database 30 stores the motion graph generated by the motion graph generation unit 20. The motion graph database 30 stores a category-specific motion graph and a steady motion graph.

  Next, the gesture generating apparatus 10 shown in FIG. 1 will be described. In FIG. 1, the gesture generation device 10 includes an input processing unit 11, a metadata generation unit 12, and a gesture data generation unit 13.

  A set of speech data and voice data is input to the gesture generation apparatus 10 as input data. The serif data is text data of the serif. The voice data is the voice data of the same set of serif data.

[Input processing section]
The input processing unit 11 performs morpheme analysis on the input data, and outputs a keyword string as a result of the morpheme analysis. For example, as a result of the morphological analysis of the speech data “Now, I must walk today”, the keyword string “Ja | Today | is | Walking | Shi | Nakya | Ne” is output. Next, the input processing unit 11 sets the temporal correspondence between the voice data and the keyword string. The correspondence in time between the speech data and the voice data is set in advance. If the speech data is synthesized speech, the correspondence between speech and speech can be obtained during speech synthesis, so the correspondence is set. In the case of voice data other than synthesized voice (recorded voice), the correspondence between voice and speech is set manually. Next, the input processing unit 11 records the duration of the line.

[Metadata generator]
The metadata generation unit 12 attaches a category label to each keyword in the keyword string output by the input processing unit 11 using a concept dictionary. For example, “WordNet” described in Non-Patent Document 3 can be used as the concept dictionary. Next, the metadata generation unit 12 selects a motion graph of the same category from the motion graph database 30 for each keyword. As a result, when a plurality of motion graphs are selected, one of the motion graphs is selected. For example, select one motion graph at random. On the other hand, when no motion graph is selected, a steady motion graph is selected.

  Next, the metadata generation unit 12 determines the timing of the gesture stroke. Specifically, the metadata generation unit 12 sets the start timing and end timing of keywords in the same category for the selected motion graph as stroke timings. However, when the steady motion graph is selected, the stroke timing is set to infinite with respect to the steady motion graph (not specifically defined).

  The metadata generation unit 12 includes metadata of audio data and keyword sequence information on time, serif duration information, motion graph information, and stroke timing information for the motion graph. And

  Note that the user may generate the metadata for the input data (serif data, voice data) manually using an authoring tool. In this case, the user arbitrarily performs selection of a motion graph, selection of a keyword corresponding to a stroke of the motion graph (setting of the timing of the stroke), setting of the temporal correspondence between the voice data and the keyword string, and the like.

[Draft data generator]
The gesture data generation unit 13 uses the metadata generated by the metadata generation unit 12 to generate gesture data that matches the speech of the speech. FIG. 6 is a flowchart showing the flow of the gesture data generation method according to the present embodiment. Hereinafter, an operation in which the gesture data generation unit 13 generates the gesture data will be described with reference to FIG.

(Step S11) The gesture data generation unit 13 selects a node that is the starting point of the gesture data from the motion graph. For example, among the first nodes in the motion data, the node having the shortest distance (highest connectivity) from the last pose of the gesture data is set as the start node.

(Step S12) The gesture data generation unit 13 searches for the optimum path from the start node on the motion graph and selects the path with the lowest cost. For this path search method, a path search technique described in Non-Patent Document 4 is used. The path search technique described in Non-Patent Document 4 searches for an optimal path from the start point by dynamic programming. Details of the optimum path search step will be described below.

  First, the cost of each path from the start point node u to all the nodes i on the motion graph is calculated by Expression (10). The first shortest path calculation operation related to the start point node u is the first operation.

  However, shortestPath (i, 1) is the cost of the path from the start node u to the node i by the first shortest path calculation operation. edgeCost (u, i) is the edge cost from node u to node i. The edge cost is calculated every time. The formula for calculating the edge cost is Equation (11).

  Note that the formula for calculating the edge cost of the steady motion graph is Expression (12).

  Next, in the k-th shortest path calculation operation after the second time, the cost of the optimal path from the start point node u to all the nodes v on the motion graph is calculated by Expression (13).

  V is a set of nodes on the motion graph. shorttestPath (v, k) is the cost of the optimum path from the start node u to the node v by the k-th shortest path calculation operation. edgeCost (i, v) is the edge cost from node i to node v.

  The second and subsequent shortest path calculation operations using this equation (13) are performed until the optimal path search end condition is satisfied.

(Step S13) The gesture data generation unit 13 determines the optimal path search end condition. The end conditions (a) to (d) for the optimum path search are shown below.
(A) When the path length exceeds a predetermined frame number N (corresponding to the self duration) except for the last node in the motion graph, the path of the search result is discarded as “excess”.
(B) When the last node in the motion graph is reached, the search result path is stored as an optimal path candidate.
(C) From the optimal path candidates, extract the optimal path candidates whose difference between the path length and the predetermined number of frames N is within a predetermined range. As the extracted optimum path candidates, when there are a path length less than the predetermined number of frames N and a path length exceeding the predetermined number of frames N, the path length is less than the predetermined number of frames N. Select the best path candidate.
(D) If there are a plurality of optimal path candidates selected in (c) above, the optimal path candidate with the lowest cost is determined as the optimal path.

(Step S14) If the optimal path search end conditions (a) to (d) are satisfied, the process proceeds to Step S15. On the other hand, if the optimal path search termination conditions (a) to (d) are not satisfied, the process returns to step S12.

(Step S15) The gesture data generation unit 13 specifies the gesture data corresponding to the optimum path from the motion graph based on the selected optimum path. Next, the gesture data generation unit 13 adjusts the gesture data corresponding to the optimum path according to the speech of the speech. FIG. 7 is an explanatory diagram of the gesture data adjustment method according to the present embodiment. The gesture data adjustment method according to the present embodiment will be described below with reference to FIG.

  As shown in FIG. 7, in the gesture data before adjustment, the timing of the stroke does not match the timing of the voice data of the corresponding keyword “walking”. For this reason, first, the stroke start timing is moved to match the voice data start timing of the corresponding keyword “walking”. Next, the stroke duration is expanded or contracted to match the voice data end timing of the corresponding keyword “walking”. The range of the expansion / contraction rate is set in advance so as not to be unnatural.

  Next, the preparation period is adjusted. Since the start timing of the preparation period coincides with the speech data start timing of the speech, the continuation time of the preparation period is expanded and contracted so that the end timing of the preparation period matches the start timing of the stroke. This stretch rate range is set in advance so as not to be unnatural.

  Next, the end period is adjusted. Regarding the end period, the duration of the end period is expanded and contracted so that the start timing is matched with the end timing of the stroke, and the end timing is matched with the speech data end timing of the speech. The range of the expansion / contraction rate is set in advance so as not to be unnatural.

(Step S16) The gesture data generation unit 13 performs randomness introduction processing on the adjusted gesture data. In this randomness introduction process, a plurality of short gesture data (referred to as random gesture data) is prepared in advance, and if there is a frame similar to the random gesture data in the preparation period or the end period, the frame is referred to. Replace with random gesture data. Specifically, the distance between the first frame of each random gesture data and each frame in the preparation period or the end period is calculated by Expression (6). When the distance calculated for the distance calculation target frame and the distance calculation target random gesture data is equal to or less than the threshold value, the distance calculation target random gesture data for the duration of the distance calculation target random gesture data from the distance calculation target frame in the preparation period or the end period The frame is replaced with the distance calculation target random gesture data. In the example of FIG. 7, the random gesture data 1000 is replaced in the preparation period and the end period.

  Random gesture data includes, for example, a gesture that raises the neck, a gesture that shakes the body, and a gesture that sticks out the tongue. By replacing the frame of the preparation period or the end period with this random gesture data, an accent can be added to the preparation period or the end period. Although the preparation period and the end period may be silent periods waiting for a line, the randomness introducing process can give the user an accent on the silent period and can prevent the user from being exposed.

  The gesture data generation unit 13 outputs the generated gesture data together with input data (serif data, voice data). With this output gesture data, it is possible to reproduce a natural gesture moving image that matches the speech of the input data.

  As mentioned above, although embodiment of this invention was explained in full detail with reference to drawings, the specific structure is not restricted to this embodiment, The design change etc. of the range which does not deviate from the summary of this invention are included.

  For example, the gesture data generation unit 13 may add a preparation period and an end period of a certain time before and after the stroke by using a predetermined steady pose for the gesture data having only a stroke. Further, the gesture data generation unit 13 may add an end period of a certain time after the stroke by using a pose of the preparation period for the gesture data having no end period. Further, the gesture data generation unit 13 may add a preparation period of a certain time before the stroke by using an end period pose for the gesture data having no preparation period.

  Also, if the stroke length of the motion graph is more than a predetermined time than the duration of the speech, switch to the steady motion graph, or a silent interval of a certain time immediately after the speech data speech corresponding to the stroke Or may be inserted.

  Further, a computer program for realizing the above-described gesture generation system 1 may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into the computer system and executed. Here, the “computer system” may include an OS and hardware such as peripheral devices.

  “Computer-readable recording medium” refers to a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a DVD (Digital Versatile Disk), and a built-in computer system. A storage device such as a hard disk.

Further, the “computer-readable recording medium” means a volatile memory (for example, DRAM (Dynamic DRAM) in a computer system that becomes a server or a client when a program is transmitted through a network such as the Internet or a communication line such as a telephone line. Random Access Memory)), etc., which hold programs for a certain period of time.
The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

DESCRIPTION OF SYMBOLS 1 ... gesture generation system, 10 ... gesture generation apparatus, 11 ... input processing part, 12 ... metadata generation part, 13 ... gesture data generation part, 20 ... motion graph generation part, 30 ... motion graph database

Claims (15)

Inputs speech data that is text data of speech, speech data of the speech, and a motion graph in which the plurality of gesture data are connected based on connectivity of the plurality of gesture data, and gesture data that matches the speech of the speech A gesture generating device for generating
A gesture data generation unit is provided that selects a path with the lowest cost on the motion graph based on the duration of the lines and adjusts the gesture data of the selected paths to match the speech of the lines. Characteristic gesture generation device.   The gesture data generation unit adjusts the start timing and the end timing of the stroke with respect to the speech data of the keyword in the speech corresponding to the stroke of the gesture data of the selected path. The gesture generating apparatus according to claim 1.   The gesture generation apparatus according to claim 2, wherein the gesture data generation unit adjusts the length of the selected path's gesture data to the duration of the line.   The said gesture data production | generation part expands / contracts the continuation time of this preparation period so that the completion timing of the preparation period of the selected path | pass gesture data may match the start timing of the said stroke. Gesture generation device.   The gesture data generation unit adjusts the start timing to the end timing of the stroke and the end timing to the speech data end timing of the lines for the end period of the selected path gesture data. The gesture generating device according to claim 3, wherein the duration of the end period is expanded or contracted.   The said gesture data generation part makes the start node the node having the best connectivity with the last pose of the gesture data among the first nodes of the gesture data included in the motion graph. The gesture generating apparatus according to any one of 5. There are multiple motion graphs by category,
The said gesture data production | generation part produces | generates the gesture data matched with the audio | voice of the said speech using the said motion graph same as the category of the keyword in the said speech, The any one of Claim 1 to 6 characterized by the above-mentioned. The gesture generating device described in 1.   The said gesture data production | generation part replaces with the frame which is similar from predetermined | prescribed gesture data with respect to the frame of the preparation period or completion | finish period of the selected gesture data of the said path | pass. The gesture generation device according to any one of the above.   The said gesture data production | generation part adds the preparation period and completion | finish period of a fixed time before and after a stroke using the predetermined steady pose with respect to the gesture data which has only a stroke, The 1st to 8 characterized by the above-mentioned. The gesture generation device according to any one of the above.   The said gesture data production | generation part adds the end period of a fixed time after a stroke using the pause of a preparation period with respect to the gesture data which does not have an end period, The any one of Claim 1 to 8 characterized by the above-mentioned. The gesture generating apparatus as described in the item.   The said gesture data production | generation part adds the preparation period of a fixed time before a stroke using the pause of an end period with respect to the gesture data without a preparation period, The any one of Claim 1 to 8 characterized by the above-mentioned. The gesture generating apparatus according to item 1.   The gesture data generation unit switches to a predetermined steady motion graph when the stroke length of the motion graph is equal to or longer than a line duration, or a voice corresponding to the stroke of the motion graph. The gesture generation device according to any one of claims 1 to 11, wherein a silent section of a fixed time is inserted immediately after a data line. The gesture generating device according to any one of claims 1 to 12,
A motion graph generation unit that sets each phase boundary of the preparation period, stroke, and end period of input gesture data to a node, and generates a motion graph with an edge based on connectivity between the nodes;
A motion graph database for storing the motion graph;
A gesture generation system characterized by comprising: Inputs speech data that is text data of speech, speech data of the speech, and a motion graph in which the plurality of gesture data are connected based on connectivity of the plurality of gesture data, and gesture data that matches the speech of the speech Is a gesture generation method of the gesture generation device for generating
The gesture generation device selects a path with the lowest cost on the motion graph based on the duration of the lines, and adjusts the gesture data of the selected paths according to the speech of the lines. A gesture generation method. Inputs speech data that is text data of speech, speech data of the speech, and a motion graph in which the plurality of gesture data are connected based on connectivity of the plurality of gesture data, and gesture data that matches the speech of the speech To the computer of the gesture generation device that generates
A computer program for executing a step of selecting a minimum cost path on the motion graph based on the duration of the serif and performing adjustment to match the voice of the serif with respect to the gesture data of the selected path.

Images