JP7417015B2 - Gesture shape learning device, gesture shape estimation device, methods thereof, and programs - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act (1) ▲1▼ Website publication date October 14, 2019 ▲2▼ Website address ICMI2019 Conference Program Website https://icmi. acm. org/2019/index. php? id=program (2) ▲1▼ Website publication date October 14, 2019 ▲2▼ Website address ICMI2019 website https://dl. acm. org/citation. cfm? id=3353736 (3) ▲1▼ Date October 14, 2019 - October 18, 2019 (Notification date: October 15, 2019) ▲2▼ Meeting name, venue ICMI 2019 Conference GRAND DUSHULAKE (China) No. 299, Yue Street, Tech■■ District, Suzhou)

The present invention relates to a technique for adding appropriately shaped iconographic gestures to words.

In speech, hand gestures (hereinafter also referred to as "gestures") that express images of concrete objects are called iconographic gestures. Non-Patent Document 1 describes a technology that uses a decision tree learning method to build an estimator that collects images on the web searched based on a word and estimates three types of shapes for that word from the SIFT features of the image. is listed.

Yuki Kadono, Yutaka Takase, and Yukiko I. Nakano, "Generating iconic gestures based on graphic data analysis and clustering," ACM/IEEE International Conference on Human-Robot Interaction, pp. 447-448, 2016.

However, the conventional technique described in Non-Patent Document 1 has a problem in that the estimation accuracy is not high.

An object of the present invention is to estimate an iconographic gesture having an appropriate shape for a word with high accuracy.

In order to solve the above problems, a gesture shape learning device according to a first aspect of the present invention stores learning data in which a plurality of images corresponding to a certain word are associated with a gesture shape corresponding to the word. A learning data storage unit, and a gesture shape learning unit that uses the learning data to learn a gesture shape determination model that receives as input a feature amount consisting of color information of each pixel extracted from an image and estimates a gesture shape corresponding to that image. and, including.

A gesture shape learning device according to a second aspect of the present invention includes a learning data storage unit that stores learning data in which a plurality of images corresponding to a certain word are associated with a gesture shape corresponding to the word; A basic shape learning unit that uses learning data to learn a basic shape determination model that estimates the basic shape of the gesture shape corresponding to the image by inputting feature quantities consisting of color information of each pixel extracted from the image. The image forming apparatus includes an aspect ratio learning unit that uses learning data to learn an aspect ratio determination model that receives a feature amount consisting of color information of each pixel as input and estimates the aspect ratio of a gesture shape corresponding to the image.

A gesture shape estimating device according to a third aspect of the present invention includes a model storage unit that stores the gesture shape determination model learned by the gesture shape learning device according to the first aspect, and color information of each pixel extracted from an input image. The gesture shape estimation unit inputs the feature amount into the gesture shape determination model and estimates the gesture shape corresponding to the input image.

A gesture shape estimating device according to a fourth aspect of the present invention includes a model storage unit that stores a basic shape determining model and an aspect ratio determining model learned by the gesture shape learning device according to the second aspect, and each pixel extracted from an input image. A basic shape estimator that inputs feature quantities consisting of color information into a basic shape determination model to estimate the basic shape of a gesture shape corresponding to the input image, and a feature consisting of color information of each pixel extracted from the input image. an aspect ratio estimation unit that inputs the amount into an aspect ratio determination model to estimate the aspect ratio of a gesture shape corresponding to the input image; and a gesture shape determination unit that determines the gesture shape corresponding to the input image from the basic shape and aspect ratio. including.

According to the present invention, an iconographic gesture having an appropriate shape for a word can be estimated with high accuracy. In particular, by estimating the basic shape of the gesture shape and the aspect ratio of the basic shape in two stages, it is possible to estimate the gesture shape with higher accuracy.

FIG. 1 is a diagram illustrating the functional configuration of a gesture shape learning device. FIG. 2 is a diagram illustrating the processing procedure of the gesture shape learning method. FIG. 3 is a diagram for explaining the configuration of a gesture shape determination model. FIG. 4 is a diagram illustrating the functional configuration of the gesture shape estimation device. FIG. 5 is a diagram illustrating a processing procedure of the gesture shape estimation method. FIG. 6 is a diagram illustrating the functional configuration of the gesture shape learning device. FIG. 7 is a diagram illustrating the processing procedure of the gesture shape learning method. FIG. 8 is a diagram illustrating the functional configuration of the gesture shape estimation device. FIG. 9 is a diagram illustrating the processing procedure of the gesture shape estimation method. FIG. 10 is a diagram illustrating the functional configuration of the gesture shape estimation device. FIG. 11 is a diagram illustrating the processing procedure of the gesture shape estimation method. FIG. 12 is a diagram illustrating the functional configuration of a computer.

Embodiments of the present invention will be described in detail below. Note that in the drawings, components having the same functions are designated by the same numbers, and redundant explanation will be omitted.

In this invention, RGB information of each pixel of an image is used as input information (feature amount) for estimating a gesture shape. Deep learning technology is used as an estimator that estimates gesture shapes from input information. Furthermore, when constructing an estimator, a mode in which a gesture shape is estimated from the feature amount in one step, and a mode in which the basic shape of the gesture shape and the aspect ratio of the basic shape are estimated in two steps will be explained. Even with one-stage estimation, it is possible to estimate the gesture shape with higher precision than in the prior art, but with two-stage estimation, it is possible to estimate the gesture shape with higher precision.

[First embodiment]
The first embodiment uses a gesture shape learning device 1 that learns a gesture shape determination model that estimates a gesture shape in one step from image features, and input data using the gesture shape determination model learned by the gesture shape learning device 1. The gesture shape estimating device 2 is configured to estimate a gesture shape from an image.

<Gesture shape learning device 1>
As illustrated in FIG. 1, the gesture shape learning device 1 of the first embodiment includes a word dictionary storage section 110, an object image storage section 120, an image size conversion section 11, a learning data generation section 12, a learning data storage section 130, It includes a gesture shape learning section 13 and a model storage section 140. The gesture shape learning method of the first embodiment is realized by the gesture shape learning device 1 performing the processing of each step illustrated in FIG. 2 .

The gesture shape learning device is, for example, a special computer configured by loading a special program into a publicly known or dedicated computer that has a central processing unit (CPU), a main memory (RAM), etc. It is a very good device. The gesture shape learning device executes each process under the control of, for example, a central processing unit. The data input to the gesture shape learning device and the data obtained through each process are stored, for example, in the main memory, and the data stored in the main memory is read out to the central processing unit as necessary. Used for other processing. Each processing unit of the gesture shape learning device may be configured at least in part by hardware such as an integrated circuit. Each storage unit included in the gesture shape learning device includes, for example, a main storage device such as a RAM (Random Access Memory), an auxiliary storage device constituted by a semiconductor memory element such as a hard disk, an optical disk, or a flash memory, or It can be configured with middleware such as relational databases and key-value stores.

With reference to FIG. 2, the processing procedure of the gesture shape learning method executed by the gesture shape learning device 1 of the first embodiment will be described.

In step S110, the gesture shape learning device 1 constructs a word dictionary database and stores the word dictionary database in the word dictionary storage unit 110. The word dictionary database is a database in which a plurality of predetermined words are associated with information on gesture shapes corresponding to the words. The plurality of words are, for example, words representing the names of about 1000 objects extracted from a dictionary or the like (hereinafter also referred to as "object name words"). Gesture shapes include, for example, "circle," "horizontal ellipse," "vertical ellipse," "square," "horizontal square," "vertical square," "equilateral triangle," "horizontal triangle," "vertical triangle," "wavy," "linear," and "horizontal diamond." Choose from 14 types: ``vertical diamond'' and ``unknown.'' Below, out of the 14 types of gesture shapes, we have integrated the ones that are selected in small numbers, and we will integrate the 7 types of gesture shapes: ``circle'', ``horizontal ellipse'', ``vertical ellipse'', ``square'', ``horizontal rectangle'', ``vertical rectangle'', and ``linear''. The type shall be used. However, the gesture shape options are not limited to the above, and may be set arbitrarily according to the usage environment. It is assumed that the association between each word and the gesture shape is performed manually by a plurality of annotators.

In step S120, the gesture shape learning device 1 constructs an object image database and stores the object image database in the object image storage unit 120. The object image database is a database in which each word included in the word dictionary database is associated with a plurality of images corresponding to that word. A plurality of images can be collected by, for example, searching for images on the Internet using each word as a search term, and downloading about 10 suitable images from the search results.

In step S11, the image size conversion unit 11 converts each image included in the object image database into an appropriate image size according to the machine learning method using the images. For example, it may be converted to an RGB image of 224 x 224 pixels.

In step S12, the learning data generation unit 12 adds a gesture shape corresponding to the image as teacher data to the image converted by the image size conversion unit 11, and generates learning data. The gesture shape corresponding to the image can be obtained by acquiring the word associated with the image in the object image database, and then acquiring the gesture shape associated with the word in the word dictionary database. The learning data generation unit 12 stores the generated learning data in the learning data storage unit 130.

In step S13, the gesture shape learning unit 13 uses the learning data stored in the learning data storage unit 130 to create a gesture shape determination model that takes as input the feature amount extracted from the image and estimates a gesture shape corresponding to the image. Learn. The gesture shape determination model is constructed using a neural network. FIG. 3 shows a specific configuration example of the gesture shape determination model. The inputs of the gesture shape determination model (Input 1, Input 2, ..., Input n in FIG. 3) correspond to each of n (for example, 10) images for a word. First, a trained VGG-16 model using deep learning is applied to each input image (224 × 224 pixel RGB 3-channel image) to obtain 4096-dimensional features (VGG -16). Next, a 4096-dimensional average vector is calculated from the n 4096-dimensional feature amounts obtained from the n images (Average in FIG. 3). In this way, we construct a gesture shape determination model that takes as input the 4096-dimensional average vector obtained from n images and outputs the gesture shape corresponding to each image. Note that at this time, each VGG-16 paired with Inputs 1 to n is not necessarily required, and in the Average in Figure 3, the feature extraction results (VGG-16) of n inputs are added and averaged. You can. Further, the use of VGG-16 is just one example, and other general models that extract feature amounts using an image as input may also be used. Similarly, the number of dimensions to be obtained may be arbitrary depending on the model. As a learning method, we calculate the likelihood of labels representing seven types of gesture shapes by applying the output of a fully connected two-layer neural network to the Softmax function. The first layer neural network (FC1 in FIG. 3) is a fully connected neural network. FC1 uses, for example, a Relu function as an activation function and obtains a 128-dimensional vector as an output. The second layer neural network (FC2 in FIG. 3) receives the output vector of FC1 and calculates the likelihood of each label corresponding to the seven types of gesture shapes using the Softmax function. As a result of machine learning using this neural network, a gesture shape determination model is obtained. The gesture shape learning unit 13 stores the learned gesture shape determination model in the model storage unit 140.

<Gesture shape estimation device 2>
The gesture shape estimation device 2 of the first embodiment includes a model storage section 140, an image size conversion section 21, and a gesture shape estimation section 22, as illustrated in FIG. The gesture shape estimation method of the first embodiment is realized by the gesture shape estimation device 2 performing the processing of each step illustrated in FIG.

The gesture shape estimation device is, for example, a special computer configured by loading a special program into a publicly known or dedicated computer having a central processing unit (CPU), a main memory (RAM), etc. It is a very good device. The gesture shape estimation device executes each process under the control of a central processing unit, for example. The data input to the gesture shape estimation device and the data obtained through each process are stored, for example, in the main memory, and the data stored in the main memory is read out to the central processing unit as necessary. Used for other processing. Each processing unit of the gesture shape estimation device may be configured at least in part by hardware such as an integrated circuit. Each storage unit included in the gesture shape estimation device includes, for example, a main storage device such as a RAM (Random Access Memory), an auxiliary storage device configured with a semiconductor memory element such as a hard disk, an optical disk, or a flash memory, or It can be configured with middleware such as relational databases and key-value stores.

With reference to FIG. 5, the processing procedure of the gesture shape estimation method executed by the gesture shape estimation device 2 of the first embodiment will be described.

The model storage unit 140 stores the gesture shape determination model learned by the gesture shape learning device 1.

The gesture shape estimation device 2 is input with n images corresponding to object name words to be estimated. These images may be downloaded and collected from the Internet, for example, as in the case of constructing the object image database, or may be collected using other methods.

In step S21, the image size converter 21 converts each input image to an image size suitable for model application. That is, the image size converter 21 converts the size of the input image in the same manner as the image size converter 11 of the gesture shape learning device 1.

In step S22, the gesture shape estimation unit 22 inputs the n input images converted by the image size conversion unit 21 into the gesture shape determination model stored in the model storage unit 140, and generates a gesture corresponding to the input image. Estimate the shape. Specifically, first, trained VGG-16 is applied to each input image to obtain 4096-dimensional features. Next, a 4096-dimensional average vector is calculated from the 4096-dimensional feature values obtained from each input image. This 4096-dimensional average vector is input to a trained neural network, and the likelihoods of labels representing seven types of gesture shapes are obtained as output results. Then, the gesture shape with the maximum likelihood of the output result is output as the gesture shape corresponding to the object name word targeted for estimation.

[Second embodiment]
The second embodiment uses a gesture shape learning device 3 that learns a gesture shape determination model that estimates a gesture shape in two stages from image features, and inputs using the gesture shape determination model learned by the gesture shape learning device 3. It consists of a gesture shape estimation device 4 that estimates gesture shapes from images. In the two-step estimation, first the basic shape of the gesture shape is estimated, and then the aspect ratio of the basic shape is estimated. As illustrated in the first embodiment, there are seven types of gesture shapes to be estimated: circle, horizontal ellipse, vertical ellipse, square, horizontal rectangle, vertical rectangle, and linear. For example, the basic shape may be set to three types: "circle,""square," and "linear," and the aspect ratio may be set to three types: "horizontal,""vertical," and "uniform." The gesture shape that is finally output is determined by the combination of the basic shape and the aspect ratio. For example, if the basic shape estimation result is "circle" and the aspect ratio estimation result is "horizontally long", the output gesture shape will be "horizontally long ellipse". Hereinafter, differences from the first embodiment will be mainly described.

<Gesture shape learning device 3>
The gesture shape learning device 3 of the second embodiment differs from the gesture shape learning of the first embodiment in that the gesture shape learning section 13 includes a basic shape learning section 131 and an aspect ratio learning section 132, as illustrated in FIG. Different from device 1. The gesture shape learning method of the second embodiment is realized by the gesture shape learning device 3 performing the processing of each step illustrated in FIG. 7 .

In step S131, the basic shape learning unit 131 uses the learning data stored in the learning data storage unit 130 to input the feature amount extracted from the image, and uses the basic shape to estimate the basic shape of the gesture shape corresponding to the image. Learn a shape decision model. The structure of the basic shape determination model is similar to the gesture shape determination model of the first embodiment shown in FIG. In the first embodiment, there are seven types of gesture shapes in the learning data: "circle," "horizontal ellipse," "vertical ellipse," "square," "horizontal square," "vertical square," and "linear." Among these gesture shapes, Learn by converting ``circle'', ``horizontal ellipse'', and ``vertical ellipse'' into ``circle'', and converting ``square'', ``horizontal rectangle'', and ``vertical rectangle'' into ``square''. The basic shape learning unit 131 stores the learned basic shape determination model in the model storage unit 140.

In step S132, the aspect ratio learning unit 132 uses the learning data stored in the learning data storage unit 130 to estimate the aspect ratio of the gesture shape corresponding to the image, using the feature amount extracted from the image as input. Learn the ratio decision model. The structure of the aspect ratio determination model is similar to the gesture shape determination model of the first embodiment shown in FIG. In the first embodiment, there are seven types of gesture shapes in the learning data: "circle," "horizontal ellipse," "vertical ellipse," "square," "horizontal square," "vertical square," and "linear." Among these gesture shapes, Learn by converting "circle" and "square" to "even", "horizontal ellipse" and "horizontal square" to "horizontal", and "vertical ellipse" and "vertical rectangle" to "vertical". The aspect ratio learning unit 132 stores the learned aspect ratio determination model in the model storage unit 140.

<Gesture shape estimation device 4>
The gesture shape estimating device 4 of the second embodiment is characterized in that the gesture shape estimating section 22 includes a basic shape estimating section 221, an aspect ratio estimating section 222, and a gesture shape determining section 223, as illustrated in FIG. This is different from the gesture shape estimation device 2 of one embodiment. The gesture shape estimation method of the second embodiment is realized by the gesture shape estimation device 4 performing the processing of each step illustrated in FIG. 9 .

In step S221, the basic shape estimation unit 221 inputs the n input images converted by the image size conversion unit 21 into the basic shape determination model stored in the model storage unit 140, and generates a gesture corresponding to the input image. Estimate the basic shape of the shape. The basic shape estimation unit 221 outputs the basic shape with the maximum likelihood of the output result of the basic shape determination model to the gesture shape determination unit 223 as the basic shape of the gesture shape.

Note that if the basic shape of the gesture shape estimated by the basic shape estimation unit 221 is a shape that does not have a different aspect ratio, such as a "linear" shape, the subsequent processing is not performed and the basic shape is used as the estimation target. Output as a gesture shape corresponding to the object name word.

In step S222, the aspect ratio estimation unit 222 inputs the n input images converted by the image size conversion unit 21 into the aspect ratio determination model stored in the model storage unit 140, and generates a gesture corresponding to the input image. Estimate the aspect ratio of a shape. The aspect ratio estimation unit 222 outputs the aspect ratio for which the likelihood of the output result of the aspect ratio determination model is maximum to the gesture shape determination unit 223 as the aspect ratio of the gesture shape.

In step S223, the gesture shape determination unit 223 determines a gesture shape based on the basic shape outputted by the gesture shape determination unit 223 and the aspect ratio outputted by the aspect ratio estimation unit 222, and determines a gesture shape corresponding to the object name word targeted for estimation. output as a gesture shape.

[Third embodiment]
In the third embodiment, the gesture shape estimation technology of the first embodiment or the second embodiment is combined with the motion label generation technology. The motion label generation technology is a technology that generates a label for the motion of the whole body that matches the content of the utterance from the utterance text obtained by transcribing the utterance (for example, see References 1 and 2).

[Reference 1] International Publication No. 2019/160104 [Reference 2] Ryo Ishii, Taichi Katayama, Ryuichiro Higashinaka, and Junji Tomita, "Generating Body Motions using Spoken Language in Dialogue," Proceedings of the 18th International Conference on Intelligent Virtual Agents (IVA '18), pp. 87-92, 2018.

Motion label generation technology takes spoken text as input and generates information indicating what kind of hand gesture to perform for each clause of the spoken text. For example, in the techniques described in References 1 and 2, a label indicating whether or not an iconographic gesture is to be performed is output for each phrase. In the third embodiment, when this label is output, the gesture shape estimation of the first embodiment or the second embodiment is performed using words included in the phrase for which the label was output and the preceding and following phrases. This allows us to estimate in more detail what shape of gesture to generate. Hereinafter, differences between the first embodiment and the second embodiment will be mainly described.

<Gesture shape estimation device 5>
As illustrated in FIG. 10, the gesture shape estimation device 5 of the third embodiment includes a model storage unit 140, an image size conversion unit 21, and a gesture shape estimation unit 22, as in the first embodiment, and further includes: It includes a motion label generation section 31 and an input image generation section 32. The gesture shape estimation method of the third embodiment is realized by the gesture shape estimation device 5 performing the processing of each step illustrated in FIG. 11 .

In step S31, the motion label generation unit 31 generates a motion label indicating whether or not to perform a gesture for each phrase from the input speech text. When the motion label representing performing a gesture is generated, the motion label generation unit 31 notifies the input image generation unit 32 of this fact.

In step S32, the input image generation unit 32 collects n images corresponding to the phrase included in the phrase and the phrases before and after the phrase for which a motion label indicating performing a gesture has been generated. Images may be collected, for example, by acquiring images corresponding to the word from the object image database used in the gesture shape learning device, or by using the word as a search term, as was done when generating the object image database. You can also search for images on the Internet and download them. The input image generation unit 32 inputs the collected n images to the image size conversion unit 21 as input images.

In step S21, the image size converter 21 converts each input image into an image size suitable for model application, and inputs it to the gesture shape estimator 22, as in the first embodiment.

In step S22, the gesture shape estimation unit 22 estimates the gesture shape from the n input images, similarly to the first embodiment.

In the third embodiment, a configuration has been described in which the gesture shape estimation device 2 of the first embodiment is combined with the motion label generation technology, but it may be configured to be combined with the gesture shape estimation device 4 of the second embodiment. In this case, the gesture shape estimation device 5 may be configured such that the gesture shape estimation section 22 includes a basic shape estimation section 221 , an aspect ratio estimation section 222 , and a gesture shape determination section 223 .

Conventional motion label generation techniques have not been able to estimate the specific shape of iconographic gestures. If the gesture shape estimation technique of the third embodiment is used, it becomes possible to estimate a more detailed gesture shape.

Although the embodiments of this invention have been described above, the specific configuration is not limited to these embodiments, and even if the design is changed as appropriate without departing from the spirit of this invention, Needless to say, it is included in this invention. The various processes described in the embodiments are not only executed in chronological order according to the order described, but also may be executed in parallel or individually depending on the processing capacity of the device that executes the processes or as necessary.

[Program, recording medium]
When the various processing functions of each device described in the above embodiments are realized by a computer, the processing contents of the functions that each device should have are described by a program. By loading this program into the storage unit 1020 of the computer shown in FIG. 12 and causing it to operate in the control unit 1010, input unit 1030, output unit 1040, etc., various processing functions in each of the above devices are realized on the computer. Ru.

A program describing the contents of this process can be recorded on a computer-readable recording medium. The computer-readable recording medium may be of any type, such as a magnetic recording device, an optical disk, a magneto-optical recording medium, or a semiconductor memory.

Further, this program is distributed by, for example, selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, this program may be distributed by storing the program in the storage device of the server computer and transferring the program from the server computer to another computer via a network.

A computer that executes such a program, for example, first stores a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. When executing a process, this computer reads a program stored in its own storage device and executes a process according to the read program. In addition, as another form of execution of this program, the computer may directly read the program from a portable recording medium and execute processing according to the program, and furthermore, the program may be transferred to this computer from the server computer. The process may be executed in accordance with the received program each time. In addition, the above-mentioned processing is executed by a so-called ASP (Application Service Provider) type service, which does not transfer programs from the server computer to this computer, but only realizes processing functions by issuing execution instructions and obtaining results. You can also use it as Note that the program in this embodiment includes information that is used for processing by an electronic computer and that is similar to a program (data that is not a direct command to the computer but has a property that defines the processing of the computer, etc.).

Further, in this embodiment, the present apparatus is configured by executing a predetermined program on a computer, but at least a part of these processing contents may be realized by hardware.

Claims (11)

Using a database in which a plurality of images corresponding to a certain word are associated with gesture shapes corresponding to the word, the gesture shapes corresponding to the images are assigned as training data, and the generated learning data is stored. a learning data storage unit;
a gesture shape learning unit that uses the learning data to learn a gesture shape determination model that receives as input a feature amount consisting of color information of each pixel extracted from the image and estimates a gesture shape corresponding to the image;
A gesture shape learning device including: Using a database in which a plurality of images corresponding to a certain word are associated with gesture shapes corresponding to the word, the gesture shapes corresponding to the images are assigned as training data, and the generated learning data is stored. a learning data storage unit;
a basic shape learning unit that uses the training data to learn a basic shape determination model that receives as input a feature amount consisting of color information of each pixel extracted from the image and estimates a basic shape of a gesture shape corresponding to the image;
an aspect ratio learning unit that uses the learning data to learn an aspect ratio determination model that receives as input a feature amount consisting of color information of each pixel extracted from the image and estimates the aspect ratio of a gesture shape corresponding to the image;
A gesture shape learning device including: a model storage unit that stores a gesture shape determination model learned by the gesture shape learning device according to claim 1;
a gesture shape estimation unit that inputs a feature amount consisting of color information of each pixel extracted from the input image to the gesture shape determination model and estimates a gesture shape corresponding to the input image;
A gesture shape estimation device including: a model storage unit that stores the basic shape determination model and the aspect ratio determination model learned by the gesture shape learning device according to claim 2;
a basic shape estimating unit that inputs a feature amount consisting of color information of each pixel extracted from the input image to the basic shape determination model to estimate the basic shape of a gesture shape corresponding to the input image;
an aspect ratio estimating unit that inputs feature quantities consisting of color information of each pixel extracted from the input image to the aspect ratio determination model to estimate the aspect ratio of the gesture shape corresponding to the input image;
a gesture shape determination unit that determines a gesture shape corresponding to the input image from the basic shape and the aspect ratio;
A gesture shape estimation device including: The gesture estimation device according to claim 3 or 4,
a motion label generation unit that generates a motion label representing a phrase to perform a gesture from the uttered text;
an input image generation unit that obtains, as the input image, an image corresponding to a word near the phrase for which the motion label has been generated;
A gesture shape estimation device further comprising: Using a database in which a plurality of images corresponding to a certain word and gesture shapes corresponding to the word are associated with the learning data storage unit, the gesture shape corresponding to the image is given as training data, and the generated image is generated. learning data is stored.
A gesture shape learning unit receives as input a feature amount consisting of color information of each pixel extracted from an image, and learns a gesture shape determination model for estimating a gesture shape corresponding to the image using the learning data,
Gesture shape learning method. Using a database in which a plurality of images corresponding to a certain word and gesture shapes corresponding to the word are associated with the learning data storage unit, the gesture shape corresponding to the image is given as training data, and the generated image is generated. learning data is stored.
The basic shape learning unit receives as input the feature amount consisting of color information of each pixel extracted from the image, and uses the learning data to learn a basic shape determination model that estimates the basic shape of the gesture shape corresponding to the image. ,
The aspect ratio learning unit receives as input the feature amount consisting of color information of each pixel extracted from the image, and learns an aspect ratio determination model for estimating the aspect ratio of the gesture shape corresponding to the image using the learning data. ,
Gesture shape learning method. A gesture shape determination model learned by the gesture shape learning method according to claim 6 is stored in the model storage unit,
a gesture shape estimation unit inputs a feature amount consisting of color information of each pixel extracted from the input image to the gesture shape determination model, and estimates a gesture shape corresponding to the input image;
Gesture shape estimation method. A basic shape determining model and an aspect ratio determining model learned by the gesture shape learning device according to claim 7 are stored in the model storage unit,
a basic shape estimating unit inputs a feature amount consisting of color information of each pixel extracted from the input image to the basic shape determination model, and estimates a basic shape of a gesture shape corresponding to the input image,
an aspect ratio estimating unit inputs a feature amount consisting of color information of each pixel extracted from an input image to the aspect ratio determination model, and estimates an aspect ratio of a gesture shape corresponding to the input image;
a gesture shape determination unit determines a gesture shape corresponding to the input image from the basic shape and the aspect ratio;
Gesture shape estimation method. A program for causing a computer to function as the gesture shape learning device according to claim 1 or 2. A program for causing a computer to function as the gesture shape estimation device according to claim 3.

Images