JP7623619B2 - Learning device, estimation device, learning method, estimation method, and program - Google Patents

Description

The present invention relates to a learning device, an estimation device, a learning method, an estimation method and a program.

With the recent development of sensing technology and machine learning technology, efforts are being made to use machines to recognize a wide variety of objects in the real world. For example, it is possible to recognize human behavior, the surrounding environment, and objects from video information obtained with a general RGB camera. By using a multispectral camera, it is becoming possible to recognize more detailed information, such as analyzing the material of an object. Furthermore, in addition to video, it is possible to recognize the content of people's conversations and emotions from audio information.

In this way, active efforts are being made to develop machine learning technologies that utilize various sensor information and modal information. For example, in Non-Patent Document 1, an image and a text describing it are given, and by directly learning the relationship between image features and text features, it becomes possible to assign a description to any image. Meanwhile, in Non-Patent Document 2, an autoencoder is used to learn a new feature representation common to multiple modalities, making it possible to solve tasks in multiple modalities with a single feature representation.

Vicente Ordonez, Girish Kulkarni, Tamara L Berg, "Im2Text: Describing Images Using 1 Million Captioned Photographs" [online], [Retrieved November 26, 2021], Internet <URL: https://papers.nips.cc/paper/2011/file/5dd9db5e033da9c6fb5ba83c7a7ebea9-Paper.pdf> Jiquan Ngiam et.al.," Multimodal Deep Learning" [online], [Retrieved November 26, 2021], Internet <URL: https://icml.cc/2011/papers/399_icmlpaper.pdf>

However, these prior art techniques are based on the premise that all modal information is attached to each piece of training data. For example, if the modals are images and language, an image and its description must represent the same instance, and data consisting of only images or only text cannot become training data. In other words, learning cannot be carried out unless image data and language data that are linked to each other exist.

Meanwhile, the accuracy of analysis is higher the more accurately the information common to the linked data is acquired. Therefore, to achieve highly accurate recognition, it is necessary to use a large amount of data linked with multiple modal information to increase the accuracy of acquiring the information common to the linked data. However, it is very difficult to prepare a large amount of data linked with multiple modal information. Therefore, with conventional techniques such as those described in the above prior art documents, it was sometimes necessary to make a large effort to estimate information common to multiple types of data.

In view of the above circumstances, the present invention aims to provide a technology that reduces the effort required to estimate information common to multiple types of data.

One aspect of the present invention is a learning device that includes a multimodal network, which is a neural network including a first encoder that acquires features of input data, a second encoder that acquires features of the input data, a first decoder that decodes the input data, and a second decoder that decodes the input data; and a network control unit that updates the multimodal network using a cross self-consistency loss that indicates the difference between a self-consistent result, which is a result of input data being converted by the first encoder, the second encoder, the first decoder, and the second decoder, and the data, and a common loss that indicates the difference between a result of the first encoder encoding data indicating a first attribute of a main subject, which is a person, living thing, thing, intangible object, or event having multiple attributes, and a result of the second encoder encoding data indicating a second attribute of the main subject that is different from the first attribute.

One aspect of the present invention is a multimodal network that is a neural network including a data acquisition unit that acquires first type target data indicating a first attribute of an estimation target and second type target data indicating a second attribute different from the first attribute among the attributes of the estimation target, a first encoder that acquires features of input data, a second encoder that acquires features of the input data, a first decoder that decodes the input data, and a second decoder that decodes the input data; and a self-agnosing function that is a result of input data being converted by the first encoder, the second encoder, the first decoder, and the second decoder. and an estimation unit that performs processing performed by the trained multimodal network obtained by a learning device comprising: a network control unit that updates the multimodal network using a cross self-consistency loss that indicates the difference between a contradiction result and the data, and a common loss that indicates the difference between a result of the first encoder encoding data indicating a first attribute of a main object which is a person, a living thing, an object, an intangible object, or an event having multiple attributes, and a result of the second encoder encoding data indicating a second attribute which is an attribute of the main object and different from the first attribute, on the first type of target data and the second type of target data.

One aspect of the present invention is a learning method comprising: a network execution step of executing a multimodal network, which is a neural network including a first encoder that acquires features of input data, a second encoder that acquires features of the input data, a first decoder that decodes the input data, and a second decoder that decodes the input data; and a network control step of updating the multimodal network using a cross self-consistency loss indicating the difference between a self-consistent result, which is a result of input data being converted by the first encoder, the second encoder, the first decoder, and the second decoder, and the data; and a common loss indicating the difference between a result of the first encoder encoding data indicating a first attribute of a main subject, which is a person, living thing, thing, intangible object, or event having multiple attributes, and a result of the second encoder encoding data indicating a second attribute of the main subject that is different from the first attribute.

One aspect of the present invention is a multimodal network that is a neural network including a data acquisition step of acquiring first type target data indicating a first attribute of an estimation target and second type target data indicating a second attribute different from the first attribute among the attributes of the estimation target, a first encoder that acquires features of input data, a second encoder that acquires features of the input data, a first decoder that decodes the input data, and a second decoder that decodes the input data, and a self-agnosing function that is a result of the input data being converted by the first encoder, the second encoder, the first decoder, and the second decoder. and an estimation step of executing, on the first type target data and the second type target data, a process executed by the trained multimodal network obtained by a learning device including: a network control unit that updates the multimodal network using a cross self-consistency loss that indicates the difference between a contradiction result and the data, and a common loss that indicates the difference between a result of the first encoder encoding data indicating a first attribute of a main target which is a person, a living thing, an object, an intangible object, or an event having multiple attributes, and a result of the second encoder encoding data indicating a second attribute which is an attribute of the main target and different from the first attribute.

One aspect of the present invention is a program for causing a computer to function as the above-mentioned learning device.

One aspect of the present invention is a program for causing a computer to function as the above-mentioned estimation device.

The present invention makes it possible to reduce the effort required to estimate information common to multiple types of data.

FIG. 1 is a first explanatory diagram illustrating an overview of a learning device according to an embodiment. FIG. 2 is a second explanatory diagram illustrating an overview of the learning device according to the embodiment. FIG. 3 is a third explanatory diagram illustrating an overview of the learning device according to the embodiment. FIG. 4 is a fourth explanatory diagram illustrating an overview of the learning device according to the embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a learning apparatus according to an embodiment. FIG. 2 is a diagram illustrating an example of a control unit included in the learning device of the embodiment. 1 is a flowchart illustrating an example of a flow of a process executed by the learning device of the embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of an estimation apparatus according to an embodiment. FIG. 2 is a diagram illustrating an example of a functional configuration of a control unit included in the estimation device of the embodiment. 4 is a flowchart showing an example of a flow of a process executed by the estimation device of the embodiment.

(Embodiment)
An overview of a learning device 1 according to an embodiment will be described with reference to Figures 1 to 4. The learning device 1 updates a mathematical model (hereinafter referred to as a "multimodal estimation model") that estimates information common to two types of data through machine learning learning.

Since the multimodal estimation model estimates information common to two types of data, when the two types of data indicate different attributes of the same object, the object to be estimated is the object whose attributes the two types of data indicate. One of the two types of data indicating the attributes of the same object is called the first type of data, and the other of the two types of data is called the second type of data. In other words, the first type of data and the second type of data are information on different attributes related to the same object.

Hereinafter, the object whose attributes the first type data and the second type data indicate will be referred to as the main object. The main object may be any person, living thing, thing, intangible object, or event, so long as it has multiple attributes. In other words, the main object may be a person with multiple attributes, a living thing with multiple attributes, an object with multiple attributes, an intangible object with multiple attributes, or an event with multiple attributes.

As described above, the multimodal estimation model estimates an object when two types of data indicate different attributes of the same object, so the main object is an example of an object to be estimated by the multimodal estimation model. Furthermore, the first type of data indicates a first attribute of the main object, and the second type of data indicates a second attribute of the main object. The second attribute (hereinafter referred to as the "second attribute") is an attribute that is different from the first attribute (hereinafter referred to as the "first attribute") among the attributes possessed by the main object.

For example, if the first type of data indicates the shape of an object, the second type of data indicates the name of that object. In other words, while the first attribute is the shape, the second attribute is, for example, the name. In this way, the difference in type between the first type of data and the second type of data is specifically the difference in the attributes they indicate.

A mathematical model is a set including one or more processes whose execution conditions and order (hereinafter referred to as "execution rules") are predetermined. Learning means updating the mathematical model using machine learning methods. Updating the mathematical model means appropriately adjusting the parameter values in the mathematical model. Executing a mathematical model means executing each process included in the mathematical model in accordance with the execution rules.

The mathematical model is updated through learning until a predetermined termination condition for learning (hereinafter referred to as the "learning termination condition") is satisfied. The learning termination condition is, for example, a condition that a predetermined number of learning rounds have been performed.

The learning device 1 includes two encoders and two decoders, and updates the processing content of each encoder and each decoder through learning until a learning termination condition is satisfied. The neural network including the two encoders and two decoders included in the learning device 1 is a neural network that represents a multimodal estimation model. Therefore, updating the processing content of each encoder and each decoder is an update of the multimodal estimation model.

A neural network is a circuit such as an electronic circuit, an electrical circuit, an optical circuit, or an integrated circuit that represents a mathematical model. Updating a neural network means updating the values of the parameters of the mathematical model that represents the neural network.

Figure 1 is a first explanatory diagram illustrating an overview of the learning device 1 of the embodiment. Figure 2 is a second explanatory diagram illustrating an overview of the learning device 1 of the embodiment. Figure 3 is a third explanatory diagram illustrating an overview of the learning device 1 of the embodiment. Figure 4 is a fourth explanatory diagram illustrating an overview of the learning device 1 of the embodiment.

The learning device 1 includes a first encoder 101, a second encoder 102, a first decoder 103, and a second decoder 104. A neural network including the first encoder 101, the second encoder 102, the first decoder 103, and the second decoder 104 (hereinafter referred to as a "multimodal network") is an example of a neural network that represents a multimodal estimation model. Therefore, updating the multimodal network means updating the multimodal estimation model.

The first encoder 101, the second encoder 102, the first decoder 103, and the second decoder 104 are each a neural network that is updated by learning. In learning of the multimodal network, the first encoder 101, the second encoder 102, the first decoder 103, and the second decoder 104 are each updated. In learning, the first encoder 101, the second encoder 102, the first decoder 103, and the second decoder 104 are each updated so as to reduce the difference between the output of the first encoder 101 and the output of the second encoder 102.

The first encoder 101 acquires the features of the input data. Hereinafter, the features acquired by the first encoder 101 are referred to as the first features. For example, first type data (001) is input to the first encoder. Therefore, the first encoder 101 acquires, for example, the first features of the first type data.

A third type of data may be input to the first encoder 101. The third type of data indicates either the first attribute or the second attribute of a person, living thing, thing, intangible object, or event (hereinafter referred to as a "secondary object") that is different from the main object. The secondary object may or may not have multiple attributes.

When the third type data is input to the first encoder 101, the first encoder 101 acquires a first feature of the third type data. In the example of Fig. 3, the first type data is shown as an example of data input to the first encoder 101, but in Fig. 3, not only the first type data but also the third type data may be input to the first encoder 101.

The second encoder 102 acquires the features of the input data. Hereinafter, the features acquired by the second encoder 102 are referred to as second features. For example, second type data (002) is input to the second encoder. Therefore, the second encoder 102 acquires, for example, the second feature of the second type data.

A third type of data may be input to the second encoder 102. By inputting the third type of data to the second encoder 102, the second encoder 102 acquires a second feature of the third type of data. In the example of FIG. 4, the second type of data is shown as an example of data input to the second encoder 102, but in FIG. 4, not only the second type of data but also the third type of data may be input to the second encoder 102.

The first decoder 103 decodes the input data. Hereinafter, the result of decoding by the first decoder 103 is referred to as the first decoding result. The first decoder 103 decodes the first feature, for example, as shown in FIG. 1. Hereinafter, the result of decoding the first feature by the first decoder 103 is referred to as the first decoding result of the first feature. The first decoder 103 decodes the second feature, for example, as shown in FIG. 2 or FIG. 3. Hereinafter, the result of decoding the second feature by the first decoder 103 is referred to as the first decoding result of the second feature.

The second decoder 104 decodes the input data. Hereinafter, the result of decoding by the second decoder 104 is referred to as the second decoding result. The second decoder 104 decodes the second feature, for example, as shown in FIG. 1. Hereinafter, the result of decoding the second feature by the second decoder 104 is referred to as the second decoding result of the second feature. The second decoder 104 decodes the first feature, for example, as shown in FIG. 2 or FIG. 4. Hereinafter, the result of decoding the first feature by the second decoder 104 is referred to as the second decoding result of the first feature.

The second encoder 102 obtains a second decoded result of the first feature, for example, as shown in Figure 3. The first encoder 101 obtains a first decoded result of the second feature, for example, as shown in Figure 4.

When a pair of first and second type data is input as the learning data, the learning device 1 performs learning using a linked loss function as the loss function. When only one of the first, second, or third type data is input as the learning data, the learning device 1 performs learning using a non-linked loss function as the loss function. The learning data is data used for learning by the learning device 1. The learning is performed so that the difference indicated by the loss function becomes small.

<About the linking loss function>
The linking loss function includes a reconstruction loss, a common loss, and a cross reconstruction loss. The reconstruction loss includes a first sub-reconstruction loss and a second sub-reconstruction loss. The first sub-reconstruction loss indicates a difference between the first type data and a first decoding result of the first type first feature. The first type first feature is a first feature of the first type data. The second sub-reconstruction loss indicates a difference between the second type data and a second decoding result of the second type second feature.

The reconstruction loss is expressed, for example, by the following equation (1):

_d1 indicates first type data. _d2 indicates second type data. _E1 indicates encoding processing by the first encoder 101. _E2 indicates encoding processing by the second encoder 102. _D1 indicates decoding processing by the first decoder 103. _D2 indicates decoding processing by the second decoder 104. The first term on the right side of equation (1) is an example of the first sub-reconstruction loss. The second term on the right side of equation (2) is an example of the second sub-reconstruction loss.

Equation (1) shows the difference between the output of the autoencoder composed of the first encoder 101 and the first decoder 103 and the output of the autoencoder composed of the second encoder 102 and the second decoder 104 and the data input to each autoencoder. Therefore, if learning is performed to reduce the reconstruction loss, each autoencoder is updated so that the difference between the input data and the output is reduced.

Hereinafter, the autoencoder composed of the first encoder 101 and the first decoder 103 is referred to as the first autoencoder. Hereinafter, the autoencoder composed of the second encoder 102 and the second decoder 104 is referred to as the second autoencoder.

The common loss indicates the difference between the first feature and the second feature. The common loss is expressed, for example, by the following equation (2):

Equation (2) shows the difference between the intermediate representation of the first autoencoder (i.e., the first feature) and the intermediate representation of the second autoencoder (i.e., the second feature). Therefore, if learning is performed to reduce the common loss, the first autoencoder and the second autoencoder are updated so that the difference between the first feature and the second feature is reduced.

As a result, by learning to reduce the common loss, the accuracy with which the multimodal estimation model estimates information common to the first type of data and the second type of data, which indicate different attributes, increases. As described above, both the first type of data and the second type of data are information indicating attributes of the main object. Therefore, the higher the accuracy with which the multimodal estimation model estimates information common to the first type of data and the second type of data, the more accurately the multimodal estimation model can estimate the main object.

The cross reconstruction loss includes a first sub-cross reconstruction loss and a second sub-cross reconstruction loss. The first sub-cross reconstruction loss indicates the difference between the first type data and the first decoding result of the second type second feature. The second sub-cross reconstruction loss indicates the difference between the second type data and the second decoding result of the first type first feature.

The cross reconstruction loss is expressed, for example, by the following equation (3):

The first term on the right side of equation (3) is an example of the first sub-crossing reconstruction loss. The second term on the right side of equation (3) is an example of the second sub-crossing reconstruction loss.

Because both the first type data and the second type data are data related to the main subject, the first type first feature and the second type second feature should be substantially identical as learning progresses. If the first type first feature and the second type second feature are substantially identical, even if the decoding of the first type first feature is performed by the second decoder 104 instead of the first decoder 103, the second decoding result of the first type first feature should be substantially identical to the first data.

Furthermore, if the first type first feature and the second type second feature are substantially identical, even if the decoding of the second type second feature is performed by the first decoder 103 instead of the second decoder 104, the first decoding result of the second type second feature should be substantially identical to the second data. Therefore, if the cross reconstruction loss is large, it means that the estimation accuracy of the multimodal estimation model is poor. Therefore, by performing learning to reduce the cross reconstruction loss, the estimation accuracy of the multimodal estimation model is improved.

Thus, the linking loss function includes the i-th sub-reconstruction loss, the common loss, and the i-th sub-cross reconstruction loss, where i is 1 or 2. More specifically, the linking loss function includes the first reconstruction loss (003), the second reconstruction loss (004), the common loss (005), the first cross reconstruction loss (006), and the second cross reconstruction loss (007).

The i-th sub-reconstruction loss indicates the difference between the i-th type data and the i-th decoded result of the i-th type, i-th feature. The i-th type, i-th feature is the i-th feature of the i-th type data. The i-th sub-reconstruction loss indicates the difference between the i-th type data and the i-th decoded result of the j-th type, j-th feature. Note that j is 1 or 2, and j and i are different values. That is, when i is 1, j is 2, and when i is 2, j is 1.

<About unlinked loss functions>
The unlinked loss function includes a crossover self-consistent loss, which includes a first sub-crossover self-consistent loss and a second sub-crossover self-consistent loss.

The first cross self-consistent loss indicates the difference between the data input to the first encoder 101 (hereinafter referred to as "first input data") and the first cross self-consistent data of the first input data. The first cross self-consistent data is the result of the first decoder 103 decoding the second feature of the second cross decoding result. The second cross decoding result is the result of the second decoder 104 decoding the first feature of the first input data.

The second cross self-consistent loss indicates the difference between the data input to the second encoder 102 (hereinafter referred to as "second input data") and the second cross self-consistent data of the second input data. The second cross self-consistent data is the result of the second decoder 104 decoding the first feature of the first cross decoding result. The first cross decoding result is the result of the first decoder 103 decoding the second feature of the second input data.

The first input data may be any type of data indicating the first attribute, and may be first type data or third type data indicating the first attribute of the secondary object. The second input data may be any type of data indicating the second attribute, and may be second type data or third type data indicating the second attribute of the secondary object. In other words, the i-th input data may be any type of data indicating the i-th attribute.

In this way, the cross self-consistency loss includes the i-th sub-cross self-consistency loss. The i-th sub-cross self-consistency loss indicates the difference between the i-th input data, which is the data input to the i-th encoder, and the i-th cross self-consistent data of the i-th input data. The i-th cross self-consistent data is the result of the j-th feature of the j-th cross decoding result being decoded by the i-th decoder. The j-th cross decoding result is the result of the i-th feature of the i-th input data being decoded by the j-th decoder.

Note that the i-th encoder means the first encoder 101 when i=1, and the second encoder 102 when i=2. The j-th encoder means the first encoder 101 when j=1, and the second encoder 102 when j=2. Note that the i-th decoder means the first decoder 103 when i=1, and the second decoder 104 when i=2. The j-th decoder means the first decoder 103 when j=1, and the second decoder 104 when j=2.

The cross self-consistency loss is expressed, for example, by the following equation (4).

The first term on the right-hand side of equation (4) is an example of the first sub-crossing self-consistent loss. The second term on the right-hand side of equation (4) is an example of the second sub-crossing self-consistent loss.

Note that the value of the jth cross self-consistency loss when the ith input data is input to the ith encoder is 0. That is, when the first input data is input to the multimodal estimation model, the value of the second term on the right side of equation (4) is 0, and when the second input data is input to the multimodal estimation model, the value of the first term on the right side of equation (4) is 0.

In the examples of Figures 1 and 2, a pair of first and second types of data was input to the multimodal network. However, there are cases where the user is not necessarily able to prepare both the first and second types of data. Furthermore, there are cases where the user prepares data that indicates either the first attribute or the second attribute, but is not the data of the main subject. Even in such cases, the cross self-consistent loss makes it possible to update the multimodal network so as to improve the accuracy of the estimation.

As shown in equation (4), the cross self-consistency loss indicates the difference between the self-consistency result of one piece of data (hereinafter referred to as "single-input data") input to a multimodal network and short-input data. The first input data and the second input data are each an example of single-input data. Below, we explain the effect of using the cross self-consistency loss using the first input data as an example of single-input data.

<Effects of cross self-consistent losses>
The self-consistent result is a result of the single input data being converted by the first encoder 101, the second encoder 102, the first decoder 103, and the second decoder 104. If the estimation accuracy of the multimodal estimation model is improved, the first feature amount and the second feature amount should be substantially identical, and if the first feature amount and the second feature amount are substantially identical, the second feature amount of the second decoded result of the first feature amount should also be substantially identical. As a result, the result of decoding the second feature amount of the second decoded result obtained from the first feature amount of the first input data by the first decoder 103 should be substantially identical to the first input data. Therefore, by performing learning so as to reduce the cross self-consistency loss, the multimodal estimation model (i.e., the multimodal network) is updated so as to improve the estimation accuracy.

In this way, the unlinked loss function includes a cross-self-consistent loss. More specifically, the unlinked loss function includes a first sub-cross-self-consistent loss (008) and a second sub-cross-self-consistent loss (009). As shown in FIG. 3 or FIG. 4, the unlinked loss function can obtain a value if any one of the first type data, the second type data, or the third type data is obtained. Therefore, by using the unlinked loss function, it is possible to update the multimodal estimation model without using two types of data, the first type data and the second type data.

As described above, the learning device 1 can update the multimodal network using a linked loss function when a pair of first type data and second type data is input. Also, the learning device 1 can update the multimodal network by using a non-linked loss function even when only one of the first type data, second type data, and third type data is input.

Therefore, even if the learning device 1 performs learning using only one of the first type data, the second type data, or the third type data in some of the multiple learnings, it is possible to update the multimodal network to improve the accuracy of estimation. In other parts of the multiple learnings, learning is performed using a pair of the first type data and the second type data, and the learning device 1 updates the multimodal network to improve the accuracy of estimation.

Figure 5 is a diagram showing an example of the hardware configuration of the learning device 1 in an embodiment. The learning device 1 has a control unit 11 including a processor 91 such as a CPU (Central Processing Unit) and a memory 92 connected by a bus, and executes a program. By executing the program, the learning device 1 functions as a device including the control unit 11, input unit 12, communication unit 13, memory unit 14 and output unit 15.

More specifically, in the learning device 1, the processor 91 reads out a program stored in the storage unit 14 and stores the read out program in the memory 92. When the processor 91 executes the program stored in the memory 92, the learning device 1 functions as a device including a control unit 11, an input unit 12, a communication unit 13, a storage unit 14, and an output unit 15.

The control unit 11 controls the operation of various functional units of the learning device 1. The control unit 11, for example, performs learning of a multimodal estimation model.

The input unit 12 includes input devices such as a mouse, a keyboard, a touch panel, etc. The input unit 12 may include an interface that connects these input devices to the learning device 1.

The communication unit 13 includes an interface for connecting the learning device 1 to an external device. The communication unit 13 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits third type data indicating the first attribute. The communication unit 13 acquires the third type data indicating the first attribute by communicating with the device that transmits the third type data indicating the first attribute. The external device is, for example, a device that transmits third type data indicating the second attribute. The communication unit 13 acquires the third type data indicating the second attribute by communicating with the device that transmits the third type data indicating the second attribute.

The external device is, for example, a device that transmits the first type of data. The communication unit 13 acquires the first type of data by communicating with the device that transmits the first type of data. The external device is, for example, a device that transmits the second type of data. The communication unit 13 acquires the second type of data by communicating with the device that transmits the second type of data.

The memory unit 14 is configured using a computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The memory unit 14 stores various information related to the learning device 1. The memory unit 14 stores various information generated as a result of processing executed by the control unit 11, for example.

The output unit 15 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The output unit 15 may include an interface that connects these display devices to the learning device 1.

6 is a diagram showing an example of the functional configuration of the control unit 11 provided in the learning device 1 of the embodiment. The control unit 11 includes a data acquisition unit 111, a data input unit 112, a learning unit 113, a communication control unit 114, a memory control unit 115, and an output control unit 116.

The data acquisition unit 111 acquires data input to the communication unit 13. Specifically, the candidates for data acquired by the data acquisition unit 111 are first type data, second type data, and third type data.

The data input unit 112 outputs the data acquired by the data acquisition unit 111 to an output destination according to the attribute indicated by each data. For example, the data input unit 112 outputs the first type of data acquired by the data acquisition unit 111 to the first encoder 101. For example, the data input unit 112 outputs the second type of data acquired by the data acquisition unit 111 to the second encoder 102. For example, the data input unit 112 outputs the third type of data acquired by the data acquisition unit 111, which indicates a first attribute, to the first encoder 101.

The data input unit 112 outputs, for example, third type data acquired by the data acquisition unit 111, which indicates a second attribute, to the second encoder 102. Each of the first type data, second type data, and third type data has information indicating the attribute indicated by each data. The information indicating the attribute may be any information as long as it indicates the attribute according to a predetermined rule. The information indicating the attribute is, for example, information that expresses a difference in attribute by a difference in data format. The difference in data format is, for example, a difference in data format between image data and text data, etc.

In this way, the attributes indicated by each piece of data are indicated by the information indicating the attributes. Therefore, the data input unit 112 can determine the attributes of each piece of data based on differences in the information indicating the attributes. As a result, the data input unit 112 can output the data acquired by the data acquisition unit 111 to an output destination according to the attributes indicated by each piece of data. Note that the information indicating the attributes of each piece of data may be input by the user to the input unit 12 or the communication unit 13.

The learning unit 113 includes a multimodal network 131 and a network control unit 132. The multimodal network 131 is a multimodal network. Therefore, the multimodal network 131 is a neural network that represents a multimodal estimation model. Therefore, the multimodal network 131 includes a first encoder 101, a second encoder 102, a first decoder 103, and a second decoder 104.

The network control unit 132 updates the multimodal network 131 based on the results obtained by the multimodal network 131. More specifically, the network control unit 132 updates the multimodal network 131 based on the results obtained by the multimodal network 131 and the data input to the multimodal network 131.

For example, when the data acquired by the data acquisition unit 111 is a pair of a first type of data and a second type of data, the network control unit 132 uses a linking loss function and updates the multimodal network 131 based on the results obtained by the multimodal network 131.

For example, when the data acquired by the data acquisition unit 111 is not a pair of the first type of data and the second type of data, the network control unit 132 uses an unlinked loss function and updates the multimodal network 131 based on the results obtained by the multimodal network 131.

The communication control unit 114 controls the operation of the communication unit 13. The memory control unit 115 controls the operation of the memory unit 14. The output control unit 116 controls the operation of the output unit 15.

Figure 7 is a flowchart showing an example of the flow of processing executed by the learning device 1 in an embodiment. The data acquisition unit 111 acquires data (step S101). Next, the data input unit 112 inputs the data acquired by the data acquisition unit 111 to an input destination according to the attributes indicated by each data (step S102). Specifically, the input destination is the first encoder 101 or the second encoder 102.

Next, the multimodal network 131 executes the multimodal estimation model on the data input in step S102 (step S103). Next, the network control unit 132 updates the multimodal estimation model based on the result of the execution of the multimodal estimation model (step S104). More specifically, the network control unit 132 updates the multimodal estimation model using either the linked loss function or the unlinked loss function depending on the data acquired by the data acquisition unit 111 based on the result of the execution of the multimodal estimation model.

Next, the network control unit 132 determines whether the learning end condition is satisfied (step S105). If the learning end condition is satisfied (step S105: YES), the process ends. On the other hand, if the learning end condition is not satisfied (step S105: NO), the process returns to step S101.

The trained multimodal estimation model obtained in this manner is used in a device that estimates an estimation target based on a pair of data on the first attribute of the estimation target and data on the second attribute of the estimation target, such as the estimation device 2 shown in FIG. 8. Note that the trained multimodal estimation model is a multimodal estimation model at the time when the learning end condition is satisfied. Hereinafter, the estimation target of the estimation device 2 is referred to as the target of interest. Hereinafter, the data on the first attribute of the target of interest is referred to as first type target data. Hereinafter, the data on the second attribute of the target of interest is referred to as second type target data. The target of interest may be an object different from the main object.

Figure 8 is a diagram illustrating an example of a hardware configuration of the estimation device 2 of an embodiment. The estimation device 2 acquires a pair of first type target data and second type target data, and estimates a target of interest indicated by the acquired first type target data and second type target data using a trained multimodal estimation model.

The estimation device 2 has a control unit 21 including a processor 93 such as a CPU and a memory 94 connected by a bus, and executes a program. By executing the program, the estimation device 2 functions as a device including the control unit 21, an input unit 22, a communication unit 23, a memory unit 24, and an output unit 25.

More specifically, in the estimation device 2, the processor 93 reads a program stored in the storage unit 24 and stores the read program in the memory 94. When the processor 93 executes the program stored in the memory 94, the estimation device 2 functions as a device including a control unit 21, an input unit 22, a communication unit 23, a storage unit 24, and an output unit 25.

The control unit 21 controls the operation of various functional units of the estimation device 2. The control unit 21 executes, for example, a trained multimodal estimation model.

The input unit 22 includes input devices such as a mouse, a keyboard, a touch panel, etc. The input unit 22 may include an interface that connects these input devices to the estimation device 2.

The communication unit 23 includes an interface for connecting the estimation device 2 to an external device. The communication unit 23 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits a pair of the first type of target data and the second type of target data. The communication unit 23 acquires the pair of the first type of target data and the second type of target data by communicating with the device that transmits the pair of the first type of target data and the second type of target data. The external device is, for example, the learning device 1. The communication unit 23 acquires a trained multimodal estimation model by communicating with the learning device 1.

The memory unit 24 is configured using a computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The memory unit 24 stores various information related to the estimation device 2. The memory unit 24 stores various information generated as a result of processing executed by the control unit 21, for example.

The output unit 25 is configured to include a display device such as a CRT display, a liquid crystal display, an organic EL display, etc. The output unit 25 may be configured to include an interface that connects these display devices to the estimation device 2.

9 is a diagram showing an example of the functional configuration of the control unit 21 provided in the estimation device 2 of the embodiment. The control unit 21 includes a data acquisition unit 211, an estimation unit 212, a communication control unit 213, a memory control unit 214, and an output control unit 215.

The data acquisition unit 211 acquires a set of data consisting of the first type of target data and the second type of target data input to the communication unit 23. The estimation unit 212 executes a trained multimodal estimation model on the data acquired by the data acquisition unit 211. That is, the estimation unit 212 executes a process of estimating an object of interest using a trained multimodal network on the first type of target data and the second type of target data acquired by the data acquisition unit 211. The communication control unit 213 controls the operation of the communication unit 23. The memory control unit 214 controls the operation of the memory unit 24. The output control unit 215 controls the operation of the output unit 25.

Figure 10 is a flowchart showing an example of the flow of processing executed by the estimation device 2 of the embodiment. The data acquisition unit 211 acquires a set of data of a first type of target data and a second type of target data (step S201). Next, the estimation unit 212 executes a trained multimodal estimation model on the data acquired in step S201 (step S202). That is, the estimation unit 212 executes processing executed by the trained multimodal network on the first type of target data and the second type of target data acquired in step S201. The processing executed by the trained multimodal network is, specifically, processing to estimate the target of interest.

The target object is estimated by executing the trained multimodal estimation model on the data acquired in step S201. After step S202, the output control unit 215 controls the operation of the output unit 25 to cause the output unit 25 to output the result of the estimation by the estimation unit 212 (step S203).

The learning device 1 configured in this manner performs learning so as to improve the accuracy of estimation of a mathematical model that estimates information common to multiple types of data. In the learning device 1, the information common to multiple types of data is specifically the main target. The learning device 1 is capable of performing learning using not only a linked loss function, but also a non-linked loss function.

Therefore, the learning device 1 can perform learning not only when linked information such as a pair of first type data and second type data exists, but also when only one of them exists, or when only third type data exists. Therefore, the learning device 1 can obtain a mathematical model that estimates information common to multiple types of data with less effort than a mathematical model obtained by a technology that can perform learning only when linked information exists. In other words, the learning device 1 can reduce the effort required to estimate information common to multiple types of data.

Furthermore, the estimation device 2 configured in this manner estimates the estimation target using the learned mathematical model obtained by the learning device 1. Therefore, the estimation device 2 can reduce the effort required to estimate information common to multiple types of data.

(Modification)
In addition, in the learning using the unlinked loss function by the learning device 1, it is preferable to perform learning using the third type data rather than the first type data or the second type data, because the occurrence of overlearning can be suppressed by performing learning not only on the main object but also on the secondary object.

Note that the unlinked loss function may further include the i-th reconstruction loss in addition to the i-th cross self-consistent loss. As described above, the value of the j-th cross self-consistent loss is 0 when training data is input to the i-th encoder. Also, the value of the j-th cross self-consistent loss is 0 when training data is input to the i-th encoder. When the unlinked loss function further includes the i-th reconstruction loss, the estimation accuracy of the trained multimodal estimation model is higher than when it does not include the i-th reconstruction loss.

Note that the learning device 1 and the estimation device 2 do not necessarily need to be implemented as different devices. The learning device 1 and the estimation device 2 may be implemented, for example, as a single device or system that combines the functions of both devices.

In addition, each functional unit of the learning device 1 and the estimation device 2 may be implemented using multiple information processing devices connected to each other so as to be able to communicate with each other via a network.

Each of the learning device 1 and the estimation device 2 may be implemented using a plurality of information processing devices communicatively connected via a network. All or part of each function of the learning device 1 and the estimation device 2 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.

The above describes in detail an embodiment of the present invention with reference to the drawings, but the specific configuration is not limited to this embodiment and also includes designs that do not deviate from the gist of the present invention.

1...Learning device, 2...Estimation device, 101...First encoder, 102...Second encoder, 103...First decoder, 104...Second decoder, 11...Control unit, 12...Input unit, 13...Communication unit, 14...Memory unit, 15...Output unit, 111...Data acquisition unit, 112...Data input unit, 113...Learning unit, 131...Multimodal network, 132...Network control unit, 114...Communication control unit, 115...Memory control unit, 116...Output control unit, 21...Control unit, 22...Input unit, 23...Communication unit, 24...Memory unit, 25...Output unit, 211...Data acquisition unit, 212...Estimation unit, 213...Communication control unit, 214...Memory control unit, 215...Output control unit, 91, 93...Processor, 92, 94...Memory

Claims (8)

a multimodal network that is a neural network including a first encoder that acquires a feature amount of input data, a second encoder that acquires a feature amount of the input data, a first decoder that decodes the input data, and a second decoder that decodes the input data;
a network control unit that updates the multimodal network using a cross self-consistency loss that indicates the difference between a self-consistency result, which is a result of input data being converted by the first encoder, the second encoder, the first decoder, and the second decoder, and the data, and a common loss that indicates the difference between a result of the first encoder encoding data indicating a first attribute of a main object, which is a person, a living thing, an object, an intangible object, or an event having multiple attributes, and a result of the second encoder encoding data indicating a second attribute of the main object that is different from the first attribute;
A learning device comprising: the network control unit updates the multimodal network using the self-consistency result of the third type of data, which is data indicating either a first attribute or a second attribute of a person, a living thing, an object, an intangible object, or an event different from the main object;
The learning device according to claim 1 . The network control unit updates the multimodal network based on a first secondary reconstruction loss indicating a difference between a result of decoding by the first decoder of a result of encoding by the first encoder of first type data, the first type data being data indicating the first attribute of the main object, and the first type data, and a second secondary reconstruction loss indicating a difference between a result of decoding by the second decoder of a result of encoding by the second encoder of second type data, the second type data being data indicating the second attribute of the main object, and the second type data.
The learning device according to claim 1 or 2. a data acquisition unit that acquires first type target data indicating a first attribute of an estimation target and second type target data indicating a second attribute, which is different from the first attribute, among attributes of the estimation target;
a multimodal network which is a neural network including a first encoder which acquires a feature amount of input data, a second encoder which acquires a feature amount of the input data, a first decoder which decodes the input data, and a second decoder which decodes the input data; and a network control unit which updates the multimodal network using a cross self-consistency loss which indicates a difference between a self-consistency result which is a result of input data being converted by the first encoder, the second encoder, the first decoder, and the second decoder and the input data, and a common loss which indicates a difference between a result of encoding data by the first encoder which indicates a first attribute of a main object which is a person, a living thing, an object, an intangible object, or an event having a plurality of attributes, and a result of encoding data by the second encoder which indicates a second attribute which is an attribute of the main object and which is different from the first attribute;
An estimation device comprising: a network execution step of executing a multimodal network, which is a neural network including a first encoder for acquiring a feature amount of input data, a second encoder for acquiring a feature amount of the input data, a first decoder for decoding the input data, and a second decoder for decoding the input data;
a network control unit step of updating the multimodal network using a cross self-consistency loss indicating the difference between a self-consistency result, which is a result of input data being converted by the first encoder, the second encoder, the first decoder, and the second decoder, and the data, and a common loss indicating the difference between a result of the first encoder encoding data indicating a first attribute of a main object, which is a person, a living thing, an object, an intangible object, or an event having a plurality of attributes, and a result of the second encoder encoding data indicating a second attribute of the main object, which is different from the first attribute;
A learning method that has a data acquisition step of acquiring first type target data indicating a first attribute of an estimation target and second type target data indicating a second attribute, which is different from the first attribute, among attributes of the estimation target;
an estimation step of executing, on the first type target data and the second type target data, a process executed by the trained multimodal network obtained by a learning device including: a multimodal network which is a neural network including a first encoder which acquires a feature amount of input data, a second encoder which acquires a feature amount of the input data, a first decoder which decodes the input data, and a second decoder which decodes the input data; and a network control unit which updates the multimodal network using a cross self-consistency loss which indicates a difference between a self-consistency result which is a result of input data being converted by the first encoder, the second encoder, the first decoder, and the second decoder and the input data, and a common loss which indicates a difference between a result of encoding data by the first encoder which indicates a first attribute of a main object which is a person, a living thing, an object, an intangible object, or an event having a plurality of attributes, and a result of encoding data by the second encoder which indicates a second attribute which is an attribute of the main object and which is different from the first attribute;
The estimation method has the following structure: A program for causing a computer to function as a learning device according to any one of claims 1 to 3. A program for causing a computer to function as the estimation device described in claim 4.

Images