JP7803837B2 - Learning device, information processing device, learning method, and program - Google Patents

Description

The present invention relates to a learning device, an information processing device, a learning method, and a program.

Technology has been proposed to recognize the mood of people participating in a communication scene. For example, Non-Patent Document 1 discloses technology that extracts features such as face, pose, gesture, and gaze from an image, and uses known deep learning techniques to realize a group-level emotion recognition model.

Xin Guo, Bin Zhu, Luisa F. Polania, Charles Boncelet, and Kenneth E. Barner. 2018. Group-Level Emotion Recognition using Hybrid Deep Models based on Faces, Scenes, Skeletons and Visual Attentions. ACM International Conference on Multimodal Interaction 2018.

The above technology attempts to achieve mood recognition by extracting visual features contained in an image, building multiple models, and fusing the results of recognition using these models. However, it can be difficult to ensure practical recognition performance using only features extracted from a single modal type of data, such as an image (e.g., facial features, facial expression features, etc.).

The present invention was made in consideration of these points and aims to improve the recognition performance of models.

A learning device according to a first aspect of the present invention includes: a first learning unit that learns parameters of each layer of a first neural network consisting of multiple layers so that, when first vector data associated with a predetermined label is input to the first neural network, the first neural network outputs the label associated with the first vector data; a setting unit that sets the parameters of each layer of the first neural network learned by the first learning unit to each layer of a second neural network consisting of multiple layers; and a second learning unit that learns parameters of each layer of the second neural network so that, when second vector data associated with the same label as the label associated with the first vector data is input to the second neural network, the second neural network outputs the label associated with the second vector data.

The learning device may further include a modal data acquisition unit that acquires modal data associated with the label, a first conversion unit that converts the modal data into the first vector data that expresses modal features of a type different from the type of modal corresponding to the modal data, and a second conversion unit that converts the modal data into the second vector data that expresses modal features corresponding to the modal data.

The first conversion unit may convert the modal data into a plurality of first vector data each having a different modal type, and the learning device may further include a learning control unit that sequentially executes the processing executed by the first learning unit, the processing executed by the setting unit, and the processing executed by the second learning unit for each of the first vector data converted by the first conversion unit, and the learning control unit may, after executing the processing executed by the second learning unit and before executing the processing executed by the first learning unit, cause the setting unit to execute a process of setting the parameters of each layer constituting the second neural network learned by the second learning unit to each layer constituting the first neural network.

The learning control unit may execute each process in sequence for each of the first vector data converted by the first conversion unit, and then execute each process in sequence again for each of the first vector data converted by the first conversion unit.

An information processing device according to a second aspect of the present invention includes a memory unit that stores, as a learning model, parameters of each layer constituting the second neural network generated by the learning device; a homogeneous data acquisition unit that acquires homogeneous data of the same modal type as the modal data; and a model application unit that applies the learning model stored in the memory unit to the homogeneous data acquired by the homogeneous data acquisition unit, thereby outputting a label corresponding to the homogeneous data.

A learning method according to a third aspect of the present invention is executed by a computer and includes the steps of: learning parameters of each layer constituting a first neural network consisting of multiple layers so that, when first vector data associated with a predetermined label is input to the first neural network, the first neural network outputs the label associated with the first vector data; setting the trained parameters of each layer constituting the first neural network to each layer constituting a second neural network consisting of multiple layers; and learning parameters of each layer constituting the second neural network so that, when second vector data associated with the same label as the label associated with the first vector data is input to the second neural network, the second neural network outputs the label associated with the second vector data.

A fourth aspect of the present invention provides a program that causes a computer to function as a first learning unit that learns parameters of each layer of a first neural network consisting of multiple layers so that, when first vector data associated with a predetermined label is input to the first neural network, the first neural network outputs the label associated with the first vector data; a setting unit that sets the parameters of each layer of the first neural network learned by the first learning unit to each layer of a second neural network consisting of multiple layers; and a second learning unit that learns parameters of each layer of the second neural network so that, when second vector data associated with the same label as the label associated with the first vector data is input to the second neural network, the second neural network outputs the label associated with the second vector data.

The present invention has the effect of improving the recognition performance of the model.

FIG. 1 is a diagram illustrating an overview of an information processing system. FIG. 2 is a diagram illustrating a functional configuration of a learning device. FIG. 1 is a diagram schematically illustrating a functional configuration of an information processing device. FIG. 2 is a diagram schematically illustrating control of processing by a learning control unit. 10 is a flowchart illustrating the flow of a learning process executed by the learning device.

[Outline of Information Processing System S]
1 is a diagram illustrating an overview of an information processing system S. The information processing system S is a system used to recognize predetermined modal data. The predetermined modal data is, for example, data indicating a state or behavior (e.g., atmosphere) between multiple people, such as image data including multiple people as subjects, audio data including the voices of multiple people, character string data expressing the state or behavior between multiple people in text, and biometric data of each of multiple people detected by a sensor.

"Recognition of specified modal data" is, for example, recognition of the atmosphere between multiple people. Atmosphere is a concept that indicates, for example, whether the atmosphere between subjects included in image data is good or bad. "Recognition of specified modal data" is not limited to this, and may also be voice recognition, image recognition, emotion recognition, etc. The information processing system S has a learning device 1 and an information processing device 2.

The learning device 1 is a computer that generates a learning model used to recognize specified modal data. The learning model is a machine learning model that uses a neural network consisting of multiple layers to output a label corresponding to vector data when vector data expressing the characteristics of a specified modal is input. Examples of specified modal data include images, audio, character strings, and sensing.

The information processing device 2 is a computer that executes recognition processing to recognize specified modal data. The information processing device 2 executes recognition processing using a learning model that reflects knowledge of multiple different modalities.

When generating a learning model, the learning device 1 uses as training data multiple vector data of different modal types associated with labels (labels common to the multiple vector data). The multiple vector data include, for example, vector data expressing the characteristics of image data containing multiple people as subjects, vector data expressing the characteristics of audio data of the subjects contained in the image data, vector data expressing the characteristics of explanatory text that describes the state or behavior of the subject contained in the image data, and vector data expressing the characteristics of biometric data of the subject contained in the image data. The label indicates, for example, the atmosphere obtained from vector data expressing the characteristics of image data containing multiple people as subjects.

"A description that describes in writing the state or behavior of a subject included in image data" is a sentence that describes a subject (including subjects other than people) included in the image data, and includes at least a noun and a verb. Specific examples of a description include, for example, "Two men are arguing," "Three women are drinking tea in a cafe," and "A graph is drawn on the whiteboard in the conference room."

By using this training data, the learning device 1 can generate a learning model that reflects knowledge of a plurality of vector data each having a different modal type.
Hereinafter, an outline of the processing executed by the learning device 1 will be described with reference to FIG.

In Figure 1, the training data T (first training data T1 and second training data T2) is training data used for learning.

The first training data T1 is data in which first vector data V1 expressing the characteristics of a first modal (e.g., speech) is associated with a label L. The first vector data V1 is, for example, vector data expressing the characteristics of a different type of modal from the modal data to be recognized in the recognition process executed by the information processing device 2.

The second training data T2 is data in which second vector data V2, which expresses the features of a second modal (e.g., an image) of a type different from the first modal, is associated with a label L that is the same as the label L included in the first training data T1. The second vector data V2 is, for example, vector data that expresses the features of a modal of the same type as the modal data to be recognized in the recognition process executed by the information processing device 2.

As an example, the label L is expressed as a five-level numerical value from 1 to 5, with the higher the numerical value, the better the atmosphere. In the training data T shown in Figure 1, the label L indicating the atmosphere is "5," indicating a good atmosphere.

The first neural network N1 is a neural network for outputting the label L contained in the first teacher data T1 from which the first vector data V1 was derived when the first vector data V1 was input. The second neural network N2 is a neural network for outputting the label L contained in the second teacher data T2 from which the second vector data V2 was derived when the second vector data V2 was input. The second neural network N2 is a neural network used in the recognition processing executed by the information processing device 2. The first neural network N1 and the second neural network N2 are neural networks composed of multiple layers including an input layer, an output layer, and an intermediate layer, and are neural networks composed of multiple layers in common.

Since this is a known technique, detailed explanation will be omitted, but the learning device 1 learns the parameters that make up the first neural network N1 by backpropagating the error between the output when first vector data V1 is input to the first neural network N1 and the label L included in the training data T. The same applies to learning the second neural network N2.

In a learning model, recognition performance can be improved when knowledge based on multiple different types of modalities is reflected rather than when knowledge based on a single type of modal is reflected. Therefore, after executing the learning process for the first neural network N1, the learning device 1 sets the parameters of each layer that makes up the first neural network N1 to each layer that makes up the second neural network N2, and then executes the learning process for the second neural network N2.

As described above, the first neural network N1 is trained to output a label L corresponding to the first vector data V1 when the first vector data V1 is input. In other words, the first neural network N1 can be said to be a neural network that learns information or knowledge based on the first modal, and therefore the parameters of each layer that makes up the first neural network N1 are considered to include knowledge based on the first modal.

Therefore, by setting the parameters of each layer constituting the first neural network N1 to each layer constituting the second neural network N2, knowledge based on the first modal can be reflected in the second neural network N2. The second neural network N2 is then trained to output the label L associated with the second vector data V2 when the second vector data V2 is input, thereby further reflecting knowledge based on the second modal in the parameters of each layer constituting the second neural network N2. This allows the learning model of the second neural network N2 to recognize vector data of a single modal type with an unknown label, based on knowledge of multiple vector data each of different modal types. As a result, the learning device 1 can improve the recognition performance of the learning model.

[Configuration of learning device 1]
FIG. 2 is a diagram schematically illustrating the functional configuration of the learning device 1. The learning device 1 includes a storage unit 11 and a control unit 12. In FIG. 2, arrows indicate the main data flows, and there may be other data flows not shown in FIG. 2. In FIG. 2, each functional block indicates a functional unit configuration, not a hardware (device) unit configuration. Therefore, the functional blocks shown in FIG. 2 may be implemented in a single device, or may be implemented separately in multiple devices. Data may be exchanged between functional blocks via any means, such as a data bus, a network, or a portable storage medium.

The storage unit 11 is a large-capacity storage device such as a ROM (Read Only Memory) that stores the BIOS (Basic Input Output System) of the computer that implements the learning device 1, a RAM (Random Access Memory) that serves as the working area for the learning device 1, an HDD (Hard Disk Drive) or SSD (Solid State Drive) that stores the OS (Operating System), application programs, and various information referenced when the application programs are executed.

The control unit 12 is a processor such as the CPU (Central Processing Unit) or GPU (Graphics Processing Unit) of the learning device 1, and functions as a learning control unit 121, a first learning unit 122, a second learning unit 123, a setting unit 124, a modal data acquisition unit 125, a first conversion unit 126, and a second conversion unit 127 by executing programs stored in the memory unit 11. The functions of the learning control unit 121, the first learning unit 122, the second learning unit 123, the setting unit 124, the modal data acquisition unit 125, the first conversion unit 126, and the second conversion unit 127 will be described in detail below.

Note that Figure 2 shows an example in which the learning device 1 is configured as a single device. However, the learning device 1 may also be implemented using multiple processors, memory, and other computing resources, such as a cloud computing system. In this case, each component of the control unit 12 is implemented by at least one of the multiple different processors executing a program.

[Configuration of information processing device 2]
FIG. 3 is a diagram schematically illustrating the functional configuration of an information processing device 2. The information processing device 2 includes a storage unit 21 and a control unit 22. In FIG. 3, arrows indicate main data flows, and data flows not shown in FIG. 3 may also exist. In FIG. 3, each functional block indicates a functional unit configuration, not a hardware (device) unit configuration. Therefore, the functional blocks shown in FIG. 3 may be implemented in a single device, or may be implemented separately in multiple devices. Data may be exchanged between functional blocks via any means, such as a data bus, a network, or a portable storage medium.

The storage unit 21 is a large-capacity storage device such as an HDD or SSD that stores the BIOS and other information of the computer that implements the information processing device 2, RAM that serves as the working area of the information processing device 2, the OS, application programs, and various information referenced when the application programs are executed.

The control unit 22 is a processor such as a CPU or GPU of the information processing device 2, and functions as a homogeneous data acquisition unit 221 and a model application unit 222 by executing programs stored in the storage unit 21. The functions of the homogeneous data acquisition unit 221 and the model application unit 222 will be described in detail below.

Note that Figure 3 shows an example in which the information processing device 2 is configured as a single device. However, the information processing device 2 may also be implemented using multiple processors, memory, and other computing resources, such as in a cloud computing system. In this case, each unit that makes up the control unit 22 is implemented by at least one of the multiple different processors executing a program.

[Learning process]
First, a description will be given of the learning process executed by the learning device 1. The learning control unit 121 controls the process executed by the first learning unit 122, the process executed by the second learning unit 123, and the process executed by the setting unit 124.

The first learning unit 122 learns the first neural network N1 using the first teacher data T1. Specifically, when first vector data V1 associated with a label L in the first teacher data T1 is input to the first neural network N1, the first learning unit 122 learns the parameters of each layer constituting the first neural network N1 so that the label L associated with the first vector data V1 is output.

The second learning unit 123 learns the second neural network N2 using the second teacher data T2. Specifically, when second vector data V2 associated with the same label L as the label L associated with the first vector data V1 in the second teacher data T2 is input to the second neural network N2, the second learning unit 123 learns the parameters of each layer constituting the second neural network N2 so as to output the label L associated with the second vector data V2.

Here, the second learning unit 123 uses the parameters of each layer constituting the first neural network learned by the first learning unit 122 as initial parameters to learn the parameters of each layer constituting the second neural network N2. Specifically, first, the setting unit 124 sets the parameters of each layer constituting the first neural network N1 learned by the first learning unit 122 to each layer constituting the second neural network N2. Then, after the parameters have been set in each layer constituting the second neural network N2, the second learning unit 123 learns the parameters of each layer constituting the second neural network N2 so that when second vector data V2 is input to the second neural network N2, the label L associated with the second vector data V2 is output.

The first vector data V1 and the second vector data V2 are, for example, vector data converted from the same modal data. The modal corresponding to the first vector data V1 and the modal data or the modal corresponding to the second vector data V2 are different types of modal.

Specifically, first, the modal data acquisition unit 125 acquires modal data associated with the label L. Then, the first conversion unit 126 converts the modal data acquired by the modal data acquisition unit 125 into first vector data V1 that expresses the characteristics of a modal type different from the type of modal corresponding to the modal data.

For example, if the modal data is an image, the first conversion unit 126 converts it into first vector data V1 that expresses the characteristics of a different type of modal (e.g., audio, text, or sensing) different from an image. The first conversion unit 126 converts the modal data into first vector data V1 using, for example, a known first converter (e.g., a model such as CLIP (Contrastive Language-Image Pre-Training) or DALL-E).

The modal corresponding to the second vector data V2 and the modal corresponding to the modal data are the same type of modal. Specifically, the second conversion unit 127 converts the modal data acquired by the modal data acquisition unit 125 into second vector data V2 that expresses the characteristics of the modal corresponding to the modal data. For example, if the modal of the modal data is an image, the second conversion unit 127 converts it into second vector data V2 that expresses the characteristics of the image. The second conversion unit 127 converts the modal data into second vector data V2 using, for example, a known second converter (for example, a model such as VGG (Visual Geometry Group)). In this way, the learning device 1 can create multiple vector data each with a different modal type.

In the above, an example has been described in which the learning device 1 converts one acquired piece of modal data into multiple vector data each having a different modal type, but this is not limiting. For example, the learning device 1 may convert multiple pieces of modal data each having a different modal type into multiple corresponding vector data.

Specifically, the modal data acquisition unit 125 first acquires a plurality of vector data, each of which is a plurality of modal data associated with a common label L and has a different modal type. The first conversion unit 126 then converts the plurality of modal data acquired by the modal data acquisition unit 125 into first vector data V1, which expresses modal features corresponding to other types of modal data that are different in modal type from the modal data to be recognized in the recognition process executed by the information processing device 2. The second conversion unit 127 also converts the plurality of modal data acquired by the modal data acquisition unit 125 into second vector data V2, which expresses modal features corresponding to modal data of the same modal type as the modal data to be recognized in the recognition process executed by the information processing device 2.

For example, if the modal data acquisition unit 125 acquires image data, audio data, and character string data as multiple modal data, and the modal data to be recognized in the recognition process executed by the information processing device 2 is an image, the first conversion unit 126 converts the audio data into first vector data V1 that expresses the characteristics of the audio, and converts the character string data into first vector data V1 that expresses the characteristics of the character string. Furthermore, the second conversion unit 127 converts the image data into second vector data V2 that expresses the characteristics of the image.

The learning device 1 may convert any of a plurality of modal data, each having a different modal type, into first vector data V1 that expresses the characteristics of the other type of modal. For example, if the modal data acquisition unit 125 acquires image data and audio data as a plurality of modal data, the first conversion unit 126 first selects modal data from the image data and audio data to use for conversion into the first vector data V1. The first conversion unit 126 then converts the selected modal data (e.g., image data) into first vector data V1 that expresses the characteristics of a modal type different from an image (e.g., a character string). For example, the first conversion unit 126 selects, from the plurality of modal data, modal data of the same type as the modal type of the modal data to be recognized in the recognition process executed by the information processing device 2, and converts the selected modal data into first vector data V1 that expresses the characteristics of a modal type different from the selected modal data.

After the first conversion unit 126 and the second conversion unit 127 convert the first vector data V1 and the second vector data V2, respectively, the learning control unit 121 sequentially causes the first learning unit 122 to execute processing based on the first vector data V1 (first teacher data T1), the setting unit 124 to execute processing, and the second learning unit 123 to execute processing based on the second vector data V2 (second teacher data T2). The learning control unit 121 may also cause the first learning unit 122 to execute processing, the setting unit 124 to execute processing, and the second learning unit 123 to execute processing using multiple first vector data V1 each having a different modal type.

Specifically, the first conversion unit 126 first converts the modal data into multiple first vector data V1, each of which has a different modal type. For example, the first conversion unit 126 converts the modal data into first vector data V1 using a different first converter for each modal type.

Then, for each piece of first vector data V1 converted by the first conversion unit 126, the learning control unit 121 sequentially causes the first learning unit 122 to execute processing based on the first vector data V1, the setting unit 124 to execute processing, and the second learning unit 123 to execute processing based on the second vector data V2. After causing the second learning unit 123 to execute processing, and before causing the first learning unit 122 to execute processing, the learning control unit 121 causes the setting unit 124 to execute processing to set the parameters of each layer constituting the second neural network N2 learned by the second learning unit 123 to each layer constituting the first neural network N1.

Figure 4 is a diagram schematically illustrating the control of processing by the learning control unit 121. Figure 4 illustrates an example in which the first conversion unit 126 converts first vector data V1a and first vector data V1b. The first teacher data T1a is data in which the first vector data V1a is associated with a label L. The first teacher data T1b is data in which the first vector data V1b is associated with a label L (the same label as the label L included in the first teacher data T1a and the label L included in the second teacher data T2). For example, the first vector data V1a is vector data representing audio features, the first vector data V1b is vector data representing character string features, and the second vector data V2 is vector data representing image features.

In this case, the learning control unit 121 first sequentially executes the processing based on the first teacher data T1a executed by the first learning unit 122 shown in FIG. 4(a), the processing executed by the setting unit 124, and the processing based on the second teacher data T2 executed by the second learning unit 123 shown in FIG. 4(b). The learning control unit 121 then executes the processing of setting the parameters of each layer constituting the second neural network N2, which is obtained after the processing based on the second teacher data T2 executed by the second learning unit 123, to each layer constituting the first neural network N1b.

The learning control unit 121 then sequentially executes the processing based on the first teacher data T1b executed by the first learning unit 122 shown in FIG. 4(c), the processing executed by the setting unit 124, and the processing based on the second teacher data T2 executed by the second learning unit 123 shown in FIG. 4(d). In this way, the learning device 1 can reflect knowledge based on multiple first modals in the parameters of each layer constituting the second neural network N2.

The learning control unit 121 may repeatedly execute each process. Specifically, the learning control unit 121 executes each process in sequence for each piece of first vector data converted by the first conversion unit 126, and then executes each process in sequence again for each piece of first vector data converted by the first conversion unit 126.

In the example shown in FIG. 4, the learning control unit 121 executes each of the processes shown in FIGS. 4(a) to 4(d), and then executes each of the processes shown in FIGS. 4(a) to 4(d) again. At this time, the learning control unit 121 causes the setting unit 124 to execute a process of setting the parameters of each layer constituting the second neural network N2 shown in FIG. 4(d) after executing each of the processes shown in FIGS. 4(a) to 4(d) to each layer constituting the first neural network N1a shown in FIG. 4(a), and then executes each of the processes shown in FIG. 4 again. In this way, the learning device 1 can improve the robustness of the recognition performance of the second neural network N2 by repeatedly executing each process.

[Recognition processing]
Next, a description will be given of the recognition processing executed by the information processing device 2. In the information processing device 2, the storage unit 21 stores, as a learning model, parameters of each layer constituting the second neural network N2 generated by the above-described learning device 1.

The homogeneous data acquisition unit 221 acquires homogeneous data. Homogeneous data is data of the same type in modal as the modal data acquired by the modal data acquisition unit 125 of the learning device 1.

The model application unit 222 applies the learning model stored in the memory unit 21 to the homogeneous data acquired by the homogeneous data acquisition unit 221, thereby outputting a label L corresponding to the homogeneous data. Specifically, the model application unit 222 first converts the homogeneous data acquired by the homogeneous data acquisition unit 221 into vector data using the second converter. The model application unit 222 then applies the learning model stored in the memory unit 21 to the converted vector data, thereby outputting a label L corresponding to the homogeneous data. This allows the information processing device 2 to recognize homogeneous data using a learning model that reflects knowledge from multiple different modals.

In the above, an example has been described in which the model application unit 222 applies the learning model to homogeneous data to output a label L corresponding to the homogeneous data, but this is not limiting. For example, the model application unit 222 may apply the learning model to heterogeneous data, the modal type of which is different from the modal data acquired by the modal data acquisition unit 125 of the learning device 1, to output a label L corresponding to the heterogeneous data.

For example, first, the model application unit 222 uses the first converter to convert the other type of data into vector data of the same modal type as the modal data acquired by the modal data acquisition unit 125 of the learning device 1. Then, the model application unit 222 applies the learning model stored in the memory unit 21 to the converted vector data, thereby outputting a label L corresponding to the other type of data.

Furthermore, for example, the model application unit 222 may use a second converter to convert the different type of data into vector data of the same type in modality, and apply the learning model to the converted vector data, thereby outputting a label L corresponding to the different type of data. Further, for example, when the same type of data acquisition unit 221 acquires unknown data of an unknown modality, the model application unit 222 first determines the modal type of the unknown data. The model application unit 222 may determine the modal type of the unknown data, for example, using known technology. Then, the model application unit 222 may convert the unknown data into vector data using a converter (first converter or second converter) corresponding to the determined modal type, and apply the learning model to the converted vector data, thereby outputting a label L corresponding to the unknown data.

[Learning process flow]
Next, the flow of the learning process will be described. Fig. 5 is a flowchart for explaining the flow of the learning process executed by the learning device 1. The process in this flowchart starts, for example, when the learning device 1 is started.

The modal data acquisition unit 125 acquires modal data associated with a label L (S1). The learning device 1 converts the modal data into multiple vector data each having a different modal type (S2). Specifically, the first conversion unit 126 converts the modal data acquired by the modal data acquisition unit 125 into one or more first vector data V1 that expresses the characteristics of a modal type different from the modal type corresponding to the modal data. Furthermore, the second conversion unit 127 converts the modal data acquired by the modal data acquisition unit 125 into second vector data V2 that expresses the characteristics of the modal type corresponding to the modal data.

The learning control unit 121 controls the execution of processes S3 to S7. The first learning unit 122 learns the parameters of each layer constituting the first neural network N1 so that when first vector data V1 associated with a label L in the first teacher data T1 is input to the first neural network N1, the first neural network N1 outputs the label L associated with the first vector data V1 (S3).

The setting unit 124 sets the parameters of each layer constituting the first neural network N1 learned by the first learning unit 122 to each layer constituting the second neural network N2 (S4). The second learning unit 123 learns the parameters of each layer constituting the second neural network N2 so that when second vector data V2 associated with the same label L as the label L associated with the first vector data V1 in the second teacher data T2 is input to the second neural network N2, the second neural network N2 outputs the label L associated with the second vector data V2 (S5).

The learning control unit 121 determines whether learning has been completed using all vector data (S6). If the learning control unit 121 determines that learning has not been completed using all vector data (NO in S6), i.e., if there is first vector data V1 that has not been used in learning, it causes the setting unit 124 to set the parameters of each layer that constitutes the second neural network N2 learned by the second learning unit 123 to each layer that constitutes the first neural network N1 (S7), and the process returns to S3. On the other hand, if the learning control unit 121 determines that learning has been completed using all vector data (YES in S6), i.e., if there is no first vector data V1 that has not been used in learning, it terminates the process.

[Effects of the learning device 1]
As described above, the learning device 1 can reflect knowledge of multiple vector data sets with different modal types in a learning model of a neural network that receives vector data as input. This allows the learning model of the second neural network N2 to recognize vector data with an unknown label L based on knowledge of multiple vector data sets with different modal types. As a result, the learning device 1 can improve the recognition performance of the learning model.

Furthermore, this invention will contribute to Goal 9 of the United Nations' Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and resilience."

The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and alterations are possible within the spirit of the invention. For example, all or part of the device can be configured by functionally or physically distributing or integrating any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of new embodiments resulting from the combination will also have the effects of the original embodiments.

REFERENCE SIGNS LIST 1 Learning device 11 Memory unit 12 Control unit 121 Learning control unit 122 First learning unit 123 Second learning unit 124 Setting unit 125 Modal data acquisition unit 126 First conversion unit 127 Second conversion unit 2 Information processing device 21 Memory unit 22 Control unit 221 Homogeneous data acquisition unit 222 Model application unit S Information processing system

Claims (7)

a first learning unit that learns parameters of each layer constituting a first neural network configured with a plurality of layers so that, when first vector data associated with a predetermined label is input to the first neural network, the first neural network outputs the label associated with the first vector data;
a setting unit that sets the parameters of each layer that constitutes the first neural network learned by the first learning unit to each layer that constitutes a second neural network that is composed of a plurality of layers;
a second learning unit that learns parameters of each layer constituting the second neural network so that, when second vector data associated with the same label as the label associated with the first vector data is input to the second neural network, the second neural network outputs the label associated with the second vector data;
A learning device having the above configuration. a modal data acquisition unit that acquires modal data associated with the label;
a first conversion unit that converts the modal data into the first vector data that expresses a feature of a modal type different from the modal type corresponding to the modal data;
a second conversion unit that converts the modal data into the second vector data that expresses modal characteristics corresponding to the modal data;
further comprising
The learning device according to claim 1 . the first conversion unit converts the modal data into a plurality of first vector data each having a different modal type;
a learning control unit that sequentially executes the process executed by the first learning unit, the process executed by the setting unit, and the process executed by the second learning unit for each of the first vector data converted by the first conversion unit;
the learning control unit, after causing the second learning unit to execute the process and before causing the first learning unit to execute the process, causes the setting unit to execute a process of setting parameters of each layer constituting the second neural network learned by the second learning unit to each layer constituting the first neural network;
The learning device according to claim 2 . the learning control unit sequentially executes each process for each of the first vector data converted by the first conversion unit, and then sequentially executes each process again for each of the first vector data converted by the first conversion unit;
The learning device according to claim 3 . a storage unit that stores, as a learning model, parameters of each layer that constitutes the second neural network generated by the learning device according to any one of claims 2 to 4;
a homogeneous data acquisition unit that acquires homogeneous data of the same type as the modal data;
a model application unit that applies a learning model stored in the storage unit to the homogeneous data acquired by the homogeneous data acquisition unit, thereby outputting a label corresponding to the homogeneous data;
An information processing device having the above. The computer executes
a step of learning parameters of each layer constituting a first neural network so that, when first vector data associated with a predetermined label is input to the first neural network constituted by a plurality of layers, the first neural network outputs the label associated with the first vector data;
a step of setting the parameters of each layer constituting the trained first neural network to each layer constituting a second neural network composed of a plurality of layers;
learning parameters of each layer constituting the second neural network so that when second vector data associated with the same label as the label associated with the first vector data is input to the second neural network, the second neural network outputs the label associated with the second vector data;
A learning method that has Computer,
a first learning unit that learns parameters of each layer constituting a first neural network configured with a plurality of layers so that, when first vector data associated with a predetermined label is input to the first neural network, the first neural network outputs the label associated with the first vector data;
a setting unit that sets the parameters of each layer that constitutes the first neural network learned by the first learning unit to each layer that constitutes a second neural network that is composed of a plurality of layers; and a second learning unit that learns the parameters of each layer that constitutes the second neural network so that, when second vector data that is associated with the same label as the label associated with the first vector data is input to the second neural network, the second neural network outputs the label associated with the second vector data.
A program to function as a