JP7190419B2 - LEARNING DEVICE, ATMOSPHERE RECOGNIZING DEVICE, LEARNING METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to a learning device, an atmosphere recognition device, a learning method, and a program.

Conventionally, techniques for recognizing the atmosphere between people participating in a communication scene have been proposed. For example, Non-Patent Document 1 describes a technique for realizing an emotion recognition model at the group level by extracting feature amounts such as faces, poses, gestures, and line of sight from images and using known deep learning techniques. is disclosed.

Xin Guo, Bin Zhu, Luisa F. Polania, Charles Boncelet, and Kenneth E. Barner. 2018. Group-Level Emotion Recognition using Hybrid Deep Models based on Faces, Scenes, Skeletons and Visual Attentions. ACM International Conference on Multimodal Interaction 2018.

The above technology is a technology that attempts to realize atmosphere recognition by extracting visual features contained in an image, constructing multiple models, and combining the results of recognition using those models. However, it may be difficult to ensure practical recognition performance only by combining low-level visual features (for example, facial features, facial expression features, etc.) extracted from images.

The present invention has been made in view of these points, and an object thereof is to provide a technique for transferring knowledge obtained from information other than images to a learning model of a neural network that receives images as input.

A first aspect of the present invention is a learning device. This apparatus acquires teacher data in which a plurality of image data including a plurality of persons as subjects, a description about the subject of each of the image data, and a label indicating the atmosphere of each of the image data are associated with each other. an acquisition unit; a first vector conversion unit that converts the description corresponding to the image data into first vector data; and a second vector conversion unit that converts the converted image data corresponding to the first vector data into second vector data. a 2-vector conversion unit, and a first neural network configured to output the label corresponding to the first vector data when the first vector data is input to the first neural network composed of a plurality of layers. Inputting the second vector data to a first learning unit that learns parameters of each layer constituting the first neural network and a second neural network that commonly includes a specific layer that is one of the layers constituting the first neural network. learning the parameters of each layer constituting the second neural network so that the data input to the specific layer in the second neural network becomes closer to the data input to the specific layer in the first neural network when the and a second learning unit.

(1) an error between data input to the specific layer in the first neural network and data input to the specific layer in the second neural network; and an error between the label and the output of the first neural network. A parameter of each layer constituting the first neural network may be updated, and the second learning unit updates the input layer, the specific layer, and the layer constituting the second neural network based on the evaluation value. Parameters of layers existing between the input layer and the specific layer may be updated.

The second learning section inputs the second vector data to the second neural network under the condition that the evaluation value satisfies a predetermined convergence condition, and the second learning section corresponds to the second vector data. A parameter of each layer constituting the second neural network may be learned so as to output a label.

The second learning unit may learn only parameters of an output layer and layers existing between the output layer and the specific layer among the layers constituting the second neural network.

A second aspect of the present invention is an atmosphere recognition device. This device includes a storage unit for storing, as a learning model, parameters of each layer constituting the second neural network generated by the learning device described above, an image data acquisition unit for acquiring image data, and a learning stored in the storage unit. a model application unit that applies a model to the image data acquired by the image data acquisition unit to output a label indicating the mood of the subject of the image data.

A third aspect of the present invention is a learning method. In this method, the processor provides teacher data in which a plurality of image data including a plurality of persons as subjects, an explanation about the subject of each of the image data, and a label indicating the atmosphere of each of the subjects of the image data are associated with each other. converting a description corresponding to the image data into first vector data; converting the image data corresponding to the converted first vector data into second vector data; a parameter of each layer constituting said first neural network so as to output said label corresponding to said first vector data when said first vector data is input to said first neural network composed of a plurality of layers; and when inputting the second vector data to a second neural network that commonly includes a specific layer that is a part of the layers constituting the first neural network, the second neural network learning the parameters of each layer of the second neural network so that the vector output by the specific layer in the network approaches the vector output by the specific layer in the first neural network.

A fourth aspect of the present invention is a program. This program provides a computer with a plurality of image data including a plurality of people as subjects, an explanation about each subject of the image data, and a label indicating the atmosphere of each subject of the image data. a function of converting the description corresponding to the image data into first vector data; and a function of converting the image data corresponding to the converted first vector data into second vector data. a parameter of each layer constituting said first neural network so as to output said label corresponding to said first vector data when said first vector data is input to said first neural network composed of a plurality of layers; and a specific layer that is part of the layers constituting the first neural network. and a function of learning the parameters of each layer constituting the second neural network so that the vector output by the specific layer in the network approaches the vector output by the specific layer in the first neural network.

In order to provide this program or update part of the program, a computer-readable recording medium recording this program may be provided, or this program may be transmitted via a communication line.

Any combination of the above-described components, and expressions of the present invention converted into methods, devices, systems, computer programs, data structures, recording media, etc. are also effective as aspects of the present invention.

According to the present invention, it is possible to transfer knowledge obtained from information other than images to a learning model of a neural network that receives images as input.

FIG. 4 is a schematic diagram for explaining an outline of processing according to the embodiment; 1 is a diagram schematically showing a functional configuration of a learning device according to an embodiment; FIG. It is a figure for demonstrating the 2nd error which concerns on embodiment. 1 is a diagram schematically showing a functional configuration of an atmosphere recognition device according to an embodiment; FIG. 6 is a flowchart for explaining the flow of learning processing executed by the learning device according to the embodiment;

<Overview of Embodiment>
A learning device according to an embodiment of the present invention utilizes a neural network composed of a plurality of layers, and when image data including a plurality of people as subjects is input, a label indicating the atmosphere obtained from the image is obtained. Generate a learning model that outputs At this time, the learning device according to the embodiment includes image data including a plurality of people as subjects, a description expressing the atmosphere of the image data in sentences, and data associated with a label indicating the atmosphere of the image data. is used as training data. By using this teacher data, the learning apparatus according to the embodiment can extract knowledge obtained from explanatory sentences as parameters of layers constituting a learning model for outputting an atmosphere from image data. Transfer features. As a result, although the learning device according to the embodiment is a learning model whose final input is only image data, it can generate a learning model in which knowledge obtained from the explanation is reflected in the learning model. can be done.

It should be noted that the “atmosphere of image data including a plurality of people as subjects” is a concept that indicates whether the atmosphere between subjects included in the image data is good or bad. Also, the “explanatory text that expresses the atmosphere of the image data in sentences” is a sentence that describes a subject (including people other than people) included in the image data, and is a sentence that includes at least nouns and verbs. Specific examples of explanations include, "Two men are arguing," "Three women are drinking tea in a cafe," and "A graph is drawn on a white board in a meeting room." ” etc.

FIGS. 1A and 1B are schematic diagrams for explaining an outline of processing according to an embodiment. Specifically, FIG. 1(a) is a diagram for explaining an atmosphere feature learning process for extracting knowledge of a description. FIG. 1(b) is a diagram for explaining a model learning process for learning an atmosphere recognition model using atmosphere feature amounts extracted by the atmosphere feature learning process. Hereinafter, an outline of processing executed by the learning device according to the embodiment will be described with reference to (a) and (b) of FIG.

In FIG. 1A, teacher data T is teacher data used for atmosphere feature learning. This training data T is data in which a plurality of image data I including a plurality of persons as subjects, a description D regarding the subject of each image data I, and a label L indicating the atmosphere of each image data I are associated with each other. be. As an example, the label L is represented by a five-level numerical value from 1 to 5, and the larger the numerical value, the better the atmosphere. In the teacher data illustrated in FIG. 1A, the label L indicating the atmosphere of the image data I is "5", indicating that the atmosphere is good.

The learning device according to the embodiment uses, for example, a known BERT (Bidirectional Encoder Representations from Transformers) to transform the descriptive text D included in the teacher data T into the first vector data V1. Further, the learning device according to the embodiment converts image data I included in teacher data T into second vector data V2 using, for example, a known VGG (Visual Geometry Group) model. Note that these vector conversion techniques are merely examples, and other techniques may be used.

The first neural network N1 is a neural network for outputting the label L included in the teacher data T from which the first vector data V1 is derived when the first vector data V1 is input. The second neural network N2 is a neural network for outputting the label L included in the teacher data T from which the second vector data V2 is derived when the second vector data V2 is input. Although the detailed description is omitted because it is a known technique, the learning device according to the embodiment is an output when the first vector data V1 is input to the first neural network N1, the label L included in the teacher data T, and The parameters that make up the first neural network N1 are learned by back-propagating the error of . The same applies to learning of the second neural network N2.

Both the first neural network N1 and the second neural network N2 are neural networks composed of a plurality of layers including an input layer, an output layer and an intermediate layer. In general, different neural networks have different parameters that make up the neural network. However, as shown in FIG. 1(a), the second neural network N2 according to the embodiment commonly includes a specific layer S, which is part of the layers constituting the first neural network N1. there is In FIG. 1A, hatched rectangles are specific layers S commonly included in the first neural network N1 and the second neural network N2.

In the learning device according to the embodiment, as atmosphere feature learning processing, in parallel with the learning of the first neural network N1, the second vector data V2 is input to the second neural network N2. The learning of the second neural network N2 is also performed so that the vector data input to the specific layer S included in the first neural network N1 approaches the vector data input to the specific layer S included in the first neural network N1.

As described above, the first neural network N1 is trained to output the label L corresponding to the first vector data V1 when the first vector data V1 is input. That is, since the first neural network N1 can be said to be a neural network that learns information or knowledge based on the descriptive text D, the specific layer S included in the first neural network N1 also includes knowledge based on the descriptive text D. it is conceivable that.

Therefore, by approximating the vector data input to the specific layer S included in the second neural network N2 to the vector data input to the specific layer S included in the first neural network N1, the knowledge based on the explanation D can be obtained. It can be transferred to a specific layer S of the second neural network N2. In other words, the learning device according to the embodiment converts the vector data input to the specific layer S when the second vector data V2 derived from the image data I is input to the second neural network N2 into It is possible to approximate the vector data input to the specific layer S when inputting the first vector data V1 to the first neural network N1.

When the atmosphere feature learning process, which is the process of transferring the knowledge based on the explanatory note D to the specific layer S of the second neural network N2, is completed, the learning device according to the embodiment transfers the second neural network obtained by the atmosphere feature learning process. The network N2 is used to perform learning of the atmosphere recognition model. The atmosphere recognition model is a machine learning model trained to output a label L indicating the atmosphere of the image data I when the image data I is input. Specifically, the structure and parameters of the second neural network N2.

The learning device according to the embodiment fixes the parameters of the layers existing from the input layer to the specific layer S in the second neural network N2 obtained by the atmosphere feature learning process, and from the specific layer S also learns only the parameters of the layers on the output layer side. In FIG. 1(b), layers indicated by dashed-dotted rectangles are layers existing between the input layer and the specific layer S in the second neural network N2. A layer indicated by a horizontal line is a layer located closer to the output layer than the specific layer S. Thus, the learning device according to the embodiment can use the vector data input to the specific layer S of the second neural network N2 to learn the parameters of the layer closer to the output layer than the specific layer S. . As a result, the learning device according to the embodiment can transfer knowledge derived from the description D, which is information other than the image, to the learning model of the second neural network N2 that receives the image as input.

<Functional Configuration of Learning Apparatus 1 According to Embodiment>
FIG. 2 is a diagram schematically showing the functional configuration of the learning device 1 according to the embodiment. The learning device 1 includes a storage section 10 and a control section 11 . In FIG. 2, arrows indicate main data flows, and there may be data flows not shown in FIG. In FIG. 2, each functional block does not show the configuration in units of hardware (apparatus), but the configuration in units of functions. Therefore, the functional blocks shown in FIG. 2 may be implemented within a single device, or may be implemented separately within a plurality of devices. Data exchange between functional blocks may be performed via any means such as a data bus, network, or portable storage medium.

The storage unit 10 includes a ROM (Read Only Memory) that stores the BIOS (Basic Input Output System) of the computer that implements the learning device 1, RAM (Random Access Memory) that serves as a work area for the learning device 1, and an OS (Operating System). ), application programs, and a large-capacity storage device such as a HDD (Hard Disk Drive) or an SSD (Solid State Drive) that stores various information referred to when the application program is executed.

The control unit 11 is a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) of the learning device 1, and executes a program stored in the storage unit 10 to obtain the teacher data acquisition unit 110, the first It functions as a vector conversion unit 111 , a second vector conversion unit 112 , a first learning unit 113 , a second learning unit 114 and an evaluation value calculation unit 115 .

Note that FIG. 2 shows an example in which the learning device 1 is composed of a single device. However, the learning device 1 may be realized by computing resources such as a plurality of processors and memories, such as a cloud computing system. In this case, each unit that configures the control unit 11 is realized by executing a program by at least one of a plurality of different processors.

The teacher data acquisition unit 110 reads and acquires the teacher data T from the storage unit 10 . The teacher data T is data in which a plurality of image data I including a plurality of persons as subjects, a description D regarding the subject of each image data, and a label L indicating the atmosphere of each image data I are associated with each other.

The first vector conversion unit 111 converts the descriptive text D corresponding to the image data I included in the teacher data T into first vector data V1. The second vector conversion unit 112 converts image data I corresponding to the first vector data into second vector data V2.

[Atmosphere Feature Learning Processing]
The first learning unit 113 sets parameters of each layer constituting the first neural network N1 so as to output a label L corresponding to the first vector data V1 when the first vector data V1 is input to the first neural network N1. to learn.

The second learning unit 114 learns the parameters of each layer forming the second neural network N2. Here, the second neural network N2 is a network that commonly includes a specific layer S, which is one of the layers constituting the first neural network N1. The second learning unit 114, when inputting the second vector data V2 to the second neural network N2, transfers the vector data input to the specific layer S in the second neural network N2 to the specific layer S in the first neural network N1. The parameters of each layer forming the second neural network N2 are learned so as to approximate the input vector data. As a result, the learning device 1 can transfer the knowledge of the explanatory text D from which the first vector data V1 is derived to the specific layer S included in the second neural network N2.

The learning performed by the first learning unit 113 and the second learning unit 114 will be described more specifically.
The evaluation value calculator 115 calculates the evaluation value V of the evaluation function F including two errors, the first error E1 and the second error E2. Here, the first error is the error between vector data input to a specific layer S in the first neural network N1 and vector data input to a specific layer in the second neural network N2. The second error is the error between the label L included in the teacher data T and the output of the first neural network N1.

FIG. 3 is a diagram for explaining the second error E2 according to the embodiment, and shows the structure of the first specific layer S1, which is the specific layer S of the first neural network N1, and the specific layer S of the second neural network N2. 2 is a diagram schematically showing the structure of a second specific layer S2 which is . In the example shown in FIG. 3, the first specific layer S1 includes N (N is a positive integer) nodes ( _first node a1 to _Nth node aN). The second specific layer S2 also has the same structure as the first specific layer S1, and includes N nodes ( _first node b1 to _Nth node bN).

_{Let a 1 , a 2} , . A vector having these numerical values as elements is vector A=(a ₁ , a ₂ , . . . , a _N ). _Similarly , let _{b 1} , _{b 2 ,} . Let vector B=(b ₁ , b ₂ , . . . , b _N ) be the element vector. At this time, the second error E2 is represented by the following formula (1).

As shown in Equation (1), the smaller the difference between the data input to the first specific layer S1 and the data input to the second specific layer S2, the smaller the second error E2.

As an example, the evaluation value V of the evaluation function F is represented by the following formula (2).
V=E1+λE2 (2)
Here, λ is a parameter for adjusting the influence of the second error E2 on the evaluation function F. When the function of formula (1) is adopted as the evaluation function F, the second learning unit 114 and the evaluation value calculation unit 115 update the parameters of each neural network so that the evaluation value V becomes smaller.

Based on the evaluation value V of the evaluation function F, the first learning unit 113 updates the parameters of each layer forming the first neural network N1. In addition, based on the evaluation value V of the evaluation function F, the second learning unit 114 selects the input layer, the specific layer S, and between the input layer and the specific layer S among the layers that make up the second neural network N2. Update the parameters of an existing layer.

As a result, the second learning unit 114 generates a learning model of the first neural network N1 that outputs the label L corresponding to the description D when the first vector data V1 derived from the description D is input. can be done. In addition, the evaluation value calculation unit 115 has a specific layer S in common with the first neural network N1 generated by the second learning unit 114, and when the image data I corresponding to the explanation D is input, the evaluation value calculation unit 115 It is possible to generate a learning model of the second neural network N2 in which input vector data is vector data similar to vector data input to the specific layer S in the first neural network N1.

[Learning processing of atmosphere recognition model]
The second learning unit 114 inputs the second vector data V2 to the second neural network N2 under the condition that the evaluation value V of the evaluation function F satisfies a predetermined convergence condition. The parameters of each layer constituting the second neural network N2 are learned so as to output the label L corresponding to .

Here, the “predetermined convergence condition” is the “end determination condition of atmosphere feature learning” referred to by the second learning unit 114 in order to determine whether the atmosphere feature learning process has ended. Concrete convergence conditions may be determined through experiments in consideration of the structure of the training data T, performance required for atmosphere recognition, calculation cost, and the like. When the formula shown in Expression (2) is employed as the evaluation value V of the evaluation function F, the predetermined convergence condition can be set such that the evaluation value V is equal to or less than a predetermined threshold Th.

In the atmosphere feature learning process, in order to transfer the knowledge derived from the descriptive text D to the specific layer S of the second neural network N2, parameters from the input layer to the specific layer S of the second neural network N2 can be said to be the process of determining Therefore, the second learning unit 114 learns only the parameters of the output layer and the layers existing between the output layer and the specific layer S among the layers constituting the second neural network N2. As a result, the second learning unit 114 uses the specific layer S to which the knowledge of the description D has been transferred to create an atmosphere recognition model that outputs the label L indicating the atmosphere of the image data I when the image data I is input. can be generated.

[Atmosphere recognition processing]
The learning process for transferring the knowledge derived from the explanatory text D to the atmosphere recognition model, which is the learning model of the neural network, has been described above. Next, atmosphere recognition processing for inputting unknown image data and recognizing the atmosphere of the image data using the atmosphere recognition model obtained by the learning process will be described.

FIG. 4 is a diagram schematically showing the functional configuration of the atmosphere recognition device 2 according to the embodiment. The atmosphere recognition device 2 includes a storage section 20 and a control section 21 . In FIG. 4, arrows indicate main data flows, and data flows not shown in FIG. 4 may exist. In FIG. 4, each functional block does not show the configuration in units of hardware (apparatus), but the configuration in units of functions. Therefore, the functional blocks shown in FIG. 4 may be implemented within a single device, or may be implemented separately within a plurality of devices. Data exchange between functional blocks may be performed via any means such as a data bus, network, or portable storage medium.

The storage unit 20 stores a ROM that stores the BIOS and the like of the computer that implements the learning device 1, a RAM that serves as a work area for the learning device 1, an OS, application programs, and various information that is referred to when the application programs are executed. It is a large-capacity storage device such as an HDD or an SSD.

The control unit 21 is a processor such as a CPU or GPU of the learning device 1, and by executing a program stored in the storage unit 20, an image data acquisition unit 210, a learning model acquisition unit 211, and a model application unit 212. Function.

Note that FIG. 4 shows an example in which the atmosphere recognition device 2 is composed of a single device. However, the atmosphere recognition device 2 may be realized by computational resources such as a plurality of processors and memories, for example, like a cloud computing system. In this case, each unit that configures the control unit 21 is realized by executing a program by at least one of the plurality of different processors.

The storage unit 20 stores, as an atmosphere recognition model, parameters of each layer forming the second neural network N2 generated by the learning device 1 described above. The image data acquisition unit 210 acquires unknown image data to be recognized. The model application unit 212 applies the learning model stored in the storage unit 20 to the unknown image data I acquired by the image data acquisition unit 210 to output a label L indicating the mood of the subject of the image data I. Thus, the atmosphere recognition apparatus 2 according to the embodiment can estimate the atmosphere of the image data I using the atmosphere recognition model to which the knowledge of the description D is transferred.

<Processing Flow of Learning Method Executed by Learning Apparatus 1>
FIG. 5 is a flowchart for explaining the flow of learning processing executed by the learning device 1 according to the embodiment. The processing in this flowchart starts, for example, when the learning device 1 is activated.

The teacher data acquiring unit 110 acquires a teacher data in which a plurality of image data I including a plurality of persons as subjects, a description D about the subject of each image data I, and a label L indicating the atmosphere of each image data I are associated with each other. Data T is acquired (S2).

The first vector conversion unit 111 converts the description D corresponding to the image data I into first vector data V1 (S4). The second vector conversion unit 112 converts the image data I corresponding to the converted first vector data V1 into second vector data V2 (S6).

The first learning unit 113 configures the first neural network N1 to output a label L corresponding to the first vector data V1 when the first vector data V1 is input to the first neural network N1 composed of a plurality of layers. The parameters of each layer forming N1 are learned (S8).

When the second learning unit 114 inputs the second vector data V2 to the second neural network N2 that commonly includes the specific layer S that is one of the layers constituting the first neural network N1, the second learning unit 114 The parameters of each layer forming the second neural network N2 are learned so that the data input to the specific layer S in the second neural network N2 is closer to the data input to the specific layer S in the first neural network N1 (S10). .

The evaluation value calculation unit 115 includes (1) a first error E1 that is an error between the data input to the specific layer S in the first neural network N1 and the data input to the specific layer S in the second neural network N2; (2) Calculate the evaluation value V of the evaluation function F including the label L included in the teacher data T and the second error E2, which is the error between the output of the first neural network N1 (S12).

Until the evaluation value V of the evaluation function F satisfies a predetermined convergence condition (No in S14), the learning device 1 repeats the processing from step S8 to step S12. When the evaluation value V of the evaluation function F satisfies a predetermined convergence condition (Yes in S14), the processing in this flowchart ends.

<Use scene of the present invention>
By introducing the atmosphere recognition device according to the present invention into a dialogue agent such as a chatbot, it becomes possible to automatically activate the dialogue agent with the result of atmosphere recognition as a trigger. As a result, according to the atmosphere recognition device 2 according to the embodiment, for example, it can be expected to increase the situations in which the dialogue agent is used, and by extension, to promote communication between humans.

<Effects of the learning device 1 according to the embodiment>
As described above, according to the learning device 1 according to the embodiment, it is possible to transfer knowledge obtained from information other than images to a learning model of a neural network that receives images as input.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist thereof. be. For example, all or part of the device can be functionally or physically distributed and integrated in arbitrary units. In addition, new embodiments resulting from arbitrary combinations of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment caused by the combination has the effect of the original embodiment.

Reference Signs List 1 Learning device 10 Storage unit 11 Control unit 110 Teacher data acquisition unit 111 First vector conversion unit 112 Second vector conversion unit 113 First Learning unit 114 Second learning unit 115 Evaluation value calculation unit 2 Atmosphere recognition device 20 Storage unit 21 Control unit 210 Image data acquisition unit 211 Learning model acquisition unit 212 ... model application unit

Claims (7)

a teacher data acquisition unit that acquires teacher data in which a plurality of image data including a plurality of persons as subjects, a description about the subject of each of the image data, and a label indicating the atmosphere of each of the image data are associated with each other;
a first vector conversion unit that converts a description corresponding to the image data into first vector data;
a second vector conversion unit that converts image data corresponding to the converted first vector data into second vector data;
parameters of each layer constituting the first neural network so as to output the label corresponding to the first vector data when the first vector data is input to the first neural network composed of a plurality of layers; a first learning unit that learns;
When the second vector data is input to a second neural network that commonly includes a specific layer that is part of the layers constituting the first neural network, the specific layer in the second neural network a second learning unit that learns the parameters of each layer constituting the second neural network so that input data approaches the data input to the specific layer in the first neural network;
A learning device with (1) an error between data input to the specific layer in the first neural network and data input to the specific layer in the second neural network; 1 error from the output of the neural network, and an evaluation value calculation unit that calculates the evaluation value of the evaluation function,
The first learning unit updates parameters of each layer constituting the first neural network based on the evaluation value,
The second learning unit calculates parameters of an input layer, the specific layer, and layers existing between the input layer and the specific layer among the layers constituting the second neural network based on the evaluation value. Update,
A learning device according to claim 1. The second learning section inputs the second vector data to the second neural network under the condition that the evaluation value satisfies a predetermined convergence condition, and the second learning section corresponds to the second vector data. learning the parameters of each layer that configures the second neural network to output a label;
3. A learning device according to claim 2. The second learning unit learns only parameters of an output layer and layers existing between the output layer and the specific layer, among the layers constituting the second neural network.
4. A learning device according to claim 3. A storage unit that stores parameters of each layer constituting the second neural network generated by the learning device according to claim 3 or 4 as a learning model;
an image data acquisition unit that acquires image data;
a model application unit that applies the learning model stored in the storage unit to the image data acquired by the image data acquisition unit, thereby outputting a label indicating the mood of the subject of the image data;
atmosphere recognition device. the processor
a step of acquiring teacher data in which a plurality of image data including a plurality of persons as subjects, a description about the subject of each of the image data, and a label indicating the atmosphere of each subject of the image data are associated;
converting a descriptive text corresponding to the image data into first vector data;
converting the image data corresponding to the converted first vector data into second vector data;
parameters of each layer constituting the first neural network so as to output the label corresponding to the first vector data when the first vector data is input to the first neural network composed of a plurality of layers; a learning step;
When the second vector data is input to a second neural network that commonly includes a specific layer that is part of the layers constituting the first neural network, the specific layer in the second neural network is a step of learning the parameters of each layer constituting the second neural network so that the output vector approaches the vector output by the specific layer in the first neural network;
How to learn to do. to the computer,
a function of acquiring teacher data in which a plurality of image data including a plurality of persons as subjects, an explanation about the subject of each of the image data, and a label indicating the mood of each subject of the image data are associated;
a function of converting a description corresponding to the image data into first vector data;
a function of converting the image data corresponding to the converted first vector data into second vector data;
parameters of each layer constituting the first neural network so as to output the label corresponding to the first vector data when the first vector data is input to the first neural network composed of a plurality of layers; ability to learn and
When the second vector data is input to a second neural network that commonly includes a specific layer that is part of the layers constituting the first neural network, the specific layer in the second neural network is A function of learning the parameters of each layer constituting the second neural network so that the output vector approaches the vector output by the specific layer in the first neural network;
program to realize

Images