JP6692271B2 - Multitask processing device, multitask model learning device, and program - Google Patents

Description

  The present invention relates to a multitask processing device, a multitask model learning device, and a program.

  As a system for evaluating and feeding back input data, a feedback system based on a rule base and a feedback system based on a comparison base are known. In a rule-based feedback system, feedback is performed by setting evaluation criteria heuristically. For example, an evaluation standard such as “○○ should be within ΔΔ” is set. Further, in the feedback system based on the comparison base, the evaluation is performed by the difference based on the example by the professional. A feedback system based on a learning base is also known (for example, Non-Patent Document 1).

Pirsiavash, “Assessing the Quality of Actions”, ECCV2014

  However, the rule-based feedback system requires professional knowledge, and the comparison-based feedback system has a problem that it is difficult to define a difference due to individual differences.

  Further, when the feedback system based on the learning base described in Non-Patent Document 1 above is applied to a feedback system that provides feedback for improving facial expressions to an input image in which a face area is reflected, as shown in FIG. From this, a key point representing the position of the eyes or nose is detected, and a score corresponding to the key point is calculated. Then, a value regarding feedback is calculated according to the relationship between the detection of the key point and the score.

  However, since it is necessary to first extract the key points, the score calculation depends on the key point detection accuracy. Further, since the information that can be used to calculate the value related to the feedback is only the key points, there is a problem that the information of the input image cannot be reflected.

  The present invention has been made to solve the above problems, and an object of the present invention is to provide a multitask processing device and a program capable of acquiring the results of a plurality of tasks for data from one model. To do.

  Another object of the present invention is to provide a multitasking model learning device and a program capable of obtaining a multitasking model for acquiring results of a plurality of tasks.

  In order to achieve the above object, the multitask processing device according to the first invention includes an evaluation task for evaluating the input data and an information presentation task for extracting information from the data. It is configured to include a task processing unit that obtains outputs of the evaluation task and the information presentation task by using a multi-task model for processing a plurality of tasks.

  The multi-task model shares a feature space between the evaluation task and the information presentation task, performs the evaluation task of evaluating data based on the feature amount of the feature space, and determines the characteristics of the feature space. It may be a multi-task DNN (Deep Neural Network) that performs the information presentation task of extracting information from data based on the amount.

  A multi-task processing device according to a first aspect of the present invention is for presenting the information so as to approach a target value when a target value is given to a plurality of or one evaluation task among the plurality of tasks. The output of the task is calculated, the calculation result of the output of the information presenting task by the task processing unit, and the calculation result of the output of the information presenting task, which is calculated so as to approach the target value, A feedback generation unit that presents information regarding the output of the information presenting task may be further included.

  The data may be image information.

  The multi-task model is a DNN (Deep Neural Network) having a plurality of layers, and the feedback generation unit includes a plurality of layers when the input data is applied to the multi-task model by the task processing unit. By modifying the output of any one of the layers, when it is evaluated as the target value by the evaluation task of the multitask model, it is extracted by the information presentation task of the multitask model. Information can be calculated.

  A multitasking model learning device according to a second aspect of the invention is a multitasking DNN (Deep Neural Network) that shares a feature space with a plurality of tasks based on learning data consisting of data and results of a plurality of tasks on the data. ) And based on the feature amount of the feature space, a learning unit that learns a multi-task DNN that performs the plurality of tasks is included.

  Further, the data may be image information.

  A program according to the present invention is a program for causing a computer to function as each unit included in the above-mentioned multitask processing device and the above-mentioned multitask model learning device.

  According to the multitask processing device and the program of the present invention, for processing a plurality of tasks including an evaluation task for evaluating data and an information presentation task for extracting information from the data with respect to the input data. By obtaining outputs for the evaluation task and the information presentation task by using the multitask model of, the effect of being able to obtain the results of a plurality of tasks for data from one model is obtained.

  According to the multitask model learning device and program of the present invention, a multitask model in which a feature space is shared by a plurality of tasks based on learning data including data and results of a plurality of tasks for the data. It is possible to obtain a multitask model for acquiring the results of multiple tasks by learning a multitask model that performs multiple tasks based on the feature amount of the feature space. can get.

It is an explanatory view for explaining a system for feeding back a specific instruction for generating a smile to the image information of the face area. It is an explanatory view for explaining an example of a system which presents feedback information for every image information on a different face region. It is a conceptual diagram of the feedback system in this Embodiment. It is an explanatory view for explaining an advantage of a learning based feedback system compared with a rule base and a comparison base. It is an explanatory view for explaining how to give learning data. It is explanatory drawing for demonstrating the example of the method of information presentation. It is explanatory drawing for demonstrating the outline | summary of the process of this Embodiment. It is a block diagram which shows the structure of the multitask processing apparatus which concerns on embodiment of this invention. It is explanatory drawing about the part of three formulation in a multitasking model. It is explanatory drawing of simultaneous learning of facial expression recognition and keypoint extraction. It is an explanatory view of the outline of processing of multi-task CNN. It is an explanatory view of an outline of learning processing of multitask CNN. FIG. 7 is an explanatory diagram illustrating a process performed by a feedback generation unit 30. FIG. 7 is an explanatory diagram illustrating a process performed by a feedback generation unit 30. FIG. 7 is an explanatory diagram illustrating a process performed by a feedback generation unit 30. It is explanatory drawing for demonstrating the outline | summary of the process of this Embodiment. It is a flow chart which shows a learning processing routine in a multitask processing device concerning an embodiment of the invention. It is a flowchart which shows the multitask processing routine in the multitask processing apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the pseudo chart of a multitasking processing routine. It is a flow chart which shows a feedback generation processing routine in a multitask processing device concerning an embodiment of the invention. It is an explanatory view for explaining the outline of an experiment. It is an explanatory view for explaining the outline of an experiment. It is explanatory drawing for demonstrating the example of an experimental result. It is explanatory drawing for demonstrating the example at the time of applying to the feedback system which gives feedback of facial expression improvement.

<Outline of Embodiment of the Present Invention>
In this embodiment, the aim is to realize a feedback system. For example, in the case of a feedback system regarding facial expression improvement, as shown in FIG. 1, a specific instruction for generating a smile is fed back to the user with respect to image information of a face area, which is an example of input data. By implementing the feedback system of the image information of the face area, it is expected to be applied to the applications as shown in the following (1) to (4), for example.

(1) What expression should an actor have in a certain situation?
(2) What kind of facial expression should you have during the interview?
(3) What is the correct mouth movement when pronouncing English?
(4) Support for people with disabilities in facial expression operations

  In this embodiment, in particular, an attempt is made to realize a feedback system that adaptively generates feedback for various input data. For example, as shown in FIG. 2, feedback information corresponding to image information is presented for each image information of different face areas.

  FIG. 3 shows a conceptual diagram of the feedback system in the present embodiment. As shown in FIG. 3, when input data such as an image or a moving image is input in [1.Input], based on the evaluation standard in [2-1.Evaluation] and the target state [Target], [ 2-2. Check of Achievement] confirms the degree of achievement. When the target state is not achieved, feedback is generated in [3-2.Feedback Proposal] according to the improvement plan in [3-1.Formulation of improvement Plan], and the input data is corrected. ..

  Here, as shown in FIG. 3, there are problems in how to obtain the evaluation standard and how to obtain a specific instruction.

  On the other hand, in the present embodiment, a feedback system based on a learning base is used to learn the evaluation standard from a large amount of data. This eliminates the need for professional knowledge and allows the evaluation criteria to be well extracted from the data. Further, as shown in FIG. 4, the feedback system based on the learning base is more effective than the rule base and the comparison base on the region that has not been discovered yet.

  Further, in the related art, Recognition, Detection, and Localization are targeted. For example, in facial expression recognition, which is an example of Recognition, emotions corresponding to facial expressions such as Happy, Sad, Angry ... Are recognized. Face detection is performed as an example of Detection. In addition, as an example of Localization, detection of key points indicating the positions of eyes and nose in the face area is performed. As described above, in the related art, the purpose is to recognize the current situation (What people do).

  On the other hand, in the present embodiment, feedback generation is performed and, for example, instructions necessary for improving facial expressions are presented. In the present embodiment, the purpose is to learn how to change (How people can improve), and since the target task is significantly different from the conventional technique, a new learning method is required.

  Further, when constructing a feedback system, two methods can be considered as methods of giving learning data. The first method is to collect all patterns of learning data from one user. The second method is to collect one pattern of learning data from one user. For example, as shown in FIG. 5, when learning data of all patterns is collected from one user, there is a merit that each person has a correct answer example, but there is a demerit that each person has a heavy burden. Further, in the case of collecting one pattern of learning data from one user, there is an advantage that the burden on each person is small, but there is a disadvantage that there is no correct answer example for each person. It is also possible for a person to annotate the correct answer after collecting the data, but it requires specialized knowledge and is not scalable. Therefore, in the present embodiment, considering the scalability of data collection, it is selected to collect one pattern of learning data from one user.

  As a method of presenting information, the format shown in FIG. 6 can be considered. As shown in FIG. 6, consider a case where image information of a face of a specific user is input and feedback is provided to make the user smile. In this case, whether the image information obtained by transforming the image information of the face of the same user is presented as feedback, or whether the image information of the face of the same user is added as an index and presented as feedback, the image information of the face of different users is displayed. It is necessary to select whether to present modified image information as feedback or to present it as feedback by adding an index to the image information of the faces of different users.

  In the present embodiment, a method of adding an index to the image information of the face of the same user and presenting it as feedback is selected in consideration of the intelligibility of the presentation information.

  As shown in FIG. 7, the present embodiment is an example of facial expression recognition, which is an example of an evaluation task for evaluating image information when a user's face image information is input, and an example of key points from the image information. An information presentation task of extracting eyes, nose, etc. is performed. Then, "smile" is set as a goal, and feedback planning is generated by utilizing the strategy to make the user smile, and the generated feedback planning is displayed as an arrow in the image information.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Configuration of Multitask Processing Device According to Embodiment of Present Invention>

  The configuration of the multitask processing device according to the embodiment of the present invention will be described. As shown in FIG. 8, a multi-task processing device 100 according to the embodiment of the present invention includes a CPU, a RAM, and a ROM that stores programs and various data for executing each processing routine described later. It can be composed of a computer. This multitask processing device 100 is functionally provided with an input unit 10, a calculation unit 20, and an output unit 40, as shown in FIG.

  The input unit 10 receives learning data. The input unit 10 also receives input data. The learning data includes data and results of a plurality of tasks for the data. In the present embodiment, a case where the learning data and the input data are image information of a human face area will be described as an example.

  The arithmetic unit 20 includes a learning data storage unit 22, a learning unit 24, a model storage unit 26, a task processing unit 28, and a feedback generation unit 30.

  The learning data storage unit 22 stores the learning data received by the input unit 10. The learning data is configured to include image information, which is an example of data, and evaluation of image information and key points of the image information, which are examples of results of a plurality of tasks on the data. As key points of the image information, information about the positions of both eyes, nose, and both ends of the mouth is added to the image information. Further, as the evaluation of the image information, information indicating a facial expression (eg, anger, happiness, normal) is added to the image information.

  The learning unit 24 is a multi-task model in which the feature space is shared by a plurality of tasks based on the learning data stored in the learning data storage unit 22, and a plurality of tasks based on the feature amount of the feature space. Learn a multitasking model. In the present embodiment, a multitasking DNN (Deep Neural Network), which is an example of a multitasking model in which a feature space is shared by an evaluation task that evaluates image information and an information presentation task that extracts key points from the image information. To learn. By using a multitasking model, it is possible to process multiple tasks and obtain outputs for multiple tasks. In the present embodiment, a case of processing an evaluation task and an information presentation task, which are examples of a plurality of tasks, will be described as an example. Therefore, the multitasking DNN performs the evaluation task based on the feature amount of the feature space, and performs the information presentation task based on the feature amount of the feature space.

  The multitasking model of the present embodiment is a model for performing an evaluation task for evaluating image information and an information presentation task for extracting a key point which is an example of information from image information. In the present embodiment, a case where the multitask model is a CNN (Convolutional Neural Network), which is an example of a DNN (Deep Neural Network) having a plurality of layers, will be described as an example. Therefore, the multitasking model used in this embodiment is a multitasking CNN having a plurality of layers.

  FIG. 9 shows an explanatory diagram of three formulation parts in the multitasking model. As shown in FIG. 9, in the present embodiment, it is necessary to formulate facial expression recognition, which is an example of an evaluation task, keypoint extraction, which is an information presentation task, and feedback planning.

In the facial expression recognition, when the image information x ₀ is input, it is formulated as an identification problem for identifying the facial expression represented by the image information x ₀ . For example, y ^c calculated according to the following equation (1) is identified as the facial expression represented by the image information x ₀ . Note that f represents a multitasking CNN, and w represents a parameter of the multitasking CNN. The subscript c represents facial expression recognition, which is an example of an evaluation task, and the subscript r represents keypoint extraction, which is an information presentation task.

(1)

Further, the key point extraction, if the image information x ₀ is input, formulated as a regression problem keypoint extraction of image information x _0. For example, y ^r calculated according to the following equation (2) is extracted as a key point of the image information x ₀ .

(2)

Further, in the feedback planning, when the image information x ₀ and the target value y ^{c *} which is the result of the target facial expression recognition are input, the feedback planning is performed according to the following formula (3), for example.

(3)

In this problem setting, since the correct answer y ^{r * of} feedback is unknown, explicit learning is impossible, but in the present embodiment, a strategy is implicitly acquired by simultaneous learning.

Specifically, as shown in FIG. 10, the relationship (strategy) between the facial expression y ^c and the key point y ^r is implicitly acquired by simultaneous learning of facial expression recognition and keypoint extraction. Then, utilizing the acquired strategies, feedback planning is performed.

As specific processing of the multitasking CNN, as shown in FIG. 11, the image information x ₀ received by the input unit 10 is applied to the multitasking CNN, and the image information x ₀ is evaluated to any facial expression. Expression recognition is performed and the key points of the image information x ₀ are extracted. Note that conv shown in FIG. 11 represents a convolutional layer, and fc represents a fully connected layer. In addition, the input image information x ₀ is processed by the multitasking CNN as shown in the following equation, and the feature amount x ^L is extracted.

Then, facial expression recognition is performed by the multitasking CNN using the extracted feature quantity x ^L as input, according to the following equation (4).

(4)

Further, the multi-task CNN inputs the extracted feature quantity x ^L and performs key point extraction according to the following equation (5).

(5)

  FIG. 12 shows an example of a specific learning process. In the present embodiment, as shown in FIG. 12, facial expression recognition and keypoint extraction are learned at the same time. In addition, the relationship between the facial expression and the key point is embedded in the shared feature space.

Then, regarding facial expression recognition, the parameters w ^s1 , ..., W ^sL , w _c of the multitasking CNN are learned so as to minimize the cross entropy error shown in the following expression (6). Incidentally, y _i ^c is the label representing facial expressions, if a smiling takes one, if not a smile take 0. N is the number of learning data.

(6)

Further, regarding key point extraction, learning of the parameters w ^s1 , ..., W ^sL , w _r of the multitask CNN is performed so as to minimize the least-square error shown in the following formula (7).

(7)

The model storage unit 26 stores the parameters w ^s1 , ..., W ^sL , w _c , and w _{r of the} multitask CNN learned by the learning unit 24.

  The task processing unit 28 uses the multitasking CNN stored in the model storage unit 26 for the image information received by the input unit 10 to obtain outputs for a plurality of tasks. In the present embodiment, the image information is evaluated by facial expression recognition, and the key points of the image information are extracted.

  For the facial expression recognition, which is an example of an evaluation task among a plurality of tasks, the feedback generation unit 30 makes a keypoint, which is an example of an information presentation task, so as to approach the target value when a target value is given. Calculate the output of the extraction. Then, the feedback generation unit 30 performs the keypoint extraction based on the calculation result of the output of the keypoint extraction by the task processing unit 28 and the calculation result of the output of the keypoint extraction calculated so as to approach the target value. Present information about the output.

  Specifically, the target value is obtained by the facial expression recognition of the multitasking CNN based on the evaluation result by the task processing unit 28, the target value of the evaluation regarding facial expression recognition, and the multitasking CNN stored in the model storage unit 26. When it is evaluated as, the key points extracted by the multi-task CNN are calculated, and the difference between the key points extracted by the task processing unit 28 and the calculated key points is presented.

  More specifically, the feedback generation unit 30 corrects the output of any one of the plurality of layers when the image information input by the task processing unit 28 is applied to the multitask CNN. The correction of the output of any one of the layers is repeated until the target value is evaluated by the facial expression recognition of the multitasking CNN. Accordingly, the feedback generation unit 30 calculates the key points extracted by the multitasking CNN when the facial expression recognition of the multitasking CNN evaluates that the target value is the target value.

  FIG. 13 shows an explanatory diagram for explaining the processing by the feedback generation unit 30. As shown in FIG. 13, when the current state of “Not Smile” is obtained as an evaluation result by the task processing unit 28, the feedback generation unit 30 sets the target value “Smile” as the correct answer and the following equation (6A ) Calculate the loss function.

(6A)

Next, as shown in FIG. 14, the feedback generation unit 30 propagates the cross entropy error by backpropagation, and outputs the output of any one of the plurality of layers of the multitasking CNN (of any one of the layers). Input) Correct x ^l . The weight w, which is a parameter of the multitasking CNN, is used as it is fixed.

Then, as shown in FIG. 15, the feedback generation unit 30 calculates the feature quantity x ^L by using the corrected output x ^l by the forward propagation, and based on the calculated feature quantity x ^L , the target Calculate the loss function with the correct value.

Further, the feedback generation unit 30 repeats the correction process of the output of any one layer by backpropagation, the calculation of the feature amount x ^L , and the calculation of the loss function until the value of the loss function becomes less than the threshold value. ..
Then, when the value of the loss function becomes less than the threshold value, the feedback generation unit 30 recalculates the key point based on the feature quantity x ^L obtained by the forward propagation using the corrected output x ^1. To do.

  Then, the feedback generation unit 30 adds the difference between the key point extracted by the task processing unit 28 and the recalculated key point to the image information and outputs it as feedback.

  As described above, in the present embodiment, the difference between the current state and the target value, which is the evaluation result by the task processing unit 28, is deeply propagated in the multitask CNN and reflected in the recalculation of the keypoint (see FIG. 16). ).

  The output unit 40 presents the feedback obtained by the feedback generation unit 30.

<Operation of the multitask processing device according to the embodiment of the present invention>

  Next, the operation of the multitask processing device 100 according to the embodiment of the present invention will be described. The multitask processing device 100 executes a learning process for learning the multitask CNN and a multitask process for presenting feedback using the multitask CNN.

  First, a learning process for learning the multitasking CNN will be described. When the input unit 10 receives the learning data, the input unit 10 stores the learning data in the learning data storage unit 22. Then, the multi-task processing device 100 executes the learning processing routine shown in FIG.

<Learning processing routine>
First, in step S100, the learning unit 24 minimizes the cross entropy error shown in the above formula (6) based on the learning data stored in the learning data storage unit 22, and the minimum shown in the above formula (7). The parameters w ^s1 , ..., W ^sL , w _c , w _r of the multitasking CNN are learned so as to minimize the squared error. As a result, a multitasking CNN that shares a feature space for facial expression recognition and keypoint extraction is obtained.

Next, in step S102, the learning unit 24 stores the parameters w ^s1 , ..., W ^sL , w _c , and w _r of the multitask CNN learned in step S100 in the model storage unit 26, and the learning processing routine. To finish.

  Next, the multitasking process of presenting feedback using the multitasking CNN will be described. Hereinafter, a case where the input unit 10 sequentially receives image information will be described as an example. When the input unit 10 receives the image information, the multitask processing device 100 executes the multitask processing routine shown in FIG. 18. The multitask processing routine shown in FIG. 18 is an example of processing executed according to the pseudo chart of the algorithm shown in FIG.

<Multitask processing routine>
In step S200, a target value for evaluation regarding facial expression recognition is set. For example, “Smile” is set as the target value.

  In step S202, the input unit 10 acquires the input image information.

  In step S203, the task processing unit 28 applies the image information received in step S202 to the multitask CNN stored in the model storage unit 26 to calculate the feature amount of the image information.

  In step S204, the task processing unit 28 evaluates the image information by facial expression recognition by the multitasking CNN stored in the model storage unit 26 based on the feature amount calculated in step S203, and the evaluation result is the current state. To get as.

  In step S206, the task processing unit 28 extracts the key point of the image information by the multitask CNN stored in the model storage unit 26 based on the feature amount calculated in step S203.

  In step S208, the feedback generation unit 30 determines whether the target value set in step S200 and the current state acquired in step S204 match. If the target value and the current state match, the process proceeds to step S210. On the other hand, if the target value and the current state do not match, the process proceeds to step S212.

  In step S210, the feedback generation unit 30 displays the key points extracted in step S206.

In step S212, the feedback generation unit 30 determines the target value of the evaluation regarding facial expression recognition set in step S200, the multitask CNN stored in the model storage unit 26, and the multitask CNN calculated in step S203. Based on the outputs x ¹ , ..., X ^{L of} each layer, the key points extracted by the multitasking CNN are recalculated when they are evaluated as target values by the facial expression recognition of the multitasking CNN. Step S212 is implemented by the feedback generation processing routine shown in FIG.

<Feedback generation processing routine>
In step S300, the feedback generation unit 30 calculates the loss function based on the feature value x ^L , the target value of the evaluation regarding facial expression recognition, and the multitask CNN stored in the model storage unit 26, and the calculation result L get ^s .

In step S302, the feedback generation unit 30 performs, based on the calculation result L ^s of the loss function acquired in step S300, for example, back propagation according to the following equation (8) to input x ^l of the specific layer. To fix.

(8)

In step S304, the feedback generation unit 30 calculates the feature amount x ^L by forward propagation based on the correction result in step S302.

In step S306, the feedback generation unit 30 determines whether the value of L ^s calculated in step S300 is equal to or less than a preset threshold value. If the value of L ^s is less than or equal to the preset threshold value, the process proceeds to step S307. On the other hand, when the value of L ^s is larger than the preset threshold value, the process returns to step S300.

  In step S307, the feedback generator 30 extracts key points of the image information.

In step S308, the feedback generation unit 30 recalculates the keypoint by the multitasking CNN stored in the model storage unit 26 based on the feature value x ^L calculated in step S303, and the recalculated key is calculated. The point is output, and the feedback generation processing routine ends.

  Next, returning to step S214 of the multi-task processing routine of FIG. 18, the difference between the key point extracted in step S206 and the key point obtained in step S212 is displayed as feedback. For example, the arrow pointing from the key point extracted in step S206 to the key point obtained in step S212 is displayed as feedback.

  In step S216, the feedback generation unit 30 determines whether to end the process. For example, when the input of the image information is completed, the multitask processing routine is ended. When the image information is continuously input, the process returns to step S202.

[Experimental result]

  Next, an experimental result regarding the multitask processing apparatus 100 according to the present embodiment will be described. As shown in FIG. 21, an experiment was conducted using four types of multitasking CNNs (a) to (d).

  The multitasking CNN in (a) corrects the output of the shallow layer. Further, the multitasking CNN of (b) corrects the output of a deeper layer than the multitasking CNN of (a). Further, the multitasking CNN of (c) corrects the output of the deepest layer. Therefore, the models (a) to (c) are obtained by changing the position of the layer of correction by backpropagation when generating feedback. On the other hand, the multi-task CNN in (d) is a CNN that separates facial expression recognition, which is an evaluation task, and keypoint extraction, which is an information presentation task, and facial expression recognition and keypoint extraction are learned by different networks. To be done.

  Table 1 shows the details of the CNN network structure used in the experiment.

  In addition, as shown in FIG. 22, the evaluation of the moving direction of the corner of the mouth was defined by FACS (Facial Action Coding System). FACS defines the criteria for physical changes in the facial expression of a certain facial expression. For example, “Anger: Lip tightener, Happiness: Lip corner puller” corresponds to the movement direction of facial expression and mouth corner. Criteria define the criteria.

  Table 2 shows a summary of the experimental results. As shown in Table 2, it can be seen that the shallower the layer to be modified by back propagation, the better the accuracy, and the deeper the layer, the worse the accuracy. From this, it can be seen that the relationship between the key points, which are the landmarks in the face area, and the facial expression expressing the emotion of the face is held in these layers (the shallow layer of the multitasking CNN or the subsequent layer). In addition, (d) which is a model of a network in which two tasks are divided has the worst result. From this, it is found that the learning by the multitask model is useful.

  Further, FIG. 23 shows an example of presentation of feedback obtained by the multitasking CNN. “To happiness” represents feedback for smiling, and it can be seen that the feedback is properly presented. Also, it can be seen that it is possible to adaptively generate feedback for various input and facial expressions of the target.

  As described above, according to the multi-task processing device according to the embodiment of the present invention, with respect to the input image information, the multi-task model for processing the plurality of tasks is used, and By obtaining the output of, the appropriate evaluation result and key point for the image information can be acquired from one model.

  Further, according to the multi-task processing device according to the embodiment of the present invention, the input image information is subjected to an evaluation task for evaluating the image information and an information presentation task for extracting a key point from the image information. By applying to the multi-task CNN of, the input image information is evaluated, and the key points of the input image information are extracted, an appropriate evaluation result and key points for the image information can be obtained from one model. Can be obtained.

  Further, according to the multi-task processing device according to the embodiment of the present invention, the multi-task CNN for evaluation is based on the evaluation result by the task processing unit, the evaluation target value by the evaluation task, and the multi-task CNN. When the task evaluates to be the target value, the key point extracted by the information presenting task of the multitask CNN is calculated, and the difference between the key point extracted by the task processing unit and the calculated key point is calculated. By presenting, it is possible to present appropriate feedback.

  Further, according to the multi-task processing device according to the embodiment of the present invention, image information is evaluated based on learning data including image information, evaluation of image information, and key points of image information. A multitasking CNN in which a feature space is shared by an evaluation task and an information presentation task for extracting key points from image information, and the evaluation task is performed based on the feature amount of the feature space. By learning the multitasking CNN that performs the information presentation task based on the feature quantity of, it is possible to obtain a multitasking model for acquiring appropriate evaluation results and key points.

  Further, by correcting the output of the subsequent layer among the multiple layers of the multitasking CNN, the output of the layer that holds the relationship between the image information evaluation and the key point of the image information is corrected, and the appropriate evaluation result is obtained. And key points can be acquired. In addition, when a deeper layer is modified from the output layer in the subsequent layers, a large number of non-linear transformations are performed up to the output, so that an output including diversity can be obtained. On the other hand, when a shallower layer is modified from the output layer in the subsequent layers, a small number of non-linear or linear transformations are performed up to the output, so that the output directly reflecting the feedback result can be obtained.

  In addition, the multitasking CNN of the present embodiment can perform two outputs for one input, and can generate feedback by propagating information between two outputs. Further, by propagating deeply in the network, input image information can also be reflected.

  Further, feedback generation can be adaptively performed on various input data, and the "rule" required for feedback generation can be learned from the data.

  The present invention is not limited to the above-described embodiments, and various modifications and applications can be made without departing from the scope of the present invention.

  For example, although cases have been described with the above embodiment as examples of using a multitasking CNN as an example of a multitasking model, the present invention is not limited to this. For example, another multitasking DNN may be used, or another multitasking model may be used instead of the DNN.

  Further, in the above-described embodiment, the case where image information is used as an example of data has been described, but the present invention is not limited to this, and for example, it can be used for acoustic signals and observation data used in the agricultural field. You may use a signal.

  In the above embodiment, the multitask processing device that performs the learning process and the multitask process has been described as an example. However, the multitask model learning device that performs the learning process and the multitask processing device that performs the multitask process are described. It may be configured as a separate device. In this case, the multitask model learning device includes a learning data storage unit 22 and a learning unit 24, and the multitask processing device includes a task processing unit 28 and a feedback generation unit.

  Further, although cases have been described with the above embodiments as examples of the evaluation task evaluation process where the facial expression recognition is evaluated as to whether or not it is a smile, the present invention is not limited to this. For example, "Regression" (example: smile degree in case of facial expression), "Ranking" (example: in case of facial expression, smiling than A, not smiling than B), etc. The evaluation task may be evaluated.

  Further, in the above-described embodiment, the process of extracting the key point from the image information is described as an example of the process of extracting the information of the information presentation task, but the present invention is not limited to this. For example, the key points of this embodiment are represented by dots, but information represented by a region such as a bounding box may be extracted.

  Further, in the above-described embodiment, the case where there is one evaluation task and one information presentation task has been described, but the present invention is not limited to this, and there are a plurality of evaluation tasks. There may be multiple presentation tasks. In this case, when a target value is given to a plurality of or one evaluation task, the output of the information presentation task is calculated so as to approach the target value.

Further, although cases have been described with the above embodiment as examples where the difference between the key points extracted by the task processing unit 28 and the recalculated key points is presented as feedback, the present invention is not limited to this. is not. For example, when presenting the area related to the feedback, the result itself recalculated with respect to the target value may be presented instead of presenting the difference.
Further, information may be presented by combining the results of a plurality of information presentation tasks. For example, if there is one that presents a region related to feedback and one that extracts key points, only the key points within the region may be presented.

  The above-mentioned multi-task processing device has a computer system inside, but the “computer system” also includes a homepage providing environment (or display environment) if a WWW system is used.

  Further, in the specification of the present application, the embodiment in which the program is preinstalled has been described, but the program can be stored in a computer-readable recording medium and provided.

10 input unit 20 arithmetic unit 22 learning data storage unit 24 learning unit 26 model storage unit 28 task processing unit 30 feedback generation unit 40 output unit 100 multi-task processing device

Claims (6)

A multi-task DNN (Deep Neural Network) having a plurality of layers for processing a plurality of tasks including an evaluation task for evaluating data and an information presentation task for extracting information from the data with respect to the input data. ) using, said evaluation task, the task processing unit for obtaining an output for said information presentation tasks,
For presenting the information when the input or output of any one layer of the multi-task DNN is modified so that the output of a plurality of or one evaluation task of the plurality of tasks approaches a target value. Recalculate the task output,
Feedback for presenting information regarding the output of the information presenting task based on the calculation result of the output of the information presenting task by the task processing unit and the recalculated calculation result of the output of the information presenting task A generator,
A multitasking processing device including. The task processing unit has a multitasking DNN having a plurality of layers, for the input image, for performing facial expression recognition of a person's face shown in the image and extracting key points representing the facial part of the person. By using, to perform facial expression recognition of the person in the image and extraction of the person's key points,
The feedback generation unit recalculates the keypoint extraction when the input or the output of any one layer of the multitasking DNN is corrected so that the facial expression recognition approaches the target value,
Presenting feedback in which the difference between the keypoint extraction result by the task processing unit and the recalculated keypoint extraction result is added to the image.
The multitasking processing device according to claim 1 . The multitasking DNN shares a feature space between the evaluation task and the information presentation task, performs the evaluation task for evaluating data based on the feature amount of the feature space, and determines the characteristics of the feature space. It is a model that performs the information presentation task of extracting information from data based on quantity .
The multitasking processing device according to claim 1 or 2 .   The multi-task processing device according to claim 1, wherein the data is image information. A multitasking DN N that shares a feature space among the plurality of tasks based on learning data composed of data and results of a plurality of tasks for the data, and based on a feature amount of the feature space, Further comprising a learning unit for learning the multitasking DNN that performs the plurality of tasks ,
The task processing unit uses the multitask DNN learned by the learning unit to obtain outputs for the evaluation task and the information presentation task,
The multitasking processing device according to claim 1. A program for a computer to function as each unit constituting the multi-task processing equipment of any one of claims 1 to 5.