JP7646497B2 - Learning device and learning method - Google Patents

Description

The present invention relates to a learning device and a learning method.

Conventionally, a method called knowledge distillation has been known as one of the learning methods for deep learning (see, for example, Patent Document 1). In knowledge distillation, a large-scale teacher model with a large number of parameters is used to train a student model with a small number of parameters. Knowledge distillation can reduce the size of the model while suppressing degradation of the model's performance.

International Publication No. 2020/161797

In a learning method that uses knowledge distillation, if a student model is trained using a teacher model uniformly regardless of the magnitude of the teacher model's inference error, there is a risk that the student model will be trained incorrectly if the teacher model has a large inference error. In other words, if the student model is trained while ignoring the inference error of the teacher model, there is a risk that the performance of the student model will be degraded.

In view of the above, the present invention aims to provide a technology that can appropriately train a student model in deep learning using knowledge distillation.

An exemplary learning device of the present invention is a learning device that trains a student model using a teacher model, and includes a first loss calculation unit that calculates a first loss, which is a loss for the learning data of the inference result of the teacher model, a second loss calculation unit that calculates a second loss, which is a loss for the learning data of the inference result of the student model, and a determination unit that determines whether or not to use the teacher model for training based on the first loss and the second loss.

An exemplary learning method of the present invention is a learning method for training a student model using a teacher model, and determines whether to use the teacher model for training based on a first loss, which is a loss of the inference result of the teacher model for the learning data, and a second loss, which is a loss of the inference result of the student model for the learning data.

According to the exemplary present invention, in deep learning using knowledge distillation, a student model can be appropriately trained.

Block diagram showing the hardware configuration of a learning device A block diagram showing the functional configuration of a processor included in the learning device. A flow chart showing the flow of training a student model in a learning device.

An exemplary embodiment of the present invention will now be described in detail with reference to the drawings.

1. Learning device
1 is a block diagram showing a hardware configuration of a learning device 10 according to an embodiment of the present invention. The learning device 10 is a learning device for machine learning, and uses knowledge distillation as a learning method. The learning device 10 is a learning device that trains a student model using a teacher model. The learning device 10 executes a learning method for training the student model using the teacher model. In detail, the learning device 10 causes the student model to learn using training data and the teacher model.

In this embodiment, the learning data is input to the learning device 10. The learning data is also learning data with a correct answer label. The learning device 10 has a teacher model and a student model. The teacher model is a large-scale deep neural network with a large number of parameters, and is a model that has already been trained. The student model is a small-scale deep neural network with a smaller number of parameters than the teacher model. The student model is a model that will be used for learning (training).

In this embodiment, the teacher model and the student model are models that perform object detection. The teacher model has higher performance than the student model. For example, the object detection rate of the teacher model is higher than that of the student model. According to this embodiment, a high-performance trained model that can perform object detection processing at high speed can be generated using knowledge distillation.

As shown in FIG. 1, the learning device 10 includes a processor 11 and a memory unit 12. The memory unit 12 stores a teacher model and a student model. The teacher model includes learned parameters. The student model includes parameters that are updated through learning. The processor 11 performs a function of performing inference on learning data using the teacher model stored in the memory unit 12, and inference on learning data using the student model. The processor 11 performs a function of updating the parameters of the student model so that the error between the inference result of the teacher model and the training result of the student model is reduced. Note that a feature map of the intermediate layer may be used instead of the inference result.

FIG. 2 is a block diagram showing the functional configuration of the processor 11 included in the learning device 10 according to an embodiment of the present invention. The functions of the processor 11 are exerted by executing arithmetic processing according to a program stored in the storage unit 12. As shown in FIG. 2, the processor 11 of this embodiment includes, as its functions, an extraction unit 110, a first inference unit 111, a second inference unit 112, a first loss calculation unit 113, a second loss calculation unit 114, a judgment unit 115, and a learning unit 116. In other words, the learning device 10 includes the extraction unit 110, the first inference unit 111, the second inference unit 112, the first loss calculation unit 113, the second loss calculation unit 114, the judgment unit 115, and the learning unit 116. The extraction unit 110 extracts a predetermined number of learning data from a plurality of learning data with correct answer labels prepared in advance to generate a mini-batch.

The first inference unit 111 uses a trained teacher model to perform inference on the input training data. The function of the first inference unit 111 is realized by the processor 11 reading in a teacher model stored in the memory unit 12. Through inference, an inference result that is the final result of the inference and a feature map that is an intermediate output are obtained. In this embodiment, the training data is image data, and the first inference unit 111 performs object detection on the input image data. The inference result of the first inference unit 111 includes the identification of the type and position of the object. The type of object is, for example, a pedestrian, a bicycle, a car, a traffic light, etc. A bounding box is used to identify the position.

The second inference unit 112 performs inference on the input learning data using a student model that is the subject of training by the teacher model. The function of the second inference unit 112 is realized by the processor 11 reading the student model stored in the memory unit 12. Through inference, an inference result that is the final result of inference and a feature map that is an intermediate output are obtained. The parameters of the student model are successively updated through learning. In this embodiment, as described above, the learning data is image data, and the second inference unit 112 performs object detection on the input image data. As with the first inference unit 111, the inference result of the second inference unit 112 includes identification of the type and position of the object.

The first loss calculation unit 113 calculates a first loss Loss_1, which is a loss of the inference result of the teacher model for the training data. The first loss Loss_1 is an error between the inference result of the teacher model and the correct label of the training data (hereinafter, may be expressed as an inference error). In this embodiment, the learning task is object detection, and the first loss Loss_1 includes, for example, a classification loss and a bounding box regression loss. The first loss Loss_1 can be calculated, for example, by the following formula (1). That is, the first loss Loss_1 can be calculated by the sum of the first to fifth terms of formula (1).

Note that the following formula (1) assumes the use of YOLO, an example of an object detection algorithm. YOLO divides the input image into an S x S grid, and for each grid, predicts the center coordinates (x, y), width and height scales (w, h), and probability (C) of an object present in B rectangular regions with a certain aspect ratio. Furthermore, when an object is present in each rectangular region, it also predicts the posterior probability p(c) indicating which class the object belongs to. YOLO combines predictions of rectangular regions, object presence probability, and class posterior probability into a single loss function.

添え字がない変数は予測値、添え字「truth」が付与された変数が正解を示す。λcoord、λnoobjは係数である。式（１）において、第１項および第２項は、矩形領域の中心座標と大きさに関する損失関数を示す。式（１）において、第３項および第４項は、存在確率に関する損失関数を示す。式（１）において、第５項はクラスの事後確率に関する損失関数を示す。

Variables without subscripts indicate predicted values, and variables with the subscript "truth" indicate correct answers. λcoord and λnoobj are coefficients. In equation (1), the first and second terms indicate loss functions related to the center coordinates and size of the rectangular region. In equation (1), the third and fourth terms indicate loss functions related to the existence probability. In equation (1), the fifth term indicates a loss function related to the posterior probability of the class.

The second loss calculation unit 114 calculates the second loss Loss_2, which is the loss of the inference result of the student model for the training data. The second loss Loss_2 is the error (inference error) between the inference result of the student model and the correct label of the training data. In this embodiment, the learning task is object detection, and the second loss Loss_2 includes, for example, a classification loss and a bounding box regression loss. The second loss Loss_2 can be calculated, for example, using equation (1) in the same way as the first loss Loss_1. That is, the second loss Loss_2 can be calculated by the sum of the first to fifth terms of equation (1).

The determination unit 115 determines whether or not to perform training using a teacher model based on the first loss Loss_1 and the second loss Loss_2. That is, according to the learning method of this embodiment, it is determined whether or not to perform training using a teacher model based on the first loss Loss_1 for the learning data of the inference result of the teacher model and the second loss Loss_2 for the learning data of the inference result of the student model. With this configuration, it is possible to configure the student model not to be trained using the teacher model when, for example, the inference error of the teacher model is larger than the inference error of the student model. That is, it is possible to suppress the distillation of erroneous knowledge of the teacher model into the student model.

In detail, when the first loss Loss_1 is smaller than the second loss Loss_2, the determination unit 115 determines that training should be performed using the teacher model. This allows knowledge of the teacher model, which is superior to the student model, to be appropriately distilled into the student model, and the performance (object detection rate, etc.) of the student model can be appropriately improved.

In addition, if the first loss Loss_1 is equal to or greater than the second loss Loss_2, the determination unit 115 determines not to use the teacher model for training. This makes it possible to prevent erroneous knowledge of the teacher model from being distilled into the student model.

The learning unit 116 learns the student model using the loss function. The learning unit 116 learns the student model so as to minimize the loss calculated by the loss function. In detail, the learning unit 116 updates the parameters of the student model using the backpropagation method. In this embodiment, the learning unit 116 changes the way the student model is learned depending on the judgment result of the judgment unit 115.

When it is determined that training should be performed using the teacher model, the learning unit 116 calculates the loss Loss_s of the student model using the second loss Loss_2 and the loss (distillation loss) Ldstl for the teacher model. The learning unit 116 learns the student model based on the calculated loss Loss_s of the student model. With this configuration, the student model can be learned using both the teacher model, which has better performance than the student model, and the learning data, and the student model can be a high-performance learning model.

The distillation loss Ldstl is the error of the inference result between the teacher model and the student model, or the error of the feature map of the intermediate layer (intermediate output). In this embodiment, the distillation loss Ldstl is the error between the feature map of the teacher model and the feature map of the student model. The distillation loss Ldstl may be, for example, the L2 loss represented by the following formula (2). However, the distillation loss Ldstl may be something other than the L2 loss, and may be, for example, the KL divergence that represents the loss between the distribution of the outputs of the teacher model and the student model.

式（２）において、ｎはサンプル（データ）の数を示す。ｆ_ｘｉ ^ｓは、サンプルｘｉの生徒モデルの出力を示す。ｆ_ｘｉ ^ｔは、サンプルｘｉの教師モデルの出力を示す。

In formula (2), n indicates the number of samples (data), f _xi ^s indicates the output of the student model for sample xi, and _{f xi} ^t indicates the output of the teacher model for sample xi.

When it is determined that training should not be performed using the teacher model, the learning unit 116 learns the student model with the loss Loss_s of the student model set as the second loss Loss_2. In this case, the student model is learned using only the learning data, but since learning can be performed by excluding the teacher model that may distill erroneous knowledge, it is possible to suppress a decrease in the performance of the student model.

2. Learning Methods
3 is a flowchart showing a flow of training a student model in the learning device 10 according to an embodiment of the present invention. Note that, before the start of the flowchart shown in FIG. 3, a trained teacher model that has been trained by a known method is created.

In step S1, the extraction unit 110 samples (extracts) a predetermined number of data from a plurality of previously prepared learning data with correct answer labels to generate a mini-batch. The learning data includes, for example, a plurality of image data. Each of the plurality of image data included in the learning data has a correct answer label. The predetermined number is an arbitrary number that is appropriately determined. For example, if the total number of previously prepared learning data (image data) is 10,000, the predetermined number is set to 100, etc. When the processing of step S1 is completed, the processing proceeds to the next step S2. Note that the mini-batch may be generated by a device other than the learning device 10. In other words, the data constituting the mini-batch may be configured to be input to the learning device 10 as appropriate. In such a configuration, the learning device 10 does not need to include the extraction unit 110.

In step S2, inference is performed by the first inference unit 111. In detail, parameters of a teacher model, which is a trained model, are read from the memory unit 12 into the processor 11, and inference is performed on the mini-batch using the teacher model. Through this inference, an inference result, which is the final result of inference, is obtained for each piece of training data that constitutes the mini-batch. When the processing of step S2 is completed, the processing proceeds to the next step S3.

In step S3, the first loss calculation unit 113 calculates the first loss Loss_1 for each learning data constituting the mini-batch using the inference result and the correct label obtained in step S2. The first loss Loss_1 is the inference error of the teacher model. The first loss Loss_1 can be calculated by the above-mentioned formula (1). When the processing of step S3 is completed, the processing proceeds to the next step S4.

In step S4, inference is performed by the second inference unit 112. In detail, parameters of the student model to be learned are read from the memory unit 12 into the processor 11, and inference is performed on the mini-batch using the student model. This inference results in an inference result that is the final result of inference for each piece of learning data that constitutes the mini-batch. When the processing of step S4 is completed, the processing proceeds to the next step S5.

In step S5, the second loss calculation unit 114 calculates the second loss Loss_2 for each learning data constituting the mini-batch using the inference result and the correct label obtained in step S4. The second loss Loss_2 is the inference error of the student model. The second loss Loss_2 can be calculated using the above-mentioned formula (1). When the processing of step S5 is completed, the processing proceeds to the next step S6.

In this embodiment, the inference and loss calculation using the teacher model is performed before the inference and loss calculation using the student model, but this is merely an example. For example, the inference and loss calculation using the student model may be performed before the inference and loss calculation using the teacher model.

In step S6, the determination unit 115 checks whether the first loss Loss_1 is smaller than the second loss Loss_2. If the first loss Loss_1 is smaller than the second loss Loss_2 (Yes in step S6), the process proceeds to the next step S7. On the other hand, if the first loss Loss_1 is equal to or greater than the second loss Loss_2 (No in step S6), the process proceeds to step S11.

In step S7, the distillation loss Ldstl is calculated. The distillation loss Ldstl can be calculated using the above formula (2). Since the inference error of the teacher model is smaller than the inference error of the student model and the inference of the teacher model is reliable, the distillation loss Ldstl is calculated to perform knowledge distillation using the teacher model. When the processing of step S7 is completed, the processing proceeds to the next step S8.

In step S8, the loss Loss_s of the student model is set to a loss obtained by using the second loss Loss_2 and the distillation loss Ldstl. In addition, the loss Loss_s of the student model may be expressed in detail by the following formula (3).
Loss_s = Loss_2 + λ・Ldstl (3)
λ is a weighting coefficient for adjusting the balance between the second loss Loss_2 and the distillation loss Ldstl. The weighting coefficient λ is a constant that is determined arbitrarily. When the process of step S8 is completed, the process proceeds to the next step S9.

In step S9, the learning unit 116 learns the student model so as to minimize the loss Loss_s of the student model. In detail, the learning unit 116 updates the parameters of the student model using the backpropagation method. When the processing of step S9 is completed, the processing proceeds to the next step S10.

In step S10, it is confirmed whether a predetermined number of learning iterations has been performed. The predetermined number of learning iterations is set in advance according to the number of learning data, etc. When the predetermined number of learning iterations has been completed (Yes in step S10), the training (learning) of the student model shown in FIG. 3 is completed. This may complete the learning of the student model and complete the trained model, but the learning process shown in FIG. 3 may also be repeated a predetermined number of times (epoch number) to complete the learning (complete the trained model).

In step S11, the loss Loss_s of the student model is set to the second loss Loss_2. This is because the inference error of the teacher model is equal to or greater than the inference error of the student model, and the inference of the teacher model is unreliable, so knowledge distillation using the teacher model is not performed. The student model is trained using only training data with correct answer labels. In other words, training using the teacher model is not performed. When the processing of step S11 is completed, the processing proceeds to the above-mentioned step S9. As a result, the student model is trained using the backpropagation method so as to minimize the second loss Loss_2 (= Loss_s).

According to the learning method shown in FIG. 3, the knowledge of the teacher model is distilled into the student model only when the inference of the teacher model is reliable, so that the knowledge of the teacher model can be appropriately distilled into the student model, and the performance of the student model can be improved. A student model that has been trained using the learning method shown in FIG. 3 has a smaller number of parameters than the teacher model, and can perform high-performance processing quickly.

<3. Important points to note>
In addition to the above embodiment, various technical features disclosed in this specification can be modified in various ways without departing from the spirit of the technical creation. In other words, the above embodiment should be considered to be illustrative in all respects and not restrictive, and the technical scope of the present invention is indicated by the claims, not the description of the above embodiment, and should be understood to include all modifications that fall within the meaning and scope of the claims. In addition, the multiple embodiments and modifications shown in this specification may be implemented in appropriate combinations to the extent possible.

10: Learning device 113: First loss calculation unit 114: Second loss calculation unit 115: Determination unit 116: Learning unit

Claims (6)

A learning device for training a student model using a teacher model, comprising: a processor;
The processor,
A first loss is calculated , which is a loss of the inference result of the teacher model with respect to the learning data;
A second loss is calculated , which is a loss of the inference result of the student model with respect to the learning data;
determining whether to perform training using the teacher model based on the first loss and the second loss;
A learning device that performs training using the teacher model if the first loss is smaller than the second loss . 2. The learning device according to claim 1, wherein when it is determined that training should be performed using the teacher model, the processor calculates a loss for the student model using the second loss and a loss for the teacher model , and learns the student model based on the calculated loss . The learning device according to claim 2 , wherein when it is determined that training should not be performed using the teacher model, the processor learns the student model by using a loss of the student model as the second loss. The learning device according to claim 1 , wherein the teacher model and the student model are models that perform object detection. A learning method for training a student model using a teacher model, comprising:
determining whether to perform training using the teacher model based on a first loss, which is a loss of the inference result of the teacher model for the learning data, and a second loss, which is a loss of the inference result of the student model for the learning data ;
A learning method, comprising: training using the teacher model if the first loss is smaller than the second loss . A program for causing a computer to execute a learning method for training a student model using a teacher model,
The computer,
calculating a first loss which is a loss of the inference result of the teacher model with respect to the training data;
Obtaining a second loss, which is a loss of the inference result of the student model for the training data;
determining whether to perform training using the teacher model based on the first loss and the second loss;
When the first loss is smaller than the second loss, training is performed using the teacher model; and
A program that serves as a means to execute a program.

Images