JP7839078B2 - Learning system - Google Patents

Description

This disclosure relates to a learning system.

Patent Document 1 discloses a technique for synchronizing the learning model on each client terminal with the master model on the integrated server before each of the multiple client terminals begins learning. Each client terminal performs machine learning on the learning model using data stored in the medical institution and sends the learning results to the integrated server. The integrated server divides the multiple client terminals into multiple client clusters and integrates the learning results for each client cluster to create a candidate master model. The integrated server evaluates the inference accuracy of each candidate master model and extracts the client terminals that are causing the accuracy to deteriorate.

International Publication No. 2021/059607

Patent Document 1 requires the cost of transmitting and storing data stored in medical institutions on client terminals. Furthermore, there is a problem of the risk of image leakage.

This disclosure was made in light of these issues, and aims to realize a learning system that can reduce the increased costs associated with image transmission and the risk of image leakage.

A learning system according to one aspect of this disclosure comprises a plurality of client terminals and a server. Each client terminal learns a first classifier that distinguishes between the distribution of first features extracted from a client image by a feature extractor and the distribution of second features extracted from a first proxy image obtained by performing a predetermined process on the client image. The client terminal transmits the first classifier to the server. The server learns the feature extractor based on the first classifier so that the distribution of third features extracted from a server image by the feature extractor approaches the distribution of the first features, and learns a first classifier by machine learning using the third features.

According to this disclosure, a learning system can be realized that reduces the increased costs associated with image transmission and the risk of image leakage.

This diagram illustrates a learning system that performs domain adaptation. This diagram illustrates the configuration of the learning system according to Embodiment 1. This figure illustrates the learning method for the feature extractor according to Embodiment 1.

The following describes specific embodiments applying this disclosure, with reference to the drawings. However, this disclosure is not limited to the following embodiments. Furthermore, for clarity, the following descriptions and drawings have been simplified as appropriate.

Background to this Disclosure First, let's explain a general federated learning method. In federated learning, a server trains a machine learning model (for example, a model for object recognition) using images collected from the web or images generated from a 3D model (also called a simulator), and distributes the initial values of the weight coefficients of the machine learning model to multiple client terminals. The training images (called server images) are also called source images.

Each client terminal collects client images (also called target images) within a house or other designated area, updates the weight coefficients of the machine learning model using these client images, and sends the updated weight coefficients to the server. Next, the server integrates (updates) the weight coefficients received from multiple client terminals and distributes the updated weight coefficients to the other client terminals.

In typical associative learning, the domains of server images and client images differ, leading to a decrease in the accuracy of the machine learning model. Therefore, domain adaptation is necessary. However, sending client images to the server raises privacy concerns; instead, server images must be sent to the client terminal.

Figure 1 illustrates a learning system 2 that performs domain adaptation. The client terminal 11 extracts features from the client image x _t ⁿ and the server image x _s using a feature extractor F _g , learns a classifier D _n to identify the client image x _t ⁿ , and sends the classifier D _n to the server 12. n is an integer from 1 to N, where N represents the number of client terminals 11. The server 12 learns a feature extractor F _g so that the distribution of features in the server image x _s approaches the distribution of features in the client image x _n (n=1 to N), and also learns a classifier C _g (also called a class classifier, corresponding to the above machine learning model), and sends the feature extractor F _g and classifier C _g to the client terminal 11.

In learning system 2, domain adaptation is performed by training the feature extractor _Fg so that the feature distribution of the server image _xs approaches the feature distribution of the client image _xtn ^. For this reason, it is necessary to send the server image _xs to the client terminal 11 and train the classifier _Dn . In this case, it is necessary to send the server image _xs to all client terminals 11, but there were problems from the standpoint of licensing and data management. The inventor of the present application came up with the learning system according to Embodiment 1 based on the above circumstances.

Embodiment 1
The learning system according to Embodiment 1 will be described below with reference to the drawings. Figure 2 is a diagram illustrating the configuration of the learning system 1 according to Embodiment 1. The learning system 1 performs domain adaptation using a proxy image that replaces the server image x _s .

^The client terminal 11 generates a proxy image _Tps ( _xtn ⁾ (also called the first proxy image) by performing a predetermined process on the client image _xtn . _Tps represents the predetermined process. The content of the predetermined process is arbitrary and can include, for example, a process to generate a random noise image, a process to add adversarial noise, a process to convert an RGB image to a BGR image, a process called dropout, a process to fractalize, a process to convert to a grayscale image, a rotation process, a process to add Gaussian noise, a process to divide into patches and swap them, or a process to overlay with other images (called mixup).

The client terminal 11 extracts features (referred to as first features) from the client image x _t ⁿ using the feature extractor F _g , and extracts features (referred to as second features) from the proxy image T _ps (x _t ⁿ ). n is an integer from 1 to N, where N represents the number of client terminals 11. The processing related to the client image x _t ⁿ is represented by a solid line, and the processing related to the proxy image T _ps (x _t ⁿ ) is represented by a dotted line. The client terminal 11 learns a classifier D _n ^k (also referred to as the first classifier) that identifies the distribution of the first features and the distribution of the second features according to equation (1), and sends the classifier D _n ^k to the server 12. There may be multiple classifiers D _n ^k . In addition, classifiers D _n ^k may be sent to other client terminals 11 for use in knowledge distillation, which will be explained below.

Furthermore, the client terminal 11 learns the classifier C _n (also called the second classifier) through unsupervised learning by knowledge distillation using the classifier C _g (also called the first classifier), which will be described later. Specifically, the client terminal 11 learns the classifier C _n according to equation (2). _{D s} represents the discriminator D _s (also called the second discriminator), which will be described later. p is a hyperparameter. KL represents the Kullback-Leibler divergence. V represents the features extracted by the feature extractor F _g . m represents an integer other than n from 1 to N. K represents the number of discriminators D _m ^k . As mentioned above, there may be multiple discriminators D _m ^k .

Classifier _Cn is trained to produce classification results similar to _{those of other classifiers Cg and Cm ,} ^taking into account the similarity of feature distributions with classifiers _Ds and _Dmk . Classifier _Cn is optimized for each client terminal 11, but classifier _{Cg may be used instead of training classifier Cn ,} as shown in the related technology in Figure 1. Classifier _Cn is transmitted to other client terminals 11.

Server 12 extracts features (referred to as third features) from the server image x _s using the feature extractor F _g . Server 12 trains the feature extractor F _g based on the classifier D _n k so that the distribution of the third features approaches the distribution of the first features, and also trains the classifier C _g using _{machine learning with the third features. Server 12 transmits the feature extractor F g} ^and the classifier C _g to the client terminal 11.

Specifically, server 12 trains the feature extractor _Fg and classifier _Cg according to equation (3). _Lce represents the loss function using cross-entropy, and _ys represents the label assigned to the server image _xs . _λs are hyperparameters. The first term of equation (3) represents normal supervised learning, and the second term represents the alignment of the feature distribution (space).

Figure 3 illustrates how server 12 trains the feature extractor _Fg . Distribution C1, enclosed by a solid line, represents the distribution of features extracted from the client image x _t ⁿ (first features). Distribution C2, enclosed by a dotted line, represents the distribution of features extracted from the proxy image T _ps (x _t ⁿ ) (second features). The code L represents a classification line that classifies distributions C1 and C2, and corresponds to the classifier D _n ^k . Distribution S, enclosed by a dashed-dotted line, represents the distribution of features extracted from the server image x _s (third features). Feature extractor _Fg is trained so that distribution S approaches distribution C1. In Figure 1, which illustrates related technologies, a classification line (discriminator) is trained to classify distribution C1 and distribution S, and a feature extractor _Fg is trained to make distribution S closer to distribution C1. However, in Embodiment 1, a classification line is trained to classify distribution C1 and distribution C2, and a feature extractor _Fg is trained to make distribution S closer to distribution C1. Therefore, in Embodiment 1, there is no need to send the server image _xs to the client terminal 11.

Returning to Figure 2, the server 12 also generates a proxy image _Tps ( _xs ) (referred to as the second proxy image) by performing the predetermined processing described above on the server image _xs . The processing related to the server image _xs is represented by a solid line, and the processing related to the proxy image _Tps ( _xs ) is represented by a dotted line. The server 12 extracts features (referred to as the fourth features) from the proxy image _Tps ( _xs ). The server 12 learns a classifier _Ds that distinguishes between the distribution of the third features and the distribution of the fourth features according to equation (4), and transmits it to the client terminal 11. The classifier _Ds is used in the knowledge distillation described above.

By repeating the processes represented by equation (1), equation (2), equation (3), and equation (4), the performance of classifier _Cn and classifier _Cg can be improved.

In conventional technology, the client terminal 11 needed to receive the server image x _s and perform domain adaptation. On the other hand, in Embodiment 1, a proxy image T _ps (x _t ⁿ ) is used instead of the server image x _s , so there is no need to send the server image x _s to the client terminal 11. Therefore, it is possible to prevent the leakage of the server image x _s while reducing the degradation of the classifier C _g 's performance due to differences in domains. In addition, the cost of transmitting and storing the server image x _s is reduced, and the risk of users leaking the server image x _s is reduced.

The program described above includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions. The program may be stored on a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drives (SSDs), or other memory technologies, CD-ROMs, digital versatile discs (DVDs), Blu-ray® discs, or other optical disc storage, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include electrical, optical, acoustic, or other forms of propagating signals.

Furthermore, this disclosure is not limited to the embodiments described above, and may be modified as appropriate without departing from its intent.

1, 2 Learning system, 11 Client terminal, 12 Server, x _t ⁿ Client image, x _s Server image, T _ps (x _t ⁿ ), T _ps (x _s ) Proxy image, F _g Feature extractor, D _n ^k , D _s Discriminator, C _n , C _g Classifier

Claims (1)

A learning system comprising multiple client terminals and a server,
Each client terminal learns a first classifier that distinguishes between the distribution of first features extracted from a client image by a feature extractor and the distribution of second features extracted from a first proxy image obtained by performing a predetermined process on the client image, and transmits the first classifier to the server.
The server is a learning system that learns the feature extractor based on the first classifier and learns the first classifier by machine learning using the third features, such that the distribution of the third features extracted from the server image by the feature extractor approaches the distribution of the first features.