JP7655446B2 - Analysis device and program - Google Patents

Description

The present disclosure relates to an analysis device and a program.

Neural networks are widely used in, for example, regression problems, classification problems, data generation, data distribution estimation, etc. In recent years, research on continuity of neural networks has also been conducted (for example, Non-Patent Document 1 and Non-Patent Document 2, etc.), and they are applied to various applications and theoretical analyses. In the Neural ODE (Ordinary Differential Equations) described in Non-Patent Document 1, the transformation from input to output is regarded as a continuous dynamical system, and a solution method for continuous differential equations is used. This method can be said to be continuity in the layer direction (horizontal direction). In addition, in the integral representation theory described in Non-Patent Document 2, the transformation in each layer is made continuous and expressed using an integral operator, and theoretical analysis of neural networks is performed using the theory of harmonic analysis. This method can be said to be continuity in the unit direction (vertical direction). Such continuity makes it possible to use operations on continuous functions such as differentiation and integration, and it can be said that performance has improved and theoretical analysis has become more sophisticated.

Meanwhile, research is being conducted into the application of mathematical concepts such as C ^* -algebras and Hilbert C ^* -modules to data analysis (for example, Non-Patent Document 3, etc.), and basic properties necessary for data analysis have been shown.

Chen, R. T. Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. Neural ordinary differential equations. In Proceedings of the Advances in Neural Information Processing Systems 31 (NeurIPS), 2018. Sonoda, S. and Murata, N. Neural network with unbounded activation functions is universal approximator. Applied and Computational Harmonic Analysis, 43(2):233-268, 2017. Hashimoto, Y., Ishikawa, I., Ikeda, M., Komura, F., Katsura, T., and Kawahara, Y. Reproducing kernel Hilbert C*-module and kernel mean embeddings. Journal of Machine Learning Research, 22(267):1-56, 2021.

However, existing research on continuity of neural networks has focused on a single model. Meanwhile, in recent years, methods such as ensemble learning and meta-learning that simultaneously train multiple models or learn features common to multiple models have been attracting attention. For this reason, it is believed that a method that targets multiple models and efficiently performs learning related to these multiple models is required.

The present disclosure has been made in consideration of the above points and provides a technology for efficiently performing learning related to multiple models.

An analysis device according to one aspect of the present disclosure includes an input unit configured to input training data for a neural network model f having a parameter θ composed of a plurality of elements that take values in a space A of all continuous functions in a compact space Z, and a parameter optimization unit configured to learn the parameter θ that minimizes a function L _reg including a predetermined loss function L, using the training data.

A technique is provided for efficiently performing learning related to multiple models.

FIG. 2 is a diagram illustrating an example of a hardware configuration of an analysis device according to the present embodiment. FIG. 2 is a diagram illustrating an example of a functional configuration of an analysis device according to the present embodiment. 10 is a flowchart illustrating an example of a parameter optimization process according to the present embodiment. FIG. 13 is a diagram showing an example of a sample. FIG. 13 is a diagram showing an example of an estimation result of a density function (part 1). FIG. 13 is a diagram (part 2) showing an example of an estimation result of a density function.

An embodiment of the present invention will be described below. In the following, an analysis device 10 capable of efficiently performing learning related to a plurality of models by continuously connecting the plurality of models will be described. In this case, mathematical concepts such as C ^* -algebras and Hilbert C ^* -modules are used to continuously express the plurality of models.

According to the analysis device 10 of this embodiment, it becomes possible to efficiently perform learning related to multiple models (i.e., learning while interacting between multiple models), and as a result, it becomes possible to efficiently analyze continuous data.

<Theoretical Framework>
The theoretical configuration of this embodiment will be described.

<Model and parameter settings>
Let A be the space of all continuous functions on a compact space Z. Note that any commutative C ^* -algebra is isomorphic to such a space A. Consider a neural network model with K hidden layers whose parameters (weights) take values in A.

The natural numbers representing the dimensions of each layer of the neural network model are denoted as N ₀ , ..., N _K+1 , where N ₀ is the dimension of the input layer, and N _K+1 is the dimension of the output layer.

For i=1, ..., K+1, let W _i be an N _i-1 ×N _i matrix with each component having a value in A. Also, let σ _i be the transformation from A ^N_i to A ^N_i , where "N_i" represents "N _i ". _{σ i} corresponds to the activation function, and by making it a nonlinear transformation, nonlinearity occurs in the transformation of the entire neural network.

In this case, consider a neural network as shown in the following formula (1) that inputs a vector A ^{N — 0} and outputs a vector A ^{N — (K+1)} , where "N_0" represents "N ₀ " and "N_(K+1)" represents "N _K+1 ".

Ｗ_１，・・・，Ｗ_Ｋ＋１のすべての成分を並べたベクトルをθ∈Ａ^Ｎとおく。ただし、Ｎ＝Ｎ_０Ｎ_１＋・・・＋Ｎ_ＫＮ_Ｋ＋１である。以下では、ｆがパラメータθを持つことを明示する場合はｆ^θと表す。Ａ^Ｎは、Ｈｉｌｂｅｒｔ空間を拡張した概念であるＨｉｌｂｅｒｔ Ｃ^＊－ｍｏｄｕｌｅと呼ばれる空間である。Ｈｉｌｂｅｒｔ Ｃ^＊－ｍｏｄｕｌｅでは、Ａに値を持つ内積が定義される。［ａ_１，・・・，ａ_Ｎ］∈Ａ^Ｎと［ｂ_１，・・・，ｂ_Ｎ］∈Ａ^Ｎに対して、Ａ^Ｎでの内積は、

Let θ∈A ^N be the vector in which all components of W ₁ , ..., W _K+1 are arranged. Here, N = N ₀ N ₁ + ... + N _K N _K+1 . In the following, when it is explicitly stated that f has a parameter θ, it will be expressed as f ^θ . ^{A N} is a space called the Hilbert C ^* -module, which is a concept that extends Hilbert space. In the Hilbert C ^* -module, an inner product with a value in A is defined. For [a ₁ , ..., a _N ]∈A ^N and [b ₁ , ..., b _N ]∈A ^N , the inner product in A ^N is:

により定義される。

It is defined by:

When training data is given, the optimal f is obtained by minimizing the loss function L after fixing σ _i and setting a loss function L that depends on the parameter θ. The loss function L is a mapping that maps the parameter θ to the element A. Considering a parameter (function value) that has a value in A is equivalent to continuously connecting multiple models. The training data includes at least a sample that represents the input to the neural network f. In addition, when the optimal f is obtained by supervised learning, the training data also includes a teacher for the sample (i.e., a correct example of the output of the neural network f when the sample is input). Note that the teacher is also called a label, etc. in classification tasks, etc.

<Loss function setting>
In order to perform learning while interacting between multiple continuous models, _Lreg , which is a regularization term added to a normal loss function L (e.g., mean square error, cross entropy error, etc.), is minimized. When Z is a finite measure space, _Lreg is defined as shown in the following formula (2).

ただし、λ∈Ａは正則化項の重みを表すハイパーパラメータである。

Here, λ∈A is a hyperparameter representing the weight of the regularization term.

<Minimization of the loss function>
Consider minimizing the loss function L using a gradient method. For a mapping L: A ^N →A and a parameter θ∈A ^N , the gradient ∇ _θ L of the A value of L at θ is defined as follows:

"There exists a ξ∈A ^N such that for every δ∈A ^N and every z∈Z,

を満たすとき、∇_θＬ＝ξと定める。」
例えば、あるＦ：Ｒ^Ｎ×Ｚ×→Ｒ（ただし、Ｒは実数全体を表す。）により、Ｌ（θ）（ｚ）＝Ｆ（θ（ｚ），ｚ）と表せるとき、各ｚ∈Ｚを通常の勾配∇_θ（ｚ）Ｆ（・，ｚ）に対応させる写像が連続であれば、この写像はＡ値の勾配∇_θＬである。

When this is satisfied, we define ∇ _θ L = ξ.
For example, for some F: R ^N × Z × → R (where R represents the set of real numbers), if L(θ)(z) = F(θ(z), z), then if the mapping that maps each z∈Z to the ordinary gradient ∇ _θ(z) F(·,z) is continuous, then this mapping is the A-valued gradient ∇ _θ L.

In this case, the gradient of L _reg (θ) defined by the above formula (2) can be calculated as shown in the following formula (3).

ただし、λ'はＡ^Ｎのベクトルで、要素がすべてλであるものとする。

Here, λ' is a vector of A ^N , and all elements are λ.

Using ∇ _θ L _reg , a gradient method scheme is constructed as shown in the following equation (4).

ただし、θ_０∈Ａ^Ｎである。また、η_ｔ∈Ａは学習率を表すハイパーパラメータである。なお、上記の式（４）は右辺で左辺を更新することを表す。

Here, θ ₀ ∈ A ^N. Also, η _t ∈ A is a hyperparameter representing a learning rate. Note that the right side of the above formula (4) indicates that the left side is updated by the right side.

Here, since it is actually impossible to handle the elements (functions) of A themselves on a computer, the calculation is performed, for example, as follows. Take a finite-dimensional subspace V of A ^N and a mapping P:A ^N →V onto V, and replace ∇ _θ L with P(∇ _θ L). Also, the hyperparameters λ and η _t are selected as constant functions. P is determined, for example, by regression at suitable points z ₁ , ..., z _m ∈ Z. Also, since Z is compact, the sequence

がｔに関して一様にＬｉｐｓｃｈｉｔｚ連続であれば、

If is uniformly Lipschitz continuous in t, then

が各点収束するならば一様収束することが示せる。そこで、Ｖとしては、Ｌｉｐｓｃｈｉｔｚ連続関数を基底として持つ有限次元の部分空間等を考える。

It can be shown that if converges at each point, then it converges uniformly. Therefore, as V, a finite-dimensional subspace having a Lipschitz continuous function as a basis is considered.

<Example of Hardware Configuration of Analysis Device 10>
An example of a hardware configuration of an analysis device 10 according to this embodiment is shown in Fig. 1. As shown in Fig. 1, the analysis device 10 according to this embodiment includes an input device 101, a display device 102, an external I/F 103, a communication I/F 104, a RAM (Random Access Memory) 105, a ROM (Read Only Memory) 106, an auxiliary storage device 107, and a processor 108. Each of these pieces of hardware is connected to each other via a bus 109 so as to be able to communicate with each other.

The input device 101 is, for example, a keyboard, a mouse, a touch panel, a physical button, etc. The display device 102 is, for example, a display, a display panel, etc. Note that the analysis device 10 does not have to have at least one of the input device 101 and the display device 102, for example.

The external I/F 103 is an interface with an external device such as a recording medium 103a. The analysis device 10 can read and write data from and to the recording medium 103a via the external I/F 103. Examples of the recording medium 103a include a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disk), a SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

The communication I/F 104 is an interface for connecting the analysis device 10 to a communication network. The RAM 105 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 106 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 107 is a storage device (storage device) such as a hard disk drive (HDD) or a solid state drive (SSD). The processor 108 is an arithmetic device such as a central processing unit (CPU) or a graphics processing unit (GPU).

The analysis device 10 according to this embodiment has the hardware configuration shown in Fig. 1, and is therefore capable of implementing the parameter optimization process described below. Note that the hardware configuration shown in Fig. 1 is merely an example, and the hardware configuration of the analysis device 10 is not limited thereto. For example, the analysis device 10 may have multiple auxiliary storage devices 107 and multiple processors 108, or may have various other hardware besides the hardware shown in the figure.

<Example of functional configuration of analysis device 10>
An example of the functional configuration of the analysis device 10 according to this embodiment is shown in Fig. 2. As shown in Fig. 2, the analysis device 10 according to this embodiment has a data input unit 201, a parameter optimization unit 202, an output calculation unit 203, and a data output unit 204. Each of these units is realized, for example, by a process in which one or more programs installed in the analysis device 10 are executed by the processor 108. The analysis device 10 according to this embodiment also has a storage unit 205. The storage unit 205 is realized, for example, by the auxiliary storage device 107 or the like. Note that the storage unit 205 may be realized, for example, by a storage device connected to the analysis device 10 via a communication network.

The data input unit 201 inputs learning data provided to the analysis device 10. The parameter optimization unit 202 updates the parameter θ according to equations (3) and (4) using the learning data input by the data input unit 201. The output calculation unit 203 calculates the output of f when a sample included in the learning data is input, when the parameter optimization unit 202 calculates the gradient ∇ _θ L _reg according to equation (3). The data output unit 204 outputs the parameter θ updated by the parameter optimization unit 202. The storage unit 205 stores various information (e.g., learning data, parameter θ, hyperparameters λ and η _t , etc.).

<Parameter optimization process flow>
The flow of the parameter optimization process according to this embodiment will be described with reference to Fig. 3. Hereinafter, M is assumed to be a sufficiently large natural number, and ε is assumed to be a sufficiently small positive real number.

The data input unit 201 inputs given learning data (generally a large number of learning data) (step S101).

The parameter optimization unit 202 initializes t=0 and also initializes the parameter θ _t =θ ₀ (step S102). Note that the parameter optimization unit 202 may initialize the parameter θ ₀ using, for example, a random number.

The parameter optimization unit 202 judges whether t<M or ∥L( _θt )∥>ε (step S103). If it is judged that t<M or ∥L( _θt )∥>ε, the parameter optimization unit 202 proceeds to step S104, and if not (i.e., t≧M and ∥L( _θt )∥≦ε), the parameter optimization unit 202 proceeds to step S107.

The parameter optimization unit 202 calculates the gradient ∇ _θ L _reg by equation (3) (step S104). At this time, the parameter optimization unit 202 calculates equation (3) after replacing ∇ _θ L in equation (3) with P(∇ _θ L). In addition, when the parameter optimization unit 202 calculates equation (3), the output calculation unit 203 calculates the output of f when a sample included in the learning data is input, thereby calculating P(∇ _θ L).

Next, the parameter optimization unit 202 updates the parameter θ _t+1 by equation (4) (step S105).

Next, the parameter optimization unit 202 updates t ← t + 1 (step S106).

The data output unit 204 outputs the parameter _θt (step S107). The data output unit 204 may output the parameter _θt to any predetermined output destination (for example, the storage unit 205, another program, another device connected via a communication network, etc.). This provides an optimal parameter θ= _θt , and as a result, an optimal f= ^fθ having this parameter is obtained.

<Application Examples>
Application examples of the analysis device 10 according to this embodiment will be described below.

<Application Example 1: Density estimation using normalizing flow>
Let Z be a compact probability space and D be a probability measure on Z. Also, let Ω be a probability space, and consider a random variable X on Z×Ω that takes values in R ^N_0 (where R is the set of real numbers and "N_0" represents "N ₀ "). For any ω∈Ω, let X(.,ω) be continuous.

In this case, if the random variable X(z, ) follows a normal distribution for any z∈Z, then the random variable

が、学習データとして与えられたサンプルの分布に従うように上記の式（１）で定義したｆ＝ｆ^θを学習する。以下、上記の数９に示す確率変数の分布の密度関数をｐ_ｄａｔａ ^θ，ｚと表す。

f = ^fθ defined in the above formula (1) is learned so that it follows the distribution of the sample given as the learning data. Hereinafter, the density function of the distribution of the random variable shown in the above formula 9 is represented as p _data ^θ,z .

Let x ₁ , ..., x _n ∈ ^{R N_0} be samples given as training data. The loss function L is defined based on the idea of maximizing the likelihood as follows:

とする。このＬに対して、図３に示すパラメータ最適化処理によりパラメータθの最適化を行う。ｆ^θの入力と出力はそれぞれＡ^Ｎ_０のベクトルとＡ^{Ｎ_（Ｋ＋１）}のベクトルであるが、入力は値がｘ_ｉの定数関数とする。出力については、推定が正しければｐ_ｄａｔａ ^θ，ｚはｚに依存しないはずである。そこで、例えば、出力として得られるＺ上の関数を、Ｚ上でＤを用いて積分する等して最終的な出力として密度関数を得る。

For this L, the parameter θ is optimized by the parameter optimization process shown in FIG. 3. The input and output of ^fθ are the vector A ^N_0 and the vector A ^N_(K+1) , respectively, but the input is a constant function whose value is x _i . As for the output, if the estimation is correct, p _data ^θ,z should not depend on z. Therefore, for example, the function on Z obtained as the output is integrated on Z using D to obtain a density function as the final output.

<Application Example 2: Few-shot learning>
Few-shot learning is a method for learning a model from a small number of samples. As a method for improving the accuracy of Few-shot learning, there is meta-learning. In meta-learning, a meta-model is learned in advance using learning data for a plurality of tasks, thereby learning features common between tasks. By using this feature, even if the number of samples for a new task is small, a model for the new task can be efficiently learned, improving accuracy. For example, in the method proposed in Reference 1, a mapping Z that maps a task to an expression in a low-dimensional space and a mapping Θ that converts the expression in the low-dimensional space into the parameters of the model are learned as meta-models. By combining such a meta-model with the parameter optimization process shown in FIG. 3, the accuracy of Few-shot learning can be further improved.

Consider a classification problem as an example. Let the samples be x ₁ , ..., x _n ∈ ^{R N_0} . Let the label for sample x _i be y _i ∈ ^{R N_(K+1)} . Let X _i be a constant function that takes a value for x _i , and Y _i be a constant function that takes a value for y _i . In this case, for sample x _i ∈ ^{R N_0} and its label y _i ∈ ^{R N_(K+1)} , f = f ^θ defined in the above formula (1) is learned so as to become a mapping that converts the constant function X _i into a constant function Y _i .

Z and Θ in the meta-model are trained in advance to obtain a representation z _new in a low-dimensional space for a new task. The neighborhood of z _new is set as Z, and the initial value θ ₀ of the model parameters for the new task is set as a mapping that restricts Θ to Z. The loss function L is the cross-entropy error

を用いる。ただし、〈・，・〉は、Ａ^ＮでのＡに値を持つ内積である。

where 〈·,·〉 is the A-valued inner product in A ^N.

<Evaluation Results>
The results of evaluation of the analysis device 10 according to this embodiment will be described below.

Suppose we are given two types of 100 samples, as shown in the left and right figures of Figure 4. In this case, the density functions of the distributions that each of the two types of 100 samples follows are estimated using the method described in Application Example 1.

Let Z = [-4, 4] x [-4, 4],

とおき、Ａ^Ｎの有限次元部分空間ＶをＳｐａｎ｛ｖ_１，・・・，ｖ_９｝^Ｎとおいた。また、Ｐは、ｖ_１，・・・，ｖ_９によるカーネルリッジ回帰とした。つまり、（ｉ，ｊ）成分が

Let V be the finite-dimensional subspace of A ^N as Span {v ₁ , ..., v ₉ } ^N. P is a kernel ridge regression with v ₁ , ..., v _9. In other words, the (i, j) component is

であるような行列をＧとしたとき、θ＝［θ_１，・・・，θ_Ｎ］^Ｔに対するＰ（θ）の第ｉ成分は、［ｖ_１，・・・，ｖ_９］（Ｇ＋μＩ）^－１［θ_ｉ（ｚ_１），・・・，θ_ｉ（ｚ_９）］^Ｔにより計算される。ここで、μ≧０はハイパーパラメータ、Ｉは単位行列である。

If G is a matrix such that θ=[θ ₁ , ..., θ _N ] ^T , the i-th component of P(θ) for θ=[θ 1 , ..., θ N ] ^T is calculated by [v ₁ , ..., v ₉ ](G+μI) ⁻¹ [θ _i (z ₁ ), ..., θ _i (z ₉ )] T, where μ≧0 is a hyperparameter and I is an identity matrix.

In addition, the random variable X(z,・) follows a normal distribution with mean z and standard deviation 1.

In this case, when the sample shown in the left diagram of FIG. 4 is given, the result of estimating the density function by the method described in Application Example 1 is shown in the left diagram of FIG. 5, and the result of separately learning nine models using normal distributions with mean z _i and standard deviation 1 for i = 1, ..., 9 and averaging their outputs is shown in the right diagram of FIG. 5. Similarly, when the sample shown in the right diagram of FIG. 4 is given, the result of estimating the density function by the method described in Application Example 1 is shown in the left diagram of FIG. 6, and the result of separately learning nine models using normal distributions with mean z _i and standard deviation 1 for i = 1, ..., 9 and averaging their outputs is shown in the right diagram of FIG. 6. Here, the method of obtaining the results shown in the right diagrams of FIG. 5 and FIG. 6 corresponds to the case where both λ and the hyperparameter μ of the kernel ridge regression in Equation (2) are set to zero in the method described in Application Example 1.

Comparing the left and right diagrams in Figures 5 and 6, it can be seen that the density function can be estimated more smoothly in the left diagram than in the right diagram. This is because the analysis device 10 according to this embodiment continuously connects multiple models and utilizes the continuous nature of the function.

The present invention is not limited to the specifically disclosed embodiments above, and various modifications, variations, and combinations with known technologies are possible without departing from the scope of the claims.

[References]
Reference 1: Rusu, AA, Rao, D., Sygnowski, J., Vinyals, O., Pascanu, R., Osindero, S., and Hadsell, R. Meta-learning with latent embedding optimization. In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019.

10 Analysis device 101 Input device 102 Display device 103 External I/F
103a Recording medium 104 Communication I/F
105 RAM
106 ROM
107 Auxiliary storage device 108 Processor 109 Bus 201 Data input section 202 Parameter optimization section 203 Output calculation section 204 Data output section 205 Storage section

Claims (5)

An input unit configured to input learning data for a neural network model f, the neural network model having a parameter θ composed of a plurality of elements whose values are in a space A of all continuous functions in a compact space Z;
A parameter optimization unit configured to learn the parameter θ that minimizes a function L _reg including a predetermined loss function L using the learning data;
An analysis device having the above configuration. The analysis device according to claim 1 , wherein the function L _reg is expressed by a mapping that maps the parameter θ to an element of A, and includes a regularization term that is expressed by an integral of the loss function L over Z. The parameter optimization unit is
A gradient ∇ θ L reg of the function L reg with respect to the parameter ^θ is calculated by calculating P(∇ θ L) instead of the gradient ∇ _θ L of the loss function L with respect to the parameter θ using a predetermined mapping P:A ^N _→ V, where N is the number of elements of the parameter θ and _V is a finite-dimensional _subspace of A _N ;
The analysis device according to claim 1 , wherein the parameter θ is learned by updating the parameter θ using the gradient ∇ _θ L _reg . The analysis device of claim 3, wherein the mapping P is a mapping determined by regression including kernel ridge regression. An input step of inputting learning data for a neural network model f having a parameter θ composed of a plurality of elements whose values are in a space A of all continuous functions in a compact space Z;
a parameter optimization procedure for learning the parameter θ that minimizes a function L _reg including a predetermined loss function L using the learning data;
A program that causes a computer to execute the following.

Images