JP7491382B2 - Importance calculation device, importance calculation method, and importance calculation program - Google Patents

Description

The present invention relates to an importance calculation device, an importance calculation method, and an importance calculation program.

Feature selection is an important method of data preprocessing in machine learning. For example, if unnecessary features are included in the data used to train a model, undesirable phenomena such as a decrease in model accuracy, an increase in training costs (computational complexity, time), model complexity, and overfitting may occur.

Breiman, Leo, et al. "RANDOM FORESTS." January 2001. Li, Zechao, et al. "Unsupervised feature selection using nonnegative spectral analysis." Twenty Sixth AAAI Conference on Artificial Intelligence. 2012.

However, conventional techniques have the problem that feature selection may not be performed efficiently. First, manual feature selection requires knowledge of statistics and machine learning. On the other hand, there are known techniques for performing feature selection automatically, but to implement such techniques, it is necessary to clearly define the purpose (task) for which the model will be used, and furthermore, supervised data with correct answers assigned is required. Assigning correct answers is generally a costly task.

In order to solve the above-mentioned problems and achieve the objective, the importance calculation device is characterized by having a latent representation calculation unit that calculates a latent representation of a data sample using the encoder portion of a trained autoencoder, and an importance calculation unit that calculates the importance of each feature contained in the data sample based on the gradient of the data sample with respect to the latent representation.

According to the present invention, feature selection can be performed efficiently.

FIG. 1 is a diagram illustrating an example of the configuration of an importance calculation device according to the first embodiment. FIG. 2 is a diagram showing an example of a data sample. FIG. 3 is a diagram illustrating the gradient. FIG. 4 is a diagram illustrating feature selection. FIG. 5 is a flowchart showing the flow of processing of the importance calculation device according to the first embodiment. FIG. 6 is a diagram showing an example of the relationship between features. FIG. 7 is a diagram showing an example of the importance of each feature. FIG. 8 is a diagram showing an example of the importance of each feature. FIG. 9 is a diagram showing an example of the importance of each feature. FIG. 10 is a diagram showing an example of the importance of each feature. FIG. 11 is a diagram showing an example in which the importance is arranged in descending order. FIG. 12 is a diagram showing an example in which the importance is arranged in descending order. FIG. 13 is a diagram illustrating an example of a computer that executes a importance calculation program.

Below, the embodiments of the importance calculation device, importance calculation method, and importance calculation program according to the present application are described in detail with reference to the drawings. Note that the present invention is not limited to the embodiments described below.

[Configuration of the first embodiment]
First, the configuration of the importance calculation device according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the importance calculation device according to the first embodiment. As shown in Fig. 1, the importance calculation device 10 receives input of data, calculates the importance, and outputs the importance of each feature of the data. In addition, the importance calculation device 10 has an interface unit 11, a storage unit 12, and a control unit 13.

The interface unit 11 is an interface for inputting and outputting data. For example, the interface unit 11 is a NIC (Network Interface Card). The interface unit 11 may also be connected to an input device such as a mouse or a keyboard, and an output device such as a display.

The memory unit 12 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or an optical disk. The memory unit 12 may be a semiconductor memory in which data can be rewritten, such as a random access memory (RAM), a flash memory, or a non-volatile static random access memory (NVSRAM). The memory unit 12 stores an operating system (OS) and various programs executed by the importance calculation device 10. The memory unit 12 stores model information 121.

The model information 121 is information for constructing a model. In this embodiment, an autoencoder is constructed using the model information 121. The autoencoder is also assumed to have been trained by minimizing the mean squared error between the input and output, for example.

For example, the model information 121 is the weights and biases of each node of a multilayer neural network that realizes the encoder and decoder included in the autoencoder. Note that the model information 121 only needs to include information for constructing the encoder among the encoders and decoders included in the autoencoder.

The control unit 13 controls the entire importance calculation device 10. The control unit 13 is, for example, an electronic circuit such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or GPU (Graphics Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array). The control unit 13 also has an internal memory for storing programs and control data that define various processing procedures, and executes each process using the internal memory. The control unit 13 also functions as various processing units by the operation of various programs. For example, the control unit 13 has a latent expression calculation unit 131, an influence calculation unit 132, an importance calculation unit 133, and a feature selection unit 134.

The latent representation calculation unit 131 calculates the latent representation of a data sample using the encoder portion of the trained autoencoder. The latent representation calculation unit 131 accepts input of a data sample and outputs a latent representation. Here, the data input to the importance calculation device 10 is called a data sample. Multiple data samples may be input to the importance calculation device 10. Furthermore, the data sample may be one that was used for training the autoencoder.

Fig. 2 is a diagram showing an example of a data sample. As shown in Fig. 2, for example, each data sample constitutes table data. That is, a data sample has items such as "Name", "Math", and "English". Also, the value of each item corresponds to a feature of the data sample. If the number of features is d, the data sample is expressed as x = [ _x1 , ..., _xd ].

The influence calculation unit 132 calculates one influence on the latent representation for each feature of the data sample. The influence calculation unit 132 accepts an input of the latent representation and outputs an influence. The latent representation is calculated by an encoder based on the data sample. If the number of dimensions of the latent representation is d', the latent representation is expressed as z = [ _z1 , ..., _zd' ]. Figure 3 is a diagram explaining the gradient. As shown in Figure 3, each feature of the data sample corresponds to each dimension of the latent representation.

For example, the influence calculation unit 132 calculates the influence by adding up the absolute values of the gradients for each dimension of the latent representation for each feature of the data sample. Here, the gradient of feature _xk (k∈{1,...,d}) for dimension _zi (i∈{1,...,d'}) of the latent representation can be written as ∂z _i /∂x _k . Therefore, the influence calculation unit 132 calculates _{the influence influence k of} feature xk as shown in formula (1).

In this way, when there are k features in a data sample (when the data sample is k-dimensional), the influence calculation unit 132 can calculate k influences, which are scalar values. In this case, there is a one-to-one correspondence between the input dimension and the influence.

There are as many gradients as the number of input dimensions, i.e., the product d x d' of the number of features of the data sample and the number of dimensions of the latent representation. However, by adding up the absolute values of the gradients corresponding to each dimension of the latent representation as in equation (1), a single influence level, which is a scalar value, can be obtained for one feature. Furthermore, by making the influence level a scalar value, users can easily understand it. Also, since the larger the absolute value of each gradient, regardless of whether it is positive or negative, it means that the greater the influence, so by first taking the absolute value and then adding them up as in equation (1), it is possible to prevent gradients from canceling each other out.

The importance calculation unit 133 calculates the importance of each feature included in the data sample based on the gradient of the data sample with respect to the latent expression. The importance calculation unit 133 accepts an input of the influence and outputs the importance. For example, the importance calculation unit 133 calculates a predetermined statistical quantity, such as the median and average of the influence calculated for multiple data samples, as the importance.

Furthermore, the importance calculation unit 133 calculates an evaluation value for evaluating the importance. For example, the importance calculation unit 133 calculates the median of the influence levels as the importance, and further calculates an evaluation value by multiplying the median by a value representing the magnitude of variation in the importance.

Specifically, first, the importance calculation unit 133 calculates the influence influence _k for a plurality of data samples. Here, a feature with a large influence on average in all data samples is considered to be important. Therefore, the importance calculation unit 133 calculates the evaluation value as the median of the influence × (75th percentile value of the influence − 25th percentile value of the influence). The median of the influence corresponds to the importance. Moreover, the value obtained by subtracting the 25th percentile value of the influence from the 75th percentile value of the influence is an example of a value representing the magnitude of variation.

Features with small evaluation values are considered to be of low importance to all data samples, and therefore can be judged to be of low importance overall. In addition, if the data samples have a nonlinear structure, it is likely that important features will differ depending on each individual data sample. Using the above evaluation values, it is possible to evaluate the variation in the influence of each feature on each data sample.

The feature selection unit 134 selects features based on the importance or evaluation value. The feature selection unit 134 accepts the importance or evaluation value of each feature as an input, and outputs the selected features. For example, the feature selection unit 134 may select features whose importance or evaluation value is equal to or greater than a predetermined threshold, or may select a predetermined number of features in descending order of importance or evaluation value. The importance calculation device 10 may output the importance or evaluation value, or may output the features selected by the feature selection unit 134.

Figure 4 is a diagram explaining feature selection. As shown in Figure 4, feature selection according to this embodiment involves dimensional compression using an autoencoder, and automatically selecting important (or unnecessary) features based on their contribution to the latent representation. In addition, since the essence of expressing data is extracted from the latent representation of the autoencoder, in this embodiment, features that contribute greatly to the latent representation are considered important.

[Processing of the First Embodiment]
5 is a flowchart showing a process flow of the importance calculation device according to the first embodiment. As shown in FIG. 5, first, the latent expression calculation unit 131 calculates the latent expression of the data sample using the encoder (step S101). Next, the influence calculation unit 132 calculates the influence of each feature on the latent expression (step S102).

Here, the importance calculation unit 133 calculates the importance of each feature based on the influence (step S103). Then, the feature selection unit 134 selects features whose importance is equal to or greater than a predetermined value (step S104). Furthermore, the importance calculation device 10 outputs the importance or the selected feature (step S105).

[Effects of the First Embodiment]
As described above, the latent representation calculation unit 131 calculates the latent representation of the data sample using the encoder part of the trained autoencoder. The importance calculation unit 133 calculates the importance of each feature included in the data sample based on the gradient of the data sample with respect to the latent representation. In this way, the importance calculation device 10 can automatically calculate the importance regardless of whether the correct answer is assigned to the data sample. As a result, according to this embodiment, feature selection can be performed efficiently.

Here, the analyst may be given a data sample for which the purpose and content of the task are unclear. According to this embodiment, even if the purpose and content of the task using the data sample are unclear, the importance of the features of the data sample can be calculated. Furthermore, it goes without saying that the importance calculated in this embodiment and feature selection using the importance can be used for learning autoencoders, but they can also be used for learning models other than autoencoders.

The influence calculation unit 132 calculates one influence on the latent expression for each feature of the data sample. The importance calculation unit 133 calculates a predetermined statistical amount of the influence calculated for multiple data samples as the importance. In this way, by summarizing the influence calculated for each of the multiple data samples into statistics such as the median and the average, it becomes possible to grasp the importance of each feature from a bird's-eye view.

The influence calculation unit 132 calculates the influence by adding up the absolute values of the gradients for each dimension of the latent representation for each feature of the data sample. This prevents gradients, which can be both positive and negative, from canceling each other out, making the influence a more meaningful index.

The importance calculation unit 133 calculates the median of the influence levels as the importance level, and further calculates an evaluation value by multiplying the median by a value that represents the magnitude of the variation in the importance level. This makes it possible to evaluate the importance of each feature in terms of variation.

[First Experiment]
An experiment performed by applying the first embodiment to actual data samples will be described below. In the first experiment, 50,000 nine-dimensional data samples x = [ _z1 , _z2 , _z3 , _x1 , _x2 , _x3 , _x4 , _x5 , _n1 ] were generated according to formula (2), and the influence and importance were calculated.

As shown in formula (2), _z1 , _z2 , and _z3 are features generated from independent distributions. _x1 , _x2 , _x3 , _x4 , and _x5 are features generated from distributions that depend on _z1 , _z2 , and _z3 . _n1 is always 0 and is a completely unnecessary feature, which is desirably removed by feature selection.

Both the encoder and decoder of an autoencoder are four-layer neural networks. The encoder layers are 9-dimensional, 20-dimensional, 30-dimensional, 50-dimensional, and 5-dimensional from the input side. In this case, d = 9 and d' = 5.

Fig. 6 is a diagram showing an example of the relationship between features. In Fig. 6, _z1 , _z2 , and _z3 tend to be independent of each other. Also, _x1 , _x2 , _x3 , _x4 , and _x5 are influenced by _z1 , _z2 , and _z3 . On the other hand, _n1 is not influenced by other features.

FIG. 7 is a diagram showing an example of the importance of each feature. FIG. 7 shows the importance (median of the influence) of each feature and the upper and lower limits of the influence. As shown in FIG. 7, n ₁ always has a low importance. In addition, there are features with large and small variances of the influence. This is considered to be due to the nonlinearity of the data samples. For this reason, it is considered that the important features differ for each data sample. For example, the importance of x ₂ is lower than that of x ₄ , but it is considered that the influence of x ₂ may exceed that of x ₄ depending on the data sample. In addition, by displaying the importance of each feature side by side as in FIG. 7, it is possible to visually grasp the important features.

[Second Experiment]
In the second experiment, the first embodiment was applied to a data sample obtained from an actual server log. The data sample was 82-dimensional server log data. The features were numerical data such as CPU usage and memory usage, which were standardized for each dimension. The number of data samples was 299,465.

The encoder and decoder of an autoencoder are both two-layer neural networks. The encoder layers are 82, 100, and 20 dimensions from the input side. In this case, d = 82 and d' = 20.

Figures 8, 9, and 10 are diagrams showing examples of the importance of each feature. As shown in Figures 8, 9, and 10, the variance in the distribution of influence varies from feature to feature. "used_mem_1", "free_mem_1", etc. in Figure 8 have very large variance. It is thought that features with such large variance have different influences depending on the data sample. In such cases, it is expected that new knowledge can be obtained by extracting data samples with the same influence.

On the other hand, "used_swap_1" in Figure 8, "read_request_1_sda1" in Figure 9, "disk_size_1" in Figure 10, etc. have very small variances. Features with such small variances can be said to have the same degree of influence on all data samples.

Figures 11 and 12 are diagrams showing examples in descending order of importance. "cached_mem_1" and "used_mem_1" in Figure 10 have high importance and a large variance in impact as shown in Figure 8. On the other hand, "disk_size_1" in Figure 11 has a low importance and a small variance in impact as shown in Figure 10. This means that features with low importance have a stable low impact in most data samples.

[System configuration, etc.]
In addition, each component of each device shown in the figure is functionally conceptual, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of it can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in part by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or can be realized as hardware by wired logic. Note that the program may be executed not only by the CPU but also by other processors such as a GPU.

Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.

[program]
As an embodiment, the importance calculation device 10 can be implemented by installing an importance calculation program that executes the above-mentioned importance calculation process as package software or online software on a desired computer. For example, the information processing device can function as the importance calculation device 10 by executing the above-mentioned importance calculation program on the information processing device. The information processing device here includes desktop or notebook personal computers. In addition, the information processing device also includes mobile communication terminals such as smartphones, mobile phones, and PHAs (Personal Handyphone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants).

The importance calculation device 10 can also be implemented as an importance calculation server device that provides a service related to the above-mentioned importance calculation process to a client, the client being a terminal device used by a user. For example, the importance calculation server device is implemented as a server device that provides an importance calculation service that inputs data and outputs the importance of each feature of the data. In this case, the importance calculation server device may be implemented as a Web server, or may be implemented as a cloud that provides a service related to the above-mentioned importance calculation process by outsourcing.

Figure 13 is a diagram showing an example of a computer that executes an importance calculation program. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. Each of these components is connected by a bus 1080.

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the program that defines each process of the importance calculation device 10 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing a process similar to the functional configuration of the importance calculation device 10 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

In addition, the setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in memory 1010 or hard disk drive 1090. Then, CPU 1020 reads out program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as necessary, and executes the processing of the above-described embodiment.

Note that the program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or wide area network (WAN)). The program module 1093 and the program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

REFERENCE SIGNS LIST 10 Importance calculation device 11 Interface unit 12 Storage unit 13 Control unit 121 Model information 131 Latent expression calculation unit 132 Influence calculation unit 133 Importance calculation unit 134 Feature selection unit

Claims (6)

a latent representation calculation unit that calculates latent representations of data samples using the encoder part of the trained autoencoder;
an importance calculation unit that calculates an importance of each feature included in the data sample based on a gradient of the data sample with respect to the latent representation;
a feature selection unit that selects the feature whose importance is equal to or greater than a predetermined threshold based on the importance calculated by the importance calculation unit;
An importance calculation device comprising: An influence calculation unit is further provided for calculating one influence on the latent representation for each feature of the data sample;
2. The importance calculation device according to claim 1, wherein the importance calculation unit calculates, as the importance, a predetermined statistic of the influence calculated for a plurality of the data samples. The importance calculation device according to claim 2, characterized in that the influence calculation unit calculates the influence by adding up the absolute values of the gradients for each dimension of the latent representation for each feature of the data sample. the importance calculation unit calculates a median of the influence levels as an importance level, and further calculates an evaluation value by multiplying the median by a value representing a degree of variation in the importance levels ;
3. The importance calculation device according to claim 2 , wherein the feature selection unit selects the features based on the importance or evaluation value calculated by the importance calculation unit . An importance calculation method executed by an importance calculation device,
A latent representation calculation step of calculating latent representations of data samples using the encoder part of the trained autoencoder;
an importance calculation step of calculating an importance of each feature included in the data sample based on a gradient of the data sample with respect to the latent representation;
a feature selection step of selecting the features whose importance is equal to or greater than a predetermined threshold based on the importance calculated by the importance calculation step;
The importance calculation method includes the steps of: An importance calculation program for causing a computer to function as an importance calculation device according to any one of claims 1 to 4.

Images