JP6668116B2 - Data interpolation device and program - Google Patents

Description

  An embodiment of the present invention relates to a data interpolation device and a program applied to a database.

  In recent years, there has been an increasing demand for taking out data stored in a database and using the results of various analyzes. For example, in a platform that links not only with medical institutions and nursing care providers but also with various health service data, such as a certain smart platinum platform, it is possible to provide various services using data analysis results The realization of such a society is being studied.

  Systems and platforms often manage data stored in a database in units of records or data sets. Note that a record means an arrangement (array) of integrated data for each case. The data set means a set in which the records are collected into one or more records.

  Here, there is almost no case where all records stored in the database include complete data, and there are many records in which data is missing for some reason. For this reason, in actual data analysis, it has been considered to delete a record in which data of a part of the attribute is missing and to perform analysis using a data set including remaining records. In addition, it has been studied to perform data interpolation (interpolation) processing on a record in which data of a part of the attribute is missing by some method, and to analyze the interpolated record together.

  Missing values (contents of missing data) are classified into three types, Missing Completely At Random (MCAR), Missing At Random (MAR), and Missing Not At Random (MNAR), according to the background of the missing data. . MCAR is a case where defects occur randomly. MAR is a case where loss occurs depending on the attribute of a certain record but does not depend on (is considered to be) dependent on a missing value. The MNAR is a case where a record is missing (considered) depending on the missing value.

  By the way, recently, various services have been provided through the Internet. For this reason, a large amount of data relating to services or personal information such as health care is deposited and stored in a database. In general, when a user wants to stop a service that he or she has received, the user often withdraws from the service and deletes all the deposited data. However, in recent years, there has been cooperation among services, and so-called opt-out, in which the provision of personal data to a third party is stopped even if the service is not stopped, can be selected. In the future, while using a plurality of services, it is expected that partial processes such as opt-out of only some services or deletion of data relating to the services will increase. For this reason, in the data analysis, it is necessary to consider not only the case where the received data is simply lost, but also the case where the data is lost due to opt-out.

  Here, opt-out refers to stopping the provision of personal data to a third party at the request of the user himself and if the user has been notified of certain matters in advance, etc. This is a mechanism that allows personal data to be provided to a third party without the consent of the user.

JP 2014-63432 A Japanese Patent No. 5546981 Japanese Patent No. 4177228 JP 2006-163465 A

“Housing Data Set”, July 7,1993, [Searched April 1, 2015] Internet <URL: https://archive.ics.uci.edu/ml/datasets/Housing>

  In general, when there is a record including missing data in a data set to be analyzed, it is desirable that a system that performs data analysis performs data analysis processing by performing data interpolation processing for the missing data. . However, on the other hand, if there is a loss due to opt-out, it is necessary to consider the privacy of the user who has requested opt-out. That is, for a record in which loss due to opt-out exists, the corresponding record itself is deleted in advance.

  Here, if there are many opt-out records, simply deleting the data may reduce the data amount of the data set provided to the data analysis system and may deteriorate the accuracy of the analysis result of the data analysis process. There is. If characteristic data cannot be obtained, effective analysis and utilization may not be possible.

  Therefore, for loss due to opt-out, it is possible to improve the accuracy of data analysis while protecting the privacy of the data provider by enabling appropriate data interpolation in consideration of the privacy of the user who is the data provider. There is a problem of realizing a data interpolation device capable of performing the above.

  The data interpolation device of the present embodiment includes a data storage unit for storing data, a data division unit, a data interpolation processing unit, and a control unit. The data division unit divides the data stored in the data storage unit into data including loss and data having no loss. The data interpolation processing means executes an interpolation process on the lost data according to the opt-out based on an interpolation accuracy different from the interpolation accuracy for general loss. The control means creates provision data combining the interpolated data interpolated by the data interpolation processing means and the data having no loss.

FIG. 1 is a block diagram for explaining a configuration of a system according to a first embodiment. 5 is a flowchart for explaining data interpolation processing according to the first embodiment. FIG. 4 is a diagram illustrating an example of an entire data set according to the first embodiment. FIG. 4 is a diagram illustrating an example of a non-missing data set according to the first embodiment. FIG. 4 is a diagram illustrating an example of a missing data set according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data set after interpolation according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data set for provision according to the first embodiment. 5 is a flowchart for explaining data analysis processing according to the first embodiment. FIG. 6 is a diagram illustrating an example of a data analysis result according to the first embodiment. FIG. 6 is a diagram illustrating an example of a data analysis result according to the first embodiment. FIG. 4 is a diagram illustrating an example of a distribution of a probability density function according to the first embodiment. FIG. 9 is a diagram illustrating an example of an entire data set according to the second embodiment. The figure which shows an example of the non-missing data set concerning 2nd Embodiment. The figure which shows an example of the missing data set regarding 2nd Embodiment. FIG. 10 is a diagram illustrating an example of a data set after interpolation according to the second embodiment. FIG. 9 is a block diagram for explaining a configuration of a system according to a third embodiment. FIG. 13 is a diagram illustrating an example of a data set according to the third embodiment. The figure which shows an example of the data set for provision regarding 3rd Embodiment. FIG. 13 is a diagram illustrating an example of a data set according to a modification of the third embodiment. FIG. 13 is a block diagram for explaining a configuration of a system according to a fourth embodiment. FIG. 13 is a sequence diagram for explaining the operation of the system according to the fourth embodiment.

Hereinafter, embodiments will be described with reference to the drawings.
[First Embodiment]
As shown in FIG. 1, the system according to the present embodiment roughly includes a data interpolation device 1, a network 2, and a data analysis device 3. The data interpolation device 1 and the data analysis device 3 are connected via a network 2 and can mutually transmit data. The network 2 is a data communication network, for example, any one of a wireless LAN (Local Area Network), a wired LAN, an optical network, a telephone network, an intranet, an Ethernet (registered trademark), and the Internet, or a combination thereof. . Although the system shown in FIG. 1 has only one data interpolating device 1 and one data analyzing device 3, the present invention is not limited to this, and may have a plurality of data interpolating devices 1 and data analyzing devices 3.

  The data interpolation device 1 is a component of a database managed by a server, for example, and stores data and executes data interpolation processing. The data interpolation device 1 includes a computer and a storage, and includes a data storage unit 10, a data division unit 11, a missing data storage unit 12, a non-missing data storage unit 13, a data interpolation processing unit 14, a data interpolation selection unit 15, It has an interpolation result correspondence table storage unit 16, an interpolated data storage unit 17, a provided data storage unit 18, a communication unit 19, and a control unit 20.

  The data storage unit 10 stores data stored as a storage target. Here, in the present embodiment, the data may mean a record or a data set in addition to the meaning of generic (overall) data. The record means an arrangement (array) of collected data for each case. The data set means a set in which the records are collected into one or more records.

  The data division unit 11 divides a data set stored in the data storage unit 10 into a data set with a loss (a loss data set) and a data set without a loss (a non-loss data set). Missing data may be referred to as missing data or missing values, and non-missing data may be referred to as non-missing data. Here, in the present embodiment, the missing data set is a data set having a record including missing data (missing value).

  The missing data storage unit 12 stores the missing data sets divided by the data dividing unit 11. The non-missing data storage unit 13 stores a non-missing data set having no loss divided by the data dividing unit 11.

  The data interpolation processing unit 14 includes data interpolation execution units 14-1 to 14-n that execute data interpolation processing by a plurality of data interpolation methods, and performs data interpolation according to the data interpolation method selected by the data interpolation selection unit 15. Execute the process. In the present embodiment, the data interpolation selection unit 15 selects a data interpolation method using regression analysis for the missing data set based on the setting of the opt-out flag. The opt-out flag is a flag indicating that there is missing data (missing value) due to opt-out.

  The interpolation result correspondence table storage unit 16 holds table information indicating the correspondence between the missing data (missing value) and the interpolated interpolation data (interpolated value). The interpolated data storage unit 17 stores the interpolated data (records including the interpolated data) after the interpolation processing by the data interpolation processing unit 14. The provision data storage unit 18 stores a provision data set in which interpolation data (records including interpolation data) and non-loss data (records without loss) are combined.

  The communication unit 19 is connected to the data analysis device 3 via the network 2 and transmits the provided data set to be analyzed stored in the provided data storage unit 18. The control unit 20 controls the operation of each of the constituent elements 10 to 19, and particularly executes a control of generating a provision data set and storing the data set in the provision data storage unit 18.

The data analyzer 3 includes a computer and a storage, and includes a data storage unit 30, a data analysis unit 31, a communication unit 32, and a control unit 33. The data storage unit 30 stores the providing data set transmitted from the data interpolation device 1 as a data set to be analyzed. The data analysis unit 31 executes a data analysis process using, for example, a regression analysis, using the data set for provision. The communication unit 32 is connected to the communication unit 19 of the data interpolation device 1 via the network 2 and receives the provided data transmitted from the communication unit 19. The control unit 33 controls the operation of each of the components 30 to 32.
[Data interpolation processing]
The data interpolation processing by the data interpolation device 1 of the present embodiment will be described with reference to the flowchart of FIG.

  First, the data interpolation processing of the present embodiment targets missing data (hereinafter, sometimes referred to as a missing value) among data stored as data to be stored in the data storage unit 10. In this case, for the sake of convenience, the missing value occurs only in a certain attribute, and only the opt-out data is used. That is, the defect targeted in the present embodiment is a defect such as MAR or MNAR as described above, which occurs because data (information) related to an individual is not registered or answered depending on the individual or attribute. . However, in practice, the occurrence of a missing value differs depending on the reason, and a missing flag is set according to the reason.

  The control unit 20 of the data interpolation device 1 acquires a data set from the data storage unit 10 and starts data interpolation processing (step S1). The control unit 20 detects the opt-out flag as the loss flag and notifies the data interpolation selection unit 15 of the opt-out flag. As a result, the data interpolation selecting unit 15 selects an appropriate data interpolation method according to the opt-out. In this case, the data interpolation processing unit 14 executes the data interpolation processing by the data interpolation execution unit 14-1 according to the selected data interpolation method.

  The data dividing unit 11 divides the data set acquired by the control unit 20 into a missing data set and a non-missing data set (Step S2). The missing data storage unit 12 stores the missing data set 500 divided by the data dividing unit 11. The non-missing data storage unit 13 stores the non-missing data set 400 divided by the data dividing unit 11.

  Here, FIG. 3 is a diagram illustrating an example of a data set 300 used in the present embodiment. The data set 300 is an entire data set including a missing data set and a non-missing data set, and indicates information on the price of a house in a certain area (see Non-Patent Document 1). FIG. 4 is a diagram showing an example of the non-missing data set 400. FIG. 5 is a diagram showing an example of the missing data set 500.

  As shown in FIG. 3, the data set 300 includes a plurality of records (here, No. 1 to No. 450). Each record has an array of data for each attribute 100 and an opt-out flag 101 set for each attribute 100. The attribute 100 includes information on the price of a house in a certain area, for example, a crime rate (CRIM), a residential land development rate (ZN), a nitric oxide concentration (NOX), an average house price (MEDV), and the like. The opt-out flag 101 is a flag (1 is significant) indicating attribute data (missing value) lost by opt-out for each record. For example, the record (No. 446) indicates that attribute (NOX) data (missing value) missing due to opt-out exists.

  As shown in FIG. 4, the non-missing data set 400 is a data set divided from the data set 300 shown in FIG. 3, and includes only records (here, No. 1 to No. 400) having no missing values. Become. Therefore, as the opt-out flag 101, an invalid flag (0) is set for each attribute. Also, as shown in FIG. 5, the missing data set 500 is a data set divided from the data set 300 shown in FIG. 3, and only records having missing values (here, No. 401 to No. 450) are included. Consists of Here, as the opt-out flag 101, a flag indicating that the attribute (NOX) data of each record is a missing value lost due to the opt-out is set.

  Returning to FIG. 2, the data interpolation processing unit 14 executes data interpolation processing for interpolating missing values corresponding to the opt-out by the selected data interpolation execution unit 14-1. First, the data interpolation processing unit 14 performs a regression analysis using the non-missing data set 400 shown in FIG. 4 (Step S3). Here, as a method of interpolating missing values, for example, a method of interpolating with an average value of peripheral values or attributes, a linear interpolation that is a univariate interpolation method, an interpolation method that uses a curve interpolation, or a regression analysis that is a multivariate interpolation method Is well known. The present embodiment is configured to perform interpolation processing by a method using the regression analysis and, as described later, data analysis processing using the regression analysis.

  As a specific example of step S3, the data interpolation processing unit 14 performs a regression analysis using the house price (MEDV) as a target variable using the non-missing data set 400 (regression analysis results are omitted). The data interpolation processing unit 14 predicts a house price (target value) using a regression coefficient calculated by regression analysis (step S4).

Further, the data interpolation processing unit 14 uses the non-missing data set 400 and the predicted house price (predicted value) to calculate the probability density function f (x, μ, σ ² ) as shown in the following equation (1). A value is obtained (step S5).

  Here, x is the price of the house in the originally included non-missing data set 400, μ is the predicted price of the house (predicted value), σ is the price of the house in the originally included non-missing data set 400, and the prediction. This is the standard deviation calculated from the difference between the values.

  The data interpolation processing unit 14 calculates the interpolation accuracy (error during the interpolation processing) based on the calculation result (distribution) of the probability density function f (step S6). Here, in the present embodiment, the error from the predicted value is 50%. That is, a value obtained by adding an error of 50% from the predicted value is calculated as an interpolation value. However, depending on the application and the method of privacy protection, interpolation processing may be performed with a change such as reducing the error. Further, depending on the purpose, the value of the error may be determined in advance based on the attribute of the data. When the determination is made in advance as described above, the processing in steps S5 and S6 becomes unnecessary.

  Further, the data interpolation processing unit 14 calculates an error value of the predicted value based on the interpolation accuracy (Step S7). That is, the data interpolation processing unit 14 calculates an error value when the data is shifted by 50% using the probability density function. For example, the data interpolation processing unit 14 calculates an absolute value “x−μ” and sets the result as an error value.

  Next, the data interpolation processing unit 14 executes the interpolation processing until there is no record with a loss from the loss data set 500 shown in FIG. 5 (step S8). Specifically, the data interpolation processing unit 14 repeats the following processing (steps S9 to S12). Here, in the present embodiment, the data of the attribute (NOX) of each record of the missing data set 500 is a missing value that is missing due to opt-out.

  Using the non-missing data set 400, the data interpolation processing unit 14 performs a regression analysis using the data of the attribute (NOX) of each record as a target variable, and predicts a missing value of the attribute (NOX) (step S9). ). The regression analysis result is omitted.

  The data interpolation processing unit 14 checks whether an opt-out flag (flag 1) is set for the attribute (NOX) (step S10). If not set, the data interpolation processing unit 14 executes the process of step S9 for the attribute (NOX) of the next record (NO in step S10, NO in S12).

  On the other hand, when the opt-out flag is set, the data interpolation processing unit 14 gives the error calculated in step S7 to the predicted value of the attribute (NOX) (addition or subtraction), and The value is calculated as an interpolation value (step S11). The data interpolation processing unit 14 repeats the interpolation process until there is no record with a loss, and can create a data set 600 in which the interpolation value of the attribute (NOX) is input to the record with the loss as shown in FIG. (YES in S12).

  Here, if errors are always added, the distribution of the entire data may be biased. Therefore, in the present embodiment, a value to which an error is added by adding or subtracting at random is set as an interpolation value. Note that, for example, when the value of the deviation is larger than the predicted value, the value can be negative. Therefore, when it is obvious that the interpolation is performed by looking at the data, it is preferable to add the values. If the interpolation does not result in an appropriate record, an interpolation process suitable for the property of the attribute is performed.

  Further, in the case where there is a plurality of defects in a certain record, the following interpolation processing procedure is performed. The data interpolation processing unit 14 uses the non-missing data set 400 to perform regression analysis using a missing attribute value (attribute data) as a target variable in a record set having no missing data. A regression coefficient can be obtained for each variable. Here, interpolation processing is performed on an attribute having a defect using a regression coefficient obtained by performing a regression analysis on the attribute. By repeating this process for each attribute, it is possible to similarly interpolate a plurality of defects.

  In addition, when regression analysis is performed on an attribute to be finally analyzed (predicted), since an attribute having a larger regression coefficient has a greater influence on data analysis, such an attribute is more accurately interpolated. By doing so, the improvement of the final analysis accuracy can be expected. Therefore, it is preferable to perform such interpolation from the viewpoint of analysis accuracy. On the other hand, in the present embodiment, not a high-precision interpolation process from the viewpoint of privacy, but an interpolation process using an interpolation method with an appropriate accuracy according to the property of the attribute. In this case, for example, the accuracy of data can be finally improved by performing an interpolation process from a variable having a small influence.

  Returning to FIG. 2, the data interpolation processing unit 14 combines the interpolated record and the record without loss (step S13). Further, the data interpolation processing unit 14 deletes the opt-out flag 101 (Step S14). As a result, the data interpolation processing unit 14 creates and outputs a data set 700 as shown in FIG. 7 as a data set to be provided after the interpolation processing (step S15).

The control unit 20 stores the data set 700 interpolated by the data interpolation processing unit 14 in the provided data storage unit 18. The control unit 20 transmits (provides) the interpolated data set 700 as a provision data set to the data analysis device 3 from the provision data storage unit 18 via the communication unit 19 and the network 2. As described above, the interpolation result correspondence table storage unit 16 holds table information indicating the correspondence between the missing value and the interpolation value after the interpolation processing. The interpolated data storage unit 17 stores the record after the interpolation processing by the data interpolation processing unit 14.
[Data analysis processing]
The data analysis processing by the data analysis device 3 of the present embodiment will be described with reference to the flowchart of FIG. The data analysis process of the present embodiment is, as a specific example, a process of predicting the price of a house.

  First, the data analysis device 3 receives the providing data set (the data set 700 in FIG. 7) transmitted from the data interpolation device 1 via the communication unit 32 (step S20). The data storage unit 30 stores the received providing data set 700 (step S21). The data analysis unit 31 acquires the provided data set 700 from the data storage unit 30, and performs a regression analysis using the data set 700 (step S22).

  Specifically, in step S22, the data analysis unit 31 performs a regression analysis using the data set 700 and using a house price as a specific example of the present embodiment as a target variable (the regression analysis result is omitted). ). The data analysis unit 31 calculates a regression coefficient using the regression analysis, and predicts the price of the house as the target value from the data set 700 using the regression coefficient (step S23).

  Next, the data analysis unit 31 evaluates the prediction accuracy of the house price (predicted value) predicted in step S4 of FIG. 2 and the house price (predicted value) predicted in step S23. That is, the data analysis unit 31 calculates the sum of squares, the variance, and the standard deviation from the difference from the original house price (MEDV) (step S24). The data analysis unit 31 compares the calculated sum of squares, variance, and standard deviation to evaluate prediction accuracy (step S25).

  9 and 10 are diagrams each showing an example of the evaluation result of the prediction accuracy. From FIGS. 9 and 10, the predicted value of the house price using only the non-missing data set 400 (the predicted value in step S <b> 4 of FIG. 2) is compared with the predicted value of the house price using the interpolated data set 700. This shows that the difference between the predicted value (the predicted value in step S23) and the original house price (the actual value) is smaller. That is, in the data analysis processing of the present embodiment, by using the data set 700 after the interpolation processing provided from the data interpolation apparatus 1, the target value (house price) is predicted as compared with the case without interpolation processing. Accuracy can be improved.

  Note that the present embodiment is a case where an interpolation process is performed based on interpolation accuracy that gives an error of, for example, 50% from a predicted value, and the error value is calculated using a probability density function. However, the error value may be changed according to the purpose of the analysis or the attribute of the data, and the calculation method may be calculated not by the probability density function but by another statistical means such as a likelihood function. FIG. 11 shows an example of the distribution of the probability density function when the missing value of the attribute (NOX) is predicted in the present embodiment. Since this function has a normal distribution, in the present embodiment, a value having an appearance probability of 50% in this distribution is calculated as an interpolation value.

  As described above, according to the first embodiment, the following operational effects can be obtained.

  That is, since an opt-out flag indicating a loss due to opt-out is set as a loss flag in the data set, in a record including a loss value, it is possible to clearly distinguish whether the loss value is an opt-out or a general loss. . Therefore, when performing the data interpolation processing, it is possible to realize the interpolation processing based on the appropriate interpolation accuracy according to the opt-out. Specifically, for example, a data set (700) in which missing values are interpolated with a correction accuracy including a 50% error from the predicted value can be generated and provided to the data analysis device 3.

  The data analysis device 3 can execute a data analysis process of predicting, for example, the price of a house as a target value by using the provided data set (700). Here, the provided data set (700) is not a data set interpolated with a high correction accuracy, but a data set interpolated with an appropriate interpolation accuracy according to the opt-out. Therefore, for example, it is possible to prevent the personal information of the user (data provider) who has designated the opt-out from being interpolated with high precision, so that the privacy of the user can be protected from the data analysis result.

  On the other hand, in the present embodiment, the accuracy is relatively reduced by using the interpolated data set in the data analysis processing using the data set in which the missing data due to the opt-out is not interpolated. It is possible to obtain a data analysis result with high (prediction accuracy). In other words, when interpolating missing data, it is possible to select a data interpolation method according to the opt-out of the data provider. As a result, the privacy of the data provider (user) can be protected and the data analysis accuracy can be improved.

Note that, in the data interpolation processing of the present embodiment, the data interpolation processing unit 14 may directly execute the interpolation processing on all the data sets 300 without dividing the missing data set and the non-missing data set. .
[Second embodiment]
FIGS. 12, 13, 14, and 15 are diagrams illustrating the second embodiment. Note that the system configuration of this embodiment is the same as that of the first embodiment shown in FIG.

  In the first embodiment, as shown in FIG. 3, the opt-out flag 101 is set for each attribute of each record of the entire data set 300. On the other hand, in the present embodiment, as shown in FIG. 12, the entire data set 1200 stored in the data storage unit 10 is set only for each record.

  In the data interpolation process according to the first embodiment, when performing the interpolation process in a loop including steps S9 to S12 in FIG. 2, an interpolation value is calculated by adding an error value every time according to the opt-out flag 101 at the time of interpolation. It is determined whether or not to perform (step S10). On the other hand, in the data interpolation processing of the present embodiment, it is possible to uniquely determine whether or not to calculate an interpolation value by adding an error value to each missing value according to the opt-out flag 101.

  Specifically, in the present embodiment, the data division unit 11 converts the entire data set 1200 stored in the data storage unit 10 into a non-missing data set 1300 as shown in FIG. The data set 1400 is divided. The missing data set 1400 is a data set having records including missing values.

  The data interpolation processing unit 14 of the present embodiment assigns the attribute (NOX) of each record (401 to 450) to the data interpolation processing for the missing data set 1400 in the loop including steps S9 to S12 in FIG. Then, an interpolation value is calculated. That is, the process shifts to the process of step S11 in accordance with the opt-out flag 101 (YES in step S10 only).

  Through such processing, the data interpolation processing unit 14 can create the data set 1500 in which the interpolation value of the attribute (NOX) is input to the record having the loss as shown in FIG. 15 (see YES in step S12). Further, the data interpolation processing unit 14 combines the interpolated record and the record with no loss, and deletes the opt-out flag 101 (see steps S13 and S14). As a result, as in the first embodiment, the data interpolation processing unit 14 creates and outputs a data set 700 as shown in FIG. 7 as a data set to be provided after the interpolation processing (see step S15). ).

As described above, also in the case of the data interpolation apparatus 1 of the present embodiment, the interpolated data set 700 is transmitted (provided) to the data analysis apparatus 3 as a provision data set. Note that the data analyzer 3 is the same as that of the first embodiment, so that illustration and description are omitted. Therefore, in the case of this embodiment, the same operation and effect as those of the first embodiment can be obtained. That is, as a result, the privacy of the data provider (user) can be protected and the data analysis accuracy can be improved.
[Third Embodiment]
FIG. 16 is a diagram illustrating a system configuration according to the third embodiment. The data interpolation device 1 of the present embodiment includes a component described as a certainty factor providing unit 21. The other configuration is the same as that of the first embodiment shown in FIG. For the data interpolation processing, refer to the flowchart in FIG.

  As described in the first embodiment, the data interpolation processing unit 14 performs the interpolation processing according to the opt-out flag 101 in the loop including steps S9 to S12 in FIG. That is, the data interpolation processing unit 14 calculates an interpolation value for the attribute (NOX) of each record (401 to 450) in, for example, the missing data set 500 illustrated in FIG. At the time of this interpolation processing, in the present embodiment, the data interpolation processing unit 14 inputs an interpolation value for the attribute (NOX) of each record (401 to 450), Give.

  Therefore, in the present embodiment, as shown in FIG. 17, the data interpolation processing unit 14 creates a data set 1700 in which the interpolated value of the attribute (NOX) is input to a record having a defect and a certainty factor 102 is added. (See YES in step S12).

  As a specific example, the certainty factor 102 corresponds to the correction accuracy set in accordance with the opt-out flag 101, and is, for example, an error value of 50%. Here, when no error value is given, the certainty factor 102 is 100%. The data interpolation processing unit 14 combines the interpolated record with the record having no loss, and creates and outputs a providing data set 1800 in which the opt-out flag 101 is deleted (see step S15). As shown in FIG. 18, the data set 1800 to be provided is one in which records having no loss are given a certainty factor 102 of 100%.

  As described above, also in the case of the data interpolation apparatus 1 of the present embodiment, the interpolated data set 1800 can be provided to the data analysis apparatus 3 as a provision data set. Therefore, in the case of the present embodiment as well, the privacy of the data provider (user) can be protected and the data analysis accuracy can be improved, as in the first embodiment.

  Here, the data analysis device 3 of the present embodiment can reflect the interpolated value of the record corresponding to the certainty factor 102 in the analysis result by using the certainty factor 102 as a weighting factor during the data analysis process. Becomes That is, in the data analysis process, the certainty factor 102 can be used as a weight for the data reliability, and as a result, the accuracy of the analysis result can be improved.

In the present embodiment, the case where an error value of 50% or 100% is used as the certainty factor 102 has been described. However, the present invention is not limited to this. It may be a value appropriately set in consideration of the attribute and number of a certain record group.
[Modification]
FIG. 19 is a diagram illustrating an example of a data set 1900 according to a modification of the third embodiment. In the data set 1900, when the opt-out flag 101 is set only for each record in the above-described second embodiment, the interpolation value of the attribute (NOX) having a defect is input, and the certainty factor 102 is added. It was done.

  Note that, even in the case of the present modification, the data interpolation processing unit 14 combines the interpolated record and the record with no loss, and deletes the opt-out flag 101, so that the provision data set 1800 as shown in FIG. And output.

Also in the case of this modification, the privacy of the data provider (user) can be protected, the accuracy of data analysis can be improved, and the certainty factor 102 can be used as a weighting factor during data analysis processing. This makes it possible to reflect the interpolated value of the record corresponding to the certainty factor 102 in the analysis result. That is, in the data analysis process, the certainty factor 102 can be used as a weight for the data reliability, and as a result, the accuracy of the analysis result can be improved.
[Fourth embodiment]
FIG. 20 is a diagram illustrating a system configuration according to the fourth embodiment. The system of the present embodiment includes an interpolation method instruction device 4 for instructing and confirming a data interpolation process. Since the other configuration is the same as that of FIG. 16, the same reference numerals are given and the description is omitted. Note that, in the system of the present embodiment, when the data interpolation device 1 does not include the certainty imparting unit 21, the configuration excluding the interpolation method instruction device 4 is the same as that of FIG.

  As shown in FIG. 20, the interpolation method instruction device 4 of the present embodiment includes a computer and a storage, and includes a reception data storage unit 40, a data storage unit 41, an interpolation result correspondence table storage unit 42, and an interpolation result confirmation unit. 43, a data interpolation method selection unit 44, a data interpolation method / accuracy storage unit 45, a communication unit 46, and a control unit 47.

  Hereinafter, the operation of the system of the present embodiment will be described with reference to the sequence diagram of FIG. 21 together with FIG.

  The interpolation method instruction device 4 is connected to each of the data interpolation device 1 and the data analysis device 3 via the network 2 connected to the communication unit 46. In the present embodiment, the interpolation method instruction device 4 stores all data sets including a missing data set and a non-missing data set in the data storage unit 41. This entire data set is, for example, the same as the data set 300 in the above-described first embodiment (see FIG. 3).

  As shown in FIG. 21, the interpolation method instruction device 4 sends the data set (300 in FIG. 3) stored in the data storage unit 41 to the data interpolation device 1 via the communication unit 46 and the network 2. (Step S30). The data interpolation device 1 receives the transmitted data set by the communication unit 19 and stores the data set in the data storage unit 10.

  Further, the interpolation method instruction device 4 determines the interpolation method and the interpolation accuracy when the data interpolation device 1 executes the interpolation processing of the missing data by the data interpolation method selection unit 44 (step S31). More specifically, the data interpolation method selection unit 44 specifies, as the selection of the interpolation method, a prediction algorithm used for the interpolation processing of the missing data, and a process of designating attribute information used when interpolating the missing data. Execute In addition, since the prediction result and the prediction accuracy change depending on these specifications, it is necessary to specify according to the analysis purpose.

  Further, as described above, the interpolation method instruction device 4 executes a process of designating how much an error from the predicted value is to be given as a determination of the interpolation accuracy. The first embodiment is a case where an error of, for example, 50% from the predicted value is given as the interpolation accuracy. It should be noted that the interpolation accuracy (addition of error) affects the analysis accuracy in the data analyzer 3 using the result, and how much error can be given to protect privacy. Since it differs depending on the target attribute, it is necessary to specify the privacy protection in accordance with the purpose / level and the content of analysis.

  The interpolation method instruction device 4 stores information indicating the interpolation method and the interpolation accuracy of the missing data selected (determined) by the data interpolation method selection unit 44 in the data interpolation method / accuracy storage unit 45. The interpolation method instruction device 4 notifies the data interpolation device 1 of the information indicating the interpolation method and the interpolation accuracy stored in the data interpolation method / accuracy storage unit 45 via the communication unit 46 and the network 2 (step S32). . After performing the interpolation method and interpolation accuracy selection process (step S31), the interpolation method instruction device 4 sends a data set to the data interpolation device 1 (step S30) and notifies the interpolation method and interpolation accuracy (step S30). Step S32) may be executed as a batch process.

  The data interpolation device 1 executes a data interpolation process as shown in the first embodiment (see FIG. 2) (Step S33). Here, the data interpolation device 1 executes the data interpolation process based on the information notified from the interpolation method instruction device 4 using the interpolation method and the interpolation accuracy of the specified missing data. The interpolation method specified by the interpolation method instruction device 4 corresponds to any of the data interpolation execution units 14-1 to 14-n in the data interpolation processing unit 14. That is, the data interpolation processing unit 14 executes data interpolation processing by the data interpolation execution unit 14-1, for example, according to the designated interpolation method. The interpolation accuracy is an error from the predicted value, and corresponds to, for example, an error of 50%. The data interpolation processing unit 14 calculates an error value of 50% using the probability density function, and adds it to the predicted value.

  The data interpolation device 1 executes the data interpolation process, combines the interpolated record with the record with no loss, and deletes the opt-out flag 101 from the provided data set after the interpolation process (700 in FIG. 7). Create (see FIG. 2). The data interpolation device 1 sends the data set after interpolation processing (700 in FIG. 7) to the interpolation method instruction device 4 via the communication unit 19 and the network 2 (Step S34).

  The interpolation method instruction device 4 receives the data set after the interpolation processing sent from the data interpolation device 1 by the communication unit 46 and stores the data set in the reception data storage unit 40. The interpolation method instruction device 4 confirms (evaluates) the interpolation result of the missing data by the interpolation result confirmation unit 43 with respect to the data set after the interpolation processing stored in the reception data storage unit 40 (step S35). At this time, the interpolation method instruction device 4 stores, in the interpolation result correspondence table storage unit 42, table information indicating the correspondence between the missing data (missing value) and the interpolated interpolation data (interpolated value).

  The interpolation result confirming unit 43 performs data interpolation on the interpolation result of the missing data based on preset reference information as to whether privacy protection is appropriate or not and improving analysis accuracy. Check (evaluate) If the interpolation processing of the data interpolation apparatus 1 is inappropriate based on the result of the confirmation by the interpolation result confirmation unit 43, the interpolation method instruction apparatus 4 repeats the processing from step S31. That is, the interpolation method instruction device 4 determines (selects) the interpolation method and the interpolation accuracy again, and causes the data interpolation device 1 to execute the interpolation process again.

  When the interpolation result confirmation unit 43 confirms that the interpolation result from the data interpolation device 1 is appropriate, the interpolation method instruction device 4 ends the loop processing (a series of repeated processing S36) from step S31. The interpolation method instruction device 4 notifies the data interpolation device 1 of the completion of the interpolation processing via the communication unit 46 and the network 2 (step S37). Upon receiving the interpolation processing completion notification from the interpolation method instruction apparatus 4, the data interpolation apparatus 1 sends the data set (700 in FIG. 7) after the completion of the interpolation processing to the data analysis apparatus 3 (step S38). The data analysis device 3 executes the same data analysis processing as in the first embodiment (see FIG. 8) (step S39).

  As described above, according to the present embodiment, it is possible to change the interpolation method and the interpolation accuracy when performing the interpolation processing of the data interpolation device 1 while checking the interpolation result of the data interpolation device 1. That is, by feeding back the interpolation method and the interpolation accuracy based on the interpolation result from the data interpolation device 1, the interpolation process of the data interpolation device 1 can be appropriately performed as required. Therefore, it is possible to realize an appropriate interpolation process in consideration of the privacy of the data provider or the accuracy of the data analysis result.

  Also in the present embodiment, at the time of the data interpolation process in step S33, the data interpolation device 1 gives the certainty factor 102 by the certainty factor adding unit 21 at the time of the interpolation process, as in the third embodiment described above. You may. In this case, the data interpolation device 1 transmits the provision data set (see FIG. 18) assigned by the certainty factor 102 to the interpolation method instruction device 4 or the data analysis device 3 via the communication unit 19 and the network 2. Send to

  Thus, the interpolation method instruction device 4 can use the certainty factor 102 together with the interpolation result to check the interpolation result of the data interpolation device 1 and reselect the interpolation method and the interpolation accuracy. Further, in the data analysis device 3, as in the third embodiment described above, it is possible to use the certainty factor 102 as a weight relating to data reliability, and as a result, to improve the accuracy of the analysis result. Becomes possible.

  Furthermore, in the present embodiment, a data set similar to that of the above-described second embodiment (see FIGS. 12, 13, 14, and 15) may be used. Further, a data set similar to that of the third embodiment (see FIGS. 17 and 18) may be used.

  In this embodiment, the data interpolation device 1 and the interpolation method instruction device 4 have been described as a system connected via the network 2. However, the present invention is not limited to this. The data interpolation device 1 may be used.

  Although several embodiments of the present invention have been described, these embodiments are provided by way of example and are not intended to limit the scope of the invention. These new embodiments can be implemented in other various forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and their equivalents.

1 ... data interpolation device, 2 ... network, 3 ... data analysis device,
4 ... Interpolation method instruction device, 10 ... Data storage unit, 11 ... Data division unit,
12: missing data storage unit, 13: non-missing data storage unit, 14: data interpolation processing unit,
15: data interpolation selection unit, 16: interpolation result correspondence table storage unit,
17: Interpolated data storage unit, 18: Provided data storage unit, 19: Communication unit, 20: Control unit,
21: confidence imparting unit, 30: data storage unit, 31: data analysis unit, 32: communication unit,
33: control unit, 40: received data storage unit, 41: data storage unit
42 ... interpolation result correspondence table storage unit; 43 ... interpolation result confirmation unit
44: data interpolation method selection unit, 45: data interpolation method / accuracy storage unit, 46: communication unit,
47 Control part.

Claims (9)

Data storage means for storing data;
Data dividing means for dividing the data stored in the data storage means into data including loss and data having no loss,
Data interpolation processing means for performing an interpolation process on the lost data according to the opt-out based on an interpolation accuracy different from the interpolation accuracy for general loss,
A data interpolating device comprising: a control unit that creates data for provision by combining the interpolated data interpolated by the data interpolation processing unit and the data having no loss. In the data stored in the data storage means, an opt-out flag indicating data lost due to the opt-out is set,
2. The data interpolation device according to claim 1, further comprising a data interpolation selection unit that selects an interpolation processing method by the data interpolation processing unit based on the opt-out flag.   The data interpolation apparatus according to claim 1, further comprising a certainty factor assigning unit that assigns data indicating a certainty factor corresponding to the interpolation accuracy in accordance with the interpolation data interpolated by the data interpolation processing unit.   4. The data interpolation apparatus according to claim 1, further comprising a unit configured to change an interpolation method by the data interpolation unit and the interpolation accuracy based on an interpolation result by the data interpolation unit. 5.   The data interpolation device according to any one of claims 1 to 4, further comprising a transmission unit that transmits the data for provision created by the control unit to a data analysis device that executes a data analysis process. A program executed by a computer incorporated in the data interpolation device,
A process of dividing the data stored in the data storage means into data including loss and data having no loss,
A process of performing an interpolation process on missing data according to opt-out based on an interpolation accuracy different from an interpolation accuracy for general loss;
A program for causing the computer to execute a procedure including a process of creating data for provision by combining the interpolated data interpolated by the interpolation process and the data having no loss. In the data stored in the data storage means, an opt-out flag indicating data lost due to the opt-out is set,
The program according to claim 6, further comprising a process of selecting a method of the interpolation process by the interpolation process based on the opt-out flag.   The program according to claim 6, further comprising a process of adding data indicating a certainty factor corresponding to the interpolation accuracy, corresponding to the interpolation data interpolated by the interpolation process.   The program according to any one of claims 6 to 8, further comprising a process of changing the method of the interpolation process by the interpolation process and the interpolation accuracy based on an interpolation result of the interpolation process.