JP7235269B2 - Data item name estimation device, data item name estimation program, and data item name estimation method - Google Patents

Description

The present invention relates to a data item name estimation device and a learning model generation method for estimating data item names of tables whose data item names are unknown, and also to a data item estimation program for realizing these.

In recent years, with the development of ICT (Information and Communication Technology), it has become possible and easy to acquire a wide variety of information in real time, and a large amount of information is collected and accumulated. When performing various analyzes and predictions using such information, it is necessary to integrate a plurality of databases in which such information is stored and create a database having a common schema.

Database consolidation requires the definition of common classifications or attributes. For this reason, conventionally, a database or data modeling specialist confirms the schema configuration with each database administrator, and manually identifies, extracts, and integrates common data. Therefore, database integration takes a lot of work time.

Also, when performing data integration, if the person who performs the work understands up to the schema structure for each data to be integrated, it is easy to determine which items should be integrated. However, if the person who performs the work does not have sufficient knowledge of the data to be integrated, and the data item name is defined as meaningless item name such as "t1", which item should be integrated? It is very difficult to judge whether it is good or not. Therefore, from this point of view as well, database integration is a time-consuming task.

In order to solve such problems, for example, Patent Document 1 discloses a classification construction support system. The classification construction support system disclosed in Patent Literature 1 first extracts the characteristic value of a data item using the record data of the data item. Next, the classification construction support system classifies each of the plurality of data items into an appropriate classification or attribute by obtaining the degree of similarity between the extracted feature value and a predefined attribute.

JP-A-2006-99236

However, in the classification construction support system disclosed in Patent Document 1, although data items having the same characteristic value can be grouped into one attribute, it is impossible to determine the name of the data item having that attribute. be.

For example, assume that "temperature" is determined as an attribute for classifying a certain data item. In this case, in order to perform data integration, it is appropriate for the data item with the attribute of "temperature" to define any item name of "average temperature", "maximum temperature", or "minimum temperature". It was difficult to identify whether or not there was.

An example of the object of the present invention is to solve the above problem and provide a data item name estimation device, a learning device, and a learning method that can estimate the item name assigned to a data item without requiring knowledge of data registered in a database. An object of the present invention is to provide a model generation method and a data item estimation program.

In order to achieve the above object, a data item name estimating device according to one aspect of the present invention includes:
a receiving unit that receives target data for estimating data item names;
an item name estimation unit that estimates an item name of the target data using the target data as input for a trained model;
with
The learned model is a model trained to output the item name of the learning data from a combination of attributes of the data items of the learning data.
It is characterized by

Further, in order to achieve the above object, a data item name estimation program in one aspect of the present invention includes:
a step of receiving target data for estimating item names of data;
a step of estimating item names of the target data using the target data as input for a trained model;
on the computer, and
The learned model is a model trained to output the item name of the learning data from a combination of attributes of the data items of the learning data.
It is characterized by

Furthermore, in order to achieve the above object, the learning model generation method in one aspect of the present invention includes:
Accepts training data including data item names,
generating a learning model that takes as input a combination of attributes of data items of the received learning data and outputs data item names corresponding to the attributes;
It is characterized by

As described above, according to the present invention, an item name assigned to a data item can be estimated without requiring knowledge of data registered in a database.

FIG. 1 is a block diagram showing a schematic configuration of a data item estimation device according to an embodiment of the present invention. FIG. 2 is a block diagram showing a specific configuration of the data item estimation device according to the embodiment of the present invention. FIG. 3 is a diagram showing an example of a learning table used in the embodiment of the invention. FIG. 4 is a diagram showing an example of a target table to be subjected to data item name estimation processing in the embodiment of the present invention. FIG. 5 is a flow chart showing the operation during learning processing of the data item name estimation device according to the embodiment of the present invention. FIG. 6 is a diagram showing an example of attribute information created in the embodiment of this invention. FIG. 7 is a diagram showing an example of attribute individual item information created in the embodiment of the present invention. FIG. 8 is a diagram showing an example of attribute combination information created in the embodiment of this invention. FIG. 9 is a flowchart showing the operation of the data item name estimation device in the embodiment of the present invention during estimation processing. FIG. 10(a) is a diagram showing an example of feature values extracted from the target table used in the embodiment of the present invention. FIG. 10(b) is a diagram showing an example of the degree of similarity calculated from the feature values shown in FIG. 10(a). FIG. 11(a) is a diagram showing an example of attribute combination information used in the embodiment of the present invention. FIG. 11B is a diagram showing an example of the frequency of appearance of data item names calculated according to the embodiment of this invention. FIG. 12 is a block diagram showing an example of a computer that implements the data item estimation device according to the embodiment of the present invention.

(Outline of invention)
In the present invention, feature values are extracted from the data items in the table whose data item name is to be estimated, the degree of similarity between the extracted feature value and the attribute is calculated, and each data item is assigned to the attribute with the high degree of similarity. Classify. Then, by applying the combination of attributes of the data items that make up the table to be inferred to the correspondence relationship between the combination of attributes of the data items learned in advance and the data item name, it is possible to Estimates whether a valid data item name is given. That is, in the present invention, learning processing and estimation processing are mainly performed. A specific description will be given below.

(Embodiment)
A data item estimation device, a data item estimation method, and a program according to embodiments of the present invention will be described below with reference to FIGS. 1 to 12. FIG.

[Device configuration]
First, the configuration of the data item estimation device according to this embodiment will be described. FIG. 1 is a block diagram showing a schematic configuration of a data item estimation device according to an embodiment of the present invention.

The data item estimation device 100 according to the present embodiment shown in FIG. 1 is a device for estimating the data item name of a table to be processed (hereinafter referred to as "target table"). As shown in FIG. 1 , the data item estimation device 100 includes a learning processing section 10 and an estimation processing section 20 .

The learning processing unit 10 first extracts, for each data item name in the learning table, the feature value of the data to which the data item name is assigned, and compares the extracted feature value with the attribute corresponding to the data item name. define the relationship between Next, the learning processing unit 10 identifies the corresponding data item name for each attribute, learns the appearance frequency of the attribute and the data item name in the learning table, and creates a learning model.

The estimation processing unit 20 first compares the feature value of each data item in the target table with the definition by the learning processing unit 10 to estimate the attribute of each data item. Next, the estimation processing unit 20 applies the estimated attribute of each data item to the learning model to estimate the data item name of the data item of the target table.

As described above, in the present embodiment, a learning table with known data item names is used to generate a learning model for estimating data item names. Data item names in unknown tables are deduced. Therefore, according to the present embodiment, it is possible to estimate the item name assigned to the data item without requiring knowledge of the data registered in the database.

Next, the configuration of data item estimation apparatus 100 according to the present embodiment will be described in more detail with reference to FIGS. 2 to 4. FIG. FIG. 2 is a block diagram showing a specific configuration of the data item estimation device according to the embodiment of the present invention. FIG. 3 is a diagram showing an example of a learning table used in the embodiment of the invention. FIG. 4 is a diagram showing an example of a target table to be subjected to data item name estimation processing in the embodiment of the present invention.

As shown in FIG. 2 , in the present embodiment, data item estimation device 100 includes storage unit 30 in addition to learning processing unit 10 and estimation processing unit 20 . The storage unit 30 stores attribute information 31, attribute individual item information 32, and attribute combination information 33, which will be described later.

Further, as shown in FIG. 2, the learning processing unit 10 includes a learning table reception unit 11, a feature extraction unit 12, an attribute information creation unit 13, an attribute individual item information creation unit 14, and an attribute combination information creation unit 15. and

The learning table reception unit 11 receives a learning table (see FIG. 3) input from the outside, and passes the received learning table to the feature extraction unit 12 . The learning table is data in a table format as shown in FIG. 3, and the format of each record data is XML (Extensible markup language), CSV (Common Separated Value), HTML (Hypertext markup language), or the like. may be of the form

For each data item name in the learning table, the feature extraction unit 12 extracts a feature value from the data of the record to which the data item name is assigned. Characteristic values to be extracted include data type (character type, numeric type), statistical information (average value, variance value), appearance frequency of the same record, and the like (see FIG. 6 described later). Further, as a method for extracting feature values from the table, there is a method disclosed in the above-mentioned Patent Document 1.

The attribute information creation unit 13 creates attribute information 31 (see FIG. 6 described later) that defines the relationship between the feature value extracted by the feature extraction unit 12 and the attribute corresponding to the data item name of the extraction source data, The created attribute information is stored in the storage unit 30 .

Specifically, when an attribute corresponding to the data item name of the extraction source data is externally set for the feature value extracted by the feature extraction unit 12, the attribute information creation unit 13 sets this feature value. Attribute information 31 that associates the attribute with the attribute is created. Also, the attribute setting may be performed manually.

The attribute individual item information creation unit 14 creates attribute individual item information (see FIG. 7) to which a corresponding data item name is assigned for each attribute included in the attribute information created by the attribute information creation unit 13 .

Specifically, when a corresponding data item name is set for each attribute included in the attribute information, the attribute individual item information creation unit 14 creates an attribute and an item to which the input data item name is assigned. Attribute individual item information is created using the item. The attribute individual item information indicates the correspondence between attributes and individual items. Note that an individual object here means an object to which a data item name is assigned. Also, the data item names may be set manually.

The attribute combination information creation unit 15 uses the attribute individual item information to create attribute combination information (see FIG. 8) indicating combinations of attributes in a learning table and corresponding data item names as a learning model. .

Further, as shown in FIG. 2, the estimation processing unit 20 includes a target table reception unit 21, a feature extraction unit 22, an attribute estimation unit 23, an item name estimation unit 24, an estimation result display unit 25, and a result editing unit. a portion 26;

The target table accepting unit 21 accepts an externally input target table (see FIG. 4) and passes the accepted target table to the feature extracting unit 22 . As shown in FIG. 4, the target table is also table format data, similar to the learning table. Also, the format of each record data in the target table may be XML (Extensible markup language), CSV (Common Separated Value), HTML (Hypertext markup language), or any other format.

The feature extraction unit 22 extracts data feature values for each data item from the records of the target table. Characteristic values to be extracted include data type (character type, numeric type), statistical information (average value, variance value), appearance frequency of the same record, and the like. Further, as a method for extracting feature values from the table, there is a method disclosed in the above-mentioned Patent Document 1.

The attribute estimation unit 23 compares the feature values extracted by the feature extraction unit 22 with the attribute information 31 to estimate the attribute of each data item. Specifically, the attribute estimation unit 23 first compares the feature value of each data item with the feature value of each attribute included in the attribute information 31 to calculate the degree of similarity. Then, the attribute estimating unit 23 identifies an attribute that has the highest degree of similarity with the characteristic value of the data item that is the target of estimation, and estimates the identified attribute as the attribute of the data item that is the target of estimation ( See later-described FIGS. 10(a) and (b)).

The item name estimation unit 24 first uses the attributes estimated by the attribute estimation unit 23 to set a combination of attributes. Next, the item name estimation unit 24 uses the attribute combination information 33 and the attribute individual item information 32 to calculate the frequency with which a specific data item name appears for each set combination of attributes. Then, the item name estimation unit 24 estimates the data item name of each data item in the target table based on the calculation result.

Specifically, the item name estimation unit 24 uses the attribute individual item information 32 to identify one or more data item names that may correspond to each attribute for which a combination is set. Then, the item name estimation unit 24 uses the attribute combination information 33 to calculate the appearance frequency (appearance probability) for each possible corresponding data item name for each combination of set attributes. Then, the item name estimation unit 24 identifies the data item name with the highest appearance frequency among the data item names that may correspond as the data item name of the estimated attribute.

The estimation result display unit 25 displays the data item names estimated by the item name estimation unit 24 on the screen of the display device connected to the data item name estimation device 100 or the display device of an external terminal device. The result editing unit 26 corrects the result of estimation by the item name estimating unit 24 when it is determined that the estimated data item name has an error and a corrected data item name is input from the outside. Also, the result editing unit 26 can inform the learning processing unit 10 of the content of correction. In this case, the learning processing unit 10 is caused to reflect the correction content on the attribute information 31 , the attribute individual item information 32 , and the attribute combination information 33 .

[Device operation]
Next, the operation of data item estimation apparatus 100 according to the present embodiment will be described with reference to FIGS. 3 to 10. FIG. Further, in the present embodiment, the data item estimation method is implemented by operating data item estimation apparatus 100 . Therefore, the description of the data item estimation method in the present embodiment is replaced with the description of the operation of data item estimation apparatus 100 below.

First, the learning phase (learning process) will be described with reference to FIGS. 5 to 8. FIG. FIG. 5 is a flow chart showing the operation during learning processing of the data item name estimation device according to the embodiment of the present invention.

As shown in FIG. 5, first, the learning table receiving unit 11 receives a learning table (see FIG. 3) inputted from the outside (step A1). Further, the learning table receiving unit 11 passes the received learning table to the feature extracting unit 12 .

Next, the feature extraction unit 12 selects one of the data items in the received learning table, and extracts a feature value from the data of the record of the selected data item (step A2).

Next, the feature extraction unit 12 determines whether or not feature values have been extracted from all data items (step A3). As a result of the determination, if feature values have not been extracted from all data items, the feature extraction unit 12 executes step A2 again. The unit 13 is instructed to execute step A4.

When it is determined as Yes in step A3, the attribute information creating unit 13 creates attribute information 31 by associating the set attribute with each data item (step A4). In addition, in step A4, attribute setting is performed externally and manually based on the data item name of the extraction source data. FIG. 6 is a diagram showing an example of attribute information created in the embodiment of this invention.

Next, the attribute individual item information creation unit 14 creates attribute individual item information 32 to which a corresponding data item name is assigned for each attribute included in the attribute information created in step A4 (step A5). Also, in step A5, the setting of the data item name is performed manually outside. FIG. 7 is a diagram showing an example of attribute individual item information created in the embodiment of the present invention.

Next, the attribute combination information creation unit 15 uses the attribute individual item information created in step A5 to generate attribute combination information indicating combinations of attributes in the learning table and corresponding data item names as a learning model. 33 is created (step A6). FIG. 8 is a diagram showing an example of attribute combination information created in the embodiment of this invention.

Next, the estimation phase (estimation processing) will be described with reference to FIGS. 9 and 10. FIG. FIG. 9 is a flowchart showing the operation of the data item name estimation device in the embodiment of the present invention during estimation processing.

As shown in FIG. 9, first, the target table receiving unit 21 receives a target table (see FIG. 4) input from the outside (step B1). The target table reception unit 21 also passes the received target table to the feature extraction unit 22 .

Next, as shown in FIG. 10A, the feature extraction unit 22 selects one of the data items of the received target table, and extracts feature values from the data of the record of the selected data item (step B2).

FIG. 10(a) is a diagram showing an example of feature values extracted from the target table used in the embodiment of the present invention. FIG. 10(b) is a diagram showing an example of the degree of similarity calculated from the feature values shown in FIG. 10(a).

Next, the feature extractor 22 determines whether feature values have been extracted from all data items (step B3). As a result of the determination, if feature values have not been extracted from all data items, the feature extraction unit 22 executes step B2 again. 23 is instructed to execute step B4.

The attribute estimating unit 23 selects one data item of the target table when it is determined as Yes in step B3. Then, as shown in FIG. 10B, the attribute estimating unit 23 compares the selected data item feature value with the feature value of each attribute included in the attribute information 31 to calculate the degree of similarity. The attribute with the highest degree is estimated as the attribute of the data item (step B4).

Next, the attribute estimation unit 23 determines whether attributes have been estimated for all data items (step B5). As a result of the determination, if attributes have not been estimated for all data items, the attribute estimation unit 23 executes step B4 again. is instructed to execute step B6.

Next, when it is determined as Yes in step B5, the attribute estimating unit 23 first sets a combination of a plurality of attributes appearing at the same time using the attributes estimated in step B4. Next, as shown in FIGS. 11A and 11B, the item name estimating unit 24 uses the attribute combination information 33 and the attribute individual item information 32 for each set combination of attributes to determine a specific data item name. is calculated (step B6).

FIG. 11(a) is a diagram showing an example of attribute combination information used in the embodiment of the present invention. FIG. 11B is a diagram showing an example of the frequency of appearance of data item names calculated according to the embodiment of this invention. In FIG. 11(b), "estimation target attribute: [attribute combination]" means an attribute to be an estimation target for a data item name in a given attribute combination. For example, "date: [place name, date]" means that the target of data item name estimation is "date" when the combination of attributes is "place name, date".

Further, in the present embodiment, the item name estimating unit 24 creates, for example, an association rule for estimating the data item name of a data item having a certain attribute from a combination of attributes. Furthermore, the item name estimating unit 24 refers to the attribute individual object information 32 to identify an individual object (data item name) having the same attribute as the estimated attribute, and uses the association rule to determine the data Calculate the probability of being selected as an item name.

Further, the creation of association rules can be performed, for example, by using correlation-based algorithms disclosed in Reference 1 or Reference 2 below.
(Reference 1) G. Piatetsky-Shapiro (1991). Discovery, analysis, and presentation of strong rules. In G. Piatetsky-Shapiro and WJ Frawley, editors, Knowledge Discovery in Databases. AAAI/MIT Press, Cambridge, MA.
(Reference 2) R. Agrawal, T. Imielinski, and A. Swami (1993). Mining association rules between sets of items in large databases. In Proceedings of the 1993 ACM SIGMOD International Conference on Management of Data, pages 207-216 .

Next, the item name estimation unit 24 selects one data item of the target table. Then, the item name estimation unit 24 compares the combination of the estimated attribute of the selected data item and other attributes appearing at the same time with the calculation result obtained in step B6, and finds The high data item name is presumed to be the data item name of the selected data item (step B7).

Next, the item name estimation unit 24 determines whether the data item names have been estimated for all data items (step B8). As a result of the determination, if the data item names have not been estimated for all data items, the item name estimation unit 24 executes step B7 again.

On the other hand, when the data item names are estimated for all data items as a result of the determination, the item name estimation unit 24 causes the estimation result display unit 25 to display the result. With this, the processing in the data item name estimation device 100 ends.

Further, in the present embodiment, the item name estimating unit 24 receives the attribute combination information 33 as an input, forms a neural network that outputs a data item name, and uses the formed neural network to estimate the data item name. Also good. Furthermore, the item name estimation unit 24 receives the attribute combination information 33 as input, constructs Bayesian inference for estimating the data item name with the highest appearance rate from the combination of attributes of the data items, and uses the constructed Bayesian inference , the data item name may be estimated.

As described above, in this embodiment, it is possible to estimate the item name given to the data item without requiring knowledge of data registered in the database. Therefore, by using the present embodiment, when data analysis or data integration is performed, data items having the same meaning and which can be integrated can be easily specified without knowledge of data.

[program]
The program in this embodiment may be a program that causes a computer to execute steps A1 to A6 shown in FIG. 5 and steps B1 to B8 shown in FIG. By installing this program in a computer and executing it, data item estimation apparatus 100 and data item estimation method according to the present embodiment can be realized. In this case, a CPU (Central Processing Unit) of the computer functions as a learning processing unit 10 and an estimation processing unit 20 to perform processing.

In addition, in the present embodiment, the storage unit 30 stores data files constituting the attribute information 31, the attribute individual item information 32, and the attribute combination information 33 in a storage device such as a hard disk provided in the computer. realizable.

Also, the program in this embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as either the learning processing unit 10 or the estimation processing unit 20 . Moreover, the storage unit 30 may be constructed on a computer different from the computer that executes the program in the present embodiment.

A computer that realizes data item estimation apparatus 100 by executing the program according to the present embodiment will now be described with reference to FIG. FIG. 12 is a block diagram showing an example of a computer that implements the data item estimation device according to the embodiment of the present invention.

As shown in FIG. 12, computer 110 includes CPU 111 , main memory 112 , storage device 113 , input interface 114 , display controller 115 , data reader/writer 116 and communication interface 117 . These units are connected to each other via a bus 121 so as to be able to communicate with each other.

The CPU 111 expands the programs (codes) of the present embodiment stored in the storage device 113 into the main memory 112 and executes them in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory). Also, the program in the present embodiment is provided in a state stored in computer-readable recording medium 120 . It should be noted that the program in this embodiment may be distributed on the Internet connected via communication interface 117 .

Further, as a specific example of the storage device 113, in addition to a hard disk drive, a semiconductor storage device such as a flash memory can be cited. Input interface 114 mediates data transmission between CPU 111 and input devices 118 such as a keyboard and mouse. The display controller 115 is connected to the display device 119 and controls display on the display device 119 .

Data reader/writer 116 mediates data transmission between CPU 111 and recording medium 120 , reads programs from recording medium 120 , and writes processing results in computer 110 to recording medium 120 . Communication interface 117 mediates data transmission between CPU 111 and other computers.

Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital); magnetic recording media such as flexible disks; An optical recording medium such as a ROM (Compact Disk Read Only Memory) can be used.

Note that data item estimation apparatus 100 according to the present embodiment can also be realized by using hardware corresponding to each part instead of a computer in which a program is installed. Further, the data item estimation device 100 may be partially realized by a program and the rest by hardware.

When performing data integration, even if one does not have knowledge of the data structure and the meaning of the data items regarding the data to be integrated, it is possible to estimate the data structure having the same data items by estimation.

10 learning processing unit 11 learning table reception unit 12 feature extraction unit 13 attribute information creation unit 14 attribute individual item information creation unit 15 attribute combination information creation unit 20 estimation processing unit 21 target table reception unit 22 feature extraction unit 23 attribute estimation unit 24 items name estimation unit 25 estimation result display unit 26 result editing unit 30 storage unit 31 attribute information 32 attribute individual item information 33 attribute combination information 100 data item name estimation device 110 computer 111 CPU
112 main memory 113 storage device 114 input interface 115 display controller 116 data reader/writer 117 communication interface 118 input device 119 display device 120 recording medium 121 bus

Claims (3)

a target table reception unit that receives a target table;
an attribute estimating unit for estimating an attribute for each of a plurality of data items in the received target table;
Using the combination of attributes of the target table as input, and using a neural network that learns the appearance probability of the data item name corresponding to the specific attribute included in each combination of attributes based on a plurality of learning tables. and an item name estimation unit for estimating a data item name corresponding to a specific attribute included in the input attribute combination . receiving a target table;
estimating an attribute for each of a plurality of data items in the received target table;
Using the combination of attributes of the target table as input, and using a neural network that learns the appearance probability of the data item name corresponding to the specific attribute included in each combination of attributes based on a plurality of learning tables. estimating a data item name corresponding to a specific attribute included in the input combination of attributes ,
A data item name estimation program that causes a computer to execute Accepts the target table,
estimating an attribute for each of a plurality of data items in the received target table;
Using the combination of attributes of the target table as input, and using a neural network that learns the appearance probability of the data item name corresponding to the specific attribute included in each combination of attributes based on a plurality of learning tables. to estimate a data item name corresponding to a specific attribute included in the input combination of attributes ,
Data item name estimation method.

Images