JP6480377B2 - Classifier learning apparatus, table type classification apparatus, method, and program - Google Patents

Description

  The present invention relates to a classifier learning device, a table type classification device, a method, and a program for classifying table types of tabular data.

  With the development of computer technology, a large amount of tabular data written in HTML on the Web and spreadsheet data created with spreadsheet software etc. has come to exist. There are several types of table data, such as a vertical or horizontal list type table, a vertical or horizontal attribute type table, a vertical or horizontal enumeration type table, a matrix type table, and other layout tables. If this table type can be understood correctly, knowledge applicable to a wide range of services such as information retrieval and question answering can be acquired.

Crestan, Eric and Patrick Pantel (2011). “Web-scale Table Census and Classification”. In: Proceedings of the 4th ACM International Conference on Web Search and Data Mining. ACM, pp. 545-554.

  As a conventional method for classifying table types, Non-Patent Document 1 and the like have been proposed. These conventional methods use features such as text length, text type (numeric, date, money, etc.), text pattern (including specific symbols), etc. for text information contained in cells in the table. However, the list type and matrix type tables having the same structure and text length as shown in FIGS. 7A and 7B cannot be identified.

  As shown in FIG. 7A, the list type table is a table in which attributes (eg, monthly fee) are arranged in the header portion and attribute values (1800 yen) are arranged in the data portion, and the matrix type table is shown in FIG. ), Entities (line XX, option A, etc.) are described in the headers of the rows and columns. According to this definition, the list type table has a high semantic similarity between the header part and the data part, but the matrix type table does not have the feature. If the semantic similarity between the header portion and the data portion can be measured, the identification accuracy between the list type and the data type can be greatly improved.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide a classifier learning device, a table type classification device, a method, and a program that can accurately classify table types of tabular data. And

  In order to achieve the above object, the classifier learning device according to the present invention is the same for each of the tabular data included in the training data set, which is a set of tabular data and a correct answer label representing a table type. A feature extraction unit that extracts a feature set used for classification, including the semantic similarity of the text of each cell in a row and the semantic similarity of the text of each cell in the same column; and the training data by the feature extraction unit Classification for learning a classifier for classifying table types of tabular data based on the feature set extracted for each of the tabular data included in the set and the correct answer label included in the training data set And a learning unit.

  In the classifier learning method according to the present invention, the feature extraction unit uses each of the same row for each of the tabular data included in the training data set that is a set of tabular data and a correct answer label indicating a table type. A feature set used for classification is extracted including the semantic similarity of cell text and the semantic similarity of each cell text in the same column, and a classifier learning unit uses the feature extraction unit to extract the training data set. A classifier for classifying the table type of the tabular data is learned based on the feature set extracted for each of the tabular data included in the table and the correct label included in the training data set.

  The table type classification device according to the present invention includes, for the input tabular data, the semantic similarity of the in-cell text of each cell in the same row and the semantic similarity of the in-cell text of each cell in the same column. , Based on a feature extraction unit that extracts a feature set used for classification, a feature set extracted by the feature extraction unit, and a classifier for classifying table types of pre-learned tabular data, A table type classification unit for classifying the table type of the input tabular data.

  In the table type classification method according to the present invention, the feature extraction unit, for the input tabular data, the semantic similarity of the in-cell text of each cell in the same row and the meaning of the in-cell text of each cell in the same column Classification for classifying the feature set used for classification, including the similarity, and the table type classification unit classifying the feature set extracted by the feature extraction unit and the table type of the tabular data learned in advance The table type of the input tabular data is classified based on the table.

  Moreover, the program of this invention is a program for functioning a computer as each part which comprises said classifier learning apparatus or table kind classification | category apparatus.

  As described above, according to the classifier learning apparatus, method, and program of the present invention, for each of the tabular data included in the training data set, the semantic similarity of the text of each cell in the same row, and By extracting a feature set used for classification including the semantic similarity of the text of each cell in the same column, a classifier capable of improving the classification accuracy of the table type can be learned.

  Further, according to the table type classification apparatus, method, and program of the present invention, for the tabular data, the semantic similarity of the text of each cell in the same row and the semantic similarity of the text of each cell in the same column are determined. By extracting the feature set used for classification, the table types can be classified with high accuracy.

It is a block diagram which shows the functional structure of the classifier learning apparatus which concerns on embodiment of this invention. It is a block diagram which shows the functional structure of a feature extraction part. It is a figure which shows an example of tabular data. It is a block diagram which shows the functional structure of the table type classification device which concerns on embodiment of this invention. It is a flowchart figure of the classifier learning process routine in the classifier learning apparatus which concerns on embodiment of this invention. It is a flowchart figure of the table type classification process routine in the table type classification device which concerns on embodiment of this invention. FIG. 4A is a diagram illustrating an example of horizontal list type tabular data, and FIG. 5B is a diagram illustrating an example of matrix type tabular data.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Configuration of Classifier Learning Device According to Embodiment of the Present Invention>
Next, the configuration of the classifier learning device according to the embodiment of the present invention will be described. As shown in FIG. 1, a classifier learning device 100 according to the present embodiment includes a CPU, a RAM, and a ROM that stores a program and various data for executing a classifier learning processing routine described later. Can be configured with a computer. The classifier learning device 100 is functionally configured to include an input unit 10, an arithmetic unit 20, and an output unit 40 as shown in FIG.

  The input unit 10 accepts a training data set that is a set of tabular data and a correct answer label representing a table type.

  The tabular data is a set of cells in a matrix format, and each cell is composed of a row number, a column number, a text character string, and a header / data flag (1 indicates header, 0 indicates data). The header / data flag is given from designation by th tag in HTML and information estimated by a method described in Non-Patent Document 2 or the like.

[Non-Patent Document 2] Jing Fang, Prasenjit Mitra, Zhi Tang, C. Lee Giles: Table Header Detection and Classification. AAAI 2012

  The table types include, for example, a vertical or horizontal list type table, a vertical or horizontal attribute type table, a vertical or horizontal enumeration type table, a matrix type table, and other layout tables.

  As shown in FIG. 1, the calculation unit 20 includes a feature extraction unit 22 and a classifier learning unit 24.

  For each of the tabular data included in the training data set, the feature extraction unit 22 performs the semantic similarity of the text of each cell in the same row for each row, and the semantic of the text in each cell in the same column for each column. A feature set used for classification including a semantic similarity feature representing similarity is extracted.

  As shown in FIG. 2, the feature extraction unit 22 includes a normal feature extraction unit 30 and a semantic similarity extraction unit 32.

  The normal feature extraction unit 30 extracts a normal feature representing the characteristics of the tabular data for each tabular data included in the training data set. As the normal feature, the feature described in Non-Patent Document 1 or the like can be used.

  The semantic similarity extraction unit 32, for each of the tabular data included in the training data set, for each row, the semantic similarity between the in-cell text of the header cell of the same row and the in-cell text of the data cell, and each A semantic similarity feature representing a semantic similarity between the in-cell text of the header cell and the in-cell text of the data cell is extracted for the column.

  Specifically, for each row (row number i), for the cell set included in the row, first, a header cell set whose header / data flag is 1 cell and data whose header / data flag is 0 cell. Divide into cell sets.

  Next, the similarity between the in-cell text of each header cell included in the header cell set and the in-cell text of each cell included in the data cell set is calculated, and the similarity average value and the similarity variance value are calculated.

  Finally, for each row, for the similarity average value and similarity variance value calculated for each header cell, the maximum value, minimum value, variance value, and average value between the header cells are calculated, and these are calculated for the table type classification. The feature value for the row is used. The feature name is row (line number i) _meaning similarity_ (header-data similarity statistic name) _ (header similarity statistic name). A method such as word2vec described in Non-Patent Document 3 can be used to calculate the similarity between the header cell and the data cell. If the word similarity cannot be measured by word2vec, the similarity is 0. If no header cell exists in the row, all the feature values are set to 0. All the feature values are set to 0 even when no data cell exists in a row.

[Non-Patent Document 3] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of NIPS, 2013.

  For each column (column number j), the cell set included in the column is calculated in the same manner as the cell set included in each row to calculate a feature value for the column in the table type classification. The feature name is column (column number j) _semantic similarity_ (header-data similarity statistic name) _ (header similarity statistic name). Taking the tabular data shown in FIG. 3 as an example, among the feature values, a calculation example in the case where the header-data similarity statistic name is the average value and the header similarity statistic name is the average value is shown below.

Row 1_Meaning similarity_Average value_Average value = 0
Row 2_Meaning similarity_Average value_Average value
= Average {average {similarity (cost, 1800 yen), similarity (cost, 2000 yen)}, average {similarity (annual, 1800 yen), similarity (annual, 2000 yen)}}
= Average {average {0.85, 0.83}, average {0.83, 0.87}} = average {0.84, 0.85} = 0.845
Row 3_Meaning similarity_Average value_Average value
= Average {Average {Similarity (cost, 21600 yen), Similarity (cost, 24000 yen)}, Average {Similarity (monthly, 21600 yen), Similarity (monthly, 24000 yen)}}
= Average {average {0.8, 0.8}, average {0.75, 0.77}} = average {0.8, 0.76} = 0.78
Column 1_Meaning similarity_Average value_Average value = 0
Column 2 _ semantic similarity _ average value _ average value = 0
Column 3_Meaning similarity_Average value_Average value
= Average {Similarity (Option A, 1800 yen), Similarity (Option A, 21600 yen)}
= Average {0.18, 0.12} = 0.15
Column 4_Meaning similarity_Average value_Average value
= Average {Similarity (Option B, 2000 yen), Similarity (Option B, 24000 yen)}
= Average {0.15, 0.15} = 0.15

  The classifier learning unit 24 determines the table type of the tabular data based on the feature set extracted for each of the tabular data included in the training data set by the feature extraction unit 22 and the correct answer label included in the training data set. Learn a classifier to classify.

  Specifically, the classifier learning unit 24 uses the feature set including the normal feature and the semantic similarity feature extracted by the feature extraction unit 22 to learn about the set of the feature set and the correct label, and outputs the output unit. At 40, the classifier is output. As a learning algorithm, any classification algorithm such as a support vector machine or a random forest method can be used.

<Configuration of Table Type Sorting Device According to Embodiment of the Present Invention>
Next, the configuration of the table type classification apparatus according to the embodiment of the present invention will be described. As shown in FIG. 4, a table type classification apparatus 150 according to the present embodiment includes a CPU, a RAM, and a ROM that stores a program and various data for executing a table type classification processing routine described later. Can be configured with a computer. The table type classification device 150 is functionally configured to include an input unit 60, a calculation unit 70, and an output unit 80 as shown in FIG.

  The input unit 60 receives tabular data to be classified.

  As shown in FIG. 4, the calculation unit 70 includes a feature extraction unit 72 and a table type classification unit 74.

  The feature extraction unit 72 extracts feature sets to be used for classification, including semantic similarity features, in the same manner as the feature extraction unit 22 for the input tabular data.

  Similar to the feature extraction unit 22 of the classifier learning device 100, the feature extraction unit 72 includes a normal feature extraction unit 30 and a semantic similarity extraction unit 32.

  The table type classifying unit 74 inputs the table based on the feature set extracted by the feature extracting unit 72 and the classifier for classifying the table type of the tabular data learned by the classifier learning device 100. Classify table type of format data.

<Operation of Classifier Learning Device According to Embodiment of Present Invention>
Next, the operation of the classifier learning device 100 according to the embodiment of the present invention will be described. When the classifier learning apparatus 100 receives a training data set by the input unit 10, the classifier learning apparatus 100 executes a classifier learning processing routine shown in FIG.

  First, in step S100, normal features are extracted for each of the tabular data included in the input training data set.

  In step S102, for each of the tabular data included in the input training data set, for each row, the semantic similarity between the in-cell text in the header cell and the in-cell text in the data cell for each row, and each column The semantic similarity feature representing the semantic similarity between the in-cell text of the header cell and the in-cell text of the data cell is extracted.

  In step S104, based on the feature set including the normal feature and the semantic similarity feature extracted in steps S100 and S102 for each of the tabular data included in the training data set, and the correct label included in the training data set. Thus, the classifier for classifying the table type of the tabular data is learned and output by the output unit 40, and the classifier learning processing routine is terminated.

<Operation of Table Type Sorting Device According to Embodiment of the Present Invention>
Next, the operation of the table type classification device 150 according to the embodiment of the present invention will be described. When the table type classification device 150 receives the tabular data through the input unit 60, the table type classification device 150 executes the table type classification processing routine shown in FIG.

  First, in step S150, normal features are extracted from the input tabular data.

  In step S152, for the input tabular data, for each row, the semantic similarity between the in-cell text of the header cell in the same row and the in-cell text of the data cell for each row, and the cell of the header cell in the same column for each column A semantic similarity feature representing a semantic similarity between the inner text and the inner text of the data cell is extracted.

  In step S154, the table type of the tabular data is changed based on the feature set including the normal feature and the semantic similarity feature extracted in steps S150 and S152 and the classifier learned by the classifier learning device 100. The data is classified and output by the output unit 80, and the table type classification processing routine is terminated.

  As described above, according to the table type classification apparatus according to the embodiment of the present invention, for tabular data, the semantic similarity of the text of each cell in the same row for each row, and the same column for each column Table types can be classified with high accuracy by extracting feature sets used for classification including the semantic similarity of the text of each cell.

  In addition, since the semantic similarity in units of rows and columns can be calculated from the input tabular data, it is possible to classify list tables and matrix tables that could not be identified by conventional features.

  Further, according to the classifier learning device according to the embodiment of the present invention, for each of the tabular data included in the training data set, for each row, the semantic similarity of the text of each cell in the same row, and each A classifier that can improve the classification accuracy of the table type can be learned by extracting a feature set used for classification including the semantic similarity of the text of each cell in the same column.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  For example, in the above embodiment, the case where the classifier learning device and the table type classification device are provided separately has been described as an example. However, the classifier learning device and the table type classification device may be realized by a single device. It may be.

  In the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium or provided via a network. Is also possible.

  The present invention can be used for knowledge acquisition and information retrieval from tabular data whose accuracy can be improved by understanding table types.

10, 60 Input unit 20, 70 Calculation unit 22, 72 Feature extraction unit 24 Classifier learning unit 30 Normal feature extraction unit 32 Semantic similarity extraction unit 40, 80 Output unit 74 Table type classification unit 100 Classifier learning device 150 Table type Classification device

Claims (5)

For each of the tabular data included in the training data set, which is a set of tabular data and a correct label representing a table type , between the in-cell text of the header cell and the in-cell text of the data cell in the same row A feature extraction unit that extracts a feature set used for classification, including semantic similarity and semantic similarity between the in-cell text of the header cell and the data cell in the same column;
Based on the feature set extracted for each of the tabular data included in the training data set by the feature extraction unit and the correct label included in the training data set, the table type of the tabular data is set to vertical Direction list type table, horizontal list type table, vertical attribute type table, horizontal attribute type table, vertical enumeration type table, horizontal enumeration type table, matrix type table, and other layouts A classifier learning unit for learning a classifier for classifying into any of the tables ;
A classifier learning device characterized by comprising: For input tabular data, the semantic similarity between the in-cell text in the header cell and the data cell in the same row, and the in-cell text in the header cell and the data cell in the same column A feature extraction unit that extracts a feature set to be used for classification, including semantic similarity between them;
Based on the feature set extracted by the feature extraction unit and a classifier for classifying the table type of the tabular data learned in advance, the table type of the input tabular data is displayed in a vertical list. Any of type table, horizontal list type table, vertical attribute type table, horizontal attribute type table, vertical enumeration type table, horizontal enumeration type table, matrix type table, and other layout tables and table type classification unit for crab classification,
A table type classification apparatus characterized by comprising: For each of the tabular data included in the training data set, in which the feature extraction unit is a set of tabular data and a correct answer label indicating a table type , the in-cell text of the header cell and the data cell in the same row semantic similarity between the text, and a semantic similarity between the cells in the text in the cell text and data cell header cell in the same column, extracting the feature set to be used for classification,
Based on the feature set extracted by the feature extraction unit for each of the tabular data included in the training data set by the feature extraction unit and the correct label included in the training data set, the tabular data The table types are : vertical list type table, horizontal list type table, vertical attribute type table, horizontal attribute type table, vertical enumeration type table, horizontal enumeration type table, matrix type table And a classifier learning method for learning a classifier for classifying into any one of the layout tables . The feature extraction unit, for the input tabular data, the semantic similarity between the in-cell text of the header cell in the same row and the in-cell text of the data cell, and the in-cell text and data cell of the header cell in the same column Extract feature sets used for classification, including semantic similarity with in-cell text ,
The table type classification unit has the feature type extracted by the feature extraction unit and the table type of the input tabular data based on a classifier for classifying the table type of the tabular data learned in advance. Vertical list type table, horizontal list type table, vertical attribute type table, horizontal attribute type table, vertical enumeration type table, horizontal enumeration type table, matrix type table, and others A table type classification method characterized by classifying into any of the layout tables . Computer, classifier learning device according to claim 1, or a program to function as each section of the table type classification apparatus according to claim 2.