JP5648008B2 - Document classification method, apparatus, and program - Google Patents

Description

  The present invention relates to a document classification method, apparatus, and program, and more particularly to a document classification method, apparatus, and program for classifying to which category a page document of a website belongs.

  A classification method for classifying a document using n-grams of words in the document is known (for example, Patent Document 1). In this classification method, a model is created by learning n-grams of words that are likely to appear in a specific category.

Koichi Hirano, Kiya Kobayashi, Junichi Takahashi, "Automatic classification of Japanese-speaking blogs", IPSJ Research Report, 2005

  However, when document classification is performed using n-grams of all the words in the document as clues, a large number of words that are not related to the body category are included, resulting in noise. As a result, there is a problem that the classification performance is degraded.

  For example, the text shown in FIG. 5 is the text in the document of the “One Piece” category, but it is related to “One Piece” such as “Necklace”, “Belt”, “Accessory”, “Shorts”, but other category names. And many words that are strongly related to other categories. Since many such words appear in the text, it causes a classification error and the classification performance deteriorates.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a document classification method, apparatus, and program that can accurately classify page documents on a Web site.

In order to achieve the above object, a document classification method according to the present invention includes:
Information about each page that is passed from the page document of the classification target page of the website including a plurality of pages to the classification target page by the site structure extraction unit, or each of the information that is reached via the classification target page Extracting site structure information representing information about the page;
Performing morpheme analysis on the site structure information extracted by the site structure extraction unit by a morpheme analysis unit;
The feature extraction means represents the notation of a plurality of morphemes excluding the morpheme of the symbol connecting the noun phrase from the plurality of morphemes obtained from the morpheme analysis result by the morpheme analysis means, and the appearance order of the plurality of morphemes Extracting a feature;
A classification model for classifying whether a page document belongs to any of a plurality of categories, which has been machine-learned in advance from the learning document by using the classification means, and extracted from the site structure information by the feature extraction means The plurality of page documents of the classification target page based on the feature representing the notation of the plurality of morphemes excluding the morpheme of the symbol connecting the noun phrases and the appearance order of the plurality of morphemes. Categorizing which of the categories belong to,
including.

The document classification apparatus according to the present invention is:
A site representing information related to each page that passes from the page document of the classification target page of the Web site including a plurality of pages until reaching the classification target page, or information related to each page reached via the classification target page Site structure extraction means for extracting structure information;
Morphological analysis means for performing morphological analysis on the site structure information extracted by the site structure extraction means;
A feature that extracts features representing notation of a plurality of morphemes excluding morphemes of symbols connecting noun phrases and appearance order of the plurality of morphemes from a plurality of morphemes obtained from a result of morpheme analysis by the morpheme analysis means Extraction means;
Pre machine learning from the learning document using the identity, a classification model for the page document to classify whether belonging to any of a plurality of categories, the at feature extracted from the site structure information said feature extracting means Yes, the page document of the page to be classified is one of the plurality of categories based on the notation of a plurality of morphemes excluding the morpheme of the symbol connecting the noun phrases and the feature indicating the appearance order of the plurality of morphemes. A classification means for classifying whether or not
It is comprised including.

The program according to the present invention is:
Computer
A site representing information related to each page that passes from the page document of the classification target page of the Web site including a plurality of pages until reaching the classification target page, or information related to each page reached via the classification target page Site structure extraction means for extracting structure information;
Morphological analysis means for performing morphological analysis on the site structure information extracted by the site structure extraction means;
A feature that extracts features representing notation of a plurality of morphemes excluding morphemes of symbols connecting noun phrases and appearance order of the plurality of morphemes from a plurality of morphemes obtained from a result of morpheme analysis by the morpheme analysis means Extraction means, and
Pre machine learning from the learning document using the identity, a classification model for the page document to classify whether belonging to any of a plurality of categories, the at feature extracted from the site structure information said feature extracting means Yes, the page document of the page to be classified is one of the plurality of categories based on the notation of a plurality of morphemes excluding the morpheme of the symbol connecting the noun phrases and the feature indicating the appearance order of the plurality of morphemes. It is a program for causing it to function as a classification means for classifying whether it belongs to.

  As described above, according to the document classification method, apparatus, and program of the present invention, features are extracted from site structure information extracted from a page document of a website, and the page document belongs to any of a plurality of categories. By classifying these, it is possible to obtain an effect that the page documents of the website can be classified with high accuracy.

It is the schematic which shows the structure of the page document classification | category apparatus which concerns on embodiment of this invention. It is a figure which shows the page document input. It is a figure which shows the example of a breadcrumb list. It is a flowchart which shows the content of the document classification processing routine in the page document classification device based on Embodiment of this invention. It is a figure which shows the example of a text.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<System configuration>
The page document classification apparatus 100 according to the embodiment of the present invention receives a page document that is an HTML text of a Web site, and displays a classification result to which category belongs among a plurality of categories prepared in advance by hand. Output. This page document classification apparatus 100 is configured by a computer including a CPU, a RAM, and a ROM storing a program for executing a document classification processing routine described later, and is functionally configured as follows. ing. As shown in FIG. 1, the page document classification device 100 includes an input unit 10, a calculation unit 20, and an output unit 30.

  The input unit 10 receives a page document that is input as a classification target and is HTML text of a page on a Web site composed of a plurality of pages. For example, HTML text as shown in FIG. 2 is input as a page document to be classified.

  The calculation unit 20 includes a text analysis unit 21, a site structure extraction unit 22, a morpheme analysis unit 23, a feature extraction unit 24, a model storage unit 25, and a classification unit 26.

  The text analysis unit 21 analyzes the HTML text of the input page document, and cuts out the site structure, text, title, and the like.

  The site structure extraction unit 22 extracts site structure information in the HTML text of the input page document based on the analysis result of the text analysis unit 21. The site structure information represents information related to each page that passes through to reach the classification target page, and is acquired from, for example, a breadcrumb list portion in the HTML text.

  The product page has site structure information, and based on the site structure information, the information on the page that passes through from the top page of the website to the product page is used. Since many pages related to the product go through from the top page of the website to the page of the product, keywords related to the product category are obtained from the site structure information. Can do. In this embodiment, a case where a breadcrumb list is used as site structure information will be described as an example.

  For example, the site structure extraction unit 22 extracts a bread crumb list as shown in FIG. 3 as site structure information. In the breadcrumb list, as in the example of FIG. 3 described above, information about each page that passes from the top page of the site to the page is described.

  The morpheme analysis unit 23 divides the text representing the site structure information extracted by the site structure extraction unit 22 into words by morpheme analysis, which is an existing technology, and further gives a part of speech to each word and outputs it. For example, if the text of the site structure information “Top> Bag / Wallet> Tote Bag” is entered, the result of morphological analysis is “Top (noun) / (blank) /> (Symbol) / (blank) / 鞄 ( Noun) / ・ (Symbol) / Wallet (noun) ... ”.

  The feature extraction unit 24 includes a plurality of morphemes obtained by removing the morphemes of symbols connecting noun phrases from a plurality of morphemes obtained from the result of morphological analysis of the site structure information (breadcrumb list) extracted by the site structure extraction unit 22. And a feature representing the appearance order of the plurality of morphemes.

  For example, “(site top (female decorative accessory (handbag)))” is extracted from the site structure “site top> female decorative accessory> handbag”. In addition, if there is a site structure such as “Home Appliance> Vacuum Cleaner> Product A”, if the feature shows that it has appeared in the order of “Home Appliance → Vacuum Cleaner → Product A”, it is another feature. May be.

  The model storage unit 25 stores a classification model for each of a plurality of categories prepared in advance. The classification model stored in the model storage unit 25 is learned by a learning unit (not shown) for each of a plurality of categories.

  For each category, the learning unit acquires positive example learning data (features obtained from page documents belonging to the category) and negative example learning data (pages not belonging to the category) obtained from the page document group that is a learning document. A classification model for classifying whether or not the input page document belongs to the category is created by machine learning using a feature obtained from the document and stored in the model storage unit 25. For example, if it is desired to classify into 10 categories, a classification model is created for each of the 10 categories. As the machine learning algorithm, for example, an algorithm such as support vector machine (SVM) or Markov Logic Network (MLN) can be used.

  The classification unit 26 receives the input page document based on the classification model of each category created by the learning device and the features extracted by the feature extraction unit 24 according to “one-versus-rest” which is a well-known technique. For each page, the category to which the page document belongs is output. Specifically, score values are calculated using each of the 10 classification models created for 10 categories, and the category with the highest score value among the categories is the category to which the page document belongs. And

  Note that a learning model may be used to create a classification model for performing multi-level classification, and for each input page document, multi-level classification may be performed based on the features extracted by the feature extraction unit 24. .

  As a classification result by the classification unit 26, the input page document and category pair is output from the output unit 30.

<Operation of page document classification device>
Next, the operation of the page document classification apparatus 100 according to this embodiment will be described. First, a document group which is a plurality of page documents as learning documents obtained from a website, and teacher information indicating which of the plurality of categories prepared in advance each of the plurality of page documents is a page. When input to the document classification device 100, a classification model is created for each category by the learning unit of the page document classification device 100 and stored in the model storage unit 25.

  Then, when a page document to be classified on a website composed of a plurality of pages is input to the page document classification device 100, the page document classification device 100 executes a document classification processing routine shown in FIG.

  First, in step S101, a page document input by the input unit 10 is received. In step S102, the text analysis unit 21 performs text analysis on the page document input in step S101. In step S103, the site structure extraction unit 22 extracts the site structure from the page document based on the analysis result in step 102.

  In step S104, the morpheme analysis unit 23 performs morpheme analysis on the site structure text extracted in step S103. In the next step S105, the feature extraction unit 24 extracts features based on the morphological analysis result of the site structure in step S103. In step S106, the classification unit 26 classifies the category to which the page document belongs based on the features extracted in step S105 and the classification model of each category stored in the model storage unit 25. In step S107, the classification result in step S106 and the page document are output by the output unit 30, and the document classification processing routine is terminated.

  As described above, according to the page document classification apparatus according to the present embodiment, a feature is extracted from site structure information extracted from a page document of a Web site, and a page document belongs to which of a plurality of categories. By classifying, the page documents of the website can be classified with high accuracy.

  In the prior art, from the body of the HTML text of the page document (DESCRIPTION), for example, “(Burberry Blossom (no (runware (also (appeared (and (bag (no (mini size (but (appeared (.))) )))))))) ... ”was extracted, but in this embodiment, focusing on the site structure (TOPIC_PASS), for example,“ (site top (women's decorative accessory (handbag))) ” By extracting features such as “”, learning is performed with less noise, and thus the classification performance can be improved.

  The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  For example, the site structure information is not limited to information about each page that passes through to reach the classification target page, but is site structure information that represents information about each page that arrives through the classification target page May be extracted. If the HTML text has no breadcrumb list, the link structure included in the THML text of the page document is extracted, the site structure information is created using the title of each linked page as a topic, and the created site structure information is used. You may make it extract a feature.

  In the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium.

DESCRIPTION OF SYMBOLS 10 Input part 20 Operation part 21 Text analysis part 22 Site structure extraction part 23 Morphological analysis part 24 Feature extraction part 25 Model storage part 26 Classification part 30 Output part 100 Page document classification device

Claims (3)

Information about each page that is passed from the page document of the classification target page of the website including a plurality of pages to the classification target page by the site structure extraction unit, or each of the information that is reached via the classification target page Extracting site structure information representing information about the page;
Performing morpheme analysis on the site structure information extracted by the site structure extraction unit by a morpheme analysis unit;
The feature extraction means represents the notation of a plurality of morphemes excluding the morpheme of the symbol connecting the noun phrase from the plurality of morphemes obtained from the morpheme analysis result by the morpheme analysis means, and the appearance order of the plurality of morphemes Extracting a feature;
The classifying means, pre machine learning from the learning document using the feature, a classification model for the page document to classify whether belonging to any of a plurality of categories, the site structure information by said feature extracting means The page document of the classification target page is based on the extracted features and the features representing the appearance of the plurality of morphemes excluding the morphemes of the symbols connecting the noun phrases and the appearance order of the plurality of morphemes. Categorizing which of a plurality of categories it belongs to;
Document classification method. A site representing information related to each page that passes from the page document of the classification target page of the Web site including a plurality of pages until reaching the classification target page, or information related to each page reached via the classification target page Site structure extraction means for extracting structure information;
Morphological analysis means for performing morphological analysis on the site structure information extracted by the site structure extraction means;
A feature that extracts features representing notation of a plurality of morphemes excluding morphemes of symbols connecting noun phrases and appearance order of the plurality of morphemes from a plurality of morphemes obtained from a result of morpheme analysis by the morpheme analysis means Extraction means;
Pre machine learning from the learning document using the feature, a classification model for the page document to classify whether belonging to any of a plurality of categories, the feature extracted from the site structure information by said feature extracting means The page document of the classification target page is based on the features representing the notation of a plurality of morphemes excluding the morpheme of the symbol connecting the noun phrases and the appearance order of the plurality of morphemes. A classification means for classifying to which one belongs,
Document classification device including Computer
A site representing information related to each page that passes from the page document of the classification target page of the Web site including a plurality of pages until reaching the classification target page, or information related to each page reached via the classification target page Site structure extraction means for extracting structure information;
Morphological analysis means for performing morphological analysis on the site structure information extracted by the site structure extraction means;
A feature that extracts features representing notation of a plurality of morphemes excluding morphemes of symbols connecting noun phrases and appearance order of the plurality of morphemes from a plurality of morphemes obtained from a result of morpheme analysis by the morpheme analysis means extracting means, and pre machine learning using the feature from learning document, a classification model for the page document to classify whether belonging to any of a plurality of categories, the site structure information by said feature extracting means The page document of the classification target page is based on the extracted features and the features representing the appearance of the plurality of morphemes excluding the morphemes of the symbols connecting the noun phrases and the appearance order of the plurality of morphemes. A program for functioning as a classification means for classifying which of a plurality of categories it belongs to.

Images