JP5214985B2 - Text segmentation apparatus and method, program, and computer-readable recording medium - Google Patents

Description

  The present invention relates to a text segmentation apparatus and method using a Web search, a program, and a computer-readable recording medium. In particular, in a field where text is used on a computer, the text is included in text (sentences such as various articles and stories). The present invention relates to a text segmentation apparatus and method using a Web search that estimates the semantic content of each sentence and divides text into semantic groups, and a computer-readable recording medium.

  In recent years, with the rapid improvement of computer performance, it has become possible to accumulate a huge amount of text (here, a set of sentences composed only of character strings) and construct a database. However, it is generally difficult to manually organize and manage stored text. Therefore, a technique called text segmentation has been developed that analyzes the stored text database and divides the text according to semantic content (called semantic paragraphs), and automatically classifies and organizes the text database with a computer. It is being applied to. For example, there is a technique for performing text segmentation using information called a concept base (see, for example, Patent Document 1).

  In this technique, a plurality of concept vectors in which a certain word and a co-occurrence pattern are converted into numerical vectors are created from learning data stored in advance. Then, text segmentation is performed using a concept base that is a collection of concept vectors. In the learning data, many texts related to one field (for example, only about the field of “politics”) are accumulated.

In addition, the conventional text segmentation mainly uses a method of evaluating semantic continuity between sentences based on the degree of connectivity between a plurality of sentences (for example, see Non-Patent Document 1). In this conventional technique, when the number of sentences to be considered when calculating the degree of connectivity is small, the possibility of excessively estimating semantic paragraphs increases instead of easily following local changes in semantic content. On the other hand, when the number of sentences to be considered is large, it is difficult to deal with text whose meaning and contents change gently instead of being able to capture global changes in meaning and contents.
JP 2002-342324 A Hearst, MA ,: Multi-Paragraph Segmention of Expository Text, 32nd Annual Meeting of the Association for Computational Linguistics, pp. 9-16 (1994)

  However, in order to improve the accuracy of the conventional text segmentation technique, large-scale learning data must be prepared. For this reason, when the learning data is small, the concept base cannot be appropriately created, and there is a problem that the accuracy of text segmentation is lowered. In addition, while it is possible to correspond to the fields included in the learning data prepared in advance, text segmentation cannot be performed on texts in different fields. For example, when only information related to “politics” and “economy” is accumulated in the learning data, text segmentation becomes difficult for text in the field of “sports”.

  The present invention has been made in view of the above points, and provides a text segmentation apparatus and method using a Web search capable of text segmentation without requiring learning data, a program, and a computer-readable recording medium. Objective.

  FIG. 1 is a principle configuration diagram of the present invention.

The present invention (Claim 1) is a text segmentation device that divides text according to content,
Text decomposing means 201 for dividing the input text into sentence units;
A sentence divided by the text decomposition means 201 is subjected to morphological analysis , and the nouns, adverbs, verbs, adjectives and adjective verbs that have been subjected to morphological analysis are extracted as search words, and the verb, the adjective, and the adjective verb are not final A search term extraction means 211 for converting into a closed form and storing it in the search term storage means 212;
Web search based on the search term, morphological analysis of the searched text , noun, adverb, verb, adjective, adjective verb among the analyzed morphemes are acquired as related words, the verb, the adjective, the A related word acquisition means 221 that converts an adjective verb that is not a terminal form into a terminal form and stores it in the related word storage means 222;
A plurality of sentences obtained by acquiring a search word from the search word storage unit 212, acquiring a related word from the related word storage unit 222, and dividing an input text using a keyword set that is a combination of the search word and the related word Connectivity determination means 231 for determining the connectivity between each other and dividing the input text by extracting a semantic paragraph that is a sentence between the valleys of the connectivity;
Have
The connectivity determination means 231
Create blocks B1 and B2 that summarize the keyword set, and use the frequency of occurrence of the word t to determine the connectivity of the i-th and i + 1-th two sentences.

(W _t ^B1 represents the frequency of the word t in the block B1, and w _t ^B2 represents the frequency of the word t in the block B2.

Takes a value between 0 and 1, and the closer to 1, the more the same words are included in block B1 and block B2)
Means to obtain
i = {1, 2,..., N},

And the block size b parameter is set to b = (b ₁ , b ₂ ,..., B _M ), and the connectivity is set for each block width.

And the average value thereof as the average connectivity C _i in the i-th and i + 1-th sentences,

Means to obtain
The average connectivity C _i (where i = (1, 2,..., N)) is used to define the average connectivity valley that is the boundary of the semantic paragraph as a condition.

And a means for obtaining a semantic paragraph based on the valley.

Further, the present invention (Claim 2 ) provides connectivity determination means 231
Means for calculating the average connectivity C _i using a plurality of thresholds C _T and extracting semantic paragraphs.

  FIG. 2 is a diagram for explaining the principle of the present invention.

The present invention (Claim 3) is a text segmentation method in an apparatus for dividing text according to content,
A text decomposition step (step 1) in which the text decomposition means divides the input text into sentence units;
The search word extracting means performs morphological analysis on the sentence divided in the text decomposition step, extracts nouns, adverbs, verbs, adjectives, and adjective verbs as morphological analysis as search words, and the verb, the adjective, and the adjective verb A search word extraction step (step 2) of converting a non-end form into a stop form and storing it in the search word storage means;
The related word acquisition means searches the web based on the search word, performs morphological analysis on the searched text, acquires nouns, adverbs, verbs, adjectives, and adjective verbs that are morphologically analyzed as related words, and the verb and the adjectives A related word acquisition step (step 3) of converting the non-terminal form of the adjective verb into a terminal form and storing it in the related word storage means;
The connectivity determination means acquires a search word from the search word storage means, acquires a related word from the related word storage means, and divides the input text using a keyword set that is a combination of the search word and the related word The connectivity between the plurality of sentences is determined (step 4), and if a valley of the average connectivity is detected (step 5), the meaning paragraph that is the sentences between the valleys of the connectivity A connectivity determination step of dividing the input text by extracting (step 6);
And
In the connectivity determination step,
Create blocks B1 and B2 that summarize the keyword set, and use the frequency of occurrence of the word t to determine the connectivity of the i-th and i + 1-th two sentences.

(W _t ^B1 represents the frequency of the word t in the block B1, and w _t ^B2 represents the frequency of the word t in the block B2.

Takes a value between 0 and 1, and the closer to 1, the more the same words are included in block B1 and block B2)
A step to obtain by
i = {1, 2,..., N},

And the block size b parameter is set to b = (b ₁ , b ₂ ,..., B _M ), and the connectivity is set for each block width.

And the average value thereof as the average connectivity C _i in the i-th and i + 1-th sentences,

A step to obtain by
The average connectivity C _i (where i = (1, 2,..., N)) is used to define the average connectivity valley that is the boundary of the semantic paragraph as a condition.

And extracting a semantic paragraph based on the valley.

Further, the present invention (Claim 4 ) is the connectivity determination step (Step 3).
Calculating the average degree of coupling C _i using a plurality of threshold C _T, it extracts the meaning paragraph.

The present invention (Claim 5 ) is a text segmentation program that causes a computer to function as each means constituting the text segmentation device according to Claim 1 or 2 .

The present invention (Claim 6 ) is a computer-readable recording medium in which the text segmentation program according to Claim 5 is stored.

  As described above, according to the present invention, it is possible to expect a text segmentation technique that does not require preparation of learning data in advance by using the concept of searching on the Web. This description can be applied as a support measure for automatically organizing and updating databases in the field of handling enormous text data and the field of distributing news articles. In addition, there is an advantage that there are few restrictions on the description content and the creation time with respect to the text to be analyzed, in that related words in a wide range of fields can be collected by Web search without using learning data.

  Furthermore, since the contents are divided into contents, it can be used as a technique for collecting only contents-related sentences including a certain keyword.

  In the present invention, nouns, adverbs, adjectives, adjective verbs, and verb end forms are used as search words, so that it is possible to deal with a wide variety of text contents and writing methods such as news articles and blog articles. In the present invention, all parts of speech obtained by morphological analysis can also be used as words that can be used as search terms. Since there are only two types of words that can be used or not used, the words without usage are used as they are, and all the words with usage are converted into ending forms, so that all words of part of speech can be searched. Available as

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

The present invention uses a new text segmentation technique that focuses on the concept of searching on the Web. Currently, a huge amount of information is accumulated on the Web, and the latest topics are always provided. That is, the Web can be regarded as a set of articles having various information. In fact,
When we investigate a certain thing, we input a search word at a search site and perform a search on the Web to check the meaning of the word and the contents of things. From this point of view, it can be said that the concept of “sports” and “ball” can be acquired to support “soccer” and “baseball” if information on the Web is appropriately used without using learning data. That is, it is possible to divide into semantic paragraphs by acquiring words according to the contents of the text based on various information on the Web, and tracking the relationship between sentences by the change of words.

[First Embodiment]
FIG. 3 shows a configuration of a text segmentation system using web search according to the first embodiment of the present invention.

  The system shown in the figure includes a computer 251, a network 252, a web 253, and a display unit 256. A network 252 is connected to the computer 251 and can access the web 253. The web 253 stores a plurality of articles 254 described in a structured language such as HTML or XML.

  The computer 251 includes a text decomposition processing unit 201, a decomposed sentence storage unit 202, a search word extraction processing unit 211, a search word storage unit 212, a related word acquisition processing unit 221, a related word storage unit 222, a connectivity determination processing unit 231, and a meaning. A paragraph storage unit 232, an input unit 241, and an output unit 242 are included. Text 255 is text that is input to the input unit 241 of the computer 251. The display unit 256 is a device for displaying the result output from the control unit 240 through the output unit 242.

  FIG. 4 is a flowchart showing an outline of a processing procedure in the first embodiment of the present invention.

  First, text is input to the input unit 241 of the computer 251 (step 101), the text input by the text division processing unit 201 is divided into sentence units (step 102), and the search word extraction processing unit 211 converts the search word from the sentence. (Step 103), the related word acquisition processing unit 221 searches the web using the search word, and acquires the related word from the search result (step 104). In the connectivity determination processing unit 231, a semantic paragraph is extracted from a keyword set in which a search word and a related word are paired (step 105). Finally, the text segmentation result is output from the output unit 242 (step 106).

  FIG. 5 shows an example of text in the first embodiment of the present invention.

  In the figure, text 255 is an example of text input to the input unit 241.

  FIG. 6 shows an example of a sentence stored in the decomposed sentence storage unit in the first embodiment of the present invention. When the text 255 is input from the input unit 241, the text is decomposed into a plurality of sentences by the text decomposition processing unit 201 and stored in the decomposed sentence storage unit 202.

  FIG. 7 is an example of general words registered in the general word list according to the first embodiment of this invention. The general word list is stored in a memory (not shown). In the figure, k is a word registered in the general word list, and is referred to by the search word extraction processing unit 211 and the related word acquisition processing unit 221. Is done.

  FIG. 8 shows an example of a search word stored in the search word storage unit in the first embodiment of the present invention. In FIG. 6, for example, “drive highway” is extracted from the first decomposed text of FIG. 6 by the search word extraction processing unit 211 and stored in the search word storage unit 212.

  FIG. 9 shows an example of related words stored in the related word storage unit according to the first embodiment of the present invention. The related word acquisition processing unit 221 extracts a related word corresponding to the search word acquired from the search word storage unit 212 from words other than those registered in a general word list of a memory (not shown), and The related word storage unit 212 stores the related word corresponding to the search word.

  FIG. 10 shows an example of a keyword set created by the connectivity determination processing unit in the first embodiment of the present invention. The keyword set shown in the figure is stored in a memory (not shown) in the connectivity determination processing unit 231. The connectivity determination processing unit 231 generates a keyword set from a set of search terms and related terms and stores the keyword set in the semantic paragraph storage unit 232. For example, in the example of FIG. 9, the search word “golf, shot, work, strike, ballistic, practice” in FIG. 8 and the related word “hit, ball, score, swing, FIG. 9 corresponding to the search word” "Golf, shot, work, untouched, ballistic, practice, hit, ball, score, swing" from the set of "flying distance, buy, round, driver, beginner, receive, purchase, feel, iron, apply" , Flight distance, buy, round, driver, beginner, receive, purchase, feel, iron, call, and the like, and these are stored in a memory (not shown).

  FIG. 11 shows an example of the semantic paragraph stored in the semantic paragraph storage unit in the first embodiment of the present invention. In the figure, for each semantic paragraph number corresponding to a certain threshold, the number of a sentence belonging to the semantic paragraph detected by the connectivity determination processing unit 231 is stored.

  Hereinafter, the processing procedure of text segmentation in the above configuration will be specifically described.

  FIG. 12 is a flowchart of the process of the related word acquisition processing unit in the first embodiment of the present invention.

  First, when the text 255 is input via the input unit 241 (step 101), the text decomposition processing unit 201 is called from the control unit 240. The text decomposition processing unit 201 reads the text 255 character by character, cuts out sentence by sentence, obtains N sentences, and stores them in the decomposed sentence storage unit 202 via the control unit 240. Here, a sentence refers to one sentence delimited by a punctuation mark “.”. As an example of the text 255, when the processing of the text decomposition processing unit 201 is performed on the text 255 as shown in FIG. 5, nine sentences decomposed into sentence units are generated as shown in FIG. 202 (step 102). The number of sentences generated in the text decomposition processing unit 201 varies depending on the input text. In addition, in the case of an input mistake of semantically following a plurality of sentences or the phrase “.”, A plurality of sentences are handled as one sentence.

  Next, the search word extraction processing unit 211 is activated by the control unit 240 for each sentence stored in the decomposed text storage unit 202. A search term refers to one or more words that are input when searching on the web. First, the search word extraction processing unit 211 performs morphological analysis on the first input sentence. Then, words classified into four nouns, adverbs, adjectives, adjective verbs, and verbs by morphological analysis are extracted as search words and stored in the search word storage unit 212 via the control unit 240. At that time, for the adjectives, adjective verbs, and verbs, the words whose inflected forms are all changed to final forms are used.

  Here, the extracted words include commonly used words such as “year” and “a” (hereinafter referred to as general words). Therefore, a general word list as shown in FIG. 7 is created in advance and stored in a memory (not shown), and a word not registered in the general word list is handled as a search word. The list of general words is as shown in FIG. The search terms stored in the search term storage unit 211 vary depending on the general word list.

Also, it is preferable to perform an AND search with an appropriate number of words when performing a web search. Therefore, if the extracted number of words is less than the predetermined threshold value S _T, the search word in the search word extraction processing unit 211 does not extract a word to the search word storing unit 212 does not store anything. Conversely, when the number S of extracted words is equal to or greater than the threshold value T, T search words are randomly selected from the S search words and stored in the search word storage unit 212. When T = 20 and S _T = 2, when the search word extraction processing unit 211 is activated, a search word as shown in FIG. 8 is stored in the search word storage unit 212.

  After the search word shown in FIG. 8 is created from the sentence of FIG. 6, the related word acquisition processing unit 221 is activated from the control unit 240 (step 201). In the related word acquisition processing unit 221, the search word first extracted by the search word extraction processing unit 211 is extracted from the search word storage unit 212 via the control unit 240 and input (step 202). Next, an AND search is performed on the web 253 connected by the network 252 using the input search word (step 204). By performing an AND search, it is possible to search on the web 253 without affecting the input order of search terms. Then, P articles 254 are acquired in ascending order of difference from the time when the text 255 was created from the web 253 referenced in the search result (step 205). Since articles created on the web 253 generally record the date of creation, the time difference from the text 255 can be measured. By using this time difference, articles strongly related to the time and contents of the text 255 can be collected from the web 253.

  As described above, the texts are collected in ascending order of the difference from the creation time of the text input when the web search is performed, but it is most preferable in terms of collecting words highly relevant to the contents of the input text. However, related words can be collected with sufficient accuracy by using P texts 254 that are referred to in the obtained search results without considering the time difference.

Here, when there is no search word corresponding to the search word storage unit 212, the related word acquisition processing unit 221 does not perform web search and stores nothing in the related word storage unit 222 (Step 203, No). ). Further, even when the number S of search words is S = T (step 203, No), the web search is not performed and the related words are not stored in the related word storage unit 222. In order to obtain appropriate related terms, the number of articles 254 obtained by web search should be as large as possible. Therefore, when the number P of texts obtained by web search is less than _PT (step 206, Yes), the search word is corrected (step 207), and the text is collected again by AND search on the web (step 204). 205). Specifically, the number of S search words is reduced to W, the SW search words having the largest number of searches are selected, and P articles 254 are collected by performing a re-search.

  Next, text is extracted from P articles 254 collected in order of time (step 2085). The article 254 is described in a structured language such as HTML or XML. Therefore, text can be obtained by analyzing a tag composed of a character string surrounded by “<” and “>” for the obtained article 254. Then, the related word acquisition processing unit 221 performs morphological analysis on the extracted text to extract nouns, adverbs, adjectives, adjective verbs, and verbs (step 209). At that time, as with the search word extraction processing unit 211, the adjectives, adjective verbs, and the verb usage forms are all extracted into words that have been converted to the final form. The number of related terms obtained varies depending on the search terms used when performing a web search and the number of articles 254 collected. If the extracted word is directly used as a related word, a general word may be handled as a related word in the same manner as the search word extraction processing unit 211. Therefore, the relevance level acquisition processing unit 221 removes the general terms by referring to the general term list as shown in FIG. 7 on the memory (not shown) as in the search term extraction processing unit 211 (step 210). Then, when there are S search words, for words extracted from P body texts, TS words are set as related words in order of appearance frequency, and the related word storage unit is connected via the control unit 240. It stores in 222 (step 211). That is, the total number of search words and relevances extracted in each sentence is set to a predetermined value T.

Then, the correction of the search word and the web search are repeated until the number P of articles 254 becomes _{equal to} or greater than _PT, and TS related words are extracted from the P articles 254 when P ≧ P _T. On the other hand, if the number of articles 254 collected is not equal to or greater than _PT even when the search word is corrected, the original S search words are left in the search word storage unit 212 and are stored in the related word storage unit 222. Do not store anything as a related term. As an example, when T = 20 and P _T = 20 with respect to the search word stored in the search word storage unit 212 of FIG. 9, the related word obtained by executing the related word acquisition processing unit 221 is FIG. As shown.

Further, when the number P of texts 254 obtained by web search is less than P _T , a keyword set consisting of a combination of a search word and a related word can be assigned to each sentence by correcting the search word and performing a re-search. Most preferred in terms. However, according to the present invention, a keyword set can be created using only search words without performing re-search in the case of less than _PT . In this case, since a plurality of keyword sets before and after the reference sentence are taken into consideration, the text content can be analyzed and the text segmentation can be performed with practical accuracy in a short calculation time.

  Finally, when the processing of the search word extraction processing unit 211 and the related word acquisition processing unit 221 is completed for all the sentences stored in the decomposed text storage unit 202, the control unit 240 switches the connectivity determination processing unit 231. to start. The connectivity determination processing unit 231 first reads out the search terms and related terms stored in the search term storage unit 212 and the related term storage unit 222 via the control unit 240 and combines them to create a keyword set. FIG. 10 shows an example of a keyword set created from the example of the search term in FIG. 8 and the example of the related term in FIG. For example, the keyword set shown in FIG. 10 is created from the search terms in FIG. 8 and the related terms in FIG. Here, the connectivity determination processing unit 231 does not create a keyword set because there is no related word corresponding to the search word when there is no search word. If there are search terms and no related terms exist, a keyword set is created using only the search terms.

  Since the keyword set is a word that reflects the content of the text, the change in the content in the text 255 can be captured by examining the change in the word included in the keyword set. Therefore, the connectivity determination processing unit 231 compares the generated keyword sets and finds a semantic paragraph composed of one sentence or a plurality of sentences that are grouped in content. The extracted semantic paragraph is stored in the semantic paragraph storage unit 232 through the control unit 240. As a comparison method, text is generally written in order from the top, so a block in which a plurality of keyword sets are collected in order from the top of the text is created and compared. Specifically, assuming that b is the size of a block, a block B1 including a keyword set from i + 1-bth to ith and a block B2 including a keyword set from i + 1th to i + bth are determined. The words in the keyword set included in the two blocks B1 and B2 are compared. A keyword set that does not include a word is not included when creating blocks B1 and B2, and a keyword set that is not empty in a sentence before the corresponding sentence in block B1, and after a corresponding sentence in block B2. Include a non-empty keyword set in the block instead. For example, if the keyword set for the j-th sentence is empty, a non-empty keyword set is found in the order of j−1, j−2,..., And included in the block B1 when the block B1 is created. On the other hand, when the block B2 is created, a non-empty keyword set is found in the order of j + 1, j + 2,..., N and included in the block B2.

After creating the block B1 and the block B2 as described above, the frequency w _{t of the} word t included in each block is calculated. Then, to evaluate by the following equation using the frequency w _t of the i-th and (i + 1) th word connectivity of two statements t.

Here, w _t ^B1 represents the frequency of the word t in the block B1, and w _t ^B2 represents the frequency of the word t in the block B2.

Takes a value between 0 and 1, and the closer to 1, the more the words contained in the block B1 and the block B2 are the same. Here, if no word is included in block B1 or block B2, the degree of connectivity

The value of is calculated as 0. The connectivity determination processing unit 231 changes i = {1, 2,..., N},

Calculate Further, the parameter of the block size b is set to b = (b ₁ , b ₂ ,..., B _M ), and the connectivity is set for each block width.

And the average value thereof is defined as the average connectivity C _i in the i-th and i + 1-th sentences.

Finally, the average connectivity C _i (where i = (1, 2,..., N)) is used to extract the average connectivity valley that is the boundary of the semantic paragraph, and the change in the text content is analyzed. Here, the “average connectivity valley” is a boundary between semantic paragraphs and appears when the content of a given text changes. The part where the content changes is a semantic paragraph, so it is called a “semantic paragraph”, and text segmentation can be performed by finding the boundary of the semantic paragraph. The valley of average connectivity satisfies the following conditions.

In the connectivity determination process 231, a semantic paragraph is extracted according to the detected valley. Specifically, when a valley is detected at the i-th, the first semantic paragraph is measured from the beginning, and the second and subsequent semantic paragraphs are searched from the location where the valley was previously detected to the i−1th. One semantic paragraph. FIG. 11 shows the result of processing of the connectivity determination processing unit 231 for the keyword set shown in FIG. 10 with the threshold C _T = 0.1. In the example of the semantic paragraph storage unit 232 in FIG. 10, valleys are detected when i = 6 and divided into two semantic paragraphs.

  When the processing of the connectivity determination processing unit 231 ends, the control unit 240 activates the output unit 242 and displays the result of text segmentation on the text 255 on the display unit 254. Specifically, referring to the semantic paragraph storage unit 232, the stored sentence number is displayed. For example, there is a method of displaying the sentence numbers shown in FIG.

As described above, the calculation time and accuracy can be adjusted by the parameters T, S _T , P _T , C _T , and b. By adjusting the parameters T, S _T , and P _T , it is possible to adjust the extraction accuracy and calculation time of related words obtained by performing a web search. For example, when collecting articles by web search, P ≧ P _T is satisfied when the number P of articles obtained reaches P _T , so that the calculation time can be reduced by terminating the collection of articles. Further, it is possible to adjust the sensitivity to capture the change in the content of the text by the parameter b, C _T. Furthermore, by using the average connectivity C _i for a plurality of parameters b, local change of contents and global change of contents are considered at the same time, and excessive semantic paragraphs that have been a problem in the conventional method are extracted. This solves the fact that semantic paragraph extraction fails when content changes slowly.

[Second Embodiment]
In the present embodiment, a case where a plurality of threshold values _CT are used as an application example of the first embodiment will be described.

  FIG. 13 shows the configuration of a text segmentation system using web search according to the second embodiment of the present invention. In the figure, the same components as those in FIG.

  A keyboard 1162 for inputting a threshold value is connected to the computer 1161 shown in FIG. Further, the computer 1161 has a configuration in which a division result selection processing unit 1141 and a generation block storage unit 1132 are added to the configuration shown in FIG.

  FIG. 14 is a flowchart showing an outline of a processing procedure in the second embodiment of the present invention.

  Text 255 is input to the input unit 241 of the computer 1161 (step 1001), the text input by the text division processing unit 201 is divided into sentence units (step 1002), and the search word extraction processing unit 211 searches the search word from the sentence. Extracted (step 1003), the related word acquisition processing unit 221 performs a search on the web using the search word, and acquires the related word from the search result (step 1004). In the connectivity determination processing unit 231, a semantic paragraph is extracted from a keyword set in which a search word and a related word are paired (step 1005). The division result selection processing unit 1141 determines whether there are a plurality of segmentation results (step 1006). If there are a plurality of segmentation results, the result is presented to the user and a threshold value is input from the keyboard 1162 (step 1007). ). The division result selection processing unit 1141 outputs the text segmentation result from the output unit 242 based on the input threshold value (step 1008).

  Hereinafter, examples of data stored in the text 255, the decomposed sentence storage unit 202, the search word storage unit 212, and the relevance degree storage unit 222 in FIG. 13 are the same as those in the first embodiment described above. Differences from the first embodiment will be described.

In the connectivity determination unit 231, it is performed on a plurality of threshold C _T. Specifically, after creating the keyword set, while using a plurality of threshold C _T in order to calculate the average connectivity of C _i, extracts the meaning paragraph. After the semantic paragraph is extracted, the semantic paragraph storage unit 232 is referred to via the control unit 240. If the division results are different, the threshold value and the semantic paragraph at that time are stored in the semantic paragraph storage unit 232 via the control unit 240. An example in which a plurality of division results are stored in the semantic paragraph storage unit 232 is shown in FIG. Here, when the same division result is obtained for a plurality of threshold values, only one threshold value and one division result are stored in the semantic paragraph storage unit 232.

  After the process of the connectivity determination processing unit 231, the control unit 240 activates the division result selection unit 1141. The division result selection unit 1141 refers to the semantic paragraph storage unit 1132 via the control unit 240. When a plurality of threshold values are stored, the division result stored in the semantic paragraph storage unit 232 is displayed on the display unit 256 via the output unit 242, and the user is requested to input the threshold value via the keyboard 1162. . The user refers to the division result displayed even on the display unit 256 and inputs one threshold value. When the threshold value is input, the control unit 240 passes the input threshold value to the division result selection processing unit 1141. Thereafter, the division result selection processing unit 1141 refers to the semantic paragraph storage unit 232 via the control unit 240, leaves only the classification result corresponding to the input threshold value, and deletes other division results. If there is only one division result stored in the semantic paragraph storage unit 232, the division result selection processing unit 1141 does not ask the user to input a threshold value and stores it in the semantic paragraph storage unit 232. The division result is output to the display unit 256 through the output unit 242.

In the above-described second embodiment, when processing for a plurality of threshold values C _T, presenting a plurality of results to select the segmentation result by inputting the threshold. Accordingly, there is an effect that text segmentation can be performed while the user adjusts whether to divide by a large change in content or to divide by a fine change in content. On the other hand, when the division processing in a batch input text can be divided automatically input text by performing the process with one threshold C _T.

  The components of the text segmentation device (computer) in the first and second embodiments described above are constructed as a program and installed in a computer used as the text segmentation device to be executed, or via a network It can be distributed.

  Further, the constructed program can be stored in a portable storage medium such as a hard disk, a flexible disk, or a CD-ROM, and can be installed or distributed in a computer.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The present invention can be applied to a technique for dividing each sentence in sentences such as various articles and stories on a computer into semantic groups.

It is a principle block diagram of this invention. It is a figure for demonstrating the principle of this invention. It is a block diagram of the text segmentation apparatus using the web search in the 1st Embodiment of this invention. It is a flowchart which shows the outline | summary of the process sequence in the 1st Embodiment of this invention. It is an example of the text in the 1st Embodiment of this invention. It is an example of the sentence stored in the decomposition | disassembly sentence memory | storage part in the 1st Embodiment of this invention. It is an example of the general word registered into the general word list | wrist in the 1st Embodiment of this invention. It is an example of the search word stored in the search word memory | storage part in the 1st Embodiment of this invention. It is an example of the related word stored in the related word memory | storage part in the 1st Embodiment of this invention. It is an example of the keyword set produced by the connectivity determination processing unit in the first embodiment of the present invention. It is an example of the sentence number stored in the semantic paragraph memory | storage part in the 1st Embodiment of this invention. It is a flowchart of the process of the related word acquisition process part in the 1st Embodiment of this invention. It is a block diagram of the text segmentation system using the web search in the 2nd Embodiment of this invention. It is a flowchart which shows the outline | summary of the process sequence in the 2nd Embodiment of this invention. It is an example of the sentence number stored in the semantic paragraph memory | storage part in the 2nd Embodiment of this invention.

Explanation of symbols

201 Text Decomposition Unit, Text Decomposition Processing Unit 202 Decomposed Sentence Storage Unit 211 Search Term Extraction Unit, Search Term Extraction Processing Unit 212 Search Term Storage Unit, Search Term Storage Unit 221 Related Word Extraction Unit, Related Word Acquisition Processing Unit 222 Related Word Storage Means, related word storage unit 231 connectivity determination unit, connectivity determination processing unit 232 semantic paragraph storage unit 241 input unit 242 output unit 251 computer 252 network 253 web 254 article 255 text 256 display unit 1132 generated in structured language Block storage unit 1141 Division result selection processing unit 1162 Keyboard

Claims (6)

A text segmentation device that divides text according to content,
Text decomposition means for dividing the input text into sentences,
A morphological analysis is performed on the sentence divided by the text decomposition means , and nouns, adverbs, verbs, adjectives, and adjective verbs that have been analyzed are extracted as search words, and the verb, the adjective, and the adjective verb are not final forms A search term extracting means for converting into a closed form and storing it in the search word storage means;
Web search based on the search term, morphological analysis of the searched text , among the analyzed morphemes, nouns, adverbs, verbs, adjectives, adjective verbs are obtained as related words, the verbs, the adjectives, A related word acquisition means for converting a non-terminal form of the adjective verb into a final form and storing it in a related word storage means;
A search word is acquired from the search word storage means, a related word is acquired from the related word storage means, and a plurality of keywords obtained by dividing the input text using a keyword set that is a combination of the search word and the related word Connectivity determination means for determining connectivity between sentences and dividing the input text by extracting a semantic paragraph that is a sentence between the valleys of the connectivity;
Have
The connectivity determination means includes
Blocks B1 and B2 that summarize the keyword set are created, and the connectivity between the i-th and i + 1-th two sentences is expressed using the appearance frequency of the word t,

(W _t ^B1 represents the frequency of the word t in the block B1, and w _t ^B2 represents the frequency of the word t in the block B2.

Takes a value between 0 and 1, and the closer to 1, the more the same words are included in block B1 and block B2)
Means to obtain
i = {1, 2,..., N},

And the block size b parameter is set to b = (b ₁ , b ₂ ,..., B _M ), and the connectivity is set for each block width.

And the average value of these as the average connectivity C _i in the i-th and i + 1-th sentences,

Means to obtain
Using the average connectivity C _i (where i = (1, 2,..., N)), a valley of average connectivity that is a boundary of a semantic paragraph is defined as a condition.

And a means for acquiring the semantic paragraph based on the valley. The connectivity determination means includes
2. The text segmentation device according to claim 1, further comprising means for calculating the average connectivity C _i using a plurality of thresholds C _T and extracting the semantic paragraph. A text segmentation method in a device for dividing text according to content,
A text decomposition step in which the text decomposition means divides the input text into sentence units;
The search word extraction means performs morphological analysis on the sentence divided in the text decomposition step, extracts nouns, adverbs, verbs, adjectives, and adjective verbs as morphological analysis as search words, and the verb, the adjective, and the adjective verb A search term extraction step of converting a non-termination type into a termination type and storing it in the search term storage means;
A related word acquisition means performs a web search based on the search word, performs a morphological analysis on the searched text, acquires nouns, adverbs, verbs, adjectives, and adjective verbs that are morphologically analyzed as related words. An adjective, a related word acquisition step of converting an adjective verb that is not a terminal form into a terminal form and storing it in a related word storage means;
The connectivity determination means acquires a search word from the search word storage means, acquires a related word from the related word storage means, and uses the keyword set that is a combination of the search word and the related word, to input the input A connectivity determination step of determining connectivity between a plurality of sentences obtained by dividing the text, and dividing the input text by extracting a semantic paragraph that is a sentence between the valleys of the connectivity;
And
In the connectivity determination step,
Blocks B1 and B2 that summarize the keyword set are created, and the connectivity between the i-th and i + 1-th two sentences is expressed using the appearance frequency of the word t,

(W _t ^B1 represents the frequency of the word t in the block B1, and w _t ^B2 represents the frequency of the word t in the block B2.

Takes a value between 0 and 1, and the closer to 1, the more the same words are included in block B1 and block B2)
A step to obtain by
i = {1, 2,..., N},

And the block size b parameter is set to b = (b ₁ , b ₂ ,..., B _M ), and the connectivity is set for each block width.

And the average value thereof as the average connectivity C _i in the i-th and i + 1-th sentences,

A step to obtain by
Using the average connectivity C _i (where i = (1, 2,..., N))

And extracting the semantic paragraph based on the valley, and a text segmentation method. In the connectivity determination step,
Wherein calculating the average degree of coupling C _i using a plurality of threshold C _T, text segmentation method as claimed in claim 3, wherein extracting the meaning paragraph.   A text segmentation program for causing a computer to function as each means constituting the text segmentation apparatus according to claim 1. A computer-readable recording medium storing the text segmentation program according to claim 5.