JP5111300B2 - Document summarization method, document summarization apparatus, document summarization program, and recording medium recording the program - Google Patents

Description

  The present invention deals with a summary for text described in a natural language, and a document D is given as an input, and a summary A that falls within the upper limit U of a specified number of characters and maximizes a certain evaluation function is output. The present invention relates to a document summarization method, a document summarization apparatus, a document summarization program, and a recording medium on which the program is recorded.

  The conventional summarization technique gives a score to “sentences” in a document by some method, ranks them, selects “sentences” until the upper limit U of the number of characters is reached, and makes a summary (see Non-Patent Document 1).

Here, the document shown in FIG. 6 will be described as input. First, the sentence score determination method for the sentence S ₅ will be described with reference to Table 1.

  The sentence is divided into words by a morphological analyzer and a part of speech tag is given. Next, based on the part-of-speech tag, whether or not to make a weighting target (whether or not to contribute to the sentence score) is determined, and a word score is acquired from an external database. Usually, nouns, verbs, and adjectives are subject to weighting. The word score can be obtained from a large-scale text database by using the tf / idf method or the like. The sum of the word scores thus obtained is used as the final sentence score.

Using the method described above, scores are assigned to all sentences S _{1 to} S _{5 in} the document of FIG.

Next, sentences are selected from sentences having a high sentence score until the character number limit U is satisfied, and these are summarized (see Non-Patent Document 2). In the case of U = 50, if the sentence S ₅ is selected after selecting the sentence S ₁ , the total number of characters becomes 57, so U is exceeded. Therefore, the summary is composed only of the sentence S _{1 as} shown in FIG.
Tomoori Hori and Sadahiro Furui, “Speech summary sentence generation method by word extraction and its evaluation”, IEICE Transactions, D-II VOL. J85-D-II No. 2 pp. 200-209 February 2002 Zechner, K.M. “Fast Generation of Abstracts from General Domain Text Corporative Extracting Relevant Sentences”, Proc. of the 16th International Conference on Computational Linguistics, pp. 15-16 986-989 (1996).

  In the conventional method, since a summary is generated by simple sentence extraction, it is not always possible to extract a sentence so as to satisfy a given character limit. In the previous example, only one sentence can be selected as a summary. As described above, there is a problem that the summarization based only on sentence extraction is not flexible in limiting the number of characters. This is a fatal problem when it is desired to display the summary in a limited space such as a portable terminal.

  An object of the present invention is to provide a document summarizing method, a document summarizing apparatus, a document summarizing program, and a recording medium on which the program is recorded, which can flexibly cope with the limitation on the number of characters. is there.

  In order to solve the above problems, the present invention provides a document summarization technique that can flexibly cope with a given character limit by using sentence shortening and sentence extraction together. However, simply shortening the sentence at a uniform reduction rate can break the context. Therefore, in the present invention, for each sentence in the document, the original sentence itself and a shortened sentence with a plurality of shortening rates are prepared, and a problem combination is selected by selecting a combination of sentences that maximizes the evaluation function. To do.

That is, in the document summarizing method according to claim 1, the sentence shortening means shortens each sentence in the input document at a plurality of designated shortening rates and outputs the original sentence and the shortened sentence. A sentence score determining step, wherein the sentence score determining means determines a sentence score that is a measure indicating suitability as a summary sentence for each of the original sentence and the abbreviated sentence for each sentence output by the sentence shortening step. And the sentence selection means, as a summary, combines the original sentence or the abbreviated sentence for each sentence in the document that has the maximum sum of the sentence scores determined by the sentence score determination step under the designated character limit. And a sentence selection step for selecting.

  The document summarization method according to claim 2 is the document summarization method according to claim 1, wherein the sentence score determination step includes a position information score obtained from the designated shortening rate, sentence appearance position information, and a designated parameter; The sentence score is determined based on a word score, which is a scale indicating the weight of words constituting the sentence.

The document summarization method according to claim 3 is the document summarization method according to claim 1, wherein the sentence selection step moves a combination of an original sentence or a shortened sentence for each sentence in the document having a maximum sum of the sentence scores. It is characterized by being obtained by statistical programming.

Further, the document summarization apparatus according to claim 4, wherein each sentence in the input document is shortened at a plurality of designated shortening rates to output an original sentence and a shortened sentence; Sentence score determination means for determining a sentence score, which is a measure indicating the appropriateness as a summary sentence, for each of the original sentence and the shortened sentence for each sentence output by the sentence shortening means, and under a specified number of characters And sentence selecting means for selecting, as a summary, a combination of original sentences or shortened sentences for each sentence in the document having the maximum sum of sentence scores determined by the sentence score determining means.

  The document summarizing apparatus according to claim 5 is the document summarizing apparatus according to claim 4, wherein the sentence score determining means includes a position information score obtained from the designated shortening rate, sentence appearance position information and designated parameters; The sentence score is determined based on a word score, which is a scale indicating the weight of words constituting the sentence.

The document summarizing apparatus according to claim 6 is the document summarizing apparatus according to claim 4, wherein the sentence selecting unit moves a combination of an original sentence or an abbreviated sentence for each sentence in the document having a maximum sum of the sentence scores. It is characterized by being obtained by statistical programming.

  A document summarization program according to claim 7 is a document summarization program that combines sentence extraction and sentence shortening for causing a computer to function as each means according to any one of claims 4 to 6.

  A recording medium according to an eighth aspect is a computer-readable recording medium on which a document summarizing program combining the sentence extraction and the sentence shortening according to the seventh aspect is recorded.

(1) According to the first to eighth aspects of the present invention, it is possible to flexibly cope with the character number limitation in document summarization. In addition, the original sentence itself and a shortened sentence with a plurality of shortening rates are prepared, sentence scores are given to them, and a combination of the original sentence or the abbreviated sentence for each sentence in the document that maximizes the sum of the sentence scores. Because it is selected as a summary, the context is not compromised.
(2) According to the second and fifth aspects of the invention, it is possible to determine a sentence score in consideration of sentence appearance position information.
(3) According to the third and sixth aspects of the invention, it is possible to efficiently obtain a combination of an original sentence or a shortened sentence for each sentence in the document having the maximum sum of sentence scores.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited to the following embodiments. FIG. 1 is a block diagram showing an embodiment of a document summarizing apparatus according to the present invention.

  In FIG. 1, reference numeral 1 denotes a document summarizing apparatus, which includes a sentence shortening device 11 as a sentence shortening means, a sentence score determining device 12 as a sentence score determining means, and a sentence selecting device 13 as a sentence selecting means. .

  A word score database 2 is provided outside the document summarizing apparatus 1 and stores a word score as a scale indicating the weight of the word. The word score database 2 and the document summarization apparatus 1 are connected by, for example, a network.

  Each function to be described later of the sentence shortening device 11, the sentence score determining device 12, and the sentence selecting device 13 is achieved by, for example, a computer. A document, a character limit, a sentence shortening rate set, and a parameter are input to the document summarizing apparatus 1 from an input device (not shown), thereby specifying a limit character number, a sentence shortening rate, and a parameter.

  The sentence shortening device 11 shortens each sentence in the document according to the inputted sentence shortening rate, and outputs a set of original sentences and abbreviated sentences (illustrated “sentence set (original sentence and abbreviated sentence)”). The set is input to the sentence score determination device 12.

  The sentence score determination device 12 calculates a sentence score based on a word score obtained by referring to the word score database 2, sentence appearance position information, sentence shortening rate, and given parameters for the input original sentence and the abbreviated sentence. Determine (sent sentence set with score).

  The sentence score in the present invention is a scale indicating the suitability as a summary sentence.

  The sentence selection device 13 selects and outputs, as a summary, a combination that maximizes the sum of the sentence scores determined by the sentence score determination device 12 among the combinations of sentences that satisfy the given character number limit.

Next, details of each of the above devices will be described.
<Sentence shortening device 11>
Now, it is assumed that a document composed of N sentences is D = {S ₁ ,. . . , S _i,. . . , S _N }, and the sentence shortening rate set R = {r ₁ ,. . . , R _j,. . . , R _k }. The sentence shortening device 11 shortens the sentence S _i (1 ≦ i ≦ N), which is an element of the set D, at a shortening rate r _j (1 ≦ j ≦ k). The shortening rate is calculated based on the number of words. If the input sentence is 10 words and the shortening rate is 0.7, the shortened sentence is 7 words.

Any sentence shortening technique may be used as long as the sentence shortening rate can be arbitrarily set. The example of FIG. 2 is the result of shortening the input sentence S ₁ consisting of 20 words by the shortening rate R = {1.0, 0.9, 0.8, 0.7, 0.6}.

The processing of the sentence shortening device 11 uses each processing method described in Non-Patent Document 1, for example.
<Sentence score determination device 12>
Based on the sentence shortening rate, the appearance position of the sentence, and the word score obtained by referring to the word score database 2, the sentence score is determined as shown in FIG. In FIG. 3, the sentence set on the input side of the sentence score determining device 12 is the sentence S _{1 to} S ₅ in FIG. 6 with the shortening rate r = 1.0, 0.9, 0.8, 0.7, 0.6. And the output side of the sentence score determination device 12 indicates the score given to each sentence of the sentence set.

First, the position information score for the i-th sentence at the shortening rate r _j is calculated by the following formula (1) using the mixed normal distribution.

S _{i in} equation (1) represents the i-th sentence in document D, and k (S _i , D) is a function that returns the relative appearance position of the i-th sentence, and is calculated by the following equation (2). To do.

Start (S _i ) in equation (2) represents the number of characters that the sentence S _i starts from the beginning of the document, and length (D) represents the total number of characters in the document D.

  These parameters may be given using human foresight knowledge, or optimum values may be automatically obtained using a machine learning method or the like.

  For example, if the following parameters are set so that the importance of sentences appearing near the beginning and near the end of a document is high and the importance of sentences with a high shortening rate is high, the position information score is as shown in the graph of FIG. Become.

Next, the sentence score is determined by the following equation (3) as the product of the position information score g _rj (k (S _i , D), λ _rj ) and the sum of the word scores w (t).

<Sentence selection device 13>
Here, it is an object of the present invention to obtain a sentence combination that satisfies a certain character limit and maximizes the expression (4), that is, a summary A.

  Therefore, in the present invention, the summary A that efficiently maximizes the expression (4) is obtained by using dynamic programming. FIG. 5 shows an example of the calculation procedure.

  In FIG. 5, N is the number of sentences in the document, U is the number of characters that is the upper limit of summarization, and Len (•) is a function that returns the number of characters in the sentence. Cost [i] [c] represents the maximum value of the summary score F (•) when the number of characters is less than or equal to c, and is a table that stores the summary score when the i-th sentence is selected. .

  In FIG. 5, first, a table Cost [i] [c] that stores a summary score at the time of selecting up to the i-th sentence, and a table G [i] that stores which shortening rate is selected at that time. [c] is prepared and initialized. The first to seventh lines in FIG.

  Next, a score is calculated when the i-th sentence is selected. The 8th to 22nd lines in FIG. However, since the character limit is given as U, the sentence may not be selected. At this time, G [i] [c] becomes zero. That is, Cost [i−1] [c] is stored in tmp in the 10th line, and G [i] [c] is set to 0 in the 11th line. In the for loop on the 12th line, the following processing is performed for each shortening rate.

  As described above, the summary A can be efficiently obtained.

Here, FIG. 7B shows an example in which a summary is created by applying the present invention to the input document of FIG. According to FIG. 7B, it can be seen that the limitation on the number of characters can be flexibly handled as compared with the summary by the conventional method of FIG. In FIG. 7B, the context of the sentences S _{1 to} S ₅ of the input document in FIG. 6 is not impaired.

  Further, since the position information score of the formula (1) is obtained based on the sentence shortening rate, the appearance position information, and the parameters of the mixed normal distribution, the sentence score can be determined in consideration of the sentence appearance position information. .

  Further, since the dynamic programming shown in FIG. 5 is used, the summary A that maximizes the sum of the sentence scores (formula (4)) can be efficiently obtained.

  The sentence-attached sentence set determined by the sentence score determination apparatus 12 of FIG. 1 may be configured to be stored in a memory (not shown) and handled.

  Further, as an embodiment of the document summarizing method of the present invention, the processing described in each of the devices 11 to 13 of the document summarizing device 1 in FIG. 1 is executed.

  That is, the sentence shortening device 11 shortens the sentence in the input document at a plurality of input sentence shortening rates and outputs the original sentence and the shortened sentence as shown in FIG. A sentence score determining step in which the apparatus 12 determines a sentence score, which is a measure indicating appropriateness as a summary sentence, for example, as shown in FIG. 3 for the original sentence and the shortened sentence output in the sentence shortening step; 13 executes a sentence selection step of selecting, as a summary, a combination of sentences that maximizes the sum of the sentence scores determined by the sentence score determination step under the input character limit.

  The sentence score determination step includes, for example, the position information score obtained by calculating the formula (1) based on the input shortening rate, sentence appearance position information, and input parameters, and the word of FIG. The sentence score is determined based on a word score, which is a scale indicating the weight of words constituting the sentence, acquired with reference to the score database 2.

  In the sentence selection step, a combination of sentences that maximizes the sum of the sentence scores is obtained by, for example, dynamic programming shown in FIG.

  In addition, some or all of the functions of each means in the document summarizing apparatus of the present embodiment can be configured by a computer program, and the program can be executed using the computer to realize the present invention. Needless to say, the procedure in the document summarization method can be constituted by a computer program, and the program can be executed by the computer, and the computer-readable recording medium, for example, FD, can be realized by the computer. (Floppy (registered trademark) Disk), MO (Magneto-Optical disk), ROM (Read Only Memory), memory card, CD (Compact Disk) -ROM, DVD (Digital Versatile D) sk) -ROM, CD-R, CD-RW, HDD, and recorded in a removable disk, or stored, it is possible or distribute. It is also possible to provide the above program through a network such as the Internet or electronic mail.

1 is a block diagram illustrating an embodiment of a document summarization apparatus according to the present invention. Explanatory drawing which shows the detail of the sentence shortening apparatus in one embodiment of the document summarization apparatus of this invention. Explanatory drawing which shows the detail of the sentence score determination apparatus in one embodiment of the document summarization apparatus of this invention. The graph which shows an example of the position information score in one example of embodiment of the document summarization apparatus of this invention. Explanatory drawing which shows an example of the calculation procedure which calculates | requires efficiently the combination of the sentence in which the sum of sentence scores becomes the maximum in one Embodiment of the document summarization apparatus of this invention. Explanatory drawing which shows an example of the document input into the document summarization apparatus of this invention. The summary when a character limit is 50 characters is represented, (a) is explanatory drawing of the summary by a conventional method, (b) is explanatory drawing of the summary by this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Document summary apparatus, 2 ... Word score database, 11 ... Sentence shortening apparatus, 12 ... Sentence score determination apparatus, 13 ... Sentence selection apparatus.

Claims (8)

A sentence shortening step in which the sentence shortening means shortens each sentence in the input document at a plurality of designated shortening rates and outputs the original sentence and the shortened sentence;
Sentence score determination means, for each of the original sentence and the shortened sentence for each sentence output by the sentence shortening step, a sentence score determining step for determining a sentence score which is a measure indicating appropriateness as a summary sentence;
The sentence selecting means selects, as a summary, a combination of the original sentence or the abbreviated sentence for each sentence in the document in which the sum of the sentence scores determined by the sentence score determining step is maximized under the designated number of characters limit. A document summarization method combining sentence extraction and sentence shortening, characterized by comprising a sentence selection step.   The sentence score determination step is based on a position information score obtained from the designated shortening rate, sentence appearance position information and a designated parameter, and a word score which is a scale indicating a weight of words constituting the sentence. The sentence summarization method combining sentence extraction and sentence shortening according to claim 1, wherein the sentence score is determined. 3. The sentence according to claim 1, wherein the sentence selection step obtains a combination of an original sentence or a shortened sentence for each sentence in the document having the maximum sum of the sentence scores by dynamic programming. A document summarization method that combines extraction and sentence shortening. A sentence shortening means for outputting the original sentence and the shortened sentence for each sentence in the input document by shortening at a plurality of designated shortening rates;
Sentence score determining means for determining a sentence score which is a measure indicating appropriateness as a summary sentence for each of the original sentence and the shortened sentence for each sentence output by the sentence shortening means;
Sentence selecting means for selecting, as a summary, a combination of original sentences or abbreviated sentences for each sentence in the document having a maximum sum of sentence scores determined by the sentence score determining means under a designated number of character limit; A document summarization apparatus that combines sentence extraction and sentence shortening characterized by being provided.   The sentence score determining means is based on a position information score obtained from the designated shortening rate, sentence appearance position information and a designated parameter, and a word score which is a scale indicating a weight of a word constituting the sentence. 5. The document summarization apparatus combining sentence extraction and sentence shortening according to claim 4, wherein the sentence score is determined. The sentence according to claim 4 or 5, wherein the sentence selection means obtains a combination of an original sentence or a shortened sentence for each sentence in the document having a maximum sum of the sentence scores by dynamic programming. Document summarization device combining extraction and sentence shortening.   A document summarization program combining sentence extraction and sentence shortening for causing a computer to function as each means according to any one of claims 4 to 6.   A computer-readable recording medium on which a document summarizing program combining sentence extraction and sentence shortening according to claim 7 is recorded.

Images