JP2590141B2 - Collocation extraction processing method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION [Summary] Computers extract collocations from digitized sentences to be used for extracting technical terms when creating dictionaries in a machine translation system and extracting keywords in an information search system. The purpose of the present invention is to provide a means for automatically extracting collocations that are considered to have a special meaning from a given digitized sentence. A word segmentation process of dividing a sentence into words, a word segmentation process of combining adjacent words within a predetermined number of words for each segmented word, and a word segmentation process of creating a word sequence. The process of counting the number of appearances in the sentence and the process of counting the number of occurrences, removing duplicated collocations between collocations with the same number of appearances, and extracting the remaining collocations And a duplicate word deleting process.

[Industrial applications]

The present invention provides a collocation extraction process for extracting collocations from digitized sentences by a computer for use in extracting technical terms in creating a dictionary in a machine translation system and extracting keywords in an information retrieval system. About the method.

[Conventional technology]

For example, when creating a dictionary to be used in a machine translation system that translates English into Japanese, it is necessary to extract collocations in which a certain combination of words has a special meaning, and to give an optimal translation to the collocation. is there.

In addition, in an information retrieval system that has a database that stores technical documents, if terms that are considered to be important in each technical document are extracted in advance as keywords, searches can be performed efficiently using the keywords. Yes, but this keyword can be used not only for words,
It is desirable to be able to specify collocations.

Conventionally, when extracting such collocations, a human reads a sentence and extracts it by empirical judgment according to each field in which the collocation is used. Therefore, there is a problem that it takes a lot of time and labor.

By the way, there is a method of automatically extracting unusual words that are considered to have special meanings such as technical terms and keywords from computerized sentences, although they are not collocations. .

In this method, when extracting words that become technical terms and keywords from the digitized text, the computer counts the number of occurrences of each word appearing in the text, and finds the word with the highest number of occurrences. , So-called stop words are uniformly removed, and the remaining words are defined as unique words.
In English, stop words are “a”, “t”
articles such as he ", ..., BE verbs such as" is "," are ", ...," t
Words such as prepositions such as "o", "in", ..., which are frequently used but are clearly not unique words.

[Problems to be solved by the invention]

According to the conventional method of simply selecting a unique word based on the number of occurrences, it is possible to automatically extract collocations that are considered to have a special meaning different from the meaning of each word by combining the preceding and following words. In addition, there is a problem that many insignificant collocations are output even if these words are combined from the extraction results of the unique words.

For example, when creating a translation dictionary, "DEEP FREEZE
It is necessary to give a technical term “freezer” to “R.” However, if only words are extracted, “DE
EP "and" FREEZER "are extracted separately, so
“DEEP” is translated to “deep”, “FREEZER” is translated to “freezer”, and “DEEP FREEZER” is translated to “deep freezer”.

Means for Solving the Problems The present invention solves the above-mentioned problems, and automatically extracts a collocation word considered to have a special meaning from a given digitized sentence while excluding as few meaningless as possible. It is intended to provide.

[Means for solving the problem]

 FIG. 1 is a diagram illustrating the principle of the present invention.

In FIG. 1, 10 is an input device such as a keyboard, and 11 is
A computer consisting of a CPU and memory, etc., 12 is a document file storing text data to be extracted
Reference numeral 3 denotes an extraction table used as a work table for extracting collocations, and 14 denotes an output device, such as a printer or an external storage device, to which the extraction results of the collocations are output.

In the input process P1, a sentence to be subjected to collocation extraction is input from the input device 10 in the same manner as, for example, a word processor process, and a document file 12 is created. The document file 12 does not necessarily have to be created by the computer 11, but may be created by another computer.

In the word division process P2, a process of reading the text of the document file 12 and dividing the text into words for each sentence is performed. The divided word information is stored in the work area.

In the collocation generation process P3, for each word of each divided sentence, a collocation is created by combining preceding and succeeding words within a predetermined number of words such as 3 or 4 words, and the collocation information is extracted. Stored in table 13.

In the appearance count counting process P4, the number of appearances in the text from which the collocation is to be extracted is counted for each created collocation, and the result is entered in the extraction table 13. This counting may be performed for each sentence, or may be performed for the entire sentence at the end.

In the duplicate word deletion process P5, while referring to and updating the extraction table 13, duplicate collocations are removed from the collocations having the same number of appearances, and the remaining collocations are extracted as desired collocations. The sorting process P51 based on the number of appearances is an auxiliary process called by the duplicate word deletion process P5 to collect collocations having the same number of occurrences.

In the output process P6, the collocations finally left in the extraction table 13 are output, for example, in alphabetical order to an output device 14 such as a printer or a magnetic disk device.

[Action]

 FIG. 2 is a diagram for explaining the operation of the present invention.

After the processing by the word division processing P2 and the processing by the collocation creation processing P3 shown in FIG. 1, it is assumed that the processing result by the appearance frequency counting processing P4 becomes, for example, an extraction table 13 shown in FIG. In other words, “CORE STORAGE” appears 10 times in the sentences for which collocations are to be extracted, and “MAGNETIC CO
"RE" appears 10 times and "MAGNETIC CORE STORAGE" appears 10 times. Focusing on the number of occurrences, "CORE STORAGE" and "MAGNETIC CORE" do not appear separately from "MAGNETIC CORE STORAGE" It is clear that “MAGN
It can be seen that "ETIC CORE STORAGE" is always used as a set of three words.
RAGE "and" MAGNETIC CORE "
RE STORAGE "only.

For example, as shown in FIG.
RAGE "10 times," MAGNETIC CORE STORAGE "10 times," MAG
NETIC CORE "appears 30 times. Focusing on this number of occurrences," CORE STORAGE "is always" MAGNETIC CORE STO "
RAGE ".
"CORE STORAGE" is removed by the duplicate word deletion process P5,
Leave "MAGNETIC CORE STORAGE" and "MAGNETIC CORE" that may appear separately.

Since processing is performed based on the number of appearances as described above, insignificant collocations created by combining preceding and succeeding words are removed, and only collocations that are likely to have significant combinations are extracted. Will be.

〔Example〕

FIG. 3 is a diagram for explaining the processing of an embodiment of the present invention, FIG. 4 is a diagram for explaining word division and collocation according to an embodiment of the present invention, and FIG. 5 is a duplicate word according to an embodiment of the present invention. FIG. 6 shows an example of deletion processing, and FIG. 6 shows a specific example of duplicate word deletion according to an embodiment of the present invention.

Hereinafter, an example of processing according to an embodiment of the present invention will be described with reference to the processing shown in FIG.

One sentence is read from the document file 12 shown in FIG.

It is determined whether or not there is a sentence depending on whether or not the reading is successful. When the statement ends, control is transferred to the processing.

Break sentences into words. In English, it can be divided by spaces or commas. In Japanese, for example, the division may be performed by a method used in a batch kanji conversion or the like in a word processor.

A collocation is created by combining words before and after within a predetermined number of words. For example, if the number of words is determined to be four, two, three, and four collocations are created.

The number of occurrences of collocations is accumulated for each collocation. The result is stored in the extraction table 13 shown in FIG. 1 or another work area.

The next one sentence is read from the document file 12, control is returned to the process, and the process is repeated in the same manner until the process for all the sentences is completed.

When the processing for all the sentences is completed, the collocation is read one by one from the extraction table 13 or another work area.

It is determined whether there is any unprocessed collocation in the extraction table 13 or another work area. When the processing for all collocations is completed, control is transferred to the processing.

Check whether the collocation is a meaningless collocation.
Here, whether or not a collocation is meaningless, for example, in English, (1) collocation ending with an article, (2) BE verb, HAVE
Judgment is made according to a predetermined rule such as a collocation including a verb. If the collocation is meaningless,
Processing proceeds to processing otherwise.

Non-significant collocations are stored in the extraction table 13.

The next collocation is read, control is returned to the process, and the process is repeated in the same manner.

When the processing for all collocations is completed, the processing for deleting duplicate words in the extraction table 13 is performed. This detailed processing will be described later with reference to FIG. The desired collocation is determined based on the result of removing the duplicate words.

In the processing by the word division processing P2 and the collocation generation processing P3 shown in FIG. 1, specifically, word division and collocation generation as shown in FIG. 4 are performed.

For example, assume that the English sentence (a) “Make hay while the sun shines.” Is handled. Fig. 4 (b)
As shown in (1), processing for dividing each word is performed. Next, the two words before and after are connected to each other to create a two-word collocation (c). Similarly, a collocation (d) based on the arrangement of three words is created. Create these collocations up to a predetermined number of words,
Each result is set as a processing target of the next occurrence count processing P4.

FIG. 5 shows in detail a part of the processing content by the duplicate word deletion processing P5 shown in FIG. FIG.
A specific example by the processing shown in the figure is shown. Here, the following assumptions are made to simplify the description.

(A) The words E, G, N, O, and R are different words.

(B) each in the text Appear n times.

(C) The maximum length of a collocation is 5 words.

Based on this assumption, to remove duplicate collocations,
The following processes [i] to [v] are performed.

[I] The collocations in the extraction table 13 are sorted in the order of appearance, first word, second word, third word, fourth word, and fifth word in the normal order. By this processing, word arrangement The contents of the extraction table 13 related to are as shown in FIG.

[Ii] Next, duplicate words are deleted from each collocation in the extraction table 13 by the processing shown in FIG. That is, the first collocation between the current collocation and the next collocation is
The word, the second word, and so on are compared in order until there are no more words. If the preceding collocation part completely matches the left part of the next collocation, the collocation is deleted, and no match is found. Then, the collocation is left in the extraction table 13.

For example, collocation Followed by collocations When there is a word, all words from the first word to the third word are matched. And delete collocations Leave. On the other hand, collocation Followed by collocations In some cases, the first word does not match, so
No deletion is performed.

By the above-described deletion processing, collocations on the extraction table 13 shown in FIG. 6 (a) are arranged as shown in FIG. 6 (b).

[Iii] Next, each collocation is right-justified as shown in FIG.

[Iv] The collocation shown in FIG.
The fifth word, the fourth word, the third word, the second word, and the first word are rearranged in the normal order. The processing result is as shown in FIG.

[V] The processing is performed in the same manner as in the above processing [ii], and those that match the next collocation are deleted. However, in the comparison with the processing shown in FIG. 5 in the above processing [ii], the fifth word, the fourth word, the third word, and the second word are compared in order of coincidence and non-coincidence.

As a result of the above processing, the extraction table 13 finally contains the collocation words as shown in FIG. Will remain. Collocation Etc. do not appear alone in this part, but always Are used integrally, so the collocation shown in FIG. 6 (e) is the collocation to be sought here.

The collocations obtained in this way are translated as if each collocation were a single word, for example, when a translation dictionary used in a machine translation system was created, enabling more appropriate translation. It can be used for things.

Also, when a collocation is treated as one keyword in the information retrieval system, it can be used for keyword registration processing of the collocation.

〔The invention's effect〕

As described above, according to the present invention, it is possible to automatically extract collocations that are considered to have a special meaning in a sequence of words from an electronic sentence, and a dictionary for machine translation can be provided. It can be used for the development of information and the extraction of keywords in information retrieval.

[Brief description of the drawings]

1 is a diagram illustrating the principle of the present invention, FIG. 2 is a diagram illustrating the operation of the present invention, FIG. 3 is a diagram illustrating the processing of an embodiment of the present invention, and FIG. 4 is word segmentation according to an embodiment of the present invention. FIG. 5 shows an example of duplicate word deletion processing according to one embodiment of the present invention, and FIG. 6 shows a specific example of duplicate word deletion according to one embodiment of the present invention. In the figure, P1 is input processing, P2 is word segmentation processing, P3 is collocation creation processing, P4 is occurrence count processing, P5 is duplicate word deletion processing, P51 is sorting processing, P6 is output processing, 10 is an input device, 11 Denotes a computer, 12 denotes a document file, 13 denotes an extraction table, and 14 denotes an output device.

Claims (1)

(57) [Claims] 1. A collocation extraction method for extracting collocation from digitized text by a computer, comprising: a word segmentation process (P2) for dividing a text to be extracted from collocation into words; , For each of the divided words, combining the preceding and succeeding words within a predetermined number of words to create a collocation word (P3), and counting the number of occurrences in the sentence for each created collocation word A collocation characterized by comprising a number-of-occurrence counting process (P4) and a duplicate word removal process (P5) for removing duplicated collocations between collocations having the same number of occurrences and extracting the remaining collocations. Extraction processing method.