JP3379643B2 - Morphological analysis method and recording medium storing morphological analysis program - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a Japanese morphological analysis method for dividing a Japanese text into words and adding information such as reading and part of speech.

[0002]

2. Description of the Related Art In word recognition in Japanese morphological analysis, words that satisfy a connectable part-of-speech chain are extracted using a word dictionary and grammatical rules, priority is given to the part-of-speech chain, and filtering is performed by the part-of-speech chain. After that, if there are homomorphic words, it is common to narrow down by the priority of the words and finally extract the word chain that becomes the first solution.

As a concrete method of assigning a priority, a minimum clause number method which gives priority to a chain having a small number of clauses (Yoshimura et al.
"Morphological analysis of solid Japanese sentences using the minimum number of clauses", Information Processing Vol. 24, No. 1, 1983), a cost minimization method that puts a cost on word adjacency rules and gives priority to a chain whose sum is the minimum (Hisamitsu et al., "Proposal of morphological analysis by minimum connection cost method and evaluation of computational complexity", IEICE Techniques, NLC 90-8, 1990) and the like and techniques combining them have been established.

However, in these conventional methods, even when a plurality of readings are possible, the word having the most common reading is selected as the first solution, so that the local proper noun (for example, the first and last name of a colleague) is selected. Correct word recognition may not be performed. For example, "Kenji Nakajima" is usually recognized as "Nakajima Kenji" in the usual morphological analysis, but
In fact, the accreditation of “Takeshi Nakajima” may be correct.

In addition, there are many proper nouns having the same notation as general nouns, but it is difficult to determine which one is the same without considering the context (eg, "Mayor of field (= place name)", "Susuki"). Nonohara (= general noun) ”), in general morphological analysis, it is generally fixed as either part of speech and recognized.

[0006]

In the conventional Japanese morphological analysis method, the reading cannot be correctly added to a word having an uncommon reading, and the person's name is erroneously read when used in text-to-speech synthesis, for example. There was a problem such as.

Further, a word whose part-of-speech is difficult to distinguish is fixed to one of the parts-of-speech and is recognized. For example, even when it is known that the description of a product, the product name is treated as a general noun. There was a problem.

An object of the present invention is to provide a morphological analysis method that improves word recognition accuracy.

[0009]

In the morphological analysis method of the present invention, a data processing device stores a Japanese text from an input device and stores it in a storage device for generating a part-of-speech chain and giving a priority. and that the described connectable word linked to grammar rules, the words of certified morphological analysis have created the word information is extracted from the word dictionary stored in the storage device, to create a word concatenation candidate sequence, The word dictionary search step output to the storage device and the data processing device satisfy the specified word information for the character string for which the word information is specified, and the word information not specified is most likely. The word having information is identified as the first solution, and the word to which the word information is given is described in Japanese text in accordance with the designation method of the target character string to which the word information is given and the word information to be given. Registered in the setting information stored device comprises a word selection step of outputting Japanese word information sequence to the output device. The word selection step is a data processing device.
But prioritized for the word chain sequence candidates, a first step of the highest chain priority to the first solution, Day
A second step in which the data processing device determines whether or not the Japanese text has a character string to which word information is given or a target character string that is a word registered in the registered word information of the setting information; When the data processing device has the target character string, a third step of determining whether or not a word that exactly matches the target character string exists in the part-of-speech chain having the first priority, and the data processing device, If there is no word that exactly matches the target character string, the word boundary matches the target character string, and if there is a part-of-speech designation, it is determined whether a word that also matches the part-of-speech exists in the word chain candidate sequence. Fourth
A step of, data processing apparatus, if there is a word object character string and words boundary coincides, with respect to the target string,
If a part of speech is specified, if there is a word that matches the part of speech and also satisfies the other specified word information, select that word, and if the other specified word information is different,
The word is duplicated including the word chain, the word information specified elsewhere is overwritten, and when no part of speech is specified for the target character string, the highest priority in the fourth step is set. A fifth step of duplicating a word having a high word chain including a part-of-speech chain and overwriting other specified word information, and a case where the data processing device specifies a part-of-speech for a target character string. Of the part-of-speech chain that has the boundary of the target word string as the word boundary, copies the part-of-speech chain before and after the chain with the highest priority that can be chained with the part-of-speech specified by the boundary, and the target word string is one word. The specified word information is filled, and the word information that is not specified is generated from the word of the original part-of-speech chain, and if there are multiple words, it is generated from the word with the highest priority and can be chained. If does not exist When a chain of words is generated and no part of speech is specified, the word chain with the highest priority is copied from the part of speech chain having the boundary of the target word string as a word boundary, and the target word string is 1 buried word information specified as a word, a sixth step of generating from the original single information about the word information not specified, the data processing device
Includes the seventh step of increasing the priority of the words dealt with in the third, fifth and sixth steps so that the word is selected in the first rank.

[0010]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Next, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a flow chart showing a morphological analysis method according to an embodiment of the present invention. A Japanese text 1 is input, a word dictionary search process 2-1 is performed, and a word chain candidate sequence 2-2 is output. , Word selection processing 2-3, and Japanese word information sequence 3
Output grammar rule 4, word dictionary 5 and setting information 6
Using.

Japanese text 1 is any Japanese text. However, word information may be added in some cases. The word information mentioned here is information to be registered in the word dictionary 5, and generally, there are many information such as notations, parts of speech, readings, meaning attributes, and accent types. Here, since the Japanese text 1 to be analyzed naturally includes the notation, word information other than the notation is added.

The word information specifying pattern for giving word information can be arbitrarily specified by the word information specifying pattern of the setting information 6. For example, in FIG. 2A, the character string to which the word information is added is enclosed in "{}", the part of speech is enclosed in "<>", and the reading is enclosed in "()". When specifying the word information, the target character string and one or more pieces of word information are indispensable, but it is not necessary to describe all the word information registered in the word information specifying pattern. For example, in the example of FIG. 2, part-of-speech and reading are specified for the target character string “Nakajima”, but only reading is specified for “Kenji”.

As described above, when the word information is designated by the character string pattern, "\" is added immediately before the character used in the character string pattern as the morphological analysis target. For example, to analyze the European Union (EU) using the word information designating pattern of FIG. 2 (a), the European Union \ (EU \) is used as the input Japanese text.

FIG. 2B shows designation of word information by XML. The target character string is <wor based on XML.
The description is that the d> element, its reading is designated by the yomi attribute of the <word> element, and the part of speech is designated by the hinti attribute of the <word> element. Also in this case, one or more attributes may be designated in the <word> element, as in FIG.

Returning to FIG. 1, the morphological analysis 2 is performed for each sentence.

In the word dictionary search process 2-1, the grammar rule 4 is used.
All connectable word chains described in 1. are extracted from the word dictionary 5, and the word chain candidate sequence 2-2 is output.

The word chain candidate sequence 2-2 is composed of a part-of-speech chain and words. For example, in FIG. 5, what is connected by a line is a part-of-speech chain (parts of speech are in <>), and one or more words are present in each part-of-speech chain.

Next, the word selection process 2-3 will be described with reference to FIG.
Will be described in detail.

In step 10, the word chain candidate sequence 2-2
Is prioritized, and the word chain with the highest priority (the word with the highest priority in the part-of-speech chain with the highest priority) is set as the first solution.

In step 11, Japanese text 1
A character string to which word information is given, or a word registered in the registered word information of the setting information 6 (hereinafter referred to as “target character string”).
Is abbreviated). If it exists, the process proceeds to step 12. Here, when there are a plurality of target character strings, the processing from step 12 onward is sequentially performed for each target character string. If it does not exist, the process ends.

In step 12, it is determined whether or not a word that exactly matches the target character string exists in the part-of-speech chain having the highest priority given in step 10. If it exists, the process proceeds to step 16. Step 1 if not present
Move to 3.

Here, the "perfect match" will be described more specifically.

The term "exact match" refers to a word in which the target character string and the word boundary completely match and satisfies all the specified word information, and the target character string and the word boundary are different, but the specified word information is Including words that meet all.

For example, "{Yamada} (Yamada) Taro" (using the designated pattern in FIG. 2A) is input as the Japanese text 1 (that is, only the reading of "Yamada" is designated),
If the word "Yamada Taro (yomi = Yamada Taro)" is included in the first part-of-speech chain, the reading from the beginning matches "Yamada" and there is no other word information specified, so this word Is treated as an exact match.

On the contrary, "{Yamada Taro} (Yamada Taro)" is input as the Japanese text 1, and "Yamada (Yomi = Yamada)" + "Taro (Yomi = Yamada)" is entered in the word chain candidate sequence 2-2. If the word chain "Tarou)" is included in the first part-of-speech chain, the notation of these two words matches the target character string, and the reading also matches, so this word chain is treated as a completely matching word (chain). .

In step 13, it is determined whether or not there is a word whose word boundary matches the word boundary and, where part-of-speech is specified, which word also matches the part-of-speech, in the word chain candidate sequence 2-2. If it exists, the process proceeds to step 14, and if it does not exist, the process proceeds to step 15.

In step 14, if a part of speech is designated for the target character string, and if there is a word that matches the part of speech and also satisfies the designated word information, that word is selected. If the other specified word information is different,
The word is copied including the word chain, and the word information specified elsewhere is overwritten.

If the part of speech is not specified for the target character string, the word of the word chain having the highest priority in step 13 is duplicated including the part of speech chain, and the word information specified elsewhere is reproduced. Overwrite.

In step 15, if a part of speech is specified for the target character string, of the part-of-speech chains having the boundary of the target word string as a word boundary, the part of speech that can be chained to the part of speech specified at that boundary is the most chainable. The part-of-speech chain before and after the chain of high priority is copied, and the target word string fills in the word information designated as one word. The word information that is not specified is generated from the word of the original part-of-speech chain (if there are multiple words, the word with the highest priority) (for example, all readings are connected). If there is nothing that can be chained, a chain with an unknown word is generated.

When no part of speech is specified, the word chain with the highest priority is copied from the part of speech chain having the boundary of the target word string as the word boundary, and the target word string is the word specified as one word. Fill in the information. The word information that is not specified is generated from the information of the original word.

In step 16, the priority of the word (chain) handled in the immediately preceding process is raised, and it is selected in the first place.

Returning to FIG. 1, grammar rule 4 describes a grammar for generating a part-of-speech chain and giving a priority.

The word dictionary 5 registers the words recognized by the morphological analysis and their word information.

The setting information 6 is a word information designation pattern (see FIG. 2) that describes a target character string to which word information is assigned in the Japanese text 1 and a designation method of the word information to be given, and once the word information is designated. In addition, it is composed of word storage information indicating whether or not to use it as registered word information, notation of a word for which the word information is designated, and registered word information for storing the designated word information.

When the word storage information of the setting information 6 is turned on in step 17, the word for which the word information is designated is registered in the registered word information in step 18, and is not registered when it is off. Further, the designated word information is canceled by deleting the word information from the registered word information.

The Japanese word information sequence 7 is a finally selected word chain sequence. In some cases, only the first solution may be narrowed down, or in some cases, the top several word chain sequences may be presented.

FIG. 4 shows an example of Japanese text 1 using the word information designating pattern of FIG. 2 (a).

The word selection process 2-3 will be described below with reference to the example of FIG. Here, first, the adjacency cost of the part of speech is used to assign the priority, and the word cost is used for selection when there are a plurality of words having the same adjacency cost (the smaller the adjacency cost and the word cost are prioritized). ).

Further, it is assumed that the grammar rule 4 describes that proper nouns: place names and proper nouns: names cannot be connected, symbols and proper nouns, and proper nouns and symbols can be connected.
Further, connection to an unknown word is allowed only when there is no other word to connect.

The word storage information of the setting information 6 is turned on.

For the first sentence, since there is no designation of the target character string, word selection is performed by the normal priority assignment in step 10, and the process is terminated.

In the second sentence, there are two designations of the target character string, that is, the reading "Nakashima" for "Nakajima" and the reading "Takeshi" for "Kenji". The word chain candidate sequence 2-2 at this time is shown in FIG. here,
The flow of FIG. 3 will be described in detail by taking “Nakajima” as an example.

In step 10, the solid part-of-speech chain (first part-of-speech chain sequence) having a small total adjacency cost is traced to "Kenji".
For, select a word that has a low word cost and is read as “kenji”, and give the priority with the space symbol Nakajima proper noun: surname Kenji Nakajima proper noun: first name Kenji personal name suffix.

The process proceeds from step 11 to step 12, and the word "Nakajima" contained in the first part-of-speech chain sequence has only the reading "Nakajima" (the reading is not "Nakashima"). Move to 13.

In step 13, the part of speech is not specified in the target character string "Nakajima" (satisfies the part of speech), and the word "Nakajima" exists, so the process proceeds to step 14.

In step 14, the word chain with the highest priority is "Nakajima (Yomi = Nakajima, Part of speech = Proper noun:
Surname) ”, this word is duplicated, the reading is replaced with the designated“ Nakashima ”, and the process proceeds to step 16 (see FIG. 6A).

In step 16, Nakajima proper noun: surname Nakashima Is set as the first solution, and the process proceeds to step 17.

The process proceeds from step 17 to step 18,
Register "Nakajima (reading = Nakashima)" in the registered word information.

For "Kenji", in step 12, there is a word in the first part-of-speech chain that exactly matches the given reading "Takeshi"("Kenji, Yomi = Takeshi"), so it is selected. Then, the process proceeds to step 16.

In step 16, instead of "Kenji, Yomi = Kenji", "Kenji, Yomi = Takeshi" is selected as the first priority word.

And finally, Space symbol Nakajima proper noun: surname Nakashima Kenji proper noun: Takeshi Natake Mr. name suffix Is output as the first solution.

In the third sentence, the word "Nakajima" at the beginning of the sentence matches the word registered in the registered word information, so the word with the reading "Nakashima" is given as the first solution in the same flow as in the second sentence. Become.

The flow of FIG. 3 will be described in detail below for the "interspace" in which the part of speech is designated. The word chain candidate sequence 2-2 in this portion is shown in FIG.

In step 10, the solid line part-of-speech chain (= word chain, which consists of one word) having a small total adjacency cost is the first solution.

The process proceeds from step 11 to step 12, and since there is no word in the first part-of-speech chain that completely matches the proper noun, which is the part of speech given to the given word information "interspace", the process proceeds to step 13.

In step 13, "interspace"
Since there is no word having a proper noun, which is the part of speech given to, the process proceeds to step 15.

In step 15, the front boundary of "inter (prefix)" and the rear boundary of "space (noun)", which is the word chain of the first solution, match the boundaries before and after the target character string "interspace". , In grammar rule 4, the symbol (")
Since a chain of a proper noun and a proper noun and a symbol (") is allowed, this part-of-speech chain of the first solution is copied and connected to a new word" interspace "(FIG. 6 (b)).
The reading of "interspace" is a combination of the readings of "inter (prefix)" and "space (noun)" to be "interspace".

In step 16, the first part-of-speech chain up to now is replaced where a thick part-of-speech chain has occurred, and the interspace proper noun interspace is the first solution.

The process proceeds from step 17 to step 18,
"Interspace (part of speech = proper noun)" is registered in the registered word information.

With respect to the fourth sentence, since there is no designation of the target character string, word selection is performed by the normal priority assignment in step 10, and the process ends.

FIG. 7 shows the configuration when the document analysis method of the present invention is carried out on a computer such as a personal computer.

The input device 21 is an input device such as a keyboard for inputting the Japanese text 1. Storage device 2
2, 23, and 24 have grammar rule 4, word dictionary 5, and
The setting information 6 is stored. The storage device 25 is a hard disk. The output device 26 is an output device such as a printer or a display that outputs a Japanese word information string.
The recording medium 27 is a recording medium such as a floppy disk, a CD-ROM, a magneto-optical disk, or the like in which a document analysis program consisting of word dictionary search 2-1 and word selection 2-3 is recorded. The data processing device 28 includes a CPU and various interfaces, reads the document analysis program from the recording medium 27, and executes it.

As described above, according to the present invention,
For some words in the input text, you can specify word information such as reading, part of speech, accent type, etc., and give priority to the part of speech chain of the grammatical rules that have been specified in advance, and specify words that satisfy the specified word information. By selecting or creating a new one if it does not exist, the specified word information is satisfied, and the unspecified word information is identified as the first solution with the word with the most probable information, and the specified word is retained once. By performing morphological analysis that certifies words satisfying the specified condition until it is canceled, the word recognition accuracy can be improved by positively providing known information.

[Brief description of drawings]

FIG. 1 is a diagram showing a morphological analysis method according to an embodiment of the present invention.

FIG. 2 is a diagram showing an example of a word information designation pattern and an example of Japanese text corresponding thereto.

FIG. 3 is a flowchart showing processing of word selection processing 2-3.

FIG. 4 is a diagram showing an example of Japanese text.

FIG. 5 is a diagram showing an example of a word chain candidate sequence and priorities given by normal priority assignment.

FIG. 6 is a diagram showing an example of adding generated words to a word chain candidate sequence.

FIG. 7 is a block diagram of an apparatus for implementing the morpheme search method of the present invention.

[Explanation of symbols]

1 Japanese text 2 Morphological analysis 2-1 Word dictionary search processing 2-2 Word chain candidate sequence 2-3 Word selection processing 3 Japanese word information string 4 Grammar rules 5 word dictionary 6 setting information 10-18 steps 21 Input device 22-25 storage device 26 Output device 27 recording media 28 Data processing device

─────────────────────────────────────────────────── ─── Continuation of front page (58) Fields surveyed (Int.Cl. ⁷ , DB name) G06F 17/21-17/28 JISST file (JOIS)

Claims (6)

(57) [Claims] 1. A morphological analysis method for dividing a Japanese text into words by a data processing device and adding word information, wherein the data processing device inputs the Japanese text from an input device to generate a part-of-speech chain and A word stored in a storage device in which a word and its word information for which a connectable word chain described in a grammatical rule stored in the storage device for giving priority is identified by morphological analysis are registered. A word dictionary search step of extracting from the dictionary, creating a word chain candidate string , and outputting the word chain candidate string to the storage device; and the data processing device , for the character string for which the word information is specified, the specified word The word information that satisfies the information and is not specified is identified as the first solution for the word that has the most probable information, and the word for which the word information is given is specified,
Register in the setting information stored in the storage device that describes the target character string to which the word information is added in Japanese text and the method of specifying the word information to be output, and output the Japanese word information string
A word selection step of outputting to a device; in the word selection step, the data processing device assigns a priority to the word chain candidate sequence , and sets a chain having the highest priority as a first solution. And whether the data processing device has a character string to which the word information is given or a target character string which is a word registered in the registered word information of the setting information in the Japanese text. A second step of determining, and the data processing apparatus, when the target character string exists, determining whether or not a word that exactly matches the target character string exists in the part-of-speech chain having the first priority. In step 3 and the data processing device, if there is no word that exactly matches the target character string, the word boundary matches the target character string, and if there is a part-of-speech designation, the part-of-speech also matches. Fourth, determining whether a word is present in the word chain candidate sequence
And if there is a word whose word boundary matches the target character string, if the part of speech is specified for the target character string, then the data processing device matches the part of speech and specifies another. If there is a word that also satisfies the specified word information, select that word, and if other specified word information is different, duplicate that word including the word chain, and specify the other word information. If no part of speech is specified for the target character string, the word of the word chain with the highest priority in the fourth step including the part-of-speech chain is duplicated, and another word is specified. The fifth step of overwriting existing word information, and when the data processing device specifies a part of speech for a target character string, of the part of speech chain having a boundary of the target word string as a word boundary, Allows chaining of parts of speech specified by boundaries Copy the part-of-speech chain before and after the highest-priority chain, fill the word information specified as one word in the target word string, and for the word information that is not specified, from the word of the original part-of-speech chain, If there is a word, the word with the highest priority is generated, if there is no chainable one, a chain with an unknown word is generated, and if no part of speech is specified, the target word is generated. Of the part-of-speech chains that have column boundaries as word boundaries, the word sequence with the highest priority is copied, the target word string fills in the word information specified as one word, and the word information that is not specified is the original word. The sixth step of generating from the information of No. 7, and the data processing device increases the priority of the words handled in the third, fifth, and sixth steps, and selects it in the first rank. Shape with steps Element analysis method. 2. The word selection step comprises the data processing.
When the device makes a similar specification for the text after the word information is specified once in the input Japanese text, saves that information, and when canceling the specification, the The method of claim 1, wherein the information is deleted. 3. The word selection step includes the data processing.
If the processing device has the word storage information of the setting information turned on,
The method according to claim 1, further comprising the step of registering a word for which word information is designated in the registered word information. 4. A morphological analysis is performed to identify connectable word chains described in grammatical rules stored in a storage device for inputting Japanese text, generating a part-of-speech chain, and assigning priorities. words have created the word information is extracted from the word dictionary stored in the storage device, to create a word concatenation candidate sequence, a word dictionary retrieval processing to output, to string word information is designated, designated The word information that satisfies the specified word information and is not specified is identified as the first solution for the word that has the most probable information, and the word information is given in Japanese text according to the specified word information. This is a word selection process of registering in the setting information stored in the storage device, which describes the target character string to be given and the method of specifying the given word information, and outputs the Japanese word information string.
A priority sequence for the word chain candidate sequence, and a sequence having the highest priority as the first solution; and a character string in which the word information is given to the Japanese text, or A second procedure of determining whether or not a target character string that is a word registered in the registered word information of the setting information exists; and, if the target character string exists, a word that exactly matches the target character string is A third step of determining whether or not the priority is in the first-ranked part-of-speech chain, and if there is no word that exactly matches the target character string,
A fourth step of determining whether or not a word whose word boundary matches the target character string and whose part of speech also matches is present in the word chain candidate string, and the target character string and the word If there are words whose boundaries match, if a part of speech is specified for the target character string,
If there is a word that matches the part-of-speech and also satisfies the other specified word information, select that word, and if the other specified word information is different, duplicate that word including the word chain. , If the word information specified elsewhere is overwritten and no part of speech is specified for the target character string, the fourth
In the procedure of step 5, the word with the highest priority word chain is duplicated including the part-of-speech chain, and the fifth step of overwriting other specified word information, and the part-of-speech is specified for the target character string. , The part of the chain of speech that has the boundary of the target word string as the word boundary,
Copy the part-of-speech chain before and after the chain of highest priority that can be chained with the part-of-speech specified at that boundary, fill the word information specified as one word in the target word string, and for the word information not specified, If there are multiple words from the original part-of-speech chain, the word with the highest priority is generated, and if there is no chainable word, a chain with an unknown word is generated. If not specified,
Of the part-of-speech chains that have the boundary of the target word string as the word boundary, the word chain with the highest priority is copied, the target word string fills the word information specified as one word, and the word information that is not specified is the original word string. of a sixth step of generating from the word information, third, fifth, raising the priority of the word addressed in the sixth step, a seventh step to make it to select the first of
Computer readable recording morphological analysis program for causing computer to execute word selection processing including
Capacity recording medium. 5. In the word selection process, when a word string is specified once in a Japanese text to be input, and when subsequent texts are also specified, the information is saved and specified. The computer-readable recording medium according to claim 4, wherein the information is deleted when the recording is canceled. 6. The word selection process further comprises the step of registering a word for which word information is designated in the registered word information when the word storage information of the setting information is ON.
The computer-readable recording medium according to claim 4.