JP4431759B2 - Unregistered word automatic extraction device and program, and unregistered word automatic registration device and program - Google Patents

Description

  The present invention relates to a dictionary creation technique for natural language processing such as machine translation, and in particular, unregistered word automatic extraction that can automatically extract or register unregistered words that are not registered in a dictionary for translation, etc. It relates to a registration device.

  A machine-readable translation dictionary is indispensable for machine translation. It can be expected that the greater the number of words registered in the translation dictionary, the higher the translation accuracy. However, in order to maintain the dictionary, it takes time and effort to collect documents, search for words that are not registered in the dictionary, search for translated words for the words, and register them in the dictionary.

  There are three stages of work described above. The first stage is the task of finding unregistered words. The second stage is the task of specifying the translation. The third stage is an operation for registering additional information such as unregistered words and translated words, part-of-speech classification, and semantic classification in the dictionary. For example, if translation fails due to a deficiency in the dictionary in an automatic translation system, the rewritten translation and the original text are matched, the original text of the rewritten part is located, and the word or phrase can be registered in the dictionary. It is possible to register a dictionary only after adding additional information such as part of speech or semantic classification. If such work is not continued properly, for example, the automatic translation system will also fail to translate at the same part, and rewrite will be necessary again. Conventionally, even if the translation work is automated, the overall work including rewrite and dictionary registration work is often not so effective in improving efficiency.

  For this reason, as one measure for improving the efficiency, a technique for automatically registering unregistered words in the translation dictionary is considered necessary. Such a function is hereinafter referred to as “automatic registration”.

As an automatic registration technique, there is one described in Patent Document 1 below. The technique described in Patent Literature 1 can be applied between Japanese and Chinese, limited to proper nouns. In this technology, a Japanese morpheme dictionary and a Japanese-Chinese translation dictionary are prepared, and when a Japanese word is registered in the Japanese morpheme dictionary but not registered in the Japanese-Chinese translation dictionary, Register the word in the Japanese-Chinese translation dictionary. That is, the registration is automated for only a part of the original words and translated words that are not registered in the translation dictionary.

Patent Document 2 discloses a technique for automatically registering a Japanese-English translation dictionary into a dictionary limited to pairs of katakana and English. In the technique disclosed in Patent Document 2, an English dictionary and a Japanese-English translation dictionary are prepared, and when an English word is registered in the English dictionary but not registered in the Japanese-English translation dictionary, the English And the corresponding katakana words are registered in the Japanese-English translation dictionary as new headings.
JP-A-4-256171 Patent No. 2995783 JP 2003-6193 A

  By the way, new words are required and used due to technological progress, etc. Of course, new words are created one after another in order to come up with “freshness” even if not necessary. This tendency is getting stronger with the generalization and everyday use of information transmission media such as television broadcasting and mobile phones. In the case of a new word that tries to produce such “fresh taste”, it is created by intentionally avoiding a word that is registered in an ordinary dictionary, synthesizing it, omitting it, or diverting it. Although there is a sense that new words created by so-called Japanese English or composition from multiple languages are abused, it is very difficult to avoid these new words and establish real-world conversations or sentences.

  On the other hand, technological progress has greatly influenced so-called natural language processing technology that attempts to support human activities by processing such languages. For example, a storage device for configuring a dictionary necessary for processing as a device has a capacity ratio of 1 / 100,000 or less compared to the mid-1980s when machine translation devices began to be commercialized in Japan. . In other words, a translation dictionary constructed using the capacity equivalent to 1 million yen of the storage device at the time of 1980 can now be constructed using the capacity equivalent to 10 yen of the storage device.

  This completely obsoletes the “common sense” of the time, such as limiting the words registered in the dictionary in order to reduce the capacity of the storage devices that make up the dictionary. That is, how to construct a dictionary that is registered in the Japanese morpheme dictionary but not registered in the Japanese-Chinese translation dictionary, or registered in the English dictionary but not registered in the Japanese-English translation dictionary Even if this can save some capacity, the cost reduction effect is too small for the disadvantages experienced by users.

  For this reason, under current circumstances, automatic registration that eliminates irregularities in dictionary registration, such as unregistered words in only some dictionaries, as in the past, is a natural process such as machine translation. There is a problem that it has only a limited effect on the usefulness of language processing.

  Therefore, an object of the present invention is to automatically extract unregistered words that can automatically extract unregistered words that are not registered in the dictionary, and unregistered that can automatically register such unregistered words in the dictionary. It is to provide an automatic word registration device.

  The unregistered word automatic extraction device according to the first embodiment of the present invention includes a rewrite pattern between a first language sentence and a second language sentence, a first language, and a second language. An unregistered word automatic extraction device that automatically extracts unregistered words in a machine-readable dictionary of a first language based on a plurality of parallel translation example sentences between a rewriting pattern, a bilingual example sentence, A sample sentence / rewrite pattern matching means for extracting a parallel translation that matches the rewrite pattern from the bilingual example sentence, and specifying a character string constituting the variable part in the sentence of the first language of the parallel translation; The character string specified by the example sentence / rewrite pattern matching means is searched in the dictionary, and the dictionary search means for determining whether or not the character string is registered, and the dictionary search means determines that it is not registered in the dictionary. For a first language character string, a pair including the first language character string. And the second language character string corresponding to the first language character string, together with the first language character string, as information to be registered in the dictionary based on the rewriting pattern matched with the parallel translation Registration information specifying means for doing this.

  The example sentence / rewrite pattern collating unit collates the rewrite pattern with the parallel translation example sentence, and extracts a character string constituting the variable part in the sentence of the first language from the parallel translation. The dictionary search means searches for this character string in the dictionary and determines whether it is registered. If not registered, the registered information specifying unit specifies the character string of the second language corresponding to the character string. The character string of the first language and the character string of the second language specified as corresponding to the character string are automatically extracted as information to be registered in the dictionary. In this process, it is not necessary that some information regarding the character string to be extracted is registered in the dictionary. There is no need to manually extract such a pair of character strings, and the labor for specifying a headline to be registered in the dictionary is greatly improved. As a result, it is possible to provide an unregistered word automatic extraction device that can automatically extract unregistered words that are not registered in the dictionary at all.

  Preferably, the unregistered word automatic extraction device associates the character string of the first language determined to be registered in the dictionary by the dictionary search unit with the rewrite pattern together with the information registered in the dictionary, and the variable part information. Further includes variable part information storage means. The registered information specifying means includes, for the first language character string determined not to be registered in the dictionary by the dictionary search means, a bilingual translation including the first language character string and a rewriting pattern in which the bilingual match. Based on this, there is a translation specifying means for specifying a character string of the second language to be registered in the dictionary as a translation corresponding to the character string of the first language, and a parallel translation including the character string of the first language. An additional information specifying means for specifying additional information to be registered in the dictionary with respect to the character string of the first language based on the variable part information stored in the variable part information storing means in association with the matched rewrite pattern; Including.

  Bilingual example sentences that match the same rewrite pattern are considered to relate to information of a common type. In other words, information to be added to a character string specified by matching a certain rewrite pattern is considered to be similar to information extracted from a dictionary that is another character string that matches the rewrite pattern. It is done. Therefore, the information specified by the additional information specifying means as described above is considered to be valid information regarding the first language character string specified first, and is added to the character string of the first language. By creating information to be registered in the dictionary, information to be registered in the dictionary can be further enhanced.

  More preferably, the additional information specifying means associates the union of the variable part information stored in the variable part information storage means in association with the rewrite pattern matched with the parallel translation including the character string of the first language in the first language. Means for specifying as additional information regarding the character string.

  By creating a union of information that should be added to a specified character string by matching a certain rewrite pattern, it is possible to create leak-free information that is valid even for the first specified character string of the first language. it can.

  The unregistered word automatic extraction device further includes an appearance count counting unit for counting the number of appearances of the character string every time a character string of the first language appears in the result of collation by the example sentence / rewrite pattern collation unit. The additional information specifying means may include the appearance count counting means among the variable part information stored in the variable part information storage means in association with the rewrite pattern in which the parallel translation including the character string of the first language is matched. Means may be included for specifying the union of the counted number of appearances as a predetermined number or more as additional information related to the first language character string.

  By taking the union of information added to a character string that appears a predetermined number of times, information extracted from the dictionary regarding character strings that frequently appear as character strings that match a specific rewrite pattern can be obtained. It is considered that the information accompanying the character string well reflects the property of the character string that matches the rewrite pattern. Therefore, by adding the union information to the specified character string of the first language, it is possible to specify the additional information regarding the character string of the first language and accurate information.

  Preferably, the unregistered word automatic extraction device counts the number of appearances for counting the number of appearances of the character string every time a character string of the first language appears in the result of collation by the example sentence / rewrite pattern collation unit. Means are further included. When the variable part information stored in the variable part information storage unit in association with the rewrite pattern in which the parallel translation including the character string in the first language is matched is larger than a predetermined threshold, the additional information specifying unit Means for specifying the union of the appearances counted by the counting means as the additional information regarding the character string of the first language is included.

  When there are a large number of variable part information associated with the rewrite pattern that matches the parallel translation including the character string of the first language, the union of those appearing more than a predetermined number of times is taken. Therefore, it can be expected that accurate information is obtained as information on the character string of the first language. On the other hand, when there is a small number of variable part information associated with the rewrite pattern, comprehensive information with as little leakage as possible can be obtained by taking the union of them.

  The unregistered word automatic extraction program according to the second aspect of the present invention, when executed by a computer, causes the computer to operate as any one of the unregistered word automatic extraction devices described above. Therefore, the same effects as those of the unregistered word automatic extraction device can be obtained.

  The unregistered word automatic registration device according to the third aspect of the present invention includes any of the above-described unregistered word automatic extraction devices and the information specified by the unregistered word automatic extraction device as being registered in the dictionary. Dictionary registration means for registering in the dictionary.

  The unregistered word automatic registration device can automatically extract a character string to be registered in the dictionary and information such as a translated word for the character string and automatically register it in the dictionary. Processing such as picking up a character string to be registered, specifying a translation, part-of-speech classification or meaning classification, creating information to be added to the character string, and registering it in a dictionary is automatically performed. As a result, it is possible to provide an unregistered word automatic registration device that can automatically extract unregistered words that are not registered in the dictionary and register them in the dictionary.

  The unregistered word automatic registration program according to the fourth aspect of the present invention, when executed by a computer, causes the computer to operate as the above-described unregistered word automatic registration device. Therefore, the same effects as those of the unregistered word automatic registration device can be obtained.

[Outline]
In one embodiment of the present invention to be described below, a complete unregistered word is automatically extracted as a phrase from a previously translated bilingual sentence, and a translation word and semantic information are added and registered in a translation dictionary. Provide a mechanism that can This mechanism eliminates the need for a series of dictionary registration tasks such as searching for unregistered words, determining translated words, and assigning semantic classifications, improving work efficiency in systems that perform natural language processing such as automatic translation systems. can do.

[Functional configuration]
Hereinafter, the configuration of the dictionary automatic registration apparatus according to the present embodiment will be described. In the following embodiment, an automatic translation system that performs Japanese-English translation is taken as an example for the sake of explanation. The source language is Japanese and the target language is English.

  FIG. 1 is a block diagram showing the configuration of an unregistered word automatic registration device 30 according to the present embodiment. Referring to FIG. 1, an unregistered word automatic registration device 30 includes a storage unit 32 for storing a rewrite pattern used for automatic translation, a Japanese and English prepared in advance, which is included in a translation system that performs Japanese-English translation. Are used together with a translation dictionary 36 included in the translation system described above, and unregistered words (character strings) are extracted from the example sentences in the translation example sentence group 34 and stored in the dictionary 36. According to this method, additional information such as translated words and semantic information is added and automatically registered in the dictionary 36.

  The unregistered word automatic registration device 30 includes a rewrite pattern extraction unit 50 for extracting a rewrite pattern from the storage unit 32, and a rewrite pattern storage unit 52 for storing the rewrite pattern extracted by the rewrite pattern extraction unit 50. Including.

  FIG. 2 shows a rewrite pattern 80 as an example of a rewrite pattern stored in the rewrite pattern storage unit 52. Referring to FIG. 2, the rewrite pattern 80 includes at least a Japanese original pattern 90 and an English translation pattern 92. Each pattern includes at least a fixed part having character string information, and position information with respect to the fixed part and a variable part having correspondence information between the variable part of the original pattern and the variable part of the translated pattern.

  In the example shown in FIG. 2, the original pattern 90 is “I sent ... to”, and among those, “I”, “to” and “Sent” are fixed parts. The translation pattern 92 is “I sent to to...”, Among which “I sent” and “to” are fixed portions. In both patterns, character strings 100 and 106 indicated by “...” And character strings 102 and 104 indicated by “˜” are variable parts. The variable parts having the same character string in the original sentence and the translated sentence correspond to each other. That is, the character strings 100 and 106 correspond to each other, and the character strings 102 and 104 correspond to each other. Further, the variable part has position information with respect to the fixed part by being inserted at a specific position in the character string of the fixed part in both the original sentence pattern and the translated sentence pattern.

  As described above, the rewrite pattern includes a pair of an original pattern and a translated pattern. Each of the original sentence pattern and the translated sentence pattern includes a fixed character string (fixed part) and a variable part. The rewrite pattern is configured so that the positional relationship between each variable portion and the fixed portion can be understood. Furthermore, the rewrite pattern is configured so that the correspondence between the variable parts can be understood between the original sentence pattern and the translated sentence pattern.

  Referring again to FIG. 1, the unregistered word automatic registration device 30 is further connected to the rewrite pattern storage unit 52, and the rewrite example stored in the rewrite pattern storage unit 52 is stored in the rewrite pattern storage unit 52. Matches the pattern, extracts example sentences that match both the original text and the translated text to one of the rewrite patterns, and supports the variable part specified by the rewrite pattern in both Japanese and English sentences of the extracted parallel translation An example sentence / rewrite pattern matching unit 54 for performing processing for specifying a character string to be performed (hereinafter referred to as “variable part character string”), a variable part character string identified by the example sentence / rewrite pattern matching unit 54, The corresponding rewrite patterns are received and analyzed, and information (word, part of speech, semantic information, etc., hereinafter referred to as “variable part information”) regarding the variable part character string of the example sentence that matches the rewrite pattern is created or created. For updating Performs management, and a variable portion analyzer 56 for performing a process of storing the rewrite pattern storage unit 52 the variable portion information in association with the rewriting pattern. For this purpose, the variable part analysis unit 56 has a function of determining whether or not the variable part character string is registered in the rewrite pattern storage unit 52 as variable part information related to the corresponding rewrite pattern.

  The unregistered word automatic registration device 30 further determines that the variable part analysis unit 56 determines that the input variable part character string is not registered in the variable part information of the corresponding rewrite pattern in the rewrite pattern storage unit 52. In response, the variable part character string is set as an unregistered word, and unregistered word information (translation to the unregistered word and additional information such as part of speech and semantic information) is changed. The variable part information of the rewrite pattern in the rewrite pattern storage unit 52 The unregistered word is automatically registered in the dictionary 36 of the translation system together with the unregistered word registration information generating unit 58 for creating based on the unregistered word information generated by the unregistered word registration information generating unit 58. And a variable part character string determined not to be registered as variable part information for the rewrite pattern stored in the rewrite pattern storage part 52 by the dictionary registration part 60 and the variable part analysis part 56. Of and a registered word dictionary search section 62 for reading the information registered by searching all the dictionary 36 unregistered words automatic automatic translation system registration device 30 is incorporated.

[Computer program structure]
As will be described later, the unregistered word automatic registration device 30 can be realized by a computer and a computer program executed on the computer. Computer program for realizing an unregistered word automatic registration process comprising an example sentence / rewrite pattern collation unit 54, a variable unit analysis unit 56, an unregistered word registration information generation unit 58, a dictionary registration unit 60, and a registered word dictionary search unit 62 The control structure will be described with reference to FIGS. The processing of the rewrite pattern extraction unit 50 can be executed independently of other processing, and the processing content is simple, so the details are omitted here.

  FIG. 3 shows an overall processing flow of this unregistered word automatic registration processing program. Referring to FIG. 3, when the execution of this program is started, first, collation example sentence group 34 shown in FIG. 1 is collated with the rewrite pattern stored in rewrite pattern storage unit 52, and either one of the rewrite patterns and the original text pattern are checked. A variable part comprising: processing for identifying example sentences that match in both the translation pattern and the translation pattern, extracting a variable part character string therein, and creating variable part information for each of the extracted variable part character strings Extraction / variable portion information creation processing is performed (step 120). When the variable part extraction / variable part information creation process is completed, among the variable part character strings extracted in step 120, the character strings that are not registered in the dictionary 36 of the translation system are both stored in the dictionary from the created variable part information. A process of registering in the dictionary 36 together with unregistered word information to be registered is performed (step 122). This is the end of the process.

  Next, details of the variable part extraction / variable part information creation process executed in step 120 of FIG. 3 will be described with reference to FIG. First, at step 140, initial setting is performed. In this initial setting, the work area is cleared. In step 142, the rewrite pattern stored in the rewrite pattern storage unit 52 is read. In step 144, it is determined whether or not reading of all rewrite patterns has been completed. If all the rewrite patterns have been read, this process ends. If all reading has not been completed, the process proceeds to step 146.

  In step 146, the example sentences in the parallel translation example sentence group 34 are read. In step 148, it is determined whether or not all example sentences have been read. If all have been read, control returns to step 142 to perform processing for the next rewrite pattern. If all are not read, the process proceeds to step 150.

  In step 150, it is determined whether or not the read example sentence matches the current rewriting pattern. This determination is made for both of whether the original sentence of the example sentence matches the original sentence pattern and whether the translated sentence of the example sentence matches the translated sentence pattern. If there is no match, control returns to step 146 to process the next example sentence. If there is a match, control proceeds to step 152.

  In step 152, a character string (variable part character string) that matches the variable part of the rewrite pattern is specified in both the original sentence and the translated sentence in the example sentence.

  Referring to FIG. 5, in step 182, it is determined whether or not this variable part character string is registered in the variable part information of the rewrite pattern in rewrite pattern storage unit 52. If registered, in step 184, 1 is added to the number of occurrences of the character string, and the process returns to step 146 in FIG. If not registered, the process proceeds to step 186.

  In step 186, the character string is searched in the dictionary 36 of the translation system. In step 188, it is determined whether this character string is registered in the dictionary 36 of the translation system. If registered, in step 190, the variable part character string and the result searched for the character string from the dictionary 36 of the translation system are used as variable part information of the rewrite pattern in the rewrite pattern storage unit 52. to add. Thereafter, the control returns to step 146. On the other hand, if it is determined in step 188 that the character string is not registered in the dictionary 36, information indicating unregistration is added to the variable part character string in step 192, and the corresponding character in the rewrite pattern storage unit 52 is added. The variable part information accompanying the rewrite pattern is added to the rewrite pattern storage unit 52. At this time, the number of appearances related to the variable part information is initialized to “1”. Control then returns to step 146 and the same processing is repeated for the next example sentence.

  Thus, when the processing shown in FIGS. 4 and 5 is completed for the combinations of all the rewrite patterns and all the example sentences, the control shifts to the dictionary update processing shown in step 122 of FIG. The details are shown in FIG. 6, steps 202 to 228 correspond to the unregistered word registration information generation unit 58 shown in FIG. 1, and step 230 corresponds to the dictionary registration unit 60.

  Referring to FIG. 6, in the dictionary update process, initial setting is first performed in step 200. In this initial setting, the work area is cleared, a list of words to be registered in the dictionary 36 (unregistered word list), and the like are performed. In step 202, the rewrite pattern stored in the rewrite pattern storage unit 52 is read. In step 210, it is determined whether reading has been completed for all rewrite patterns. If completed, control proceeds to step 230. Otherwise, control proceeds to step 212.

  In step 212, it is determined whether or not there is a variable part character string with information indicating unregistered in the variable part information accompanying the rewrite pattern. If not, control returns to step 202 and the process is repeated for the next rewrite pattern. If there is a variable part character string with information indicating unregistered, the control proceeds to step 214.

  In step 214, the variable part character string to which information indicating unregistration is attached is read. In step 216, it is determined whether or not all of the variable part character strings to which information indicating unregistration has been read out of the variable part information related to the rewrite pattern. If all have been read, control returns to step 202, and processing is executed for the next rewrite pattern. If all of them have not yet been read, a process for specifying a translation for the variable part character string is executed in step 218.

  Details of the processing performed in step 218 will be described. As shown in FIG. 2, the rewrite pattern includes an original sentence pattern and a translated sentence pattern, and the variable part thereof includes information indicating the correspondence between the variable part character strings. Using this correspondence relationship, it is possible to determine which character string corresponds to the variable part character string of the original sentence among the variable part character strings of the translated sentence. This character string can be considered as a translation for the original variable part character string. In step 218, the translation word for the variable part character string of the original text is specified using the rewrite pattern in this way.

  Subsequently, in step 220, it is determined whether or not the number of variable part information accompanying the rewrite pattern is larger than a predetermined threshold value. If the number of variable part information is larger than the threshold value, the process proceeds to step 226, and if not, the process proceeds to step 224.

  In step 226, among the variable part information accompanying the rewrite pattern, the union of information such as part-of-speech classification and semantic classification included in the variable part information having the number of appearances equal to or greater than a certain value is taken, and this is taken as the variable part information. Registered information related to character strings. On the other hand, in step 224, a union of information such as part-of-speech classification and semantic classification included in all variable part information is taken and used as registration information regarding the variable part character string.

  As described above, in the present embodiment, when there are a large number of variable part information associated with the rewrite pattern in which the parallel translation including the character string of the first language is matched, a union of those appearing a predetermined number of times or more is obtained. Take. Information obtained by taking the union of information to be added to a character string that frequently matches a certain rewrite pattern is highly likely to be valid for the first-specified character string of the first language. Therefore, it can be expected that accurate information is obtained as information on the character string of the first language. On the other hand, when there is a small number of variable part information associated with the rewrite pattern, comprehensive information with as little leakage as possible can be obtained by taking the union of them.

  Of course, such a process may not be performed, and the union of all variable part information may be taken at any time. Even in this case, by taking the union of the information to be added to the specified character string by matching a certain rewrite pattern, the leakage of the first language specified first is also valid. It is thought that no information can be created.

  Thereafter, the control proceeds to step 228, and the variable part character string and unregistered word information attached to the character string are added to the unregistered word list. Thereafter, the control returns to step 214 to execute processing for the next variable part information associated with the rewrite pattern.

  In this way, when the processing of step 202 to step 228 is executed for all variable part information of all rewrite patterns, the control moves from step 210 to step 230.

  In step 230, a process of adding to the dictionary 36 of the translation system from the top of the unregistered word list to which the unregistered word information is added in step 228 is executed. When this process ends, the dictionary update process ends.

[Computer hardware configuration]
FIG. 7 shows an example of the external appearance of a computer system that executes the above-described computer program, and FIG. 8 shows an example of a block diagram thereof.

  Referring to FIG. 7, the computer system 330 includes a computer 340 having an FD (flexible disk) drive 352 and a CD-ROM (compact disk read only memory) drive 350, a keyboard 346, a mouse 348, and a monitor 342. including.

  Referring to FIG. 8, in addition to the FD drive 352 and the CD-ROM drive 350, the computer 340 includes a CPU (central processing unit) 356 and a bus 366 connected to the CPU 356, the FD drive 352, and the CD-ROM drive 350. And a read only memory (ROM) 358 for storing a boot-up program and the like, and a random access memory (RAM) 360 connected to the bus 366 for storing a program command, a system program, work data, and the like. Computer system 330 further includes a printer 344.

  Although not shown here, the computer 340 may further include a network adapter board that provides a connection to a local area network (LAN).

  A computer program for causing the computer system 330 to operate as the unregistered word automatic registration device 30 shown in FIG. 1 is stored in the CD-ROM 362 or FD 364 inserted in the CD-ROM drive 350 or FD drive 352, and Transferred to the hard disk 354. Alternatively, the program may be transmitted to the computer 340 through a network (not shown) and stored in the hard disk 354. The program is loaded into the RAM 360 when executed. The program may be loaded directly into the RAM 360 from the CD-ROM 362, from the FD 364, or via a network.

  This program includes a plurality of instructions for causing the computer 340 to operate as the unregistered word automatic registration device 30 of this embodiment. Some of the basic functions required to perform this operation are provided by operating system (OS) or third party programs running on the computer 340 or various toolkit modules installed on the computer 340. Therefore, this program does not necessarily include all functions necessary for realizing the unregistered word automatic registration device 30 of this embodiment. This program calls only an instruction to execute an operation as the above-described unregistered word automatic registration device 30 by calling an appropriate function or “tool” in a controlled manner so as to obtain a desired result. It only has to be included. The operation of computer system 330 is well known and will not be repeated here.

[Operation]
The unregistered word automatic registration device 30 according to the present embodiment whose configuration has been described above operates as follows. Referring to FIG. 1, it is assumed that a storage unit 32 of the translation system and a dictionary 36 of the translation system are connected to the unregistered word automatic registration device 30 in advance. Also, it is assumed that the parallel translation example sentence group 34 is prepared in advance, and the parallel translation example sentences are mutually correct.

  First, the rewrite pattern extraction unit 50 extracts a rewrite pattern from the storage unit 32 of the translation system, and stores it in the rewrite pattern storage unit 52 in the format shown in FIG. When all the rewrite patterns are extracted from the storage unit 32 of the translation system and stored in the rewrite pattern storage unit 52, the rewrite pattern extraction unit 50 ends the operation.

  Subsequently, the example sentence / rewrite pattern matching unit 54 starts operating. Specifically, the example sentence / rewrite pattern matching unit 54 first reads the first rewrite pattern from the rewrite pattern storage unit 52. Further, the first example sentence for translation is read from the example sentence group 34 for translation. The example sentence / rewrite pattern matching unit 54 compares the read rewrite pattern with the example sentence for translation, the original sentence of the example sentence for translation matches the original pattern of the rewrite pattern, and the translation of the example sentence for translation is the translation pattern of the rewrite pattern. It is determined whether or not they match. If there is no such match, this bilingual example sentence will not match the rewriting pattern being processed. Therefore, the example sentence / rewrite pattern matching unit 54 reads the next example sentence for translation from the example sentence group 34 for translation, and executes the same processing.

  If the original text of the translation example sentence matches the original pattern of the rewrite pattern, and the translation of the example sentence for translation matches the translation pattern of the rewrite pattern, the example sentence / rewrite pattern matching unit 54 The variable part character string of the original sentence is extracted from the parallel translation example sentence using the example sentence. The example sentence / rewrite pattern matching unit 54 further determines whether or not the variable part character string of the original text is registered as variable part information associated with the rewrite pattern being processed. If registered, 1 is added to the number of appearances of the variable part information, and the process proceeds to the next example sentence for bilingual translation. If not registered, the following processing is executed.

  That is, the sample sentence / rewrite pattern matching unit 54 provides the extracted variable portion character string of the original text to the registered word dictionary search unit 62. The registered word dictionary search unit 62 searches the dictionary 36 of the translation system for a registered word including the variable part character string as a headline, and information regarding whether or not the character string is registered and Gives all of the retrieved information as information about the registered words to the variable part analysis unit 56.

  When the registered word dictionary search unit 62 returns information indicating that the variable part character string is not registered in the dictionary 36 of the translation system, the variable part analyzing unit 56 is information indicating that the variable part character string is not registered. And is additionally registered in the rewrite pattern storage unit 52 as variable part information accompanying the rewrite pattern being processed. If registered, the information given from the registered word dictionary search unit 62 is added to the variable part character string, and is registered in the rewrite pattern storage unit 52 as variable part information accompanying the rewrite pattern being processed.

  When collation of all the parallel translation example sentences for a certain rewrite pattern is completed, the example sentence / rewrite pattern collation unit 54 reads the next rewrite pattern from the rewrite pattern storage unit 52 and executes the same processing. When this process is completed for all the rewrite patterns, the variable part analysis unit 56 gives a signal indicating that the process has been completed to the unregistered word registration information generation unit 58 and ends the operation.

  When the unregistered word registration information generation unit 58 receives a signal indicating that the processing is completed from the variable unit analysis unit 56, it first initializes the unregistered word list. Further, the unregistered word registration information generation unit 58 scans the rewrite pattern storage unit 52 and sequentially reads the rewrite patterns having variable part information to which information indicating that they are not registered is added. The unregistered word registration information generation unit 58 uses the variable part information of the read rewrite pattern to generate unregistered word information for the variable part character string in the read variable part information as follows. . That is, the unregistered word registration information generation unit 58 includes the variable part that the variable part character string occupied in the original sentence among the character strings corresponding to the variable part included in the translation pattern of the rewrite pattern among the input parallel translations. The character string of the part corresponding to is specified as a translation for the variable part character string. Further, the unregistered word registration information generation unit 58 takes the union of the part-of-speech classification or the semantic classification included in the variable part information accompanying the rewriting pattern being processed, so that the part-of-speech classification and variable part character string being processed and Create information about semantic classification. When the number of variable part information accompanying the rewrite pattern is larger than the threshold value, information on the part-of-speech classification and the semantic classification is created using only the variable part information whose number of appearances is equal to or greater than a certain value. The unregistered word registration information generation unit 58 adds the unregistered word information thus created to the unregistered word list.

  The example sentence / rewrite pattern collation unit 54 repeats the above-described process for each of the variable part information to which a plurality of unregistered information is added in the rewrite pattern.

  The unregistered word registration information generation unit 58 further reads out the next rewrite pattern from the rewrite pattern storage unit 52 and repeats the same processing. When the process for all the rewrite patterns is completed, an unregistered word list is given to the dictionary registration unit 60 and a signal indicating that the unregistered word registration information generation process is completed is given to the dictionary registration unit 60.

  When a signal indicating that the unregistered word registration information generation process has been completed is received from the unregistered word registration information generation unit 58, the dictionary registration unit 60 receives the unregistered word list received from the unregistered word registration information generation unit 58. The variable part character string and its translation, part of speech classification, semantic classification, and other information are added to the translation system dictionary 36 in order from the beginning. When this process is completed for all unregistered word information included in the unregistered word list, the unregistered word automatic registration device 30 ends the operation.

[Specific operation example]
The operation of the above-mentioned unregistered word automatic registration device 30 will be described using a specific example. Here, a rewrite pattern having the original pattern “recently new technology has become a hot topic” as a rewrite pattern (this is referred to as “first rewrite pattern”) is extracted from the storage unit 32 of the translation system. It is assumed that it is registered in the rewrite pattern storage unit 52. In the parallel translation example sentence group 34, as a parallel translation example sentence matching the rewriting pattern, a parallel translation example sentence having the original sentence “a new technology called ubiquitous has recently become a hot topic” (this is the first parallel translation example sentence). And a bilingual example sentence (this is referred to as a second bilingual example sentence) having the original sentence “A new technology called the Internet has recently become a hot topic”. In the former, it is assumed that the variable part of the translation corresponding to the word “ubiquitous” includes “ubiquitous computing”. Further, “ubiquitous” is not registered in the dictionary 36 of the translation system, and “Internet” is registered. For the “Internet”, information such as part-of-speech classification “common noun” and semantic classification “technical term” is registered in the dictionary 36 of the translation system.

  The example sentence / rewrite pattern collation unit 54 reads the rewrite pattern stored in the rewrite pattern storage unit 52 and collates with the example sentence group for translation included in the example sentence group 34 for translation to extract the variable part character string of the original sentence. If the variable part information including the extracted variable part character string is already registered in the rewrite pattern storage unit 52 as the variable part information of the rewrite pattern being processed, the example sentence / rewrite pattern matching unit 54 stores the variable part information. Add 1 to the number of appearances and proceed to the next parallel translation. If the variable part information has not been registered in the rewrite pattern storage unit 52, the following processing is performed.

  When the example sentence / rewrite pattern matching unit 54 reads the first rewrite pattern, the first example sentence for translation included in the example sentence group for translation 34 first matches the original pattern of the rewrite pattern. Therefore, the example sentence / rewrite pattern matching unit 54 extracts the word “ubiquitous” from the example sentence for translation as a variable part character string. The variable part analysis unit 56 determines whether variable part information including this word is registered in the rewrite pattern storage unit 52 as being associated with the first rewrite pattern. It is assumed that it is not registered here. In this case, the variable part analysis unit 56 searches the dictionary 36 of the translation system using the registered word dictionary search unit 62, but the corresponding word is not registered and information cannot be obtained. Therefore, the variable part analysis unit 56 adds information indicating unregistration to the variable part character string, and additionally registers it in the rewrite pattern storage unit 52 as variable part information associated with the first rewrite pattern.

  The example sentence / rewrite pattern collation unit 54 further extracts the above-described second example sentence for translation from the example sentence group 34 for translation as the example sentence for translation that matches the first rewrite pattern. As a result of the variable part analysis part 56 searching the dictionary 36 of the translation system using the registered word dictionary search part 62 for the variable part character string, this word is registered as described above, and the information is stored in the registered word dictionary. The search unit 62 returns to the variable part analysis unit 56. This information includes information such as part-of-speech classification “common noun” and semantic classification “technical term”. The variable part analysis part 56 adds these pieces of information to the rewrite pattern storage part 52 as variable part information accompanying the first rewrite pattern together with the variable part character string “Internet”.

  In this way, many other parallel translation example sentences that match the first rewriting pattern are extracted, and it is checked whether or not the variable part character string is registered in the dictionary 36 of the translation system. For words registered in the dictionary 36, information obtained from the dictionary is added to the rewrite pattern storage unit 52 as variable part information accompanying the first rewrite pattern. The variable part information already registered in the rewrite pattern storage unit 52 is incremented each time the corresponding variable part character string is extracted from the bilingual example sentence group 34 by the example sentence / rewrite pattern matching part 54. To go.

  After the example sentence / rewrite pattern matching unit 54 finishes the process for all the rewrite patterns, the process by the unregistered word registration information generation unit 58 is started. The unregistered word registration information generation unit 58 scans the rewrite pattern storage unit 52 and extracts variable part information to which information indicating unregistration is added. Here, it is assumed that the unregistered word registration information generation unit 58 has extracted the word “ubiquitous”. The unregistered word registration information generation unit 58 applies the original sentence pattern and the translated sentence pattern included in the first rewriting pattern to the original bilingual example sentence from which the word is extracted, and changes the translation corresponding to “ubiquitous”. The character string “ubiquitous computing” is specified as the sub character string. Further, the variable part analysis unit 56 includes a part of speech classification or meaning included in the part registered as variable part information associated with the first rewrite pattern in the rewrite pattern storage unit 52 as part of speech classification and semantic classification for “ubiquitous”. A union such as a classification is obtained, and the result is set as a part-of-speech classification and a semantic classification for “ubiquitous”. If the number of variable part information associated with the first rewrite pattern is larger than a predetermined threshold, the unregistered word registration information generation unit 58 uses only the variable part information having a large number of appearances, Similarly, a union such as part-of-speech classification or semantic classification is obtained, and the result is defined as part-of-speech classification and semantic classification for “ubiquitous”. Here, it is assumed that the part of speech classification and the semantic classification obtained from the dictionary 36 of the translation system for the word “Internet” are generated as the part of speech classification and the semantic classification of the word “ubiquitous”.

  The unregistered word registration information generation unit 58 includes the variable part character string “ubiquitous” obtained in this way, the corresponding translated word “ubiquitous computing”, the part of speech classification “common noun”, the semantic classification “technical term”, and the like. Unregistered word information is generated and added to the unregistered word list.

  The dictionary registration unit 60 that has obtained this unregistered word list adds the heading “ubiquitous” to the dictionary 36 of the translation system. The translation is “ubiquitous computing”, the part of speech classification includes “common nouns”, and the semantic classification includes “technical terms”.

[Effect of the embodiment]
As described above, according to this embodiment, by preparing the bilingual example sentence group 34, words that are not registered at all in the translation system dictionary 36 are automatically additionally registered. Can do. Moreover, not only the character string that becomes the heading but also information such as the translation, part-of-speech classification, and semantic classification are automatically generated and registered. Therefore, operations such as picking up unregistered words, determining translated words, assigning semantic classifications, and the like, which are conventionally required, are unnecessary. As a result, the translation system dictionary 36 can always be kept up-to-date with very little effort.

  The dictionary 36 thus obtained is not limited to a translation system, and can be used for general purposes in processing related to natural language processing. As a result, it is possible to provide an unregistered word automatic registration device capable of automatically registering completely unregistered words that are unregistered in all dictionaries used for natural language processing in the dictionary. Further, according to this embodiment, an effect of improving the utilization efficiency of a system that performs natural language processing, for example, an automatic translation system can be obtained.

[Possible variants]
In the present embodiment, the dictionary registration unit 60 shown in FIG. 1 automatically performs additional registration in the dictionary 36 of the translation system. If there is a possibility that many unnecessary headlines may be added to the dictionary 36 of the translation system in the case of complete automatic addition registration, a dialog asking the user whether or not to register when adding headings by the dictionary registration section 60 The heading may be added to the translation system dictionary 36 only when the user gives an instruction. Even in this case, it is not necessary to manually collect information such as extraction of headings to be registered, specification of translated words, part-of-speech classification, and semantic classification, and the maintenance of the dictionary 36 of the translation system is performed more efficiently than before. be able to. In addition, since the translation system dictionary 36 can be maintained with such a small amount of effort, the performance of the entire natural language processing system using the translation system dictionary 36 can always be kept constant, and the appearance of new words, etc. Natural language processing can be executed with high accuracy.

  In this embodiment, the rewrite pattern as shown in FIG. 2 is used. However, the rewrite pattern is not limited to such an example. As described above, as the rewriting pattern, it is only necessary to specify the character string of the fixed part, the position information of the variable part with respect to the fixed part, and the correspondence information of the variable part between the original sentence and the translated sentence in the original sentence and the translated sentence. An example of a rewrite pattern different from the above is shown in FIG.

  In the example shown in FIG. 9, the rewrite pattern is structured. That is, the rewrite pattern 390 includes information indicating the number of fixed parts of the original text (original text fixed number), a character string of each original text fixed part (original text fixed parts 1, 2, and 3), and the number of fixed parts of the translated text. Information (number of translation fixed parts), character strings of each translation fixed part (translation fixed parts 1 and 2), number of variable parts (number of variable parts), and information indicating the position of each variable part (variable part 1 position, Variable portion 2 position).

  Even if the rewrite pattern is structured in this way, the same processing as in the above-described embodiment can be realized. In the example shown in FIG. 9, the name of each part does not necessarily have to be included in the rewrite pattern. The sections may be separated by a predetermined separation symbol, and their meaning may be specified by the data position.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each claim in the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are included. Including.

It is a block diagram of unregistered word automatic registration device 30 concerning one embodiment of the present invention. It is a figure which shows an example of a rewriting pattern. It is a schematic flowchart of the computer program in the case of implement | achieving the unregistered word automatic registration apparatus 30 with a computer. 4 is a flowchart of the first half of variable part extraction / variable part information creation processing executed in step 120 of FIG. 3. It is a flowchart of the latter half of variable part information extraction and variable part information creation processing. It is a flowchart of the dictionary update process performed by step 122 of FIG. It is a figure which shows the external appearance of the computer system 330 which implement | achieves the unregistered word automatic registration apparatus 30 shown in FIG. FIG. 8 is a block diagram of the computer system 330 shown in FIG. 7. It is a figure which shows another example of a rewriting pattern.

Explanation of symbols

30 Automatic unregistered word registration device 32 Storage unit 34 of translation system 36 Example sentence group for translation 36 Dictionary 50 of translation system Rewrite pattern extraction unit 52 Rewrite pattern storage unit 54 Example sentence / rewrite pattern matching unit 56 Variable unit analysis unit 58 Unregistered word Registered information generation unit 60 Dictionary registration unit 62 Registered word dictionary search unit 80, 390 Rewrite pattern 90 Original sentence pattern 92 Translated sentence pattern 100, 102, 104, 106 Variable part

Claims (8)

Based on a rewrite pattern between a sentence in a first language and a sentence in a second language, and a plurality of parallel translation example sentences between the first language and the second language, the first language An unregistered word automatic extraction device that automatically extracts unregistered words in a machine-readable dictionary of languages,
The sentence of the first language and the sentence of the second language of the rewrite pattern each include a fixed character string and a specific character string indicating a variable part corresponding to each other,
The unregistered word automatic extraction device is:
The rewriting pattern and the parallel translation example sentence are collated, a parallel translation that matches the rewrite pattern is extracted from the parallel translation example sentence, and in the rewrite pattern that matches the parallel translation in the sentence of the first language in the parallel translation and said first language sentence of the specific string sentence, rewriting pattern checking means for that identifies a character string existing at a position corresponding to,
The character string specified by the for sentence-rewrite pattern checking means to search in the dictionary, the dictionary retrieval section you determine whether or not it is registered,
For the character string of the first language determined not to be registered in the dictionary by the dictionary search means, based on the bilingual translation including the character string of the first language and the rewriting pattern to which the bilingual match The second language character string existing at the position corresponding to the specific character string in the second language sentence in the matched rewriting pattern in the second language sentence in the parallel translation , it said with a string of a first language and a registration information identification module that identifies the information to be registered in the dictionary, unregistered word automatic extractor. Part-of- speech classification or semantic classification registered in the dictionary in relation to the character string of the first language for the character string of the first language determined to be registered in the dictionary by the dictionary search means further comprising a variable portion information memory means you stored as the variable portion information in association with the rewriting pattern with,
The registration information specifying means includes:
For the character string of the first language determined not to be registered in the dictionary by the dictionary search means, based on the bilingual translation including the character string of the first language and the rewriting pattern to which the bilingual match By identifying a character string existing in a position corresponding to the specific character string in the sentence of the second language in the matched rewriting pattern in the sentence of the second language in the parallel translation , a translation word identifying means that identifies the string of the second language to be registered in the dictionary as a translation corresponding to the character string of the first language,
By selecting according to predetermined criteria a portion or all of the variable portion information translation including the character string of said first language is stored in the matching rewrite pattern in association with the variable portion information memory means, wherein and a first language the part of speech classification or semantic classification pressurized information identification module with that identifies as the additional information to be registered in the dictionary corresponding to the character string of the unregistered word the automatic extraction apparatus according to claim 1 . The additional information specifying unit, the union of the variable portion information stored in the variable portion information storage means in association with the rewriting pattern translation matches including the string of said first language, the first language including hand stage that identifies as the additional information related to the character string of the unregistered word the automatic extraction apparatus according to claim 2. Each time the string of the first language appear on a result of matching by the use sentence-rewrite pattern checking means further includes a current number counting means leave you count the number of occurrences of the string,
The additional information specifying unit, among the variable portion information stored in the variable portion information storage means in association with the rewriting pattern translation matches including the string of said first language, counted by the occurrence number counting means has been the union of occurrence counts not less than a predetermined number of times, including hand stage that identifies as the additional information related to the character string of said first language, unregistered words automatic extraction apparatus according to claim 2. Each time the string of the first language appear on a result of matching by the use sentence-rewrite pattern checking means further includes a current number counting means leave you count the number of occurrences of the string,
The additional information specifying unit, the number is larger than a predetermined threshold value of the variable portion information stored in the variable portion information storage means in association with the rewriting pattern translation matches including the string of said first language Occasionally, among the variable portion information, the union of the occurrence frequency counted not less than a predetermined number of times by the appearance count counting means, hand stage that identifies as the additional information related to the character string of said first language The unregistered word automatic extraction device according to claim 2, comprising: Computer, claims 1 to make function as all means of unregistered words automatic extraction apparatus according to any one of claim 5, unregistered word automatic extraction program. An unregistered word automatic extraction device according to any one of claims 1 to 5,
The unregistered word information identified as to be registered in the dictionary by the automatic extraction apparatus further comprises a dictionary registration unit to register to the dictionary, unregistered word automatic registration device. The computer causes function as all means of unregistered words automatic registration apparatus according to claim 7, unregistered word automatic registration program.

Images