JP6678087B2 - Bilingual sentence extraction device, bilingual sentence extraction method and program - Google Patents

Description

  The present invention relates to a technique for creating a bilingual corpus.

  In recent years, it has been recognized that it is important to create a large number of high-quality bilingual corpora for use in statistical machine translation and text mining. In general, the creation of a bilingual corpus requires a great deal of cost, and there is a technical problem in how to efficiently create such a corpus. As a method of creating a bilingual corpus, for example, a document in one language that constitutes a bilingual document is translated into another language, and the number of word matches between the created translation and the sentence in the other language is measured. There is a known method of associating sentences (see Non-Patent Document 1).

Tatsuya Ishizaka, Masao Uchiyama, Eiichiro Sumida, Kazuhide Yamamoto, "Building a Large-Scale Open Source Japanese-English Parallel Corpus", Information Processing Society of Japan, 2009-NL-191, p.1-6, May 2009

  However, in the conventional corpus creation method, as a result of associating sentences with focusing only on the number of matching words, there is a case where sentences that are not translated when viewed as a whole sentence are associated with each other.

  The present invention has been made in view of such circumstances, and it is an object of the present invention to create a higher-quality bilingual corpus as compared with a case where sentences are associated based only on the number of matching words. I do.

  In order to solve the above-mentioned problem, the present invention provides a bilingual document acquisition unit that acquires a bilingual document in a first language and a second language, and a sentence in the first language and the second A bilingual sentence acquisition unit that matches a sentence of a language using the bilingual dictionary of the first language and the second language, and acquires one or more bilingual sentences of the first language and the second language; A translation model generation unit that generates a translation model based on the obtained one or more translated texts, and, for each of the obtained one or more translated texts, configures the translated text using the generated translation model. A translation unit for translating the sentence in the first language into the second language; and for each of the obtained one or more bilingual sentences, a sentence in the first language translated into the second language; Editing for calculating the editing distance between the corresponding second language sentence And a release calculator, among the obtained one or more sentence pairs, providing a bilingual sentence extraction device and a translated sentence selection unit editing the calculated distance is selected greater translated sentence than the threshold value.

  The present invention is also directed to a bilingual sentence extraction method executed by one or more computers, wherein a step of acquiring a bilingual document in a first language and a second language, the first bilingual document constituting the acquired bilingual document is provided. Matching a sentence of a language and a sentence of the second language using a bilingual dictionary of the first language and the second language to obtain one or more bilingual sentences of the first language and the second language Generating a translation model based on the obtained one or more translations, and constructing the translation using the generated translation model for each of the obtained one or more translations. Translating the sentence in the first language into the second language, and for each of the obtained one or more bilingual sentences, the sentence in the first language translated into the second language; Between the corresponding second language sentence Calculating an edit distance, among the obtained one or more sentence pairs, providing a translated sentence extraction method comprising the steps of: editing the calculated distance is selected greater translated sentence than the threshold value.

  The present invention also provides a computer with a step of acquiring a bilingual document in a first language and a second language, and transmitting the sentence in the first language and the sentence in the second language constituting the acquired bilingual document to the computer. Matching using a bilingual dictionary of a first language and the second language to obtain at least one bilingual sentence of the first language and the second language, based on the obtained one or more bilingual sentences Generating a translation model, and for each of the obtained one or more parallel translations, using the generated translation model to translate the first language sentence constituting the bilingual translation into the second language. Translating and, for each of the obtained one or more translations, editing between the first language sentence translated into the second language and the second language sentence corresponding to the sentence Calculating a distance; and Among the one or more sentence pairs provides a program for executing a step of editing the calculated distance is selected greater translated sentence than the threshold value.

  According to the present invention, a bilingual corpus with higher quality can be created as compared with the case where sentences are associated based only on the number of matching words.

FIG. 2 is a block diagram illustrating an example of a configuration of a bilingual sentence extraction device 1. It is a flowchart which shows an example of a bilingual sentence extraction process. FIG. 4 is a diagram illustrating an example of a bilingual document. FIG. 4 is a diagram illustrating an example of data in a bilingual sentence storage unit 106. FIG. 4 is a diagram illustrating an example of data in a bilingual sentence storage unit 106. FIG. 4 is a diagram illustrating an example of data in a bilingual sentence storage unit 106.

1. Embodiment 1-1. Configuration FIG. 1 is a block diagram showing an example of the configuration of a bilingual sentence extraction device 1 according to the present embodiment. The bilingual sentence extraction device 1 is a computer including an arithmetic processing device such as a CPU and a storage device such as an HDD. The bilingual sentence extraction device 1 includes a bilingual document storage unit 101, a bilingual document acquisition unit 102, a word division unit 103, a bilingual dictionary storage unit 104, a bilingual sentence acquisition unit 105, a bilingual sentence storage unit 106, It has functions of a model generation unit 107, a translation unit 108, an edit distance calculation unit 109, a bilingual sentence selection unit 110, and a bilingual sentence editing unit 111. Among these functions, the functions of the bilingual document storage unit 101, the bilingual dictionary storage unit 104, and the bilingual sentence storage unit 106 are realized by a storage device. Other functions are realized by the arithmetic processing device executing a program stored in the storage device.

  The bilingual document storage unit 101 stores bilingual documents in a first language and a second language. Here, the first language is Japanese and the second language is English. A bilingual document is a pair of a Japanese document and an English document created by translating the document into English. The bilingual document is, for example, a pair of a patent publication of a Japanese patent application and a patent publication of a US patent application belonging to the same patent family. Or, a pair of a Japanese newspaper article and an English version of the newspaper article. Or, a pair of an English version of the open source software and a Japanese translation of the manual.

  The bilingual document acquisition unit 102 acquires a bilingual document from the bilingual document storage unit 101.

  The word division unit 103 divides the bilingual document acquired by the bilingual document acquisition unit 102 into sentences, and divides each sentence into words. For Japanese documents, morphological analysis is performed to divide sentences into sentences using clues as clues, and to divide each sentence into words. At that time, the inflected words may be converted into the basic form. For English documents, the sentences are divided into sentences using periods as clues, and each sentence is divided into words using spaces as clues. At that time, the ending word may be analyzed to convert the inflected word into the basic form. Also, uppercase letters may be converted to lowercase letters, and plural forms may be converted to singular forms.

  The bilingual dictionary storage unit 104 stores a bilingual dictionary. Here, the bilingual dictionary is a set of pairs of Japanese words and English words having the same meaning as the words.

  The bilingual sentence acquisition unit 105 matches the Japanese sentence and the English sentence cut out by the word division unit 103 using a bilingual dictionary stored in the bilingual dictionary storage unit 104, and obtains at least one of Japanese and English. Get the bilingual sentence of Specifically, the bilingual sentence acquisition unit 105 translates the Japanese sentence cut out by the word division unit 103 into English, calculates the similarity between the created translated sentence and each sentence in English, and calculates the calculated similarity. A pair of the English sentence with the maximum degree and the Japanese sentence described above is acquired as a bilingual sentence. Here, the similarity is a value calculated based on the number of words that match between the created translated sentence and the English sentence. More specifically, it is a value represented by the ratio of the number of independent words that match between the translated sentence and the number of all independent words included in the English sentence. For example, the bilingual sentence acquisition unit 105 acquires a bilingual sentence using DP (Dynamic Programming) matching as in the bilingual corpus creation method described in the above-mentioned non-patent document. As another example, the bilingual sentence acquisition unit 105 uses the DP matching as described in Takehito Utsuro, et al. "Bilingual Text Matching using Bilingual Dictionary and Statistics," COLING, p. May be obtained. Here, the bilingual sentence is a pair of a Japanese sentence and an English sentence created by translating the sentence into English. In other words, it is a pair of a Japanese sentence and an English sentence having the same meaning as the sentence.

  The bilingual sentence storage unit 106 stores one or more bilingual sentences (in other words, a bilingual corpus) acquired by the bilingual sentence acquiring unit 105. At this time, the bilingual sentence storage unit 106 stores each bilingual sentence in association with a bilingual sentence ID for identifying the bilingual sentence.

  The translation model generation unit 107 generates a translation model based on one or more translations stored in the translation storage unit 106. At this time, the translation model generation unit 107 generates a translation model using, for example, a Moses decoder (http://www.statmt.org/moses/). For a Moses decoder, see, for example, Philipp Koehn, et al. "Moses: Open Source Toolkit for Statistical Machine Translation," Annual Meeting of the Association for Computational Linguistics, demonstration session, Prague, Czech Republic, June 2007.

  The translation unit 108 uses the translation model generated by the translation model generation unit 107 for each of one or more translations stored in the bilingual sentence storage unit 106 to translate the Japanese sentence constituting the translation into English. Translate to The translation unit 108 stores the created translation in the bilingual sentence storage unit 106 in association with the original Japanese sentence.

  The edit distance calculation unit 109 compares, for each of one or more parallel sentences stored in the parallel sentence storage unit 106, a Japanese sentence translated into English by the translation unit 108 and an English sentence corresponding to the sentence. The edit distance between them is calculated. Here, the editing distance is a value calculated based on the number of editing operations required to change a Japanese sentence translated into English into an English sentence corresponding to the sentence. Specifically, the editing distance calculation unit 109 calculates TER (Translation Error Rate) as the editing distance. Here, the editing operation is specifically four operations of insertion, deletion, replacement, and rearrangement. For TER, see, for example, Matthew Snover, et al. "A study of translation edit rate with targeted human annotation," Proceedings of Association for Machine Translation in the Americas, p. 223-231, 2006. After calculating the TER, the edit distance calculation unit 109 stores the calculated TER in the bilingual sentence storage unit 106 in association with the corresponding bilingual sentence.

  The bilingual sentence selection unit 110 selects, from one or more bilingual sentences stored in the bilingual sentence storage unit 106, a bilingual sentence whose TER calculated by the edit distance calculating unit 109 is larger than a threshold value. Here, the threshold value is set, for example, such that a predetermined percentage of bilingual sentences among one or more bilingual sentences stored in the bilingual sentence storage unit 106 are selected. When the bilingual sentence is selected, the editing distance calculating unit 109 notifies the bilingual sentence editing unit 111 of the bilingual sentence ID of the bilingual sentence as a deletion target.

  The bilingual sentence editing unit 111 deletes the bilingual sentence identified by the bilingual sentence ID notified from the bilingual sentence selection unit 110 from the bilingual sentence storage unit 106.

1-2. Operation The operation of the bilingual sentence extraction device 1 will be described. FIG. 2 is a flowchart illustrating an example of a bilingual sentence extraction process performed by the bilingual sentence extraction device 1.

  In step S1 of the bilingual sentence extraction process, the bilingual document acquisition unit 102 of the bilingual sentence extraction device 1 acquires a bilingual document from the bilingual document storage unit 101. FIG. 3 is a diagram illustrating an example of a bilingual document.

  When the bilingual document acquisition unit 102 acquires the bilingual document, the word division unit 103 divides the acquired bilingual document into sentences and divides each sentence into words (step S2). For Japanese documents, morphological analysis is performed to divide sentences into sentences using clues as clues, and to divide each sentence into words. For English documents, the sentences are divided into sentences using periods as clues, and each sentence is divided into words using spaces as clues.

  When the bilingual document is divided into sentences by the word dividing unit 103 and each sentence is divided into words, the bilingual sentence acquiring unit 105 converts the Japanese sentence and the English sentence cut out by the word dividing unit 103 into words. Matching is performed using the bilingual dictionaries stored in the bilingual dictionary storage unit 104 to obtain one or more bilingual sentences in Japanese and English (step S3). When one or more bilingual sentences are acquired, the bilingual sentence acquisition unit 105 stores each bilingual sentence in the bilingual sentence storage unit 106 in association with the bilingual sentence ID (step S4). FIG. 4 is a diagram illustrating an example of data in the bilingual sentence storage unit 106 in which the bilingual sentence is stored by the bilingual sentence acquisition unit 105.

  When one or more bilingual sentences are stored in the bilingual sentence storage unit 106 by the bilingual sentence acquiring unit 105, the translation model generating unit 107 generates a translation model based on the stored one or more bilingual sentences (step S5). .

  When the translation model is generated by the translation model generation unit 107, the translation unit 108 uses the generated translation model for each of the one or more bilingual sentences stored in the bilingual sentence storage unit 106. The constituent Japanese sentences are translated into English (step S6). The translation unit 108 stores the created translation in the bilingual sentence storage unit 106 in association with the original Japanese sentence. FIG. 5 is a diagram illustrating an example of data in the bilingual sentence storage unit 106 in which a translation is stored by the translation unit 108.

  When the translation unit 108 translates the Japanese sentence constituting the bilingual sentence into English, the edit distance calculation unit 109 translates each of the one or more bilingual sentences stored in the bilingual sentence storage unit 106 into English. The TER between the sentence in Japanese and the English sentence corresponding to the sentence is calculated (step S7). After calculating the TER, the edit distance calculation unit 109 stores the calculated TER in the bilingual sentence storage unit 106 in association with the corresponding bilingual sentence. FIG. 6 is a diagram illustrating an example of data in the bilingual sentence storage unit 106 in which the TER is stored by the edit distance calculation unit 109. The “overall” of the TER shown in FIG. 3 is a value obtained by summing the number of times of each operation of insertion, deletion, replacement, and rearrangement.

  When the TER is calculated by the edit distance calculation unit 109, the bilingual sentence selection unit 110 selects a bilingual sentence having a TER larger than a threshold value from one or more bilingual sentences stored in the bilingual sentence storage unit 106 (step). S8). When the bilingual sentence is selected, the editing distance calculating unit 109 notifies the bilingual sentence editing unit 111 of the bilingual sentence ID of the bilingual sentence as a deletion target. For example, if the threshold is set to “5”, the bilingual sentence selection unit 110 notifies the bilingual sentence editing unit 111 of the bilingual sentence ID “032” of the bilingual sentences shown in FIG. 6 as a deletion target.

When the bilingual sentence ID is notified from the bilingual sentence selection unit 110, the bilingual sentence editing unit 111 deletes the bilingual sentence identified by the bilingual sentence ID from the bilingual sentence storage unit 106 (step S9).
The above is the description of the bilingual sentence extraction process.

  According to the bilingual sentence extraction device 1 described above, after the bilingual sentence acquisition unit 105 acquires the bilingual sentence from the bilingual document using DP matching, the bilingual sentence selection unit 110 selects the bilingual sentence from the acquired bilingual sentences. , TER are selected. Therefore, according to the bilingual document storage unit 101, a higher quality bilingual corpus can be created as compared with the case where sentences are associated based only on the number of matching words.

2. Modifications The above embodiment may be modified as described below. One or more of the modifications described below may be combined with one another.

2-1. Modification 1
The bilingual sentence extraction device 1 according to the above embodiment may be a computer system including a plurality of computers. The storage device included in the bilingual sentence extraction device 1 according to the above embodiment may be connected to the bilingual sentence extraction device 1 via a communication line such as the Internet.

2-2. Modification 2
In the above embodiment, the first language may be English and the second language may be Japanese. Further, the combination of the first language and the second language may be arbitrarily selected from natural languages such as German, French, Chinese, and Korean in addition to Japanese and English.

2-3. Modification 3
The translation model generation unit 107 may generate a translation model using a decoder other than the Moses decoder. For example, a translation model may be generated using a Pharaoh decoder. For the Pharaoh decoder, see, for example, Philipp Koehn, "Pharaoh: A Beam Search Decoder for Phrase-Based Statistical Machine Translation Models," Proceedings of the 6th Conference of the Association for Machine Translation in the Americas, p.115-124, 2004. See also.

2-4. Modification 4
In the above-described embodiment, the translation model generation unit 107 is omitted, and the translation unit 108 uses a predetermined translation model for each of one or more bilingual sentences stored in the bilingual sentence storage unit 106. The Japanese sentence constituting the sentence may be translated into English.

2-5. Modification 5
The editing distance calculation unit 109 according to the above embodiment may calculate a value other than TER as the editing distance. For example, a Levenshtein distance, a Damerau-Levenshtein distance, or a Jaro-Winkler distance may be calculated.

As another example, the edit distance calculation unit 109, for each of one or more bilingual sentences stored in the bilingual sentence storage unit 106, corresponds to a Japanese sentence translated into English by the translating unit 108 and a corresponding sentence. BLUE or RIBES between English sentences may be calculated. For BLUE, for example,
See Kishore Papineni, et al. "BLUE: a method for automatic evaluation of machine translation," Proc. Of the 40th Annual Meeting of the Association for Computational Linguistics, p.311-318, July 2002. For details on RIBES, see, for example, Tsutomu Hirao, et al., "RIBES: Automatic Evaluation of Translation Based on Rank Correlation", Proc. Of the 17th Annual Meeting of the Linguistic Processing Society, p.1115-1118, March 2011. thing.

2-6. Modification 6
The bilingual sentence selection unit 110 according to the above-described embodiment, after selecting bilingual sentences having a TER greater than the threshold value, selecting a bilingual sentence other than the selected bilingual sentence and storing the bilingual sentence stored in the bilingual sentence storage unit 106. The bilingual sentence ID may be notified to the bilingual sentence editing unit 111 as a bilingual sentence to be left without being deleted. In this case, the bilingual sentence editing unit 111 deletes the bilingual sentence other than the bilingual sentence identified by the bilingual sentence ID notified from the bilingual sentence selection unit 110 from the bilingual sentence storage unit 106.

2-7. Modification 7
A program for implementing each function of the bilingual sentence extraction device 1 may be provided via a recording medium readable by a computer device. Here, the recording medium is, for example, a magnetic recording medium such as a magnetic tape or a magnetic disk, an optical recording medium such as an optical disk, a magneto-optical recording medium, or a semiconductor memory. This program may be provided via a network such as the Internet.

DESCRIPTION OF SYMBOLS 1 ... Translation sentence extraction apparatus, 101 ... Translation document storage part, 102 ... Translation document acquisition part, 103 ... word division part, 104 ... Translation dictionary storage part, 105 ... Translation sentence acquisition part, 106 ... Translation sentence storage part, 107 ... Translation model generation unit, 108: translation unit, 109: edit distance calculation unit, 110: bilingual sentence selection unit, 111: bilingual sentence editing unit

Claims (3)

A bilingual document acquisition unit for acquiring bilingual documents in the first language and the second language;
The sentence of the first language and the sentence of the second language constituting the acquired bilingual document are matched using the bilingual dictionary of the first language and the second language, and the first language and the second language are translated. A bilingual sentence acquisition unit that acquires one or more bilingual sentences in two languages;
A translation model generation unit that generates a translation model based on the obtained one or more translations;
For each of the obtained one or more translations, using the generated translation model, a translation unit that translates the first language sentence constituting the translation into the second language,
For each of the obtained one or more translated sentences, an edit for calculating an edit distance between the sentence of the first language translated into the second language and the sentence of the second language corresponding to the sentence A distance calculator,
A bilingual sentence selection unit that selects a bilingual sentence whose calculated editing distance is greater than a threshold value from the one or more acquired bilingual sentences. A bilingual sentence extraction method executed by one or more computers,
Obtaining a bilingual document in a first language and a second language;
The sentence of the first language and the sentence of the second language constituting the acquired bilingual document are matched using the bilingual dictionary of the first language and the second language, and the first language and the second language are translated. Obtaining one or more bilingual sentences in two languages;
Generating a translation model based on the obtained one or more translations;
Translating the sentence in the first language constituting the translated sentence into the second language using the generated translation model for each of the obtained one or more translated sentences;
Calculating, for each of the obtained one or more parallel sentences, an edit distance between a sentence in the first language translated into the second language and a sentence in the second language corresponding to the sentence; When,
Selecting a bilingual sentence whose calculated editing distance is greater than a threshold from the one or more obtained bilingual sentences. On the computer,
Obtaining a bilingual document in a first language and a second language;
The sentence of the first language and the sentence of the second language constituting the acquired bilingual document are matched using the bilingual dictionary of the first language and the second language, and the first language and the second language are translated. Obtaining one or more bilingual sentences in two languages;
Generating a translation model based on the obtained one or more translations;
Translating the sentence in the first language constituting the translated sentence into the second language using the generated translation model for each of the obtained one or more translated sentences;
Calculating, for each of the obtained one or more parallel sentences, an edit distance between a sentence in the first language translated into the second language and a sentence in the second language corresponding to the sentence; When,
Selecting a bilingual sentence whose calculated edit distance is greater than a threshold value from the one or more acquired bilingual sentences.

Images