JP4829685B2 - Translation phrase pair generation apparatus, statistical machine translation apparatus, translation phrase pair generation method, statistical machine translation method, translation phrase pair generation program, statistical machine translation program, and storage medium - Google Patents

Description

  The present invention relates to a translation phrase pair generation device, a statistical machine translation device, a translation phrase pair generation method, a statistical machine translation method, a translation phrase pair generation program, a statistical machine translation program, and a storage medium.

  A technique (statistical machine translation) for automatically constructing a machine translation system from parallel translation data obtained by collecting a large number of pairs of sentences in a source language and a target language has been proposed.

  FIG. 3 is an explanatory diagram showing phrase translation. Phrase-based statistical machine translation uses a translation model for word strings rather than word units. First, consider all the possibilities of dividing the source language sentence at the top of Fig. 3 "Japan's prime minister is Koizumi" into substrings. As one of the possibilities, it is divided into three substrings “Japan”, “Prime Minister” and “I am Koizumi” as shown in Figure 3.

  Next, each divided partial character string is translated using a translation model to create a translated language sentence (a character string in the target language) “The prime minister”, “of Japan”, and “is Koizumi”. Furthermore, all sorts of character strings in these target languages are considered as translation sentence candidates.

  From the enormous translation candidates generated in this way, the most likely translation “The prime minister of Japan is Koizumi” is searched and output. However, in practice, it is not practical to search all these possibilities to find an optimal solution, so it is common to narrow down candidates by adding various constraints to find a sub-optimal solution.

  The problem of machine translation is formulated as a problem of finding an optimal translation destination word string eh that satisfies the following (Formula 1) corresponding to the translation source word string f (see Non-Patent Document 1). In this specification, in order to make the explanation easy to understand, a Japanese character string is used as an example of the translation source word string f, and an English character string is used as an example of the translation destination word string e.

h _m (e, f) is a feature function for scoring a pair of source word string f and destination word string e, λ _m (m = 1... M) is their scaling factor, and E is all possible translations. A destination word string (that is, a translation destination language) is used.

As each feature function _h m (e, f), Non-Patent Document 2, which was taken following seven log is used.
・ Phrase translation probability: φ (e | f)
・ Phrase translation probability: φ (f | e)
・ Lexical weight: lex (e | f)
・ Lexical weight: lex (f | e)
Phrase penalty: ω ^{length (e)} (where ω is a constant and length () is a function that returns the word string length)
-N-gram language model: P _LM (e)
・ Distortion model

  Each phrase translation probability or lexical weight is a function for evaluating the likelihood as a translation, and is also called a translation model. In the above equation, since it is impossible in reality to use all elements of E as candidates, translation candidates are generated as follows and E is approximated.

  FIG. 4 is an explanatory diagram showing sentence division and translation candidates. First, a translation source sentence is divided into word strings, and translation target word string candidates for each translation source word string are generated using a translation model. A set of pairs of source word strings and target word strings (hereinafter referred to as phrase pairs) that cover the source sentence without overlap or gaps (in this case, it is obtained by tracing the effective graph from the beginning to the end of the sentence) A translation candidate is a combination of all the elements of a phrase pair set).

  In this specification, one phrase pair is enclosed in square brackets and expressed in the form of “a phrase of a translation destination word string e translated from a translation source word string f (a phrase of the translation source word string f)”. One sentence is composed of one or more phrase pairs.

For example, in the example of FIG. 4, the following set is appropriate as a translation candidate.
・ {"Japanese", "The prime minister", "Koizumi", "is"}
・ {"Japanese", "the prime minister", "is Koizumi"}

By rearranging the elements of each set, the following huge translation candidate sentences are generated.
・ “Japanese” “The prime minister” “is Koizumi”
・ "The prime minister""Japanese""isKoizumi"

  However, even if the number of translation candidates is limited so far, it is practically impossible to obtain an optimal solution from all these candidates. Therefore, a suboptimal solution is obtained by the method of Non-Patent Document 3.

About each scaling factor (lambda) _m , it can set automatically so that the translation accuracy in the bilingual corpus for scaling factor learning may become the maximum by using the method of a nonpatent literature 1.

The four translation models (φ (e | f), φ (f | e), lex (e | f), and lex (f | e)) are expressed as the following phrase table.

These values for the translation candidate sentence are calculated as the cumulative score of the phrase pair constituting the translation candidate. For example, φ (e | f) for “of Japan”, “The prime minister” and “is Koizumi” is φ (e | f) = 0.3 × 0. Since 6 × 0.2 = 0.036, and the value of the corresponding feature function is the log value, logφ (e | f) ≈−1.44.

Assuming that the scaling factors of the feature functions corresponding to the four translation models are 0.1, 0.2, 0.3, and 0.4, respectively, an overall score for the translation model of the following translation candidate sentences is calculated.
・ "The prime minister""ofJapan""isKoizumi"
The calculation results are as follows.
0.1 x log (0.6 x 0.3 x 0.2)
+ 0.2 × log (0.5 × 0.35 × 0.1)
+ 0.3 × log (0.4 × 0.2 × 0.3)
+ 0.4 × log (0.3 × 0.25 × 0.4)

  The score of the translation candidate sentence is calculated in the same manner in consideration of not only the overall score related to the translation model but also other feature functions. As a translation result, the translation candidate with the highest score is selected (repeatedly, but in reality, an optimal solution is not required, and quasi-optimization is requested by the method of Non-Patent Document 3).

FIG. 5 is a configuration diagram showing the translation phrase pair generation device 1a. The phrase table 19a is created from parallel translation data in which a large number of pairs of sentences in the translation source language and sentences in the translation destination language are collected. The phrase table 19a is created by the following steps (1) to (3).
(1) The word association unit 14a performs word association by creating the word association 16a from the parallel corpus 11a.
(2) The phrase pair extraction unit 15a extracts the corresponding phrase pair 17a from the bilingual corpus 11a and the word correspondence 16a.
(3) The score adding unit 18a scores the phrase pair 17a and creates the phrase table 19a.

FIG. 6 is an explanatory diagram showing phrase extraction. Phrase extraction corresponds to the above (1) and (2). First, word association is performed using an IBM model (Non-Patent Document 4) or the like. The “filled rectangle” in FIG. 6 represents word correspondence. The word correspondence in this example is obtained by setting a set {1-1, 2-2, 3-3, 3-4, 4-0} of “Japanese word position−English word position” with the word position as an index (however, Word position 0 means not corresponding anywhere). Next, based on this word correspondence, the phrase pair 17a is extracted so that the “filled rectangle” corresponding to each word in the source language is closed and the “filled rectangle” corresponding to each word in the target language is also closed. In this example, the following 9 sets are extracted.
・ This
・ Is
・ A pen
・ A pen
・ This is
・ This is a pen
・ This is a pen
・ Is a pen
・ Is a pen

  The score adding unit 18a calculates a score for each phrase pair 17a from the list of phrase pairs 17a extracted from all sentence pairs of the parallel translation data. The calculation method of each score is based on Non-Patent Document 2.

  FIG. 7 is a configuration diagram showing the translation phrase pair generation device 1b. The translation phrase pair generation device 1b obtains the phrase table 19b after (1) word normalization as preprocessing for word association. Specifically, the word expression normalization unit 13b creates a normalized bilingual corpus 12b from the bilingual corpus 11b. The word association unit 14b creates a word association 16b from the normalized parallel corpus 12b. The remaining configuration is the same as in FIG.

Several methods can be considered as a word normalization method of the word expression normalization unit 13b. In normalization of English words, a method of lowering all of the words, or a word ending ending called stemming is deleted. Techniques are often used. Since the word-normalized bilingual corpus is used only for word association and the phrase pair 17b is extracted using the original bilingual corpus, a word string that appears in the phrase table is not word-normalized.
FJOch, "Minimum error rate training in statistical machine translation", in Proc. Of the 41st Annual Meeting of the Association for Computational Linguistics (ACL), pp. 160-167, 2003 P.Koehn, FJOch, and D.Marcu, `` Statistical phrase-based translation '', in Proc. Of Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL), pp. 127-133, 2003 P. Koehn, Pharaoh, `` a beam search decoder for phrase-based statistical machine translation models '', in Proc of the 6th Conforence of the Association for Machine Translation in the Americas (AMTA), pp. 115-124, 2004 PFBrown, SADPietra, VJDPitra and RLMercer, `` the Mathematics of Statistical Machine Translation '', Parameter Estimation, Computational Linguistics, Vol. 19, No. 2, pp. 263-311, 1993.3

  A translation model for a word string is represented by a set of triplets of a translation source language character string, a translation destination language character string, and a score. This translation model is configured by using the word association result by performing word association using the co-occurrence statistics of words included in both languages.

  However, the co-occurrence statistic used for word association cannot be obtained when the learning data used for learning the translation model is small, and as a result, there is a problem that a highly accurate translation model cannot be obtained.

  Therefore, it is easier to stop co-occurrence statistics by stabilizing the word notation by unifying English words in lower case in advance or deleting the English endings (stemming). Widely done. Until now, word associations were generated only from normalized bilingual corpora.

  However, normalization causes a side effect that it is difficult for any learning data to be stable and to extract performance. In other words, normalizing the word notation means that some information is lost. Therefore, the translation model created based on word correspondence obtained by normalizing the word notation normalizes the word notation as the learning data increases. There was a problem that the accuracy was lower than the one created without conversion.

  For example, the Japanese word “Japan” can easily correspond to the English words “Japan” and “Japanese”. However, when learning data is small, it is more stable to obtain co-occurrence statistics by stemming English words and converting “Japan” and “Japanese” to “Japan”. However, when seeking word correspondence between two sentences, “Japanese likes Japan” and “Japanese people like Japan”, it is difficult to distinguish which “Japan” corresponds to “Japan” or “Japanese”. Such normalization also has side effects because it is necessary.

  Therefore, the main object of the present invention is to solve the above-described problems and to create a phrase pair that can be translated with high accuracy in any learning data.

  In order to solve the above-described problem, the present invention provides a phrase pair generation device that generates a phrase pair used for statistical machine translation, and is a first unnormalized first untranslated corpus. Creating a word correspondence and creating a normalized second word correspondence from the normalized second bilingual corpus, the first bilingual corpus, and the first word correspondence to the first A phrase pair is extracted, a second phrase pair is extracted from the first bilingual corpus and the second word correspondence, a third phrase pair is extracted from the second bilingual corpus and the first word correspondence, A phrase pair extraction unit for extracting a fourth phrase pair from the second bilingual corpus and the second word correspondence; the first phrase pair; the second phrase pair; Zupea, and, with respect to the fourth phrase pair, respectively perform scoring, and having a score adding unit for storing a phrase table, a.

  Thereby, it is possible to generate a larger number of translated phrase pairs than in the conventional method.

  The present invention accepts an input of a translation source language sentence and stores the first phrase pair, the second phrase pair, the third phrase pair, and the fourth phrase stored in the phrase table by the translated phrase pair generation device. Based on the phrase pair, the translated language sentence is output.

  Thereby, it is possible to generate a larger number of translated phrase pairs than in the conventional method.

  The present invention is a translation phrase pair generation method for generating a phrase pair used for statistical machine translation, in which a computer creates a non-normalized first word correspondence from a non-normalized first parallel corpus Generating a normalized second word correspondence from the normalized second bilingual corpus, extracting the first bilingual corpus, and the first phrase pair from the first word correspondence, Extracting a second phrase pair from the first bilingual corpus and the second word correspondence; extracting a second phrase pair from the second bilingual corpus and the first word correspondence; and the second bilingual corpus; , Extracting the fourth phrase pair from the second word correspondence, the first phrase pair, the second phrase pair, the third phrase pair, and the fourth frame Against Zupea, respectively perform scoring, and executes the instructions stored in the phrase table, a.

  Thereby, it is possible to generate a larger number of translated phrase pairs than in the conventional method.

  In the present invention, a computer receives an input of a translation source language sentence, and the first phrase pair, the second phrase pair, the third phrase pair stored in the phrase table by the translated phrase pair generation method, and A translated language sentence is output based on the fourth phrase pair.

  Thereby, it is possible to generate a larger number of translated phrase pairs than in the conventional method.

  The present invention is a translation phrase pair generation program for causing a computer to execute the translation phrase pair generation method.

  Thereby, it is possible to generate a larger number of translated phrase pairs than in the conventional method.

  The present invention is a statistical machine translation program for causing a computer to execute the statistical machine translation method.

  Thereby, it is possible to generate a larger number of translated phrase pairs than in the conventional method.

  The present invention is characterized by storing the program.

  Thereby, it is possible to generate a larger number of translated phrase pairs than in the conventional method.

  The present invention is a phrase pair generation device that generates a phrase pair used for statistical machine translation, and includes one bilingual corpus that is not normalized and N-1 types of word normalization techniques for the bilingual corpus. By creating N-1 bilingual corpora each applied once, a total of N bilingual corpora, and a word association unit for creating N word correspondence groups from the N bilingual corpora, the N A phrase pair extraction unit that extracts N square phrase pairs by combining the N-word parallel corpus and the N word correspondence, and scoring each of the N square phrase pairs. And a score adding unit for storing in the phrase table.

  Thereby, it is possible to generate a larger number of translated phrase pairs than in the conventional method.

  The present invention is a phrase pair generation device for generating a phrase pair used for statistical machine translation, and N words obtained by applying N kinds of word normalization techniques once each from one bilingual corpus without normalization A phrase pair extraction unit that extracts N phrase pairs by a combination of a word association unit that creates a correspondence group, the one bilingual corpus, and the N word correspondences, and the N phrase pairs And a score adding unit that performs scoring and stores in a phrase table.

  Thereby, it is possible to generate a larger number of translated phrase pairs than in the conventional method.

  According to the present invention, by generating from a bilingual corpus that has not been normalized, a larger number of translation phrase pairs can be generated than in the conventional method. Therefore, any learning data can be translated with high accuracy on average compared to the case where only the translation model without word normalization is used or the case where only the translation model with word normalization is used. Can be pulled out.

  In addition, the user does not have to make a priori decision on whether or not to normalize the bilingual corpus according to the learning data or what kind of normalization method is used, thus reducing the burden on the decision. To do.

  Hereinafter, an embodiment of a translation system to which the present invention is applied will be described in detail with reference to the drawings.

  FIG. 1 is a configuration diagram showing a translation phrase pair generation device 1c. Compared with FIG. 7, there are differences in the configuration shown below.

  First, the word association unit 14c corresponds to a word without normalization corresponding to the bilingual corpus 11c (first bilingual corpus) without word normalization and the normalized bilingual corpus 12c (second bilingual corpus), respectively. 16d (corresponding to the first word) and normalized word correspondence 16c (corresponding to the second word) are obtained.

Next, the phrase pair extraction unit 15c extracts the following four types of phrase pairs (multiple sets of phrase pairs).
As the first phrase pair 17c, the phrase pair is extracted from the non-normalized bilingual corpus 11c using the unnormalized word correspondence 16d.
As the second phrase pair 17d, the phrase pair is extracted from the bilingual corpus 11c that has not been normalized using the normalized word correspondence 16c.
As the third phrase pair 17e, the phrase pair is extracted from the normalized parallel corpus 12c using the non-normalized word correspondence 16d.
As the fourth phrase pair 17f, the phrase pair is extracted from the normalized parallel corpus 12c using the normalized word correspondence 16c.

An example of the phrase table 19 (part) is shown below.

  In the phrase table 19, the score of the phrase pair that has not been extracted is set to 0.001, which is a very small value. In the example of the phrase table 17, the phrase pair “# 1” is 0.001 because it is not included in the first phrase pair 17c.

  FIG. 2 is a configuration diagram showing the statistical machine translation apparatus 2. An example of a Japanese-English machine translation system that translates from Japanese to English is shown. The statistical machine translation apparatus 2 includes parallel translation data 21 for scaling factor learning, a scaling factor learning unit 22, a scaling factor 23, a solution search unit 24, a phrase table 25, and a language model 26. These components can be realized by the conventional methods of Non-Patent Document 1 and Non-Patent Document 3, for example.

  The contents stored in the phrase table 19 (see FIG. 1) and the contents stored in the phrase table 25 are the same. In order to make this content the same, for example, data may be copied from the phrase table 17 to the phrase table 25, or a link referring to the phrase table 17 from the phrase table 25 may be set.

A function h _m (e, f) for evaluating a translation candidate sentence is constructed from a set of triplets of individual phrase pairs and the corresponding scores stored in the phrase table 25. The scaling factor learning unit 22 performs scaling using the method of Non-Patent Document 1 so that the translation accuracy in the parallel translation data 21 for scaling factor learning including the feature function other than the translation model (language model 26) is maximized. Factor 23 is determined.

  The solution search unit 24 weights each feature function by the scaling factor 23.

  In the above embodiment, by using two types of “parallel translation corpus” and “normalized parallel corpus” stored in the phrase table 25, variations of 2 2 = 4 types of phrase pairs are obtained. Further, the word expression normalization unit 13c creates N-1 bilingual corpuses each applying N-1 types of word normalization techniques once, and includes a total of N corpora including one bilingual corpus that is not normalized. It can be extended to use a bilingual corpus. Similarly, N-1 word correspondences, each of which applies N-1 types of word normalization methods once, are created and expanded to use a total of N word correspondences including one word correspondence that is not normalized. it can. N is a natural number of 2 or more.

  In this case, it is possible to acquire a multi-set of N 2 parallel corpus to be extracted, N word correspondences to be used, and a total of N square phrase pairs. Alternatively, it is possible to acquire a multi-set of a total of N phrase pairs with one bilingual corpus without normalization and N word correspondences to be used. As described above, in the present embodiment, it is possible to increase the variations of the translation model by combining the variations of the bilingual corpus.

  In the present invention described above, the translation model obtained without word normalization and the translation model obtained after word normalization are used in a weighted manner. The weighting of the translation model is characterized in that weighting learning data different from the learning data obtained by learning the translation model is used and the translation accuracy in the weight determination learning data is set to be the highest.

  The present invention described above can be widely modified without departing from the spirit thereof as follows.

  For example, in this specification, translation from an arbitrary language system to an arbitrary language system can be applied as the translation source word string f and the translation destination word string e.

  The translation phrase pair generation device 1c and the statistical machine translation device 2 each store a memory as a main storage unit used when performing arithmetic processing, an arithmetic processing device that performs the arithmetic processing, and each table. The computer is configured to include at least auxiliary storage means such as a hard disk drive (HDD). The memory is constituted by a RAM (Random Access Memory) or the like. Arithmetic processing is realized by an arithmetic processing unit configured by a CPU (Central Processing Unit) executing a program on a memory. In addition to each device, the present embodiment includes a program for causing each device to perform arithmetic processing, and a computer-readable storage medium storing the program.

It is a block diagram which shows the translation phrase pair production | generation apparatus regarding one Embodiment of this invention. It is a block diagram which shows the statistical machine translation apparatus regarding one Embodiment of this invention. It is explanatory drawing which shows the phrase translation regarding one Embodiment of this invention. It is explanatory drawing which shows the sentence division and translation candidate regarding one Embodiment of this invention. It is a block diagram which shows the translation phrase pair production | generation apparatus (conventional method 1) regarding one Embodiment of this invention. It is explanatory drawing which shows the phrase extraction regarding one Embodiment of this invention. It is a block diagram which shows the translation phrase pair production | generation apparatus (conventional method 2) regarding one Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Translation phrase pair production | generation apparatus 2 Statistical machine translation apparatus 14 Word matching part 15 Phrase pair extraction part 18 Score addition part 19 Phrase table

Claims (9)

A translation phrase pair generator for generating phrase pairs used for statistical machine translation,
A word associating unit that creates a non-normalized first word correspondence from the unnormalized first bilingual corpus and creates a normalized second word correspondence from the normalized second bilingual corpus; ,
Extracting a first phrase pair from the first bilingual corpus and the first word correspondence; extracting a first phrase pair from the first bilingual corpus and the second word correspondence; and the second bilingual corpus, And a phrase pair extraction unit that extracts a third phrase pair from the first word correspondence, extracts the second parallel corpus, and a fourth phrase pair from the second word correspondence;
A score adding unit that performs scoring for each of the first phrase pair, the second phrase pair, the third phrase pair, and the fourth phrase pair, and stores the score in a phrase table;
The translation phrase pair production | generation apparatus characterized by having.   The input of the translation source language sentence is accepted, and the first phrase pair, the second phrase pair, the third phrase pair, and the stored in the phrase table by the translated phrase pair generation device according to claim 1, A statistical machine translation device that outputs a translated language sentence based on a fourth phrase pair. A translation phrase pair generation method for generating a phrase pair used for statistical machine translation,
Computer
Creating a non-normalized first word correspondence from the non-normalized first bilingual corpus, and creating a normalized second word correspondence from the second bilingual corpus that is normalized;
Extracting a first phrase pair from the first bilingual corpus and the first word correspondence; extracting a first phrase pair from the first bilingual corpus and the second word correspondence; and the second bilingual corpus, And extracting a third phrase pair from the first word correspondence, extracting the second parallel corpus, and a fourth phrase pair from the second word correspondence;
The first phrase pair, the second phrase pair, the third phrase pair, and the fourth phrase pair are each scored and stored in a phrase table;
The translation phrase pair generation method characterized by performing this.   The computer accepts input of a translation source language sentence, and the first phrase pair, the second phrase pair, the third phrase pair stored in the phrase table by the translated phrase pair generation method according to claim 3, And the statistical machine translation method characterized by outputting a translation destination language sentence based on said 4th phrase pair.   A translation phrase pair generation program for causing a computer to execute the translation phrase pair generation method according to claim 3.   A statistical machine translation program for causing a computer to execute the statistical machine translation method according to claim 4.   A computer-readable storage medium storing the program according to claim 5 or 6. A phrase pair generation device for generating a phrase pair used for statistical machine translation,
One bilingual corpus without normalization and N-1 bilingual corpora obtained by applying N-1 types of word normalization methods once to the bilingual corpus are generated as a total of N bilingual corpora, A word association unit for creating N word correspondence groups from the N parallel corpora;
A phrase pair extraction unit that extracts N square phrase pairs by combining the N bilingual corpora and the N word correspondences;
A score adding unit for scoring each of the N square phrase pairs and storing it in the phrase table;
The translation phrase pair production | generation apparatus characterized by having. A phrase pair generation device for generating a phrase pair used for statistical machine translation,
A word associating unit that creates N word correspondence groups each applying N kinds of word normalization techniques once from one bilingual corpus that is not normalized;
A phrase pair extraction unit that extracts N phrase pairs by combining the one bilingual corpus and the N word correspondence;
A score adding unit for scoring each of the N phrase pairs and storing it in a phrase table;
The translation phrase pair production | generation apparatus characterized by having.

Images