JP6535607B2 - Preprocessing model learning device, method and program - Google Patents

Description

  The present invention relates to a preprocessing model learning apparatus, method, and program, and more particularly, to a preprocessing model learning apparatus, method, and program for rewriting a sentence for language processing.

  In the prior art, a plurality of methods for performing rewriting based on the rewriting rule have been proposed (see Non-Patent Document 1 and Non-Patent Document 2).

  Also, conventionally, techniques for learning a model using a specific domain text are known. For example, in the case of model learning for translation, a specific domain text refers to a Japanese-side target corpus used when learning a translation model. In addition, a Japanese corpus used as a model learning corpus at the time of construction of a system for syntactic analysis and information extraction, or in the case of processing for converting to a description of a specific domain (for example, newspaper notation or spoken language etc.) Points to the text.

Yoshimi Yoshihiko, Sada Ichiko and Fukumochi Yoji, "Original Text Preediting for Robust English-to-Japanese Machine Translation System", Natural Language Processing, 2000 Sakamoto Akiko, Tanaka Hiroyuki, "Japanese Pre-editing for Spoken Language Machine Translation", The 21st Annual Meeting of the Association for Speech Processing, 2015

  However, in language processing such as translation, there is a phenomenon that language processing can not be appropriately performed because the language expression of the input sentence to be processed does not match the language expression of the target corpus used as the model learning corpus.

  For example, in the case of translation processing, when the input sentence "This is delicious" is translated, the translation result is "This Oishi-I", and "It is delicious" can not be correctly analyzed, resulting in an incorrect translation. On the other hand, if the input sentence is rewritten to translate "It is delicious", the translation result will be "It tastes great", and "Delicious" will be correctly analyzed and a meaningful translation.

  The present invention has been made to solve the above-mentioned problems, and provides a preprocessing model learning device, method, and program that can learn a preprocessing model for performing high-accuracy language processing. The purpose is

  In order to achieve the above object, a preprocessing model learning device according to a first aspect of the present invention performs language processing predetermined for the rewrite candidate for each of the rewrite candidates generated for the input character string. Evaluating the likelihood of the result of performing the language processing on the rewrite candidate based on the result of performing the step of selecting the pair of the character string and the rewrite candidate having the highest evaluation as a correct pair A preprocessing model for selecting a rewrite candidate in the language processing preprocessing based on an evaluation selection unit, the correct pair selected by the evaluation selection unit, and the features extracted from each of the rewrite candidates And a preprocessing model learning unit that learns.

Further, in the preprocessing model learning device according to the first invention, the preprocessing model learning unit is
Generating a lattice having a graph structure including an edge corresponding to a node corresponding to each partial character string representing the character string and each of the rewrite candidates and a node corresponding to the partial character string to be connected; The pre-processing model may be learned such that a character string represented by a path having a maximum score calculated using the pre-processing model among the respective paths is a rewrite candidate of the correct pair.

  In the preprocessing model learning method according to the second aspect of the invention, the evaluation selecting unit performs predetermined language processing on the rewrite candidate for each of the rewrite candidates generated for the input character string. Evaluating likelihood of a result of performing the language processing on the rewrite candidate based on the result, and selecting a pair of the character string and the rewrite candidate having the highest evaluation as a correct solution pair; Before the preprocessing model learning unit selects a rewriting candidate in the language processing preprocessing based on the correct pair selected by the evaluation selecting unit and the feature extracted from each of the rewriting candidates And learning the process model.

  Further, in the preprocessing model learning method according to the second invention, in the step for the preprocessing model learning unit to learn, a node corresponding to each partial character string representing the character string and each of the rewrite candidates and concatenation are provided. A lattice which is a graph structure having edges connecting nodes corresponding to the partial character strings to be generated, and a character represented by a path having the largest score calculated using the preprocessing model among the paths of the lattice The pre-processing model may be learned such that a row is a candidate for rewriting the correct pair.

  A program according to a third invention is a program for causing a computer to function as each unit of the preprocessing model learning device according to the first invention.

  According to the preprocessing model learning device, method, and program of the present invention, for each of the rewrite candidates, based on the result of performing predetermined language processing on the rewrite candidates and the predetermined correct data. The likelihood of the result of language processing on the rewrite candidate is evaluated, and a pair of the character string and the highest-rated rewrite candidate is selected as the correct pair, and the selected correct pair and the language processing By learning a preprocessing model for selecting a rewriting candidate in preprocessing of language processing, based on the language model score of each of the partial character strings in the language model created from the target corpus used for learning of The effect of being able to learn a pre-processing model for performing high-accuracy language processing is obtained.

It is a figure which shows an example of the language model which used the object corpus. It is a block diagram which shows the structure of the pre-processing model learning apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the entry word set of lexeme "Oyau". It is a figure which shows an example of the rewriting candidate and the value of a semantic similarity of a higher order of the semantic similarity with "Harahe". It is a figure which shows an example of a rewriting candidate table. It is a figure which shows an example of a production | generation of a lattice. It is a figure which shows the example of calculation of the score of a feature. It is a flowchart which shows the rewriting candidate table creation process routine in the pre-processing model learning apparatus which concerns on embodiment of this invention. It is a flowchart which shows the pre-processing model learning processing routine in the pre-processing model learning apparatus which concerns on embodiment of this invention. It is a flowchart which shows the lattice production | generation processing routine in the pre-processing model learning apparatus which concerns on embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Overview of Embodiment of the Present Invention>

  First, an outline of the embodiment of the present invention will be described.

  The technique proposed in the embodiment of the present invention acquires a plurality of expressions of notation fluctuation and wording as rewrite candidates in advance, and develops them as one lattice. Using the score of the language model created from the target corpus, which is a learning corpus of the target sentence, and the score of the semantic similarity of the rewrite candidate, the top N rewrite candidates suitable for language processing are output. The target corpus is prepared by preparing text to be rewritten and performing morphological analysis on the text. In addition, it is assumed that a language model is created in advance using a target corpus and combining writing, part of speech, and score. FIG. 1 shows an example of a language model using a target corpus. The optimal rewrite candidate is the one closest to the language expression of the target corpus. Then, pre-processing is performed to select the rewrite candidate in the pre-processing using the correct pair of the character string before rewriting and the rewriting candidate obtained by translating the top N rewriting candidates and using the correct data Learn a model.

  In the embodiment of the present invention, although a case of learning a preprocessing model for performing machine translation is described, the present invention is not limited to this and is applied to all language processing such as syntactic analysis and automatic summary. be able to.

<Configuration of Preprocessing Model Learning Device According to Embodiment of the Present Invention>

  Next, the configuration of the preprocessing model learning apparatus according to the embodiment of the present invention will be described. As shown in FIG. 2, the preprocessing model learning device 100 according to the embodiment of the present invention is a program for executing a CPU, a RAM, a rewriting candidate table creating processing routine and a preprocessing model learning processing routine described later. And a ROM storing various data, and can be configured by a computer. The preprocessing model learning device 100 functionally includes an input unit 10, an arithmetic unit 20, and an output unit 90 as shown in FIG.

  The input unit 10 includes, as text sets, UniDic (https://osdn.jp/projects/unidic/) and JUMAN (http://nlp.ist.i.kyoto-u.ac.jp/index.php?JUMAN Etc. and large-scale texts collected from SNSs such as Twitter (R), and output to the rewrite candidate acquisition unit 22. Also, the input unit 10 receives a character string to be rewritten.

  Arithmetic unit 20 includes rewrite candidate acquiring unit 22, preprocessing unit 24, language model 26, main processing model 28, main processing unit 60, evaluation selecting unit 62, and preprocessing model learning unit 64. It consists of

  The rewrite candidate acquisition unit 22 creates a rewrite candidate table storing rewrite candidates for the input notation from the dictionary and the large-scale text received by the input unit 10.

  The rewrite candidate acquisition unit 22 includes a dictionary candidate acquisition unit 30, a synonym phrase acquisition unit 32, a synonym predicate acquisition unit 34, and a similarity setting unit 36. In the present embodiment, the case where the rewrite candidate is acquired by each of the dictionary candidate acquisition unit 30, the synonym phrase acquisition unit 32, and the synonym predicate acquisition unit 34 of the rewrite candidate acquisition unit 22 will be described as an example. For example, rewriting candidates may be obtained by other methods such as rewriting candidates created manually, rewriting candidates using similarity of reading, and the like.

  The dictionary candidate acquisition unit 30 acquires rewrite candidates at a plurality of levels for each of the input notations, using the dictionary accepted by the input unit 10. Specifically, using the lexical elements of the dictionary (using UniDic), the representative notation (using JUMAN), etc. as the entry words, the same lexical element registered in the dictionary and the entry word set having the representative notation are rewritten Define as a candidate group. FIG. 3 shows an example of the entry word set of the lexical element "Ohashi" in UniDic. For each of the created rewrite candidate groups, each of the dictionary entry words may be used as an input notation, and each entry word other than the input notation of the same group may be used as a rewrite candidate. As described above, the rewriting candidate obtained by the dictionary candidate acquiring unit 30 can absorb word-level writing fluctuation.

  The synonym phrase acquiring unit 32 acquires, using the large-scale text received by the input unit 10, a pair of character string at the word level and the phrase level using the semantic similarity as a rewrite candidate for the input notation. Specifically, when a short sentence (within n characters) on Twitter (R) is divided by punctuation marks, symbols, etc., a group of character strings that is within 10 characters is registered in the dictionary, and each of the character strings is analyzed by analysis. Find the semantic vector for Word2vec (see Reference 1) or the like may be used to calculate the semantic vector. Then, based on the semantic vector of each of the character strings, the semantic similarity of each pair of character strings is estimated using cosine similarity or the like to obtain a rewrite candidate for the input notation.

Reference 1: Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. In Proceedings of Workshop at ICLR, 2013.

  Here, a pair of character strings whose semantic similarity is within a predetermined threshold is defined as a rewrite candidate for the input notation. FIG. 4 shows an example of the rewriting candidate of the upper rank and the value of the semantic similarity with respect to the character string “Harahe ー”. The rewriting candidate obtained by the synonym phrase acquiring unit 32 enables rewriting to a rewriting candidate that is semantically similar even if the notations are not similar, and the semantic space of the rewriting candidate is expanded.

  The synonym predicate acquiring unit 34 acquires, using the large-scale text received by the input unit 10, rewriting candidates in which the function word of the predicate is rewritten with respect to the input notation which is a predicate. Specifically, predicate normalization analysis (see reference 2) is performed on a large-scale text, and candidates having the same semantic label and predicate are defined as rewriting candidates.

Reference 2: Izumi Atsuko, Imamura Kenji, Kikui Genichiro, Fujita Atsushi, Sato, "Predator paraphrasing functional verb expressions directed to normalization", The 15th Annual Conference of the Association for Language Processing, 2009

  For example, "mich", "might", "mighty", and "michata" are examples of rewrite candidates that have the same functional word semantic label and predicate as opposed to "muri + comple". It can be mentioned.

  Especially in Japanese, function words of predicates are redundant and expressions are also diverse, so function words of such various predicates should be used as rewriting candidates according to the rewrite candidates acquired by the synonym predicate acquiring unit 34 Can. In addition, since detailed expressions of function words can not often be identified by methods such as semantic similarity, using a model specialized for function words of predicates makes it possible to use various models that are semantically identical and have high reliability. It is possible to obtain a predicate rewriting variation.

  The similarity setting unit 36 assigns semantic similarity to the rewrite candidates acquired by each of the dictionary candidate acquisition unit 30, the synonym phrase acquisition unit 32, and the synonym predicate acquisition unit 34, and the input notation, the rewrite candidate, and the input A rewrite table composed of each combination of the semantic similarity of the rewrite candidate to the notation is stored in the rewrite candidate DB 38. The rewrite candidate acquired by the dictionary candidate acquisition unit 30 and the synonym predicate acquisition unit 34 is assigned a value of 1 as a semantic similarity. On the other hand, the rewriting candidate acquired by the synonymous phrase acquiring unit 32 is assigned the semantic similarity calculated by the semantic vector. The reason for setting the degree of similarity in this way is that the dictionary candidate acquiring unit 30 and the synonym predicate acquiring unit 34 acquire only certain rewrite candidates whose synonym determination has been made by manual check in advance. This is because, in the phrase acquisition unit 32, the rewrite candidate automatically acquired by the function of the semantic similarity is not necessarily a reliable candidate. By using the semantic similarity itself as a feature, the degree of similarity of the meaning is reflected.

  The rewrite candidate DB 38 stores the rewrite candidate table created by the rewrite candidate acquisition unit 22. FIG. 5 shows an example of the rewrite candidate table. In addition, it is also possible to add a predetermined rule like the rule 1 of FIG. Further, in the present embodiment, the rewrite candidate table further stores the source from which the rewrite candidate has been acquired.

  In the above-described rewrite candidate table, collation is performed using the notation and the part of speech as a key, but the part of speech can be omitted. In the following description, an example in which the part of speech is omitted for simplicity is shown.

  The preprocessing unit 24 is configured to include a lattice generation unit 40 and an Nbest solution generation unit 50.

  The language model 26 is a model created for a combination of notation and part of speech using a specific domain text to be rewritten (in the present embodiment, a target corpus used for learning for translation processing). Here, the language model score of the language model 26 is a score representing the likelihood of the expression in the target corpus.

  The lattice generation unit 40 performs dictionary lookup on the character string received by the input unit 10 using the rewrite candidate table generated by the rewrite candidate acquisition unit 22 by the processing of each unit described below, and includes each of the rewrite candidates. A lattice is generated, which is a graph structure including nodes corresponding to partial strings and edges connecting nodes corresponding to partial strings to be connected.

  The lattice generation unit 40 includes a morphological analysis unit 42, a rewrite candidate table reference unit 44, and a rewrite candidate lattice generation unit 46.

  The morphological analysis unit 42 morphologically analyzes the character string received by the input unit 10, and outputs the input morpheme obtained by the analysis to the rewrite candidate table reference unit 44.

  The rewrite candidate table reference unit 44 acquires, for each of the input morphemes obtained by the morpheme analysis unit 42, the rewrite candidate table of the rewrite candidate DB 38 using the input morpheme as the reference key of the input notation and acquires the rewrite candidate set.

  The rewrite candidate lattice generation unit 46 generates a lattice by expanding the rewrite candidates for each of the input morphemes using the rewrite candidate set acquired by the rewrite candidate table reference unit 44.

  Specifically, as shown in FIG. 6, rewrite candidate lists matching the input morphemes are listed in order from the left. Then, each of the input morpheme and the rewrite candidate is used as a node, and a graph structure in which consecutive nodes are connected by an edge is generated as a lattice.

  The Nbest solution generation unit 50 generates the lattice generated by the lattice generation unit 40, the semantic similarity of the rewrite candidate corresponding to each node in the lattice, and the language model of each of the partial character strings in the language model 26 by the following processing. Based on the score, among the paths consisting of lattice edges, a character string represented by the top N paths with a score is generated as each of the input character string rewrite candidates. Here, the score of the path is obtained based on the semantic similarity of the rewrite candidate corresponding to the partial character string of each node on the path and the language model score.

  First, the Nbest solution generation unit 50 generates the lattice generated by the lattice generation unit 40, the semantic similarity of the rewrite candidate corresponding to each node in the lattice, and the language model score of each of the partial character strings in the language model 26. Based on each path in the generated lattice, the score of each feature of the path using the language model score, the semantic similarity, and the rewrite flag is calculated. Then, based on the calculated scores of each feature, for example, using dynamic programming, the path having the top N total scores shown below in the lattice is calculated.

  Here, α, β, and γ are weights of each feature including the language model score, the semantic similarity, and the rewriting flag. The values of α and β are experimentally determined based on the data in advance. The rewrite flag is a variable that takes a value of 1 in the case of a rewritten node and a value of 0 in the case of other nodes. The total score can be calculated by adding up the scores of each feature. The total score is calculated by the following equation (1). The values of α, β, and γ may be values of a pre-processing model learned in advance by a pre-processing model learning unit 64 described later.

Total score = _{i i} (α x language model score + β x (-log (meaning similarity))
+ Γ × rewrite flag)
... (1)

  Where i represents a substring of a node on the path. For example, as shown in FIG. 7, the score of each node is calculated by multiplying α by the language model score corresponding to the substring of the node.

  For example, if the route corresponding to the input character string is "Oha yo / / / っ / / /-", the total score of the route is "1.2 + 2.3 + 2.1 + 1.9 + 1.7 + 5.7 = 14.9" Become. In addition, if the route 1 including the rewrite candidate is "Good morning / stomach / cold / I /", the total score of the route 1 is "1.0 + 2.1 + 1.9 + 1.8 + 3.8 + 0.5 * (-log (1)) + 0.5 * (− log (0.6)) = 10.86 ”. Paths including other rewriting candidates are similarly calculated. In the route of the above example, since the route 1 has the smallest score, the route 1 is the top first route.

  In addition, when the total score of several path | routes is the same, it selects according to a priority. For example, the input notation may be given the top priority, and the next priority may be in the order of character code.

  The processing model 28 is a pre-learned translation model for translating a source language sentence into a target language sentence. The translation model is assumed to be learned with the source language as English and the target language as Japanese. As a translation model, an external translation system or the like may be used. The present processing model 28 is not limited to the translation model, and may be any language processing model such as syntactic analysis and automatic summary depending on the learning target of the preprocessing model.

    The main processing unit 60 performs a translation process on each of the rewrite candidates generated by the Nbest solution generation unit 50 of the preprocessing unit 24 using the main processing model 28, and outputs a translation result. For example, if the character string before rewriting is "Oha-yo-ha-he-he", the translation result is "Whoa Hayo belly heh was over", but the generated rewriting candidate is translated as follows: . If the rewriting candidate is "Good morning," the translation result will be "Good morning hungry." If the rewrite candidate is "Good morning I was hungry", the translation result will be "Good morning I was reduced stomach". If the rewrite candidate is "Good morning I'm hungry", the translation result will be "Good morning I was hungry". If it is the rewrite candidate "Good morning," the translation result will be "Good morning hungry." In addition, if the character string before rewriting is "television has been seen", the translation result is "I chat Terebimi", but the generated rewriting candidate is translated as follows. If the rewrite candidate is "TV", the translation result is "I saw TV". If it is "TV has been found", the translation result will be "It was chai Terebimi".

  The evaluation and selection unit 62 applies, to each of the rewrite candidates generated by the Nbest solution generation unit 50, the rewrite candidate based on the result of performing a predetermined translation process on the rewrite candidate in the main processing unit 60. The likelihood of the result of translation processing is evaluated, and a pair of the character string accepted by the input unit 10 and the rewriting candidate with the highest evaluation is selected as a correct pair.

  Here, in order to evaluate the likelihood of the translation process, the translation correct answer data determined in advance when the translation process is performed on the character string received by the input unit 10, and RIBIES, which is an automatic evaluation system for the translation process. Use to give an evaluation score for the translation. An evaluation score in automatic evaluation of the translation result of the original character string before rewriting and the translation result of the rewriting candidate is calculated, and the character string before rewriting and the rewriting candidate are ranked. The rewrite candidate with the highest evaluation score in the automatic evaluation is taken as the correct sentence of the rewrite. If the original character string before rewriting has the highest evaluation value, the sentence that is not rewritten is the correct data. Other automatic evaluation systems such as BLUE may be used besides RIBIES.

  An example of the evaluation score of the likelihood of translation processing is shown below. The evaluation score in the automatic evaluation is as follows. Here, the reference translation which becomes correct data is "Good morning. I am hungry".

  The evaluation score is 4.9 for the translation result "Good morning I was hungry" of the rewrite candidate "Good morning I was hungry". For the translation result "Good morning hungry" of the rewrite candidate "Good morning", the evaluation score is 4.1. The evaluation score is 4.1 for the translation result "Good morning hungry" of the rewrite candidate "Good morning hungry". For the translation result "Good morning I was reduced stomach" of the rewriting candidate "Good morning I was hungry", the evaluation score is 4.0. In addition, the evaluation score is 3.3 for the translation result “Whoa Hayo belly heh was over” of the original string “Oha yo yo ha hen ー” before the rewrite.

  The above evaluation scores are ranked, and the 1st rewrite candidate "Good morning" is determined to be the correct rewrite data (positive example), and the original pre-rewrite character string "Oh yo yo ha ha ー ー" The pair of is the correct pair.

  The preprocessing model learning unit 64 rewrites in the language processing preprocessing based on the correct pair selected by the evaluation selecting unit, the character string received by the input unit 10, and the features extracted from each of the rewrite candidates. Train a preprocessing model to select candidates. Here, the preprocessing model learning unit 64 has a graph structure including an edge corresponding to a character string, a node corresponding to each partial character string representing each of the rewrite candidates, and a node corresponding to the partial character string to be connected. The preprocessing model is learned so that the character string represented by the path having the largest score calculated using the preprocessing model among the paths of a certain lattice is a candidate for rewriting the correct pair.

  Specifically, for the character string accepted by the input unit 10, the preprocessing model learning unit 64 first uses each of the rewrite candidates obtained by the Nbest solution generation unit 50 by the same method as the lattice generation unit 40. Perform dictionary lookup to generate a lattice. Alternatively, among the lattices created by the lattice generation unit 40, paths relating to each of the rewrite candidates obtained by the Nbest solution generation unit 50 may be used. At this time, morpheme information is pre-assigned or reanalyzed.

  Next, the preprocessing model learning unit 64 learns the parameters α, β, and γ based on the correct pair obtained by the evaluation selection unit 362 and the lattice. The score function using the preprocessing model learned by the preprocessing model learning unit 364 is defined as follows.

F = α * f1 (rewrite candidate feature) + β * f2 (language model feature) + γ * f3 (rewrite candidate similarity feature)

  Here, f1 (rewrite candidate feature) is a binary feature that is 1 when the corresponding rewrite candidate is used and 0 otherwise. It is a feature defined for each rewrite candidate, such as f1 (は → 腹 し た), f1 (→→ 腹). For example, in the case of a path passing through "good morning", f1 (good morning → good morning) = 1. Also, in the case of a route that does not pass "Good morning", f1 (Good morning → Good morning) = 0. f2 (language model feature) represents the language model score of the corresponding path obtained from the language model 26. f3 (rewrite candidate similarity feature) represents a similarity function value with the original unrewritten character string of the rewrite candidate adopted in the corresponding route. For example, in the case of the route 1 “oh ah ha ha ha ー ー ー”, the route 2 “good morning you are hungry”, the similarity function value is

  f3 (path 1 → path 2) = 0.7 (oh aw → good morning) + 0.6 (a ha ha ー ー 腹 腹 腹) = 1.3

It becomes. For the generated lattice, learning of the parameters α, β, and γ so that the sentence represented by the optimal path when the above score function is calculated matches (becomes close to) the rewrite candidate that is the correct sentence of the correct pair I do. The learning of the parameters α, β, and γ can be performed using perceptron learning or CRF.
<Operation of Preprocessing Model Learning Device According to the Embodiment of the Present Invention>

  Next, the operation of the preprocessing model learning device 100 according to the embodiment of the present invention will be described. When the input unit 10 receives a Japanese dictionary and large-scale text and outputs it to the rewrite candidate acquisition unit 22, the preprocessing model learning device 100 executes a rewrite candidate table creation processing routine shown in FIG.

  First, in step S200, using the dictionary received by the input unit 10, rewrite candidates at a plurality of levels are acquired for each of the input notations.

  In step S202, using the large scale text received by the input unit 10, a pair of character level and word level character strings using semantic similarity is acquired as a rewrite candidate for the input notation.

  In step S204, using the large-scale text accepted by the input unit 10, a rewrite candidate obtained by rewriting the function word of the predicate is acquired for the input notation which is a predicate.

  In step S206, a semantic similarity is added to the rewrite candidate acquired in each of steps S200 to S204, and a rewrite table including combinations of the input notation, the rewrite candidate, and the semantic similarity of the rewrite candidate for the input notation is provided. It creates and stores it in the rewrite candidate DB 38, and ends the rewrite candidate table creation processing routine.

  Then, when the character string to be rewritten is received in the input unit 10, the preprocessing model learning device 100 executes a preprocessing model learning processing routine shown in FIG.

  First, in step S102, dictionary substitution is performed on the character string accepted by the input unit 10 using the rewrite candidate table generated in step S100, and nodes corresponding to each partial character string including the rewrite candidate are connected. Create a lattice, which is a graph structure consisting of edges connecting nodes corresponding to the substrings.

  In step S104, based on the lattice generated in step S102, the semantic similarity of the rewriting candidate corresponding to each node in the lattice, and the language model score of each of the substrings in the language model 26, from the edge of the lattice Among the respective routes, the character string represented by the route having the top N scores is generated as each of the rewrite candidates of the input character string.

  In step S106, translation processing is performed on each of the rewrite candidates generated in step S104 using the processing model 28 to obtain a translation result.

  In step S108, for each of the rewrite candidates generated in step S104, a translation process is performed on the rewrite candidates based on the result of performing a predetermined translation process on the rewrite candidates in step S106. The likelihood is evaluated, and the pair of the character string and the rewriting candidate with the highest evaluation is selected as the correct pair.

  In step S110, dictionary extraction is performed on each of the character strings accepted by the input unit 10 using each of the rewrite candidates obtained in step S104 to generate a lattice.

  In step S112, based on the correct pair selected in step S108, the character string received by the input unit 10, and the features extracted from each of the rewrite candidates, a preprocessing model is used among the lattice paths. The pre-processing model for selecting a rewrite candidate in the pre-processing of language processing is learned so that the character string represented by the path with the largest calculated score becomes the rewrite candidate of the correct pair.

  The above step S102 is realized by the lattice generation processing routine shown in FIG.

  In step S300, the character string received by the input unit 10 is morphologically analyzed, and the input morpheme obtained by the analysis is output to the rewrite candidate table reference unit 44.

  In step S302, for each of the input morphemes output in step S300, the rewrite candidate table of the rewrite candidate DB 38 is referred to using the input morpheme as the reference key of the input notation, and the rewrite candidate set is acquired.

  In step S304, using the rewrite candidate set acquired in step S302, rewrite candidates are expanded for each of the input morphemes to generate a lattice, and the lattice generation processing routine is ended.

  As described above, according to the preprocessing model learning apparatus according to the embodiment of the present invention, rewriting candidates are each provided with rewriting candidates based on the result of performing predetermined language processing on the rewriting candidates. The likelihood of the result of the language processing is evaluated, and the pair of the character string and the highest-rated rewriting candidate is selected as the correct pair, and extracted from each of the selected correct pair and the rewriting candidate By learning a pre-processing model for selecting a rewrite candidate in the pre-processing of language processing based on the selected feature, it is possible to learn a pre-processing model for performing high-precision language processing.

  Further, it is possible to select an optimal candidate based on automatic evaluation from among a plurality of automatically acquired rewrite candidates and their combination. For example, in the case of setting by a rule, it is necessary to define unique rewriting such as “mich” → “seen”, and in order to put a plurality of candidates in order, it is necessary to define an order function. In the technique of (1), it is possible to output the most suitable candidate based on the criteria of the language model of the target corpus among "Michi" → "I saw" "I saw" "I saw" and so on.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the scope of the present invention.

  For example, in the above-described embodiment, the preprocessing model learning device 100 acquires rewriting candidates by the processing of the rewriting candidate acquiring unit 22 and the preprocessing unit 24, and language processing predetermined for the rewriting candidates in the main processing unit 60 is performed. Although the pre-processing model learning apparatus 100 does not include the rewrite candidate acquisition unit 22, the pre-processing unit 24, and the main processing unit 60 in the configuration, the pre-processing model learning device 100 uses the input unit 10 in advance. Each of the rewrite candidates and the result of the language processing may be received, and the processing of each unit of the evaluation selection unit 62 and the preprocessing model learning unit 64 may be performed.

10 input unit 20, 220 operation unit 22 rewrite candidate acquisition unit 24 pre-processing unit 26 language model 28 main processing model 30 dictionary candidate acquisition unit 32 synonymous phrase acquisition unit 34 synonymous predicate acquisition unit 36 similarity setting unit 40 lattice generation unit 42 Morphological analysis unit 44 Rewriting candidate table reference unit 46 Rewriting candidate lattice generation unit 50 Nbest solution generation unit 60 Main processing unit 62 Evaluation selection unit 64 Preprocessing model learning unit 90 Output unit 100 Preprocessing model learning device

Claims (7)

Rewriting candidate with a headword having the same lexical element and representative notation registered in a predetermined dictionary, generated for an input character string, a word and a phrase whose semantic similarity to the character string satisfies the condition For each of the rewriting candidates including the rewriting candidate according to and the rewriting candidate obtained by rewriting the function word of the predicate of the character string , based on the result of performing predetermined language processing on the rewriting candidate, An evaluation selection unit which evaluates the likelihood of the result of the language processing on the rewrite candidate and selects the pair of the character string and the rewrite candidate with the highest evaluation as a correct pair;
A preprocessing that learns a preprocessing model for selecting a rewriting candidate in the preprocessing of the language processing based on the correct pair selected by the evaluation selecting unit and the features extracted from each of the rewriting candidates Model learning unit,
Pre-processing model learning device including. The pre-processing model learning unit
Generating a lattice having a graph structure including an edge corresponding to a node corresponding to each partial character string representing the character string and each of the rewrite candidates and a node corresponding to the partial character string to be connected; The front processing model according to claim 1, wherein among the paths, a character string represented by a path with the largest score calculated using the pre-processing model represents a candidate for rewriting the correct pair. Processing model learning device. The pre-processing model learning unit relates to the correctness pair selected by the evaluation selecting unit, the feature extracted from each of the rewrite candidates, and the language model score representing the likelihood of each writing of the rewrite candidate The preprocessing model for selecting a rewriting candidate in the preprocessing of the language processing is learned on the basis of the feature regarding the similarity between each of the character string and each of the rewriting candidates. Preprocessing model learning device as described. The evaluation and selection unit is a candidate for rewriting by a headword having the same lexical element and representative notation registered in a predetermined dictionary generated for the input character string, and the semantic similarity to the character string is a condition Result of performing predetermined language processing on the rewriting candidate for each of the rewriting candidates including the rewriting candidate by the word and the phrase satisfying the above, and the rewriting candidate which is the rewriting of the function word of the predicate of the character string Evaluating likelihood of a result of performing the language processing on the rewrite candidate, and selecting a pair of the character string and the rewrite candidate having the highest evaluation as a correct answer pair;
Before the preprocessing model learning unit selects a rewriting candidate in the language processing preprocessing based on the correct pair selected by the evaluation selecting unit and the feature extracted from each of the rewriting candidates Learning the processing model;
Pre-processing model learning method including: The step that the pre-processing model learning unit learns is:
Generating a lattice having a graph structure including an edge corresponding to a node corresponding to each partial character string representing the character string and each of the rewrite candidates and a node corresponding to the partial character string to be connected; 5. The pre-processing model according to claim 4 , wherein a character string represented by a path having a maximum score calculated using the pre-processing model among the paths represents a candidate for rewriting the correct pair. Processing model learning method. The step that the pre-processing model learning unit learns is:
The correct pair selected by the evaluation selection unit, the feature extracted from each of the rewrite candidates, the feature related to the language model score representing the likelihood of each writing of the rewrite candidate, the character string, and the rewrite The preprocessing model learning method according to claim 4 or 5, wherein a preprocessing model for selecting a rewriting candidate in the preprocessing of the language processing is learned based on the feature related to the degree of similarity with each of the candidates. A program for causing a computer to function as each unit of the preprocessing model learning device according to any one of claims 1 to 3 .