JP7705314B2 - Document processing device, document processing method, and document processing program - Google Patents

Description

The present invention relates to a document processing device, a document processing method, and a document processing program, and is suitable for use in a document processing device, a document processing method, and a document processing program that generate distributed representations of words.

Traditionally, in the field of natural language processing, a technique called distributed representation is known that represents words as vectors consisting of multiple continuous values and can express characteristics between words.

For example, Patent Document 1 is an example of a document related to this technical field. Patent Document 1 discloses a natural language device that aims to "correct, in a simple manner, the values of word vectors that express the features of words with multidimensional vectors" and that "includes a thesaurus storage unit 166 that stores word categories and a first word vector storage unit 164 that stores a distributed representation model including word vectors that express the features of words with multidimensional vectors, and calculates a category vector that represents the word vectors of words that belong to the same category in a category vector calculation unit 144, and corrects the word vector in a word vector correction unit 146 to bring the word vector closer to the category vector to which the word belongs."

Distributed representations use a large-scale corpus to learn the features of each word from its surrounding context, making it possible to acquire vectors that represent the features of a word and its associations with other words. However, when the learning corpus contains words that appear infrequently or when the contexts in which a word appears are biased, there is an issue in that it is not possible to acquire accurate features of the word.

To address the above-mentioned issues, Patent Document 1 describes a mechanism that uses a thesaurus as external knowledge to correct the distributed representation of each word according to the category to which the word belongs, so that the vector approaches a category vector that represents the category.

JP 2021-009538 A

However, the technology disclosed in Patent Document 1 only applies corrections to words represented in the trained distributed representation space, and does not disclose how to handle words that are not included in the trained distributed representation space (for example, subwords split into partial strings, unknown words, and other words that do not fully express meaning by themselves). Therefore, the natural language device disclosed in Patent Document 1 has the problem that it cannot create distributed representations for words that are not included in the trained distributed representation space, even if the words are included in the thesaurus.

The present invention has been made in consideration of the above points, and aims to propose a document processing device, document processing method, and document processing program that can appropriately embed the meaning of words whose meaning cannot be adequately represented in a pre-trained distributed representation space, without additional learning.

In order to solve the above problem, the present invention provides a document processing device that generates embedded representations of words included in text input from outside, the document processing device comprising: a class vector storage unit that stores class vectors which are reference vectors for each type of word; a morphological analysis unit that divides the text into word strings by morphological analysis and assigns information indicating the type to which each word string belongs; a word embedded representation generation unit that converts the word strings divided by the morphological analysis unit into embedded representations for each word included in the word strings; and a word embedded representation correction unit that corrects the embedded representations generated by the word embedded representation generation unit, In this case, the morphological analysis unit divides the text into word strings by morphological analysis, and then further divides the low-frequency words included in the word strings into sub-words of partial strings, the word variance representation generation unit converts the low-frequency words into a plurality of variance representations consisting of variance representations for each of the sub-words included in the low-frequency words, and the word variance representation correction unit combines the plurality of variance representations of the low-frequency words generated by the word variance representation generation unit with a position vector that fixes the order of the sub-words in the low-frequency word, and then combines it with the class vector of the type to which the low-frequency word belongs, and replaces the variance representation after the combination as the variance representation of the low-frequency word .

In addition, in order to solve the above problem, the present invention provides a document control method by a document processing device that generates embedded representations of words included in text input from outside, the document processing device storing a class vector that is a reference vector for each type of word, the document processing device comprising: a morphological analysis step in which the document processing device divides the text into word strings by morphological analysis and assigns information indicating a type to which each word string belongs; a word embedded representation generation step in which the document processing device converts the word strings divided in the morphological analysis step into embedded representations for each word included in the word strings; and a word embedded representation correction step in which the document processing device corrects the embedded representations generated in the word embedded representation generation step, In a document processing method, when a low-frequency word having a frequency lower than a certain level is included, in the morphological analysis step, the text is divided into a word string by morphological analysis, and the low-frequency word included in the word string is further divided into sub-words of partial strings, in the word variance generation step, the low-frequency word is converted into a plurality of variance representations consisting of variance representations for each of the sub-words included in the low-frequency word, and in the word variance representation correction step, a position vector that fixes the order of the sub-words in the low-frequency word is combined with the plurality of variance representations of the low-frequency word generated in the word variance generation step, and then combined with the class vector of the type to which the low-frequency word belongs, and the variance representation after the combination is replaced as the variance representation of the low-frequency word .

In order to solve the above problem, the present invention provides a document control program to be executed by a computer constituting a document processing device that generates embedded representations of words included in externally input text, the document processing device storing a class vector that is a reference vector for each type of word, and the document processing device causes the computer to execute a morphological analysis process that divides the text into word strings by morphological analysis and assigns information indicating the type to which each word string belongs, a word embedded representation generation process that converts the word strings divided by the morphological analysis process into embedded representations for each word included in the word strings, and a word embedded representation correction process that corrects the embedded representations generated by the word embedded representation generation process, and the document processing device causes the computer to execute a morphological analysis process that converts the word strings divided by the morphological analysis process into embedded representations for each word included in the word strings, and When a low-frequency word is included, in the morphological analysis process, the text is divided into a word string by morphological analysis, and the low-frequency word included in the word string is further divided into subwords of partial strings, in the word variance generation process, the low-frequency word is converted into a plurality of variance representations consisting of variance representations for each of the subwords included in the low-frequency word, and in the word variance correction process, a position vector that fixes the order of the subwords in the low-frequency word is combined with the plurality of variance representations of the low-frequency word generated in the word variance generation process, and then combined with the class vector of the type to which the low-frequency word belongs, and the variance representation after the combination is replaced as the variance representation of the low-frequency word.

According to the present invention, for words whose meaning cannot be adequately expressed in a pre-trained distributed representation space, the meaning of the word can be appropriately embedded without additional training. Problems, configurations, and effects other than those described above will become clear from the description of the embodiments below.

1 is a block diagram showing an example of a functional configuration of a document processing apparatus 100 according to a first embodiment of the present invention. FIG. 1 is a diagram showing an example of the configuration of a document processing system 200 including a document processing device 100. 1 is a block diagram showing an example of a hardware configuration of a document processing apparatus 100. FIG. 13 is a flowchart illustrating an example of a processing procedure for class vector creation processing. FIG. 13 is a diagram showing an example of a screen display of a scatter diagram of word variances of proper nouns. 13A and 13B are diagrams illustrating an example of a screen display when correcting class vector candidates. 13 is a flowchart showing an example of a processing procedure for distributed representation generation processing. 13 is a flowchart illustrating an example of a processing procedure for a word embedded expression correction process. 11 is a diagram illustrating an example of a process of processing an input sentence by the distributed representation generation process. FIG. FIG. 11 is a block diagram showing an example of a functional configuration of a document processing apparatus 1000 according to a second embodiment of the present invention. 13 is a flowchart illustrating an example of a processing procedure for an extended word vector creation process. 13 is a flowchart illustrating an example of a processing procedure for a word embedded expression correction process according to the second embodiment. 13 is a diagram illustrating an example of a process of processing an input sentence by a distributed representation generation process in the second embodiment. FIG. FIG. 13 is a diagram showing an example of an explanation of a pictogram.

The following describes an embodiment of the present invention in detail with reference to the drawings.

(1) First embodiment In the first embodiment, an example of a document processing device will be described that generates a reference vector (class vector) representing each type of proper noun for each type of proper noun and combines the class vector with the distributed representations of the words that make up the proper noun to generate a distributed representation of the proper noun with embedded meaning. The proper nouns used in this embodiment are under-learned words whose meaning cannot be adequately expressed in a pre-trained distributed representation space, and correspond to an example of a low-frequency word that has appeared infrequently in previous learning and has not been able to achieve sufficient accuracy.

(1-1) Configuration FIG. 1 is a block diagram showing an example of a functional configuration of a document processing device 100 according to a first embodiment of the present invention. The document processing device 100 is a computer (information processing device) in which a document processing program is installed, and as shown in FIG. 1, includes an input unit 101, a morphological analysis unit 102, a morphological analysis dictionary 103, a pre-trained neural language model 104 (a word variance generation unit 105, a sentence variance generation unit 106), an output unit 107, a target word extraction unit 108, a word variance correction unit 109, a class vector confirmation unit 110, a position vector storage unit 111, a class vector storage unit 112, and a class vector generation unit 113. Each of the above units will be described in detail after the description of FIG. 2 and FIG. 3. Note that the connection relationships between the components shown in FIG. 1 are only representative connection examples, and do not represent all connection relationships.

Fig. 2 is a diagram showing an example of the configuration of a document processing system 200 including a document processing device 100. When the document processing system 200 is realized, for example, as a client-server system, as shown in Fig. 2, a server 201, which is a client server, and multiple user terminals 203 used by users are communicatively connected via a network 202. The user terminals 203 are, for example, information processing devices such as personal computers (PCs) and smartphones. The network 202 may be any network such as the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network).

When the document processing system 200 is realized as a client-server system, the document processing program executed by the document processing device 100 is installed on the server 201. Therefore, the server 201 performs morphological analysis, generation of distributed representations, generation of class vectors, generation of extended word vectors, etc. as the document processing device 100. In this case, the user terminal 203 becomes an interface that inputs the document to be processed, transmits the document to be processed to the server 201, and receives the processing results from the server 201.

In this embodiment, the document processing system 200 can also be realized as a standalone system. In this case, the document processing program executed by the document processing device 100 is installed in the user terminal 203, and the server 201 and the network 202 are not required. Therefore, the user terminal 203 functions as the document processing device 100 and performs input of documents to be processed, data conversion of the input documents, output of the processing results, etc.

Figure 3 is a block diagram showing an example of the hardware configuration of the document processing device 100. The information processing device 300 shown in Figure 3 is an information processing device that realizes the document processing device 100, and corresponds to the server 201 and the user terminal 203 shown in Figure 2.

As shown in FIG. 3, the information processing device 300 has a processor 301, a storage device 302, an input device 303, an output device 304, and a communication interface (communication IF) 305. The processor 301, the storage device 302, the input device 303, the output device 304, and the communication IF 305 are connected by a bus 306, which is an internal communication line.

The processor 301 is a processor that controls the information processing device 300, and is, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The storage device 302 is a device that provides a working area for the processor 301, and is a non-temporary or temporary recording medium that stores various programs and data. Specifically, the storage device 302 is, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a HDD (Hard Disk Drive), or a flash memory. Note that the storage device 302 does not necessarily have to be a recording medium installed inside the information processing device 300, and some or all of the various programs and data may be stored outside (external storage medium, cloud, etc.) connected to the information processing device 300. Each processing unit of the document processing device 100 shown in FIG. 1 is mainly realized by the processor 301 reading and executing various predetermined programs. In addition, each storage unit (including the morphological analysis dictionary 103) of the document processing device 100 shown in FIG. 1 is realized by the storage device 302. The input device 303 is a device for inputting data, and specifically includes, for example, a keyboard, a mouse, a touch panel, a numeric keypad, a scanner, a microphone, or a biosensor. The output device 304 is a device for outputting data, and specifically includes, for example, a display, a printer, or a speaker. The communication IF 305 is an interface that connects to the network 202 and transmits and receives data to and from outside the information processing device 300, and specifically includes, for example, a network interface card (NIC).

The functional configuration of the document processing device 100 shown in FIG. 1 will be described.

The input unit 101 is a module that reads text data, and all processing and registration of text data is performed via the input unit 101. In the following explanation, the text contained in the text data read by the input unit 101 may be referred to simply as text.

The morphological analysis unit 102 performs morphological analysis of the text input from the input unit 101, and outputs the results to the word distributed representation generation unit 105 of the pre-trained neural language model 104 and the target word extraction unit 108. As pre-processing for the morphological analysis, the morphological analysis unit 102 performs normalization possible on a character-by-character basis by standardizing alphanumeric symbols and katakana to full-width characters, and converting different characters such as old characters into commonly used kanji.

Morphological analysis is a process in which text is divided into word strings using the morphological analysis dictionary 103 and reading and part-of-speech information is assigned. The morphological analysis dictionary 103 is a database that contains reading and part-of-speech information for each word, as well as the word occurrence cost and inter-word connection cost learned from a large text corpus. Therefore, in the morphological analysis process by the morphological analysis unit 102, the text of the text data is divided into word strings at appropriate positions in Japanese based on the information in the morphological analysis dictionary 103.

The morphological analysis dictionary 103 is composed of a system dictionary, which is a read-only dictionary prepared in advance on the system side, and a user dictionary, which is a readable and writable dictionary that the user can register and modify at will. By editing the user dictionary, the user can register new proper nouns in the morphological analysis dictionary 103. Note that some proper nouns may be registered in the system dictionary in advance. The system dictionary of the morphological analysis dictionary 103 is the same dictionary as the morphological analysis dictionary used when creating the pre-trained neural language model 104 described below.

The morphological analysis unit 102 further obtains a vocabulary list from the pre-trained neural language model 104 described below, and processes words to match the vocabulary list. The vocabulary list is a list of vocabulary included in the pre-trained neural language model 104. Due to the structure of the neural language model, it is necessary to limit the number of vocabulary words, so the total number of vocabulary words is reduced by performing subword division, which divides words divided by normal morphological analysis into subwords of smaller partial strings, and unknown word processing, which replaces low-frequency words (words that have a low frequency of occurrence in previous learning) with special tokens that indicate unknown words (words that are not treated as sufficiently learned words).

Therefore, the morphological analysis unit 102 re-divides the results of morphological analysis using the morphological analysis dictionary 103 into subword units defined in the vocabulary list, and if a word does not exist in the vocabulary list, it replaces it with a special token indicating an unknown word.

The pre-trained neural language model 104 is a language model that has a neural network and is trained on a large-scale corpus, and converts the word strings output by the morphological analysis unit 102 into distributed representations that are multidimensional real vectors. The pre-trained neural language model 104 has a word distributed representation generation unit 105 and a sentence distributed representation generation unit 106.

An example of a pre-trained neural language model 104 having a word distributed representation generation unit 105 and a sentence distributed representation generation unit 106 is BERT (Bidirectional Encoder Representations from Transformers). In this case, the embedding layer in BERT corresponds to the word distributed representation generation unit 105, and the encoder layer corresponds to the sentence distributed representation generation unit 106.

The word distributed representation generation unit 105 converts the word string output by the morphological analysis unit 102 into a distributed representation (word distributed representation) which is a multidimensional real vector for each word via a neural network. More specifically, the word distributed representation generation unit 105 converts the words into word IDs, further converts these into one-hot vectors, and then inputs the one-hot vectors into the neural network to obtain a distributed representation (word distributed representation) for each word. The word IDs are consecutive integers that are uniquely assigned to words in the vocabulary list.

A one-hot vector is a vector in which one component is "1" and the remaining components are all "0". A one-hot vector is a vector with the same dimension as the number of word IDs, and a word with a word ID of "n" is converted into a one-hot vector in which the nth element is "1". The word distributed representation generation unit 105 inputs the one-hot vector converted as described above into a neural network and converts it into a word distributed representation, which is a real vector.

The sentence variance generating unit 106 synthesizes the word variances output by the word variance generating unit 105 (or replaced by the word variance correction unit 109 described below) to generate a variance for the entire sentence (or word string). Note that the pre-trained neural language model 104 is not limited to BERT, and any language model with an architecture similar to BERT may be used as long as it can convert a word string into a variance for each word and a variance for the entire sentence. Also, the word variance generating unit 105 may use Word2Vec, and the sentence variance generating unit 106 may be configured to take the average of the word variances output by the word variance generating unit 105 to generate the sentence variance.

The output unit 107 outputs the distributed representation of the entire sentence (or word string) generated by the sentence distributed representation generation unit 106 via a predetermined output means via the output device 304 (such as display output to a display or data output to an output file). At this time, the output unit 107 may output the distributed representation (distributed vector) for the entire sentence, or may output the distributed vector for each word.

The target word extraction unit 108 extracts words of a predetermined part of speech (e.g., proper nouns such as place names, personal names, and restaurant names) that correspond to the class vectors stored in the class vector storage unit 112 from the word string output by the morphological analysis unit 102. Note that the words extracted here are words before subword division and unknown word processing are performed according to the vocabulary of the pre-trained neural language model 104 described above. The target word extraction unit 108 also extracts word variances of word strings that correspond to the extracted proper nouns from the word variances output by the word variance generation unit 105.

The word variance correction unit 109 generates word variances for the proper noun by combining the word variances for the multiple words constituting the proper noun output by the target word extraction unit 108 with the class vector stored in the class vector storage unit 112 and the position vector stored in the position vector storage unit 111. The word variance correction unit 109 then outputs the generated word variances for the proper noun to the word variance generation unit 105, replaces the word variances for the proper noun in the word variance generation unit 105 with the generated word variances, and inputs the replaced word variances to the sentence variance generation unit 106.

The class vector confirmation unit 110 has a function of outputting the class vector generated by the class vector generation unit 113 described below as a class vector candidate via the output device 304, and accepting manual correction of the class vector candidate from the user via the input device 303. When the class vector confirmation unit 110 accepts manual correction of the class vector candidate, it transfers the class vector candidate after manual correction to the class vector generation unit 113, and registers the class vector candidate as a class vector in the class vector storage unit 112 (alternatively, it may be registered in the class vector storage unit 112 directly from the class vector confirmation unit 110).

The position vector storage unit 111 stores the position vectors that the word embedded representation correction unit 109 synthesizes into the word embedded representations. The position vectors provide position information for the embedded representations of each word that constitutes a proper noun by adding up absolute values corresponding to their positions. One technique for realizing position vectors is, for example, the position encoding of the Transformer.

The class vector storage unit 112 stores the class vectors that the word variance correction unit 109 synthesizes into the word variances. The class vector storage unit 112 stores class vectors created for each type of proper noun by the class vector creation process described below by the class vector generation unit 113.

The class vector generation unit 113 generates class vectors that the word variance correction unit 109 synthesizes into word variances, and stores them in the class vector storage unit 112. The generation of class vectors is performed before the document processing device 100 performs document processing.

(1-2) Class Vector Creation Process Fig. 4 is a flowchart showing an example of the processing procedure of the class vector creation process. The class vector creation process shown in Fig. 4 is a process executed at any timing before the execution of the distributed representation generation process described later, in which the document processing device 100 shown in Fig. 1 creates a class vector and registers it in the class vector storage unit 112.

According to FIG. 4, first, the class vector generation unit 113 reads the vocabulary list of the pre-trained neural language model 104 via the input unit 101 (step S401). The vocabulary list is a list of all vocabulary words (all vocabulary words held by the word variance generation unit 105) included in the pre-trained neural language model 104, and can be acquired from the pre-trained neural language model 104. If the vocabulary list read in step S401 includes subwords, the target word extraction unit 108 excludes the subwords and extracts only words that are not divided into subwords (step S402). The vocabulary list includes information on whether a word is a subword, and the target word extraction unit 108 excludes the subwords based on this information. If the vocabulary list does not include subwords, the process of step S402 is omitted.

Next, the morphological analysis unit 102 refers to the morphological analysis dictionary 103 to check the part of speech of the words included in the vocabulary list loaded in step S401 other than the subwords extracted in step S402, and extracts proper nouns (step S403). Then, the class vector generation unit 113 converts the proper noun words extracted by the morphological analysis unit 102 in step S403 into word variance representations via the word variance representation generation unit 105 (step S404).

When all proper nouns extracted by the morphological analysis unit 102 are converted into word variance representations in step S404, the class vector generation unit 113 creates variance representations by averaging the variance representations for each proper noun type in the word variance representation generation unit 105, and sets these as class vectors for each proper noun type (step S405). Note that the class vectors created by the class vector generation unit 113 in step S405 can be corrected by the user in step S407 described below (i.e., they are not necessarily the class vectors that are ultimately registered in the class vector storage unit 112), and therefore can also be called class vector candidates.

Furthermore, in order to visualize the word distributed representations and class vectors of the proper nouns created in step S405, the class vector generation unit 113 performs dimensionality reduction on the distributed representations of the words and creates a two-dimensional scatter plot (step S406). For example, principal component analysis can be used for dimensionality reduction, but any other method can be used as long as it can summarize the features that represent the differences in the distributed representations of each proper noun.

Next, the class vector generation unit 113 transmits the scatter plot created in step S406 to the class vector confirmation unit 110, which outputs it using the output device 304 (a display device such as a monitor) (step S407).

Figure 5 shows an example of a screen display of a scatter plot of word distributed representations of proper nouns. Scatter plot 500 shown in Figure 5 is a specific example of a scatter plot output in step S407 of Figure 4. Scatter plot 500 shows a group of distributed representations 501 of place names such as "Tokyo" and "Osaka", and a group of distributed representations 502 of restaurant names such as "○× Cafe" and "○△ Shokudo". Furthermore, cross marks 503 and 504 shown in Figure 5 represent the distributed representations of class vector candidates generated from the distributed representations of the proper nouns of place names and restaurant names (restaurants in the figure).

Here, if the group of distributed representations of each proper noun contains a distributed representation with a value that is significantly different from the majority of distributed representations, the average of the group of distributed representations is not necessarily appropriate as a class vector. Therefore, in step S407, the class vector confirmation unit 110 may accept manual corrections by the user using the input device 303 (mouse, keyboard, etc.) for the class vector candidates shown in the output scatter plot (for example, the x marks 503 and 504 in the scatter plot 500 in FIG. 5).

Figure 6 is a diagram showing an example of a screen display when correcting a class vector candidate. The scatter diagram 600 shown in Figure 6 is mostly the same as the scatter diagram 500 in Figure 5, with the addition of a cross mark 601. The cross mark 601 is the position of the place name class vector that the user considers appropriate, and the user performs a predetermined operation to select the cross mark 601 on the scatter diagram 600, and it is determined that the place name class vector candidate is corrected from the cross mark 503 to the cross mark 601. The appropriate position of the class vector candidate can be determined arbitrarily by the user, and specifically, for example, it may be determined based on the average of the vectors in the distributed representation group of each proper noun, or may be determined based on any distributed representation, or may be determined by other determination methods. Then, when the correction of the class vector candidate is determined, the class vector confirmation unit 110 (or the class vector generation unit 113 that receives information from the class vector confirmation unit 110) creates a new class vector candidate according to the above determination. Furthermore, if the user determines that no correction is required for the class vector candidates of each distributed representation group, a predetermined operation indicating confirmation is performed, and confirmation of the class vector candidates displayed in the scatter plot 500 (or scatter plot 600) in step S407 is terminated.

Then, depending on the result of the correction or confirmation of the class vector candidates in step S407, the class vector generation unit 113 registers each of the finally confirmed class vector candidates as a class vector in the class vector storage unit 112 (step S408), and completes the class vector generation process. Note that in step S408, the class vector is registered in the class vector storage unit 112 as data paired with the proper noun type. As a result, the word distributed representation correction unit 109 can obtain the class vector of the proper noun type by referring to the class vector storage unit 112 using the proper noun type, such as "restaurant" or "place name", as a key.

(1-3) Distributed Representation Modification Processing Fig. 7 is a flowchart showing an example of the processing procedure of the distributed representation generation processing. The distributed representation generation processing shown in Fig. 7 is processing in which the document processing device 100 shown in Fig. 1 converts text included in text data into sentence distributed representations. The distributed representation generation processing is executed, for example, when a text input sentence (text data) is input to the input unit 101 during operation of the document processing device 100 after the class vector creation processing shown in Fig. 4 is completed.

Figure 8 is a flowchart showing an example of the processing procedure of the word dispersed representation correction process. The word dispersed representation correction process shown in Figure 8 corresponds to the processing of step S705 of the dispersed representation generation process shown in Figure 7, and is executed by the target word extraction unit 108 and the word dispersed representation correction unit 109. Also, Figure 9 is a diagram explaining an example of the processing process of an input sentence by the dispersed representation generation process.

The distributed representation generation process in this embodiment will be described in detail below with reference to the flowcharts in Figures 7 and 8, and also with reference to the specific example in Figure 9 as appropriate.

According to FIG. 7, first, the input unit 101 reads the input sentence 901, which is text data (step S701). FIG. 9 shows that the input sentence 901 is the text "Where is the ○×△ cafe?"

Next, the morpheme analysis unit 102 performs a morpheme analysis on the input sentence 901 input in step S701 (step S702). As shown in FIG. 9, the input sentence 901 is divided by the morpheme analysis into three morphemes (words) 902-904, namely, "○×△ cafe", "wa", and "doko". In addition, part-of-speech information 905-907 is assigned to each morpheme 902-904. Specifically, the morpheme 902 of "○×△ cafe" is assigned part-of-speech information 905 of "proper noun-restaurant", the morpheme 903 of "wa" is assigned part-of-speech information 906 of "particle", and the morpheme 904 of "doko" is assigned part-of-speech information 907 of "pronoun".

Next, the morpheme analysis unit 102 refers to the vocabulary list of the pre-trained neural language model 104 and divides the morphemes 902 to 904 into subwords (step S703). In the case of FIG. 9, specifically, the proper noun "O x △ cafe" (morpheme 902) is divided into four subword tokens 908 to 911: "O", "X", "△", and "cafe". In addition, if the particle "wa" (morpheme 903) and the pronoun "doko" (morpheme 904) are pre-trained words registered in the vocabulary list of the pre-trained neural language model 104 and do not need to be divided more finely than they are currently, two tokens 912 and 913 are generated without division into subwords.

The vocabulary list of the pre-trained neural language model 104 is learned from the frequency of occurrence in the training corpus during pre-training of the pre-trained neural language model 104. Therefore, in step S703, even proper nouns may not be divided into subwords, and even common nouns may be divided into subwords. That is, if a proper noun appears frequently in pre-training, sufficient accuracy has been obtained and therefore division into subwords is not necessary, whereas if a common noun appears infrequently in pre-training, sufficient accuracy has not been obtained and therefore division into subwords is performed.

Next, the word embedding generation unit 105 converts the tokens (word strings) 908 to 913 after processing in step S703 into word embedding expressions 914 to 919, respectively (step S704).

Next, the target word extraction unit 108 and the word dispersed representation correction unit 109 execute a word dispersed representation correction process (step S705). The word dispersed representation correction process executed in step S705 is a process of correcting the dispersed representation of a word (proper noun word 902 in this example) that was divided into meaningless subwords in step S703, among the morphemes (words) 902 to 904 output in step S702, to generate an dispersed representation of a single word, and as mentioned above, an example of the detailed processing procedure is shown in FIG. 8. The word dispersed representation correction process will be described in detail below with reference to FIG. 8.

As shown in FIG. 8, first, the target word extraction unit 108 focuses on the first word 902 among the words 902 to 904 output by the morphological analysis unit 102 in step S702 (step S801).

Next, the target word extraction unit 108 refers to the part-of-speech information 905 of the focused word 902 and determines whether it is a proper noun to be processed (step S802). The process of step S802 is to determine whether the focused word 902 is a word to be targeted for the word embedding conversion process performed in steps S803 to S807 described below. In this case, the proper noun word 902 divided into subwords in step S703 of FIG. 7 corresponds to the word to be processed. In step S802, if it is a proper noun to be processed (YES in step S802), the process proceeds to step S803, and if it is not a proper noun to be processed (NO in step S802), the process proceeds to step S808 described below.

Referring to FIG. 9, since the part of speech information 905 of "proper noun-restaurant" is a proper noun, the result of step S802 is "YES", and the process proceeds to step S803. In step S803, the word hull correction unit 109 obtains word hulls 914 to 917 of the words that make up the focused word 902 from the word hull generation unit 105. As described above, the word hulls 914 to 917 were generated by the word hull generation unit 105 in step S704 of FIG. 7.

Next, the word ambiguity correction unit 109 synthesizes position vectors for the word ambiguities 914 to 917 obtained in step S803 (step S804).

The processing of step S804 will be described in detail. First, the word shared expression correction unit 109 obtains from the position vector storage unit 111 the position vectors (position vectors 920 to 923 shown in FIG. 9) to be synthesized with each word of the word shared expressions 914 to 917. In FIG. 9, the position vectors 920 to 923 are written as "P1", "P2", "P3", and "P4" in order, which indicate the order of positions. That is, the position vector 920 "P1" is the vector indicating the first position, and the position vectors 921 to 923 are vectors indicating the second, third, and fourth positions, respectively.

Furthermore, in step S804, the word hull correction unit 109 adds the acquired position vectors 920-923 to the word hulls 914-917. Specifically, the word hull 914 of the first word "○" is synthesized with the position vector 920 of "P1" indicating the first word, and the word hull 915 of the second word "×" is synthesized with the position vector 921 of "P2" indicating the second word, and similarly for the third and fourth words. The above synthesis process is necessary to generate a unique vector, and by synthesizing the word hulls 914-917 with the position vectors 920-923, the word order of the subwords in word 902 can be maintained.

Then, after combining the position vectors 920-923 with the word embedded representations 914-917 of all words in step S804, the word embedded representation correction unit 109 takes the arithmetic mean of these word embedded representations with the combined position vectors, and combines them into one embedded representation (step S805).

Next, the word distributed representation correction unit 109 combines one distributed representation combined in step S805 with a class vector corresponding to the distributed representation (class vector 924 shown in FIG. 9) (step S806).

To explain the process of step S806 in more detail, first, the word distributed representation correction unit 109 refers to the class vector storage unit 112 using the part-of-speech information 905 as a key, and obtains the class vector 924 of "proper noun-restaurant" that corresponds to the part-of-speech information 905.

Furthermore, in step S806, the word shared representation correction unit 109 multiplies the shared representation generated in step S805 by a predetermined coefficient and adds the result to the class vector 924. This predetermined coefficient is for adjusting the shared representation of the synthesized proper noun so that it is close to the class vector. By making the shared representation of the synthesized proper noun close to the class vector, it is possible to generate a unique shared representation for each proper noun while including the meaning of the class. For example, the above-mentioned predetermined coefficient can be determined by the following method. First, in the class vector creation process of FIG. 4, the maximum value α of the Euclidean distance between the class vector and the shared representation of the proper noun that is the basis of the class vector is obtained in advance. Furthermore, the processes of steps S803 to S805 of FIG. 8 are performed for all proper nouns for each class to obtain shared representations. Then, the average of these is calculated, and the maximum value β of the Euclidean distance between this average and each shared representation is obtained. Then, the ratio (α/β) of the obtained α and β is set as the predetermined coefficient.

In this manner, in step S806, the word embedded expression correction unit 109 generates an embedded expression 922 that is synthesized. As shown in FIG. 9, the embedded expression 922 represents an embedded expression of a single proper noun, "○△× cafe."

Next, the word hull correction unit 109 replaces the word hulls 914 to 917 held by the word hull generation unit 105 with the word hull 925 newly generated in step S806, and completes the processing from step S802 onwards for the word 902 (step S807).

Next, the word variance correction unit 109 (or the target word extraction unit 108) checks whether there are any unprocessed words (i.e., words not focused on in step S801 or step S809) among the words 902 to 904 output by the morphological analysis unit 102 in step S702 of FIG. 7 (step S808). If there are any unprocessed words (YES in step S808), the processing in steps S802 and after is repeated, focusing on one of the words (step S809). On the other hand, if there are no unprocessed words (NO in step S808), the word variance correction processing is terminated.

Specifically, for example, in step S808 after the processing of steps S802 to S807 for word 902 is completed, since words 903 and 904 have not been processed (not been focused on), the focus is on word 903 in step S809, for example, and the processing of step S802 is performed. However, since the part of speech information 906 of word 903 is "particle," it is determined in step S802 that it is not the proper noun to be processed (NO in step S802), and no special conversion processing is performed, and the process proceeds to step S808. As a result, in the example of FIG. 9, word hull 918 converted from word 903 in step S704 becomes word hull 926. In the next step S808, since word 904 has not been processed, the focus is on word 904 in step S809, and the processing of step S802 is performed again. However, since word 904 is also not a proper noun to be processed, step S802 returns NO, and the process proceeds to step S808 without any special conversion processing being performed. As a result, in the example of FIG. 9, word hull 919 converted from word 904 in step S704 becomes word hull 927. In step S808 after the above processing, there are no more unprocessed words (i.e., processing of all words 902 to 904 has been completed), so step S808 returns NO, and the word hull correction processing ends.

Returning to the explanation of FIG. 7, after the word dispersed representation correction process of step S705 has been performed, the sentence dispersed representation generation unit 106 synthesizes the word dispersed representations (e.g., word dispersed representations 925 to 927 shown in FIG. 9) corrected by the word dispersed representation correction unit 109 in step S705 to generate a sentence dispersed representation of the entire input sentence 901 (step S706), and ends the dispersed representation generation process.

As described above, according to the document processing device 100 of this embodiment, the class vector creation process of FIG. 4 can generate a class vector representing each type of low-frequency word for the distributed representation of the low-frequency word (e.g., a proper noun) for which sufficient semantic accuracy has not been obtained in the pre-learning. Then, the word distributed correction process of FIG. 8 performed in the distributed representation generation process of FIG. 7 can correct (replace) the word distributed representation of the low-frequency word by adding sufficient meaning to the low-frequency word by synthesizing a position vector that fixes the order of subwords with the distributed representation of the low-frequency word included in the input sentence and synthesizing it with the class vector. Thus, the document processing device 100 can generate a unique sentence distributed representation in which the meaning of the low-frequency word is appropriately embedded for a sentence including a low-frequency word (e.g., a proper noun) for which sufficient semantic accuracy has not been obtained in the pre-learning, without performing additional learning, by performing the class vector creation process of FIG. 4 and then executing the distributed representation generation process of FIG. 7.

(2) Second embodiment In the second embodiment, an example of a document processing device will be described in which a reference vector (class vector) representing a symbol is generated for a distributed representation of an emoji symbol, and the distributed representation of an explanation of the emoji and the class vector are combined to generate a distributed representation of the symbol with embedded meaning. The emojis used in this embodiment are under-learned words whose meaning cannot be adequately expressed in a pre-trained distributed representation space, and correspond to an example of an unknown word that has not been pre-trained.

(2-1) Configuration Fig. 10 is a block diagram showing an example of a functional configuration of a document processing device 1000 according to the second embodiment of the present invention. The document processing device 1000 is configured by adding an extended word vector generation unit 1003 and an extended word vector storage unit 1004 to the configuration of the document processing device 100 according to the first embodiment. Note that a target word extraction unit 1001 and a word sparse expression correction unit 1002 in the document processing device 1000 are processing units corresponding to the target word extraction unit 108 and the word sparse expression correction unit 109 in the document processing device 100, but new reference numerals are assigned to them because they perform some different processing.

The document processing device 1000 can be included in the document processing system 200 shown in FIG. 2, similar to the document processing device 100 according to the first embodiment. Also, the document processing device 1000 is realized by an information processing device 300 having the hardware configuration example shown in FIG. 3, similar to the document processing device 100 according to the first embodiment. Specifically, the extended word vector generation unit 1003 is realized by the processor 301 reading and executing various predetermined programs, and the extended word vector storage unit 1004 is realized by the storage device 302.

The extended word vector generation unit 1003 acquires, via the sentence dispersed representation generation unit 106, the sentence explanatory text of the word (an unknown word, in this example, a symbol representing an emoji) that is the basis of the extended word vector, generates dispersed representations (extended word vectors) to which a symbolic meaning is assigned, and registers the generated dispersed representations in the extended word vector storage unit 1004. The extended word vector storage unit 1004 stores the extended word vectors generated by the extended word vector generation unit 1003. The generation of extended word vectors will be described in detail in the extended word vector creation process shown in FIG. 11, which will be described later.

Figure 14 is a diagram showing an example of an explanation of an emoji. In Figure 14, multiple emojis 1401 and explanations 1402 for each emoji 1401 are shown linked to each other. If the emoji 1401 is an emoji symbol defined in UNICODE, the explanation 1402 contains the UNICODE explanation of the corresponding symbol. The emoji symbol and explanation defined in UNICODE are registered in the system dictionary of the morphological analysis dictionary 103. However, the description of the explanation 1402 is not limited to this, and the user may create it independently. Furthermore, symbols other than those defined in UNICODE can also be used as targets for generating extended word vectors by being registered in the user dictionary of the morphological analysis dictionary 103, for example.

As described above, in this embodiment, emoji symbols are used as an example of extended words (unknown words), and the extended word vector generation unit 1003 generates extended word vectors for each emoji 1401 by performing an extended word vector creation process in pre-processing, and registers the generated extended word vectors in the extended word vector storage unit 1004.

(2-2) Extended word vector creation process Fig. 11 is a flowchart showing an example of the processing procedure of the extended word vector creation process. The extended word vector creation process shown in Fig. 11 is a process executed at an arbitrary timing before the execution of the distributed representation generation process, similar to the class vector creation process shown in Fig. 4 in the first embodiment, in which the document processing device 1000 creates extended word vectors and registers them in the extended word vector storage unit 1004.

According to FIG. 11, first, the class vector generation unit 113 generates a class vector representing multiple symbols (emoticons 1401 in FIG. 14) (step S1101). The procedure for generating the class vector in step S1101 is almost the same as the procedure for the class vector creation process shown in FIG. 4, and in step S403 in FIG. 4, symbols are extracted instead of proper nouns. The class vectors of the symbols generated in step S1101 are registered in the class vector storage unit 112.

Next, the class vector generation unit 113 calculates the Euclidean distance between the class vector of the symbol generated in step S1101 and the distributed representation of each symbol, and finds the maximum Euclidean distance (step S1102).

Next, the extended word vector generation unit 1003 converts all the explanatory sentences 1402 of the emojis into sentence dispersed representations via the input unit 101, the morphological analysis unit 102, the word dispersed representation generation unit 105, and the sentence dispersed representation generation unit 106 (step S1103). In step S1103, the explanatory sentences 1402 of all the target emojis 1401 are converted into sentence dispersed representations. The procedure for generating sentence dispersed representations in step S1103 is the same as the dispersed representation generation process shown in FIG. 7 (however, the word dispersed representations are not modified in step S705).

Next, the extended word vector generation unit 1003 calculates the average of the sentence variances of all the explanatory sentences created in step S1103 (step S1104). Then, the Euclidean distance between the calculated average of the sentence variances of all the explanatory sentences and the variances of each explanatory sentence is calculated, and the maximum value of the Euclidean distance is obtained (step S1105).

Next, the extended word vector generation unit 1003 uses the maximum Euclidean distance of the symbol embedded representation calculated in step S1102 and the maximum Euclidean distance of the sentence embedded representation of the explanatory sentence calculated in step S1105 to calculate the ratio of the latter maximum value to the former maximum value (step S1106).

Next, the extended word vector generation unit 1003 subtracts the average of the distributed representations of all the explanatory sentences from the sentence distributed representation of the explanatory sentence, and multiplies the ratio of the Euclidean distances calculated in step S1106 to generate a distributed representation (extended word vector) of the emoji corresponding to each explanatory sentence (step S1107).

Finally, the extended word vector generation unit 1003 registers the extended word vector generated in step S1107 in the extended word vector storage unit 1004 (step S1108), and ends the extended word vector creation process.

By executing the extended word vector creation process as described above, the extended word vector generation unit 1003 can generate a unique distributed representation (extended word vector) for each explanatory sentence (in other words, for each emoji corresponding to the explanatory sentence) while incorporating the properties of a symbol.

(2-3) Word Distributed Representation Generation Process The dispersed representation generation process executed in the second embodiment will be described. The dispersed representation generation process in this embodiment is a process in which the document processing device 1000 shown in FIG. 10 converts text included in text data into sentence dispersed representations. The dispersed representation generation process in this embodiment is executed, for example, when a text input sentence (text data) is input to the input unit 101 during operation of the document processing device 1000 after the class vector creation process shown in FIG. 4 and the extended word vector creation process shown in FIG. 11 are completed. Note that the dispersed representation generation process in the second embodiment is mostly executed in the same processing procedure as the dispersed representation generation process shown in FIG. 7 in the first embodiment, and the word dispersed representation correction process in step S705 is executed in a processing procedure specific to the second embodiment.

Fig. 12 is a flowchart showing an example of the processing procedure of the word dispersed representation correction process in the second embodiment. The word dispersed representation correction process shown in Fig. 12 corresponds to the processing of step S705 of the dispersed representation generation process shown in Fig. 7 in the second embodiment, and is executed by the target word extraction unit 1001 and the word dispersed representation correction unit 1002. Fig. 13 is a diagram explaining an example of the processing process of an input sentence by the dispersed representation generation process in the second embodiment.

The distributed representation generation process in this embodiment will be described in detail below in accordance with the flowcharts in Figs. 7 and 12, with reference to the specific example in Fig. 13 as appropriate. Note that detailed description of the process in Fig. 7 that is common to the first embodiment may be omitted.

In the distributed representation generation process, the input unit 101 first reads the input sentence 1301, which is text data (step S701). FIG. 13 shows an example of the input sentence 1301, which is text including an emoji 1302. Note that the emoji 1302 is an emoji to which the description 1402, "smiling face," is added in FIG. 14.

Next, the morpheme analysis unit 102 performs a morpheme analysis on the input sentence 1301 input in step S701 (step S702). As shown in FIG. 13, the input sentence 1301 is divided into two morphemes (words) 1303 and 1304 by the morpheme analysis. In addition, part-of-speech information 1305 and 1306 are assigned to each morpheme 1303 and 1304. Specifically, part-of-speech information 1305 of "interjection" is assigned to morpheme 1303, and part-of-speech information 1306 of "symbol-emoji" is assigned to morpheme 1304.

Next, the morpheme analysis unit 102 refers to the vocabulary list of the pre-trained neural language model 104 and divides the morphemes 1303 and 1304 into subwords (step S703). Specifically, since the morpheme 1303, which is an interjection, exists in the vocabulary list, it is converted into the token 1307 "thank you" without being divided into subwords. On the other hand, the morpheme 1304, which is an emoji symbol, does not exist in the vocabulary list, so it is converted into the special token "UNK" (token 1308) that represents an unknown word in the pre-trained neural language model 104.

Next, the word embedding generation unit 105 converts the tokens (word strings) 1307 and 1308 after processing in step S703 into word embedding expressions 1309 and 1310, respectively (step S704).

Next, the word embedded representation correction process shown in FIG. 12 is executed (step S705). The word embedded representation correction process executed in step S705 in this embodiment is a process of correcting the embedded representation of a word (in this example, the emoji word 1304) that has been divided into meaningless subwords among the morphemes (words) 1303 and 1304 output in step S702, and generating an embedded representation of a single word, and as described above, an example of the detailed processing procedure is shown in FIG. 12. The word embedded representation correction process will be described in detail below with reference to FIG. 12.

According to FIG. 12, first, the target word extraction unit 1001 focuses on the first word 1303 of the words 1303 and 1304 output in step S702 (step 1201). Then, based on the part-of-speech information 1305 of the focused word 1303, the target word extraction unit 1001 refers to the extended word vector storage unit 1004 and determines whether or not the focused word 1303 is a word to be processed (step S1202). In step S1202, if the extended word vector of the focused word 1303 is registered in the extended word vector storage unit 1004, it can be determined that the focused word 1303 is a word to be processed. In step S1202, if the focused word 1303 is an emoji to be processed (YES in step S1202), the process proceeds to step S1203, and if the focused word 1303 is not an emoji to be processed (NO in step S1202), the process proceeds to step S1207.

Specifically, in the case of word 1303, since the part of speech information 1305 is "interjection", it is determined in step S1202 that it is not to be processed (NO in step S1202), and the process proceeds to step S1207.

In step S1207, the word variance correction unit 1002 (or the target word extraction unit 1001) checks whether there are any unprocessed words (i.e., words not focused on in step S1201 or step S1208) among the words 1303, 1304 output in step S702. If there are any unprocessed words (YES in step S1207), the process focuses on one of the words (step S1208) and repeats the process from step S1202 onwards. On the other hand, if there are no unprocessed words (NO in step S1207), the word variance correction process ends.

Specifically, when word 1303 is the focus, word 1304 has not been processed, so the result of step S1207 is "NO," and word 1304 is focused on in step S1208, and the process proceeds to step S1202.

In step S1202, when word 1304 is focused on, the part-of-speech information 1306 is "symbol-emoji" and the extended word vector is registered in the extended word vector storage unit 1004, so it is determined to be a processing target (YES in step S1202) and the process proceeds to step S1203.

In step S1203, the word variance correction unit 1002 refers to the class vector storage unit 112 to obtain the class vector 1312 of the symbol of the word 1304 focused on in step S1202. The word variance correction unit 1002 also refers to the extended word vector storage unit 1004 to obtain the extended word vector 1311 corresponding to the emoji (symbol) word 1304 (step S1204).

Next, the word embedded representation correction unit 1002 adds the extended word vector 1311 obtained in step S1204 and the class vector 1312 obtained in step S1203 to generate a word embedded representation 1314 for the emoji word 1304 (step S1205).

Next, the word hull correction unit 1002 replaces the word hull 1310 held by the word hull generation unit 105 with the word hull 1314 newly generated in step S1205 (step S1206).

When the processing of step S1206 is completed, the process proceeds to step S1207. In this case, however, since the processing from step S1202 onwards has been completed for all words 1303 and 1304, the result is determined to be "NO" and the word embedded expression correction process ends.

Returning to the explanation of FIG. 7, after the word dispersed representation correction process of step S705 has been performed, the sentence dispersed representation generation unit 106 synthesizes the word dispersed representations (e.g., the word dispersed representations 1313 and 1314 shown in FIG. 13) corrected by the word dispersed representation correction unit 1002 in step S705 to generate a sentence dispersed representation of the entire input sentence 1301 (step S706), and ends the dispersed representation generation process.

As described above, according to the document processing device 1000 of this embodiment, the extended word vector creation process of FIG. 11 generates a class vector representing a plurality of unknown words for a distributed representation of an unknown word (e.g., an emoji symbol) for which sufficient semantic accuracy has not been obtained in pre-learning, and generates a unique distributed representation (extended word vector) for each unknown word using an explanatory text for the plurality of unknown words. Then, the word distributed correction process of FIG. 12 performed in the distributed representation generation process of FIG. 7 combines the distributed representation (extended word vector) of the unknown word and the class vector for the unknown word included in the input sentence, thereby giving sufficient meaning to the unknown word and correcting (replacing) the word distributed representation of the unknown word. Thus, the document processing device 1000 executes the distributed representation generation process of FIG. 7 through the class vector creation process of FIG. 11, and thereby generates a unique sentence distributed representation in which the meaning of the unknown word is appropriately embedded for a sentence including an unknown word (e.g., an emoji symbol) for which sufficient accuracy has not been obtained in pre-learning, without performing additional learning. In the above explanation, emoticon symbols are used as an example, but the document processing device 1000 according to this embodiment can also process unknown words such as emoticons composed of symbols in the same way if there is an explanatory text corresponding to the symbols as shown in FIG. 14. In this case, the emoticons are assumed to be registered in the morphological analysis dictionary 103.

The present invention is not limited to the above-described embodiments, but includes various modified examples. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described. It is also possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

Furthermore, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in hardware, in part or in whole, for example by designing them as integrated circuits. Furthermore, the above-mentioned configurations, functions, etc. may be realized in software, by a processor interpreting and executing a program that realizes each function. Information on the programs, tables, files, etc. that realize each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

In addition, the control lines and information lines shown in the drawings are those that are considered necessary for explanation, and do not necessarily show all control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected.

100, 1000 Document processing device 101 Input unit 102 Morphological analysis unit 103 Morphological analysis dictionary 104 Pre-trained neural language model 105 Word distributed representation generation unit 106 Sentence distributed representation generation unit 107 Output unit 108, 1001 Target word extraction unit 109, 1002 Word distributed representation correction unit 110 Class vector confirmation unit 111 Position vector storage unit 112 Class vector storage unit 113 Class vector generation unit 200 Document processing system 201 Server 202 Network 203 User terminal 300 Information processing device 301 Processor 302 Storage device 303 Input device 304 Output device 305 Communication interface 306 Bus 1003 Extended word vector generation unit 1004 Extended word vector storage unit

Claims (9)

A document processing device that generates embedded representations of words included in an externally input text,
a class vector storage unit that stores a class vector which is a reference vector for each type of word;
a morphological analysis unit that divides the text into word strings by morphological analysis and assigns information indicating a type to each word string;
a word embedding generation unit that converts the word string divided by the morphological analysis unit into an embedding representation of each word included in the word string;
a word embedding correction unit that corrects the embeddings generated by the word embedding generation unit;
Equipped with
When the text includes a low-frequency word whose occurrence frequency in pre-learning is lower than a predetermined level,
The morphological analysis unit divides the text into word strings by morphological analysis, and then further divides the low-frequency words included in the word strings into subwords of partial character strings;
The word embedding generation unit converts the low-frequency word into a plurality of embedding expressions including embedding expressions for the subwords included in the low-frequency word;
the word variance correction unit combines a position vector that fixes the order of the subwords in the low-frequency word with a plurality of variance expressions of the low-frequency word generated by the word variance generation unit, combines the position vector with the class vector of the type to which the low-frequency word belongs, and replaces the variance expression after the combination as the variance expression of the low-frequency word. 2. The document processing device according to claim 1, further comprising a class vector generation unit that generates, for each type to which the low-frequency words belong, a class vector for each type based on a distributed representation of the low-frequency words belonging to the same type, and registers the generated class vector in the class vector storage unit. The class vector generation unit
The document processing device according to claim 2, characterized in that an average of a distributed representation generated by the word distributed representation generation unit for one word among a plurality of the low-frequency words belonging to the same type, or a distributed representation generated by the word distributed representation generation unit for a plurality of the low-frequency words belonging to the same type, is generated as a class vector of the type to which the low-frequency word belongs. The word embedding correction unit,
The document processing device according to claim 3, characterized in that the distributed representation of the low-frequency word generated by the word distributed representation generation unit is multiplied by a predetermined coefficient for adjusting the distributed representation after synthesis so that the distributed representation after synthesis is in the vicinity of the class vector, and then synthesized with the class vector of the type to which the low-frequency word belongs, and the distributed representation after synthesis is replaced as the distributed representation of the low-frequency word. a class vector confirmation unit that outputs the class vector of the type to which the low-frequency word belongs, generated by the class vector generation unit, as a class vector candidate and requests a user to confirm it, and when a correction request is made by the user, corrects the class vector candidate in accordance with the request;
3 . The document processing device according to claim 2 , wherein the class vector generating unit registers the class vector candidate corrected or confirmed by the class vector confirming unit in the class vector storage unit as a class vector of a type to which the low-frequency word belongs. a sentence embedded representation generation unit that generates an embedded representation of the entire text by synthesizing the embedded representations generated by the word embedded representation generation unit or the word embedded representation correction unit for each word included in the word string that constitutes the text; and
The document processing device according to claim 1 , further comprising: an output unit that outputs the distributed representations of the entire text generated by the sentence distributed representation generation unit. a class vector generating unit that generates a class vector for each type to which the low-frequency words belong based on a distributed representation of a plurality of the low-frequency words belonging to the same type, and registers the generated class vector in the class vector storage unit;
An explanation for each of a plurality of unknown words that have not been pre-trained is maintained,
the class vector generation unit generates a class vector for each type to which the unknown words belong based on a distributed representation of the unknown words belonging to the same type, and registers the generated class vector in the class vector storage unit;
the sentence embedded representation generation unit converts a plurality of explanatory sentences for the plurality of unknown words into embedded representations;
an extended word vector generation unit that generates a distributed representation of each unknown word based on a comparison between the distributed representations of the plurality of explanatory sentences and a distributed representation of an explanatory sentence for each unknown word;
an extended word vector storage unit that stores the distributed representations of each unknown word generated by the extended word vector generation unit;
Further equipped with
If the text contains any of the unknown words,
the morphological analysis unit divides the text into word strings, and then converts the unknown words included in the word strings into predetermined tokens;
the word embedding generation unit generates embedding expressions of the predetermined tokens,
The document processing device according to claim 6, characterized in that the word embedded representation correction unit combines the embedded representation of the unknown word stored in the extended word vector storage unit with the class vector of the type to which the unknown word belongs stored in the class vector storage unit, and sets the embedded representation after the combination as the embedded representation of the unknown word. A document control method using a document processing device that generates embedded representations of words included in an externally input text, comprising:
The document processing device stores a class vector, which is a reference vector for each type of word,
a morphological analysis step in which the document processing device divides the text into word strings by morphological analysis and assigns information indicating a type to each word string;
a word embedding generation step of converting the word string divided in the morphological analysis step into an embedding representation of each word included in the word string;
a word embedding correction step of correcting the embeddings generated in the word embedding generation step by the document processing device;
Equipped with
When the text includes a low-frequency word whose occurrence frequency in pre-learning is lower than a predetermined level,
In the morphological analysis step, after dividing the text into word strings by morphological analysis, the low frequency words included in the word strings are further divided into subwords of partial strings;
In the word embedding step, the low-frequency word is converted into a plurality of embedding expressions each including an embedding expression for each of the subwords included in the low-frequency word;
a word variance correction step of generating a word variance representation for each low-frequency word, the word variance representation for each low-frequency word, and a position vector for fixing the order of the subwords in the low-frequency word are combined with the class vector of the type to which the low-frequency word belongs, and the combined variance representation is replaced as the variance representation of the low-frequency word. A document control program to be executed by a computer constituting a document processing device that generates embedded representations of words included in an externally input text, the document control program comprising:
The document processing device stores a class vector, which is a reference vector for each type of word,
a morphological analysis process for dividing the text into word strings by morphological analysis and adding information indicating the type to which each word string belongs;
a word embedding generation process for converting the word string divided by the morphological analysis process into an embedding representation for each word included in the word string;
a word embedding correction process for correcting the embeddings generated in the word embedding generation process;
causing the computer to execute the
When the text includes a low-frequency word whose occurrence frequency in the pre-learning is lower than a predetermined level,
In the morphological analysis process, after dividing the text into word strings by morphological analysis, the low frequency words included in the word strings are further divided into subwords of partial strings;
In the word embedded representation generation process, the low-frequency word is converted into a plurality of embedded representations including embedded representations for each of the subwords included in the low-frequency word;
a position vector that fixes the order of the subwords in the low-frequency word is synthesized with the multiple embedded representations of the low-frequency word generated in the word embedded representation generation process, and then the position vector is synthesized with the class vector of the type to which the low-frequency word belongs, and the embedded representation after the synthesis is replaced as the embedded representation of the low-frequency word.

Images