JP5513985B2 - CHARACTER VECTOR GENERATION DEVICE, CHARACTER VECTOR GENERATION METHOD, PROGRAM, AND COMPUTER-READABLE RECORDING MEDIUM CONTAINING THE PROGRAM - Google Patents

Description

  The present invention relates to a character string vector generating device, a character string vector generating method, a program, and a computer-readable recording medium storing the program.

  A vector as a semantic expression of a word can quantify the semantic similarity between words, and thus is applied to language processing such as search and contributes to improvement in accuracy.

  There is a method described in Non-Patent Document 1 as a method of generating a vector that is a semantic representation of a word. This method takes text as input, takes different sets of words present in the text, and generates a co-occurrence matrix between the sets. The co-occurrence matrix corresponding to the word pair is a value obtained by adding the frequency at which arbitrary words co-occur in a predetermined range of text (typically one sentence) over all the predetermined ranges in the text. Store in the element inside. Each row vector in the co-occurrence matrix represents what word the word corresponding to the row co-occurs with (that is, what word has as a neighbor). In general, words that have close meanings have words that are common neighbors, and the corresponding row vectors are close. Therefore, the concept of a word is represented by a corresponding row vector. This row vector is called a co-occurrence vector.

  The generated co-occurrence matrix is decomposed into singular values, converted into a co-occurrence matrix with a reduced number of columns, and the dimensionality of the co-occurrence vector is reduced, thereby improving the accuracy of similarity between vectors. . A vector obtained by reducing the dimensionality of the co-occurrence vector is called a concept vector.

  The co-occurrence vector and the concept vector are collectively called a word vector. Semantic similarity between words is quantified by the similarity between corresponding word vectors.

  The document vector as the semantic expression of the document is the sum or centroid of the word vectors of the words in the document. Semantic similarity between documents is quantified by the similarity between corresponding document vectors.

H. Schutze, Dimensions of Meaning, Proc. Of Supercomputing '92, pp.786-796, 1992

  In the method of Non-Patent Document 1, morphological analysis is performed to acquire words in the input text. In morphological analysis, text is divided into words in a word dictionary. When there is a word that is not in the word dictionary in the text, the word may not be divided properly. As a result, the extracted word is incorrect and the co-occurrence frequency between words is not counted correctly. There is a problem with the quality of the vector. Also, after generating a word vector group, when generating an input document vector, the input document may not be properly segmented. As a result, the extracted word is not correct, resulting in improved document vector quality. There's a problem.

  Moreover, even if a word vector group is generated from a certain text, there is a problem that a word vector cannot be obtained for a word that does not exist in the text. For this reason, when generating a vector of an input document, if there is a word without a word vector in the input document, the meaning of the word is not reflected in the document vector, and there is a problem in the quality of the document vector.

  As described above, there are problems in the quality of word vectors and document vectors due to word division errors, and there are problems in the quality of document vectors due to the amount of word vectors. In addition to this, there is a problem in accuracy over the entire language processing using word vectors.

  The present invention has been made to solve the above-described problem, and provides a technique for generating a character string vector that can generate an appropriate vector as a semantic expression for an arbitrary character string. With the goal.

In order to solve the above-described problem, the invention according to claim 1 is a character string vector generation device that generates a character string vector as a row vector of a matrix, and each sentence of the sentence in the text is input from the input text. Character string set extraction means for extracting different sets A and B of character strings that are continuous finite number of character strings from the character position , and each row corresponding to the character string in the set A, and each column A matrix corresponding to the character string in the set B is generated, and the character string C in the set A existing in a predetermined range that is a string of a finite number of continuous words in the text, and the set A matrix in which the frequency of co-occurrence of the character string D in B is added over all the predetermined ranges in the text and the elements in the matrix corresponding to the character string C and the character string D are used. First matrix generation means for generating and outputting Configured as a string vector generation device comprising.

Further, the invention according to claim 2 is a character string vector generation device that generates a character string vector as a row vector of a matrix, and from an input text, a finite number of continuous characters from each character position of the sentence in the text. Character string set extraction means for extracting different sets A and B of character strings, and L (L is an integer of 1 or more) components for each character string in the set B components from the group of the set of components numbers extracted from the number, from an already group set excluding the group of imparting ingredients number set, randomly select one component numbers set to impart the component number set in the character string Numbering means, each row corresponding to a character string in the set A, a matrix in which each column corresponds to a component number in the component number set is generated, and a continuous finite number of words in the text Previous in a given range that is a column A value obtained by adding the frequency of co-occurrence of the character string C in the set A and the component number E associated with the character string in the set B over all the predetermined ranges in the text, A character string vector generation device is provided, comprising: a second matrix generation unit that generates and outputs a matrix having elements in the matrix corresponding to the column C and the component number E.

  The invention according to claim 3 is the character string vector generation device according to claim 1 or 2, wherein the matrix is reduced in number by performing singular value decomposition on the output matrix. A singular value decomposition means for converting into a matrix is further provided as a character string vector generation device.

  According to a fourth aspect of the present invention, in the character string vector generating device according to any one of the first to third aspects, the output is performed for each input character string in the document. A document vector generation unit that searches a set of character strings corresponding to the rows of the matrix, obtains a row vector corresponding to the searched character string, and uses a sum or centroid of the row vectors as a vector of the document; This is configured as a character string vector generation device characterized by that.

  Further, as described in claims 5 to 8, the present invention can also be configured as a character string vector generation method executed by the character string vector generation device according to claims 1 to 4. According to a ninth aspect of the present invention, there is provided a computer program configured to cause a computer to function as each unit in the character string vector generating device according to any one of the first to fourth aspects. May be. Furthermore, as described in claim 10, the present invention can also be configured as a computer-readable recording medium storing the program described in claim 9.

  In the technology according to the present invention, for each character string in the set A, a row vector of a matrix generated by using any of the inventions according to claims 1 to 3 corresponding to the character string, It is a string vector of columns. A set of pairs of character strings in the set A and character string vectors corresponding to the character strings is referred to as a character string vector database.

  In the inventions according to claims 1 to 3, there is no problem caused by an error in word division because there is no morphological analysis of text if both sets A and B are a set of general character strings. The co-occurrence frequency of column pairs can be appropriately counted, and the quality of the generated character string vector is improved. In addition, when generating a vector of an input document using the invention according to claim 4, there is no problem caused by an error in word division because the document is not subjected to morphological analysis. The column vector can be appropriately acquired, and an effect is achieved that a high-quality document vector can be generated.

  In the inventions according to claims 1 to 3, a character string vector database may be generated by taking a set of character strings consisting of up to several characters as the set A. When using this character string vector database, even for a character string that does not exist in the set A, the character string can be decomposed into a certain set of character strings in the set A. The character string vector of the character string can be generated by performing the process of claim 4 using the input document in the invention.

  In the inventions according to claims 1 to 3, as the set A, a character string vector database may be generated by taking a union of a set of words and a set of character strings composed of up to several characters. Good. When using this character string vector database, the corresponding character string vector is used for words existing in the set A, and for words that do not exist, the word is assigned to the character string in the set A. Since it can be decomposed into a certain set, the character string vector of the word can be generated by performing the process of claim 4 using the word as an input document in the invention according to claim 4. Thus, since a character string vector of a word that does not exist in the character string vector database can also be generated, the character string vector of the word of the document can be appropriately acquired when generating a vector of the input document, and a high-quality document vector Can be generated.

  The invention according to claims 1 to 3 takes a set of words as set B, or takes a union of a set of words and a set of character strings consisting of up to several characters. Variations are also included.

  In the invention according to claim 2, the number of dimensions of the character string vector can be made constant (L) while holding the co-occurrence information of an arbitrary character string pair. As a result, it is possible to generate a character string vector with a low calculation amount of processing that uses the co-occurrence information of a pair of arbitrary character strings while using the information.

  The effects described above are the same effects in the inventions according to claims of other categories corresponding to the above claims.

3 is a functional configuration diagram of a character string vector generation device 100 in Embodiment 1. FIG. It is a figure which shows the example of input text (15). It is a figure which shows the example of a 2-gram arrangement | sequence. 5 is a flowchart of processing executed by a first matrix generation unit 12 or a second matrix generation unit 23. It is a figure which shows the example of the matrix in Example 1-1. It is a figure which shows the example of the arrangement | sequence a in Example 1-1 and Example 2-1. It is a figure which shows the example of the arrangement | sequence b in Example 1-1. It is a figure which shows the example of arrangement | sequence b 'in Example 1-1. It is a figure which shows the example of the matrix in Example 1-1. It is a figure which shows an example of the content of a word dictionary. It is a figure which shows an example of the content of the unnecessary word table. It is a figure which shows an example of the content of the unnecessary part of speech table. FIG. 3 is a diagram showing a result of morphological analysis of the text of FIG. It is a figure which shows the example of the character string set A in Example 1-2 and Example 2-2. It is a figure which shows the example of the character string set B in Example 1-2 and Example 2-2. It is a figure which shows the example of the matrix in Example 1-2. It is a figure which shows the example of the arrangement | sequence a in Example 1-2 and Example 2-2. It is a figure which shows the example of the arrangement | sequence b in Example 1-2. It is a figure which shows the example of arrangement | sequence b 'in Example 1-2. It is a figure which shows the example of the matrix in Example 1-2. It is a function block diagram of the character string vector production | generation apparatus 200 in Example 2. FIG. It is a figure which shows the example of the character string set B with a component number in Example 2-1. It is a figure which shows the example of the matrix in Example 2-1. It is a figure which shows the example of the arrangement | sequence b in Example 2-1. It is a figure which shows the example of arrangement | sequence b 'in Example 2-1. It is a figure which shows the example of the matrix in Example 2-1. It is a figure which shows the example of the arrangement | sequence a in Example 2-1. It is a figure which shows the example of the arrangement | sequence a in Example 2-1. It is a figure which shows the example of the arrangement | sequence g in Example 2-1. It is a figure which shows the example of the arrangement | sequence b in Example 2-1. It is a figure which shows the example of arrangement | sequence b 'in Example 2-1. It is a figure which shows the example of the matrix in Example 2-1. It is a figure which shows the example of the character string set B with a component number in Example 2-2. It is a figure which shows the example of the matrix in Example 2-2. It is a figure which shows the example of the arrangement | sequence b in Example 2-2. It is a figure which shows the example of arrangement | sequence b 'in Example 2-2. It is a figure which shows the example of the matrix in Example 2-2. It is a figure which shows the example of the arrangement | sequence a in Example 2-2. It is a figure which shows the example of the arrangement | sequence a in Example 2-2. It is a figure which shows the example of the arrangement | sequence g in Example 2-2. It is a figure which shows the example of the arrangement | sequence b in Example 2-2. It is a figure which shows the example of arrangement | sequence b 'in Example 2-2. It is a figure which shows the example of the matrix in Example 2-2. It is a function block diagram of the character string vector production | generation apparatus 301 in Example 3 (example 1). It is a function block diagram of the character string vector production | generation apparatus 302 in Example 3 (example 2). It is an example of the character string vector database 2 (19) in Example 3-1. It is an example of the character string vector database 2 (19) in Example 3-2. It is a function block diagram of the character string vector generation apparatus 401 in Example 4 (Example 1). It is a function block diagram of the character string vector production | generation apparatus 402 in Example 4 (example 2). It is a figure which shows the example of the input document in Example 4-1. It is a figure which shows the example of the 2-gram row | line in Example 4-1. It is a figure which shows the example of the input document in Example 4-2. It is a figure which shows the example of the morphological analysis result in Example 4-2. It is an example of the word string extracted from a morphological analysis result in Example 4-2. It is a figure which shows the example of the 1-gram row | line in Example 4-2.

  Hereinafter, with reference to the drawings, Examples 1 to 4 which are forms for carrying out the present invention will be described.

[Example 1]
FIG. 1 shows a configuration example of a character string vector generation device 100 according to the first embodiment of the present invention. As shown in FIG. 1, a character string vector generation device 100 according to the first embodiment includes a character string set extraction unit 11, a first matrix generation unit 12, a text storage unit 101, a character string set storage unit 102, and a character string vector storage. A unit 103 is provided.

  The character string set extraction unit 11 reads the text (15) from the text storage unit 101 and inputs it, and the characters that are different sets of character strings that are strings of a finite number of consecutive characters in the input text (15). Column set A (16) and character string set B (17) are extracted and stored in the character string set storage unit. Note that “different sets of character strings” are sets of different character strings. Hereinafter, similarly, “different sets of X” mean sets of X different from each other.

  Assuming that the above sets A and B are the same, one character string set may be extracted and stored, and the character string set may be treated as both set A and set B.

  For example, the character string set extraction unit 11 takes different sets of strings of n consecutive characters (called n-gram) in the text for an integer constant n of 1 or more, and uses this as a character string set. And Among the n-gram sets, for example, a character string set may be limited to a high-frequency n-gram set. Further, for a plurality of n, a union of n-gram sets for each n may be taken and used as a character string set.

  Further, the character string set extraction unit 11 may perform morphological analysis on the input text and extract different sets of words in the text as character string sets. In this case, the words employed in the character string set may be limited to content words such as nouns, verbs, and adjectives. In addition, among the word sets, for example, a character string set may be limited to a high-frequency word set.

  Moreover, it is good also as a character string set by taking the union of several n-gram sets and a word set. At this time, even if the n-gram set and the word set have the same elements as the character strings, they are regarded as different elements, and the n-gram set and the word set have no common elements. You may catch it.

  The first matrix generation unit 12 reads the text (15), the character string set A (16), and the character string set B (17) from the text storage unit 101 and the character string set storage unit 102 as inputs, and each row includes the set A. A matrix corresponding to each character string in the text and corresponding to each character string in the set B, and existing in a predetermined range that is a sequence of a finite number of consecutive words in the text. The matrix corresponding to the character string C and the character string D, the value obtained by adding the frequency of occurrence of the character string C in the character set A and the character string D in the character set B over all the predetermined ranges in the text. A character string vector database 1 (18) which is a set of pairs of character strings in the set A and row vectors of the matrix corresponding to the character strings; Store in 103.

  Note that the first matrix generation unit 12 instead of outputting and storing the character string vector database 1 (18) to the character string vector storage unit 103, for example, a character string vector generation device via a communication network It is good also as outputting to the apparatus outside 100.

  Hereinafter, as detailed examples of processing in Example 1, Example 1-1 in which co-occurrence between n-grams is taken, and Example 1-2 in which an n-gram set is added to a word set in a co-occurrence line between words explain.

<Example 1-1>
The character string set extraction unit 11 takes different sets of n-grams from the input text (15) for an integer constant n of 1 or more as follows.
(1) Clear the n-gram array. The array is a predetermined area in a memory or the like in the character string vector generation device 100. The same applies to the other embodiments.
(2) Perform the following process for each sentence in the text.

  (2.1) Numbering each character so that the first character of the sentence is the first character, the position immediately before the first character is the 0th character, and thereafter the front of the sentence decreases and the rear of the sentence increases. To go. When the number of characters in the sentence is s, each position from the (2-n) th character to the sth character (however, if n ≧ s + 3, the (s-n + 3) th character to the 0th character N) is taken, and the n-gram is inserted into the n-gram array. At this time, if the n-gram is not already in the n-gram array, it is inserted (if the frequency is also counted, the frequency of the n-gram in the n-gram array is set to 1), and nothing is done. (When the frequency is also counted, the frequency of the n-gram in the n-gram array is further increased by 1).

  The n-gram arrays obtained in this way are set as character string sets A and B. As described above, the character string set A or B may be a string limited to a high-frequency n-gram set in the n-gram array.

  Hereinafter, a specific example when n = 2 will be described.

  FIG. 2 is an example of the input text (15). “Buy clock and bread and ate bread” in FIG. 2 is one sentence in the text.

  FIG. 3 shows a 2-gram array obtained by the processing of the character string set extraction unit 11 when n = 2. In FIG. 3, a 2-gram obtained from one sentence in FIG. 2 is displayed. Here, “0 o'clock” in FIG. 3 indicates a 2-gram from the 0th character in the sentence, and “ta ○” indicates that it is a 2-gram from the s character in the sentence. Show. Hereinafter, the 2-gram array in FIG.

  FIG. 4 shows a flowchart of processing executed by the first matrix generation unit 12. Hereinafter, the process of each step in Example 1-1 will be described.

  Step 31) Generate a matrix such that each row corresponds to an n-gram in set A and each column corresponds to an n-gram in set B as shown in FIG. To store. Each element in the matrix represents the co-occurrence frequency of the corresponding n-gram in set A and n-gram in set B. In this step, 0 is set as an initial value.

  Step 32) As the control 1, a predetermined range in the text to be processed is calculated for calculating the frequency with which the n-gram in the set A and the n-gram in the set B co-occur. The predetermined range is a string of a finite number of continuous words such as one sentence, one paragraph, and one document. This determination can be made, for example, by storing information indicating a predetermined range in the storage unit of the character string vector generation device 100 in advance and reading the information from the storage unit.

  When the predetermined range is one sentence, the first sentence in the text is first processed. When the processing from step 33 regarding the sentence to be processed is completed, the next sentence is set as the processing target. If the processing after step 33 relating to the last sentence is completed, there is no sentence to be processed, and thus this flowchart is ended. The same applies to the case where the predetermined range is other.

  Step 33) The array a which is an array of n-gram and its frequency in the set A and the array b which is an array of n-gram and its frequency in the set B are cleared.

  Step 34) As control 2, an n-gram sequence in the range of the text to be processed is defined as follows.

  In each sentence in the range, the first character of the sentence is the first character, the position immediately before the first character is the 0th character, and thereafter each character is such that the front of the sentence decreases and the rear of the sentence increases. Number. When the number of characters in the sentence is s, each position from the (2-n) th character to the sth character (however, if n ≧ s + 3, the (s-n + 3) th character to the 0th character N-grams from (excluded) are taken in order to form an n-gram sequence of the sentence. The n-gram sequence of each sentence in the range is connected to the sentence in the order of the sentences, and is defined as the n-gram sequence in the range.

  After determining as described above, an n-gram to be processed in the range is determined as follows.

  First, the first n-gram of the n-gram sequence in the range is set as a processing target. When the processing of step 35 regarding the n-gram to be processed is completed, the next n-gram is set as the processing target. If the process of step 35 regarding the last n-gram is completed, there is no n-gram to be processed, and the process proceeds to step 36.

  Step 35) Search the set A with the n-gram to be processed. If hit, do the following: The array a is searched with the n-gram. If there is no hit, the n-gram and its frequency 1 pair are added to the array a. If there is a hit, the frequency of the n-gram in sequence a is increased by 1.

  Search set B by n-gram to be processed. If hit, do the following: The array b is searched with the n-gram. If there is no hit, the n-gram and its frequency 1 pair are added to the sequence b. If there is a hit, the frequency of the n-gram in sequence b is increased by 1.

  With the above processing, the array a in FIG. 6 and the array b in FIG. 7 are obtained from the range of the text shown in FIG. 2 when there is no n-gram to be processed in step 34.

  Step 36) The n-gram to be processed in the array a is determined as follows.

  First, the first n-gram in the array a is a processing target. When the processing in step 37 regarding the n-gram to be processed is completed, the next n-gram is set as the processing target. If the process of step 37 regarding the last n-gram is completed, there is no n-gram to be processed, and the process proceeds to step 32.

  Step 37) Add the frequency of each n-gramV in array b to the frequency in array a of n-gramU to be processed to the component corresponding to n-gramV in the row vector corresponding to n-gramU To do. Here, as a method of addition, self-co-occurrence that counts the co-occurrence with the n-gram itself for the processing target n-gram in the processing target range (even if there are multiple things of the same n-gram) There are two types: a method and a self-non-cooccurrence method that does not count.

1) Self-co-occurrence method
A value obtained by multiplying the frequency of each n-gramV in array b by the frequency in array a of n-gramU to be processed is added to the component corresponding to n-gramV in the row vector corresponding to n-gramU.

2) Self-non-co-occurrence method
If array b ′ is generated as a copy of array b and n-gram U to be processed is in array b ′, the frequency of n-gram U in array b ′ is decreased by 1.
A value obtained by multiplying the frequency of each n-gram V in the array b ′ by the frequency in the array a of the n-gram U is added to the component corresponding to the n-gram V of the row vector corresponding to the n-gram U.
• Clear array b '.

  As a specific example of the self-non-co-occurrence method, consider a case where, in step 32, the matrix is FIG. 5, and one range of text shown in FIG. FIG. 6 is a diagram obtained by subtracting 1 from the frequency of the n-gram “pan” in the array b in FIG. 7 when the n-gram “pan” in the array a in FIG. Each frequency of array b ′ of 8 is multiplied by frequency 2 in array a of n-gram “pan” and added to the row vector of n-gram “pan”. When there is no n-gram to be processed in step 36, the matrix of FIG. 9 is obtained.

  In the description of the flowchart of FIG. 4 in Example 1-1 above, when there are a plurality of identical n-grams in the range of text to be processed, each is processed as a separate item, but the range of text to be processed Even if there are multiple identical n-grams, there can be a processing method that treats them as one. When this processing method is used, step 35 is as follows.

  Step 35) Search the set A with the n-gram to be processed. If hit, do the following: The array a is searched with the n-gram. If there is no hit, the n-gram and its frequency 1 pair are added to the array a.

  Search set B by n-gram to be processed. If hit, do the following: The array b is searched with the n-gram. If there is no hit, the n-gram and its frequency 1 pair are added to the sequence b.

  From the range of the text shown in FIG. 2, when there are no n-grams to be processed in step 34, the array a has the frequency of each n-gram in FIG. You can get one with the frequency of each n-gram of 7.

<Example 1-2>
Next, Example 1-2 will be described. First, the character string set extraction unit 11 performs morphological analysis on the input text (15) with reference to the word dictionary. The word dictionary may be stored in the storage unit of the character string vector generation device 100, or may be stored in an external device and accessed via a communication network.

  FIG. 10 is a diagram showing an example of the contents of the word dictionary. In the example of FIG. 10, in the word dictionary, one record is information related to one word, and one record is composed of two items separated by a comma. The first item is a word notation, and the second item is the part of speech information of the word. You may make it register a closing form as a new item with respect to a use word.

  The character string set extraction unit 11 determines whether the word obtained as a result of the morphological analysis is a word used in subsequent processing (referred to as a necessary word) or a word not used (referred to as an unnecessary word). In general, content words such as nouns, verbs, and adjectives are necessary words.

  Whether or not a word is a necessary word is determined by, for example, storing an unnecessary word table describing unnecessary words and an unnecessary part of speech table describing parts of speech of unnecessary words in the storage unit of the character string vector generation device 100. Judgment by reference.

  FIG. 11 shows an example of the contents of the unnecessary word table. The ending form of each word that is considered unnecessary (the word notation is the ending form for words that do not have a ending form) is described as one record. FIG. 12 is an example of the contents of the unnecessary part-of-speech table. Each part-of-speech information that is unnecessary is described as one record.

  The result of morphological analysis of the text in FIG. 2 is shown in FIG. As shown in FIG. 13, each word is separated by “/”. Each word is composed of a word notation, an end form, part-of-speech information, and a necessary word flag, and each word is separated by “,”. For words for which no final form is registered, the final form is derived from the word notation and part-of-speech information after the morphological analysis, and for words that do not have a final form, the word notation is used as the final form. If the final form matches a record with an unnecessary word table, or if the part of speech information matches a record with an unnecessary part of speech table, the required word flag is set to 0, and if not, the required word flag is set to 1.

Then, the character string set extraction unit 11 takes different sets of n-grams for different sets of words and an integer constant n of 1 or more from the morphological analysis result. This process is performed as follows.
(1) Clear the word array and n-gram array.
(2) The following processing is performed for each sentence in the morphological analysis result.

(2.1)-Insert the final form in the morpheme analysis result with the required word flag set to 1 into the word array. At this time, if the final form is not already in the word sequence, it is inserted (if the frequency is counted, the frequency of the final form in the word sequence is set to 1), and nothing is done if the frequency is counted. In addition, the frequency of the terminal form in the word sequence is increased by 1).
・ Generate a non-divided sentence by concatenating word expressions in the morphological analysis result.
-Number each character so that the first character of the non-split sentence is the first character, the position immediately before the first character is the 0th character, and thereafter the front of the sentence decreases and the rear of the sentence increases. Go. When the number of characters in the sentence is s, each position from the (2-n) th character to the sth character (however, if n ≧ s + 3, the (s-n + 3) th character to the 0th character N) is taken, and the n-gram is inserted into the n-gram array. At this time, if the n-gram is not already in the n-gram array, it is inserted (if the frequency is also counted, the frequency of the n-gram in the n-gram array is set to 1), and nothing is done. (When the frequency is also counted, the frequency of the n-gram in the n-gram array is further increased by 1).
-Clear the unsplit sentence.

  The word array and n-gram array obtained in this way may be limited to high-frequency words and high-frequency n-grams, respectively.

  The character string set extraction unit 11 regards the word array and the n-gram array obtained as described above as a set having no common elements, and combines them into a character string set A. In the character string set A, even if a word and an n-gram have the same character string, they are regarded as different things. Also, let the word array be a character string set B.

  Hereinafter, a specific example when n = 1 is described.

  FIG. 14 shows a character string set A and FIG. 15 shows a character string set B obtained by performing the processing of the character string set extraction unit 11 with n = 1 from the morphological analysis result of FIG. FIG. 14 displays the word array and the 1-gram array so as to be separated by ruled lines, and displays the word and 1-gram obtained from one sentence in FIG. FIG. 15 displays words obtained from one sentence in FIG.

  FIG. 4 shows a flowchart of processing executed by the first matrix generation unit 12. Hereinafter, processing of each step in Example 1-2 will be described.

  Step 31) Generate a matrix such that each row corresponds to a character string (word or n-gram) in set A and each column corresponds to a word in set B, as shown in FIG. It stores in storage means such as. Each element in the matrix represents the co-occurrence frequency of the corresponding character string in the set A and the word in the set B. In this step, 0 is set as an initial value.

  Step 32) As control 1, a predetermined range in the result of morpheme analysis to be subjected to processing for calculating the frequency of co-occurring character strings in set A and words in set B is determined. The predetermined range is a string of a finite number of continuous words such as one sentence, one paragraph, and one document. This determination can be made, for example, by storing information indicating a predetermined range in the storage unit of the character string vector generation device 100 in advance and reading the information from the storage unit.

  When the predetermined range is one sentence, the first sentence in the morphological analysis result is first processed. When the processing from step 33 regarding the sentence to be processed is completed, the next sentence is set as the processing target. If the processing after step 33 relating to the last sentence is completed, there is no sentence to be processed, and thus this flowchart is ended. The same applies to the case where the predetermined range is other.

  Step 33) The array a which is an array of pairs of character strings and their frequencies in the set A, the array b which is an array of pairs of words and their frequencies in the set B, and the undivided text are cleared.

  Non-divided text in which word notations in the range of the morpheme analysis result to be processed are connected is generated and stored in the storage means.

  Step 34) As the control 2, an n-gram sequence in the range of the morphological analysis result to be processed is determined as follows.

  In each sentence in the non-split text in the range, the first character of the sentence is the first character, the position immediately before the first character is the 0th character, and thereafter the front of the sentence decreases and the rear of the sentence increases Then, number each letter. When the number of characters in the sentence is s, each position from the (2-n) th character to the sth character (however, if n ≧ s + 3, the (s-n + 3) th character to the 0th character N-grams from (excluded) are taken in order to form an n-gram sequence of the sentence. The n-gram sequence of each sentence in the non-divided text in the range is connected to the sentence in the order of the sentences, and is defined as the n-gram sequence in the range.

  After determining as described above, an n-gram to be processed in the range is determined as follows.

  First, the first n-gram of the n-gram sequence in the range is set as a processing target. When the processing of step 35 regarding the n-gram to be processed is completed, the next n-gram is set as the processing target. If the process of step 35 regarding the last n-gram is completed, there is no n-gram to be processed, and the process proceeds to step 36.

  Step 35) In the array a, a word and an n-gram are treated as different characters even if they match as a character string.

  Search set A by n-gram to be processed. If hit, do the following: The array a is searched with the n-gram. If there is no hit, the n-gram and its frequency 1 pair are added to the array a. If there is a hit, the frequency of the n-gram in sequence a is increased by 1.

  If the word notation character in the morphological analysis result corresponding to the first character of the n-gram to be processed is the first character of the word notation and the required word flag of the word notation is 1, the word notation The following processing is performed on the final form of processing.

  Search set A with the final form to be processed. If hit, do the following: The sequence a is searched with the end form. If there is no hit, add the terminal form and its frequency 1 pair to sequence a. If there is a hit, the frequency of the terminal form in sequence a is increased by one.

  Search set B with the final form to be processed. If hit, do the following: The sequence b is searched with the end form. If there is no hit, the pair of the terminal form and its frequency 1 is added to the sequence b. If there is a hit, the frequency of the terminal form in sequence b is increased by one.

  With the above processing, the array a in FIG. 17 and the array b in FIG. 18 are obtained from the range of the morphological analysis results shown in FIG. 13 when there is no n-gram to be processed in step.

  In steps 34 and 35, the frequency of the n-gram and the word in the processing target range is obtained simultaneously. In steps 34 and 35, only the frequency of the n-gram is calculated based on the non-divided text in the range. In addition to these steps, a step of obtaining a word frequency based on the morphological analysis result in the range may be provided.

  Step 36) The character string to be processed in the array a is determined as follows.

  First, the first character string in the array a is a processing target. When the processing in step 37 regarding the character string to be processed is completed, the next character string is set as the processing target. When the process of step 37 regarding the last character string is completed, there is no character string to be processed, and the process proceeds to step 32.

  Step 37) The n-gram in the array a and the word in the array b are treated as different objects even if they match as a character string.

  A value obtained by multiplying the frequency of each word V in the array b by the frequency in the array a of the character string U to be processed is added to the component corresponding to the word V in the row vector corresponding to the character string U. Here, as a method of addition, a self-co-occurrence method for counting the co-occurrence with the character string itself for the processing target character string in the processing target range (even if there are a plurality of the same character string), There are self-non-co-occurrence methods that do not count, and each has the following processing contents.

1) Self-co-occurrence method
A value obtained by multiplying the frequency of each word V in the array b by the frequency in the array a of the character string U to be processed is added to the component corresponding to the word V in the row vector corresponding to the character string U.

2) Self-non-co-occurrence method
If array b ′ is generated as a copy of array b and character string U to be processed is in array b ′, the frequency of array U of character string U is reduced by 1.
A value obtained by multiplying the frequency of each word V in the array b ′ by the frequency in the array a of the character string U is added to the component corresponding to the word V in the row vector corresponding to the character string U.
• Clear array b '.

  As a specific example of the self-non-co-occurrence method, consider a case where, in step 32, the matrix is FIG. 16, and one range of the morphological analysis results shown in FIG. When the word “pan” in the array a in FIG. 17 is determined as a processing target in step 36, the array b in FIG. 19 obtained by subtracting 1 from the frequency of the word “pan” in the array b in FIG. 18 is obtained in step 37. Each frequency of 'is multiplied by the frequency 2 in the array a of the word “pan” and added to the row vector of the word “pan”. When there is no character string to be processed in step 36, the matrix of FIG. 20 is obtained.

  In the description of the flowchart of FIG. 4 in Example 1-2 above, when there are a plurality of the same words in the range of the morphological analysis results to be processed, or when there are a plurality of the same n-grams, A processing method that treats them as one, even if there are multiple identical words or n-grams that are the same in the scope of the morphological analysis results to be processed, even though they are processed separately Is also possible. When this processing method is used, step 35 is as follows.

  Step 35) In the array a, a word and an n-gram are treated as different characters even if they match as a character string.

  Search set A by n-gram to be processed. If hit, do the following: The array a is searched with the n-gram. If there is no hit, the n-gram and its frequency 1 pair are added to the array a.

  If the word notation character in the morphological analysis result corresponding to the first character of the n-gram to be processed is the first character of the word notation and the required word flag of the word notation is 1, the word notation The following processing is performed on the final form of processing.

  Search set A with the final form to be processed. If hit, do the following: The sequence a is searched with the end form. If there is no hit, add the terminal form and its frequency 1 pair to sequence a.

  Search set B with the final form to be processed. If hit, do the following: The sequence b is searched with the end form. If there is no hit, the pair of the terminal form and its frequency 1 is added to the sequence b.

  From the range of the morphological analysis results shown in FIG. 13, when the n-gram to be processed in step 34 disappears, the array a has the frequency of each character string in FIG. A word with the frequency of each word in FIG. 7 set to 1 is obtained.

[Example 2]
FIG. 21 shows a configuration example of the character string vector generation device 200 according to the second embodiment of the present invention. As shown in FIG. 21, the character string vector generation device 200 according to the second embodiment includes a character string set extraction unit 11, a component number assigning unit 22, a second matrix generation unit 23, a text storage unit 101, and a character string set storage unit 102. , And a character string vector storage unit 103.

  The character string set extraction unit 11 is the same as the character string set extraction unit 11 in the first embodiment.

  The component number assigning unit 22 reads out the character string set B (17) from the character string set storage unit 102 and uses it as an input. The component number-attached character string set B (24) associated with the component number is output and stored in the character string set storage unit 102.

  The second matrix generation unit 23 reads the text (15), the character string set A (16), and the component numbered character string set B (24) from the text storage unit 101 and the character string set storage unit 102, and inputs each row. Generates a matrix in which each corresponds to a character string in set A and each column corresponds to a component number in the component number set, and a predetermined range that is a sequence of a finite number of consecutive words in the text The value obtained by adding the frequency of the occurrence of the character string C in the set A and the component number E associated with the character string in the set B over all the predetermined ranges in the text Stored in the element in the matrix corresponding to the column C and the component number E, the character string vector database 1 (a set of pairs of the character string in the set A and the row vector of the matrix corresponding to the character string 18) is output and stored in the character string vector storage unit 103.

  The second matrix generation unit 23, instead of outputting and storing the character string vector database 1 (18) to the character string vector storage unit 103, for example, a character string vector generation device via a communication network It is good also as outputting to the apparatus outside 200.

  Hereinafter, as a detailed example of processing in Example 2, Example 2-1 in which co-occurrence between n-grams is taken, and Example 2-2 in which an n-gram set is added to a word set in a co-occurrence line between words explain.

<Example 2-1>
The character string set extraction unit 11 takes different sets of n-grams from the input text (15) for an integer constant n of 1 or more. The processing content is the same as the processing content of the character string set extraction unit 11 of Example 1-1.

  Hereinafter, a specific example when n = 2 will be described.

  The 2-gram array in FIG. 3 obtained by the processing of the character string set extraction unit 11 from the input text (15) in FIG.

  The component number assigning unit 22 associates each character string in the set B with several component numbers in a component number set (referred to as set F) composed of L component numbers.

  Assume that the set F is {1, 2,..., L} in this embodiment. The number of component numbers associated with one character string is determined as a constant K. The constant K is determined as follows so that different component numbers are associated with different character strings.

  Number of combinations of K component numbers extracted from L component numbers

は、以下の式で表される。

Is represented by the following equation.

そして、集合Bの要素数をMとしたとき、

And when the number of elements in set B is M,

となるようにKを定める。Kをこのように定めることにより、異なる文字列には異なる成分番号集合を対応付けることができる。

Determine K to be By defining K in this way, different component number sets can be associated with different character strings.

The processing of the component number assigning unit 22 is performed as follows.
(1) The following processing is performed for each character string of set B.

  (1.1) From the set of K component numbers extracted from L component numbers, a set of component numbers excluding the set of component number sets assigned so far, one component number set is randomly selected, The component number set is assigned to the character string.

  FIG. 22 is a diagram showing an example of the component number-attached character string set B obtained by the processing of the component number assigning unit 22 when the character string set B of FIG. 3 is input and K = 2. Each character string is associated with a component number set consisting of two elements that is unique to the character string. The component number set is a second item, and each component number is separated by a colon.

  In the following description of the processing of the second matrix generation unit 23 in Example 2-1, the set B means the character string set B with component numbers.

  FIG. 4 shows a flowchart of processing executed by the second matrix generation unit 23. Hereinafter, the process of each step in Example 2-1 will be described.

  Step 31) Generate a matrix in which each row corresponds to an n-gram in set A and each column corresponds to a component number in set F, as shown in FIG. Store. Each element in the matrix represents the co-occurrence frequency of the corresponding n-gram in the set A and the component number in the set F. In this step, 0 is set as an initial value.

  Step 32) As the control 1, a predetermined range in the text to be processed is calculated for calculating the frequency with which the n-gram in the set A and the component number in the set F co-occur. The predetermined range is a string of a finite number of continuous words such as one sentence, one paragraph, and one document. The determination here can be performed, for example, by storing information indicating a predetermined range in the storage unit of the character string vector generation device 200 in advance and reading the information from the storage unit.

  When the predetermined range is one sentence, the first sentence in the text is first processed. When the processing from step 33 regarding the sentence to be processed is completed, the next sentence is set as the processing target. If the processing after step 33 relating to the last sentence is completed, there is no sentence to be processed, and thus this flowchart is ended. The same applies to the case where the predetermined range is other.

  Step 33) The array a which is a pair of n-gram and its frequency in the set A and the array b which is a pair of the component number and its frequency in the set F are cleared.

  Step 34) As control 2, an n-gram sequence in the range of the text to be processed is defined as follows.

  In each sentence in the range, the first character of the sentence is the first character, the position immediately before the first character is the 0th character, and thereafter each character is such that the front of the sentence decreases and the rear of the sentence increases. Number. When the number of characters in the sentence is s, each position from the (2-n) th character to the sth character (however, if n ≧ s + 3, the (s-n + 3) th character to the 0th character N-grams from (excluded) are taken in order to form an n-gram sequence of the sentence. The n-gram sequence of each sentence in the range is connected to the sentence in the order of the sentences, and is defined as the n-gram sequence in the range.

  After determining as described above, an n-gram to be processed in the range is determined as follows.

  First, the first n-gram of the n-gram sequence in the range is set as a processing target. When the processing of step 35 regarding the n-gram to be processed is completed, the next n-gram is set as the processing target. If the process of step 35 regarding the last n-gram is completed, there is no n-gram to be processed, and the process proceeds to step 36.

  Step 35) Search the set A with the n-gram to be processed. If hit, do the following: The array a is searched with the n-gram. If there is no hit, the n-gram and its frequency 1 pair are added to the array a. If there is a hit, the frequency of the n-gram in sequence a is increased by 1.

  Search set B by n-gram to be processed. If hit, do the following: The following is performed for each component number associated with the n-gram. Search the sequence b by the component number. If there is no hit, the pair of the component number and its frequency 1 is added to the sequence b. If there is a hit, the frequency of the component number in sequence b is increased by 1.

  With the above processing, the array a in FIG. 6 and the array b in FIG. 24 are obtained from the range of the text shown in FIG. 2 when there is no n-gram to be processed in step.

  Step 36) The n-gram to be processed in the array a is determined as follows.

  First, the first n-gram in the array a is a processing target. When the processing in step 37 regarding the n-gram to be processed is completed, the next n-gram is set as the processing target. If the process of step 37 regarding the last n-gram is completed, there is no n-gram to be processed, and the process proceeds to step 32.

  Step 37) A value obtained by multiplying the frequency of each component number V in array b by the frequency in array a of n-gram U to be processed is added to the component corresponding to component number V in the row vector corresponding to n-gram U. To do. Here, as the method of addition, the co-occurrence with the component number of the n-gram itself is counted with respect to the processing target n-gram in the processing target range (even if there are multiple items of the same n-gram). There are a self-co-occurrence method and a self-non-co-occurrence method that does not count, and each has the following processing contents.

1) Self-co-occurrence method
A value obtained by multiplying the frequency of each component number V in the array b by the frequency in the array a of the n-gram U to be processed is added to the component corresponding to the component number V of the row vector corresponding to the n-gram U.

2) Self-non-co-occurrence method
Generate array b ′ as a copy of array b.
Search the set B with the n-gram U to be processed and obtain the corresponding component number set.
If the component number set is not empty, each component number of n-gramU is reduced by 1 from the frequency in the component number array b ′.
A value obtained by multiplying the frequency of each component number V in the array b ′ by the frequency in the array a of the n-gram U is added to the component corresponding to the component number V of the row vector corresponding to the n-gram U.
• Clear array b '.

  As a specific example of the self-non-co-occurrence method, consider a case where the matrix is FIG. 23 and one range of the text shown in FIG. When the n-gram “pan” in the array a in FIG. 6 is determined to be processed in step 36, the component in the array b in FIG. 24 is compared with the component numbers 16 and 38 of the n-gram “pan” in step 37. 25. The row vector of n-gram “pan” is obtained by multiplying each frequency of array b ′ in FIG. 25 obtained by subtracting 1 from the frequencies of numbers 16 and 38 by frequency 2 in array a of n-gram “pan”. Add to. When there is no n-gram to be processed in step 36, the matrix of FIG. 26 is obtained.

  In the description of the flowchart of FIG. 4 in Example 2-1, a processing method that does not count the frequency of n-grams in the range of text to be processed can be realized by performing steps 33, 35, and 37 as follows. . This processing method can be configured not to include a process of counting the frequency of n-grams and a process of counting the frequency of co-occurrence of n-grams in the range of text to be processed when generating a matrix.

  Step 33) The array a which is an n-gram array in the set A and the array b which is a pair of the component number and its frequency in the set F are cleared.

  Step 35) The set A is searched for n-grams to be processed, and if hit, the n-gram is added to the array a. In the array a, n-grams are stored in the order of addition, and are added even when the same n-gram is already in the array a.

  Search set B by n-gram to be processed. If hit, do the following: The following is performed for each component number associated with the n-gram. Search the sequence b by the component number. If there is no hit, the pair of the component number and its frequency 1 is added to the sequence b. If there is a hit, the frequency of the component number in sequence b is increased by 1.

  From the range of the text shown in FIG. 2, the array a in FIG. 27 and the array b in FIG. 24 are obtained when there is no n-gram to be processed in step.

  Step 37) The frequency of each component number V in the array b is added to the component corresponding to the component number V in the row vector corresponding to the n-gram U to be processed. Here, as the method of addition, the co-occurrence with the component number of the n-gram itself is counted with respect to the processing target n-gram in the processing target range (even if there are multiple items of the same n-gram). There are a self-co-occurrence method and a self-non-co-occurrence method that does not count, and each has the following processing contents.

1) Self-co-occurrence method
The frequency of each component number V in the array b is added to the component corresponding to the component number V in the row vector corresponding to the n-gram U to be processed.

2) Self-non-co-occurrence method
Generate array b ′ as a copy of array b.
Search the set B with the n-gram U to be processed and obtain the corresponding component number set.
If the component number set is not empty, each component number of n-gramU is reduced by 1 from the frequency in the component number array b ′.
The frequency of each component number V in the array b ′ is added to the component corresponding to the component number V in the row vector corresponding to n-gramU.
• Clear array b '.

  In the description of the flowchart of FIG. 4 in Example 2-1 above, when there are a plurality of identical n-grams in the range of the text to be processed, each is processed as a different item. Even if there are multiple n-grams in the range of text to be processed, there can be a processing method that handles them as one. In the case of this processing method, steps 33, 35 and 37 are as follows.

  Step 33) An array a which is an n-gram array in set A, an array b which is an array of component numbers and their frequencies in set F, and an array g which is an n-gram array in set B Is cleared.

  Step 35) Search the set A with the n-gram to be processed. If hit, do the following: The array a is searched with the n-gram. If there is no hit, the n-gram is added to the array a.

  Search set B by n-gram to be processed. If hit, do the following: The array g is searched with the n-gram. If not, do the following: The n-gram is added to the array g. The following is performed for each component number associated with the n-gram. Search the sequence b by the component number. If there is no hit, the pair of the component number and its frequency 1 is added to the sequence b. If there is a hit, the frequency of the component number in sequence b is increased by 1.

  From the range of the text shown in FIG. 2, the array a in FIG. 28, the array g in FIG. 29, and the array b in FIG. 30 are obtained when there is no n-gram to be processed in step.

  Step 37) The frequency of each component number V in the array b is added to the component corresponding to the component number V in the row vector corresponding to the n-gram U to be processed. Here, as a method of addition, there are a self-co-occurrence method for counting the co-occurrence with the component number of the n-gram itself for the processing target n-gram in the processing target range, and a self-non-co-occurrence method not counting, The processing contents are as follows.

1) Self-co-occurrence method
The frequency of each component number V in the array b is added to the component corresponding to the component number V in the row vector corresponding to the n-gram U to be processed.

2) Self-non-co-occurrence method
Generate array b ′ as a copy of array b.
Search the set B with the n-gram U to be processed and obtain the corresponding component number set.
If the component number set is not empty, each component number of n-gramU is reduced by 1 from the frequency in the component number array b ′.
The frequency of each component number V in the array b ′ is added to the component corresponding to the component number V in the row vector corresponding to n-gramU.
• Clear array b '.

  As an example of the self-non-co-occurrence method, consider a case where the matrix is FIG. 23 and one range of text shown in FIG. When the n-gram “pan” in the array a in FIG. 28 is determined to be processed in step 36, the component in the array b in FIG. 30 is compared with the component numbers 16 and 38 in the n-gram “pan” in step 37. Each frequency of array b ′ in FIG. 31 obtained by subtracting 1 from the frequencies of numbers 16 and 38 is added to the row vector of n-gram “pan”. When there is no n-gram to be processed in step 36, the matrix of FIG. 32 is obtained.

<Example 2-2>
Next, Example 2-2 will be described. First, the character string set extraction unit 11 performs morphological analysis on the input text (15) with reference to the word dictionary. The word dictionary may be stored in the storage unit of the character string vector generation device 200, or may be stored in an external device and accessed via a communication network.

  From the morphological analysis results, different sets of n-grams are taken for different sets of words and an integer constant n of 1 or more. The processing content is the same as the processing content of the character string set extraction unit 11 of the embodiment 1-2.

  Hereinafter, a specific example when n = 1 is described.

  The character string array of FIG. 14 obtained by the processing of the character string set extraction unit 11 from the morphological analysis result of FIG. 13 is referred to as a character string set A, and the word array of FIG.

  The component number assigning unit 22 associates each word in the set B with several component numbers in a component number set (referred to as set F) composed of L component numbers. The processing content is the same as the processing content of the component number assigning unit 22 of Example 2-1.

  FIG. 33 is a component number-attached character string set B obtained by processing of the component number assigning unit 22 when the character string set B of FIG. 15 is input and K = 2. Each character string is associated with a component number set consisting of two elements that is unique to the character string. The component number set is a second item, and each component number is separated by a colon.

  In the following description of the processing of the second matrix generation unit 23 in Example 2-2, the set B means the character string set B with component numbers.

  FIG. 4 shows a flowchart of processing executed by the second matrix generation unit 23. Hereinafter, the process of each step in Example 2-2 will be described.

  Step 31) Generate a matrix in which each row corresponds to a character string (word or n-gram) in set A and each column corresponds to a component number in set F, as shown in FIG. Store in storage means such as memory. Each element in the matrix represents the co-occurrence frequency of the corresponding character string in the set A and the component number in the set F. In this step, 0 is set as an initial value.

  Step 32) As a control 1, a predetermined range in the morphological analysis result to be processed is calculated for calculating the frequency with which the character string in the set A and the component number in the set F co-occur. The predetermined range is a string of a finite number of continuous words such as one sentence, one paragraph, and one document. The determination here can be performed, for example, by storing information indicating a predetermined range in the storage unit of the character string vector generation device 200 in advance and reading the information from the storage unit.

  When the predetermined range is one sentence, the first sentence in the morphological analysis result is first processed. When the processing from step 33 regarding the sentence to be processed is completed, the next sentence is set as the processing target. If the processing after step 33 relating to the last sentence is completed, there is no sentence to be processed, and thus this flowchart is ended. The same applies to the case where the predetermined range is other.

  Step 33) The array a which is an array of pairs of character strings and their frequencies in the set A, the array b which is an array of pairs of component numbers and their frequencies in the set F, and the undivided text are cleared.

  Non-divided text is generated by concatenating word notations in the range of the morphological analysis result to be processed.

  Step 34) As the control 2, an n-gram sequence in the range of the morphological analysis result to be processed is determined as follows.

  In each sentence in the non-split text in the range, the first character of the sentence is the first character, the position immediately before the first character is the 0th character, and thereafter the front of the sentence decreases and the rear of the sentence increases Then, number each letter. When the number of characters in the sentence is s, each position from the (2-n) th character to the sth character (however, if n ≧ s + 3, the (s-n + 3) th character to the 0th character N-grams from (excluded) are taken in order to form an n-gram sequence of the sentence. The n-gram sequence of each sentence in the non-divided text in the range is connected to the sentence in the order of the sentences, and is defined as the n-gram sequence in the range.

  After determining as described above, an n-gram to be processed in the range is determined as follows.

  First, the first n-gram of the n-gram sequence in the range is set as a processing target. When the processing of step 35 regarding the n-gram to be processed is completed, the next n-gram is set as the processing target. If the process of step 35 regarding the last n-gram is completed, there is no n-gram to be processed, and the process proceeds to step 36.

  Step 35) In the array a, a word and an n-gram are treated as different characters even if they match as a character string.

  Search set A by n-gram to be processed. If hit, do the following: The array a is searched with the n-gram. If there is no hit, the n-gram and its frequency 1 pair are added to the array a. If there is a hit, the frequency of the n-gram in sequence a is increased by 1.

  If the word notation character in the morphological analysis result corresponding to the first character of the n-gram to be processed is the first character of the word notation and the required word flag of the word notation is 1, the word notation The following processing is performed on the final form of processing.

  Search set A with the final form to be processed. If hit, do the following: The sequence a is searched with the end form. If there is no hit, add the terminal form and its frequency 1 pair to sequence a. If there is a hit, the frequency of the terminal form in sequence a is increased by one.

  Search set B with the final form to be processed. If hit, do the following: The following is performed for each component number associated with the final shape. Search the sequence b by the component number. If there is no hit, the pair of the component number and its frequency 1 is added to the sequence b. If there is a hit, the frequency of the component number in sequence b is increased by 1.

  From one range of the morphological analysis results shown in FIG. 13, the array a in FIG. 17 and the array b in FIG. 35 are obtained when there is no n-gram to be processed in step.

  In steps 34 and 35, the frequency of the n-gram, the word, and the component number in the range to be processed is obtained at the same time, but in steps 34 and 35, only the frequency of the n-gram based on the non-divided text in the range is obtained. In addition to these steps, a step of obtaining the frequency of words and component numbers based on the morphological analysis results in the range may be provided.

  Step 36) The character string to be processed in the array a is determined as follows.

  First, the first character string in the array a is a processing target. When the processing in step 37 regarding the character string to be processed is completed, the next character string is set as the processing target. When the process of step 37 regarding the last character string is completed, there is no character string to be processed, and the process proceeds to step 32.

  Step 37) It is assumed that the n-gram in the array a does not exist in the set B.

  A value obtained by multiplying the frequency of each component number V in the array b by the frequency in the array a of the character string U to be processed is added to the component corresponding to the component number V of the row vector corresponding to the character string U. Here, as a method of addition, self-co-occurrence that counts the co-occurrence with the component number of the character string itself for the process target character string in the process target range (even if there are multiple things of the same character string) There are two types: a method and a self-non-cooccurrence method that does not count.

1) Self-co-occurrence method
A value obtained by multiplying the frequency of each component number V in array b by the frequency in array a of character string U to be processed is added to the component corresponding to component number V of the row vector corresponding to character string U.

2) Self-non-co-occurrence method
Generate array b ′ as a copy of array b.
Search the set B with the character string U to be processed and obtain the corresponding component number set.
If the component number set is not empty, each component number of the character string U is reduced by 1 from the frequency in the component number array b ′.
A value obtained by multiplying the frequency of each component number V in the array b ′ by the frequency in the array a of the character string U is added to the component corresponding to the component number V of the row vector corresponding to the character string U.
• Clear array b '.

  As a specific example of the self-non-cooccurrence method, consider a case where the matrix is FIG. 34 and one range of the morphological analysis results shown in FIG. When the word “pan” in the array a in FIG. 17 is determined as a processing target in step 36, the component numbers 11, 91 in the array b in FIG. 35 are compared with the component numbers 11, 91 in the word “pan” in step 37. Each frequency of the array b ′ in FIG. 36 obtained by subtracting 1 from each of the frequencies is multiplied by the frequency 2 in the array “a” of the word “pan” and added to the row vector of the word “pan”. When there is no character string to be processed in step 36, the matrix of FIG. 37 is obtained.

  In the description of the flowchart of FIG. 4 in Example 2-2 above, a processing method that does not count the frequency of character strings in the range of the morpheme analysis result to be processed is realized by performing steps 33, 35, and 37 as follows: it can. This processing method can be configured not to include the process of counting the frequency of character strings and the process of counting the co-occurrence frequency of character strings in the range of the morphological analysis results to be processed when generating a matrix.

  Step 33) The array a which is an array of character strings in the set A, the array b which is an array of the component number and its frequency in the set F, and the non-divided text are cleared.

  Non-divided text is generated by concatenating word notations in the range of the morphological analysis result to be processed.

  Step 35) In the array a, a word and an n-gram are treated as different objects even if they match as a character string, and are stored in different areas.

  If the set A is searched for n-grams to be processed and hit, the n-gram is added to the array a. In the array a, n-grams are stored in the order of addition, and are added even when the same n-gram is already in the array a.

  If the word notation character in the morphological analysis result corresponding to the first character of the n-gram to be processed is the first character of the word notation and the required word flag of the word notation is 1, the word notation The following processing is performed on the final form of processing.

  If the set A is searched for the termination form to be processed and hit, the termination form is added to the array a. In array a, the terminal forms are stored in the order of addition, and are added even if the same terminal form is already in array a.

  Search set B with the final form to be processed. If hit, do the following: The following is performed for each component number associated with the final shape. Search the sequence b by the component number. If there is no hit, the pair of the component number and its frequency 1 is added to the sequence b. If there is a hit, the frequency of the component number in sequence b is increased by 1.

  From one range of the morphological analysis results shown in FIG. 13, the array a in FIG. 38 and the array b in FIG. 35 are obtained when there is no n-gram to be processed in step.

  Step 37) It is assumed that the n-gram in the array a does not exist in the set B.

  The frequency of each component number V in the array b is added to the component corresponding to the component number V in the row vector corresponding to the character string U to be processed. Here, as a method of addition, self-co-occurrence that counts the co-occurrence with the component number of the character string itself for the process target character string in the process target range (even if there are multiple things of the same character string) There are two types: a method and a self-non-cooccurrence method that does not count.

1) Self-co-occurrence method
The frequency of each component number V in the array b is added to the component corresponding to the component number V in the row vector corresponding to the character string U to be processed.

2) Self-non-co-occurrence method
Generate array b ′ as a copy of array b.
Search the set B with the character string U to be processed and obtain the corresponding component number set.
If the component number set is not empty, each component number of the character string U is reduced by 1 from the frequency in the component number array b ′.
The frequency of each component number V in the array b ′ is added to the component corresponding to the component number V in the row vector corresponding to the character string U.
• Clear array b '.

  In the description of the flowchart of FIG. 4 in the above Example 2-2, when there are a plurality of identical words or n-grams in the range of the morphological analysis results to be processed, Treated as a separate item. In the range of morpheme analysis results to be processed, even when there are a plurality of identical words or when there are a plurality of identical n-grams, a processing method that treats them as one can be considered. In the case of this processing method, steps 33, 35 and 37 are as follows.

  Step 33) An array a that is an array of character strings in set A, an array b that is an array of component numbers and their frequencies in set F, an array g that is an array of words in set B, and a non- Clear the split text.

  Non-divided text is generated by concatenating word notations in the range of the morphological analysis result to be processed.

  Step 35) In the array a, a word and an n-gram are treated as different characters even if they match as a character string.

  Search set A by n-gram to be processed. If hit, do the following: The array a is searched with the n-gram. If there is no hit, the n-gram is added to the array a.

  If the word notation character in the morphological analysis result corresponding to the first character of the n-gram to be processed is the first character of the word notation and the required word flag of the word notation is 1, the word notation The following processing is performed on the final form of processing.

  Search set A with the final form to be processed. If hit, do the following: The sequence a is searched with the end form. If there is no hit, the end form is added to sequence a.

  Search set B with the final form to be processed. If hit, do the following: The sequence g is searched with the end form. If not, do the following: Add the end form to array g. The following is performed for each component number associated with the final shape. Search the sequence b by the component number. If there is no hit, the pair of the component number and its frequency 1 is added to the sequence b. If there is a hit, the frequency of the component number in sequence b is increased by 1.

  From the range of the morphological analysis results shown in FIG. 13, when there is no n-gram to be processed in step 34, the array a in FIG. 39, the array g in FIG. 40, and the array b in FIG. 41 are obtained.

  Step 37) It is assumed that the n-gram in the array a does not exist in the set B.

  The frequency of each component number V in the array b is added to the component corresponding to the component number V in the row vector corresponding to the character string U to be processed. Here, as a method of addition, there are a self-co-occurrence method for counting the co-occurrence with the component number of the character string itself and a self-non-co-occurrence method not counting for the character string to be processed in the processing target range. It becomes processing contents.

1) Self-co-occurrence method
The frequency of each component number V in the array b is added to the component corresponding to the component number V in the row vector corresponding to the character string U to be processed.

2) Self-non-co-occurrence method
Generate array b ′ as a copy of array b.
Search the set B with the character string U to be processed and obtain the corresponding component number set.
If the component number set is not empty, each component number of the character string U is reduced by 1 from the frequency in the component number array b ′.
The frequency of each component number V in the array b ′ is added to the component corresponding to the component number V in the row vector corresponding to the character string U.
• Clear array b '.

  As an example of the self-non-co-occurrence method, consider the case where the matrix is FIG. 34 and one range of the morphological analysis results shown in FIG. When the word “pan” in the array “a” in FIG. 39 is determined to be processed in step 36, the component numbers 11, 91 in the array “b” in FIG. Each frequency of the array b ′ shown in FIG. 42 obtained by subtracting 1 from each frequency is added to the row vector of the word “pan”. When there is no character string to be processed in step 36, the matrix of FIG. 43 is obtained.

<Example 3>
The third embodiment is an embodiment in which the singular value decomposition unit 13 is provided in the character string vector generation devices 100 and 200. FIG. 44 shows a configuration of a character string vector generation device 301 provided with the singular value decomposition unit 13 in the character string vector generation device 100 of the first embodiment. FIG. 45 shows a configuration of a character string vector generation device 302 provided with the singular value decomposition unit 13 in the character string vector generation device 200 of the second embodiment.

  The singular value decomposition unit 13 reads the character string vector database 1 (18) from the character string vector storage unit 103 and uses it as an input, and performs singular value decomposition on the matrix of the character string vector database, thereby converting the matrix into a column Convert to a matrix with fewer numbers, and output the character vector database 2 (19), which is a set of pairs of character strings in set A and row vectors of the converted matrix corresponding to the character strings. And stored in the character string vector storage unit 103.

<Example 3-1>
In the character string vector database 1 (18), a case where a character string corresponding to a row is an n-gram will be described as Example 3-1. Example 3-1 corresponds to the case where the character string vector database 1 (18) is, for example, FIG. 9 of Example 1-1, or FIG. 26 or FIG. 32 of Example 2-1.

  Here, when the number of dimensions of the matrix in the character string vector database 1 (18) is J, a constant H satisfying J ≧ H ≧ 1 is taken. A character string vector database 2 (19) as shown in FIG. 46 is obtained by converting the matrix into a matrix having H columns by singular value decomposition.

<Example 3-2>
In the character string vector database 1 (18), the case where the character strings corresponding to the rows are words and n-grams will be described as Example 3-2. Example 3-2 corresponds to, for example, the case where the character string vector database 1 (18) is that of FIG. 20 of Example 1-2, or FIG. 37 or 43 of Example 2-2.
Here, when the number of dimensions of the matrix in the character string vector database 1 (18) is J, a constant H satisfying J ≧ H ≧ 1 is taken. A character string vector database 2 (19) as shown in FIG. 47 is obtained by converting the matrix into a matrix having the number of columns H by singular value decomposition.

  Thus, by generating a character string vector with a reduced number of dimensions, the accuracy of similarity between character string vectors is improved.

<Example 4>
In the fourth embodiment, the document vector generation unit 14 is provided in the character string vector generation devices 100, 200, 301, and 302. FIG. 48 shows a configuration of a character string vector generation device 401 provided with the document vector generation unit 14 in the character string vector generation device 301 of FIG. FIG. 49 shows a configuration of a character string vector generation device 402 provided with the document vector generation unit 14 in the character string vector generation device 302 of FIG. As shown in FIGS. 48 and 49, in this embodiment, a document information storage unit 104 for storing documents and document vectors is also provided.

  48 and 49, the singular value decomposition unit 13 is provided. However, the singular value decomposition unit 13 is not essential, and the singular value decomposition unit 13 may not be provided. That is, the document vector generation unit 14 uses the character string output from the first matrix generation unit 12 or the second matrix generation unit 23 instead of the character string vector database 2 (19) output from the singular value decomposition unit 13 as an input. The vector database 1 (18) may be input.

  In the fourth embodiment, the character string vector database 1 (18) and the character string vector database 2 (19) are collectively referred to as a character string vector database.

  The document vector generation unit 14 reads from the document information storage unit 104 and inputs the character string corresponding to the row of the matrix of the character string vector database for each character string in the document (20). The set is searched, a row vector corresponding to the searched character string is obtained, the sum or centroid of the row vector is output as the vector of the document, and stored in the document information storage unit 104.

  Hereinafter, as a specific example of Example 4, Example 4-1, in which a character string corresponding to a row is an n-gram character string vector database, and a character string vector corresponding to a row is a word and an n-gram character string vector A description will be given of Example 4-2 taking a database.

<Example 4-1>
In this embodiment, as the character string vector database, as shown in FIG. 9 of Embodiment 1-1, FIG. 26 or FIG. 32 of Embodiment 2-1, or FIG. 46 of Embodiment 3-1. A character string vector database whose character string corresponding to is n-gram is taken. At this time, the document vector of the input document is obtained as follows.

  The n-gram sequence in the input document is defined as follows.

  In each sentence in the input document, the first character of the sentence is the first character, the position immediately before the first character is the 0th character, and thereafter, each character so that the front of the sentence decreases and the rear of the sentence increases. Number. When the number of characters in the sentence is s, each position from the (2-n) th character to the sth character (however, if n ≧ s + 3, the (s-n + 3) th character to the 0th character N-grams from (excluded) are taken in order to form an n-gram sequence of the sentence. The n-gram sequence of each sentence in the input document connected in the order of the sentences is defined as the n-gram sequence in the input document. Here, when there are a plurality of identical n-grams in the n-gram sequence, only one of them may be used.

  A character string vector database is searched for each n-gram in the n-gram sequence, and a corresponding character string vector is obtained. The n-gram columns that are hit by the search are p1, p2,..., Pt, and the character string vector of each pr (1 ≦ r ≦ t) is v (pr).

または、
v(pr)（1≦r≦t）の重心である

Or
It is the center of gravity of v (pr) (1 ≦ r ≦ t)

または、

Or

を長さ1に正規化したベクトル、のうちのいずれかを算出し、入力文書の文書ベクトルとて出力する。どの方法で文書ベクトルを算出するかは予め設定しておくことができる。

One of the vectors normalized to length 1 is calculated and output as the document vector of the input document. Which method is used to calculate the document vector can be set in advance.

  As the character string vector database, when the character string vector database corresponding to the line is 2-gram as shown in FIGS. 9, 26, 32, and 46, the input document of FIG. The two-gram columns are extracted, the character string vector database is searched with each 2-gram of FIG. 51, the corresponding character string vectors are obtained, and the above formula is applied to these vector groups to input Generate a document vector for the document.

<Example 4-2>
In this embodiment, as the character string vector database, as shown in FIG. 20 of Embodiment 1-2, FIG. 37 or FIG. 43 of Embodiment 2-2, or FIG. 47 of Embodiment 3-2. A character string vector database of word and n-gram character strings corresponding to is taken. At this time, the document vector of the input document is obtained as follows.

  The input document is subjected to morpheme analysis, and the end form in the morpheme analysis result with the necessary word flag set to 1 is set as a word string in the input document. Here, when there are a plurality of identical words in the word string, only one of them may be used.

  The character string vector database is searched for each word in the word string, and the corresponding character string vector is obtained. A string of words hit by the search is set as p1, p2,..., Pt, and a character string vector of each pr (1 ≦ r ≦ t) is set as v (pr).

または、
v(pr)（1≦r≦t）の重心である

Or
It is the center of gravity of v (pr) (1 ≦ r ≦ t)

または、

Or

を長さ1に正規化したベクトル、のうちのいずれかを算出し、入力文書の文書ベクトルとて出力する。

One of the vectors normalized to length 1 is calculated and output as the document vector of the input document.

  For words in the word string that did not hit even after searching the character string vector database, the character string vector of the word is obtained by performing the processing of Example 4-1 using the word as an input document. It may be generated. That is, an n-gram string in the word is extracted, a character string vector database is searched for each n-gram in the n-gram string, and the character string vector of the word is obtained by synthesizing the obtained character string vectors. Generate.

  As described above, the input document vector may be generated by adding the word that has not been hit by the search and the generated character string vector to the word string and the character string vector string.

  When the character string vector database is a character string vector database in which the character string corresponding to the line is a word and 1-gram as shown in FIG. 20, FIG. 37, FIG. 43, and FIG. 47, the input document of FIG. 53, the morpheme analysis result of FIG. 53 is acquired, the word string of FIG. 54 is extracted from the morpheme analysis result of FIG. 53, and the character string vector database is searched for each word of FIG.

  Since the words “tea” and “drink” are hit, the corresponding character string vector is acquired.

  Since the word “Chicago” does not hit, the 1-gram column in FIG. 55 is extracted using the word as an input document, the character string vector database is searched with each 1-gram in FIG. 55, and the obtained character string vector is obtained. A character string vector of the word “Chicago” is generated by combining.

  A document vector of the input document is generated by synthesizing the character string vectors of the respective words in FIG.

  Each character string vector generation device described so far can be realized, for example, by causing a general computer including a CPU, a memory, a hard disk, and the like to execute a program corresponding to the processing means described in each embodiment. is there. The program may be distributed by being recorded on a computer-readable recording medium such as a portable memory, or may be downloaded from a server on the network. As described above, when the character string vector generation device is realized by a computer, processing data such as text and character strings is stored in a memory or the like, and is appropriately read and processed by a CPU according to a program instruction. Become.

  In addition, each character string vector generation device described so far may be realized by one computer or may be realized by connecting a plurality of computers via a network. For example, configuring one or more of the storage units 101 to 104 as an external server device is an example of a case where the character string vector generation device is realized by a plurality of computers.

  The present invention can be applied to a technique for generating a character string vector that is a semantic representation of a character string.

  The present invention is not limited to the above-described embodiments, and various modifications and applications are possible within the scope of the claims.

100, 200, 301, 302, 401, 402 Character string vector generator
11 String set extractor
12 First matrix generator
13 Singular value decomposition
22 Ingredient number assignment unit
23 Second matrix generator
101 Text storage
102 String set storage
103 String vector storage
104 Document information storage
14 Document vector generator

Claims (10)

A character string vector generation device for generating a character string vector as a matrix row vector,
Character string set extraction means for extracting different sets A and B of character strings, which are strings of a finite number of consecutive characters from each character position of the sentence in the text, from the input text;
A predetermined range that is a sequence of a finite number of consecutive words in the text by generating a matrix in which each row corresponds to a character string in the set A and each column corresponds to a character string in the set B The value obtained by adding the frequency of the co-occurrence of the character string C in the set A and the character string D in the set B over all the predetermined ranges in the text is present in the character string C. And a first matrix generation means for generating and outputting a matrix as an element in the matrix corresponding to the character string D, and a character string vector generation device. A character string vector generation device for generating a character string vector as a matrix row vector,
Character string set extraction means for extracting different sets A and B of character strings, which are strings of a finite number of consecutive characters from each character position of the sentence in the text, from the input text;
For each character string in the set B, from a set group obtained by excluding an already assigned component number set group from a set of component number sets extracted from L (L is an integer of 1 or more) component numbers A component number assigning unit that randomly selects one component number set and assigns the component number set to the character string ;
Each row is associated with a character string in the set A, a matrix is generated in which each column is associated with a component number in the component number set, and a predetermined finite number of words in the text. The frequency that the character string C in the set A and the component number E associated with the character string in the set B coexist in the range is added over all the predetermined ranges in the text A character string vector generation device comprising: a second matrix generation unit that generates and outputs a matrix in which the value obtained is an element in the matrix corresponding to the character string C and the component number E.   3. The character according to claim 1, further comprising singular value decomposition means for converting the matrix into a matrix having a smaller number of columns by performing singular value decomposition on the output matrix. Column vector generator.   For an input document, a set of character strings corresponding to the row of the output matrix is searched for each character string in the document, and a row vector corresponding to the searched character string is obtained. 4. The character string vector generation device according to claim 1, further comprising: a document vector generation unit that uses the sum or the center of gravity of the document as a vector of the document. 5. A character string vector generation method executed by a character string vector generation device that generates a character string vector as a row vector of a matrix,
A character string set extraction step for extracting different sets A and B of character strings, which are strings of a finite number of consecutive characters from each character position of the sentence in the text, from the input text;
A predetermined range that is a sequence of a finite number of consecutive words in the text by generating a matrix in which each row corresponds to a character string in the set A and each column corresponds to a character string in the set B The value obtained by adding the frequency of the co-occurrence of the character string C in the set A and the character string D in the set B over all the predetermined ranges in the text is present in the character string C. And a first matrix generation step of generating and outputting a matrix as an element in the matrix corresponding to the character string D. A character string vector generation method comprising: A character string vector generation method executed by a character string vector generation device that generates a character string vector as a row vector of a matrix,
A character string set extraction step for extracting different sets A and B of character strings, which are strings of a finite number of consecutive characters from each character position of the sentence in the text, from the input text;
For each character string in the set B, from a set group obtained by excluding an already assigned component number set group from a set of component number sets extracted from L (L is an integer of 1 or more) component numbers A component number assigning step that randomly selects one component number set and assigns the component number set to the character string ;
Each row is associated with a character string in the set A, a matrix is generated in which each column is associated with a component number in the component number set, and a predetermined finite number of words in the text. The frequency that the character string C in the set A and the component number E associated with the character string in the set B coexist in the range is added over all the predetermined ranges in the text A second matrix generation step of generating and outputting a matrix in which the value obtained is an element in the matrix corresponding to the character string C and the component number E.   7. The character according to claim 5, further comprising a singular value decomposition step of converting the matrix into a matrix having a smaller number of columns by performing singular value decomposition on the output matrix. Column vector generation method.   For an input document, a set of character strings corresponding to the row of the output matrix is searched for each character string in the document, and a row vector corresponding to the searched character string is obtained. The character string vector generation method according to claim 5, further comprising: a document vector generation step in which a sum or a center of gravity is used as a vector of the document.   The program for functioning a computer as each means in the character string vector generation apparatus described in any one of Claims 1 thru | or 4.   A computer-readable recording medium storing the program according to claim 9.

Images