JP5284117B2 - Word segmentation apparatus and method - Google Patents

Description

  The present invention relates to a word division apparatus and method.

  Conventionally, morphological analyzers and other dictionary information are used as division into words for abbreviation generation. In the case of English word division, a search is performed with reference to a token database or the like to divide words (for example, see Patent Document 1).

Special table 2008-515107 gazette

However, when a morphological analyzer or the like is used, proper word division may not be performed unless an appropriate word is registered in the dictionary. For example, in a name composed of a new word or coined word, a basic word included in the name becomes a word (unknown word) in which no word is registered in a dictionary such as a morphological analyzer, and word division cannot be performed. In particular, in an area that targets information that is updated daily, such as program guide data (EPG (Electronic Program Guide) data) of TV programs, new celebrities appear and new program names appear, so they are likely to become unknown words. . Furthermore, words in which katakana and English are mixed in hiragana and kanji cannot be distinguished and are easily determined as unknown words. It is difficult to sufficiently deal with such unknown words with basic word data or data stored in the dictionary of the morphological analyzer, and the dictionary needs to be updated frequently.
However, it is expensive to manually update the dictionary for unknown words. When morphological analysis is used in this way, it is difficult to divide new words and coined words into abbreviation generation words.

  The present invention has been made to solve the above-described problem, and even if a new word or coined word such as a character string such as a name that cannot be divided by a morphological analyzer, word division can be performed. Apparatus and methods are provided.

  In order to solve the above-described problem, a word segmentation device according to the present invention divides an input character string into two parts between an input unit that receives an input of a character string and obtains an input character string, and all characters of the input character string. Dividing to obtain a divided character string consisting of a first half character string that is a character string of the first half part after division and a second half character string that is a character string of the second half part after division by a number one less than the number of characters of the input character string Means, a first frequency which is a number indicating the frequency of occurrence of the input character string, a second frequency which is a number indicating the frequency of appearance of the first half character string, and a number indicating the frequency of appearance of the second character string A plurality of the divided characters according to a ratio of the acquisition means for acquiring the third frequency, the value of the first frequency, and the smaller one of the second frequency value and the third frequency value. The divided character string having the smallest ratio among the columns is defined as the optimum divided character string. First determination means for determining, and at least one of an optimal first half character string that is the first half character string included in the optimally divided character string and an optimal second half character string that is the second half character string included in the optimal divided character string is further divided A second determination means for determining, as a basic word, a character string satisfying the stop condition among the optimum first half character string and the optimum latter half character string. Features.

  According to the word segmentation apparatus and method of the present invention, even if a character string such as a name is a new word or coined word that cannot be segmented by a morphological analyzer, it is possible to segment the word.

The block diagram which shows the word division | segmentation apparatus which concerns on 1st Embodiment. The flowchart which shows operation | movement of a phrase division part. The figure which shows an example of a word division process. The figure which shows an example of character string frequency data. The figure which shows another example of a word division | segmentation process. The block diagram which shows the word division | segmentation apparatus which concerns on a modification. The block diagram which shows the word division | segmentation apparatus which concerns on 2nd Embodiment. The flowchart which shows operation | movement of an abbreviation production | generation part. The block diagram which shows the word division | segmentation apparatus concerning 3rd Embodiment. The flowchart which shows operation | movement of an abbreviation production | generation part and an abbreviation candidate selection part.

Hereinafter, a word segmentation apparatus and method according to embodiments of the present invention will be described in detail with reference to the drawings. In the following embodiments, the same reference numerals are assigned to the same numbered parts, and repeated description is omitted.
(First embodiment)
The word segmentation device according to the present embodiment of the present invention uses an external character string frequency data or the like without using a dictionary or word data, thereby dividing an input character string using only the character string frequency. To divide.
The word division device according to the first embodiment will be described with reference to FIG.
The word segmentation apparatus 100 according to the present embodiment includes an input unit 101, a segmentation unit 102, a character string frequency measurement unit 103, a segment position determination unit 104, a stop condition determination unit 105, and an output unit 106. In the following text, the two-dividing unit 102, the character string frequency measuring unit 103, the division position determining unit 104, and the stop condition determining unit 105 are collectively referred to as a word / phrase dividing unit 107.

The input unit 101 accepts a character string such as a name, an organization name, a program or product name, and sends it to the bipartition unit 102. A character string input to the input unit 101 may be referred to as an input character string below. The character string includes the case of only one character.
The two-dividing unit 102 divides the input character string received from the input unit 101 into two and sends it to the character string frequency measuring unit 103. The detailed operation of the two-dividing unit 102 will be described with reference to FIG.

The character string frequency measuring unit 103 uses the external database 108 to determine the frequencies of all the input character strings divided into two (hereinafter referred to as divided character strings) received from the two dividing unit 102 and the input character strings before the division. The measurement result is sent to the division position determination unit 104. Here, the external database 108 is character string frequency data on web, existing document data, existing character string frequency dictionary, and the like. The frequency is a number indicating the frequency (appearance frequency) of occurrence of a specific character string in an external database such as a web or in an existing document. The frequency is, for example, the number of hits of a character string in a web database, the number of hits of a web document or content including a character string, and the like. In existing document data, the number of appearances of a specific character string in a document and the number of documents including the specific character string.
The division position determination unit 104 determines the most suitable division position of the input character string using the frequency data of the set of divided character strings obtained from the character string frequency measurement unit 103 and the frequency data of the input character string, and determines the stop condition. The data is sent to the determination unit 105. The detailed operation of the division position determination unit 104 will be described with reference to FIG.
The stop condition determination unit 105 determines whether or not the divided character string input from the division position determination unit 104 satisfies the stop condition. If the stop condition is not satisfied, the divided character is recursively until the stop condition is satisfied. The string is sent to the two-dividing unit 102 to generate a new divided character string. If the stop condition is satisfied, the divided character string is sent to the output unit 106. In the following, a word output after the processing of the phrase dividing unit 107 is defined as a basic word indicating that the input character string is divided at a position most suitable for the condition under a certain condition. For example, the first name is divided into the last name and the first name, and the abbreviation is divided so that the abbreviation is a word that is the basis of the abbreviation.
The output unit 106 receives the divided character string that satisfies the stop condition in the stop condition determination unit 105 and outputs the divided character string to the outside as a basic word.

Next, the operation of the phrase dividing unit 107 when a character string is input will be described with reference to the flowchart of FIG. 2 and FIGS. 3 and 4.
As an example of the input character string, a case where “Taberu no Tobira” is input as a program name is shown. In the present embodiment, an example in which character string frequency measurement is performed using an external database 108 will be described.
First, in S <b> 201, “Tabera no Tobira” is input to the input unit 101 as an input character string. Then, the input character string “Tateru no Tobira” is sent to the two-dividing unit 102.

Next, in S202, the two-dividing unit 102 performs all divisions to divide the input character string “Tateru no Tobira” into two. An example in which all the divisions are performed will be described with reference to FIG. If i (division after the i-th character) representing the division position is used, six divisions in which i is from 1 to 6 are performed. In other words, “Tabe / Take no Tobira”, “Tabe / Toru no Tobira”, “Tabe no Tobira”, “Tabe no Tobira”, “Tabe no Tobira”, “Tabe no Tobira” The input character string is divided between all characters “/ ra” to generate a divided character string. However, the “/” symbol represents a word division position. Here, j (j = 1 is the first half character string, j = 2 is the second half character string as the first half character string that is the first half character string after the division and the second half character string that is the second half character string after the division. A character string) is used to denote Sij. In the case of the division of “Tabe no Tora”, “Ta” is the first half character string S11, “Beru no Tora” is the second half character string S12, and similarly “Tabe” is S21, “Ru no Tora”. ”Is called S22,...“ Tate no Tobi ”is called S61, and“ R ”is called S62. Further, the input character string “Tateru no Tobira” before the division is called S0. The order of division may be divided in order from the first character, or may be divided randomly. In any division order, the input character string may be divided without any overlap between all characters.
In step S203, the character string frequency measurement unit 103 receives the characters S0, Si1, and Si2 (i = 1 to n, where n is (character string length-1 of S0)) received from the bipartition unit 102. Measure the column frequency. For example, the database 108 that is a character string frequency measurement target performs measurement using the external character string frequency data.
As an example of character string frequency data measurement, an API (Application Programming Interface) for web search is used, and a search result is obtained by using a character string in which each character string whose frequency is to be measured is enclosed in double quotation marks (""). It is obtained by acquiring the number (number of documents). At this time, when the character string is “eat”, the search word is “eat”. Further, when measuring the frequency of each character string in the existing document data, it is obtained by measuring the frequency of each character string to be frequency-measured with respect to document data such as data held in the organization, for example. . Furthermore, the frequency measurement using the existing character string frequency dictionary is to prepare a table holding data as shown in FIG. 4 and obtain the frequency from this table. The data of FIG. 4 creates a character string (character string data 401) having the number of characters up to m characters (m is a natural number) instead of words from the document data, and the frequency of each character string (character string frequency data 402). It is obtained by measuring in the document data.
Here, the character string frequency of the character string S obtained from the database 108 is expressed as Hit (S0), Hit (Si1), Hit (Si2) (i = 1 to n), and acquired in association with each other.

In S204, the division position determination unit 104 determines an optimal division position from the character string frequency data 402 acquired in S203. Here, the degree of association R is calculated from the character string sets Si1 and Si2 and the frequency of the character string S0 before division. The relevance R is calculated from the following equation (1), for example.
R (i) = Hit (S0) / min (Hit (Si1), Hit (Si2)) (1)
Using this equation, the division position of i = k that minimizes this is calculated, and Sk1 and Sk2 are output. An example of calculating R (i) when i = 4 will be described with reference to FIG. 3. When i = 4, the divided character string is S41 = “eat”, S42 = “door”, It is presumed that it will be divided into “Tabe no Tobira”. Further, when calculating the relevance R, the character string frequency of “Tateru no Tobira”, which is S0, is Hit (S0) = 11,000,000, and the character string frequency of “Tabe no Tora” is Hit (S41) = 1890000, It is assumed that the character string frequency of “Bira” is Hit (S42) = 12900000. As a result of calculating the relevance R (4) using these character string frequencies, R (4) = 5.851 (here, rounded to the fourth decimal place) is obtained from the equation (1). In calculating min (Hit (Si1), Hit (Si2)), if either Hit (Si1) or Hit (Si2) is 0, it may be excluded from the candidates as not applicable.

In this way, the degree of relevance R is calculated in the same manner for other division positions i, and the value with the smallest degree of relevance R is estimated as the optimum position for dividing the input character string. Here, it is assumed that R (4) is the smallest, and the process proceeds to S205 with “eat” and “door” as basic words.
Similarly, the case where the input character string is the character string “Himechakarin” will be described with reference to FIG. Similarly to “Tabera no Tobira”, the processing of S202 and S203 is performed on “Himechamakalin”, and the word division frequency is measured for each of the seven word divisions by the two division unit 102. To do. Next, as an example, when i = 5, S51 = "Himechama" and S52 = "Karin", and the frequency of these character strings is measured. Hit (S51) = 931000, Hit (S52) = 17300000, and the character string frequency of “Himechakarin” is Hit (S0) = 899000, and the relevance R (5) using the equation (1) at this time Calculate If R (5) is the smallest than the other relevance, R (5) is estimated to be divided into “Himechama / Karin” and “Himechama” and “Karin” are used as basic words in S205. move on.

  The expression of R (i) is called a Simpson coefficient that measures the relationship between words (Simpson (A, B) = A∩B / min (A, B) when measuring the relationship between word A and word B)). It is an improvement of the formula. When indexing is performed by removing unique character strings, calculation may not be performed well even if the Simpson coefficient is used as it is. For this reason, Formula R (i) that can perform word division only by using the external character string frequency data 402 is used.

In S205, the stop condition determination unit 105 determines whether the divided character string received from S204 satisfies the stop condition. The stop condition may be, for example, the case where each divided character string is 2 or less Kanji characters, 4 or less katakana or hiragana characters, or the number of mora instead of the number of characters.
Here, if the division of “Tateru” and “Toba” is obtained from the input character string “Tateru no Tobira” in “S204”, “Tateno” is less than 4 hiragana characters and is output as it is. The process proceeds to S206. Also, since “Tobira” is less than 4 characters of Katakana and Hiragana, this is output as it is and the process proceeds to S206. On the other hand, under the same stop condition, when the division of “Himechama” and “Karin” is obtained in S204 from the input character string “Himechakarin”, “Karin” is less than 4 hiragana characters. Therefore, it is output as it is. However, since “Himechama” has more than 4 hiragana characters and does not satisfy the stop condition, “Himechama” is returned to S201, and a further division position is obtained. In this way, the character string dividing process is repeated recursively until all the divided character strings satisfy the stop condition.
In S206, the divided character string satisfying the stop condition in S205 is output as a basic word. Here, “eat” and “door” are output. The processing of the phrase dividing unit 107 is terminated by the above-described steps.

(Modification)
Next, FIG. 6 shows a modification of the word segmentation apparatus 100 shown in FIG.
A word segmentation apparatus 600 according to this modification includes a morpheme analysis unit 601 in addition to the word segmentation apparatus 100 shown in FIG. The other configurations included in the word segmentation apparatus 600 perform the same operations as those in the first embodiment described above, and thus description thereof is omitted here.
In the first embodiment described above, it is assumed that input is performed in a state where the start position and end position of an unknown word can be recognized mainly in units of words such as program names and names. Not only the unit but also the input in a state where the end position of the unknown word is not known in the sentence, the unknown word can be extracted in units of words and divided into basic words by performing morphological analysis.
The morpheme analysis unit 601 receives an input character string from the input unit 101 and performs a morphological analysis. As a result, the morpheme analysis unit 601 determines that the word is an unknown word and becomes a single word, or treats a character string that has become a single word by handling a name, etc. The data is sent to the dividing unit 107. Instead of the morphological analyzer, words may be divided using the particles “ha”, “no”, etc. as division positions. For example, in the case of an input character string of “Tashiba Suro's Friday Entertainment”, the morpheme analysis unit 601 performs morpheme analysis. As a result, if the personal name “Tashiba Suro” or the coined word “Entertainment” becomes an unknown word, the “Tashiba Suro” or “Entertainment” is given as an input to the word segmentation unit 107 to split the first and last names. And word division.
Further, the word / phrase dividing unit 107 may give a person's name as an input and perform first / last name division. Further, the phrase dividing unit 107 may give a compound word as an input and divide it into basic words constituting the compound word.

  According to the first embodiment described above, even an unknown word such as a new word or coined word that cannot be divided by a morphological analyzer can be divided. Even when a sentence is input, an unknown word can be extracted by performing morphological analysis, and word division can be performed on the unknown word. Further, since the calculation is performed by determining an appropriate two-divided position, the calculation can be performed with less access to the outside and the amount of calculation than when calculating the cohesion degree from all the partial character strings.

(Second Embodiment)
A second embodiment according to the present invention will be described with reference to FIG. The word division device 700 according to the present embodiment further includes an abbreviation generation unit 701 in addition to the word division device 100 shown in FIG. The other configurations included in the word segmentation apparatus 700 perform the same operations as those of the word segmentation apparatus 100 of the first embodiment described above, and thus description thereof is omitted here.
The difference from the word segmentation apparatus 100 according to the first embodiment is that an abbreviation candidate is acquired by performing abbreviation generation after performing word division into basic words.
The abbreviation generation unit 701 generates abbreviation candidates from the basic words received from the phrase dividing unit 107 according to the abbreviation generation rules and sends them to the output unit 106. Alternatively, each character string may be used as a word for machine learning as a word to generate abbreviation candidates and send them to the output unit 106. The abbreviation generation rules specify, for example, the positions of basic words and characters used for abbreviation generation among the basic words obtained by dividing the input word. For example, a method of generating by combining character positions (for example, beginning and end of a word) or adding processing for removing long sound and sound repellent. As an example, if the official name is “Chuo Expressway between Capitals” (City / Central / Automobile / Road), the abbreviation “Select one letter at the end of the first, second and fourth basic words” If it is a production rule, “between” of “between capitals”, “center” of “center”, and “road” of “road” are selected from each basic word and are abbreviated as “middle road”. In addition, if the official name is “third junior high school” (third / junior high school), the abbreviation generation rule “select one letter at the end of the first basic word and one letter at the beginning of the second basic word” “Third” “three” and “junior high school” “middle” are selected and abbreviated “three middle”. Furthermore, in the abbreviation generation rule “excludes long sound”, the basic word “super” is abbreviated as “spa”. Also, for machine learning, for example, there is a technique such as CRF (Conditional Random Fields), and it is determined whether each character is a character used as an abbreviation.

An example of processing in the abbreviation generation unit 701 will be described with reference to the flowchart of FIG.
First, when the input character string to the input unit 101 is “Tateru no Tobira”, the basic words “Tateru” and “Tobira” are obtained as a result of the word division processing in the word division unit 107, respectively. To do.
In step S <b> 801, two words (input basic word 804), “eat” and “tobira”, are input to the abbreviation generation unit 701.
Next, in S802, when abbreviation generation is performed using abbreviation generation that obtains two initial characters at the time of applying the abbreviation generation rule, the abbreviation “tabetobi” is used as the basic word “taberuno” and “tobira”. Generated. If there are a plurality of abbreviation generation rules, a plurality of abbreviation candidates are also generated (abbreviation candidate 805). After the abbreviation candidate 805 is generated, the process proceeds to S803.
In step S803, the abbreviation candidate 805 generated in step S802 is output. The process in the abbreviation generation unit 701 is terminated by the above steps.

In addition, the word division unit 107 may store the number of word divisions, and may input not only the division position but also the number of divisions to the abbreviation generation unit 701. The division position and the number of divisions are assumed to be stored in the stop condition determination unit 105, but are not particularly limited and may be stored anywhere. By using not only the division position but also the number of divisions as an input to the abbreviation generation unit 701, for example, the abbreviation generation unit 701 outputs an abbreviation candidate 805 composed of division positions with a small number of divisions. Alternatively, the candidates may be ranked and output in ascending order of the number of divisions. This is because the smaller number of divisions consists of basic words that are more easily selected as the basis for the abbreviation candidate 805. For example, in the case of the input character string “Himechakamarin”, the character string “Himechama” is divided again after being divided into basic words “Himechama” and “Karin” in the first division. , And “Hime” and “Chama”. At this time, the information of only the division position is composed of three basic words “hime / chama / karin”.
Here, if the number of divisions is given as additional information, it will be divided into “hime / 2 chama / 1 karin”. However, “/ i” represents the i-th division position. If the abbreviated name generation rule is “created by arbitrarily connecting two words beginning with two mora”, three candidates “Himechama”, “Himekari”, and “Chamakari” are generated. However, if the number of divisions is given as additional information, and the abbreviation generation rule at this time is, for example, “created from two-word prefix 2 mora divided at a position where the number of divisions is small”, only “Himekari” The abbreviation candidate 805 can be selected.

  According to the second embodiment described above, it is possible to automatically generate an abbreviation from a basic word according to an abbreviation generation rule. Further, since the division is repeated recursively, it is possible to measure a division position that is easier to divide from information indicating how many divisions have been made.

(Third embodiment)
A third embodiment according to the present invention will be described with reference to FIG. The word division device 900 according to the present embodiment further includes an abbreviation candidate selection unit 901 in addition to the word division device 700 according to the second embodiment shown in FIG. The character string frequency measurement unit 103 is different from the first embodiment and the second embodiment in that character string frequency data is exchanged with the abbreviation candidate selection unit 901, but otherwise performs the same operation.
Other configurations included in the word segmentation apparatus 900 perform substantially the same operations as those of the word segmentation apparatus 700 according to the second embodiment described above, and thus description thereof is omitted here.
The difference from the second embodiment is that the abbreviation candidates 805 generated by the abbreviation generation unit 701 are not output as they are, but the frequency of the abbreviation candidates 805 or the abbreviation candidates 805 and the input characters are output for a plurality of abbreviation candidates 805. The abbreviation candidate 805 whose co-occurrence frequency with the column is smaller than the threshold value is deleted, ranking is performed with reference to the co-occurrence frequency, and only the selected abbreviation is sent to the output unit 106. The threshold is a value that is statistically determined by the number of appearances of the character string. In other words, if the frequency of the character string of the abbreviation candidate 805 is equal to or higher than the threshold, it can be said that the abbreviation candidate 805 has a high accuracy rate as an abbreviation of the input character string, that is, high reliability. Conversely, if the frequency of the character string of the abbreviation candidate 805 is smaller than the threshold, the reliability of the abbreviation candidate 805 as an abbreviation is low. The co-occurrence frequency is a frequency (number of appearances) indicating how much the input character string and the abbreviation candidate 805 appear in association with each other. For example, it is conceivable that the relationship between the input character string and the abbreviation candidate is searched on the Web and obtained by measuring the frequency.

  The abbreviation candidate selection unit 901 receives the abbreviation candidate 805 generated by the processing of the abbreviation generation unit 701, or acquires and references the co-occurrence frequency measured from the character string frequency measurement unit 103, and ranks a plurality of abbreviation candidates. Then, the abbreviated candidate candidates having higher ranks are sent to the output unit 106, or only the abbreviated candidate candidates selected from the abbreviated candidate candidates by the threshold are sent to the output unit 106.

An example of abbreviation candidate selection processing based on basic words in the abbreviation generation unit 701 and the abbreviation candidate selection unit 901 in the present embodiment will be described with reference to the flowchart of FIG. If the input character string to the input unit 101 is “Tateru no Tobira”, it is assumed that, as a result of the word division processing in the phrase dividing unit 107, divided words “Tateru” and “Tobira” are obtained. .
First, in S 1001, two words (input basic word 804) “eat” and “tobira” are input to the abbreviation generation unit 701.
Next, in S1002, in the abbreviation generation unit 701, when an abbreviation generation rule that obtains two initial characters is applied, the abbreviation “Tabetobi” is generated with “Tabe no” and “Tobira” as basic words. . At this time, if there are a plurality of abbreviation generation rules, some abbreviation candidates 805 are generated in addition to “Tabetobi”.

In step S <b> 1003, the abbreviation candidate selection unit 901 selects and ranks a plurality of abbreviation candidates 805 received from the abbreviation generation unit 701. For example, two words of the input character string “Tabe no Tobira” and each abbreviation candidate 805 are used as search words, and the number of search results (co-occurrence frequency) is obtained by the Web search API through the character string frequency measurement unit 103. If there are three words “Tabetobi”, “Tatobi”, and “Tabeto” as abbreviation candidates 805, the search word A ““ Tabera no Tobira ”“ Tabetobi ”” and the search term B “″ The frequency on the Web is obtained by using the three search words “Tab” and the search word C “Taber no Tobira” “Tabet”. As a result, when the frequency is A: 86700, B: 0, C: 85200 (abbreviation candidate frequency 1005), the “tabetobi” having the highest frequency is selected, and the process proceeds to S1004. When selecting the abbreviation candidate 805 using a threshold, for example, the abbreviation candidate 805 having a frequency lower than the threshold is not selected and output is not performed, or conversely, the abbreviation candidate having a frequency equal to or higher than the threshold is used. 805 may select all and perform processing such as outputting.
Further, selection processing may be performed by simultaneously performing thresholding and ranking. For example, assume that three abbreviation candidates 805 are selected and output. When the top three abbreviation candidates 805 selected by ranking have the frequency of the top two abbreviation candidates 805 equal to or higher than the threshold value, and the frequency of the third abbreviation candidate 805 is equal to or lower than the threshold value, The abbreviation candidate 805 has high reliability as an abbreviation, and one abbreviation candidate 805 smaller than the threshold value is selected, but it can be said that the reliability as an abbreviation is low.

  In addition, the situation where the number of cases changes due to the indexing of the search engine may be considered. For example, when the search word C is searched from the search word A and the frequency on the Web is obtained, the frequencies are A: 85200, B: 0, C: 86700 (abbreviation candidate frequency 1005), respectively. “Tabet” is the most frequent candidate. However, the two character strings “Tabetobi” (words with a non-zero frequency) and “Tabet” with the highest frequency are inclusive relations (“Tabet” + “Bi” ⇒ “Tabetobi”) In fact, “Tabetobi” is the correct abbreviation, but both “Tabetobi” and “Tabetobi” may be registered as words in the search engine. In this case, a document that includes “tabetobi” but is indexed only by “tabet” does not appear as the number of cases.

In order to consider such a case, the search result high-order summary (snippet) is examined, and the character string inclusion relationship of the abbreviation candidate 805 (a partial character string from the beginning of a certain character string or a partial character string from the back is To see if it matches another string. A character string in which one candidate (longer character length) of two arbitrary abbreviation candidates 805 includes the other (shorter character length) is searched with the shorter abbreviation candidate 805. It is checked whether the shorter abbreviation candidate 805 appearing in the result summary (snippet) is not a part of the longer abbreviation candidate 805. If there is an inclusion relationship, it is determined that the frequency on the Web with the shorter character string length is due to the longer frequency, and the abbreviation candidate 805 with the longer character string length is selected. In the case of “Tabetobi” and “Tabetobi”, “Tabetobi” is a partial character string from the beginning of “Tabetobi”. When “tabetto” appearing in the search result frequently appears in the phrase “... tabetobi ...”, “tabetobi” is selected as the abbreviation candidate 805, and the process proceeds to S1004.
In step S <b> 1004, the output unit 106 outputs the abbreviation candidate 805 received from the abbreviation candidate selection unit 901. The selection process of the abbreviation candidate 805 from the basic word is completed by the above steps.

  Further, by using the information about how many times the input character string is divided when the input character string is divided into two, which is held in the phrase dividing unit 107, for example, the abbreviation candidate selecting unit 901 starts from a division position with a small number of divisions. The abbreviation candidate 805 may be output with priority. As described above, it is assumed that as a result of dividing the input character string “Himechakarin” by the phrase dividing unit 107, the division position and the number of divisions become “Hime / 2chama / 1karin”. If the abbreviation generation rule is “created by connecting two mora of arbitrary two words in the basic word”, three abbreviation candidates “Himechama”, “Himekari”, “Chamakari” 805 is generated. Then, the number of divisions is given as additional information to each abbreviation candidate 805. At this time, in S1003 of FIG. 10, when priority is given to a word created from two words divided by a smaller number of divisions, “/ 0 hime / 1”, “/ 0 hime / 2 chama”, “/ The order of the sum of the separators i, “2 Chama / 1 kari”, in ascending order is “Himekari” (total is 1), “Himechama” (sum is 2), and “Chamakari” (sum is 3). I will do it.

  According to the third embodiment described above, abbreviations can be automatically generated from the divided basic words. Furthermore, since it is not necessary to prepare connection probability data, it is assumed that the character strings (basic words and original words) after word splitting are indexed even in data such as search engines used for frequency measurement. In addition, the relationship before and after the word division can be related even in an unknown state. Further, there are a plurality of abbreviation generation rules, and when a plurality of abbreviation candidates are generated, a desired abbreviation can be automatically selected from a plurality of abbreviation candidates by selecting according to the frequency or the number of divisions.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

100, 600, 700, 900 ... word dividing device, 101 ... input unit, 102 ... dividing unit, 103 ... character string frequency measuring unit, 104 ... dividing position determining unit, 105 ... Stop condition determination unit, 106 ... output unit, 107 ... phrase division unit, 108 ... database, 401 ... character string data, 402 ... character string frequency data, 601 ... morphological analysis Part 701 ... abbreviation generation part 804 ... input basic word 805 ... abbreviation candidate, 901 ... abbreviation candidate selection part, 1005 ... abbreviation candidate frequency.

Claims (12)

An input means for receiving input of a character string and obtaining an input character string;
Dividing the input character string into two parts between all characters of the input character string, and dividing the input character string into a first half character string that is the first half character string after the division and a second half character string that is the second half character string after the division Dividing means for obtaining a character string by a number one less than the number of characters of the input character string;
A first frequency that is a number indicating the frequency of appearance of the input character string, a second frequency that is a frequency indicating the frequency of appearance of the first half character string, and a number indicating the frequency of appearance of the second character string. An acquisition means for acquiring three frequencies;
A divided character in which the ratio of the plurality of divided character strings is minimized by a ratio of the first frequency value and the smaller one of the second frequency value and the third frequency value. First determination means for determining a column as an optimally divided character string;
A stop condition indicating a condition in which at least one of the optimum first half character string that is the first half character string included in the optimum division character string and the optimum second half character string that is the second half character string included in the optimum division character string is not further divided. When satisfying, comprises a second determination means for determining, as a basic word, a character string that satisfies the stop condition among the optimal first half character string and the optimal second half character string,
The acquisition means acquires the first frequency, the second frequency, and the third frequency, which are results of searching the input character string or the divided character string in a first external database. Word splitting device. An input means for receiving input of a character string and obtaining an input character string;
Dividing the input character string into two parts between all characters of the input character string, and dividing the input character string into a first half character string that is the first half character string after the division and a second half character string that is the second half character string after the division Dividing means for obtaining a character string by a number one less than the number of characters of the input character string;
A first frequency that is a number indicating the frequency of appearance of the input character string, a second frequency that is a frequency indicating the frequency of appearance of the first half character string, and a number indicating the frequency of appearance of the second character string. An acquisition means for acquiring three frequencies;
A divided character in which the ratio of the plurality of divided character strings is minimized by a ratio of the first frequency value and the smaller one of the second frequency value and the third frequency value. First determination means for determining a column as an optimally divided character string;
A stop condition indicating a condition in which at least one of the optimum first half character string that is the first half character string included in the optimum division character string and the optimum second half character string that is the second half character string included in the optimum division character string is not further divided. When satisfying, comprises a second determination means for determining, as a basic word, a character string that satisfies the stop condition among the optimal first half character string and the optimal second half character string,
The acquisition means stores a plurality of character strings, and for each character string, the first frequency, the second frequency, and the first frequency from a second database that stores the character string and the frequency of the character string in association with each other. A word segmentation device characterized by acquiring three frequencies. An input means for receiving input of a character string and obtaining an input character string;
Dividing the input character string into two parts between all characters of the input character string, and dividing the input character string into a first half character string that is the first half character string after the division and a second half character string that is the second half character string after the division Dividing means for obtaining a character string by a number one less than the number of characters of the input character string;
A first frequency that is a number indicating the frequency of appearance of the input character string, a second frequency that is a frequency indicating the frequency of appearance of the first half character string, and a number indicating the frequency of appearance of the second character string. An acquisition means for acquiring three frequencies;
A divided character in which the ratio of the plurality of divided character strings is minimized by a ratio of the first frequency value and the smaller one of the second frequency value and the third frequency value. First determination means for determining a column as an optimally divided character string;
A stop condition indicating a condition in which at least one of the optimum first half character string that is the first half character string included in the optimum division character string and the optimum second half character string that is the second half character string included in the optimum division character string is not further divided. When satisfying, comprises a second determination means for determining, as a basic word, a character string that satisfies the stop condition among the optimal first half character string and the optimal second half character string,
The acquisition means acquires the existing document data, and measures the frequency of occurrence of the input character string, the first half character string, and the second half character string in the document data, thereby obtaining the first frequency, The word division apparatus characterized by acquiring the second frequency and the third frequency. The second determination means does not satisfy the stop condition of the optimal first half character string and the optimal second half character string when at least one of the optimal first half character string and the optimal second half character string does not satisfy the stop condition. the string sent to the dividing means as a new input character string, said dividing means any one of claims 1, wherein the obtaining the new division string on the basis of the new input string of claim 3 1 The word segmentation device according to item. The second determination means does not satisfy the stop condition of the optimal first half character string and the optimal second half character string when at least one of the optimal first half character string and the optimal second half character string does not satisfy the stop condition. A character string is sent to the dividing means as a new input character string, and the dividing means obtains a new divided character string based on the new input character string,
Selecting N character strings (N is an integer of 0 or more) at the designated character positions as the abbreviated candidate character strings at positions from the first character for each of the basic words in the plurality of basic words; The generator according to any one of claims 1 to 3 , further comprising generating means for generating a first abbreviation of at least two characters or more by a specified combination of the plurality of abbreviation candidate character strings. Word divider. A plurality of said first abbreviation, occurrence frequency representing at least one of the document data beauty existing Oyo in the first database, the frequency of the input character string and the first abbreviation is co-occur in the same document When the selection is performed based on at least one of the threshold value indicating the level of the co-occurrence frequency and the order in which the first abbreviation with the higher co-occurrence frequency is positioned higher, the co-occurrence frequency 6. The method according to claim 5 , further comprising selection means for preferentially selecting the first abbreviation as the rank is higher when selecting the first abbreviation having a value equal to or higher than the threshold. The word segmentation device described. When the co-occurrence frequency of the second abbreviation selected from the first abbreviations is greater than the co-occurrence frequency of the third abbreviation that is an abbreviation that includes all of the second abbreviations, the third abbreviation is given priority for output. The word segmentation device according to claim 6 . Said generating means, to any one of claims 7 claim 5, characterized in that to generate the abbreviation combining the abbreviation candidate character strings in the order of the basic word division number is small in the input string The word segmentation device described. When the character string is a sentence, the character string is further subjected to morphological analysis, and further includes an analysis unit that acquires a character string that is included in the character string and is one word as a one-word character string,
9. The word dividing device according to claim 1 , wherein the dividing unit acquires the one-word character string as an input character string and divides the input character string. The input means accepts the input of the character string, obtains the input character string,
The dividing means divides the input character string into two parts between all characters of the input character string, and the first half character string that is the first half character string after the division and the second half character string that is the second half character string after the division A divided character string consisting of is obtained by a number one less than the number of characters of the input character string,
The acquisition means indicates a first frequency that is a number indicating the frequency of appearance of the input character string, a second frequency that is a number indicating the frequency of appearance of the first half character string, and a frequency of occurrence of the second character string. Get a third frequency that is a number,
The first determination means determines the ratio of the plurality of divided character strings based on a ratio between the first frequency value and the smaller one of the second frequency value and the third frequency value. Is determined as the optimal divided character string.
At least one of the optimal first half character string that is the first half character string included in the optimal divided character string and the optimal second half character string that is the second half character string included in the optimal divided character string is further divided by the second determination unit. When the stop condition indicating no condition is satisfied, a character string satisfying the stop condition is determined as a basic word among the optimal first half character string and the optimal second half character string ,
The acquisition means acquires the first frequency, the second frequency, and the third frequency, which are results of searching the input character string or the divided character string in a first external database. How to split words. The input means accepts the input of the character string, obtains the input character string,
The dividing means divides the input character string into two parts between all characters of the input character string, and the first half character string that is the first half character string after the division and the second half character string that is the second half character string after the division A divided character string consisting of is obtained by a number one less than the number of characters of the input character string,
The acquisition means indicates a first frequency that is a number indicating the frequency of appearance of the input character string, a second frequency that is a number indicating the frequency of appearance of the first half character string, and a frequency of occurrence of the second character string. Get a third frequency that is a number,
The first determination means determines the ratio of the plurality of divided character strings based on a ratio between the first frequency value and the smaller one of the second frequency value and the third frequency value. Is determined as the optimal divided character string.
At least one of the optimal first half character string that is the first half character string included in the optimal divided character string and the optimal second half character string that is the second half character string included in the optimal divided character string is further divided by the second determination unit. When the stop condition indicating no condition is satisfied, a character string satisfying the stop condition is determined as a basic word among the optimal first half character string and the optimal second half character string,
The acquisition means stores a plurality of character strings and stores, for each character string, the first frequency, the second frequency, and the second frequency from a second database that stores the character string and the frequency of the character string in association with each other. A word division method characterized by acquiring three frequencies. The input means accepts the input of the character string, obtains the input character string,
The dividing means divides the input character string into two parts between all characters of the input character string, and the first half character string that is the first half character string after the division and the second half character string that is the second half character string after the division A divided character string consisting of is obtained by a number one less than the number of characters of the input character string,
The acquisition means indicates a first frequency that is a number indicating the frequency of appearance of the input character string, a second frequency that is a number indicating the frequency of appearance of the first half character string, and a frequency of occurrence of the second character string. Get a third frequency that is a number,
The first determination means determines the ratio of the plurality of divided character strings based on a ratio between the first frequency value and the smaller one of the second frequency value and the third frequency value. Is determined as the optimal divided character string.
At least one of the optimal first half character string that is the first half character string included in the optimal divided character string and the optimal second half character string that is the second half character string included in the optimal divided character string is further divided by the second determination unit. When the stop condition indicating no condition is satisfied, a character string satisfying the stop condition is determined as a basic word among the optimal first half character string and the optimal second half character string,
The acquisition means acquires the existing document data and measures the frequency at which the input character string, the first half character string, and the second half character string appear in the document data, thereby obtaining the first frequency, The word dividing method, wherein the second frequency and the third frequency are acquired.

Images