JP7378680B2 - Information processing device, update method, and update program - Google Patents

Description

The present disclosure relates to an information processing device, an update method, and an update program.

word2vec is known. word2vec is an unsupervised learning method that can learn the semantic features of words using a corpus (i.e., unlabeled sentences). The word vectors of learned words are called distributed representations. The word vectors of the learned words can be used in document searches and the like.

On the other hand, a method is known that uses external information such as knowledge of relationships between words and knowledge of word attributes as training information to improve the accuracy of distributed representation. For example, a method has been proposed for learning the relationships between words while preserving the semantic information of the words by using external information for the relationships between words and category information (Non-Patent Document 1). ).

Chang Xu et al. “RC-NET: A General Framework for Incorporating Knowledge into Word Representations”, 2014 Tomas Mikolov et al. "Distributed Representations of Words and Phrases and their Compositionality"

In Non-Patent Document 1, the word vector of the target word is updated by adding the same weight to all words in the same category. However, adding the same weight to words that appear less frequently has a negative effect on the distributed representation of updated word vectors. Therefore, the method of Non-Patent Document 1 cannot be said to be desirable.

The purpose of this disclosure is to improve distributed representation.

An information processing device according to one aspect of the present disclosure is provided. The information processing device includes an acquisition unit that acquires a preprocessed corpus that is a corpus that has undergone preprocessing, and a category dictionary that is information that indicates a correspondence relationship between words and categories; a category determination unit that determines the category of a plurality of words included in the processing corpus, and a word vector of the target word based on a target word that is one word among the plurality of words; a word vector creation unit that creates a word vector of the same category word based on the same category word that is one of the words of the target word and is a word of the same category as the target word; , an appearance frequency calculation unit that calculates the appearance frequency of the target word in the preprocessing corpus and the appearance frequency of the same category word in the preprocessing corpus; a regularization term calculation unit that calculates a regularization term using a word vector, an appearance frequency of the target word, and an appearance frequency of the same category word; and a regularization term calculation unit that calculates a regularization term using the regularization term. and an update unit that updates.

According to the present disclosure, distributed representation can be improved.

FIG. 2 is a block diagram showing functions of the information processing device according to the first embodiment. FIG. 2 is a diagram showing hardware included in the information processing apparatus according to the first embodiment. 3 is a flowchart illustrating an example of processing executed by the information processing apparatus according to the first embodiment. FIG. 3 is a diagram illustrating a semantic space according to the first embodiment. FIG. 2 is a block diagram showing functions of an information processing device according to a second embodiment. 12 is a flowchart illustrating an example of processing executed by the information processing apparatus according to the second embodiment. FIG. 7 is a diagram showing a specific example of displaying similar synonyms and selecting similar synonyms according to the second embodiment.

Embodiments will be described below with reference to the drawings. The following embodiments are merely examples, and various modifications can be made within the scope of the present disclosure.

Embodiment 1.
FIG. 1 is a block diagram showing the functions of the information processing apparatus according to the first embodiment. The information processing device 100 is a device that executes an update method. For example, the information processing device 100 is a personal computer, a server, a smartphone, or a tablet device. First, the hardware included in the information processing device 100 will be explained.

FIG. 2 is a diagram showing hardware included in the information processing apparatus according to the first embodiment. The information processing device 100 includes a processor 101, a volatile storage device 102, a nonvolatile storage device 103, a network IF (Interface) 104, an input IF 105, and a display IF 106.

Processor 101 controls the entire information processing device 100 . For example, the processor 101 may be a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field d Programmable Gate Array). Processor 101 may be a multiprocessor. Further, the information processing device 100 may include a processing circuit. Furthermore, the processor 101 may be a microcomputer or a SoC (System on Chip).

The volatile storage device 102 is the main storage device of the information processing device 100. For example, the volatile storage device 102 is a RAM (Random Access Memory). The nonvolatile storage device 103 is an auxiliary storage device of the information processing device 100. For example, the nonvolatile storage device 103 is a ROM (Read Only Memory), an HDD (Hard Disk Drive), or an SSD (Solid State Drive).

Network IF 104 communicates with network 10 . Note that the network 10 is a wired network or a wireless network.
The input IF 105 receives information or signals from a keyboard, touch panel, mouse, etc. Note that the information processing apparatus 100 does not need to have the input IF 105.
Display IF 106 outputs information to the display. Note that the information processing device 100 does not need to have the display IF 106.

Returning to FIG. 1, the functions of the information processing apparatus 100 will be explained.
The information processing device 100 includes a storage unit 110, an acquisition unit 120, a preprocessing unit 130, a category determination unit 140, a word vector creation unit 150, an appearance frequency calculation unit 160, a regularization term calculation unit 170, and an update unit 180.

The storage unit 110 may be realized as a storage area secured in the volatile storage device 102 or the nonvolatile storage device 103.
Part or all of the acquisition unit 120, preprocessing unit 130, category determination unit 140, word vector creation unit 150, appearance frequency calculation unit 160, regularization term calculation unit 170, and update unit 180 are realized by a processing circuit. Good too. In addition, some or all of the acquisition unit 120, preprocessing unit 130, category determination unit 140, word vector creation unit 150, appearance frequency calculation unit 160, regularization term calculation unit 170, and update unit 180 are executed by the processor 101. It may also be realized as a program module. For example, the program executed by processor 101 is also referred to as an update program. For example, the update program is recorded on a recording medium such as a CD or a flash memory. Further, the update program may be stored in the storage unit 110. The update program may be obtained via the network 10.

The storage unit 110 stores a corpus 111 and a category dictionary 112. The corpus 111 may also be called learning data. Furthermore, the corpus 111 may be considered as a database in which sentences are registered. The category dictionary 112 is information indicating the correspondence between words of nouns or noun phrases and categories. Note that the category may be an expression of a superordinate concept of a noun or a noun phrase, a product category, or a class name of a named entity. Further, for example, the class name is a person's name, place name, etc.

The acquisition unit 120 acquires the corpus 111 and the category dictionary 112 from the storage unit 110. Further, the acquisition unit 120 may acquire the corpus 111 and the category dictionary 112 from an external device. Illustrations of external devices are omitted.

The preprocessing unit 130 preprocesses the corpus 111. For example, the preprocessing unit 130 performs morphological analysis and word normalization. Here, the storage unit 110 may store a corpus that has been subjected to preprocessing. If the preprocessed corpus is stored in the storage unit 110, the acquisition unit 120 acquires the preprocessed corpus. Further, when a preprocessed corpus is stored in the storage unit 110, the information processing device 100 does not include the preprocessing unit 130. A corpus that has been preprocessed is called a preprocessed corpus. Further, the acquisition unit 120 may acquire the preprocessed corpus from an external device.

The category determining unit 140 uses the category dictionary 112 to determine the categories of a plurality of words included in the preprocessing corpus. In detail, the category determining unit 140 uses the category dictionary 112 to determine the category of a word, a noun or a noun phrase, included in the preprocessing corpus.
Detailed functions of the word vector creation section 150, appearance frequency calculation section 160, regularization term calculation section 170, and update section 180 will be explained later.

Next, the processing executed by the information processing apparatus 100 will be explained using a flowchart.
FIG. 3 is a flowchart illustrating an example of processing executed by the information processing apparatus according to the first embodiment.
(Step S11) The preprocessing unit 130 determines whether a processing request has been accepted. If the processing request is accepted, the process proceeds to step S12. If no processing request has been received, the preprocessing unit 130 waits.
(Step S12) The preprocessing unit 130 performs preprocessing on the corpus 111 including the target word.

(Step S13) The category determination unit 140 uses the category dictionary 112 to determine the category of the noun or noun phrase word included in the preprocessing corpus.
(Step S14) The word vector creation unit 150 creates a word vector of the target word based on the target word, which is one of the plural words included in the preprocessing corpus. For example, the word vector creation unit 150 creates a word vector of the target word using the target word and word2vec. Further, the word vector creation unit 150 creates a word vector of the same category word based on the same category word that is one of the plurality of words and is a word in the same category as the target word.

(Step S15) The appearance frequency calculation unit 160 calculates the appearance frequency f(w) of the target word in the preprocessing corpus as a weight, based on the preprocessing corpus. Furthermore, the appearance frequency calculation unit 160 calculates the appearance frequency f(w _t′ ) of the same category words in the preprocessed corpus as a weight, based on the preprocessed corpus.

(Step S16) The regularization term calculation unit 170 uses the word vector of the target word, the word vector of the same category word, the frequency of appearance f(w) of the target word, and the frequency of appearance f(w _t′ ) of the same category word. Then, the regularization term E(w _t , w _t′ ) is calculated. In other words, the regularization term calculation unit 170 uses the word vector of the target word, the word vector of the same category word, the frequency of appearance f(w) of the target word, and the frequency of appearance f(w _t′ ) of the same category word. , which is a distance in the semantic space, and calculates a regularization term E(w _t , w _t' ) based on the distance according to the frequency of appearance. Specifically, the regularization term calculating unit 170 calculates the regularization term E(w _t , w _t′ ) using equation (1).

Note that V indicates a vocabulary set of the corpus 111. w _t is a word vector of the target word. w _t' is a word vector of words in the same category. d(w _t , w _t' ) is the distance between the word vector of the target word and the word vector of the same category word. Note that Euclidean distance is used for the distance. For the distance, the reciprocal of cosine similarity (cos similarity) or "1-cos similarity" may be used.

Here, distance in the semantic space will be explained.
FIG. 4 is a diagram illustrating the semantic space of the first embodiment. In FIG. 4, the semantic space is represented in two dimensions. FIG. 4 shows a target word 20, a high-frequency word 21 that is a word that appears frequently, and a low-frequency word 22 that is a word that appears less frequently. The target word 20, the high frequency word 21, and the low frequency word 22 are words in the same category.

FIG. 4 shows trust regions 23 and 24. The word vectors of the high-frequency words 21 tend to have small variance. Therefore, the trust region 23 is small. The word vectors of the low frequency words 22 tend to have a large variance. Therefore, the trust region 24 is large. Further, FIG. 4 shows words 31 and 32 in a different category from the target word 20.

When the appearance frequency f(w _t′ ) of words in the same category is low, the regularization term calculation unit 170 calculates a regularization term E(w _t , w _{t )} that reduces the influence of the low frequency words 22 on the target word 20. _′ ) is calculated. Furthermore, when the appearance frequency f(w _t′ ) of words in the same category is high, the regularization term calculation unit 170 calculates a regularization term E(w _t , w _t' ) is calculated.

(Step S17) The updating unit 180 updates the word vector w _t of the target word using the regularization term E(w _t , w _t' ). In detail, the update unit 180 updates the word vector w _t of the target word based on an objective function using the regularization term E (w _t , w _t′ ) and the objective function of skip-gram negative sampling. . Specifically, the updating unit 180 updates the word vector w _t of the target word using equation (2). Note that the first term and the second term in the summation symbol in equation (2) are the same as the objective function of skip-gram negative sampling described in Non-Patent Document 2. Therefore, description of similar parts will be omitted.

Note that J indicates an objective function. σ(x) (=1/(1+exp(-x))) indicates a sigmoid function. k indicates the number of pseudo negative examples of Skip-gram negative sampling. Pn indicates the sampling distribution of pseudo-negative examples. Note that Pn is usually a unigram distribution or a distribution obtained by raising a unigram to the 0.75th power. w _n indicates a word vector of a pseudo-negative example. w _c indicates a word vector of words co-occurring with w _t . Note that the co-occurring words are words that exist within N characters before and after the target word. Further, N is a predetermined integer.

(Step S18) The update unit 180 determines whether the termination requirements are satisfied. Note that, for example, the termination requirement is that the word vector w _t of the target word no longer fluctuates. Further, for example, the termination requirement is that the number of times steps S16 and 17 are executed exceeds a predetermined threshold.
If the termination requirements are not met, the process proceeds to step S16. If the termination requirements are met, the process ends. The updating unit 180 stores the word vector w _t of the target word in the storage unit 110.
Here, repeating steps S16 to S18 may be expressed as learning.

According to the first embodiment, the information processing apparatus 100 adds the same weight to all words in the same category and does not update the word vector of the target word. The information processing device 100 calculates a regularization term according to the frequency of appearance, and uses the regularization term to update the word vector of the target word. For example, when the frequency of appearance of words in the same category is low, the information processing device 100 uses a regularization term to reduce the influence of words that appear less frequently on the target word. Therefore, the information processing device 100 can improve the distributed representation of the word vector of the target word.

Embodiment 2.
Next, a second embodiment will be described. In the second embodiment, matters that are different from the first embodiment will be mainly explained. In the second embodiment, explanations of matters common to the first embodiment will be omitted.

FIG. 5 is a block diagram showing the functions of the information processing device according to the second embodiment. Components in FIG. 5 that are the same as those shown in FIG. 1 are designated by the same reference numerals as those shown in FIG. Further, in FIG. 5, illustration of the preprocessing unit 130, the category determining unit 140, the word vector creating unit 150, the appearance frequency calculating unit 160, the regularization term calculating unit 170, and the updating unit 180 is omitted.

The information processing device 100 further includes a display section 191 , a determining section 192 , a determining section 193 , and a relearning section 194 .
A part or all of the display section 191, the determination section 192, the determination section 193, and the relearning section 194 may be realized by a processing circuit. Further, a part or all of the display section 191, the determination section 192, the determination section 193, and the relearning section 194 may be realized as a module of a program executed by the processor 101.

The storage unit 110 stores a database 113. The database 113 may also be referred to as a trained model. The database 113 shows the correspondence between a plurality of words included in the corpus 111 and a plurality of word vectors corresponding to the plurality of words. Note that each of the plurality of word vectors is a word vector updated in the first embodiment.
The storage unit 110 also stores relearning target information 114. The relearning target information 114 will be explained later.

The functions of the display section 191, determination section 192, determination section 193, and relearning section 194 will be explained later.

Next, the processing executed by the information processing apparatus 100 will be explained using a flowchart.
FIG. 6 is a flowchart illustrating an example of processing executed by the information processing apparatus according to the second embodiment.
(Step S21) The display unit 191 displays a search user interface screen. For example, the display unit 191 displays a search user interface screen on the display.
(Step S22) The acquisition unit 120 determines whether a search keyword has been input via the search user interface screen. When a search keyword is input by the user, the acquisition unit 120 acquires the search keyword input by the user. The process then proceeds to step S23. If a search keyword has not been input, the acquisition unit 120 waits until a search keyword is input.

(Step S23) The determining unit 192 uses the database 113 to determine synonyms of the search keyword. Specifically, the determining unit 192 detects a word vector similar to the word vector of the search keyword from the database 113, and determines a word corresponding to the detected word vector as a synonym of the search keyword. Note that, for example, whether or not they are similar is determined using a threshold value.

(Step S24) The display unit 191 displays synonyms of the search keyword. Note that the display of synonyms for a search keyword is also called a suggestion.
(Step S25) The determination unit 193 determines whether the user has adopted the synonym.

Here, the display of synonyms and the selection of synonyms will be specifically illustrated.
FIG. 7 is a diagram showing a specific example of displaying similar synonyms and selecting similar synonyms according to the second embodiment. FIG. 7 shows that "hogehoge" has been input as a search keyword. The display unit 191 displays a synonym of the search keyword "hogehoge", "fugafuga". When the user selects the synonym "Fugafuga", the display unit 191 displays the search keyword "Hogehoge" and the synonym "Fugafuga". If the user does not select the synonym "Fugafuga", the display unit 191 displays the search keyword "Hogehoge".
In this way, the user looks at the display screen and determines whether or not to adopt synonyms.

If the user adopts synonyms, the process ends. If the user does not select synonyms, the process proceeds to step S26.

Here, if the user does not adopt the similar synonym, this means that the similar synonym is not a synonym of the search keyword for the user. That is, this means that the similar synonym should not exist near the search keyword in the semantic space. Therefore, the relearning unit 194 registers the search keyword and the displayed synonyms in the relearning target information 114 as relearning target words.

(Step S26) The determination unit 193 determines whether the requirements for relearning are satisfied. For example, a requirement for relearning is that the number of words registered in the relearning target information 114 exceeds a threshold value. Further, the determination unit 193 may determine that the requirements for relearning are satisfied at the timing when two words are registered in the relearning target information 114.
If the relearning requirements are met, the process proceeds to step S27. If the relearning requirements are not met, processing ends.

(Step S27) The acquisition unit 120 acquires update information of the category dictionary 112. For example, the update information may be created by a user.
(Step S28) The relearning unit 194 updates the category dictionary 112 based on the updated information. The reason for updating the category dictionary 112 is to update the word vectors of the search keyword and the word vectors of synonyms to appropriate word vectors during relearning described later.

(Step S29) The relearning unit 194 uses the updated category dictionary 112 and the relearning target information 114 to execute a process for updating the word vector of the search keyword and the word vector of the synonym. In detail, the relearning unit 194 performs steps S12 to S18 by regarding the plurality of words registered in the relearning target information 114 as the corpus 111 or the preprocessing corpus. Thereby, each of the plurality of words registered in the relearning target information 114 becomes a target word. Then, each word vector of the plurality of words is updated.
The relearning unit 194 registers a plurality of words registered in the relearning target information 114 and a plurality of updated word vectors in the database 113.

According to the second embodiment, the information processing device 100 can display synonyms desired by the user by relearning the plurality of words registered in the relearning target information 114.

The features of each embodiment described above can be combined with each other as appropriate.

10 network, 20 target word, 21 high frequency word, 22 low frequency word, 23 trust region, 24 trust region, 31, 32 word, 100 information processing device, 101 processor, 102 volatile storage device, 103 nonvolatile storage device, 110 storage unit, 111 corpus, 112 category dictionary, 113 database, 114 relearning target information, 120 acquisition unit, 130 preprocessing unit, 140 category determination unit, 150 word vector creation unit, 160 appearance frequency calculation unit, 170 regularization term calculation unit, 180 update unit, 191 display unit, 192 determination unit, 193 determination unit, 194 relearning unit.

Claims (5)

an acquisition unit that acquires a preprocessed corpus that is a preprocessed corpus and a category dictionary that is information indicating a correspondence between words and categories;
a category determination unit that determines categories of a plurality of words included in the preprocessing corpus using the category dictionary;
A word vector of the target word is created based on the target word that is one of the plurality of words, and a word that is one of the plurality of words and is in the same category as the target word. a word vector creation unit that creates a word vector of the same category word based on the same category word;
an appearance frequency calculation unit that calculates the appearance frequency of the target word in the preprocessing corpus and the appearance frequency of the same category word in the preprocessing corpus, based on the preprocessing corpus;
a regularization term calculation unit that calculates a regularization term using the word vector of the target word, the word vector of the same category word, the frequency of appearance of the target word, and the frequency of appearance of the same category word;
an updating unit that updates a word vector of the target word using the regularization term;
An information processing device having: further comprising a pre-processing section,
The acquisition unit acquires a corpus,
The preprocessing unit preprocesses the corpus,
the preprocessing corpus is a corpus preprocessed by the preprocessing unit;
The information processing device according to claim 1. storage section and
A decision section,
A display section;
A determination section;
Re-study club and
It further has
The storage unit stores a database indicating correspondence between the plurality of words and a plurality of word vectors corresponding to the plurality of words,
One word vector among the plurality of word vectors is an updated word vector,
The acquisition unit acquires a search keyword input by a user,
The determining unit determines synonyms of the search keyword using the database,
The display unit displays the synonyms,
The determination unit determines whether the user has adopted the synonym,
If the user does not adopt the synonym, the relearning unit registers the search keyword and the synonym as a relearning target word in relearning target information,
The acquisition unit acquires update information of the category dictionary when the user does not adopt the synonym,
When the update information is acquired, the relearning unit updates the category dictionary based on the update information, and uses the updated category dictionary and the relearning target information to search for the search keyword. executing a process for updating the word vector of and the word vector of the synonym;
The information processing device according to claim 1 or 2. The information processing device
Obtain a preprocessed corpus, which is a corpus that has been preprocessed, and a category dictionary, which is information indicating the correspondence between words and categories,
determining categories of a plurality of words included in the preprocessing corpus using the category dictionary;
A word vector of the target word is created based on the target word that is one of the plurality of words, and a word that is one of the plurality of words and is in the same category as the target word. Create a word vector of the same category words based on the same category words,
Based on the preprocessing corpus, calculating the frequency of appearance of the target word in the preprocessing corpus and the frequency of appearance of the same category word in the preprocessing corpus,
Calculating a regularization term using the word vector of the target word, the word vector of the same category word, the frequency of appearance of the target word, and the frequency of appearance of the same category word,
updating a word vector of the target word using the regularization term;
How to update. In the information processing device,
Obtain a preprocessed corpus, which is a corpus that has been preprocessed, and a category dictionary, which is information indicating the correspondence between words and categories,
determining categories of a plurality of words included in the preprocessing corpus using the category dictionary;
A word vector of the target word is created based on the target word that is one of the plurality of words, and a word that is one of the plurality of words and is in the same category as the target word. Create a word vector of the same category words based on the same category words,
Based on the preprocessing corpus, calculating the frequency of appearance of the target word in the preprocessing corpus and the frequency of appearance of the same category word in the preprocessing corpus,
Calculating a regularization term using the word vector of the target word, the word vector of the same category word, the frequency of appearance of the target word, and the frequency of appearance of the same category word,
updating a word vector of the target word using the regularization term;
An update program that performs the process.

Images