JP7555701B2 - Text processing apparatus, method, device and computer-readable recording medium - Google Patents

Description

The present disclosure relates to the field of text processing, and more specifically to a text processing apparatus, method, device, and computer-readable recording medium.

In conventional text processing, the final text processing result can be obtained by processing the input source text in processes such as text conversion and text generation.

In some cases, the user can specify some external information for the text processing process to obtain more ideal results. Such external information may be important information in the text specified by the user, or other text information associated with the source text. To increase the possibility that such external information will appear in the text processing results, a text processing method is required that fully takes the external information into account in the text processing process.

To fully take external information into account in the text processing process, the present disclosure provides a text processing method, apparatus, device, and computer-readable recording medium.

According to one aspect of the present disclosure, there is provided a text processing apparatus including an encoding unit configured to encode a source text to obtain a source text encoding hidden state, a decoding unit configured to determine a decoded hidden state, and an output unit configured to determine an output word probability distribution based on external information, the source text encoding hidden state, and the decoded hidden state to determine an output word.

In some embodiments, the output unit is configured to determine, based on the external information, among the candidate output words, a word whose probability is equal to or greater than an output probability threshold and is included in the external information as a candidate output word for a current time step.

In some embodiments, the output unit is further configured to determine candidate probabilities of the candidate output words based on the joint probability between the candidate output word of the current time step and the candidate sequence determined in the immediately preceding time step, and the similarity between the candidate sequence determined in the immediately preceding time step and the external information, and to determine a predetermined number of candidate output words with the highest candidate probabilities as output words.

In some embodiments, the encoding unit is further configured to encode the extrinsic information to obtain an extrinsic information encoded hidden state, and the output unit is configured to determine a similarity between the extrinsic information encoded hidden state and the decoded hidden state, and output the extrinsic information as an output word if the similarity is greater than or equal to a similarity threshold for the current time step.

In some embodiments, the output unit is further configured to determine, if the similarity is less than the current similarity threshold, the word with the highest probability in the output word probability distribution as the output word for the current time step, and to adjust the similarity threshold for the current time step to determine an adjusted similarity threshold that is less than the similarity threshold for the current time step and is to be used as the similarity threshold for the next time step.

In some embodiments, the text processing device further includes an attention generation unit configured to determine an attention distribution for a current time step based on the external information, the source text encoding hidden state, and the decoded hidden state, and the output unit is configured to determine an output word probability distribution based on the attention distribution, the source text encoding hidden state, and the decoded hidden state to determine an output word.

In some embodiments, the encoding unit and the decoding unit are trained by the steps of: encoding a training source text to obtain a training source text encoding hidden state; determining a training decoding hidden state; determining an output word for a current time step based on the external information, the training source text encoding hidden state, and the training decoding hidden state; and adjusting parameters in the encoding unit and the decoding unit so that the difference between the training output word and the word contained in the external information is minimized.

According to another aspect of the present disclosure, there is further provided a text processing method including the steps of encoding a source text to obtain a source text encoding hidden state, determining a decoded hidden state, and determining an output word probability distribution based on external information, the source text encoding hidden state, and the decoded hidden state to determine output words.

In some embodiments, the method further comprises determining, based on the external information, among the candidate output words, a word whose probability is equal to or greater than an output probability threshold and is included in the external information as a candidate output word for a current time step.

In some embodiments, the method further includes a step of determining candidate probabilities of the candidate output words based on the joint probability of the candidate output words of the current time step and the candidate sequence determined in the immediately preceding time step, and the similarity between the candidate sequence determined in the immediately preceding time step and the external information, and determining as output words a predetermined number of candidate output words with the highest candidate probabilities.

In some embodiments, the method further comprises the steps of encoding the extrinsic information to obtain an extrinsic information encoded hidden state, determining a similarity between the extrinsic information encoded hidden state and the decoded hidden state, and outputting the extrinsic information as an output word if the similarity is greater than or equal to a similarity threshold for the current time step.

In some embodiments, the method further includes determining the most probable word in the output word probability distribution as the output word for the current time step if the similarity is less than the current similarity threshold, and adjusting the similarity threshold for the current time step to determine an adjusted similarity threshold to be used as the similarity threshold for the next time step, the adjusted similarity threshold being less than the similarity threshold for the current time step.

In some embodiments, the method further includes determining an attention distribution for a current time step based on the external information, the source text encoding hidden state, and the decoded hidden state, and determining an output word probability distribution based on the external information, the source text encoding hidden state, and the decoded hidden state to determine an output word includes determining an output word probability distribution based on the attention distribution, the source text encoding hidden state, and the decoded hidden state to determine an output word.

According to yet another aspect of the present disclosure, there is provided a text processing device including a processor and a memory storing computer-readable programming commands, the text processing device performing the above-described text processing method when the computer-readable programming commands are executed by the processor.

According to yet another aspect of the present disclosure, there is provided a computer-readable recording medium having recorded thereon computer-readable commands that, when executed by a computer, cause the computer to perform the text processing method as described above.

By utilizing the text processing method, apparatus, device, and computer-readable recording medium disclosed herein, in the text generation process, the attention distribution for the current time step is determined using external information, and/or the output words for the current time step are determined based on the external information, thereby effectively taking the content of the external information into account in the text processing process, improving the probability of generating the external information in the text generation process, and improving the effectiveness of generating text when the external information is taken into account.

The embodiments of the present disclosure will be described in more detail with reference to the drawings. The above and other objects, features, and advantages of the present disclosure will become more apparent. The drawings are intended to provide a deeper understanding of the embodiments of the present disclosure, and are used to interpret the present disclosure together with the embodiments of the present disclosure as part of the specification, and are not intended to limit the present disclosure. In the drawings, the same reference numerals generally indicate the same parts or steps.

1 shows a schematic block diagram of a text processing device according to the present disclosure; 1 illustrates a schematic embodiment of determining candidate output words based on output probability distributions according to an embodiment of the present application; 1 illustrates a schematic embodiment of determining candidate output words based on output probability distributions according to an embodiment of the present application; 1 shows a schematic block diagram of an attention generation unit according to an embodiment of the present disclosure. 1 illustrates a schematic process of an attention generation unit according to an embodiment of the present application determining an attention distribution for a current time step. FIG. 2 shows another schematic block diagram of an attention generation unit according to an embodiment of the present application. FIG. 2 shows another schematic block diagram of a text processing apparatus according to an embodiment of the present application. 1 shows a schematic flow chart of a text processing method according to the present application; 1 illustrates a schematic flow chart of determining attention distribution for a current timestep based on external information according to an embodiment of the present application; 13 illustrates another schematic flowchart of determining attention distribution for a current timestep based on external information according to an embodiment of the present application. 4 shows a schematic flowchart of another text processing method according to an embodiment of the present application; 4 shows an exemplary flowchart of another text processing method according to an embodiment of the present application. FIG. 2 is a schematic diagram of a computing device according to an embodiment of the present disclosure.

The technical solutions of the embodiments of the present disclosure will be described below clearly and comprehensively with reference to the drawings in the embodiments of the present disclosure. It is clear that the described embodiments are only some of the embodiments of the present disclosure, and not all of the embodiments. All other embodiments obtained by a person skilled in the art based on the embodiments of the present disclosure without creative labor are included in the scope of protection of the present disclosure.

Unless otherwise defined, technical or scientific terms used herein have the ordinary meaning that one skilled in the art would understand. The words "first", "second" and similar words used herein do not indicate any order, number or importance, but are merely used to distinguish different components. Similarly, words such as "comprise" and "include" and similar words refer to the part or thing that appears before the word including the part or thing listed after the word and its equivalents, and do not exclude other parts or things. Words such as "connect" and "connect to each other" and similar words are not limited to physical or mechanical connections, but also include electrical connections, whether direct or indirect. Words such as "upper", "lower", "left", "right" and the like are used merely to indicate relative positional relationships, and after the absolute position of the described object is changed, the relative positions may be changed accordingly.

Below, the principles of the present disclosure are described using the generation of a text summary as an example. However, those skilled in the art will appreciate that the method provided by the present disclosure may be applied to other text processing processes, such as, for example, a text conversion process, a machine translation process, etc., without departing from the principles of the present disclosure.

FIG. 1 shows a schematic block diagram of a text processing device according to the present disclosure. As shown in FIG. 1, the text processing device 100 may include an encoding unit 110, a decoding unit 120, an attention generating unit 130, and an output unit 140. The text processing device 100 may perform text processing on a source text I to generate a corresponding text processing result. For example, the text processing device 100 may be used to generate a summary for the source text I. The source text I may include at least one sentence, and each sentence includes at least one word.

The text processing device 100 provided by the present disclosure can receive external information and perform a text processing process on the source text based on the external information. In some embodiments, the external information refers to predefined text information expected as a processing result of the source text. In some examples, the external information is at least one word or sentence in the source text. In some other examples, the external information is a word or sentence at a predetermined position in the source text, for example, the first sentence, the last sentence, or any other specified position of the source text. In some other examples, the external information is additional text associated with the source text. For example, the title of the source text. In some implementations, the external information may be additional text determined based on a user's input. The present application is not limited here to a method for determining the external information. In fact, the external information used in the text processing process may be determined in any possible manner.

When processing source text using the text processing device 100, the probability that the external information will appear in the text processing results can be increased by taking external information into consideration at each stage of text processing. For example, if the title of a sentence in the source text is determined as external information, words and/or sentences in the sentence title will definitely appear or are highly likely to appear in the summary of the source text output from the text processing device 100 provided by the present application.

When performing text processing methods on a computer, the computer often cannot process text data directly, so when processing source text and/or external information, the source text must first be converted into numerical data.

In some embodiments, the source text I is implemented in the form of a natural language. In this case, the text processing device 100 may further include a preprocessing unit (not shown). The preprocessing unit may be used to convert the source text into numerical data before the source text is input to the encoding unit. For example, a word segmentation process may be performed on each sentence in the source text I, thereby dividing each sentence into multiple words. Then, for example, the multiple words obtained by the word segmentation process may be converted into word vectors of a specific dimension using a word embedding method.

Similarly, the external information may be transformed to obtain at least one word vector corresponding to the external information for subsequent text processing.

In some embodiments, the source text I according to the present disclosure may be realized in the form of numerical data, for example, the source text I may be represented by at least one word vector. In this case, the source text I may be directly processed by the encoding unit 110. Preprocessing of the natural language may also be performed by a preprocessing device provided independently of the text processing device 100.

In the following, no distinction is made between whether the external information and the source text are in the form of natural language or in the form of numerical data, and when it is necessary to process the external information and/or the source text in the form of natural language using a computer, a person skilled in the art may convert the external information and/or the source text in the form of natural language into numerical data as necessary.

The encoding unit 110 may be configured to encode a source text I to be processed in order to obtain a source text encoding hidden state h.

In some embodiments, the encoding unit 110 may be implemented as an encoding network. An exemplary encoding network includes a long short-term memory (LSTM) network, and an LSTM network system may be applied to tasks such as machine translation, text summary generation, etc. Note that the encoding network may be implemented as any machine learning model capable of encoding word vectors.

For example, given at least one word vector corresponding to a source text I as input, the encoder may output source text encoded hidden states _h1 , _{h2, h3... corresponding to each word vector x1, x2 , x3 ... respectively .} The number of source text encoded hidden states and the number of source text word vectors may be the same or different. For example, to generate k word vectors based on a source text I, the encoder may process these k word vectors to generate k corresponding source text encoded hidden states, where k is an integer greater than 1.

The decoding unit 120 may be used to determine a decoded hidden state s. In some embodiments, the decoding unit 120 may receive a decoded hidden state s _t-1 of a previous time step t-1 and an output word x _t obtained by the text processing device in the previous time step, and process s _t-1 and x _t to obtain a decoded hidden state s _t of a current time step. In the processing of the first time step, s ₀ and x ₁ may be determined as default initial values. The decoded hidden state s may include multiple decoded hidden states s ₁ , s ₂ , s ₃ ... corresponding to the source text I.

In some embodiments, the decoder 120 may be implemented as a decoding network. Exemplary decoding networks include long-short-term memory networks. Note that the decoding network may be implemented as any machine learning model capable of decoding the output of the encoding network.

In some embodiments, the encoding and decoding networks may be represented as a sequence to sequence (Seq2Seq) model, which may be used to convert one input sequence, e.g., "WXYZ" (e.g., as input text), to another output sequence, e.g., "AXY" (e.g., as a text summary).

The attention generation unit 130 may be configured to determine an attention distribution A based on the source text encoding hidden state h and the decoded hidden state s, and to output the attention distribution A for a subsequent text processing process of the current time step.

In some implementations, the attention distribution A ^t at the current time step t may be the encoded attention distribution of the source text.

ｉは単語ベクトルのインデクス番号であり、ｈ_ｉは第ｉ個の単語ベクトルに対応するソーステキスト符号化隠れ状態であり、ｖ^Ｔ、Ｗ_ｈ、Ｗ_Ｓ、ｂ_ａｔｔｎはトレーニング対象である学習パラメータであり、ｈは現在タイムステップのソーステキスト符号化隠れ状態であり、ｓ_ｔは現在タイムステップの復号化隠れ状態である。 In some implementations, at each time step t, the coding attention distribution a t of the source text of the current time step may be determined using the source text coding hidden state h _t and the decoded hidden state s _t of the current time step. For example, the coding attention distribution ^{a t of} the source text may be determined using equations (1) and (2).
^at = softmax( ^et ) (1)
t denotes the current time step, softmax denotes the normalized exponential function, and e ^t is determined by equation (2) as follows:

i is the index number of the word vector, h _i is the source text encoding hidden state corresponding to the i-th word vector, v ^T , W _h , W _S , and b _attn are the learning parameters to be trained, h is the source text encoding hidden state of the current time step, and s _t is the decoding hidden state of the current time step.

In some other embodiments, the attention generation unit 130 may determine an attention distribution A ^t for the current time step including the external information based on the external information and the attention distribution a ^t of the source text determined by equation (1), and output the attention distribution A ^t including the external information for the processing process of each subsequent part.

In some implementations, the attention distribution A ^t for the current time step including external information may be determined after adjusting the attention distribution a ^t for the source text using the external information.

In some other implementations, the attention distribution A ^t of the current time step including external information may include both the attention distribution a ^t of the source text and the attention distribution a′ ^t of the external information.

The process of determining the attention distribution A ^t including external information will be described below with reference to FIGS. 3A, 3B, and 4, and will not be described in detail here.

The output unit 140 may be configured to determine an output word probability distribution based on the attention distribution A, the source text encoding hidden state h, and the decoding hidden state s to determine an output word O for the current time step.

Ｖ’、Ｖ、ｂ、ｂ’は、出力部におけるトレーニング対象である学習パラメータであり、ｈ_ｔ ^＊はアテンション分布ａ^ｔに基づいて決定したコンテキストベクトルである。例えば、式（４）によってｈ_ｔ ^＊を決定してもよい。

Ａ_ｉ ^ｔはアテンション生成部から出力されるアテンション分布Ａｔにおける第ｉ個要素であり、ｈ_ｉは第ｉ個の単語ベクトルのソーステキスト符号化隠れ状態である。 The output word probability distribution may include a generation probability distribution P _vocab . The generation probability distribution P _vocab may be determined according to equations (3) and (4).

V', V, b, and b' are learning parameters to be trained in the output unit, and h _t ^* is a context vector determined based on the attention distribution a ^t . For example, h _t ^* may be determined by equation (4).

A _i ^t is the i-th element in the attention distribution A t output from the attention generation unit, and h _i is the source text encoding hidden state of the i-th word vector.

In some implementations, the output word probability distribution may include the attention distribution A ^t output from the attention generator 130 .

For example, the output word probability distribution may be determined by weighting the generated probability distribution and the attention distribution A.

In some implementations, weighting factors Pgen for the generation probability distributions and attention distributions may be determined based on the source text encoded hidden state of the current time step, the decoded hidden state, the attention distribution and the _decoder input _xt of the current time step.

σは、例えばｓｉｇｍｏｉｄ関数のような活性化関数を示し、ｗ_ｈ ^Ｔ、ｗ_ｓ ^Ｔ、ｗ_ｘ ^Ｔ及びｂ_ｐｔｒは、トレーニングパラメータであり、ｈ_ｔ ^＊はタイムステップｔで式（４）によって決定されるパラメータであり、ｓ_ｔはタイムステップｔの復号化隠れ状態であり、ｘ_ｔはタイムステップｔの復号化部の入力であり、すなわち、直前のタイムステップｔ－１の出力部の出力である。式（５）によって決定される重み係数Ｐ_ｇｅｎは、スカラーの形によって実装されてもよい。重み係数Ｐ_ｇｅｎを利用して生成確率分布Ｐ_{ｖｏｃａｂ}とアテンション分布Ａ^ｔとに対して加重平均を行うことで出力単語確率分布を取得することができる。 For example, a weighting coefficient P _gen for performing weighted addition on the generated probability distribution and the attention distribution may be represented by equation (5).

σ denotes an activation function such as a sigmoid function, w _h ^T , w _s ^T , w _x ^T and b _ptr are training parameters, h _t ^* is a parameter determined by equation (4) at time step t, s _t is the decoded hidden state at time step t, and x _t is the input of the decoder at time step t, i.e., the output of the output unit at the previous time step t-1. The weighting coefficient P _gen determined by equation (5) may be implemented in the form of a scalar. The weighting coefficient P _gen can be used to perform a weighted average of the generation probability distribution P _vocab and the attention distribution A ^t to obtain the output word probability distribution.

When the attention distribution A ^t includes both the attention distribution a ^t of the source text and the attention distribution a′ ^t of the external information, the weighting coefficient parameters of the attention distribution a ^t of the source text and the attention distribution a′ ^t of the external information may be the same or different. Hereinafter, the method of determining the weighting coefficient parameters of the generation probability distribution P _vocab , the attention distribution a ^t of the source text, and the attention distribution a′ ^t of the external information will be described with reference to FIG. 4, and will not be described in detail here.

In some embodiments, the output unit 140 may determine the word with the highest probability in the output word probability distribution as the output word for the current time step.

In some other embodiments, the output unit 140 may determine and output a word probability distribution based on external information to determine the output word for the current time step.

In some implementations, the output unit 140 determines, based on the external information, among the candidate output words, a word whose probability is equal to or greater than an output probability threshold and is included in the external information as a candidate output word for the current time step. In some examples, the output unit 140 may determine the candidate output word by using a beam search principle.

For example, the output unit 140 may determine at least two words at each time step as candidate output words for the current time step, and then use the candidate output words in the text processing process for the next time step. Similarly, the output unit 140 may determine at least two candidate output words for the next time step.

Specifically, for example, taking the number of candidate output words as two, two candidate output words a and b can be output at time step t. Then, the candidate output words a and b can be used in the text processing process of the next time step to determine the candidate output words c and d at time step t+1.

2A and 2B show an example embodiment of determining candidate output words based on an output probability distribution according to an embodiment of the present application.

In some embodiments, when determining candidate output words for each time step, a predetermined number M (in the above example, M is 2) of words with the highest output probability in the output probability distribution are determined as candidate output words. M is an integer equal to or greater than 2.

The two words with the highest output probabilities in the output word probability distribution shown in Figure 2A are w3 and w11, so w3 and w11 are determined as candidate output words.

In some other embodiments, when determining candidate output words for each time step, N words with the highest output probability in the output probability distribution are selected and determined in a predefined manner, and M words out of these N words are determined as candidate output words. N is an integer greater than M. In some implementations, the value of N may be specified in advance.

In some other implementation methods, an output probability threshold may be determined in advance, and M words out of N words whose output probabilities are greater than the output probability threshold may be determined as candidate output words.

If there is no word included in the external information among the N words with the highest output probabilities, the M words with the highest output probabilities among these N words may be determined as candidate output words.

When there is a word included in the external information among the N words with the highest output probability, if the number n of words included in the external information that are present in these N words is M or more, the M words included in the external information that have the highest output probability among these N words are determined as candidate output words, and if the number n of words included in the external information that are present in the N words is smaller than a predetermined number M, the word included in the external information among the N words and the M-n words with the highest output probability among the remaining N-n words are determined as candidate output words.

As shown in FIG. 2B, the two highest words in the output word probability distribution are w3 and w11, and the words with output probabilities greater than the preset output probability threshold include w3, w7, and w11, and w2 and w7 are included in the external information.

In this case, since w7 is included in the external information and the output probability of w7 is greater than the output probability threshold, w7 and w11 are selected as candidate output words rather than selecting w3, which has a higher output probability.

In this way, the probability that a word in the external information will be determined as the output word can be increased.

At least four output candidate sequences ac, ad, bc, bd can be determined using the candidate output words a, b output at time step t and the candidate output words c, d at time step t+1, and the output probability of each output candidate sequence is determined using a joint probability method, and the two with the highest output probabilities among the four output sequence candidates ac, ad, bc, bd are determined as candidate texts after time step t+1.

For example, the output probabilities of candidate output words a, b, c, d can be denoted as _Pa , _Pb , _Pc , and _Pd , respectively. The output candidate sequences ac, ad, bc, bd can be denoted as _Pa = _Pa * _Pc , _Pa = _Pa * _Pd , _Pbc = _Pb * _Pc , and _Pbd = _Pb * _Pd, respectively. If _Pa > _{Pa > Pb > Pbd} , the output sequences ac, ad are used for subsequent text processing at time step t+1.

Ｐ（ｙ_ｔ｜ｘ）はタイムステップｔで単語ｘを出力する確率を示し、ｈは、外部情報を示し、ｓｉｍ（ｙ_＜ｔ，ｈ）はタイムステップｔの前に生成した候補テキスト系列と外部情報との間の類似度を示す。 In some embodiments, the output candidate sequence may be determined based on external information. For example, the penalty value of the output candidate sequence may be determined using Equation (6). The penalty value determined by Equation (6) may adjust the joint output probability of the output candidate sequence.

P(y _t |x) denotes the probability of outputting word x at time step t, h denotes the external information, and sim(y _{< t} , h) denotes the similarity between the candidate text sequence generated before time step t and the external information.

In one implementation, any possible text similarity algorithm may be used to determine the similarity between the candidate text sequences generated before time step t and the external information. For example, a cosine similarity method may be used to determine the similarity between the candidate text sequences generated before time step t and the external information.

Using the above formula (6), the higher the similarity between the candidate text sequence generated before time step t and the external information, the more the penalty value is used to increase the output probability of the output candidate sequence. In some implementations, the penalty value s(x,y) can be multiplied or added to the output probability in the output candidate sequence to achieve the effect of determining the output candidate sequence based on the similarity between the candidate text sequence generated before time step t and the external information.

That is, by determining the penalty value to be used for the output candidate sequence based on external information, the probability that the external information will appear in the candidate text sequence can be increased, thereby increasing the probability that the external information will appear in the final output text processing result.

In another implementation, the output unit may be further configured to determine a similarity between the external information and the source text coding hidden state, and to determine a word to be output at the current time step based on the similarity between the external information and the source text coding hidden state.

For example, the external information may be encoded using the encoding unit 110 to obtain the external information encoded hidden state.

The output unit 140 may be configured to determine a similarity between the extrinsic information encoded hidden state and the decoded hidden state. If the similarity between the extrinsic information encoded hidden state and the decoded hidden state is equal to or greater than a predefined similarity threshold, the output unit outputs the extrinsic information as the output for the current time step.

If the extrinsic information is a word, the extrinsic information may be output as the word of the current time step. If the extrinsic information is a sentence, the extrinsic information may be inserted directly after the text sequence generated before the current time step t.

Note that the text sequence already generated before the current time step t may be generated based on the most probable word in the output probability distribution, or may be generated based on some candidate output words with the most probabilities in the output probability distribution. The process described in the implementation method may be adopted to determine the candidate output words, and will not be described in detail here.

If the similarity between the external information encoded hidden state and the decoded hidden state is less than a predefined similarity threshold, the output unit determines an output word probability distribution for the current time step based on the results output from the decoding unit and the attention generation unit, and determines an output word for the current time step based on the output word probability distribution for the current time step.

Using the above method, if the similarity between the result output from the decoding unit and the external information is relatively high, the result output from the decoding unit may be directly replaced with the external information. That is, in this case, the result of the text sequence determined after the output of the current time step is the result obtained by inserting the external information after the text sequence determined after the output of the previous time step.

Then, when processing the next time step, the decoder can use the extrinsic information to encode the extrinsic information to obtain the decoded hidden state of the next time step, so that the subsequent decoding process can use the results of the extrinsic information, thereby ensuring semantic consistency between the results obtained in the subsequent decoding and the inserted extrinsic information.

When the external information is a word, the decoded hidden state of the previous time step and the external information are used as input to the decoding unit, and the decoded hidden state of the current time step can be obtained.

When the external information contains multiple words, the decoding unit performs several loop processes. The inputs to the decoding unit in the first loop are the decoded hidden state of the previous time step and the first word of the external information, and the inputs to the decoding unit in subsequent loops are the decoded hidden state obtained in the previous loop and the next word of the external information. By performing several loops, it is possible to process each word in the external information, and a decoded hidden state containing all the external information is obtained and used as the decoded hidden state for the current time step.

In some implementations, after the external information has already replaced the result output by the decoding unit and been inserted into the text processing result, the similarity comparison between the external information encoded hidden state and the decoded hidden state is not performed.

In some examples, the similarity threshold may be implemented as a function with respect to a given time step t.

As described above, if the similarity between the external information encoded hidden state and the decoded hidden state is smaller than a predefined similarity threshold, the operation of replacing the decoder output with the external information and outputting it is not performed, and the output result is determined based on the output word probability distribution. In this case, in order to increase the probability that the external information appears in the final text processing result, the similarity threshold of the current time step is adjusted to determine an adjusted similarity threshold, which is smaller than the similarity threshold of the current time step, and which is used as the similarity threshold of the next time step.

ε_{ＳＩＭ，ｔ＋１}はタイムステップｔ＋１に使用される類似度閾値であり、ε_{ＳＩＭ，ｔ}はタイムステップｔに使用される類似度閾値であり、ｆ（ｔ）は時間ｔに関する単調減少関数である。例えば、ｆ（ｔ）は式（８）のように実装されてもよい。

ｔは現在タイムステップであり、ｋは、ソーステキストの長さであり、ｅは自然対数である。一部の変更例において、ｋは、ソーステキスト長さに関する関数で表示されてもよい。例えば、ｋをβとソーステキスト長さの積で表示されてもよく、βは０より大きく１より小さい事前定義されたパラメータである。 For example, equation (7) is used to adjust the similarity threshold.

where ε _SIM,t+1 is the similarity threshold used for time step t+1, ε _SIM,t is the similarity threshold used for time step t, and f(t) is a monotonically decreasing function with respect to time t. For example, f(t) may be implemented as in equation (8).

t is the current time step, k is the length of the source text, and e is the natural logarithm. In some variants, k may be expressed as a function of the source text length. For example, k may be expressed as a product of β and the source text length, where β is a predefined parameter greater than 0 and less than 1.

By using the above method to perform a monotonically decreasing adjustment of the similarity threshold for each time step, the similarity between the external information and the output result of the decoding unit is reduced to a lower level in the text processing process, increasing the probability that the similarity between the external information and the output result of the decoding unit will be greater than the similarity threshold for the current time step. In other words, the probability that the external information will appear in the final text processing result increases.

The text processing device 100 provided by the present application uses external information to determine the attention distribution for the current time step during the text summary generation process, and/or determines the output words for the current time step based on the external information, thereby effectively taking the content of external information into account in the text processing process, increasing the probability of generating external information during the text generation process, and improving the effectiveness of text generation when external information is taken into account.

When implementing the text processing device 100 provided by the present application, a person skilled in the art may make any combination of the above technical solutions. For example, in the process of performing text processing on source text in the text processing device 100, only the attention generation unit may be used to generate an attention distribution including external information based on external information and used in subsequent text processing, and the external information may not be taken into account in the subsequent text processing process. For example, only the output unit may be used to determine the word to be output in the current time step based on external information, and the external information may not be taken into account in the previous encoding, decoding and attention generation processes. Furthermore, the possibility of including external information in the text processing result may be further increased by taking external information into account in the process of generating the attention distribution for the current time step in the attention generation unit and determining the word to be output in the current time step in the output unit.

3A shows a schematic block diagram of an attention generation unit according to an embodiment of the present disclosure. The attention generation unit 300 shown in FIG. 3A can be used to adjust the attention distribution a ^t of the source text based on external information and determine the attention distribution A′ including the external information.

As shown in FIG. 3A, the attention generation unit 300 includes a source text attention determination unit 310 and a content selection unit 320.

The source text attention determiner 310 is used to determine the coding attention distribution a ^t of the source text based on the source text coding hidden state and the decoded hidden state. In some embodiments, the coding attention distribution a ^t of the source text can be determined using Equation (1).

The content selector 320 may be used to determine a selection probability for each word in the source text. In some embodiments, the content selector 320 may be used to determine a selection probability distribution for the source text based on external information, the selection probability distribution including a selection probability for each word in the source text.

In some embodiments, the content selector 220 may use a content selection network (e.g., an LSTM network) to process the source text I and determine a first selection probability for each word in the source text.

A reference text processing result ref (i.e., a text processing result of a predetermined training data) is used to train the content selection network used. In the training process of the content selection network, a tagging sequence generated based on a source text I and the reference text processing result ref is input to the content selection network for processing. The tagging sequence and the word sequence of the source text I have the same length, and the value of the i-th element of the tagging sequence identifies whether the i-th word of the source text I is included in the reference text processing result ref. By using the above method to train the content selection network, the content selection network can process the source text I and output a first selection probability result for each word in the source text, where the first selection probability indicates the probability that this word in the source text I will be selected based on the content selection network and appear in the final text processing result.

In some embodiments, for at least one word included in the external information in the source text, a selection probability of the at least one word may be determined to be at least a predefined probability value λ, for example, a second selection probability of each word included in the external information in the source text is determined to be a predefined probability value λ, and a second selection probability of other words not included in the external information is determined to be 0.

A selection probability for each word in the source text may be determined based on the first selection probability and the second selection probability. For example, the selection probability for each word in the source text may be determined as the sum of the first selection probability and the second selection probability. As can be seen from the above, the selection probability for the words included in the external information is equal to or greater than a predefined probability value λ.

Based on the selection probability distribution, the content selection unit 220 is used to determine, for each word in the source text, the attention of the word according to the selection probability of the word, to obtain an attention distribution A. In some embodiments, the content selection unit 220 may be configured to determine the attention used for the word in the attention distribution of the current time step as zero if the selection probability of the word is lower than a predetermined selection probability threshold ε. Also, the content selection unit 220 may be configured to determine the attention used for the word in the attention distribution of the current time step as the attention of the word in the encoding attention distribution a ^t of the source text if the selection probability of the word is equal to or greater than a predetermined selection probability threshold ε.

By using the attention generation unit, a selection probability can be generated for each word in the source text; that is, when determining the attention for each word, at least both the magnitude of attention calculated using formula (1) and the selection probability of the word should be considered. If the selection probability of the word is lower than a predetermined selection threshold, it may be considered that the word has a low probability of appearing in the current time step, and therefore the attention of the word does not need to be considered in the subsequent text processing process.

はコンテンツ選択によって決定される現在タイムステップのアテンション分布であり、ｘは現在出力しようとする単語であり、ｊは現在タイムステップの番号であり、ｙ_{１：ｊ－１}は既に出力したテキスト系列、ｐ（ａ_ｊ ^ｉ｜ｘ，ｙ_{１：ｊ－１}）は、ソーステキストの符号化アテンション分布であり、例えばｐ（ａ_ｊ ^ｉ｜ｘ，ｙ_{１：ｊ－１}）は、前記式（１）、（２）によって計算された結果である。ｑは上記第一選択確率であり、λ＊ｈｉｎｔ＿ｔａｇは上記第二選択確率である。外部情報に含まれる第ｉ個の単語に対して、ｑ＋λｈｉｎｔ＿ｔａｇの第ｉ項の値はｑ_ｉ＋λであってもよく、外部情報に含まれていない第ｋ個の単語に対して、ｑ＋λｈｉｎｔ＿ｔａｇの第ｋ項の値はｑ_ｋであってもよい。 Equation (9) may represent the result of the attention distribution determined using the content selection unit.

is the attention distribution of the current time step determined by content selection, x is the word to be currently output, j is the number of the current time step, _y1:j-1 is the text sequence already output, p(a _j ⁱ |x, y _1:j-1 ) is the coding attention distribution of the source text, for example, p(a _j ⁱ |x, y _1:j-1 ) is the result calculated by the above formulas (1) and (2). q is the first selection probability, and λ*hint_tag is the second selection probability. For the i-th word included in the external information, the value of the i-th term of q+λhint_tag may be q _i +λ, and for the k-th word not included in the external information, the value of the k-th term of q+λhint_tag may be q _k .

By setting the selection probability of a word included in the external information to at least a predefined probability value λ, if the predefined probability value is greater than a predetermined selection probability threshold ε, the word in the external information can be prevented from being filtered out in the content selection step, ensuring that the word in the external information enters the subsequent text processing process, thereby increasing the probability that the word in the external information will appear in the text processing results. Note that in some implementations, the predefined probability value λ may be set to be equal to or less than the predetermined selection probability threshold ε. In this case, by determining the selection probability for each word to be the sum of the first selection probability and the second selection probability, the selection probability of the word in the external information can be increased, and the effect of increasing the probability that the word in the external information will appear in the text processing results can be implemented.

Figure 3B shows an example process for determining the attention distribution for the current time step using the attention generator shown in Figure 3A.

As shown in FIG. 3B, taking four words in the source text as an example, the content selection network is used to select the attention of the first and third words for use in the subsequent text processing process.

Fig. 4 shows another schematic block diagram of the attention generation unit of the embodiment of the present application. As shown in Fig. 4, the attention generation unit 400 may include a source text attention determination unit 410 and an external information attention determination unit 420. The attention distribution A determined by using the attention generation unit shown in Fig. 4 includes both the attention distribution a ^t of the source text and the attention distribution a' ^t of the external information.

In some implementations, the source text attention determiner 410 utilizes equation (2) to determine an encoded attention parameter e _i ^t for each word in the source text based on the source text encoded hidden state of the current time step and the decoded hidden state of the current time step.

The external information attention determination unit 420 is used to determine an external attention parameter for each word in the source text, and the external attention parameter of the word included in the external information is determined to be a predetermined first external attention parameter, and the external attention parameter of the word not included in the external information is determined to be a predetermined second external attention parameter. In some implementations, the first external attention parameter may be set as λ′, and the second external attention parameter may be set to 0, where λ′ is a value greater than 0.

Determine a word-by-word attention parameter to be used for the source text based on the coded attention parameter and the external attention parameter, for example, calculate the sum of the coded attention parameter for each word and the external attention parameter to obtain the attention parameter _e'i ^' of the word.

Then, the attention distribution of the current time step of the source text is determined based on the attention parameter _e'i ^t . For example, the attention of the current time step for each word used in the source text can be obtained by applying a softmax function to the attention parameter _e'i ^t .

Using the above method, the attention parameter of the words included in the external information in the source text can be adjusted with the predefined external attention parameter, and the adjustment for the attention of the words included in the external information can be implemented. Note that, when the first external attention parameter is a hyperparameter λ' that is greater than 0 and the second external attention parameter is set to 0, the attention distribution of each word in the source text can be adjusted based on the external information, so that the attention of the words included in the external information becomes more important.

In the above example, the principle of the present application is described by taking the first external attention parameter as λ' and the second external attention parameter as 0 as an example, but the scope of the present application is not limited thereto. Those skilled in the art may set the parameters of the external attention for each word used in the source text based on the actual situation, and only the effect of making the attention of the words included in the external information more important can be finally implemented. For example, the first external attention wp parameter is set as λ ₁ ', and the second external attention parameter is set as λ ₂ ', where λ ₁ ' and λ ₂ ' are any real numbers, and only λ ₁ '>λ ₂ ' is satisfied.

In some embodiments, the source text attention determiner 410 may be used to determine the coding attention distribution a ^t of the source text using the above formulas (1) and (2). The external information attention determiner 420 may be used to determine the coding attention distribution a′ ^t of the external information.

In this case, the extrinsic information is encoded using the encoding unit 110 shown in FIG. 1 to obtain the extrinsic information encoded hidden state h'. The extrinsic information attention unit 420 determines the extrinsic information encoding attention distribution based on the extrinsic information encoded hidden state h' and the decoded hidden state s.

For example, the above equations (1) and (2) are used to determine the extrinsic information coding attention distribution a′ ^t , and the source text coding hidden state h in equations (1) and (2) should be replaced with the extrinsic information coding hidden state h′.

In some implementations, the external information coding attention distribution ^a't and the source text coding attention distribution ^a't are calculated using the share parameter formulas (1) and (2), i.e., the parameters ^vT , _Wh , _Ws , and ^battn used to calculate the external information coding attention distribution ^a't and the source text coding attention distribution _at may be the same. In other implementations, the external information coding attention distribution ^a't and the source text coding attention distribution ^at may be trained on the training parameters used to calculate the external information coding attention distribution a't and the source text coding attention distribution at, i.e., the parameters ^vT , _Wh , _Ws , and ^battn used to calculate the external information coding attention distribution ^a't and the source _text coding attention distribution at may be different.

The attention generation unit 400 shown in Figure 4 generates an encoded attention distribution ^a't of external information and an encoded attention distribution at of source text, and the output unit shown in Figure 1 performs further processing on the encoded attention distribution ^a't of external information and the encoded attention distribution ^at of source text to determine the output word probability distribution for the current time step.

When the attention distribution A ^t includes both the attention distribution a ^t of the source text and the attention distribution a′ ^t of the external information, the output word probability distribution may be represented as a weighted average of the generation probability distribution, the attention distribution a ^t of the source text and the attention distribution a′ ^t of the external information.

生成確率分布Ｐ_{ｖｏｃａｂ}は、ソーステキストの符号化隠れ状態、復号化隠れ状態及びソーステキストの符号化アテンション分布に基づいて式（３）によって決定されるのであってもよく、

は、ソーステキストの符号化アテンション分布における第ｉ個の単語のアテンションを示し、

は、外部情報の符号化アテンション分布における第ｉ個の単語のアテンションを示している。Ｐ_{ｇｅｎｅｒａｔｏｒ}、Ｐ_{ｐｏｉｎｔｅｒ}、及びＰ_Ｔはそれぞれ生成確率分布Ｐ_{ｖｏｃａｂ}、ソーステキストの符号化アテンション分布ａ^ｔ及び外部情報の符号化アテンション分布ａ’^ｔ用の重み係数を示している。 In some implementations, the output word probability distribution may be determined by equation (10).

The generating probability distribution P _vocab may be determined based on the encoded hidden state of the source text, the decoded hidden state, and the encoded attention distribution of the source text according to equation (3):

denotes the attention of the i-th word in the coded attention distribution of the source text,

denotes the attention of the i-th word in the extrinsic information coding attention distribution. P _generator , P _pointer , and P _T denote the weighting coefficients for the generating probability distribution P _vocab , the source text coding attention distribution a ^t, and the extrinsic information coding attention distribution a′ ^t, respectively.

In some implementations, P _generator , P pointer , and P T may be determined based on the encoded hidden state of the source text at the current time step t, the decoded hidden state, the encoded attention distribution of the extrinsic information, and the output of the output _unit at the previous time step _t −1.

σは活性化関数を示し、例えばｓｉｇｍｏｉｄ関数であり、

はトレーニングしようとするパラメータであり、ｈ_ｔ ^＊はタイムステップｔでソーステキストの符号化隠れ状態ｈと復号化隠れ状態ｓに基づき式（３）、（４）によって決定されるパラメータであり、ｓ_ｔはタイムステップｔでの復号化隠れ状態であり、ｘ_ｔはタイムステップｔで復号化部への入力であり、すなわち、直前のタイムステップｔ－１で出力部の出力であり、ａ’^ｔはアテンション生成部４００から出力される外部情報の符号化アテンション分布である。 For example, P _generator , P _pointer , and _PT can be determined based on equation (11).

σ denotes an activation function, e.g., a sigmoid function,

is the parameter to be trained, h _t ^* is a parameter determined by equations (3) and (4) based on the encoded hidden state h and the decoded hidden state s of the source text at time step t, s _t is the decoded hidden state at time step t, x _t is the input to the decoding unit at time step t, i.e., the output of the output unit at the immediately preceding time step t-1, and a′ ^t is the encoded attention distribution of the external information output from the attention generation unit 400.

In some embodiments, when the output unit determines the output word probability distribution, it may take into account probability distribution results determined by other methods. For example, the importance of each word in the source text I may be determined by considering the correlation between multiple sentence vectors formed by word vectors in the source text I. The output word probability distribution may further include a word probability distribution formed by the above importance. For those skilled in the art, the method of generating the output word probability distribution is not limited to this, and the output word probability distribution may include various forms of word probability distributions without departing from the principles of the present disclosure.

Using the above method, the attention generation unit 400 may determine an attention distribution ^a't at the current time step of the external information, and use the attention distribution ^a't at the current time step of the external information to determine the output probability distribution of the current time step. In the embodiment provided by the present disclosure, the attention distribution of the external information, rather than the features of the external information, is used to determine the output probability distribution of the current time step, thereby avoiding the influence of invalid information in the features of the external information on the output probability distribution of the current time step.

Figure 5 shows another exemplary embodiment of a text processing device according to an embodiment of the present application.

As shown in FIG. 5, the text processing device 500 may include an encoding unit 510, a decoding unit 520, an attention generation unit 530, an output unit 540, and a post-processing unit 550. The encoding unit 510, the decoding unit 520, the attention generation unit 530, and the output unit 540 may be implemented using the encoding unit 110, the decoding unit 120, the attention generation unit 130, and the output unit 140 described in FIGS. 1 to 3, and will not be described in detail here.

The post-processing unit 550 may be configured to perform post-processing on the candidate text based on the external information to determine an output text that includes the external information.

As described above, the encoding unit 510, decoding unit 520, attention generation unit 530, and output unit 540 can be implemented in accordance with the encoding unit 110, decoding unit 120, attention generation unit 130, and output unit 140 described in Figures 1 to 3, so that the output unit 540 can output text processing results including external information.

If the results output from the output unit 540 already contain external information, the results output from the output unit 540 may be directly treated as the results of text processing.

If the result output from the output unit 540 does not include external information, the result output from the output unit 540 is set as candidate text, and the post-processing unit 550 performs post-processing on the candidate text based on the external information to determine the output text that includes the external information.

In some embodiments, the external information may include pre-specified information. For example, the external information may be a pre-specified sentence, or a sentence in the source text that includes a pre-specified word.

If the pre-specified external information is a sentence, the post-processing unit 550 may be configured to determine a similarity between the sentence in the candidate text and the external information. If the similarity is greater than a pre-defined candidate similarity threshold, the sentence in the candidate text may be replaced with the external information.

If the pre-determined external information is a word, the post-processing unit 550 may be configured to determine a similarity between a sentence in the candidate text and the external information, and replace the sentence in the candidate text with the external information if the similarity is greater than a pre-defined candidate similarity threshold.

In some implementations, if the similarity is greater than a predefined candidate similarity threshold, the post-processing unit 550 may be configured to delete the sentence in the candidate text and replace it with the sentence as external information or the sentence in the deleted candidate text that contains the word as external information.

In some examples, the external information is inserted into the remaining information based on a correlation in the source text between the external information in the source text and the redundant information in the candidate text. For example, the external information is inserted into the redundant information of the candidate text based on the order in which the external information and the redundant information in the candidate text appear in the source text.

In some other implementations, if the similarity is less than a predefined candidate similarity threshold, the post-processing unit 550 inserts external information into the candidate text based on the correlation in the source text between the external information and sentences in the candidate text.

The similarity between the external information and each sentence in the candidate text may be compared. If the similarity between the external information and each sentence in the candidate text is smaller than a predetermined candidate similarity threshold, this means that the generated text processing result does not contain information similar to the external information. In this case, a direct spelling match may be performed between the external information and the candidate text to determine the final text processing result.

For example, the final text processing result may be determined by inserting external information into the candidate text according to the order in which the external information and sentences in the candidate text appear in the source text.

By using the text processing device according to the present application, it is possible to effectively add external information to the text processing results, and it is possible to ensure that the expected external information is added to the text processing results.

As described above, the encoding unit 110, the decoding unit 120, and the attention generation unit 130 in the text processing device shown in FIG. 1 to FIG. 3 all contain parameters that need to be trained. Therefore, it is necessary to use machine learning to train at least one of the encoding unit 110, the decoding unit 120, and the attention generation unit 130.

In some embodiments, the encoding unit, the attention generating unit, and the decoding unit are trained using a predefined source text training set. The source text training set includes a plurality of training source texts.

The text processing device shown in FIG. 1 can be used to process the training source text and obtain a training text processing result for the training source text. For example, an encoding unit can be used to encode the training source text to obtain an encoded hidden state of the training source text. Then, a decoding unit can be used to determine a training decoded hidden state. Then, an attention generation unit can be used to determine a training attention distribution for the current time step based on the external information, the training source text encoded hidden state, and the training decoded hidden state. An output unit can be used to determine a training output word probability distribution based on the training attention distribution, the training source text encoded hidden state, and the training decoded hidden state, thereby determining the training output words.

By adjusting the parameters in the encoding unit, the attention generation unit, and the decoding unit, the loss function used in the training process is minimized, and training is implemented for the encoding unit, the attention generation unit, and the decoding unit.

Ｐ（ｗ_ｔ ^＊）はタイムステップｔの正解単語がタイムステップｔのトレーニング出力単語確率分布における確率値であり、（ｐ_Ｔ ^ｔ _{ｌａｂｅｌ}－ｐ_Ｔ ^ｔ）はトレーニング出力単語確率分布における正解の確率値とトレーニング出力単語確率分布における外部情報の確率値間の差異である。トレーニング出力単語に外部情報に含まれる単語が現れた時、（ｐ_Ｔ ^ｔ _{ｌａｂｅｌ}－ｐ_Ｔ ^ｔ）の値が小さく、トレーニング出力単語に外部情報に含まれる単語が現れてない時、（ｐ_Ｔ ^ｔ _{ｌａｂｅｌ}－ｐ_Ｔ ^ｔ）の値が大きい。 In some examples, the loss function used in the training process may be implemented as the following equation (12):

P( _wt ^* ) is the probability value of the correct ^word at time step t in the training output word probability distribution at time step t, and ⁽ _pTtlabel - _{pTt )} is the difference between the probability value of the correct answer in the training output word probability distribution and the probability value of the external _information in the training output word probability distribution. When a word included in the external information appears in the training output words, the value of ( _pTtlabel - ^{pTt )} is ^small , and when a word included in the external information does not appear in the training output words, the value of _{( pTtlabel - pTt} ⁾ is large.

Ｔはテキスト処理プロセスにおける全体タイムステップであり、ｔは現在タイムステップを示し、Ｐ（ｗ_ｔ ^＊）はタイムステップｔの正解単語がタイムステップｔにトレーニング出力単語確率分布における確率値を示し、－ｌｏｇＰ（ｗ_ｔ ^＊）は負の対数尤度／クロスエントロピー項を示し、

は、ソーステキストの収束メカニズムの損失項を示し、ａ_ｉ ^{ａｒｔｉｃｌｅ ｔ}は現在タイムステップｔのソーステキストアテンション分布であり、

は前の全てのタイムステップソーステキストアテンション分布の合計であり、（ｐ_Ｔ ^ｔ _{ｌａｂｅｌ}－ｐ_Ｔ ^ｔ）はトレーニング出力単語確率分布における正解の確率値とトレーニング出力単語確率分布における外部情報の確率値間の差異であり、

は、外部情報の収束メカニズム損失項であり、ａ_ｉ ^Ｔ＿ｔは現在タイムステップｔの外部情報アテンション分布

である。γ、βは事前設定したハイパーパラメータである。 In another example, the loss function loss used in the training process can be implemented by the following equations (13) and (14).

where T is the overall time step in the text processing process, t denotes the current time step, P( _wt ^* ) denotes the probability that the correct answer word at time step t is in the training output word probability distribution at time step t, and −logP( _wt ^* ) denotes the negative log-likelihood/cross-entropy term.

denotes the loss term of the source text convergence mechanism, a _i ^{article t} is the source text attention distribution at the current time step t,

is the sum of all previous time step source text attention distributions, (p _T ^t _label -p _T ^t ) is the difference between the probability value of the correct answer in the training output word probability distribution and the probability value of the external information in the training output word probability distribution,

is the convergence mechanism loss term of the extrinsic information, and a _i ^T_t is the extrinsic information attention distribution at the current time step t.

γ and β are predefined hyperparameters.

FIG. 6 illustrates an exemplary flow chart of the text processing method of the present application. As shown in FIG. 6, in step S602, the source text may be encoded to obtain a source text encoding hidden state. In some embodiments, the source text may be encoded using an encoding network. Exemplary encoding networks include long short-term memory (LSTM) networks, and LSTM network-based systems are applied to tasks such as machine translation, text summary generation, etc. As can be seen, the encoding network may be implemented as any machine learning model that encodes word vectors.

For example, when at least one word vector corresponding to a source text I is input, the encoding network outputs source text encoded hidden states _h1 , _{h2, h3... corresponding to each word vector x1, x2 , x3 ... respectively} . The number of source text encoded hidden states and the number of source text word vectors may be the same or different. For example, when generating k word vectors based on a source text I, the encoding network processes the k word vectors to generate k corresponding source text encoded hidden states, where K is an integer greater than 1.

In step S604, a decoded hidden state can be determined. In some embodiments, the decoding unit 120 receives a decoded hidden state s _t-1 of the previous time step t-1 and an output word x _t obtained by the previous time step text processing device, and obtains a decoded hidden state s _t of the current time step by processing s _t-1 and x _t . In the processing of the first time step, s ₀ and x ₁ are determined as default initial values. The decoded hidden state s may include multiple decoded hidden states s ₁ , s ₂ , s ₃ ... corresponding to the source text S. An exemplary decoding network includes a long short-term memory network. Note that the decoding network may be implemented by any machine learning model that performs decoding on the output of the encoding network.

In step S606, an attention distribution for the current time step may be determined based on external information, the source text encoded hidden state, and the decoded hidden state.

In some implementations, the attention distribution A ^t at the current time step t is the coded attention distribution of the source text. For example, the coded attention distribution a ^t of the source text can be determined using equations (1) and (2).

In some other embodiments, an attention distribution A ^t for the current time step including the external information is determined based on the external information and the attention distribution a ^t of the source text determined based on Equation (1), and the attention distribution A ^t including the external information is output for use in subsequent text processing processes.

Figure 7 shows an exemplary flowchart for determining the attention distribution for the current time step based on external information according to an embodiment of the present application.

In step S702, a coding attention distribution of the source text is determined based on the source text coding hidden state and the decoded hidden state. In some embodiments, the coding attention distribution of the source text, a ^t, can be determined using Equation (1).

In step S704, a selection probability distribution for the source text is determined based on external information, the selection probability distribution including a selection probability used for each word in the source text.

In some embodiments, a content selection network (e.g., an LSTM network) is used to process the source text I to determine a first selection probability for each word in the source text.

The content selection network processes the source text I and outputs a first selection probability result for each word in the source text, where the first selection probability represents the probability that this word in the source text I will be selected based on the content selection network and appear in the final text processing result.

In some embodiments, for at least one word included in the external information in the source text, a selection probability of the at least one word is determined to be at least a predefined probability value λ, for example, a second selection probability of each word included in the external information in the source text is determined to be a predefined probability value λ, and the second selection probability of other words included in the external information is determined to be 0.

Based on the first and second selection probabilities, a selection probability of each word in the source text is determined. For example, the selection probability of each word in the source text can be determined as the sum of the first and second selection probabilities. As can be seen, for words included in the external information, the selection probability is equal to or greater than a predefined probability value λ.

In step S706, the attention distribution can be obtained by determining the attention for each word in the source text based on the selection probability of the word.

Based on the selection probability distribution, step S706 may include, for each word in the source text, determining an attention for the word based on the selection probability of the word to obtain an attention distribution A. In some embodiments, step S706 may include determining an attention to be used for the word in the attention distribution of the current time step as 0 when the selection probability of the word is less than a predetermined selection probability threshold ε. Also, step S706 may include determining an attention to be used for the word in the attention distribution of the current time step as an attention for the word in the encoding attention distribution a ^t of the source text when the selection probability of the word is equal to or greater than a predetermined selection probability threshold ε.

Using the above attention generation method, a selection probability can be generated for each word in the source text; that is, when determining the attention for each word, at least both the magnitude of attention calculated using formula (1) and the selection probability of the word should be considered. When the selection probability of the word is lower than a predetermined selection threshold, it can be determined that the probability of the word appearing in the current time step is very low, so the attention of the word does not need to be considered in the subsequent text processing process.

The selection probability of a word included in the external information is set to at least a predefined probability value λ, and if the predefined probability value is greater than a predetermined selection probability threshold ε, the word in the external information is guaranteed not to be filtered in the content selection step, thereby ensuring that the word in the external information is processed in the subsequent text processing process, thereby increasing the probability that the word in the external information will appear in the text processing results. Note that in some implementations, the predefined probability value λ may be set to be equal to or less than the predetermined selection probability threshold ε. In this case, the selection probability of each word is determined to be the sum of the first selection probability and the second selection probability, thereby implementing the effect of increasing the selection probability of the word in the external information and increasing the probability that the word in the external information will appear in the text processing results.

Figure 8 shows another exemplary flowchart of steps for determining the attention distribution for the current time step based on external information according to an embodiment of the present application.

In step S802, the coding attention of the source text for the current time step can be determined.

In some embodiments, the coding attention parameter e _i ^{t for each word in the source text can be determined based on the source text coding hidden state of the current time step and the decoded hidden state of the current time step using equation (2), and then the coding attention distribution a t for} the source text is determined using equations (1) and (2).

In step S804, the extrinsic information coding attention for the current time step can be determined.

In some embodiments, an external attention parameter for each word in the source text can be determined, where the external attention parameter for words included in the external information is determined to a first predetermined external attention parameter, and the external attention parameter for words not included in the external information is determined to a second predetermined external attention parameter. In some implementations, the first external attention parameter is set to λ′ and the second external attention parameter is set to 0, where λ′ is a value greater than 0.

Based on the coded attention parameter determined in step S802 and the external attention parameter, the attention parameter to be used for each word in the source text can be determined. For example, the attention parameter e′ _i ^t for a word may be determined by calculating the sum of the coded attention parameter and the external attention parameter for that word.

An attention distribution for the current time step of the source text may then be determined based on the attention parameter _e'i't . For example, a softmax _{function may be applied to the attention parameter e'i't} ^{to determine} the attention for the current time step to be used for each word of the source text.

Using the above method, the attention parameter of the words included in the external information in the source text can be adjusted using a predefined external attention parameter, and the adjustment of the attention of the words included in the external information can be implemented. Note that, when the first external attention parameter is set to a hyperparameter λ' greater than 0 and the second external attention parameter is set to 0, the attention distribution of each word in the source text can be adjusted based on the external information, so that the attention of the words included in the external information is more important.

In some embodiments, the extrinsic information can be encoded to obtain an extrinsic information encoded hidden state h', and the extrinsic information encoded attention distribution ^a't can be determined using the above formulas (1) and (2), and the source text encoded hidden state h in formulas (1) and (2) should be replaced with the extrinsic information encoded hidden state h'.

In some implementations, when calculating the external information coding attention distribution ^a't and the source text coding attention distribution ^a't , respectively, the share parameters are used to calculate Equations (1) and (2), i.e., the parameters ^vT , _Wh , _Ws , and _battn used when calculating the external information coding attention distribution ^a't and the source text coding ^attention distribution at may be the same. In other implementations, training parameters for calculating the external information coding attention distribution ^a't and the source text coding attention distribution at are trained, respectively, i.e., the parameters ^vT , _Wh , _Ws , and _battn used when calculating the external information coding attention ^{distribution a't} and the source text coding attention distribution ^at may be different.

Referring back to FIG. 6, in step S608, an output word probability distribution may be determined based on the attention distribution, the source text encoding hidden state, and the decoding hidden state.

The output word probability distribution may include a generation probability distribution P _vocab . Equations (3) and (4) can be used to determine the generation probability distribution P _vocab .

In some implementations, the output word probability distribution may include the attention distribution A ^t for the current timestep.

For example, the output word probability distribution can be determined by performing a weighted addition of the generated probability distribution and the attention distribution A.

In some implementations, the weighting factors Pgen for the generation probability distribution and the attention distribution are determined based on the source text encoding hidden _state of the current time step, the source text decoding hidden state, the attention distribution and the output of the previous time step decoding network.

For example, the weighting coefficient P _gen for performing weighted addition on the generated probability distribution and the attention distribution can be expressed as in Equation (5).

When the attention distribution A ^t includes both the attention distribution a ^t of the source text and the attention distribution a′ ^t of the external information, the weighting coefficient parameters of the attention distribution a ^t of the source text and the attention distribution a′ ^t of the external information may be the same or different.

When the attention distribution A ^t includes both the attention distribution a ^t of the source text and the attention distribution a′ ^t of the external information, the output word probability distribution can be expressed as a weighted average of the generation probability distribution, the attention distribution a ^t of the source text and the attention distribution a′ ^t of the external information.

In some implementations, equation (10) may be used to determine the output word probability distribution.

In some implementations, P generator , P pointer , and P T may be determined based on the encoded hidden state of the source text at the current time step t, the decoded hidden state of the source text, the encoded attention distribution of the extrinsic information, and the output of _the decoding network at the previous time step _t -1. For example, Equation ( _{11) is used to determine P generator ,} P _pointer , and P _T.

In some implementations, step S608 may include determining the most probable word in the output word probability distribution as the output word for the current time step.

Figure 9 shows an exemplary flowchart of a text processing method according to an embodiment of the present application.

As shown in FIG. 9, in step S902, the source text is encoded to obtain a hidden source text encoding state.

In step S904, the decoded hidden state can be determined. In some embodiments, the decoded hidden state can be determined using step S604 shown in FIG. 6, which will not be described again.

In step S906, an output word probability distribution is determined based on the external information, the source text encoding hidden state, and the decoded hidden state, and an output word is determined.

In some other embodiments, step S906 may further determine and output a word probability distribution based on external information to determine the output word for the current time step.

In one implementation, step S906 may include determining, based on the external information, a word whose probability in the candidate output word is equal to or greater than an output probability threshold and is included in the external information as a candidate output word for the current time step.

For example, at least two words may be determined for each time step as candidate output words for the current time step, and then the candidate output words may be used in the text processing process for the next time step. Similarly, at least two candidate output words may be determined for the next time step.

Specifically, for example, when the number of candidate output words is two, two candidate output words a and b can be determined at time step t. Then, the candidate output words a and b can be used in the text processing process of the next time step to determine candidate output words c and d at time step t+1.

In some embodiments, when determining candidate output words for each time step, a predetermined number M (in the above example, M is 2) of words with the highest output probability in the output probability distribution are determined as candidate output words. M is an integer equal to or greater than 2.

In some other embodiments, when determining candidate output words for each time step, N words with the highest selection output probability in the output probability distribution may be determined based on a predefined method, and M words out of these N words may be determined as candidate output words. N is an integer greater than M. In some implementations, the value of N may be specified in advance.

In some other implementations, an output probability threshold may be determined in advance, and M words from among N words whose output probabilities are greater than the output probability threshold may be determined as candidate output words.

If there is no word included in the external information among the N words with the highest output probability, the M words with the highest output probability among these N words are determined as candidate output words.

When there is a word included in the external information among the N words with the highest output probability, if the number n of words included in the external information that are present in these N words is M or more, the M words included in the external information that have the highest output probability among these N words are determined as candidate output words. If the number n of words included in the external information that are present in these N words is less than a predetermined number M, the words included in the external information among these N words and the M-n words with the highest output probability among the remaining N-n words are determined as candidate output words.

At least four output candidate sequences ac, ad, bc, bd can be determined using the candidate output words a, b output at time step t and the candidate output words c, d at time step t+1. The output probability of each output candidate sequence is determined using a joint probability method, and the two with the highest output probabilities among the four output sequence candidates ac, ad, bc, bd are determined as the candidate text after time step t+1.

For example, the output probabilities of candidate output words a, b, c, d can be denoted as _Pa , _Pb , _Pc , and _Pd , respectively. The output candidate sequences ac, ad, bc, bd can be denoted as _Pa = _Pa * _Pc , _Pa = _Pa * _Pd , _Pbc = _Pb * _Pc , and _Pbd = _Pb * _Pd, respectively. If _Pa > _{Pa > Pb > Pbd} , the output sequences ac, ad are used for subsequent text processing at time step t+1.

In some embodiments, the output candidate sequence may also be determined based on external information. For example, the penalty value for the output candidate sequence may be determined using equation (6).

In one implementation, any possible text similarity algorithm may be used to determine the similarity between the candidate text sequences generated before time step t and the external information. For example, a cosine similarity method may be used to determine the similarity between the candidate text sequences generated before time step t and the external information.

Using the above formula (6), the higher the similarity between the candidate text sequence generated before time step t and the external information, the more the penalty value is used to increase the output probability of the output candidate sequence. In some implementations, the penalty value s(x,y) can be multiplied or added to the output probability in the output candidate sequence to achieve the effect of determining the output candidate sequence based on the similarity between the candidate text sequence generated before time step t and the external information.

That is, by determining the penalty value to be used for the output candidate sequence based on external information, the probability that the external information will appear in the candidate text sequence can be increased, thereby increasing the probability that the external information will appear in the final output text processing result.

In another implementation, step S906 may include determining a similarity between the external information and the source text coding hidden state, and determining a word to be output at the current time step based on the similarity between the external information and the source text coding hidden state.

For example, the extrinsic information may be encoded using a coding network to obtain an extrinsic information encoded hidden state.

Step S906 may include determining a similarity between the extrinsic information encoded hidden state and the decoded hidden state. If the similarity between the extrinsic information encoded hidden state and the decoded hidden state is greater than or equal to a predefined similarity threshold, the extrinsic information may be output as the output for the current time step.

If the extrinsic information is a word, the extrinsic information may be output as the word of the current time step. If the extrinsic information is a sentence, the extrinsic information may be inserted directly after the text sequence generated before the current time step t.

Note that the text sequence already generated before the current time step t may have been generated based on the most probable word in the output probability distribution, or may be generated based on some of the most probable candidate output words in the output probability distribution. The candidate output words to be adopted may be determined by the process described in the implementation method, and will not be described in detail here.

If the similarity between the extrinsic information encoded hidden state and the decoded hidden state is less than a predefined similarity threshold, an output word probability distribution for the current time step is determined based on the results output from the decoding network, and an output word for the current time step is determined based on the output word probability distribution for the current time step.

Using the above method, when the similarity between the result output from the decoding network and the external information is higher, the result output from the decoding network may be directly replaced with the external information. That is, in this case, the result of the text sequence determined after the output of the current time step is the result obtained by inserting the external information after the text sequence determined after the output of the previous time step.

Then, when processing the next time step, we use the decoding network to encode the extrinsic information to obtain the decoded hidden state of the next time step, so that the later decoding process can use the results of the extrinsic information, thereby ensuring semantic consistency between the results obtained in the later decoding and the inserted extrinsic information.

When the external information is a word, the decoded hidden state of the previous time step and the external information can be used to process it as the input of the decoding network to obtain the decoded hidden state of the current time step.

When the external information contains multiple words, the decoding network performs several loop processes. The inputs to the decoding network in the first loop are the decoded hidden state of the previous time step and the first word of the external information, and the inputs to the decoding network in subsequent loops are the decoded hidden state obtained in the previous loop and the next word of the external information. By performing several loops, it is possible to process each word in the external information, and a decoded hidden state containing all the external information is obtained, which becomes the decoded hidden state of the current time step.

In some implementations, after the external information has already been substituted into the results output by the decoding network and inserted into the text processing results, a similarity comparison between the external information encoded hidden state and the decoded hidden state is not performed.

In some examples, the similarity threshold may be implemented as a function with respect to a given time step t.

As described above, if the similarity between the external information encoded hidden state and the decoded hidden state is smaller than a predefined similarity threshold, the operation of replacing the output of the decoding network with the external information is not performed, and the output result is determined based on the output word probability distribution. In this case, in order to increase the probability that the external information appears in the final text processing result, an adjusted similarity threshold is determined by adjusting the similarity threshold of the current time step, the adjusted similarity threshold is smaller than the similarity threshold of the current time step, and the adjusted similarity threshold is used as the similarity threshold of the next time step.

For example, adjust the similarity threshold using equation (7).

By performing a monotonically decreasing adjustment of the similarity threshold at each time step, the similarity between the external information and the output result of the decoding unit is reduced to a lower level in the text processing process, so that the probability that the similarity between the external information and the output result of the decoding network will be greater than the similarity threshold for the current time step increases. In other words, the probability that the external information will appear in the final text processing result increases.

Figure 10 shows an exemplary flowchart of a text processing method according to an embodiment of the present application.

In step S1002, the source text is encoded to obtain the source text encoded hidden state.

In step S1004, the decoded hidden state is determined.

In step S1006, output words for each time step are determined based on the source text encoded hidden state and the decoded hidden state, and candidate texts are determined.

In step S1008, post-processing is performed on the candidate text based on the external information to determine an output text that includes the external information.

If the result output from step S1006 does not include external information, the result output from step S1006 is set as candidate text, and post-processing is performed on the candidate text based on the external information to determine an output text that includes external information.

In some implementations, the external information may include pre-specified information. For example, the external information may be a pre-specified sentence, or a sentence in the source text that includes a pre-specified word.

If the pre-specified external information is a sentence, a similarity between the sentence in the candidate text and the external information may be determined. If the similarity is greater than a pre-defined candidate similarity threshold, the sentence in the candidate text may be replaced with the external information.

If the predetermined external information is a word, a similarity between a sentence in the candidate text and the external information may be determined, and if the similarity is greater than a predetermined candidate similarity threshold, the sentence in the candidate text may be replaced with the external information.

In some implementations, if the similarity is greater than a predefined candidate similarity threshold, the sentence in the candidate text may be deleted and replaced with an external sentence or a sentence containing the external word in the deleted candidate text.

In some examples, the external information is inserted into the remaining information based on a correlation in the source text between the external information in the source text and the redundant information in the candidate text. For example, the external information is inserted into the redundant information of the candidate text based on the order in which the external information and the redundant information in the candidate text appear in the source text.

In some other implementations, if the similarity is less than a predefined candidate similarity threshold, external information is inserted into the candidate text based on correlations in the source text between the external information and sentences in the candidate text.

The similarity between the external information and each sentence in the candidate text may be compared. If the similarity between the external information and each sentence in the candidate text is smaller than a predetermined candidate similarity threshold, this means that the generated text processing result does not contain information similar to the external information. In this case, a direct spelling match may be performed between the external information and the candidate text to determine the final text processing result.

For example, the final text processing result may be determined by inserting external information into the candidate text according to the order in which the external information and sentences in the candidate text appear in the source text.

By utilizing the above text processing method according to the present application, it is possible to effectively add external information content to the text processing results, and it is possible to ensure that the expected external information content is added to the text processing results.

By using the text processing method of the present application, in the text generation process, the content of the external information can be effectively taken into account in the text processing process by determining the attention distribution of the current time step using external information and/or determining the output words of the current time step based on the external information, thereby increasing the probability of generating external information in the text generation process and improving the effect of generating text when external information is taken into account.

The method or apparatus according to the embodiment of the present application may also be implemented by the structure of a computing device shown in FIG. 11. FIG. 11 shows the structure of the computing device. As shown in FIG. 11, the computing device 1100 may include a bus 1110, one or at least two CPUs 1120, a read-only memory (ROM) 1130, a random access memory (RAM) 1140, a communication port 1150 connected to a network, input/output components 1160, a hard disk 1170, etc. The memory device in the computing device 1100, such as the ROM 1130 or the hard disk 1170, may record various data or files used in the processing and/or communication of the method for detecting a target in a video provided by the present application and programming commands executed by the CPU. The computing device 1100 may include a user interface 1180. Of course, the structure shown in FIG. 11 is exemplary, and when implementing a different device, one or more components in the computing device shown in FIG. 11 may be omitted according to actual needs.

The embodiments of the present application may further be implemented by a computer-readable recording medium. Computer-readable commands are recorded on the computer-readable recording medium of the embodiments of the present application. When the computer-readable commands are executed by a processor, they may be executed with reference to the methods of the embodiments of the present application described in the above drawings. The computer-readable recording medium may include, but is not limited to, volatile memory and/or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and/or cache. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, a flash memory, etc.

As will be understood by those skilled in the art, the contents described herein may be modified and improved in various ways. For example, the various devices or components described above may be implemented in hardware, software, firmware, or a combination of some or all of these.

Also, as shown in the present application and the claims, unless the context clearly indicates an exception case, words such as "a," "one," "one," "a kind," and/or "the" do not refer to the singular but may include the plural. In general, "comprise" or "comprises" merely indicates the inclusion of steps and elements already clearly labeled, but these steps and elements are not exhaustive and the method or device may include other steps or elements.

Furthermore, although various references are made herein to certain units in the systems according to the embodiments of the present application, any number of different units may be used and executed on the user side and/or server side. The units are illustrative, and different aspects of the systems and methods may use different units.

Note that in this application, the operations performed by the system according to the embodiment of the application are described using flowcharts. Note that the previous or subsequent operations do not have to be performed in order. Conversely, each step may be processed in the reverse order or simultaneously. Additionally, other operations may be added to these processes, or a step or multiple steps may be omitted from these processes.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. Unless otherwise defined, for example, terms normally defined by dictionaries have meanings consistent with their meaning in the context of their related art and are not to be construed in an idealized or formal sense.

The above is a description of the present invention, but is not intended to limit the present invention. Although several exemplary embodiments of the present invention have been described, those skilled in the art will understand that various modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present invention. Therefore, these modifications are intended to be included within the scope of the present invention, which is defined by the claims. Note that the above is a description of the present invention, and is not limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are also intended to be included within the scope of the appended claims. The present invention is limited to the claims and their equivalents.

Claims (5)

A text processing device that processes word vectors of a source text based on a sequence-to-sequence model while taking into account external information, comprising:
an encoding unit configured to encode word vectors of a source text using an encoding network to obtain a source text encoding hidden state;
a decoder configured to decode an output of the encoder utilizing a decoding network to determine a decoded hidden state;
an attention generating unit configured to calculate, in each time step, an encoding attention parameter for each word in the source text using the source text encoding hidden state and the decoded hidden state of the current time step, and to calculate an encoding attention distribution of the source text by applying a softmax function to the encoding attention parameter, as an attention distribution of the current time step;
an output unit configured to obtain a context vector based on the attention distribution and the source text encoding hidden state of a current time step , calculate a generation probability distribution based on the context vector and the decoded hidden state, calculate an output word probability distribution by performing a weighted addition of the generation probability distribution and the attention distribution , determine a word in the output word probability distribution whose output probability is equal to or greater than a preset output probability threshold and which is included in the external information as a candidate output word of a current time step, determine an output candidate sequence based on the candidate output word of the current time step and the candidate output word of a next time step, determine an output probability of the output candidate sequence based on a joint probability of an output probability of the candidate output word of the current time step and an output probability of the candidate output word of the next time step, determine a penalty value of the output candidate sequence based on a similarity between a candidate text sequence determined in a previous time step and the external information, adjust the output probability of the output candidate sequence using the penalty value such that the higher the similarity, the greater the penalty value increases the output probability of the output candidate sequence, and determine the candidate output word in a predetermined number of sequences having the highest output probability of the adjusted output candidate sequence as the output word of the current time step ,
A text processing device, wherein the external information is predefined text information expected as a result of processing a source text. The encoding unit and the decoding unit
In the encoding unit, encoding the training source text using an encoding network to obtain a training source text encoding hidden state;
in the decoding unit, decoding the output of the encoding unit using a decoding network to determine training decoded hidden states;
determining training output words for a current time step based on the extrinsic information, the training source text encoded hidden states, and the training decoded hidden states;
2. The text processing apparatus according to claim 1 , wherein the training is performed by adjusting parameters in the encoding unit and the decoding unit so that a difference between the training output words and words included in the external information is minimized. A text processing method for performing processing on word vectors of a source text based on a sequence-to-sequence model while taking into account external information, comprising:
encoding word vectors of a source text using an encoding network to obtain source text encoding hidden states;
using a decoding network to decode an output of the encoding network to determine a decoded hidden state;
At each time step, calculating an encoding attention parameter for each word in the source text using the source text encoding hidden state and the decoded hidden state of the current time step, and calculating an encoding attention distribution of the source text by applying a softmax function to the encoding attention parameter, which is the attention distribution of the current time step;
determining an output candidate sequence based on a joint probability of an output probability of the candidate output word of the current time step and an output probability of the candidate output word of the next time step; determining a penalty value of the output candidate sequence based on a similarity between a candidate text sequence determined in a previous time step and the external information; adjusting the output probability of the output candidate sequence using the penalty value such that the penalty value increases the output probability of the output candidate sequence as the similarity increases; and determining the candidate output word in a predetermined number of sequences having the highest output probability of the adjusted output candidate sequence as the output word of the current time step . A processor;
a memory having computer readable programming instructions stored therein;
A text processing device which performs the text processing method of claim 3 when the computer readable programming instructions are executed by the processor. A computer readable recording medium having computer readable instructions recorded thereon which, when executed by a computer, cause the computer to carry out the text processing method of claim 3 .