JP5743952B2 - Quotation part extraction apparatus, method, and program - Google Patents

Description

  The present invention relates to a citation extraction device, method, and program, and more particularly, to a citation extraction device, method, and program for extracting a portion to be cited from a document.

  Conventionally, a sequence of a plurality of words is extracted from a document as a single unit, such as extraction of a specific expression (see Non-Patent Document 1, for example). In specific expression extraction, etc., using a discriminator for extracting a target part from a document, a B tag is used for the word at the beginning of the extraction range, an I tag is used for a word other than the start of the extraction range, and An O tag is assigned to extract a group of words to which a B tag and an I tag are assigned. CRF (Conditional Random Fields) is known as a discriminator used for solving such sequence labeling problems such as named entity extraction and morphological analysis (for example, see Non-Patent Document 2).

Yuta Tsuboi, Hisayoshi Kashima, Taku Kudo, "Development of Discrimination Model in Language Processing-From HMM to CRF", 2006 Annual Conference of the Language Processing Society of Japan John Lafferty, Andrew McCallum, Fernando Pereira, "Conditional random fields: Probabilistic models for segmenting and labeling sequence data", 2001.

  However, Non-Patent Documents 1 and 2 do not disclose the application of a conventional method for extracting a specific expression to citation extraction. For example, when a document “e-mail to Mr. A that says“ I am a little late ”” is input through voice input or the like, it is not explicit where the quotation part (the range of “”) is. Even if the above-described conventional method for extracting a specific expression is applied to such a document, it is not possible to appropriately extract the citation part.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide a citation extraction apparatus, method, and program capable of extracting a citation from a document to which a citation is not explicitly given. And

  According to a first aspect of the present invention, there is provided a quoting part extracting device that divides document data into a plurality of parts, and that each part divided by the dividing means has a feature according to characteristics of a quoting part extracted from the document data. With each of the parts, with each of the plurality of types of tags indicating whether or not it is a citation part as candidates, information indicating whether or not the restriction based on the citation part to be extracted is satisfied is attached to each part. Based on a constrained lattice generating means for generating a lattice and an extraction model for extracting a citation part from the lattice, an optimal candidate for each part is selected from the constrained lattice generated by the constrained lattice generating means, Extraction means for extracting, as a citation part, a portion in which a tag indicating the citation part is selected as an optimal candidate.

  According to the citation extraction apparatus according to the first invention, the dividing means divides the document data into a plurality of parts, and the constrained lattice generating means extracts from the document data into each part divided by the dividing means. A feature according to the characteristics of the quotation part is given, and for each part, information indicating whether or not the restriction based on the quotation part to be extracted is satisfied with each of a plurality of types of tags indicating whether or not the quotation part is satisfied as candidates. Generate a constrained lattice attached to each part. Then, the extraction means selects the optimal candidate for each part from the constrained lattice generated by the constrained lattice generation means based on the extraction model for extracting the citation part from the lattice, and the tag indicating the citation part is optimal. The part selected as a candidate is extracted as a citation part.

  In this way, a constrained lattice is generated with constraints according to the characteristics of the citation part to be extracted, and the citation part is extracted from the document using the citation part extraction model. Extract citations from missing documents.

  Further, the citation extraction device according to the first invention is a word other than the word that appears at the end of the document data among the words in the document data that match a keyword predetermined according to the characteristics of the citation to be extracted. The constrained lattice generating means generates the constrained lattice for the document data concealed by the concealing means, and the extracting means comprises: When the selected part includes a concealed word, the concealed word can be restored and extracted as the citation part. Thereby, it is possible to prevent erroneous extraction of the citation part without looking at a wide context.

  In the first invention, the dividing unit can divide the document data into phrases. As a result, it is possible to suppress the sparsity of the features assigned to each phrase, and it is possible to provide features that also take into account the information of parts located at distant positions.

  In addition, the citation extraction device according to the first aspect of the invention provides a feature according to the characteristics of the citation to each part divided by the dividing means, and determines whether each part is the citation. The extraction model is generated based on a lattice generation means for generating a lattice with each of a plurality of types of tags shown as candidates, a lattice generated by the lattice generation means, and a correct position of a citation in the document data. Learning means for learning. Thereby, it can have the structure for producing | generating the extraction model used when extracting a quotation part.

  According to a second aspect of the present invention, there is provided a quoting part extraction device that divides document data into a plurality of parts, and that divides the document data into parts divided by the dividing means according to the characteristics of the citation part extracted from the document data. A lattice generating means for generating a lattice with candidates for each of a plurality of types of tags indicating whether or not each part is a citation part, a lattice generated by the lattice generating means, and the document Learning means for learning an extraction model for extracting a citation part from a lattice based on the correct position of the citation part in the data can be configured. Using the extraction model generated in this way, it is possible to extract a citation from a document to which a citation is not explicitly given.

  In the citation extraction method according to the third invention, the dividing unit divides the document data into a plurality of parts, and the constrained lattice generation unit extracts each part divided by the dividing unit from the document data. A feature corresponding to the characteristics of the quoted part to be given is given, and each of the plurality of types of tags indicating whether or not the quoted part is selected as a candidate for each part, and indicates whether or not the constraint based on the extracted quoted part is satisfied A constrained lattice in which information is given to each part is generated, and an extracting unit extracts each of the constrained lattices generated by the constrained lattice generating unit based on an extraction model for extracting a citation from the lattice. In this method, an optimum candidate for a part is selected, and a part in which a tag indicating a citation part is selected as an optimum candidate is extracted as a citation part.

  According to a fourth aspect of the present invention, the quoting part extracting method divides the document data into a plurality of parts by the dividing means, and the lattice generating means extracts from the document data into the parts divided by the dividing means. A feature corresponding to the characteristics of the part is given, and for each of the parts, a lattice is generated with each of a plurality of types of tags indicating whether the part is a quoted part, and learning means is generated by the lattice generating means. This is a method of learning an extraction model for extracting a citation from a lattice based on the lattice and the correct position of the citation in the document data.

  Moreover, the quotation part extraction program which concerns on 5th invention is a program for functioning a computer as each means which comprises said quotation part extraction apparatus.

  As described above, according to the citation extraction apparatus, method, and program of the present invention, a constraint lattice to which a restriction according to the characteristics of the citation to be extracted is generated is generated, and the citation extraction model is used. Since the citation part is extracted from the document, the citation part can be extracted from the document to which the citation part is not explicitly given.

It is a functional block diagram which shows the structure of the quotation part extraction apparatus which concerns on this Embodiment. It is the schematic which shows an example of the lattice produced | generated by the lattice production | generation part. It is the schematic which shows an example of the restricted lattice produced | generated in the restricted lattice production | generation part. It is the schematic which shows an example of the optimal candidate selected from the lattice with restrictions. It is a flowchart which shows the content of the learning process routine in this Embodiment. It is a flowchart which shows the content of the application process routine in this Embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  The quotation part extraction apparatus 10 according to the present embodiment is configured by a computer including a CPU, a RAM, and a ROM that stores a program for executing a quotation part extraction processing routine including a learning process and an application process described later. can do.

  As shown in FIG. 1, this computer can be functionally represented by a configuration including a learning unit 12 and an application unit 14, and the learning unit 12 further includes a phrase division unit 20 and a keyword concealment unit 22. And a lattice generation unit 24 and a citation extraction learning unit 26. The application unit 14 includes a phrase division unit 20, a keyword concealment unit 22, a constrained lattice generation unit 28, It can be expressed by a configuration including the citation extraction unit 30. The phrase division unit 20 and the keyword concealment unit 22 are functions common to the learning unit 12 and the application unit 14. The phrase dividing unit 20 is an example of the dividing unit of the present invention, the keyword concealing unit 22 is an example of the concealing unit of the present invention, and the citation extraction learning unit 26 is an example of the learning unit of the present invention. The part extraction unit 30 is an example of the extraction means of the present invention.

  Here, the citation part to be extracted in this embodiment is defined as follows.

  For “someone or something” (a), quote “b” in the document “something” (b), “something” (c), “something” (d). Suppose that

  For example, in the case of a document “send me a little late for email”, a = A, b = slightly late, c = email, d = send.

  Also, in the case of “note queuing at 3 o'clock tomorrow”, a = self or a system capable of automatically taking notes, b = waiting at 3 o'clock tomorrow, c = memo, d = recording.

  It is not necessarily a condition that all of “a” to “d” are included in the document. Further, as a definition of the citation part, not only b but also b + c and other combinations may be defined.

  The phrase dividing unit 20 outputs a phrase-delimited sentence obtained by dividing an input document (text data) into a plurality of parts by dividing the input document (text data) by using a conventionally known classifier such as CRF. By dividing the document into phrase units, the sparseness of the features can be suppressed when features are added to each part (each clause) in the lattice generation unit 24 and the constrained lattice generation unit 28 described later. Compared with the case where each word is divided, it is possible to give a feature that also considers information of a portion at a distant position.

  The keyword concealing unit 22 refers to the keyword dictionary 40 in which keywords according to the characteristics of the quoting part to be extracted are registered in advance, and a word other than the word that appears at the end of the document among the words that match the keyword in the phrase delimiter Output the concealed phrase delimiter sentence that conceals the word. For example, in the case of a document “Email Mr. A to“ Email Mr. B ””, “Mail” that appears for the first time is a word in the quote part, but the word “Mail” Since a feature with a very strong power is given as the feature given when extracting the citation part defined as above, "Email Mr. A to Mr. B" As described above, there is a possibility that an erroneous part is extracted as a citation part starting from “mail” that appears first. In order to prevent such erroneous extraction, it is necessary to look at a relatively wide context. However, by concealing the word that matches the keyword, the window size when assigning features to each phrase (reference range) ) Can be prevented and the above-mentioned erroneous extraction can be prevented without looking at a wide context.

  Further, the keyword concealment unit 22 outputs concealment information such as the concealed word and the position of the word to the citation unit extraction unit 30 at the time of application for extracting the citation unit from the document.

  The lattice generation unit 24 assigns features corresponding to the characteristics of the citation to each clause of the concealed clause delimiter output from the keyword concealment unit 22. For example, a feature indicating content such as “the next phrase contains the word“ mail ”” or “the next phrase contains the word“ noun ””. Can be granted. In the case of a document input by voice, pause information may be used as a feature. The pause information is an important clue for dividing a sentence, and is also important information as information indicating the start position or end position of the citation section. Therefore, a feature such as “there is pause information before the phrase” can be given. Moreover, when using a learning device, it is an advantage that there is no need to consider the dependence of these features.

  In addition, the lattice generation unit 24 uses, as candidates for each phrase, a B tag indicating the head portion of the citation portion, an I tag indicating a portion other than the head portion of the citation portion, and an O tag indicating a portion other than the citation portion. And outputs a lattice containing the above features for each clause.

  The citation extraction learning unit 26 learns a citation extraction model 42 for extracting a citation from the lattice output from the lattice generation unit 24 and correct data indicating the correct position of the citation in the input document. . FIG. 2 shows an example in which the correct tag of each phrase is colored with respect to the lattice output from the lattice generation unit 24. Note that the lattice includes features for each phrase, but the features are not shown in FIG. The same applies to FIGS. 3 and 4 described later. As a learning method, a discriminative learning method similar to a general sequence labeling problem can be used. Various features can be used by using discriminative learning.

  Similar to the lattice generation unit 28, the constrained lattice generation unit 28 adds features to each clause of the concealed clause delimiter output from the keyword concealment unit 22, and for each clause, a B tag, an I tag, and A lattice with an O tag as a candidate is generated. In addition, the constrained lattice generation unit 28 assigns, for each candidate, constraint information indicating whether or not the constraint based on the extracted citation unit is satisfied. For example, a restriction can be provided that an I tag clause appears only after a B tag clause (an I tag clause does not appear after an O tag clause or at the beginning of a sentence). In addition, a condition that the citation part appears only once in the document may be set, and based on this condition, a restriction may be provided that the B tag clause appears only once.

  FIG. 3 shows an example of a constrained lattice in which constraint information is given to each candidate. A candidate that satisfies the constraint is given “valid” in parentheses after the symbol indicating the tag, and “invalid” is given to a candidate that does not satisfy the constraint. FIG. 3 shows an example in which a plurality (two in this case) of restricted lattices are generated when a plurality of constraints are appropriately combined.

  In addition, although information of “valid” and “invalid” is also given to the lattice used in the citation extraction learning unit 26 shown in FIG. 2, the correct tag is selected using the correct position of the citation in FIG. This also indicates that the correct tag satisfies the constraints as described above, and therefore the correct tag candidate matches the candidate given “valid”.

  The citation extraction unit 30 uses the citation extraction model 42 generated by the citation extraction learning unit 26 from the constrained lattice generated by the constrained lattice generation unit 28 and uses the most likely candidate (optimum) for each phrase. Select (Candidate). Here, first, the most likely constrained lattice is selected from a plurality of constrained lattices, and the optimal candidate is selected by searching the optimal route from the selected constrained lattice. FIG. 4 shows a lattice in which the selected candidate is colored. In the example of FIG. 4, the upper restricted lattice shown in FIG. 3 is selected, and the optimum candidate is selected from the candidates given “valid” of the selected restricted lattice.

  In addition, the citation extraction unit 30 extracts a phrase in which the B tag and the I tag are selected as optimal candidates, and the words concealed by the keyword concealment unit 22 are based on the concealment information output from the keyword concealment unit 22. Return to the original and extract the quoted part. In addition, if the phrase ends with a quote particle such as "~ tte", the part before the quote particle is extracted as a quote part, and an output sentence with quote part information attached with information indicating the extracted quote part is output. To do.

  Next, the operation of the citation extraction device 10 according to the present embodiment will be described. When an input sentence for learning (an input sentence including information on the correct position of the quotation part) is input to the quotation part extraction apparatus 10, the learning part 12 executes the learning processing routine shown in FIG. A part extraction model 42 is generated. When an input sentence (an input sentence in which the correct position of the citation part is not explicit) is input to the citation part extraction apparatus 10 in a state where the citation part extraction model 42 is generated, the application part At 14, the citation is extracted from the input sentence by executing the application process routine shown in FIG. Hereinafter, each process is explained in full detail.

First, in the learning processing routine, in step 100, the phrase dividing unit 20 acquires an input sentence (text data) for learning. Here, it is assumed that an input sentence for learning as shown below is acquired. In addition, in this input sentence for learning, correct data indicating the correct position of the quoted part as shown below is given. Input sentence: Mail A to Mr. B and correct data: “Email B”
Then, the phrase dividing unit 20 divides the acquired learning input sentence for each phrase and divides it into a plurality of parts, and outputs a phrase-delimited sentence as shown below.
Sentence breaks: To A / B / email / email

Next, in step 102, the keyword concealment unit 22 refers to the keyword dictionary 40 in which keywords corresponding to the characteristics of the quoting part to be extracted are registered in advance, and the keywords in the phrase delimiter sentence output in step 100 are A concealed phrase delimiter sentence that conceals words other than the word that appears at the end of the document among the matching words is output. When the word “mail” is registered in the keyword dictionary 40, a concealed phrase delimiter as shown below is output for the input sentence. In the following, the concealed part is represented by “x”.
Hidden paragraph breaks: To Mr. A / To Mr. B / X / Mail

  Next, in step 104, the lattice generation unit 24 assigns a feature corresponding to the characteristics of the quoting part to each clause of the concealed clause-delimited sentence output in step 102. In addition, the lattice generation unit 24 uses, as candidates for each phrase, a B tag indicating the head portion of the citation portion, an I tag indicating a portion other than the head portion of the citation portion, and an O tag indicating a portion other than the citation portion. (For example, FIG. 2) is generated, and a lattice including the above-described feature for each clause is output.

  Next, in step 106, the citation extraction learning unit 26 uses the lattice output in step 104 and the correct answer data given to the learning input sentence acquired in step 100 to determine the citation unit. The citation extraction model 42 for extraction is learned. The citation extraction model 42 generated by learning is stored in a predetermined storage area, and the learning processing routine is terminated.

Next, in the application processing routine, in step 110, the phrase dividing unit 20 acquires an applied input sentence (text data) to which the citation extraction is applied. Here, an application input sentence as shown below is acquired.
Applicable input sentence: E-mail is sent to Mr. A, and then the phrase dividing unit 20 divides the acquired applied input sentence into a plurality of parts by dividing the acquired input sentence for each phrase, as shown below. Outputs paragraph breaks.
Sentence breaks: To Mr. A / To Mr./C

Next, in step 112, the keyword concealment unit 22 refers to the keyword dictionary 40 in the same manner as in step 102 of the learning process, and converts the phrase delimiter sentence output in step 110 into a concealed phrase delimiter sentence. And the concealment information such as the concealed word and the position of the word. Here, a concealed clause-delimited sentence and concealment information as shown below are output for the applied input sentence.
Concealed sentence break sentence: To Mr. A / To Mr./C/Choose/Email Concealment information: Third sentence, X → Mail

  Next, in step 114, the constrained lattice generation unit 28 adds a feature to each clause of the concealed clause-delimited sentence output in step 112, and for each clause, a B tag, an I tag, and an O tag. Generate a lattice with candidate. Further, the constrained lattice generation unit 28 generates a constrained lattice (for example, FIG. 3) in which constraining information indicating whether or not the constraint based on the extracted citation part is satisfied is given to each candidate. Output a constrained lattice including features.

Next, in step 116, the citation extraction unit 30 reads the citation extraction model 42 stored in the predetermined area, and selects the optimum candidate for each phrase from the constrained lattice output in 114 above. Then, the phrase in which B tag and I tag are selected as the optimal candidates is extracted (for example, FIG. 4), and the word concealed in step 112 is restored based on the concealment information output in step 112 above. To extract the citation. Furthermore, when the last part of a phrase is a quote particle such as “~ tte”, the part before the quote particle is extracted as a quote part, and information indicating the extracted quote part (“” part as shown below) ) Is output to the input sentence, and the application processing routine is terminated.
Output text with quoted part information: Email Mr. A to “Email C”

  As described above, according to the citation extraction device according to the present embodiment, a lattice with a constraint according to the characteristics of the citation to be extracted is generated, and the document is extracted from the document using the citation extraction model. Since the citation part is extracted, the citation part can be extracted from a document to which the citation part is not explicitly given. In addition, by concealing words other than the word that appears at the end of the document among the words that match the keyword according to the characteristics of the citation part to be extracted, erroneous extraction of the citation part is prevented without looking at a wide context. be able to. Further, by dividing the document into clauses, it is possible to suppress the sparsity of the features given to each clause, and it is possible to give features that also take into account information at a distant portion.

  The present invention is not limited to the above embodiment, and various modifications and applications can be made without departing from the gist of the present invention.

  For example, in the above embodiment, the case where the citation extraction model is learned and generated by an identification learning device such as CRF has been described. However, the citation extraction model may be generated on a rule basis. For example, in the case of an input sentence including pose information “mail (pause) is likely to be late”, when “pause information” follows “mail”, the next word of the pose information is a quote part. By describing the rules, a citation extraction model can be generated.

  Moreover, in the said embodiment, as multiple types of tags which show whether it is a quotation part, B tag which shows the head part of a quotation part, I tag which shows parts other than the head among quotation parts, and parts other than a quotation part However, the present invention is not limited to this, and a tag that can distinguish whether each part is a citation part or a part other than the citation part may be used. For example, you may use 4 types which added E tag which shows the last part of a quotation part, and may use 3 types, E tag, I tag, and O tag except B tag.

  In the above embodiment, the learning unit and the application unit are configured by the same computer. However, the learning unit and the application unit may be configured by separate computers.

  Further, in the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium. Moreover, you may comprise the quotation part extraction apparatus of this invention by hardware, such as a semiconductor integrated circuit for implement | achieving the said process.

DESCRIPTION OF SYMBOLS 10 Quotation part extraction apparatus 12 Learning part 14 Application part 20 Clause division part 22 Keyword concealment part 24 Lattice generation part 26 Quote part extraction learning part 28 Constrained lattice generation part 30 Quote part extraction part 40 Keyword dictionary 42 Quote part extraction model

Claims (6)

A dividing means for dividing the document data into a plurality of parts;
Each part divided by the dividing means is given a feature according to the characteristics of the quotation part extracted from the document data, and each of the plural types of tags indicating whether or not the quotation part is a candidate for each part. , A constrained lattice generation means for generating a constrained lattice in which information indicating whether or not the constraint based on the extracted citation part is satisfied is given to each part;
Based on the extraction model for extracting the citation part from the lattice, the optimum candidate of each part is selected from the constrained lattice generated by the constrained lattice generation means, and the tag indicating the citation part is selected as the optimal candidate Extraction means for extracting the part as a quote part,
Quotation part extraction device including Including concealment means for concealing words other than the word appearing at the end of the document data among the words in the document data matching a predetermined keyword according to the characteristics of the citation part to be extracted,
The constrained lattice generation means generates the constrained lattice for the document data after concealment by the concealment means,
The citation extraction device according to claim 1, wherein, when the selected part includes a concealed word, the extraction unit restores the concealed word and extracts it as the citation part.   The citation extraction apparatus according to claim 1, wherein the dividing unit divides the document data into phrases. A feature corresponding to the characteristics of the quoting part is given to each part divided by the dividing unit, and a lattice is generated with each of the plurality of types of tags indicating whether or not the quoting part is a candidate for each part. Lattice generating means to
Learning means for learning the extraction model based on the lattice generated by the lattice generation means and the correct position of the citation in the document data;
The citation extraction device according to claim 1, comprising: The dividing means divides the document data into a plurality of parts,
The constrained lattice generation means assigns a feature according to the characteristics of the citation part extracted from the document data to each part divided by the division means, and indicates a plurality of types indicating whether each part is a citation part Each of the tag as a candidate, generating a constrained lattice that gives information indicating whether or not the constraint based on the citation part to be extracted is satisfied,
Based on the extraction model for extracting the citation part from the lattice, the extraction means selects the optimum candidate of each part from the constrained lattice generated by the constrained lattice generation means, and the tag indicating the citation part is optimal A citation part extraction method that extracts a part selected as a candidate as a citation part. A citation extraction program for causing a computer to function as each means constituting the citation extraction device according to any one of claims 1 to 4 .