JP3338570B2 - Prediction device for processing end time in batch processing of natural language - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to relates to the prediction apparatus of the processing end time in the batch processing of natural language.

2. Description of the Related Art Natural language processing for performing processing such as language translation and character recognition for natural languages such as Japanese and English takes a considerable amount of processing time when the amount of documents (texts) to be processed is enormous. However, since it is not possible to accurately predict the processing end time, the user often simply waits or leaves the processing end in vain, which is inconvenient in that the user cannot make a schedule. The improvement is desired.

[0003]

2. Description of the Related Art Conventionally, in natural language processing, there has been a function of displaying a progress by a document number or a percentage being processed, but an end time cannot be predicted. Japanese Patent Application Laid-Open No. 63-284677 is a document that discloses a technique for estimating an end time. The technology described in this publication is shown in FIG. 5 as an explanatory diagram of a conventional example.

[0004] In FIG. Is the block diagram,
B. Is a flowchart. When a sentence to be processed first is input from an external storage device or a keyboard, an input sentence holding unit 9
At this time, the sentence length measuring unit 94 counts the number of characters of the input sentence (S1 in FIG. 5). When the holding of the input sentence is completed, a predicted processing time is calculated (S2 and S3). in this case,
The processing capacity is held in a processing capacity holding unit 95 in which the processing capacity is held in advance, and the predicted processing time is obtained by dividing the number of characters measured by the sentence length measuring unit 94 by the processing capacity (characters / second). The calculation result is held in the predicted processing time holding unit 97 and displayed as desired (S5 and S6), and then natural language processing is processed by the natural language processing unit 93 (S7). When the content is processed (S8), the process returns to step S1, and the same operation is repeated while newly holding the input sentence.

In this prior art publication, as another flowchart, after measuring the input character length, a natural language process is performed, and a predicted processing time calculation and a result holding step are performed. This is an example in which the processing capacity is not stored in the processing capacity holding unit in advance, but the processing capacity is calculated from the length of the sentence that has been processed in the process and the processing time. To keep,
The specific configuration is unknown because detailed contents are not described.

[0006]

According to the above conventional example,
The principle is to calculate the predicted processing time by measuring the number of characters held in the input sentence holding unit and dividing by the number of characters that can be processed in one second held in the processing capacity holding unit.

On the other hand, when a sentence composed of a large number of characters is subjected to natural language processing such as language translation of a sentence composed of a large number of characters, the sentence processing ability to determine how many characters can be processed in one second is determined in advance. It is difficult to obtain an accurate numerical value because the setting changes depending on the content, and there is a problem that if the text is long, the error increases and the predicted value is not trusted.

Therefore, at present, the end time of the language processing is roughly roughly predicted by the user based on empirical rules. The present invention provides an apparatus for predicting the processing end time in batch processing of natural language, which can accurately predict the end time according to the progress of processing when batch processing a long sentence to be processed by natural language. The purpose is to provide.

[0009]

FIG. 1 is a block diagram showing the principle of the present invention. In FIG. 1, 1 is a processing device, 10 is a sentence identification unit, 11 is a table storing the number of sentences for each sentence and data generated during the course of processing, 12 is a natural language processing unit, and 13 is a timer unit , 14 is a table setting unit, 15 is an end time prediction unit, 15a is an end time calculation unit for each number of characters with actual data, 15b is an end time calculation unit for each number of characters without actual data, 15c is a total end time unit, and 2 is natural language processing. A document storage unit 3 in which a document to be processed is stored and a processing result is also stored is a display device.

The present invention measures the number of characters in each sentence for all texts when batch processing a long text including many natural language sentences, and sets the number of sentences for each number of characters in a table. Each time the natural language processing of one sentence is completed, the processing time is registered in the table as the processing time corresponding to the number of characters, and the end time and processing of each sentence by the number of characters processed The end processing time is obtained and displayed by calculation including prediction for the number of characters for which no is performed.

[0011]

The sentence identification unit 10 of the processing device 1 sequentially retrieves the data of all the documents from the document storage unit 2 in which the natural language is stored, and includes the data for each sentence (character string delimited by a period, a period, etc.). The number of characters is identified, and the contents of the table 11 are updated based on the identification result. In the table 11, positions for storing the number of sentences and other data are provided for each sentence according to the number of characters of each sentence (not for each character but for a number of characters having a fixed width). The number is set so as to be sequentially incremented (+1).
When the identification of all the sentences in the document storage unit 2 is completed in this way, the table 11 stores all the number of sentences of each sentence of the entire document.

Thereafter, the natural language processing section 12 starts processing. The natural language processing unit 12 measures the time from the start to the end of processing for one sentence by the timer unit 13, and when the processing ends, the number of characters of the processed sentence and the processing time obtained from the timer unit 13 are stored in a table. It is supplied to the setting unit 14. The table setting unit 14 stores the result data corresponding to the number of characters in the table 11. The result data includes the number of processed sentences and the accumulated processing time. If there is previous result data, the content is updated with the current data.

Next, the end time prediction unit 15 is started. The end time predicting unit 15 first drives the end time calculating unit 15a for the number of characters having actual data with reference to the table 11 for the number of characters having actual data, and calculates the number of unprocessed sentences using the actual data in the table 11. Calculate the time required to process. Next, when there is the number of characters for which the actual data in the table 11 is not set, the end time calculating unit 15b for each character without actual data uses the actual data of the end time calculating unit 15a for each character with actual data to calculate the number of characters. Calculate the time to process the number of each sentence. When the processing of each of the calculation units 15a and 15b ends, the end time totaling unit 15c adds the respective calculation results, and the addition result is displayed on the display device 3.

[0014]

FIG. 2 is a flowchart of the embodiment, and FIGS. 3 and 4 show specific examples (part 1) and (part 2) of tables whose contents are set and changed according to processing.

The flowchart of FIG. 2 is executed by a processing unit (corresponding to 1 in FIG. 1) constituted by a CPU and a memory, and is previously stored in a file device such as a hard disk (corresponding to the document storage unit 2 in FIG. 1). A natural language document is stored. (1) Initial processing First, the number of sentences in the entire document is obtained. In this case, a delimiter of a sentence is detected, the sentence is classified according to the number of characters included therein, and a table is created (S1 in FIG. 2).

In this processing, the document data is sequentially taken out from the file device, a sentence break is identified (a predetermined period, a period, etc.), and the number of characters constituting one sentence is counted. Table (1 in FIG. 1)
An operation of setting the number of sentences corresponding to the number of characters of one sentence (corresponding to 1) is performed.

When one sentence is identified, the same logic (same criterion) as that of one sentence in natural language processing is used. In counting the number of characters, white space is not counted.
Katakana character strings are treated as one character, and the number of characters is calculated as the number of words in English and European languages.

Furthermore, when a table is created, its configuration (the number of classes, the range of the number of characters for each class, etc.) is dynamically set. In other words, many sentences included in the entire document are identified, and the maximum number of characters and the minimum number of characters included in one sentence can be determined. From the numerical values, the optimal number of each type in the table and the number of characters included in the range of each type are determined. I do.

FIG. In the specific example of the table set by the initial processing shown in, the maximum number of characters in one sentence is 80.
Characters, since the minimum is 10 characters or less, the classification of the number of characters contained in one sentence (indicated by) is 0 to 10, 11 to 2
This is an example of classification into eight groups of 0,..., 71 to 80. As a result of identifying all the documents, as shown in FIG. Is set. For example, there are 30 sentences having a character number of 0 to 10 characters. FIG. The number of processed sentences (indicated by) and the accumulated processing time (indicated by) provided as other items are set when the natural language processing is executed. (2) Natural language processing When the classification of all the sentences in the document is completed, it is determined whether there is an original text to be processed (S2 in FIG. 2). If there is no original text, the processing is ended. If there is, the natural language processing is started. You. First, the processing start time is recorded for one sentence to be processed (S3), and natural language processing (for example, translation processing) is performed (S4). When the processing ends, the processing end time is recorded (S5). Next, the number of processed sentences (indicated by) corresponding to the number of characters in the table (indicated by) is counted up (6), and the processing time is detected by calculating the difference between the processing end time and the processing start time. The processing cumulative time according to the number of characters is recorded (S7).

FIG. Is an example of recording the number of processed sentences corresponding to the number of characters of the processed sentence.
This represents a state in which only one sentence of a 60-character sentence is processed, and the number of processed sentences indicated by in the table is accumulated (added) each time a sentence included in the same number of characters is processed.

C. of FIG. Is an example of recording of the accumulated processing time corresponding to the number of characters of the processed sentence. (A sentence having 51 to 60 characters) is set as the accumulated processing time indicated by the processing time (25 seconds). This time is accumulated (added) each time a sentence with the same number of characters is processed.

In step S3, the processing time is recorded (accumulated from the second time) corresponding to the number of characters. Apart from this, the processing time is recorded for each processed sentence, and the sample group table is stored. You may make it create. in this case,
If there is a sample of the processing time for each sentence that is statistically effective during the processing, replace the statistical abnormal value of the processing time for one sentence with the average value and calculate the accumulated processing time to obtain a more accurate processing time. Prediction is also possible. (3) Prediction of end time for each number of characters The processing end time of an unprocessed sentence is predicted and calculated for each number of characters assuming that it is proportional to the number of processed sentences.

A. Prediction in the case where there is actual data First, the processing end time of the remaining sentences of the number of characters is calculated and set based on the number of characters having actual data in proportion to each character (S8 in FIG. 2).

As shown in FIG. In the example of the table, since the actual processing time of one sentence of 51 to 60 characters is 25 seconds, the time required to process the remaining 10 sentences in the total sentence number 11 having this number of characters is directly proportional. As thing 2
5 (seconds) × 10 = 250 (seconds) is obtained by calculation.
This value is held as a predicted value in the memory of the processing device as a predicted time of the end of processing of 51 to 60 characters.

FIG. Shows the result of the prediction process for the number of characters having actual data. b. Prediction when there is no actual data Next, for the number of characters that have no actual data, the processing end time for each number of characters is proportionally predicted from other actual data (S9 in FIG. 2). In other words, when there is no actual data in the number of characters, it is assumed that the processing time is directly proportional to the number of characters, and the prediction calculation is performed from other actual data. However, when there are a plurality of other result data, the calculation is performed using the result data having the closest number of characters.

In FIG. If the number of characters is 51 to 60 characters, the average of one sentence is 25 seconds (in this case, the result data of only one sentence, but if the result data of a plurality of sentences is obtained, this number is averaged) ), For other numbers of characters, a prediction time is obtained for the remaining number of sentences for each number of characters by calculation in direct proportion as shown in the following three examples.

Number of characters 71 to 80 25 seconds × 5 sentences × 80/60 = 1
66 seconds 61-70 characters 25 seconds × 10 sentences × 70/60 = 2
91 seconds Number of characters 41-50 25 seconds x 3 sentences x 50/60 =
The predicted time is obtained by the same calculation for the other character counts of 62 seconds. As shown in the figure, the prediction time for the total number of characters is obtained. (4) Display of predicted end time The process end time for each character is totaled to predict the process end time of the entire document (S10 in FIG. 2). Next, the total obtained time is displayed on the display device as the predicted processing time (S
11). In the example of FIG. 7, the predicted time for each number of characters is added, and the total is displayed on the display device (3 in FIG. 1).

In the processing flow of FIG. 2, if the next sentence exists after step S10, the next sentence is sent to S4 to S1.
When the process 0 is executed and the actual data (the number of processed statements and the accumulated processing time) increases, the prediction time is corrected according to the accumulation of the actual data, and the accuracy increases.

FIG. The correction of the predicted time when actual data is obtained for another number of characters after the predicted time of FIG. This will be described with reference to an example of correction of the estimated time shown in FIG.

FIG. Is B. in FIG. In this example, one sentence having 61 to 70 characters is processed after the predicted time shown in FIG. 1 is obtained, and the processing time is 33 seconds. in this case,
Since the processing of 61 to 70 characters as result data takes 33 seconds, the time required for processing the remaining nine sentences having the same number of characters is 9 × 33 = 297 seconds. This time corresponds to B. in FIG. Since it is different from the predicted value of (291 seconds), it is corrected to 297 seconds.

As described above, as actual data is generated for each number of characters as the processing proceeds sequentially, the respective predicted times are corrected, and when the number of processed sentences for the same number of characters increases, the accumulated processing time becomes longer. It is clear that the average processing time per sentence approaches the actual processing time.

Therefore, at the point of time shortly after the start of the natural language processing, the accuracy of the prediction time is not sufficient, but the user can be notified of the approximate time until the end, and as the processing progresses, the accuracy of the prediction time increases. High end times can be signaled.

[0033]

Processing time documents a large amount of natural language, according to the equipment of the present invention according to the present invention is, it is possible to predict with high accuracy by using the number of characters constituting the text in the document, the user is expected There is no need to wait in vain for the processing end time, and language processing can be performed efficiently.

[Brief description of the drawings]

FIG. 1 is a principle configuration diagram of the present invention.

FIG. 2 is a diagram showing a flowchart of the embodiment.

FIG. 3 is a diagram illustrating a specific example (part 1) of a table whose contents are set and changed according to processing;

FIG. 4 is a diagram showing a specific example (part 2) of a table whose contents are set and changed according to processing;

FIG. 5 is an explanatory diagram of a conventional example.

[Explanation of symbols]

 Reference Signs List 1 processing device 10 sentence identification unit 11 table 12 natural language processing unit 13 timer unit 14 table setting unit 15 end time prediction unit 15a end time calculation unit by number of characters with actual data 15b end time calculation unit by number of characters without actual data 15c total end time Unit 2 document storage unit 3 display device

Continuation of the front page (56) References JP-A-61-84779 (JP, A) JP-A-62-229471 (JP, A) JP-A-62-241069 (JP, A) JP-A-63-109575 (JP) JP-A-63-220364 (JP, A) JP-A-63-284677 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G06F 17/21-17/28

Claims (2)

(57) [Claims] 1. A processing device for batch processing natural languages.
The document storage unit in which the natural language document is stored and the display device are connected.
And the processing device is configured to store all natural language document data in the document storage unit.
For data, identify the number of characters that make up the sentence
And a natural language process for each sentence in the document storage unit.
The processing time is measured, and the characters in the table
Natural language processing unit set as actual data corresponding to the number
When being activated every completion of processing the sentence, sets the table
Using the actual data for each number of characters,
Calculate the processing end time for the number of unprocessed statements and calculate the processing end time
And an end time prediction unit for predicting
Prediction device for processing end time in batch processing of natural language. 2. The natural language processing unit according to claim 1, wherein the natural language processing unit stores actual data in the table.
The number of processed statements and the total processing time corresponding to the number of characters
Set the product value, and the end time prediction unit corresponds to each number of characters in the table
Sentence that has actual data
Processing the number of unprocessed sentences using the actual data for the number of characters
End time calculation unit for each number of characters with actual data to calculate time
And the number of characters for which there is no actual data
Data that calculates the processing time for the number of unprocessed statements using
And an end time calculating unit for each character.
Prediction device for processing end time in batch processing of natural language.