JP7304772B2 - Image analysis device, image analysis method, and program - Google Patents

Description

The present invention relates to an image analysis device, an image analysis method, and a program.

In recent years, various techniques have been proposed for recognizing a character area in a document image by recognizing characters in the document image.

For example, Patent Document 1 below discloses a technique for recognizing a word region in a document image. In this technology, a word region is recognized by blurring a document image, combining adjacent characters, and extracting the combined portion.

Japanese Patent No. 5522408

The technique disclosed in Japanese Patent Laid-Open No. 2002-200310 can recognize the area of each word in the document image, and is particularly effective when performing processing on a word-by-word basis. However, the art does not consider recognizing paragraphs. Therefore, for example, when it is desired to process by paragraph, it is difficult to recognize the paragraph and perform the process.

SUMMARY OF THE INVENTION In view of the above problems, an object of the present invention is to provide an image analysis apparatus, an image analysis method, and a program capable of easily setting paragraphs in a document image.

To solve the above-described problems, an image analysis apparatus according to an aspect of the present invention provides a blurred image including a second object obtained by processing a first object included in the document image by blurring processing on the document image. a block setting unit that sets a block indicating a rectangular region containing one of the second objects in the blurred image; and based on the block set in the blurred image: and a determination unit that determines the type of the document image to be determined .

An image analysis method according to an aspect of the present invention includes generating a blurred image including a second object obtained by processing a first object included in the document image by blurring a document image; , setting a section indicating a rectangular area including one of the second objects , and determining the type of the document image to be determined based on the section set in the blurred image. and are executed by a processor.

A program according to an aspect of the present invention comprises a computer, an image processing unit that generates a blurred image including a second object obtained by processing a first object included in the document image by blurring processing on the document image; a section setting unit for setting a section indicating a rectangular area including one of the second objects for the blurred image; It functions as a determination unit that determines the type .

According to the present invention, it is possible to easily set paragraphs in a document image.

1 is a diagram showing a configuration example of an image analysis system according to an embodiment of the present invention; FIG. It is a figure which shows the example of the document image which concerns on the same embodiment. It is a figure which shows the example of the blurred image which concerns on the same embodiment. It is a figure which shows the setting example of the division with respect to the blurred image which concerns on the same embodiment. It is a figure which shows the example of a synthesis|combination of the division with respect to the document image which concerns on the same embodiment. FIG. 10 is a diagram showing a setting example of partitions for a blurred image when the degree of blurring is increased according to the same embodiment; FIG. 10 is a diagram showing a setting example of partitions for a blurred image when the degree of blurring is weakened according to the same embodiment; It is a flowchart which shows the flow of a process in the image-analysis apparatus which concerns on the same embodiment.

<Configuration of image analysis system>
BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a diagram showing a configuration example of an image analysis system 1 according to an embodiment of the present invention. As shown in FIG. 1, the image analysis system 1 includes an image analysis device 10, a similarity calculation server 20, and a network 30. FIG. The image analysis apparatus 10 and the similarity calculation server 20 are connected by a network 30 so that they can transmit and receive information to each other.

The image analysis device 10 is a device that performs processing for determining the type of image (hereinafter also referred to as “determination processing”). The image analysis device 10 is implemented by a terminal such as a PC (Personal Computer), a tablet terminal, or a smart phone, for example.

The image analysis device 10 determines, for example, the type of document image. A document image is an image of a document composed of objects (first objects) such as characters, lines, symbols, graphics, tables, and images. The document may be a document with a multi-column structure. Examples of document image types include articles, magazines, catalogs, and newspapers. Note that the configuration of the document image, the configuration of the document, and the type of the document image are not limited to these examples.

The image analysis apparatus 10 generates a blurred image by performing a blurring process on a document image to be determined, and sets sections for the blurred image. Here, a section is a rectangular area that includes one processed object (second object) obtained by processing an object included in a document image by blurring processing. The processed object is, for example, an object blurred by blurring, or an object formed by combining a plurality of objects into one due to blurring. After setting the partitions, the image analysis device 10 transmits the blurred image with the partitions set to the similarity calculation server 20 via the network 30 .

The similarity calculation server 20 is a device that performs processing for calculating the similarity of images (hereinafter also referred to as “similarity calculation processing”). The similarity calculation server 20 is implemented by a computer such as a server device, for example.

The similarity calculation server 20 calculates the similarity of images by, for example, a machine learning technique. For example, the similarity calculation server 20 calculates the similarity of images using a trained model. A trained model is a model that has learned the relationship between the layout of the section and the type of the document image.

The learning method of the trained model is, for example, supervised learning. A trained model is generated by training a learning model such as a CNN (Convolutional Neural Network) using a training data set. A CNN is a model composed of three layers: an input layer, a hidden layer (hidden layer), and an output layer. Data (input data) to be estimated by the CNN is input to the input layer. The output layer outputs data (estimation results) indicating the results of learning by the CNN. The hidden layer performs the core processing of learning. For example, a hidden layer converts an input into a value expressed by a function called an activation function (transfer function) and outputs the value. For example, the activation function is a normalized linear function, a sigmoid function, a step function, etc., but is not limited to these, and any function may be used.

In CNN, when data is output from a unit in a certain layer to a unit in a deeper layer, data to which a weight W according to the coupling coefficient of the node connecting the units and a bias component b are added is output. be. The learning model performs operations between units on input data (input data), and outputs output data from the output layer.

The learning data set in the present embodiment is information that associates the layout of sections as an input with the type of document image as an output.

In the process of learning, the learning model is fed with input data of a training data set. The learning model adjusts the parameters (weight W and bias component b) of the learning model so that the data output from the output layer with respect to the input data (output data) approaches the output of the data set for learning. to train the learning model.

For example, error backpropagation is used to adjust the parameters (weight W and bias component b) of the learning model. In the error backpropagation method, the degree of divergence between the data output from the output layer of the learning model and the output of the learning data and set is derived as a loss function. Any index may be used for the degree of divergence here, but for example, the square of the error (square error), cross entropy, or the like is used. In the error backpropagation method, the combination of the weight W and the bias component b is determined (updated) so that the loss function is minimized in the direction from the output layer to the input layer. This makes the learning model learn and improves the accuracy of estimation.

Note that the learning model is not limited to CNN. Techniques such as decision trees, hierarchical Bayes, and SVM (Support Vector Machine) may be used as learning models, for example.

The similarity calculation server 20 inputs the blurred image received from the image analysis device 10 to the learned model, thereby acquiring the output (type and similarity of the document image) from the learned model. The output from the trained model is the similarity with each type of document image, for example, "80% similarity to articles, 50% similarity to magazines, 8% similarity to newspapers". is information indicating Note that the degree of similarity here is the degree of similarity between the layout of the blocks in the blurred image to be determined and the layout of the blocks used for learning by the trained model.

The similarity calculation server 20 transmits the output from the trained model to the image analysis device 10 . Then, the image analysis apparatus 10 determines the type of document image to be determined based on the received information indicating the degree of similarity. For example, the image analysis apparatus 10 determines the type of document image showing the highest degree of similarity as the type of document image to be determined.

It should be noted that the trained model according to the present embodiment is stored in a storage medium included in the similarity calculation server 20, but the storage destination of the trained model is not limited to this example. For example, the trained model may be stored in a storage medium included in the image analysis device 10. Thereby, the image analysis device 10 can obtain the similarity using the trained model stored in the storage medium of the device without accessing the similarity calculation server 20 .

<Configuration of image analysis device>
Here, the configuration of the image analysis device 10 will be described in more detail. As shown in FIG. 1 , the image analysis apparatus 10 includes an image acquisition section 110 , a control section 120 , a communication section 130 , a storage section 140 and an output section 150 .

(Image acquisition unit 110)
The image acquisition unit 110 has a function of acquiring an image to be subjected to determination processing. For example, the image acquisition unit 110 acquires a document image as a target of determination processing, and outputs the acquired document image to the control unit 120 . Note that the image acquisition unit 110 may output the acquired document image to the output unit 150 .

The image acquisition unit 110 can acquire document images by various means. For example, the image acquisition unit 110 acquires document images stored in the storage unit 140 . Further, when the image analysis apparatus 10 has an imaging device, the image acquisition unit 110 may acquire an image of a printed document or the like captured by the imaging device as the document image. Also, the image acquisition unit 110 may acquire the document image from an external device. As an example, when a scanner device is connected to the image analysis device 10 as an external device, the image acquisition unit 110 acquires an image generated by scanning a printed document or the like with the scanner device as a document image. Note that the external device may be connected to the image analysis device 10 either wired or wirelessly. Also, the image acquisition unit 110 may acquire the document image from a web service or the like via the network 30 .

Here, an example of a document image will be described with reference to FIG. FIG. 2 is a diagram showing an example of a document image 40 according to an embodiment of the invention. A document image 40 shown in FIG. 2 is an image of a two-column article. In the paper, the title of the paper is described at the top, "The xxth National Convention of the Society of XXXX", and the text of the paper is described in two columns under the title with a ruled line between them. In the first row of the text, the heading "1.****" and its text, the heading "2.****" and its text, and the title of Table 1 and its table are described. there is In the second row of the text, the title of Table 2 and its table, the text with the heading “2.****” continuing from the first row, and the heading “3.****” and The text is provided.

In the following, an example in which the document image 40 of the two-column paper shown in FIG. 2 is the target of the determination process will be described.

(control unit 120)
The control unit 120 has a function of controlling the overall operation of the image analysis apparatus 10 . The control unit 120 is implemented, for example, by causing a CPU (Central Processing Unit) included as hardware in the image analysis apparatus 10 to execute a program. To realize this function, the control unit 120 includes an image reading unit 1202 , an image processing unit 1204 , a section setting unit 1206 , a section synthesizing unit 1208 , a determination unit 1210 and a post-processing unit 1212 .

(Image reading unit 1202)
An image reading unit 1202 has a function of reading a document image. For example, the image reading unit 1202 reads a document image input from the image acquisition unit 110 and outputs the read document image to the image processing unit 1204 . When reading a document image, the image reading unit 1202 converts the document image to grayscale. Note that the image reading unit 1202 may read the document image as a color image.

(Image processing unit 1204)
The image processing unit 1204 has a function of processing document images. For example, the image processing unit 1204 processes the document image input from the image reading unit 1202 and outputs the processed document image to the section setting unit 1206 . Note that the image processing unit 1204 may output the processed document image to the output unit 150 . As an example of processing, the image processing unit 1204 applies blurring processing and binary conversion processing to the document image.

First, the image processing unit 1204 blurs the document image. The blurring process is a process of blurring the processing target. The image processing unit 1204 generates a blurred image by blurring the document image by blurring processing. Blur processing is performed using, for example, a smoothing filter. A smoothing filter is, for example, a Gaussian filter. The Gaussian filter averages the pixel values of the pixels in a predetermined area centered on one focused pixel (focused pixel) among the pixels of the image by weighting them according to the distance from the focused pixel. The size of the predetermined area can be set to, for example, 3 pixels×3 pixels or 5 pixels×5 pixels. By averaging the pixel values of the pixels in a predetermined area centering on the pixel of interest, the change in color density in the predetermined area becomes smooth, resulting in a blurry image.

If a Gaussian filter is applied to a pixel representing an object at the boundary between a pixel representing an object and a pixel representing something other than the object in the document image, the pixel value of the pixels in a predetermined area centered on the target pixel is averaged. At this time, if the pixel values of the pixels that indicate objects other than the object contained in the predetermined area become close to the pixel values of the pixels that indicate the object due to averaging, the pixel values that are close to the pixel values of the pixels that indicate the object are used. As the number of certain pixels increases, the size of the object increases. Accordingly, when the image processing unit 1204 performs the blurring process on the document image, the size of the object in the document image can be made larger than the size before the blurring process. The image processing unit 1204 enlarges the size of the object by the blurring process, so that the object becomes larger from the pixel indicating the object toward the outside of the object. Therefore, when a plurality of objects are arranged adjacent to each other, the image processing unit 1204 can combine the adjacent objects by blurring processing.

Blur processing can generate various processed objects that are processed objects in the document image. For example, when a character string is blurred, a processed object is generated in which adjacent characters are combined. Also, when multiple lines of text are blurred, a processed object is generated in which adjacent lines are combined. Further, when the table is blurred, a processed object is generated in which the ruled lines indicating the frame of the table and the character strings in the table are combined. Further, when the ruled lines are blurred, a processed object is generated in which the ruled lines are blurred and thickened.

In the blurring process, the degree of blurring indicating the intensity of blurring is set so as to be adjustable. For example, the degree of blurring is desirably set so that multiple objects in the document image are properly combined. Appropriately connecting here means connecting a plurality of objects so that each processing object has a meaning. For example, if the document image is a thesis, the degree of blurring is set so as to generate meaningful processed objects such as the title, headline, and text of the thesis.

In addition, in the case of a document image including sentences such as a paper, the degree of blurring is set according to the distance between sentences, so that the processing objects can be appropriately divided into groups of sentences. An appropriate group of sentences here is, for example, a paragraph. The distance between sentences differs between, for example, one line break and two line breaks.

For example, when it is desirable to generate a processing object in which a plurality of paragraphs are separated from each other, the degree of blurring is determined according to the distance between the plurality of paragraphs (between two lines of text sandwiching the boundary between the paragraphs). Set to separate objects. For example, if the distance between paragraphs is short, multiple paragraphs may join together if the degree of blurring is too strong. Therefore, the degree of blurring is set to be strong enough not to merge two paragraphs. On the other hand, if the distance between paragraphs is long, there is a possibility that multiple paragraphs will not be merged even if the degree of blurring is strong to some extent. Thus, the degree of blurring may be set as strong as possible to the extent that two paragraphs do not merge.

When it is desirable to generate a processed object in which multiple paragraphs are combined, the degree of blurring may be set so that the multiple paragraphs are combined according to the distance between the multiple paragraphs. For example, if the distance between paragraphs is short, multiple paragraphs may be merged even if the degree of blurring is somewhat weak. Therefore, the degree of blurring may be set weakly so that the two paragraphs are not separated. On the other hand, if the distance between paragraphs is long, the paragraphs may not join together if the degree of blurring is too weak. Therefore, the degree of blurring is set strong enough to combine multiple paragraphs.

Also, the degree of blurring is desirably set so that the meaning of the blurred character string is not recognized. Specifically, the degree of blurring is set to such an extent that the user cannot understand the meaning indicated by the blurred character string when the user views the blurred image. As a result, even if the document image to be determined contains personal information, the personal information can be blurred to such an extent that it cannot be recognized, so that the personal information can be protected.

In addition, since the personal information is protected by the blurring process, it is possible to prevent the personal information from being leaked to the outside even if an external service such as a cloud service is used for the process using the blurred image. Therefore, it becomes easier for the user to use an external service for processing using a blurred image.

Note that the image processing unit 1204 initially sets the weakest degree of blurring. The weakest degree of blurring here is the degree that the meaning of the blurred character string is not recognized. When changing the degree of blurring, the image processing unit 1204 gradually increases the degree of blurring from the initial setting. For example, if the type of the document image is not determined as a result of the determination processing based on the blurred image generated with the default degree of blurring, the image processing unit 1204 sets the degree of blurring higher than the initial setting. do.

Here, an example of a blurred image will be described with reference to FIG. FIG. 3 is a diagram showing an example of a blurred image 42 according to an embodiment of the invention. A blurred image 42 shown in FIG. 3 is an image obtained by blurring the document image 40 shown in FIG. In the blurred image 42, blurred characters and symbols are indicated by black circles. Also, the ruled lines such as the frame lines of the blurred table are shown thicker than before the blurring process.

In the blurred image 42 shown in FIG. 3, the title of the article at the top shown in FIG. 2 is combined with a plurality of characters to form one processed object. The ruled line under the title is a blurred and thickened processed object. The headline "1.****" in the first row of the text and its text are each divided into two processing objects by combining a plurality of characters. The headline "2.****" in the first row of the text and its text are each divided into two processing objects by combining a plurality of characters. The title of Table 1 in the first row of the text and its table form one processed object by combining the characters of the title, the characters in the table, and the ruled lines of the table. The title of Table 2 in the second row of the text and its table form one processed object by combining the characters of the title, the characters in the table, and the ruled lines of the table. In the second row of the text, the text with the heading "2.****" continuing from the first row is separated from the first row of text, and multiple characters are combined to form one processing object. It's becoming The headline "3.****" in the second row of the text and its text are each divided into multiple characters that are combined into processed objects, and the text is further divided into multiple characters that are combined. are three machining objects.

After the blurring processing, the image processing unit 1204 performs binary conversion processing on the blurred image. Binary conversion processing is processing for converting an image to be processed into two gradations of white and black. For example, the image processing unit 1204 converts the blurred image into two gradations of white and black based on the pixel value of each pixel of the blurred image. Specifically, the image processing unit 1204 sets a value (for example, 1) indicating black to a pixel whose pixel value is equal to or greater than a predetermined threshold value, and sets a value indicating white to a pixel whose pixel value is less than the predetermined threshold value. (eg 0) is set. In the case of the blurred image 42 shown in FIG. 3, 1 is set to pixels that have been blurred and turned black, and 0 is set to other white pixels. The image processing unit 1204 then outputs the blurred image after the binary conversion processing to the partition setting unit 1206 .

(Section setting unit 1206)
The partition setting unit 1206 has a function of setting partitions for the blurred image. For example, the block setting unit 1206 sets blocks for the blurred image input from the image processing unit 1204 and outputs the block-set blurred image to the block synthesizing unit 1208 and the determination unit 1210 . Note that the partition setting unit 1206 may output the blurred image with the partitions set to the output unit 150 .

Specifically, the partition setting unit 1206 sets partitions based on the pixel values of the binary-converted blurred image input from the image processing unit 1204 . First, the section setting unit 1206 detects a processing object in the blurred image based on the pixel values set for each pixel of the blurred image. For example, pixels whose pixel value is set to 1 (i.e., black pixels) constitute the processing object, and pixels whose pixel value is set to 0 (i.e., white pixels) constitute non-processing objects. It is a pixel that After detection, the partition setting unit 1206 sets partitions for the detected processing object.

When setting the partition, the partition setting unit 1206 extracts object information indicating object information for each processing object for which the partition is set. The object information is, for example, information indicating the type of processed object such as header, footer, title, headline, text, table, or graphic. The partition setting unit 1206 determines the type of object based on, for example, the shape, size, area, and position information of the partition, and extracts object information.

In addition, the section setting unit 1206 performs labeling processing on the set section. The labeling process is a process of adding labeling information to the set partitions. The labeling information includes, for example, the partition number, size, area, position information, extracted object information, and the like.

Here, an example of a blurred image in which sections are set will be described with reference to FIG. 4 . FIG. 4 is a diagram showing a setting example of partitions for a blurred image according to the embodiment of the present invention. In the blurred image shown in FIG. 4, the division is indicated by a rectangular frame of two-dot chain lines.

In the blurred image 44 shown in FIG. 4, a section is set for each processing object determined in the blurred image 42 shown in FIG. Specifically, divisions are set for each of the title of the paper and the ruled line under the title. In addition, a section is set for each of the headline "1.****" in the first row of the text and the text. In addition, a section is set for each of the headline "2.****" in the first row of the text and the text. In addition, a section is set for the title of Table 1 in the first row of the text and the processed object that combines the table. In addition, a section is set for the title of Table 2 in the second row of the text and the processing object that combines the table. Also, in the second row of the text, a section is set for the text of the headline "2.****" continuing from the first row. In addition, sections are set for each of the headline "3.****" in the second row of the text and the three texts.

Note that in the present embodiment, a simple rectangular section is set for the blurred image regardless of the shape of the processing object. If a complex shaped section corresponding to the shape of the object to be processed is set in the blurred image, the trained model that performs processing using the blurred image needs to perform processing that takes into account the complex shape. Processing becomes heavy. On the other hand, the simpler the shape of the parcel set in the blurred image, the less the trained model can consider the shape of the parcel. Therefore, by making the shape of the partition rectangular regardless of the shape of the processing object, it is possible to reduce the processing load on the learned model.

(Section combining unit 1208)
The section synthesizing unit 1208 has a function of synthesizing a section frame indicating a section set in the blurred image with the document image. For example, the section synthesizing unit 1208 synthesizes a section frame with the document image based on the blurred image with the section set input from the section setting unit 1206 and outputs the synthesized document image to the output unit 150 . As a result, the section set as the blurred image is visualized on the document image, and the user can visually recognize the section. Note that the section synthesizing unit 1208 may synthesize not only the section frame but also the labeling information and the object information with the document image.

Here, an example of a document image synthesized with partition frames will be described with reference to FIG. FIG. 5 is a diagram showing an example of synthesizing sections with respect to a document image according to an embodiment of the present invention. In the document image shown in FIG. 5, the sections are indicated by solid-line rectangular frames.

A document image 46 shown in FIG. 5 is obtained by synthesizing solid-line division frames at the same positions as the divisions indicated by two-dot chain lines in the blurred image 44 in FIG. It is an image in which labeling information indicating the number of the partition is synthesized near the partition frame. As an example, “No. The partition frame shows "No. 7" and the number of each partition.

(Determination unit 1210)
The determination unit 1210 has a function of determining the type of document image. For example, the determination unit 1210 determines the type of the document image based on the division set in the blurred image input from the division setting unit 1206 and outputs the determination result to the post-processing unit 1212 .

For example, first, the determination unit 1210 uses the learned model to acquire the degree of similarity between the layout of the section set in the blurred image and the layout of the section used for learning by the learned model. In this embodiment, the trained model is stored in the similarity calculation server 20 . Therefore, the determination unit 1210 causes the communication unit 130 to transmit the blurred image with the partitions set to the similarity calculation server 20 when using the learned model. The similarity calculation server 20 calculates the similarity by inputting the received blurred image to the learned model, and transmits the calculated similarity to the image analysis device 10 via the network 30 . After receiving the degree of similarity from the degree of similarity calculation server 20 , the communication unit 130 outputs the received degree of similarity to the determination unit 1210 . Thereby, the determination unit 1210 can acquire the degree of similarity.

After obtaining the degree of similarity, the determination unit 1210 determines the type of the document image to be determined based on the obtained degree of similarity. For example, the determination unit 1210 determines the type of the document image by performing threshold determination on the acquired similarity.

If there is a degree of similarity equal to or greater than a predetermined threshold among the obtained degrees of similarity, the determination unit 1210 selects the type of the document image corresponding to the maximum similarity among the degrees of similarity equal to or greater than the predetermined threshold as the type of the document image to be determined. Determined as the type of document image. For example, assume that the predetermined threshold is 50%, the degree of similarity to articles is 80%, the degree of similarity to magazines is 75%, and the degree of similarity to newspapers is 8%. In this case, the determining unit 1210 determines that the paper with the maximum similarity of 50% or more is the document image type to be determined.

If there is no degree of similarity equal to or greater than a predetermined threshold among the obtained degrees of similarity, the determination unit 1210 re-obtains the degree of similarity using a blurred image generated by changing the degree of blurring, and calculates the re-obtained degree of similarity. The type of the document image to be determined is determined based on the degree. For example, assume that the predetermined threshold is 50%, the degree of similarity to articles is 40%, the degree of similarity to magazines is 25%, and the degree of similarity to newspapers is 8%. In this case, the determination unit 1210 assumes that the type of the document image to be determined cannot be determined, and redoes the determination processing from the generation of the blurred image.

At this time, the image processing unit 1204 changes the degree of blurring and regenerates the blurred image. Specifically, the image processing unit 1204 makes the degree of blurring after the change stronger than the degree of blurring before the change. Next, the partition setting unit 1206 sets partitions in the regenerated blurred image. Then, the determination unit 1210 re-acquires the similarity using the blurred image in which the section is set, and re-determines the type of the document image to be determined based on the re-acquired similarity. Here, if the type of the document image to be determined cannot be determined, the processing is restarted from generation of the blurred image. As described above, when the type of the document image to be determined cannot be determined, the determination accuracy can be improved by repeating the determination process while changing the degree of blurring. If the type of the document image to be determined cannot be determined even after repeating the determination process a predetermined number of times, the determination unit 1210 may return an error.

Here, with reference to FIG. 6, an example of a blurred image in which sections are set by increasing the degree of blurring will be described. FIG. 6 is a diagram showing a setting example of partitions for a blurred image when the degree of blurring is increased according to the embodiment of the present invention. The diagram on the left side of FIG. 6 is a diagram showing a setting example of the partitions before the degree of blurring is changed. The diagram on the right side of FIG. 6 is a diagram showing a setting example of the partitions after the degree of blurring is changed.

Before the degree of gradation is changed, as shown in the left diagram of FIG. 6, section 51 of the headline and section 52 and section 53 of the body of the headline are set in two paragraphs, respectively. In this state, the degree of blurring is strongly changed to the extent that the paragraph in which the section 52 is set and the paragraph in which the section 53 is set are combined. As the degree of blurring increases, the object becomes more blurred, so the size of the black circle indicating the degree of blurring of the object also increases. Therefore, when the degree of blurring is changed strongly, as shown in the right side of FIG. 6, a section 54 that is larger than before the change in the degree of blurring is set in the headline. In addition, since the degree of blurring is strongly changed to the extent that two paragraphs of the text are combined, as shown in the diagram on the right side of FIG. One partition 55 is set.

(Post-processing unit 1212)
The post-processing unit 1212 has a function of executing processing according to the determination result of the document image type. The post-processing unit 1212 can execute various processes based on the determination result input from the determination unit 1210 .

The post-processing unit 1212 executes processing according to the type of document image, for example. The post-processing unit 1212 has, for example, a processing list in which types of document images and processing are associated with each other. When the judgment result is input from the judgment unit 1210, the post-processing unit 1212 refers to the judgment result and the processing list, selects from the list the type of document image indicated by the judgment result, and executes the corresponding processing. For example, if the type of the document image is an article, a process related to the article is executed, and if the type of the document image is a magazine, a process related to the magazine is executed.

As an example of processing, the post-processing unit 1212 executes OCR (Optical Character Recognition) processing. OCR processing is performed on the document image synthesized with the partition frame, and characters within the partition frame are recognized. At this time, the post-processing unit 1212 executes OCR processing for each partition frame. As a result, the post-processing unit 1212 can reduce the number of objects to be processed as compared with the case where the OCR processing is executed for each object, so that the cost required for the OCR processing can be reduced.

In addition, the post-processing unit 1212 executes OCR processing for each partition frame, so that the OCR processing can be executed without being affected by the structure of sentences in the document image (for example, writing direction, columns, etc.). can be done. As an example, when the text in the document image has a two-column configuration like the document image 46 in which sections are set as shown in FIG. , the sentence in the first row and the sentence in the second row can be recognized separately. Therefore, the post-processing unit 1212 can prevent the text in the first line and the text in the second line from being partially mixed in the processing result. In addition, since the post-processing unit 1212 can also recognize sentences in the second row after recognizing sentences in the first row, the sentences in the first row and sentences in the second row can be connected in the correct order. can also recognize.

In addition, since the OCR process is executed for each partition frame, the object to be recognized by the OCR process can be changed by adjusting the object included in the partition frame by changing the degree of blurring.

Here, with reference to FIG. 7, an example of changing the partition setting when recognizing characters in each cell of a table by OCR processing will be described. FIG. 7 is a diagram showing a setting example of partitions for a blurred image when the degree of blurring is weakened according to the embodiment of the present invention. The diagram on the left side of FIG. 7 is a diagram showing a setting example of the partitions before the degree of blurring is changed. The diagram on the right side of FIG. 7 is a diagram showing a setting example of the partitions after the degree of blurring is changed.

Before the degree of blurring is changed, as shown in the diagram on the left side of FIG. 7, one section 60 is created for one processing object in which the title of the table, the ruled line indicating the frame of the table, and the characters in the table are combined. is set. In this state, for example, in order to recognize characters in each cell in the table, it is necessary to set a partition for each processing object in each cell. Therefore, the degree of shading is changed to a weak degree so that the ruled lines indicating the frame lines of the table are separated from the characters in the table. As the degree of blurring becomes weaker, the object becomes smaller and blurry, so the size of the black circle indicating the degree of blurring of the object also becomes smaller. Therefore, by weakening the degree of blurring to such an extent that the ruled lines indicating the frame of the table and the characters in the table are separated, as shown in the right side of FIG. In contrast, partitions 61 to 64 can be set respectively.

Also, as an example of processing, the post-processing unit 1212 executes reading processing of sentences in the document image. The voice at the time of reading is output from the output unit 150, for example. In the reading process, for example, characters recognized by the post-processing unit 1212 through OCR processing are read out. Therefore, the post-processing unit 1212 can read out the sentences in an appropriate order even if the sentences in the document image are arranged in columns. For example, when the sentences in the document image are two-column sentences, the post-processing unit 1212 reads out the sentences in the second row after reading out the sentences in the first row.

Also, the post-processing unit 1212 may execute reading processing based on object information. For example, the post-processing unit 1212 refers to the object information set in the section, and reads aloud for each type of processed object. Specifically, the post-processing unit 1212 first reads out all headlines and then reads out the text. Further, whether or not to read aloud may be set according to the type of the processed object. For example, assume that all pages of a document image consisting of multiple pages have the same header and footer. In this case, whether or not to read out the header and footer may be set to none. This prevents the post-processing unit 1212 from reading out the same header and footer on every page. Therefore, by setting whether or not to read aloud for each type of processing object, redundant reading by the post-processing unit 1212 can be prevented.

(Communication unit 130)
The communication unit 130 has a function of communicating with an external device. Communication unit 130 transmits information input from control unit 120 to an external device. As an example, the communication unit 130 transmits the blurred image in which the divisions input from the control unit 120 are set to the similarity calculation server 20 via the network 30 . Communication unit 130 outputs information received from an external device to control unit 120 . As an example, the communication unit 130 outputs the similarity received from the similarity calculation server 20 via the network 30 to the control unit 120 .

(storage unit 140)
The storage unit 140 has a function of storing various information. The storage unit 140 includes a storage medium such as a HDD (Hard Disk Drive), a flash memory, an EEPROM (Electrically Erasable Programmable Read Only Memory), a RAM (Random Access read/write memory), a ROM (Read Only Memory), or any of these Consists of any combination of storage media. A non-volatile memory, for example, can be used for the storage unit 140 .

Note that the storage unit 140 may store a learned model as an example of various information. Accordingly, the determination unit 1210 can acquire the similarity without accessing the similarity calculation server 20 by accessing the storage unit 140 and using the learned model.

(Output unit 150)
The output section 150 has a function of outputting according to the input from the control section 120 . The output unit 150 is realized by, for example, a display device such as a liquid crystal panel or a touch panel, or an audio output device such as a speaker.

For example, the output unit 150 can display various images input from the control unit 120 . For example, output unit 150 displays a document image input from image acquisition unit 110 . Also, the output unit 150 displays the blurred image input from the image processing unit 1204 . Also, the output unit 150 displays a blurred image in which the divisions input from the division setting unit 1206 are set. Also, the output unit 150 displays the document image in which the partition frames input from the partition synthesizing unit 1208 are synthesized. Thereby, the user can visually recognize the image on which various processes have been performed.
Also, the output unit 150 may output the voice input from the control unit 120 .

<Process flow>
The configuration example of the image analysis apparatus 10 has been described above. Next, the flow of processing in the image analysis apparatus 10 according to this embodiment will be described. FIG. 8 is a flow chart showing the flow of processing in the image analysis device 10 according to the embodiment of the present invention.

First, the image reading unit 1202 of the image analysis apparatus 10 reads the document image acquired by the image acquiring unit 110 in gray scale (S102).

Next, the image processing unit 1204 blurs the document image read by the image reading unit 1202 to generate a blurred image (S104). After the blurring process, the image processing unit 1204 performs binary conversion process on the blurred image (S106).

Next, the partition setting unit 1206 sets partitions for the binary-converted blurred image (S108). After setting the partitions, the partition setting unit 1206 performs labeling processing on the blurred image with the partitions set (S110).

Next, the section synthesizing unit 1208 synthesizes a section frame indicating a section with the document image based on the blurred image in which the section is set (S112). The document image combined with the partition frame is displayed on the output unit 150 so as to be visible to the user, for example.

Next, the determination unit 1210 performs determination processing. First, the determination unit 1210 transmits the blurred image with the division set to the similarity calculation server 20, and acquires the similarity (S114). After obtaining the degree of similarity, the determination unit 1210 performs threshold determination on the obtained degree of similarity (S116).

If there is a degree of similarity equal to or greater than the threshold (S116/YES), the determination unit 1210 determines the type of document image corresponding to the maximum degree of similarity as the type of document image to be determined. Then, the post-processing unit 1212 executes post-processing according to the determined type of document image (S118).

If the degree of similarity equal to or greater than the threshold does not exist (S116/NO), the determination unit 1210 cannot determine the type of the document image to be determined, so the control unit 120 causes the image processing unit 1204 to change the degree of blurring (S120). , the processing is repeated from S104.

As described above, the image analysis apparatus 10 according to the present embodiment first generates a blurred image including a processed object obtained by processing an object included in the document image by blurring the document image. Next, the image analysis device 10 sets a section indicating a rectangular area including one processing object in the generated blurred image. As a result, the image analysis apparatus 10 can set a section for each paragraph only by setting a section for each processing object, and can divide the paragraphs in the document image. Therefore, the image analysis apparatus 10 can easily set paragraphs in the document image.

Furthermore, the image analysis device 10 determines the type of the document image to be determined based on the section set as the blurred image. Specifically, the image analysis apparatus 10 first uses a learned model that has learned the relationship between the layout of sections and the type of document image to determine the layout of sections in the blurred image and the section used for learning by the learned model. Get the similarity with the layout of . When calculating the degree of similarity, the trained model compares only the layouts of simple rectangular partitions, so the degree of layout similarity can be easily calculated without performing complicated comparison processing. Further, the image analysis apparatus 10 can easily obtain the degree of similarity simply by inputting the document image into the trained model.

Next, if there is a degree of similarity equal to or greater than a predetermined threshold among the obtained degrees of similarity, the image analysis apparatus 10 determines the type of the document image corresponding to the maximum similarity degree among the degrees of similarity equal to or greater than the predetermined threshold, Determined as the type of document image to be determined. In this manner, the image analysis apparatus 10 can easily determine the type of the document image by performing the threshold value determination on the easily acquired similarity. Further, the image analysis apparatus 10 can easily determine the layout of the document image from the layout of the section set as the blurred image.

On the other hand, if none of the obtained similarities exceeds the predetermined threshold, the image analysis apparatus 10 reacquires the similarities using a blurred image generated by changing the degree of blurring. Next, the image analysis apparatus 10 determines the type of document image to be determined based on the reacquired similarity. As described above, when it is difficult to make a determination based on the acquired similarity, the image analysis apparatus 10 acquires the similarity again using a blurred image with a different degree of blurring, and repeats the determination process. As a result, the image analysis apparatus 10 can improve the probability of being able to determine the type of the document image, and also improve the accuracy of determination.

In addition, the image analysis apparatus 10 executes processing using a blurred image in which divisions are set or a document image in which division frames are combined in the post-processing. As a result, the image analysis apparatus 10 can reduce the number of search targets by setting the block as a search target when searching for processing targets in post-processing, compared to when each object is set as a search target. can be done. Therefore, the image analysis apparatus 10 can reduce the load in the post-processing by executing the process using the blurred image in which the division is set or the document image in which the division frame is synthesized.

The embodiments of the present invention have been described above. Note that the image analysis apparatus 10 in the above-described embodiment may be realized by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in this recording medium may be read into a computer system and executed. It should be noted that the "computer system" referred to here includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible discs, magneto-optical discs, ROMs and CD-ROMs, and storage devices such as hard discs incorporated in computer systems. Furthermore, "computer-readable recording medium" refers to a program that dynamically retains programs for a short period of time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It may also include something that holds the program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client in that case. Further, the program may be for realizing a part of the functions described above, or may be capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be implemented using a programmable logic device such as an FPGA (Field Programmable Gate Array).

Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configurations are not limited to those described above, and various design changes can be made without departing from the gist of the present invention. It is possible to

1 image analysis system 10 image analysis device 20 similarity calculation server 30 network 110 image acquisition unit 120 control unit 130 communication unit 140 storage unit 150 output unit 1202 image reading unit 1204 image processing unit 1206 section setting unit 1208 section synthesis unit 1210 determination unit 1212 post-processing unit

Claims (7)

an image processing unit that generates a blurred image including a second object obtained by processing a first object included in the document image by blurring the document image;
a partition setting unit that sets a partition indicating a rectangular area including one of the second objects in the blurred image;
a determination unit that determines the type of the document image to be determined based on the section set in the blurred image;
An image analysis device comprising: The determination unit uses a learned model that has learned the relationship between the layout of the section and the type of the document image to determine the layout of the section in the blurred image and the layout of the section used for learning by the learned model. 2. The image analysis apparatus according to claim 1 , wherein a degree of similarity with a layout is obtained, and the type of the document image to be determined is determined based on the degree of similarity. If there is a degree of similarity equal to or greater than a predetermined threshold among the obtained degrees of similarity,
3. The image according to claim 2, wherein said determination unit determines, as the type of said document image to be determined, the type of said document image corresponding to said degree of similarity having a maximum value among said degrees of similarity equal to or greater than a predetermined threshold. analysis equipment. If there is no similarity above a predetermined threshold among the obtained similarities,
The determination unit re-acquires the similarity using the blurred image generated by changing the degree of blurring, and determines the type of the document image to be determined based on the re-acquired similarity. The image analysis apparatus according to claim 2 or 3 . 5. The image analysis device according to claim 4 , wherein the image processing unit makes the degree of blurring after the change stronger than the degree of blurring before the change. generating a blurred image including a second object obtained by processing a first object included in the document image by blurring the document image;
setting a section indicating a rectangular area including one of the second objects in the blurred image;
Determining the type of the document image to be determined based on the section set in the blurred image;
A processor-implemented image analysis method comprising: the computer,
an image processing unit that generates a blurred image including a second object obtained by processing a first object included in the document image by blurring the document image;
a partition setting unit that sets a partition indicating a rectangular area including one of the second objects in the blurred image;
a determination unit that determines the type of the document image to be determined based on the section set in the blurred image;
A program that acts as a

Images