JP6760589B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing apparatus and an image processing method.

In recent years, a technique for recognizing characters contained in an image has been utilized. For example, a technique of recognizing characters using an OCR library and a technique of recognizing characters using an image pattern matching method are utilized.

Japanese Unexamined Patent Publication No. 8-44819

In the above character recognition technology, there is a problem that characters that face each other (upright) in an image can be recognized, but characters that do not face each other are difficult to recognize. For example, it is difficult to recognize characters included in an image of an object on which characters are printed, taken from a position other than the front, or characters arranged diagonally in the image.

In consideration of the above circumstances, an object of the present invention is to recognize characters in various orientations contained in an image.

The image processing apparatus of the present invention includes a template image input unit for inputting a template image, a character area specifying unit for specifying a character area indicating a character recognition target portion on the input template image, and the template image and the character. A template area storage unit that stores an area in association with each other, a character learning data generation unit that generates character learning data related to a character to be recognized, a machine learning unit that performs machine learning using the generated character learning data, and a machine learning unit. Whether the learning result storage unit that stores the machine learning result, the sample image input unit that inputs the sample image, the input sample image, and the template image stored in the template area storage unit match. An image that creates a face-to-face image of a portion corresponding to the character area on the sample image when the judgment flow generation unit that generates the judgment flow for determining whether or not the sample image matches the template image. An image creation flow generation unit that generates a creation flow, a character recognition flow generation unit that generates a character recognition flow that recognizes characters included in the facing image based on the result of the machine learning, and the determination flow. A flow storage unit that stores the image creation flow and the character recognition flow, an inspection image input unit that inputs an inspection image, and the determination flow, the image creation flow, and the character recognition flow for the inspection image. Is provided with a character recognition unit that recognizes characters by applying.

In a preferred embodiment of the present invention, the character learning data generation unit generates, as the character learning data, a plurality of character image data having different inclinations within a predetermined angle range for each of the characters to be recognized. ..

In a preferred embodiment of the present invention, the character recognition unit identifies a plurality of character areas including characters to be recognized, sets a connection between the character areas, and splines over the plurality of the character areas based on the connection. The curve is acquired, and the slope of each character region is estimated based on the spline curve.

In the image processing method of the present invention, a template image is input, a character area indicating a character recognition target portion on the input template image is specified, and the template image and the character area are stored in association with each other. To do, to generate character learning data for the character to be recognized, to perform machine learning using the generated character learning data, to store the result of the machine learning, and to input a sample image. When the input sample image and the stored template image are matched or not, a determination flow for determining whether or not the input sample image is matched is generated, and when the sample image and the template image are matched. A character that generates an image creation flow that creates a facing image of a portion corresponding to the character area on the sample image and recognizes a character included in the facing image based on the result of the machine learning. Generating a recognition flow, storing the determination flow, the image creation flow, and the character recognition flow, inputting an inspection image, and for the inspection image, the determination flow and the image creation. By applying the flow and the character recognition flow, the character is recognized.

According to the present invention, characters in various orientations included in the inspection image are recognized.

It is a block diagram which shows the physical structure of the image processing apparatus of 1st Embodiment. It is a block diagram which shows the logical structure of the image processing apparatus of 1st Embodiment. It is a flowchart which shows outline the image processing operation of 1st Embodiment. It is a flowchart which shows the template / character area registration operation. It is a flowchart which shows the character learning operation. It is a flowchart which shows the flow generation operation. It is a flowchart which shows the character recognition operation. It is a figure which illustrates the template image. It is a figure which illustrates the character area. It is a figure which illustrates the image shown in the character learning data. It is explanatory drawing which shows the image created by the image creation flow. It is explanatory drawing which shows the process executed in the character recognition flow. It is a flowchart which shows the inclination estimation operation of a character area. It is explanatory drawing which shows the specific state of the adjacent character area.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

1. 1. First Embodiment 1 (1). Physical Configuration FIG. 1 is a block diagram showing a physical configuration of the image processing apparatus IA of the present embodiment. The image processing device IA includes a network interface 10, an external storage medium input / output unit 12, an operation input unit 14, a display output unit 16, a CPU (Central Processing Unit) 18, a RAM (Random Access Memory) 20, and a ROM (Read Only Memory). 22 and HDD (Hard Disk Drive) 24 are provided. The image processing device IA may be, for example, a personal computer or a hardware device such as an image sensor.

The network interface 10 communicates with other devices via a wired or wireless network. The external storage medium input / output unit 12 executes input (read) from an external storage medium such as an SD memory card or a USB flash memory and output (write) to the external storage medium. The operation input unit 14 receives an operation signal from an input device such as a keyboard. The display output unit 16 transmits a display signal to an output device such as a display. The CPU 18 performs various controls and calculations by executing programs stored in the RAM 20 and the ROM 22 which are the main storage devices. The HDD 24 is an auxiliary storage device that stores a program expandable on the RAM 20 and various data. In addition, instead of or in addition to the HDD 24, a storage medium such as an SSD (Solid State Drive) may be adopted.

The image processing device IA of the present embodiment is roughly an device that processes an image captured by the photographing device SA and executes character recognition. The image captured by the photographing device SA is input to the image processing device IA via the network interface 10 or the external storage medium input / output unit 12.

1 (2). Logical Configuration FIG. 2 is a block diagram showing a logical configuration of the image processing apparatus IA of the present embodiment. The image processing device IA includes a template image input unit 100, a character area specifying unit 102, a character learning data generation unit 104, a machine learning unit 106, a sample image input unit 108, a determination flow generation unit 110, an image creation flow generation unit 112, and characters. It includes a recognition flow generation unit 114, an inspection image input unit 116, and a character recognition unit 118. The above functional block is realized by the CPU 18 executing a computer program stored in the main storage device of the image processing device IA.

Further, the image processing device IA includes a template area storage unit 130, a learning result storage unit 132, and a flow storage unit 134. Each of the above storage units is composed of a RAM 20 and an HDD 24.

The template image input unit 100 inputs a template image from the photographing device SA. The character area specifying unit 102 specifies a character area indicating a character recognition target portion on the input template image. The template area storage unit 130 stores the template image and the character area in association with each other.

The character learning data generation unit 104 generates character learning data related to the character to be recognized. The machine learning unit 106 performs machine learning using the generated character learning data. The learning result storage unit 132 stores the result of machine learning.

The sample image input unit 108 inputs a sample image stored in the HDD 24. The determination flow generation unit 110 generates a determination flow for determining whether or not the input sample image matches the template image stored in the template area storage unit 130. The image creation flow generation unit 112 generates an image creation flow that creates a facing image of a portion corresponding to a character area on the sample image when the sample image and the template image are matched. The character recognition flow generation unit 114 generates a character recognition flow that recognizes characters included in the facing image based on the result of machine learning. The flow storage unit 134 stores the above determination flow, image creation flow, and character recognition flow.

The inspection image input unit 116 inputs an inspection image from the photographing apparatus SA. The character recognition unit 118 recognizes characters by applying the determination flow, the image creation flow, and the character recognition flow to the input inspection image.

1 (3). Image processing operation An example of the image processing operation of the present embodiment will be described below. 3 to 7 are flowcharts showing the image processing operation of this example. A schematic operation is shown in FIG. 3, and details of each operation included in FIG. 3 are shown in FIGS. 4 to 7, respectively. As shown in FIG. 3, the image processing operation of the present embodiment includes a template / character area registration operation (S1000), a character learning operation (S2000), a flow creation operation (S3000), and a character recognition operation (S4000). Is done.

The details of the template / character area registration operation (S1000) will be described with reference to FIG. The template image input unit 100 inputs the template image TI input from the photographing device SA to the character area specifying unit 102 (S1010). The template image TI is an image used for detecting a characteristic pattern included in the template image TI in another image (such as an inspection image II described later). The template image TI may be stored in the HDD 24 in advance. An example of the template image TI is shown in FIG.

The character area specifying unit 102 specifies a character recognition target portion (that is, a character area TA) on the template image TI supplied from the template image input unit 100 (S1020). More specifically, the character area specifying unit 102 describes an area on the template image TI specified by the user by operating the operation input unit 14, or a character on the template image TI detected by automatic recognition. The area is specified as the character area TA. An example of the character area TA specified on the template image TI is shown in FIG.

The character area specifying unit 102 associates the template image TI input from the template image input unit 100 with the character area on the template image TI, and stores the template image TI in the template area storage unit 130.

The details of the character learning operation (S2000) will be described with reference to FIG. The character learning data generation unit 104 generates character learning data used in machine learning described later (S2010). The character learning data is image data showing a plurality of variations of recognition target characters (for example, numbers from "0" to "9"). The character learning data generation unit 104 automatically generates character learning data corresponding to the font and angle range specified by the user by operating the operation input unit 14. For example, if the user specifies "number" as the character to be recognized, "Gothic" and "Mincho" as the font, and "-15 ° to + 15 °" as the angle range, the character. The learning data generation unit 104 has a plurality of images in the Gothic font in the angle range "-15 ° to + 15 °" and the Mincho font in the angle range "-15 ° to" for each number from "0" to "9". Create multiple images at "+ 15 °". As a result, for each of the recognition target characters, a plurality of character image data having different inclinations within a predetermined angle range are created for each designated font. FIG. 10 shows an example of an image shown by the character learning data.

The machine learning unit 106 performs machine learning using the character learning data generated by the character learning data generation unit 104 (S2020). The result of machine learning is used to recognize the characters to be recognized contained in a given image. The learning result storage unit 132 stores the result of machine learning in step S2020. With respect to the above machine learning, a suitable algorithm may be arbitrarily selected. Examples of algorithms selected include backpropagation, nearest neighbor classification, perceptrons, and support vector machines.

The details of the flow creation operation (S3000) will be described with reference to FIG. The sample image input unit 108 inputs the sample image SI stored in the HDD 24 (S3010). The sample image SI is an image of the same type as the template image TI.

The determination flow generation unit 110 generates a determination flow for determining whether or not the sample image SI input by the sample image input unit 108 and the template image TI stored in the template area storage unit 130 match (S3020). ). In the determination flow, matching can be determined by various determination methods. For example, when the ratio at which the plurality of feature points extracted from the sample image SI and the plurality of feature points extracted from the template image TI match exceeds a predetermined threshold value, it may be determined that both images match. .. The feature points are, for example, points having a higher brightness than the surroundings and points having a lower brightness than the surroundings. Further, the edges may be extracted instead of the feature points, and the matching may be determined according to the ratio of the edges matching.

The image creation flow generation unit 112 creates an image creation flow that creates a facing image of a portion corresponding to the character area TA in the sample image SI when it is determined by the determination flow that the sample image SI and the template image TI match. Is generated (S3030). In the image creation flow, as shown in FIG. 11, the corresponding part CA on the sample image SI corresponding to the character area TA on the template image TI specified in step S1020 is extracted, and the corresponding part CA is directly opposed to the corresponding part CA. The image is created.

The character recognition flow generation unit 114 generates a character recognition flow that recognizes characters included in the facing image based on the machine learning result stored in the learning result storage unit 132 (S3040). In the character recognition flow, as shown in FIG. 12, after a plurality of character area TRs included in the facing image are specified, character recognition processing using the result of machine learning is executed for each character area TR. The recognition result is output.

The details of the character recognition operation (S4000) will be described with reference to FIG. 7. The inspection image input unit 116 inputs the inspection image II input from the photographing device SA into the character recognition unit 118 (S4010). The character recognition unit 118 is included in the inspection image II by applying the determination flow, the image creation flow, and the character recognition flow created in steps S3020, S3030, and S3040 described above to the inspection image II. Recognize characters (S4020). That is, the determination flow determines whether or not the inspection image II and the template image TI match, the image creation flow creates a facing image of the portion corresponding to the character area TA in the inspection image II, and the character recognition flow determines. The characters included in the above facing image are recognized.

1 (4). Effect of the present embodiment According to the above configuration of the present embodiment, the recognition of the characters included in the inspection image II is executed based on the flow created by using the sample image SI. The above character recognition is performed based on the result of machine learning in step S2000. In the machine learning of step S2000, a plurality of character image data having different inclinations within a predetermined angle range are used for learning. Therefore, characters in various orientations included in the inspection image are recognized.

2. 2. Second Embodiment The second embodiment of the present invention will be described below. In each of the embodiments illustrated below, for elements having the same action and function as those in the first embodiment, the reference numerals referred to in the above description will be used and the respective description will be omitted as appropriate.

In specifying the character area TR by the character recognition flow, it is preferable that the inclination of each character area TR is obtained. This is because it is possible to perform character recognition according to the inclination of the character area TR. In the second embodiment, the inclination of each character area TR is estimated in specifying the character area TR.

FIG. 13 is a flowchart showing the inclination estimation operation of the character area TR. The character recognition unit 118 identifies a plurality of character areas TR included in the inspection image II (face-to-face image) (S5010). When specifying the character area TR, it is preferable to extract an area having a size assumed to be a character and specify it as the character area TR.

The character recognition unit 118 identifies other character areas TR that may be adjacent to each character area TR (S5020). More specifically, as shown in FIG. 14, the character region TR existing within a predetermined angle range (−θ to + θ) and within a predetermined distance (α × r to β × r) when viewed from the character region TR of interest. Is specified as an adjacent character area NTR that may be adjacent. A connection (link) is set between a certain character area TR and an adjacent character area NTR. At this stage, a plurality of concatenations from one character area TR may be set for a plurality of adjacent character area NTRs.

The character recognition unit 118 calculates the existence probability of the adjacent character area NTR for each of the character area TRs (target character area OTR) in which the adjacent character area NTRs exist on both sides (S5030). Here, the target character area OTR is called the "kth", the adjacent character area NTR before the target character area OTR is called the "(k-1) th", and the adjacent character area NTR next to the target character area OTR is called. It is called "(k + 1) th". In the above step S5030, the existence probability P _{k + 1} of the "(k + 1) th" adjacent character area NTR is calculated for each target character area OTR (the "kth" character area). k is any integer greater than or equal to 2.

ここで、ベクトル

は（ｋ−１）番目の隣接文字領域ＮＴＲの座標からｋ番目の対象文字領域ＯＴＲの座標までのベクトルを示し、ベクトル

はｋ番目の対象文字領域ＯＴＲの座標から（ｋ＋１）番目の隣接文字領域ＮＴＲの座標までのベクトルを示す。また、ｄ１は（ｋ−１）番目の隣接文字領域ＮＴＲの座標とｋ番目の対象文字領域ＯＴＲの座標との距離を示し、ｄ２はｋ番目の対象文字領域ＯＴＲの座標と（ｋ＋１）番目の隣接文字領域ＮＴＲの座標との距離を示す。 The existence probability P _{k + 1} of the (k + 1) th adjacent character region NTR is calculated by the following equation (1).

Here vector

Indicates a vector from the coordinates of the (k-1) th adjacent character region NTR to the coordinates of the kth target character region OTR.

Indicates a vector from the coordinates of the k-th target character area OTR to the coordinates of the (k + 1) th adjacent character area NTR. Further, d1 indicates the distance between the coordinates of the (k-1) th adjacent character region NTR and the coordinates of the kth target character region OTR, and d2 is the coordinates of the kth target character region OTR and the (k + 1) th target character region OTR. Indicates the distance from the coordinates of the adjacent character area NTR.

In step S5030, it is preferable that only the character area TR whose distance between the character areas TR is 1 / n to n times the character width is the target of calculation.

The character recognition unit 118 executes the first connection deletion operation for deleting unnecessary connections (S5040). More specifically, the character recognition unit 118 determines whether or not the existence probability P _{k + 1} of the (k + 1) th adjacent character area NTR is equal to or greater than a predetermined threshold value for the kth target character area OTR, and exists. If the probability P _{k + 1} is less than the threshold value, the connection between the k-th target character area OTR and the (k + 1) -th adjacent character area NTR is deleted.

The character recognition unit 118 executes a second connection deletion operation for deleting unnecessary connections (S5050). More specifically, the character recognition unit 118 has the minimum value of the existence probability P _{k + 1} in a certain adjacent character area NTR when a plurality of (k + 1) th adjacent character area NTRs exist for the kth target character area OTR. If exceeds the maximum value of the existence probability P _{k + 1 in} the other adjacent character area NTR, the connection with the other adjacent character area NTR is deleted.

It is preferable that the first connection deletion operation (S5040) and the second connection deletion operation (S5050) are repeatedly executed until there are no more connections to be deleted.

The character recognition unit 118 executes a third connection deletion operation for deleting unnecessary connections (S5060). More specifically, the character recognition unit 118 has (k + 1) th existence probabilities P _{k + 1} and (k + 2) when a plurality of (k + 1) th adjacent character region NTRs exist for the kth target character region OTR. The connection with the adjacent character region NTR having the smallest product (composite existence probability) with the third existence probability P _{k + 2} is deleted. When a plurality of (k + 2) th adjacent character region NTRs exist in the (k + 1) th adjacent character region NTR, the above composite existence probability is calculated based on the largest existence probability P _{k + 2} .

The character recognition unit 118 acquires a spline curve over a plurality of character regions TR based on the remaining concatenation, and estimates the slope of each character region TR (S5070). The slope of each character region TR is acquired as the direction of the perpendicular of the tangent line at the coordinates determined based on the derivative obtained by differentiating the acquired spline curve at the coordinates of each character region TR.

According to the above configuration of the present embodiment, the inclination is estimated for each of the specified character areas TR. The accuracy of character recognition is improved by executing the subsequent character recognition processing based on the estimated inclination.

3. 3. Modifications The above embodiments are variously modified. A specific mode of modification is illustrated below. Two or more embodiments arbitrarily selected from the above embodiments and the following examples can be appropriately merged as long as they do not conflict with each other.

3 (1). Modification 1
In the above embodiment, the image processing device IA, which is one device, executes the image processing operation. However, an image processing system composed of a plurality of devices may perform an image processing operation. For example, template image input unit 100, character area identification unit 102, character learning data generation unit 104, machine learning unit 106, sample image input unit 108, judgment flow generation unit 110, image creation flow generation unit 112, character recognition flow generation unit The 114, the inspection image input unit 116, and the character recognition unit 118 may be implemented by a system including two or more server devices.

3 (2). Modification 2
Even if each function executed by the CPU in the image processing device IA is executed by a programmable logic device such as FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), or GPGPU (General-Purpose computing on Graphics Processing Units). Good.

10 ... Network interface, 12 ... External storage medium input / output unit, 14 ... Operation input unit, 16 ... Display output unit, 18 ... CPU, 20 ... RAM, 22 ... ROM, 24 ... HDD, 100 ... template image input unit, 102 ... character area identification unit, 104 ... character learning data generation unit, 106 ... machine learning unit, 108 ... sample image input unit, 110 ... judgment flow generation unit, 112 ... ... image creation flow generation unit, 114 ... character recognition flow generation unit, 116 ... inspection image input unit, 118 ... character recognition unit, 130 ... template area storage unit, 132 ... learning result storage unit, 134 ... Flow storage unit, CA ... equivalent location, IA ... image processing device, II ... inspection image, NTR ... adjacent character area, OTR ... target character area, SA ... photographing device, SI ... sample image, TA …… Character area, TI …… Template image, TR …… Character area.

Claims (3)

Template image input section for inputting template images and
A character area specifying part that specifies a character area indicating a character recognition target location on the input template image,
A template area storage unit that stores the template image and the character area in association with each other,
A character learning data generator that generates character learning data related to the character to be recognized,
A machine learning unit that performs machine learning using the generated character learning data,
A learning result storage unit that stores the machine learning results,
Sample image input section for inputting sample images and
A determination flow for determining whether or not the input sample image and the template image stored in the template area storage unit match.
An image creation flow for creating a facing image of a portion corresponding to the character area on the sample image when the sample image and the template image are matched.
A flow storage unit that stores a character recognition flow that recognizes characters included in the facing image based on the result of the machine learning, and a flow storage unit.
Inspection image input section for inputting inspection images and
An image processing device including a character recognition unit that recognizes characters by applying the determination flow, the image creation flow, and the character recognition flow stored in the flow storage unit to the inspection image. And
The character recognition unit
Specify a plurality of character areas including the character to be recognized, set the connection between the character areas, and set the connection.
A part of the set connection is deleted based on the predetermined conditions.
Obtain a spline curve over a plurality of the character regions based on the remaining concatenation, excluding the deleted concatenation from the configured concatenation.
An image processing device that estimates the inclination of each character region based on the spline curve .
The character learning data generation unit
The image processing apparatus according to claim 1, which generates a plurality of character image data having different inclinations within a predetermined angle range for each of the characters to be recognized as the character learning data.
Entering a template image and
Identifying the character area indicating the character recognition target location on the input template image, and
To store the template image in association with the character area,
Generating character learning data for characters to be recognized and
Performing machine learning using the generated character learning data,
To memorize the result of the machine learning
Entering a sample image and
The determination flow for determining whether or not the input sample image and the stored template image match, and the character area on the sample image when the sample image and the template image match. To memorize an image creation flow for creating a facing image of a portion corresponding to the above, and a character recognition flow for recognizing a character included in the facing image based on the result of the machine learning.
Entering the inspection image and
An image processing method comprising recognizing characters by applying the stored determination flow, image creation flow, and character recognition flow to the inspection image .
The recognition of the characters
Specify a plurality of character areas including the character to be recognized, set the connection between the character areas, and set the connection.
A part of the set connection is deleted based on the predetermined conditions.
Obtain a spline curve over a plurality of the character regions based on the remaining concatenation, excluding the deleted concatenation from the set concatenation.
An image processing method for estimating the inclination of each character region based on the spline curve .

Images