JP7800473B2 - Information processing device - Google Patents

Description

The present invention relates to an information processing device.

Technologies for detecting character strings aligned along the X-axis or Y-axis in a captured image have been known (see, for example, Patent Documents 1 to 5). Also known is a technique for rotating character strings tilted relative to the X-axis or Y-axis so that they are parallel to the X-axis or Y-axis, and then reading the rotated character string. Furthermore, techniques for performing image processing such as edge extraction before character recognition are known.

JP 2012-103755 A Japanese Patent Application Publication No. 7-121658 Japanese Patent Application Publication No. 9-237318 Japanese Patent Application Laid-Open No. 2019-128727 Japanese Patent Application Laid-Open No. 2022-52716

For example, if you write two lines of text on a rectangular piece of paper, wrap that paper around a cylinder, and then photograph an image of the cylinder, the characters will be lower in alignment, the characters closer to the left and right edges will be smaller, and the space between characters will be narrower. In this state, character recognition may not be able to accurately read each character in the text.

Therefore, the present invention aims to provide an information processing device that can improve the accuracy of character recognition for character strings arranged in a curved shape.

The information processing device according to the present invention comprises an input means for inputting an image of an object having a curved surface on which a character string arranged in a curved shape is written; a detection means for detecting from the image a first ellipse that passes through multiple upper endpoints of the character string and a second ellipse that passes through multiple lower endpoints of the character string; and a conversion means for converting the image of the curved surface of the object into an image of a plane based on the central coordinates of the first ellipse and the second ellipse, the radii of the major axes of the first ellipse and the second ellipse, and the radii of the minor axes of the first ellipse and the second ellipse.

In the above configuration, the object is a cylinder, and the conversion means defines the difference between the Y coordinates of the centers of the first ellipse and the second ellipse as the Y-direction length H of the planar image, defines half the perimeter of either the first ellipse or the second ellipse as the X-direction length W of the planar image, defines the lower right end of the planar image as the origin, defines upward and leftward directions as positive, defines the coordinates of an arbitrary point R on the planar image as (x, y), and defines the angle by which the radii of the major axes of the first ellipse and the second ellipse rotate clockwise from the horizontal direction as θ. The conversion means calculates the angle θ from the coordinate x of point R and the length W, and defines the upper and lower endpoints of a vertical line segment passing through point R as points P. , point Q, the coordinates of points P' and Q' on the cylinder corresponding to the coordinates of point P and point Q may be calculated based on the central coordinates, major axis radius, minor axis radius, and angle θ of the first and second ellipses, the coordinates of the points that internally divide the Y coordinates of points P' and Q' in the equation y:(H-y) may be set as the Z coordinate of point R' on the cylinder corresponding to the coordinates of point R, and the X and Y coordinates of either point P' or point Q' may be set as the X and Y coordinates of point R', thereby calculating the X, Y, and Z coordinates of point R', and detecting the color of point R' from surrounding pixels using bilinear interpolation, and setting the color of point R' as the color of point R.

In the above configuration, if the character string is composed of multiple lines of character strings, the first ellipse may be an ellipse that passes through multiple upper endpoints of the topmost character string of the multiple lines of character strings, and the second ellipse may be an ellipse that passes through multiple lower endpoints of the bottommost character string of the multiple lines of character strings.

The information processing device according to the present invention may input a captured image of a curved cylinder on which a character string arranged in a curved shape is written, and obtain the results of character recognition of the character string using a learning model that outputs the results of character recognition of the character string. The learning model is generated using a well-known machine learning algorithm such as a neural network.

This invention can improve the accuracy of character recognition for character strings arranged in a curved pattern.

FIG. 1 is a block diagram showing an information processing apparatus according to the present embodiment. FIG. 2 is a flowchart showing the process executed by the information processing device. 3A and 3B are diagrams illustrating the ellipse fitting process in step S8 of FIG. FIG. 4 is an explanatory diagram of the image conversion process in step S9 of FIG. FIG. 5 is a diagram showing an example of a character string after the image conversion process in step S9 of FIG.

The following describes an embodiment of the present invention with reference to the drawings.

FIG. 1 is a block diagram showing an information processing device according to this embodiment. The information processing device 100 in FIG. 1 includes a CPU (Central Processing Unit) 101 as a processor, RAM (Random Access Memory) 102 and ROM (Read Only Memory) 103 as memories, a HDD (Hard Disk Drive) 104, an interface (IF) 105, and a network interface (IF) 106. The information processing device 100 can be realized, for example, by a computer, a server, or a smartphone. The CPU 101 is connected to the RAM 102, ROM 103, HDD 104, IF 105, and network IF 106 via a bus 107. IF 105 or network IF 106 functions as an input means, and the CPU 101 functions as a detection means and a conversion means.

IF 105 is, for example, a USB (Universal Serial Bus) and is connected to a camera 110 serving as an imaging device. In this embodiment, the camera 110 is connected externally to the information processing device 100, but may also be built into the information processing device 100. Network IF 106 is an interface for connecting the information processing device 100 to an external network. The camera 110 may be connected to the information processing device 100 via network IF 106.

The HDD 104 stores images captured by the camera 110 and programs for executing various image processing and character recognition processes. These programs correspond to the flowcharts described below. The HDD 104 also stores the results of various image processing and character recognition processes. An SSD (Solid State Drive) may be used instead of the HDD 104.

Figure 2 is a flowchart showing the processing executed by the information processing device 100. The processing in Figure 2 is implemented by the CPU 101 based on a program stored in the HDD 104.

First, an image captured by camera 110 is input (step S1). In this embodiment, the object captured by camera 110 is a cylinder with a curved surface on which a string of characters is written in a curved manner. The string of characters includes letters, numbers, codes, and symbols. A curved cylinder is, for example, a component such as a rotating shaft, bolt, or plug. For example, a string of characters written in a curved manner is a string of characters such as a product name or model number written in a straight line along the curved surface of the cylinder. In the captured image of the cylinder, the string of characters is arranged in a curved manner, with the arrangement of the characters descending or ascending as you move toward the left or right end (see, for example, Figure 3 (A)).

Next, CPU 101 detects the object from the captured image (step S2) and extracts the curved surface area where the curved text is written (step S3). Next, CPU 101 performs a process to remove the alpha channel from the image of the area extracted in S3 (step S4). The alpha channel is auxiliary data provided for each pixel in addition to the color representation data, and generally represents the opacity of the pixel.

Next, the CPU 101 performs binarization processing on the image after the alpha channel has been removed (step S5), and performs processing to enlarge or reduce the line width of the character string included in the image after the binarization processing (step S6). If the line width of the character string is larger than the range of a preset threshold, the line width of the character string is reduced to fit within the range of the preset threshold; if the line width of the character string is smaller than the range of the preset threshold, the line width of the character string is enlarged to fit within the preset threshold.

Next, the CPU 101 removes the outer frame corresponding to the outer shape of the object from the image after line width expansion or reduction processing (step S7). The outer frame corresponding to the outer shape of the object is removed because it is information unnecessary for character recognition.

Next, the CPU 101 performs an ellipse fitting process on the image after the outer frame has been removed, i.e., the image of the character string arranged in a curved shape (step S8). Details of the ellipse fitting process will be described later.

Next, based on the information about the multiple ellipses detected in the ellipse fitting process, the CPU 101 performs image conversion processing to convert the image of the curved surface of the object into a flat image so that the curvedly arranged character strings written on the curved surface of the object are arranged in a straight line (step S9). Details of the image conversion processing will be described later.

Next, CPU 101 performs optical character recognition (OCR) on the planar image after image conversion (step S10), outputs the results of the OCR (step S11), and ends the processing in FIG. 2. The results of the OCR may be stored in HDD 104, or may be output to an external device (not shown) via network IF 106 or IF 105.

The processing of steps S2 to S10 in FIG. 2 may be performed by machine learning. In this case, the information processing device 100 acquires the results of character recognition of a string of characters using a learning model that takes as input, for example, a captured image of a cylinder with a curved surface on which a string of characters is written in a curved manner, and outputs the results of character recognition of the string of characters. The learning model is generated using a well-known machine learning algorithm such as a neural network. The training data to be input into the machine learning algorithm can be created, for example, by using the camera 110 to capture images of a cylinder with a curved surface on which a string of characters is written in a curved manner under various conditions, and linking the results of character recognition of the string of characters to the captured images obtained.

Figures 3(A) and (B) are explanatory diagrams of the ellipse fitting process in step S8 of Figure 2. As shown in Figure 3(A), assume that two lines of text, "ABCD 1234-EF," are written on the curved surface of a cylinder.

In the ellipse fitting process, CPU 101 detects each of the blocks "A," "B," "C," "D," "1," "2," "3," "4," "E," and "F," and extracts the top and bottom endpoints of each block. The state after the top and bottom endpoints of each block have been extracted is shown in the right diagram of Figure 3(A).

If two lines of text, "ABCD 1234-EF," are written on the curved surface of a cylinder, four point clouds will be created: the multiple upper endpoints of "ABCD" on the first line, the multiple lower endpoints of "ABCD" on the first line, the multiple upper endpoints of "1234-EF" on the second line, and the multiple lower endpoints of "1234-EF" on the second line. However, "-" is omitted as a connection, resulting in six point clouds, separated by "1234" and "EF." Additionally, flat parts such as "E" and "F" have multiple endpoints displayed instead of just one.

Next, as shown in FIG. 3(B), CPU 101 detects ellipses E1 to E4 from the point cloud of the multiple upper endpoints of "ABCD" on the first line, the point cloud of the multiple lower endpoints of "ABCD" on the first line, the point cloud of the multiple upper endpoints of "1234-EF" on the second line, and the point cloud of the multiple lower endpoints of "1234-EF" on the second line. Ellipse E1 passes through the point cloud of the multiple upper endpoints of "ABCD" on the first line, ellipse E2 passes through the point cloud of the multiple lower endpoints of "ABCD" on the first line, ellipse E3 passes through the point cloud of the multiple upper endpoints of "1234-EF" on the second line, and ellipse E4 passes through the point cloud of the multiple lower endpoints of "1234-EF" on the second line. CPU 101 performs ellipse fitting on each point cloud and finds the optimal value of the ellipse using the steepest descent method.

As shown in Figure 3 (B), the CPU 101 detects four ellipses E1 to E4 of the same size, aligns the center points of the ellipses E1 to E4 vertically, and fixes the top, bottom, left, and right of the ellipses E1 to E4. This ellipse detection makes it possible to determine the tilt of the cylinder.

The ellipse used for ellipse fitting is given by the following equation (1).
g(x,y)= ^ax2 +bxy+ ^cy2 +dx+ey+f (1)
If D=4ac−b ² >0 and g(x,y)=0, then x,y are points on the ellipse.

In ellipse fitting, the equation of the ellipse that passes closest to the point cloud is obtained by finding the coefficients a, b, c, d, and e that minimize Σ[g(x, y)] ² . Furthermore, in ellipse fitting, simultaneous fitting is performed so that the x-coordinate of the center and the major axis of multiple ellipses in the following equations (2) and (3) are the same. Note that if the major and minor axes of each ellipse are aligned with the x-axis and y-axis, respectively, the coefficient b = 0.
Ellipse E1 a1x ² + c1y ² + d1x + e1y + f1 = 0 (2)
Ellipse Em amx ² + cmy ² + dmx + emy + fm = 0 (m = integer greater than or equal to 2) (3)

As an ellipse fitting method, for example, the following document 1 can be referred to.
Reference 1: Andrew Fitzgibbon, Maurizio Pilu, “Direct linear least squares fitting of an ellipse”, November 2000, IEEE Transactions on Pattern Analysis and Machine Intelligence 21(5)

In Figure 3(B), CPU 101 detects ellipses E1 to E4 from each of the four point groups. However, if the character string consists of multiple lines of character strings, CPU 101 may detect ellipse E1 that passes through multiple upper endpoints of the topmost line of the character string, and ellipse E4 that passes through multiple lower endpoints of the bottommost line of the character string. In this case, the number of ellipses detected can be reduced, thereby reducing the amount of calculations performed by CPU 101 and shortening processing time.

Figure 4 is an explanatory diagram of the image conversion process of step S9 in Figure 2. Figure 5 is a diagram showing an example of a character string after the image conversion process of step S9 in Figure 2. First, it is assumed that the CPU 101 has previously detected the coordinates (Cx1, Cy1) of the center point C1 of the ellipse E1, the coordinates (Cx2, Cy2) of the center point C2 of the ellipse E4, the radius A1 of the major axis of the ellipse E1, the radius A2 of the major axis of the ellipse E4, the radius B1 of the minor axis of the ellipse E1, and the radius B2 of the minor axis of the ellipse E4 in the ellipse fitting process before the image conversion process.

In the image conversion process, as shown in Figure 4, the CPU 101 converts the image of the curved surface of the object (cylinder β) into a flat image α based on the coordinates of the center points C1 and C2 of the ellipses E1 and E4 detected in the ellipse fitting process, the radii A1 and A2 of the major axes of the ellipses E1 and E4, and the radii B1 and B2 of the minor axes of the ellipses E1 and E4, so that the curvedly arranged character strings written on the curved surface of the object (cylinder β) are arranged in a straight line.

The CPU 101 calculates the difference (= Cy1 - Cy2) between the Y coordinates of the center points C1 and C2 of the ellipses E1 and E4 as the length H in the Y direction of the planar image α, and calculates half the perimeter of either the ellipse E1 or the ellipse E4 as the length W in the X direction of the planar image α. The lengths H and W are rounded off to integers. For example, half the perimeters of the ellipses E1 and E4 can be calculated using the following formulas (4) and (5), respectively.
Half the perimeter of the ellipse E1 = π [(3(A1 + B1) - √(A1 + 3B1) (3A1 + B1))] / 2 (4)
Half the perimeter of ellipse E4 = π [(3(A2 + B2) - √(A2 + 3B2) (3A2 + B2))] / 2 (5)

Let the bottom right corner of the planar image α be the origin O, the upward and leftward directions be positive, and the coordinates of any point R on the planar image α be (x, y).

Next, the CPU 101 determines the color of the point R(x, y) according to the following procedures (A) to (D).
(A) When the angle at which the radii A1 and A2 of the major axes of the ellipse E1 and the ellipse E4 rotate clockwise from the horizontal direction is θ, the CPU 101 calculates the angle θ from the coordinate x and length W of the point R.
Because the object is a cylinder β, the length W can be expressed as W = πr (r is the radius of the circle), and the coordinate x of point R can be expressed as x = 2πr × θ/2π = rθ. Therefore, the angle θ can be expressed as θ = x/r = x/(W/π) = πx/W. Therefore, the angle θ can be calculated from the coordinate x of point R and the length W.

(B) If the upper and lower endpoints of a vertical line segment passing through point R are points P and Q, the CPU 101 calculates the coordinates of points P' and Q' of ellipses E1 and E4 corresponding to the coordinates of point P and point Q based on the coordinates of center points C1 and C2 of ellipses E1 and E4, radii A1 and A2 of the major axes of ellipses E1 and E4, radii B1 and B2 of the minor axes of ellipses E1 and E4, and angle θ.
The coordinates of point P' are (Cx1+A1cos θ, Cy1-B1sin θ), and the coordinates of point Q' are (Cx2+A2cos θ, Cy2-B2sin θ).

(C) The CPU 101 calculates the X, Y, and Z coordinates of point R' by setting the coordinates of the point that divides the Y coordinates of points P' and Q' internally into y:(H-y) as the Z coordinate of point R' on the cylinder β that corresponds to the coordinates of point R, and setting the X and Y coordinates of either point P' or point Q' as the X and Y coordinates of point R'.
For example, the X, Y and Z coordinates of point R' can be expressed as (Cx2+A2cos θ, Cy2-B2sin θ, y).

(D) The CPU 101 detects the color of point R' using bilinear interpolation from the four surrounding pixels and sets the color of point R' as the color of point R. For example, the four pixels are the upper right, lower right, upper left, and lower left pixels adjacent to point R'. Bilinear interpolation is a well-known interpolation method that uses the four surrounding pixels, so a detailed explanation of it will be omitted.

By repeating steps (A) through (D) above for all points on the planar image α, the image of the curved surface of the object (cylinder β) can be converted into the planar image α. As a result, the curved character strings written on the curved surface in Figure 3(A) are arranged in a straight line on the planar image α, as shown in Figure 5.

As described above, according to this embodiment, the information processing device 100 includes an IF 105 or network IF 106 that inputs an image of an object having a curved surface on which a character string arranged in a curved manner is written, and a CPU 101 that detects from the image an ellipse E1 (first ellipse) that passes through multiple upper endpoints of the character string and an ellipse E4 (second ellipse) that passes through multiple lower endpoints of the character string, and converts the image of the curved surface of the object into a planar image α based on the center coordinates of the ellipse E1 and the ellipse E4, the radii A1 and A2 of the major axes of the ellipses E1 and E4, and the radii B1 and B2 of the minor axes of the ellipses E1 and E4. This allows the image of the curved surface of the object to be converted into a planar image α. Furthermore, since the character strings arranged in a curved manner on the curved surface of the object in FIG. 3(A) are arranged in a linear manner on the planar image α as shown in FIG. 5, the accuracy of character recognition of character strings arranged in a curved manner can be improved.

The object is a cylinder, and the CPU 101 defines the difference between the Y coordinates of the centers of ellipses E1 and E4 as the Y-direction length H of the planar image α, defines half the perimeter of either ellipse E1 or E4 as the X-direction length W of the planar image α, defines the lower right end of the planar image α as the origin, defines upward and leftward directions as positive, and defines the coordinates of any point R on the planar image α as (x, y). Furthermore, when the angle at which the radius of the major axis of the planar image α rotates clockwise from the horizontal is defined as θ, the CPU 101 calculates the angle θ from the coordinate x of point R and the length W. If the upper and lower endpoints of a vertical line segment passing through point R are defined as points P and B2, the CPU 101 calculates the coordinates of points P' and Q' on the cylinder corresponding to the coordinates of points P and Q based on the center points C1 and C2 of the ellipses E1 and E4, the radii A1 and A2 of the major axes, the radii B1 and B2 of the minor axes, and the angle θ. Furthermore, the CPU 101 sets the coordinates of the point that divides the Y coordinates of points P' and Q' internally in the ratio y:(H-y) as the Z coordinate of point R' on the cylinder corresponding to the coordinates of point R, and sets the X and Y coordinates of either point P' or point Q' as the X and Y coordinates of point R' to calculate the X, Y, and Z coordinates of point R', detects the color of point R' from surrounding pixels using bilinear interpolation, and sets the color of point R' as the color of point R. In this way, the image of the curved surface of the object can be converted into a planar image α.

Although the preferred embodiment of the present invention has been described in detail above, the present invention is not limited to such specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention as set forth in the claims.

100 Information processing device 101 CPU
102 RAM
103 ROM
104 HDD
105 Interface 106 Network interface 107 Bus 110 Camera

Claims (3)

an input means for inputting an image of an object having a curved surface on which character strings arranged in a curved shape are written;
a detection means for detecting a first ellipse passing through a plurality of upper endpoints of the character string from the image and a second ellipse passing through a plurality of lower endpoints of the character string from the image;
and a conversion means for converting an image of the curved surface of the object into an image of a plane based on the central coordinates of the first ellipse and the second ellipse, the radii of the major axes of the first ellipse and the second ellipse, and the radii of the minor axes of the first ellipse and the second ellipse. the object is a cylinder,
The conversion means
a difference between the Y coordinates of the centers of the first ellipse and the second ellipse is defined as a length H in the Y direction of the image of the plane;
a length W in the X direction of the image of the plane is set to half the perimeter of either the first ellipse or the second ellipse;
The lower right corner of the image of the plane is set as the origin, and the upward and leftward directions are set as positive directions. The coordinates of an arbitrary point R in the image of the plane are set as (x, y),
When the angle at which the radii of the major axes of the first ellipse and the second ellipse rotate clockwise from the horizontal direction is defined as θ, the angle θ is calculated from the coordinate x of the point R and the length W;
When the upper and lower end points of a vertical line segment passing through the point R are defined as points P and Q, the coordinates of points P' and Q' on the cylinder corresponding to the coordinates of the point P and the point Q are calculated based on the central coordinates of the first ellipse and the second ellipse, the radius of the major axis, the radius of the minor axis, and the angle θ;
The coordinates of the point that divides the Y coordinates of the points P' and Q' internally into y:(H-y) are set as the Z coordinate of the point R' on the cylinder that corresponds to the coordinates of the point R, and the X coordinate and Y coordinate of either the point P' or the point Q' are set as the X coordinate and Y coordinate of the point R', thereby calculating the X coordinate, Y coordinate and Z coordinate of the point R';
2. The information processing apparatus according to claim 1, wherein the color of said point R' is detected from surrounding pixels by bilinear interpolation, and the color of said point R' is set as the color of said point R. 3. The information processing device according to claim 1, wherein, when the character string is composed of multiple lines of character strings, the first ellipse is an ellipse that passes through multiple upper endpoints of the topmost character string of the multiple lines of character strings, and the second ellipse is an ellipse that passes through multiple lower endpoints of the bottommost character string of the multiple lines of character strings.