JP7579095B2 - Image Processing Device - Google Patents

Description

This disclosure relates to an image processing device.

Image processing devices that are used in digital still cameras and the like and that correct distortion caused by oblique shooting of an image of a subject plane are known (see Patent Document 1 below). This conventional image processing device includes an imaging means for imaging the subject plane, a plane measurement means for measuring the orientation of the subject plane relative to the imaging means, and an image distortion correction means for correcting distortion caused by oblique shooting of the image captured by the imaging means based on the measured orientation of the subject plane (Patent Document, Claim 1). With this conventional image processing device, even if an arbitrary subject plane is captured from an arbitrary viewpoint and posture, image distortion can be easily corrected by measuring the plane (Patent Document, Paragraphs 0016 and 0077).

JP 2000-307947 A

For example, to evaluate the wear condition of a mechanical part based on an image, a front-on image is required in which the front of the part is directly facing the imaging surface. However, the front of a part is not necessarily flat, and may have unevenness due to the adhesion of foreign matter. When such a part is used as a subject, the conventional image processing device described above has difficulty measuring the orientation of the mechanical part based on the imaging means, and there is a risk that it will not be possible to correct distortion caused by oblique shooting of the image captured by the imaging means.

This disclosure provides an image processing device that can convert an image of a subject's uneven surface captured from an oblique angle into a front-on image in which the subject surface is directly facing the imaging surface.

One aspect of the present disclosure is an image processing device that acquires an image of a target surface of an object from an imaging device and acquires the distance between the imaging device and the target surface from a distance sensor to convert the image of the target surface into a front-facing image, and that sets a first point group, a second point group, and a third point group spaced apart from each other on the target surface, acquires the distance from the imaging device to each point of the first point group, the distance to each point of the second point group, and the distance to each point of the third point group from the distance sensor, calculates a first plane group by performing a plane calculation process for all combinations of points to calculate a plane based on one point selected from the first point group, one point selected from the second point group, and one point selected from the third point group, calculates a first average plane that has a normal vector that is an average vector of unit normal vectors of all planes in the first plane group and has a minimum sum of distances between the first point group, the second point group, and all points included in the third point group, and converts the image of the target surface into the front-facing image based on the first average plane.

According to the above aspect of the present disclosure, it is possible to provide an image processing device capable of converting an image of a subject's uneven surface captured obliquely into a front-on image in which the surface is directly facing the imaging surface.

FIG. 1 is a block diagram illustrating an embodiment of an image processing device according to the present disclosure. 2 is a side view of a work machine showing an example of a processing target of the image processing device of FIG. 1 . FIG. 2 is a flow chart showing an example of a processing flow by the image processing apparatus of FIG. 1 . FIG. 2 is a flow chart showing an example of a processing flow by the image processing apparatus of FIG. 1 . 2 is an image diagram of a target surface and a plurality of point clouds displayed on a display device of the condition evaluation device of FIG. 1 . FIG. 2 is an image diagram showing an example of a mechanical part and a guide line displayed on the display device of FIG. 1 . 2 is a conceptual diagram of a first group of planes and unit normal vectors calculated by the image processing device in FIG. 1 . 4 is an explanatory diagram of a plane calculation process executed by the image processing device in FIG. 1 . 3 is a conceptual diagram of a first mean plane and a mean vector calculated by the image processing device in FIG. 1 . 4A and 4B are conceptual diagrams of a second group of planes and average vectors calculated by the image processing device in FIG. 1 . 2 is a conceptual diagram of a rotation process of the first and second mean planes by the image processing device of FIG. 1 . 2 is a conceptual diagram of a rotation process of the first and second mean planes by the image processing device of FIG. 1 . 2 is a conceptual diagram of a rotation process of the first and second mean planes by the image processing device of FIG. 1 . 2 is a conceptual diagram of a rotation process of the first and second mean planes by the image processing device of FIG. 1 . 2 is a conceptual diagram of a calculation process of a frontal image by the image processing device of FIG. 1 . 2 is a conceptual diagram of a calculation process of a frontal image by the image processing device of FIG. 1 . A plan view of a portion of a sprocket and a side view of two types of sprockets. 4 is an explanatory diagram of a secondary rotation of a frontal image by the image processing device of FIG. 1 .

Below, an embodiment of an image processing device according to the present disclosure will be described with reference to the drawings.

FIG. 1 is a block diagram showing one embodiment of an image processing device 150 according to the present disclosure. The image processing device 150 of this embodiment constitutes, for example, a part of a condition evaluation device 100 for a machine component. The condition evaluation device 100 evaluates the condition, such as wear, of a component constituting a machine, for example, based on an image of the component. The condition evaluation device 100 includes, for example, an imaging device 110, a distance sensor 120, an input device 130, a display device 140, and the image processing device 150 of this embodiment.

The imaging device 110 is, for example, a digital camera, and is equipped with an optical system including a lens, an electronic system including a light receiving element, and a housing that holds these. The imaging device 110 may also be, for example, a mobile information terminal equipped with an imaging function, such as a smartphone. The distance sensor 120 is, for example, an image sensor that measures the distance to an object using a time-of-flight (TOF) method. The distance sensor 120 is, for example, equipped with a pulse-modulated light source, and measures the distances to multiple points on the object's surface based on phase difference information between the timing of light emission from the light source and the timing of light reflected from the object being received by the image sensor.

The imaging device 110 or the distance sensor 120 also visualizes multiple points on the target surface to which the distance is to be measured. Specifically, the imaging device 110 or the distance sensor 120 includes a visualization light source that irradiates multiple points on the target surface to which the distance is to be measured with a visible light laser, for example. This visualization light source may also serve as a distance measurement light source, for example. The imaging device 110 or the distance sensor 120 may also visualize multiple points on the target surface to which the distance is to be measured by other methods. The imaging device 110 may also include the distance sensor 120.

The input device 130 includes, for example, an operation button, an operation dial, or a touch sensor provided on the imaging device 110, a touch panel provided on the display device 140, or a keyboard or microphone connected to the image processing device 150 separately from the imaging device 110 and the display device 140. The operator of the condition evaluation device 100 and the image processing device 150 inputs necessary information to the image processing device 150 via the input device 130, for example.

The display device 140 may include, for example, a viewfinder, a liquid crystal display device, or an organic EL display device provided in the imaging device 110, or a liquid crystal display device, an organic EL display device, or a mobile information terminal such as a smartphone that is connected to the image processing device 150 separately from the imaging device 110.

The image processing device 150 is, for example, a microcontroller including an input/output unit 151, a processing device 152 such as a central processing unit (CPU), a storage device 153 such as a RAM or ROM, various control programs stored in the storage device 153, and a timer (not shown). The image processing device 150 realizes various processes and functions (described below) by executing the various control programs stored in the storage device 153 using the processing device 152.

The input/output unit 151 is connected to the imaging device 110, the distance sensor 120, the input device 130, and the display device 140 so as to be able to communicate with each other. The processing device 152 is connected to the input/output unit 151 so as to be able to communicate with each other, and transmits and receives control signals and information between the imaging device 110, the distance sensor 120, the input device 130, the display device 140, and the storage device 153 via the input/output unit 151. The storage device 153 stores, for example, various control programs and data, and stores the processing results of the processing device 152 and information input from the imaging device 110, the distance sensor 120, and the input device 130.

Figure 2 is a side view showing an example of a work machine 200. The evaluation target of the condition evaluation device 100 of this embodiment shown in Figure 1 is, for example, a part of the work machine 200. Also, the image processing target of the image processing device 150 of this embodiment shown in Figure 1 is, for example, a part of the work machine 200. The work machine 200 is, for example, a hydraulic excavator. Note that the work machine 200 is not limited to a hydraulic excavator and may be another work machine.

The work machine 200 includes, for example, a traveling device 210, a rotating body 220, and a front work machine 230. The rotating body 220 is provided on the traveling device 210 so as to be rotatable via a rotating mechanism equipped with a hydraulic motor or an electric motor. The front work machine 230 includes, for example, a link mechanism in which a boom 231, an arm 232, and a bucket 233 are connected via joints. The front work machine 230 can perform desired work such as excavation work or leveling work by extending and retracting hydraulic cylinders 231S, 232S, and 233S using a hydraulic device.

The traveling device 210 is, for example, a crawler-type traveling device including a sprocket 211, an idler 212, a link assembly 213, an upper roller 214, a lower roller 215, and a shoe plate 216. The sprocket 211 is connected, for example, to a hydraulic motor driven by a hydraulic device via a power transmission mechanism. The link assembly 213 is stretched across the sprocket 211, the idler 212, the upper roller 214, and the lower roller 215.

The shoe plates 216 are attached to the outside of the annular link assembly 213, and the shoe plates 216 located on the lower side come into contact with the ground. With this configuration, when the hydraulic motor rotates, the sprocket 211 rotates, and the annular link assembly 213 with the shoe plates 216 attached to the outside rotates around the idler 212, upper roller 214, and lower roller 215, causing the work machine 200 to travel.

For example, mechanical parts such as the sprocket 211, idler 212, link assembly 213, upper roller 214, and lower roller 215 of the traveling device 210 are subject to wear due to friction between the parts and soil that has gotten between the parts. The image processing device 150 of this embodiment treats such mechanical parts as objects, captures the object surface of the object using the imaging device 110, and processes the image of the object surface to convert it into a front-on image.

Next, the operation of the image processing device 150 of this embodiment will be described with reference to Figures 3A and 3B. Figures 3A and 3B are flow diagrams showing an example of the processing flow by the image processing device 150 of Figure 1.

For example, an operator of the condition evaluation device 100 photographs the target surface of the sprocket 211 of the work machine 200 using the imaging device 110, and converts the image of the target surface into a frontal image using the image processing device 150 to evaluate the condition of the sprocket 211, such as wear. When the operator starts the condition evaluation device 100, the image processing device 150 first executes process P1, which displays and acquires settings.

In this process P1, the image processing device 150 causes the display device 140 to display, for example, a threshold value to be used in process P13, which will be described later. The operator checks the threshold value displayed on the display device 140. This threshold value is, for example, a threshold value for the height of the unevenness relative to a first mean plane H1, which will be described later. Furthermore, if the operator wishes to change the threshold value displayed on the display device 140, he or she operates the input device 130 to input a new threshold value. In this case, the image processing device 150 obtains the new threshold value input by the operator from the input device 130 and stores it in the storage device 153.

In addition, in this process P1, the image processing device 150 causes the display device 140 to display, for example, the necessity of a secondary rotation, which rotates the front-on image of the target surface 211f of the sprocket 211 obtained in the process described below. When the operator wishes to change the necessity of the secondary rotation displayed on the display device 140, he or she operates the input device 130 to newly input the necessity of the secondary rotation. In this case, the image processing device 150 obtains the necessity of the secondary rotation input by the operator from the input device 130 and stores it in the storage device 153.

In addition, in this process P1, the image processing device 150, for example, causes the display device 140 to display whether or not automatic shooting is to be performed by the imaging device 110. When the operator wishes to change the whether or not automatic shooting is to be performed displayed on the display device 140, the operator operates the input device 130 to input a new whether or not automatic shooting is to be performed. In this case, the image processing device 150 obtains the whether or not automatic shooting is to be performed input by the operator from the input device 130 and stores it in the storage device 153.

Next, the image processing device 150 executes a process P2 to determine whether or not to perform a secondary rotation, for example, to rotate the front-on image of the target surface of the sprocket 211. If the presence or absence of secondary rotation stored in the storage device 153 is "yes," the image processing device 150 executes a process P3 to display guide lines.

In process P3, the image processing device 150 displays guide lines corresponding to the contour shape of the sprocket 211, which is the part to be evaluated, on the display device 140, such as the viewfinder of the imaging device 110. The operator adjusts the attitude of the imaging device 110, for example, so that the guide lines displayed on the display device 140 match the contour shape of the sprocket 211.

Next, the image processing device 150 executes, for example, process P4 to determine whether or not automatic shooting is performed by the imaging device 110. If the information on whether or not automatic shooting is performed stored in the storage device 153 is "yes" (YES), the image processing device 150 executes process P5 to determine whether or not the conditions for automatic shooting are satisfied.

Figure 4 is an image diagram of the target surface 211f of the sprocket 211 and a plurality of point clouds on the target surface displayed on a display device 140 such as a viewfinder of the imaging device 110. Figure 5 is an image diagram showing an example of the sprocket 211 and guide line GL displayed on the display device 140. For example, as shown in Figure 4, the image processing device 150 sets a first point cloud G1, a second point cloud G2, and a third point cloud G3 that are spaced apart from each other on the target surface 211f of the sprocket 211, which is the evaluation target of the condition evaluation device 100.

The first point group G1, the second point group G2, and the third point group G3 each include a plurality of points p. The first point group G1, the second point group G2, and the third point group G3 may each include n points p, where n is a natural number equal to or greater than 2. The image processing device 150 can easily change, for example, the value of n, which is the number of points p, the position of each point group, and the position of the points p included in each point group, depending on the application.

The number of points p included in each of the first point group G1, the second point group G2, and the third point group G3 is not particularly limited, but may be, for example, 2 or more, 3 or more, 5 or more, or 10 or more. The distance between any two points p included in any one point group is closer than, for example, the distance between any one point p included in one of the two point groups and any one point p included in the other point group. In addition, the image processing device 150 sets the positions of the multiple points p included in each point group so that the three points p selected one by one from each point group do not exist on a straight line.

The distance sensor 120 measures the distance to each point p included in each of the first point group G1, the second point group G2, and the third point group G3, and outputs the distance to the image processing device 150. In addition, the imaging device 110 or the distance sensor 120, for example, as described above, irradiates a visible light laser from a light source to visualize multiple points p on the target surface 211f to which the distance is to be measured.

In process P5, in which the image processing device 150 determines whether the conditions for automatic photography are satisfied, the image processing device 150 displays a guide line GL on the display device 140, for example, as shown in the upper diagram of FIG. 5. The operator adjusts the orientation of the imaging device 110 so that a reference point, such as the tip of the tooth of the sprocket 211, is positioned on the guide line GL, for example, as shown in the lower diagram of FIG. 5. More specifically, the image processing device 150 detects the outline of the sprocket 211 on the guide line GL as a reference point, for example, based on distance information obtained from the distance sensor 120.

In this case, in process P5, when the image processing device 150 determines that the number of reference points on the guide line GL has decreased from three or more to two, as shown in FIG. 5, it determines that the condition is satisfied (YES). Then, the image processing device 150 executes process P8, for example, to acquire the coordinates of the reference point of the sprocket 211 from the distance sensor 120 and to capture an image of the target surface 211f of the sprocket 211, which is the subject, using the imaging device 110.

On the other hand, in process P2 for determining whether or not to perform the secondary rotation, if the presence or absence of secondary rotation is "no" (NO), the image processing device 150 executes process P6 for displaying, for example, a plurality of point clouds including the first point cloud G1, the second point cloud G2, and the third point cloud G3. In this process P6, the image processing device 150 irradiates a visible light laser from a light source such as the imaging device 110 or the distance sensor 120, and displays a plurality of points p of each of the first point cloud G1, the second point cloud G2, and the third point cloud G3 on the target surface 211f on the display device 140. Furthermore, the image processing device 150 executes process P7 for determining whether or not the operator has performed a shooting operation.

The operator performs manual shooting operations, for example by adjusting the orientation of the imaging device 110 so that the first point group G1, the second point group G2, and the third point group G3 displayed on the display device 140 are appropriately positioned on the target surface 211f, and pressing the shutter button of the imaging device 110. If the image processing device 150 determines that the operator has performed an imaging operation (YES), it executes process P8 in which the imaging device 110 captures an image of the target surface 211f of the sprocket 211. If the image processing device 150 determines in process P7 that the operator has not performed an imaging operation (NO), it repeats the determination process P7.

In addition, in the process P4 for determining whether automatic photography is performed as described above, if the determination of whether automatic photography is performed is "no" (NO), the image processing device 150 executes process P7 for determining whether the operator has performed a photography operation. The operator performs manual photography operation by adjusting the orientation of the imaging device 110 so that the guide line GL displayed on the display device 140, such as the viewfinder of the imaging device 110 in the process P3 described above, is positioned on the reference point of the tooth tip of the sprocket 211.

If the image processing device 150 determines that the operator has performed an imaging operation (YES), it executes process P8, in which the imaging device 110 captures an image of the target surface 211f of the sprocket 211, and if it determines that the operator has not performed an imaging operation (NO), it repeats determination process P7.

Next, the image processing device 150 executes a process P9, for example, as shown in FIG. 4, to superimpose the image of the target surface 211f of the sprocket 211 captured in process P8 on the screen of the display device 140 along with the first point group G1, the second point group G2, and the third point group G3. Furthermore, the image processing device 150 executes a process P10 to acquire, for example, the success or failure of the image capture.

In this process P10, the image processing device 150, for example, causes the display device 140 to display a confirmation display such as text, an icon, or an image for confirming whether the image capture was successful. The operator, for example, visually checks each point p of the first point group G1, the second point group G2, and the third point group G3 visualized on the display device 140, and the confirmation display. The operator, for example, inputs the success or failure of the image capture to the input device 130 by pressing a touch panel in accordance with the confirmation display on the display device 140.

More specifically, for example, if each point p is located on the target surface 211f of the sprocket 211, the operator inputs "yes" to the input device 130, indicating that the imaging was successful, and otherwise inputs "no" to the input device 130, indicating that the imaging was unsuccessful. The image processing device 150 acquires from the input device 130 the success or failure of the imaging input by the operator via the input device 130.

Next, the image processing device 150 executes process P11 to determine whether the imaging acquired from the input device 130 was successful. In this process P11, if the image processing device 150 determines that the imaging acquired in process P10 was not successful (NO), it returns to the above-mentioned process P2. On the other hand, in this process P11, if the image processing device 150 determines that the imaging acquired in process P10 was successful (YES), it executes process P12 to calculate the first mean plane.

Figure 6 is a conceptual diagram of the first plane group PG1 calculated in process P12, and the unit normal vectors NV1, NV2, ..., NVn of each plane included in the first plane group PG1. In process P12, the image processing device 150 acquires, for example, the distance from the distance sensor 120 to each point p in the first point group G1 shown in Figure 4, the distance to each point p in the second point group G2, and the distance to each point p in the third point group G3.

The image processing device 150, for example, sets an orthogonal coordinate system consisting of the X-axis, Y-axis, and Z-axis with the distance sensor 120 as the origin, and calculates the coordinates of each point p. Furthermore, the image processing device 150 executes a plane calculation process for all combinations of points p, in which a plane is calculated based on one point p selected from the first point group G1, one point p selected from the second point group G2, and one point p selected from the third point group G3. This allows the image processing device 150 to calculate the first plane group PG1 as shown in FIG. 6.

Figure 7 is an explanatory diagram of the plane calculation process executed by the image processing device 150 in process P12. For example, the coordinates of an arbitrary point p1 included in the first point group G1 are (x1, y1, z1), the coordinates of an arbitrary point p2 included in the second point group G2 are (x2, y2, z2), and the coordinates of an arbitrary point p3 included in the third point group G3 are (x3, y3, z3). If the equation of the plane passing through these points p1, p2, and p3 is ax + by + cz + d = 0, the unit normal vector NV of this plane is parallel to the vector V = (a, b, c).

Here, if vector α = (x, y, z) = (x2 - x1, y2 - y1, z2 - z1) and vector α' = (x', y', z') = (x3 - x1, y3 - y1, z3 - z1), then the cross product of these is α x α' = (yz' - y'z, zx' - z'x, xy' - x'y). The cross product α x α' of vector α and vector α' is parallel to the unit normal vector NV of the above plane.

Therefore, if (a, b, c) = α x α', then d = -(ax1 + by1 + cz1) (∵ ax1 + by1 + cz1 + d = 0). From the above, the equation of the above plane can be found, and the unit normal vector NV is (α x α') / |α x α'|. Note that -1 times this unit normal vector NV also becomes a unit normal vector on the opposite side of the above plane, but if the first point group G1, the second point group G2, and the third point group G3 are fixed, all of the ⁿ³ unit normal vectors NV will be aligned on the same side of the above plane.

Next, the image processing device 150 executes a process P13 in which, for example, the unevenness of the target surface 211f of the sprocket 211 is displayed in the color of each point p. Figure 8 is a conceptual diagram of the first mean plane H1 and mean vector V1 calculated by the image processing device 150 in this process P13.

In this process P13, the image processing device 150 first calculates a first average plane H1. The first average plane H1 is a plane whose normal vector is an average vector V1 obtained by averaging the unit normal vectors NV of all the planes in the first plane group PG1, and which has a minimum sum of distances to all the points p included in the first point group G1, the second point group G2, and the third point group G3. The number of all the points p included in the first point group G1, the second point group G2, and the third point group G3 is, for example, 3n. If the formula for the first average plane H1 is ax+by+cz+d=0, a, b, and c can be found from the above average vector, and d can be easily calculated using the least absolute value method or the like.

Furthermore, in process P13, the image processing device 150 acquires a threshold value that has been set in advance or that has been newly set in process P1 and stored in the storage device 153. As described above, this threshold value is the threshold value of the height h of the unevenness relative to the first mean plane H1. Based on the coordinates of each point p in the first point group G1, the second point group G2, and the third point group G3, the image processing device 150 recognizes, as a protruding point, a point p whose distance from the distance sensor 120 is closer than the first mean plane H1 and whose protruding height h from the first mean plane H1 is equal to or greater than the threshold value Th [mm].

Furthermore, based on the coordinates of each point p in the first point group G1, the second point group G2, and the third point group G3, the image processing device 150 recognizes as a sunken point a point whose distance from the distance sensor 120 is farther than the first mean plane H1 and whose height h of the sunken point from the first mean plane H1 is equal to or greater than a threshold value Td [mm]. For example, the image processing device 150 causes the display device 140 to display the protruding points, the sunken points, and other points in different colors. For example, the image processing device 150 causes the display device 140 to display the heights at which the protruding points protrude from the first mean plane H1 and the heights at which the sunken points sink from the first mean plane H1.

Next, the image processing device 150 executes process P14 to acquire the operation content. In this process P14, the image processing device 150 causes the display device 140 to display, for example, images, characters, or icons indicating options for the operation content. The options for the operation content are, for example, "take a picture again," "change the threshold," and "proceed to the next process." The image processing device 150 acquires the operation content input by the operator via the input device 130, for example, a touch panel.

For example, if the operator recognizes that the raised points displayed on the display device 140 in process P13 are due to removable foreign matter such as mud adhering to the target surface 211f of the sprocket 211, the operator can select the option to "take a new photograph" after removing the foreign matter. Also, if the target surface of the subject of the imaging device 110 is, for example, part of the surface of a large-diameter sphere that can be regarded as a pseudo-flat surface, it is possible that a point p on the target surface will be recognized as a raised point or a depressed point even if there are no bumps or depressions. In such a case, the operator can select the option to "proceed to next process" to use all points p on the target surface without excluding points p recognized as raised points or depressed points.

Next, the image processing device 150 executes process P15 to determine whether or not to perform re-shooting. In this process P15, if the image processing device 150 determines that the operation content acquired in the above-mentioned process P14 is "re-shooting" (YES), it returns to the above-mentioned process P2. Also, in this process P15, if the image processing device 150 determines that the operation content acquired in the above-mentioned process P14 is not "re-shooting" (NO), it executes the next process P16.

In process P16, the image processing device 150 determines whether the operation content acquired in the above-mentioned process P14 is "changing the threshold value." If the image processing device 150 determines that the operation content is "changing the threshold value" (YES), it executes process P17 to acquire the threshold value, as in the above-mentioned process P1, and returns to the above-mentioned process P13. If the image processing device 150 determines that the operation content is not "changing the threshold value" (NO), it executes process P18 to acquire the presence or absence of exclusion.

In process P18, the image processing device 150, for example, causes the display device 140 to display "normal points" and "all points" as options. The operator selects an option via the input device 130, for example, by pressing a touch panel. The image processing device 150 acquires the option input by the operator from the input device 130.

Next, the image processing device 150 executes process P19 to determine whether or not exclusion is required. If the option "all points" is obtained in the above-mentioned process P18, the image processing device 150 determines in process P19 that there is no exclusion (NO), and executes process P20 to obtain the first mean plane H1 calculated in the above-mentioned process P12 and the mean vector V1, which is its normal vector. In this case, the image processing device 150 uses the first mean plane H1 in process P22 to calculate the next rotation angle.

On the other hand, if the option "normal point" is obtained in the above-mentioned process P18, the image processing device 150 determines in process P19 that there is exclusion (YES) and executes process P21 to calculate the second mean plane H2.

Figure 9 is a conceptual diagram of process P21 for calculating the second plane group PG2. The second plane group PG2 is calculated in the same manner as the first plane group PG1 calculated in the above-mentioned process P12, except that points p recognized as protruding points or depressed points in the above-mentioned process P13 are excluded and other points p are used. By excluding protruding points and depressed points in this way, for example, a plane having a unit normal vector NVn whose angle is significantly different from the unit normal vectors NV1, NV2, and NV3 of the other planes in the first plane group PG1 shown in Figure 6 can be excluded from the second plane group PG2 shown in Figure 9.

Furthermore, in process P21, the image processing device 150 calculates a second mean plane H2 in the same manner as the first mean plane H1 calculated in the above-mentioned process P13. In this case, the image processing device 150 uses the second mean plane H2 in process P22, which calculates the next rotation angle. The second mean plane H2 is a plane whose normal vector is the average vector V2 obtained by averaging the unit normal vectors NV of all the planes in the second plane group PG2, and which is the plane that has the smallest sum of distances to the remaining points p, excluding the protruding points and the depressed points, from all the points p in the first point group G1, the second point group G2, and the third point group G3.

Next, the image processing device 150 executes a process P22 to calculate a rotation angle for converting the first mean plane H1 or the second mean plane H2, which is not a front-facing image, into a front-facing image. This process P22 will be described below with reference to Figs. 10 to 14. Here, a front-facing image is, for example, an image that faces the imaging surface of the imaging device 110. The imaging surface is, for example, the light receiving surface of an imaging element, which is a surface perpendicular to the optical axis of the lens of the imaging device 110.

The image processing device 150, for example, determines a point (x', y', z') on the first mean plane H1 or the second mean plane H2 that corresponds to all coordinates (x, y) of the image G captured by the imaging device 110 as shown in FIG. 10. If it is assumed that there is no distortion of the lens of the imaging device 110, this point (x', y', z') may be determined based on a vector from (x, y) toward the lens. Furthermore, the point (x', y', z') may be determined taking into account the distortion of the lens of the imaging device 110.

Assume that the first average plane H1 or the second average plane H2, which is not a frontal image, is expressed by the formula: a x + by + c z + d = 0, and (a, b, c) ≠ (0, 0, c). In this case, for example, as shown in Figure 11, the coordinate system can be reset as z ← z-d/c, and a, b, and c can be normalized to a plane passing through the origin expressed by the formula: a x + by + c z + d = 0 and a ² + b ² + c ² = 1, without loss of generality. Any point p on this plane is expressed by
(c, 0, -a) ⊥ (a, b, c),
(0, c, -b)⊥(a, b, c),
Using parameters s and t, it is expressed by the following equation (1).

p(x,y)=s(c,0,-a)+t(0,c,-b)...(1)

This constitutes the plane of the double triangle shown in FIG.
When this point p is rotated around the y axis by a rotation angle θ, where θ=arctan(a/c), a point p′ expressed by the following equation (2) is obtained.

p'(x',y')=s(c・secθ,0,0)
+t(b・sinθ,c,−b・cosθ)...(2)

This point p' includes the x-axis (it becomes the x-axis when t = 0) and forms the plane of the double triangle shown in Figure 12. If this point p' is rotated around the x-axis by a rotation angle φ, where φ = arctan(b · cos θ/c), then point p" is obtained, as expressed by the following equation (3).

p"(x",y")=s(c・secθ,0,0)
+t (b・sinθ, b・cosθ・sinφ+c・cosφ, -b・cosθ)
=(s・D+t・a・b/D, t・c/D, 0) ...(3)

However, in equation (3), D = ( ^a2 + ^c2 ) ^1/2 . Since this point p" coincides with the xy plane (z coordinate is always 0), it is a front-facing image and forms the plane of the double triangle shown in Figure 13. As described above, in process P22, the image processing device 150 calculates the rotation angles θ and φ for converting the first mean plane H1 or the second mean plane H2 into a front-facing image by rotating it so that it is parallel to the imaging surface of the imaging device 110.

Next, the image processing device 150 executes a process P23 for calculating a front-facing image, as shown in FIG. 3B. In this process P23, the image processing device 150 obtains a point (x", y", z") corresponding to all coordinates (x, y) of the image G captured by the imaging device 110 on the first mean plane H1 or the second mean plane H2 rotated so as to be parallel to the imaging surface of the imaging device 110, as shown in FIG. 14.

To convert the image of the target surface 211f of the sprocket 211, which is not a front-on image, into a front-on image, the color of the pixel at coordinates (x, y) is transferred to the pixel at coordinates (x", y"). The color of a pixel is generally represented by the numerical values of each of the three primary colors (R, G, B). Even if x and y are positive values, this does not necessarily mean that x", y" will be positive values.

In the example shown in FIG. 15, for example, the area of the pixel with coordinates (x, y) in the pre-conversion image corresponds to the area of coordinates (x", y") in the post-conversion image. This area of coordinates (x", y") in the post-conversion image spans, for example, four pixels in the post-conversion image. In this case, the numerical values of the three primary colors (R, G, B) of the pixel in the pre-conversion image are multiplied by the percentage of the area that the pixel in the pre-conversion image overlaps with each pixel in the post-conversion image, and the result is added to the numerical values of the three primary colors (R, G, B) of each pixel in the post-conversion image.

More specifically, in the example shown in FIG. 15, the area at coordinates (x", y") in the converted image overlaps with 1/4 of the area of each of the four pixels in the converted image. Also, the numerical values of the three primary colors of the pixel at coordinates (x, y) in the pre-conversion image are, for example, (R, G, B) = (128, 64, 192). In this case, (R, G, B) = (32, 16, 48), which is 1/4 of the numerical values of the three primary colors of the pixel at coordinates (x, y) in the pre-conversion image, is added to the numerical values of the three primary colors (R, G, B) of each of the four pixels in the converted image. This makes it possible to determine the numerical values of the three primary colors of the four pixels in the converted image.

As described above, in process P23 for calculating a front-facing image, the image processing device 150 makes the numerical values of the three primary colors of the pixel at coordinates (x, y) in the image before conversion correspond to the numerical values of the three primary colors of the pixel in the image after conversion. This makes it possible to generate a front-facing image in which the first mean plane H1 or the second mean plane H2 is directly opposed to the imaging surface of the imaging device 110 from the image of the target surface 211f of the sprocket 211 captured by the imaging device 110.

Next, the image processing device 150 executes process P24 to determine whether or not a secondary rotation is to be performed, as shown in FIG. 3B, for example. In this process P24, if the information on whether or not a secondary rotation is to be performed, which is stored in advance in the storage device or input in the above-mentioned process P1, is "no," the image processing device 150 determines that a secondary rotation is not to be performed (NO), and ends the process shown in FIG. 3A and FIG. 3B.

In addition, in process P24, if the secondary rotation information stored in advance in the storage device or input in the above-mentioned process P1 is "Yes," the image processing device 150 determines that secondary rotation will be performed (YES) and executes process P25 to calculate a front-on image.

Figure 16 shows a plan view of a portion of a sprocket 211 to explain whether or not secondary rotation is required, and a side view of two types of sprockets 211A and 211B.

As the sprocket 211 of the work machine 200 shown in FIG. 2, for example, one of two types of sprockets 211A and 211B as shown in FIG. 16 can be used. One of the sprockets, 211A, has, for example, a tooth width at the tooth tip 211t that is roughly equal to the tooth width at the tooth base 211b, and the other sprocket, 211B, has, for example, a tooth width at the tooth tip 211t that is narrower than the tooth width at the tooth base 211b.

For sprocket 211A, where the tooth width at the tooth tip 211t is roughly equal to the tooth width at the tooth bottom 211b, an image is taken of a plane perpendicular to the rotation axis as the target surface 211f, and the image is converted into a front-on image. This allows the condition of the teeth of sprocket 211, which are most susceptible to wear, to be properly evaluated.

On the other hand, in the case of sprocket 211B, where the tooth width at the tooth tip 211t is narrower than the tooth width at the tooth base 211b, even if an image is taken with a plane perpendicular to the rotation axis as the target surface 211f and the image is converted into a front-on image, the side of the tooth is inclined at a certain angle Ψ with respect to the target surface 211f. Therefore, in this state, it is not possible to properly evaluate the condition of the tooth part of sprocket 211B, which is most susceptible to wear.

Therefore, for sprocket 211B, where the tooth width of the tooth tip 211t is narrower than the tooth width of the tooth bottom 211b, it is necessary to convert the image of target surface 211f perpendicular to the rotation axis into a front-facing image, and then perform a secondary rotation to rotate this front-facing image by a predetermined angle Ψ. This makes it possible to obtain a front-facing image of target surface 211f' by treating the side of the tooth of sprocket 211B as a new target surface 211f'.

More specifically, the image of the target surface 211f of the sprocket 211B shown in the lower diagram of Figure 5, captured in the above-mentioned process P8, is converted into a front-facing image as shown in the upper diagram of Figure 17 through the above-mentioned processes P12 to P23. Here, as shown in Figure 17, a Cartesian coordinate system is set with an x-axis whose positive direction is to the right of the front-facing image, a y-axis whose positive direction is to the top of the front-facing image, and a z-axis perpendicular to the front-facing image.

In process P25, the image processing device 150 rotates the front-facing image by a predetermined angle λ around the z-axis so that the guide line GL passing through the reference point of the tooth tip of the sprocket 211B shown in the upper diagram of Figure 17 becomes parallel to the x-axis, to obtain a front-facing image of the target surface 211f as shown in the center of Figure 17. The image processing device 150 further rotates this front-facing image of the target surface 211f as shown in the center of Figure 17 by a predetermined angle Ψ around the x-axis.

As a result, as shown in the lower diagram of FIG. 17, a front-on image after the secondary rotation can be obtained in which the side of the tooth of the sprocket 211B is set as the target surface 211f' and the target surface 211f' is directly facing the imaging surface of the imaging device 110. This ends process P25, and the image processing device 150 ends the process shown in FIG. 3A and FIG. 3B. The front-on image may be stored in the storage device 153.

The operation of the image processing device 150 of this embodiment is described below.

For example, to evaluate the condition of a mechanical part based on an image, such as measuring the amount of wear on the sprocket 211 of the work machine 200, a front-on image of the mechanical part is required in order to improve the evaluation accuracy. However, a mechanical part, such as the sprocket 211, may have unevenness on the target surface 211f, or may have foreign matter such as mud or oil attached to the target surface 211f. When such a mechanical part is used as a subject, the conventional image processing device described above has difficulty measuring the orientation of the mechanical part based on the imaging means, and may not be able to correct distortion caused by oblique shooting of the image captured by the imaging means.

In contrast, the image processing device 150 of this embodiment is a device that acquires an image of the target surface 211f from the imaging device 110 and acquires the distance from the target surface 211f from the distance sensor 120 to convert the image of the target surface 211f into a front-facing image, and has the following functions. The image processing device 150 sets a first point group G1, a second point group G2, and a third point group G3 that are spaced apart from each other on the target surface 211f, and acquires the distance from the distance sensor 120 to each point p of the first point group G1, the distance to each point p of the second point group G2, and the distance to each point p of the third point group G3. The image processing device 150 also executes a plane calculation process for all combinations of points to calculate a plane based on one point p selected from the first point group G1, one point p selected from the second point group G2, and one point p selected from the third point group G3, to calculate the first plane group PG1. The image processing device 150 also calculates a first average plane H1 that uses an average vector V1 obtained by averaging the unit normal vectors NV of all planes in the first plane group PG1 as a normal vector and that minimizes the sum of the distances between the first point group G1, the second point group G2, and the third point group G3 and all points p. The image processing device 150 then converts the image of the target surface 211f into a front-facing image based on the first average plane H1.

With this configuration, the image processing device 150 of this embodiment can convert an image of the target surface 211f having unevenness of a machine part such as a sprocket 211, which is a subject, photographed obliquely into a front-facing image in which the target surface 211f is directly faced to the imaging surface of the imaging device 110. More specifically, as described above, the image processing device 150 of this embodiment calculates a first plane group PG1 by setting a first point group G1, a second point group G2, and a third point group G3 that are spaced apart from each other on the target surface 211f, and further calculates a first average plane H1 from the first plane group PG1. As a result, the influence of local unevenness or foreign matter on the target surface 211f is diluted, reduced, or removed, and a first average plane H1 that is substantially parallel to the target surface 211f can be obtained. By converting the image of the target surface 211f into a front-facing image based on this first average plane H1, an image of the target surface 211f photographed obliquely into a front-facing image can be converted.

Moreover, the image processing device 150 of this embodiment sets n points in each of the first point group G1, the second point group G2, and the third point group G3, and calculates the first plane group PG1 including ⁿ³ planes by plane calculation processing. With this configuration, the image processing device 150 of this embodiment can calculate a more average first plane group PG1 because the first point group G1, the second point group G2, and the third point group G3 each have an equal number of points p.

The image processing device 150 of this embodiment also displays the image of the target surface 211f acquired from the imaging device 110 on the display device 140 together with the first point group G1, the second point group G2, and the third point group G3. With this configuration, the image processing device 150 of this embodiment allows the operator to adjust the positions of the first point group G1, the second point group G2, and the third point group G3, and to determine whether imaging by the imaging device 110 was successful.

The image processing device 150 of this embodiment also displays points p whose distance to the first mean plane H1 is equal to or greater than a threshold value in different colors from other points p on the display device 140. With this configuration, the image processing device 150 of this embodiment allows the operator to adjust the positions of the first point group G1, the second point group G2, and the third point group G3, and to determine whether or not imaging by the imaging device 110 has been successful.

In addition, the image processing device 150 of this embodiment excludes points p whose distance to the first mean plane H1 is equal to or greater than a threshold value from the first point group G1, the second point group G2, and the third point group G3, and performs a plane calculation process for all combinations of the remaining points p to calculate the second plane group PG2. In addition, the image processing device 150 calculates a second mean plane H2 that uses an average vector V2 obtained by averaging the unit normal vectors NV of all planes in the second plane group PG2 as a normal vector and that has the smallest sum of distances to the remaining points p. Then, the image processing device 150 calculates the rotation angles θ and φ of the second mean plane H2 so that the normal vector of the second mean plane H2 is parallel to the normal vector of the image plane of the frontal image, and converts the image of the target surface 211f into a frontal image based on the rotation angles θ and φ. With this configuration, the image processing device 150 of this embodiment can convert the image of the target surface 211f into a frontal image by removing the effects of local unevenness and foreign matter on the target surface 211f.

In addition, the image processing device 150 of this embodiment acquires a threshold value for the distance to the first mean plane H1 from the input device 130. With this configuration, the image processing device 150 of this embodiment allows the operator to change the threshold value according to the situation.

The image processing device 150 of this embodiment also calculates the rotation angles θ and φ of the first mean plane H1 so that the normal vector of the first mean plane H1 is parallel to the normal vector of the image plane of the front-facing image, and converts the image of the target surface 211f into a front-facing image based on the rotation angles θ and φ. With this configuration, the image processing device 150 of this embodiment can calculate the first mean plane H1 using all points p included in the first point group G1, the second point group G2, and the third point group G3, and further convert the image of the target surface 211f into a front-facing image based on the first mean plane H1.

The image processing device 150 of this embodiment also obtains an image of the target surface 211f', which is an inclined surface inclined relative to the target surface 211f, from the imaging device 110, and obtains the distance from the target surface 211f', which is an inclined surface, from the distance sensor 120 to convert the image of the target surface 211f' into a front-on image. With this configuration, the image processing device 150 of this embodiment can obtain a front-on image of an inclined surface inclined relative to the target surface 211f, such as the side of a tooth of a sprocket 211B as shown in the lower diagram of Figure 16, and can more accurately evaluate the condition of wear, etc.

The mechanical part condition evaluation device 100 of this embodiment also includes an imaging device 110, a distance sensor 120, and the image processing device 150 described above. With this configuration, the condition evaluation device 100 of this embodiment can achieve the same effect as the image processing device 150 described above, and can convert an image of the subject's uneven target surface 211f captured obliquely into a front-on image in which the target surface is directly facing the imaging surface. In addition, if the mechanical part condition evaluation device 100 includes a storage device 153, as described above, the front-on image can be stored in the storage device 153, and when a front-on image is subsequently captured, the difference between the image and the previous front-on image can be compared to calculate the degree of wear.

As described above, according to this embodiment, it is possible to provide an image processing device 150 that can convert an image of the uneven target surface 211f of a subject captured obliquely into a front-on image in which the target surface is directly opposed to the imaging surface of the imaging device 110.

The above describes in detail an embodiment of the image processing device according to the present disclosure using the drawings, but the specific configuration is not limited to this embodiment, and even if there are design changes, etc., within the scope that does not deviate from the gist of this disclosure, they are included in this disclosure.

110 Imaging device 120 Distance sensor 130 Input device 140 Display device 150 Image processing device 211f Target surface 211f' Target surface (inclined surface)
G1 First point group G2 Second point group G3 Third point group H1 First mean plane H2 Second mean plane NV Unit normal vector NV1 Unit normal vector NV2 Unit normal vector NV3 Unit normal vector NVn Unit normal vector p Point PG1 First plane group PG2 Second plane group V1 Mean vector V2 Mean vector θ Rotation angle φ Rotation angle

Claims (5)

1. An image processing device that acquires an image of a target surface of an object from an imaging device, acquires a distance between the imaging device and the target surface from a distance sensor, and converts the image of the target surface into a frontal image,
setting a first point group, a second point group, and a third point group on the target surface such that the first point group, the second point group, and the third point group are spaced apart from each other and each point selected from the first point group is not on the same line ;
n points (n is a natural number equal to or greater than 2) are set in each of the first point group, the second point group, and the third point group;
acquiring, from the distance sensor, a distance from the imaging device to each point of the first point group, a distance from the imaging device to each point of the second point group, and a distance from the imaging device to each point of the third point group;
calculating a first group of planes by executing a plane calculation process for calculating a plane based on one point selected from the first group of points, one point selected from the second group of points, and one point selected from the third group of points for all combinations of points;
The first plane group including ⁿ³ planes is calculated by the plane calculation process ;
calculating a first average plane that is a normal vector obtained by averaging unit normal vectors of all planes in the first plane group, and that minimizes the sum of distances between the first point group, the second point group, and all points included in the third point group;
2. An image processing device comprising: a first mean plane that converts an image of the target surface into the front-facing image based on the first mean plane. The image processing device according to claim 1 , wherein the image of the target surface acquired from the imaging device is displayed on a display device together with the first point cloud, the second point cloud, and the third point cloud. 3. The image processing apparatus according to claim 2 , wherein points whose distance to said first mean plane is equal to or greater than a threshold value are displayed in different colors on said display device from other points. excluding points whose distances to the first mean plane are equal to or greater than a threshold value from the first point group, the second point group, and the third point group, and performing the plane calculation process on all combinations of remaining points to calculate a second plane group;
a second average plane having a minimum sum of distances to the remaining points is calculated by averaging the unit normal vectors of all the planes in the second group of planes as a normal vector;
Calculating a rotation angle of the second mean plane so that the normal vector of the second mean plane is parallel to a normal vector of an image plane of the frontal image;
4. The image processing device according to claim 1 , wherein the image of the target surface is converted into the front-facing image based on the rotation angle. Calculating a rotation angle of the first mean plane so that the normal vector of the first mean plane is parallel to a normal vector of an image plane of the frontal image;
The image processing apparatus according to claim 1 , wherein the image of the target surface is converted into the front-facing image based on the rotation angle.

Images