JP7799225B2 - Image processing device, image processing method and program - Google Patents

Description

This disclosure relates to an image processing device, an image processing method, and a program.

Currently, infrastructure inspections are primarily performed visually by inspectors. Image recognition-based deterioration detection technology is being widely considered as an alternative to visual inspection. For example, Non-Patent Documents 1 and 2 describe methods for detecting cracks in concrete or asphalt. The methods described in Non-Patent Documents 1 and 2 use deep learning techniques to detect minute cracks in acquired images with high accuracy.

Mohamed Alipour, Devin K Harris, and Gregory R miller S. M. S. et al. “Robust Pixel-Level Crack Detection Using Convolutional Neural Networks”, Journal of Computing in Civil Engineering, 33(6), 04019040, 2019. K. Zenko, A. Igo, Y. Minamimenra, K. Kuroki, and K. Okubo, "Evaluation of crack rate using deep learning from pavement images captured by an in-vehicle camera," Journal of the Japan Society of Civil Engineers (E1), Vol. 73, No. 3, pp. I_97-I_105, 2017.

The methods described in Non-Patent Documents 1 and 2 are based on the assumption that only concrete surfaces are captured in the acquired images, which imposes limitations on the image capture, such as the requirement that only concrete surfaces be captured within the camera's field of view. To efficiently inspect infrastructure facilities on-site using images, it is necessary to use images of various facilities captured simultaneously using equipment such as an MMS (Mobile Mapping System), rather than images captured under the aforementioned limitations. However, when attempting to capture various infrastructure facilities simultaneously using an MMS or similar device, it is difficult to capture the size of the infrastructure facilities being inspected uniformly in the captured images, and the size of the infrastructure facilities in the images may be so large that it is impossible to determine whether they have deteriorated.

For example, consider the case of capturing images of infrastructure facilities such as guardrails or signs using an MMS. In this case, it is difficult to maintain a constant shooting distance of the infrastructure facilities from the vehicle due to factors such as the lane of travel of the vehicle equipped with the camera, the road alignment, and the location of the infrastructure facilities. It is also difficult to consistently capture the infrastructure facilities at a consistent size or composition in the captured image. As a result, infrastructure facilities located far away may appear within the field of view of the captured image. Areas where distant infrastructure facilities are captured may not be able to detect any deterioration in the infrastructure facilities, making it undesirable to subject such areas to inspection. Therefore, a method is needed to appropriately determine inspection areas to inspect for deterioration of objects such as infrastructure facilities in images captured by the camera.

The purpose of this disclosure, made in consideration of the above-mentioned problems, is to provide an image processing device, image processing method, and program that can appropriately determine an inspection area for inspecting whether or not an object, such as infrastructure equipment, has deteriorated in an image of the object.

In order to solve the above problem, the image processing device disclosed herein is an image processing device that determines an inspection area in an image of an object captured to check for the presence or absence of deterioration of the object, and includes: a depth estimation unit that estimates the depth, which is the distance in the depth direction in the image, for each pixel that constitutes the image; an inspection area determination unit that determines the depth of the area of the object in the image where the ratio of the size of the object to the size of the deterioration to be inspected is equal to or greater than a predetermined value as a depth threshold, and determines an area closer to the depth threshold as the inspection area based on the depth for each pixel estimated by the depth estimation unit; and a result output unit that outputs the inspection area determination result obtained by the inspection area determination unit.

In addition, to solve the above problem, the image processing method disclosed herein is an image processing method by an image processing device that determines an inspection area in an image of an object captured to check for the presence or absence of deterioration of the object, and includes the steps of: estimating the depth, which is the distance in the depth direction in the image, for each pixel constituting the image; determining the depth of the area of the object in the image where the ratio of the size of the object to the size of the deterioration to be inspected is equal to or greater than a predetermined value as a depth threshold; determining an area closer to the depth threshold as the inspection area based on the estimated depth for each pixel; and outputting the inspection area determination result.

In addition, to solve the above problem, the program disclosed herein causes a computer to operate as the above-mentioned image processing device.

The image processing device, image processing method, and program disclosed herein make it possible to appropriately determine an inspection area in an image of an object to check for deterioration of the object.

FIG. 1 is a diagram illustrating an example of the configuration of an image processing device according to an embodiment of the present disclosure. 2 is a diagram for explaining depth estimation by a depth estimation unit shown in FIG. 1 . FIG. 2 is a flowchart showing an example of the operation of the image processing device shown in FIG. 1 . 2 is a flowchart showing an example of the operation of a determination unit shown in FIG. 1 . FIG. 10 is a diagram for explaining the number of pixels of an object required for inspection. This is an image of a guardrail as an object. This is an annotation image in which information about the pixel area of the guardrail is added to the image shown in FIG. 6A. FIG. 6B is a diagram showing the depth estimation results of the pixel region of the guardrail in the image shown in FIG. 6A. FIG. 10 is a diagram showing an example of the distribution of GR width and depth. FIG. 8 is a diagram illustrating an example of an index (Accuracy) calculated based on the distribution shown in FIG. 7 . 1. FIG. 4 is a diagram for explaining another example of a method for determining a depth threshold α by the threshold determination unit shown in FIG. 1. FIG. 10 is a diagram for explaining yet another example of a method for determining a depth threshold α by the threshold determination unit shown in FIG. FIG. 2 is a diagram illustrating an example of the configuration of a threshold value determination unit illustrated in FIG. 1 . FIG. 2 is a diagram illustrating an example of a hardware configuration of the image processing apparatus illustrated in FIG. 1 .

Embodiments of the present disclosure are described below with reference to the drawings.

Figure 1 is a diagram showing an example configuration of an image processing device 10 according to one embodiment of the present disclosure. The image processing device 10 according to this embodiment determines an inspection area in an image of an object, such as infrastructure equipment, captured by a monocular camera to check for deterioration of the object.

As shown in Figure 1, the image processing device 10 of this embodiment includes an image input unit 11, a depth estimation unit 12, an inspection area determination unit 13, and a result output unit 14.

The image input unit 11 receives an image of an object such as infrastructure equipment. The image input unit 11 outputs the input image to the depth estimation unit 12.

The depth estimation unit 12 estimates the depth, which is the distance in the depth direction at each pixel constituting the image output from the image input unit 11. The depth estimation unit 12 estimates the depth for each pixel using, for example, a depth estimation model (e.g., Midas) based on a convolutional neural network (CNN), a type of deep learning method.

Figure 2 shows an example of the results of depth estimation by the depth estimation unit 12. Figure 2 shows an example of an output image that the depth estimation unit 12 outputs as a result of depth estimation when an image of a guardrail is used as the input image to the depth estimation unit 12.

The depth estimation unit 12 inputs the input image into the depth estimation model and outputs an output image that indicates the relative distance in the depth direction for each pixel as a numerical value (0 (black) to 255 (white)), as shown in Figure 2. By generating such an image, it becomes possible to estimate how far away an object in the image is. Note that Figure 2 shows an example in which the smaller the distance in the depth direction, the larger the value indicating the depth, and the larger the distance in the depth direction, the smaller the value indicating the depth, but this is not limited to this.

Referring again to Figure 1, the depth estimation unit 12 outputs the estimation result to the inspection area determination unit 13.

The inspection area determination unit 13 determines the depth of the area of the object in the image where the ratio of the size of the object to the size of the defect to be inspected is equal to or greater than a predetermined value as the depth threshold α to be used in determining the inspection area. Then, based on the depth of each pixel estimated by the depth estimation unit 12, the inspection area determination unit 13 determines the area closer to the depth threshold α (the area closer to the depth threshold α) as the inspection area. As shown in FIG. 1, the inspection area determination unit 13 includes a threshold determination unit 131 and a determination unit 132.

The threshold determination unit 131 determines the depth threshold α. In this embodiment, a value between 0 and 255 is set as the depth for each pixel, so the depth threshold α is greater than or equal to 0 and less than or equal to 255 (0≦α≦255). However, this is not limited to this, and the depth threshold α can be set appropriately depending on how the depth for each pixel is expressed. Details of how the threshold determination unit 131 determines the depth threshold α will be described later.

The determination unit 132 determines the area closer to the depth threshold α determined by the threshold determination unit 131 as the inspection area based on the depth of each pixel estimated by the depth estimation unit 12.

The result output unit 14 outputs the results of the inspection area determination by the inspection area determination unit 13. The result output unit 14 may, for example, extract and output an image of the area determined to be the inspection area by the inspection area determination unit 13 from the image input to the image input unit 11. Furthermore, the result output unit 14 may, for example, superimpose an image showing the inspection area determined by the inspection area determination unit 13 on the image input to the image input unit 11 and output the image.

Next, we will explain the operation of the image processing device 10 related to this embodiment.

Figure 3 is a flowchart showing an example of the operation of the image processing device 10 of this embodiment, and is a diagram for explaining the image processing method performed by the image processing device 10 of this embodiment.

The depth estimation unit 12 estimates the depth for each pixel constituting the image of the object that is input via the image input unit 11 (step S11). As described above, the depth estimation unit 12 estimates the depth for each pixel using a depth estimation model such as Midas.

The inspection area determination unit 13 (threshold determination unit 131) determines the depth of the area where the ratio of the size of the object to the size of the deterioration to be inspected is equal to or greater than a predetermined value as the depth threshold α. Then, the inspection area determination unit 13 (determination unit 132) determines the area closer to the depth threshold α as the inspection area based on the depth of each pixel estimated by the depth estimation unit 12 (step S12).

Figure 4 is a diagram for explaining the operation of determining the inspection area by the inspection area determination unit 13 (judgment unit 132).

The determination unit 132 acquires the depth p _i,j of pixel i,j estimated by the depth estimation unit 12 (step S121).

Next, the determination unit 132 determines whether the depth p _i,j is equal to or greater than a depth threshold α (p _i,j ≧α) (step S122).

If it is determined that p _i,j ≧α is satisfied (pixel i,j is in a region closer than the depth threshold α) (step S122: Yes), the determination unit 132 sets the determination result for pixel i,j to R _i,j =1 (step S123).

If it is determined that p _i,j ≧α is not satisfied (step S122: No), the determining unit 132 sets the determination result for pixel i,j to R _i,j =0 (step S124).

When the above-described process is completed for all pixels, the determination unit 132 determines the region consisting of pixels i and j for which the determination result R _i,j = 1 as the inspection region (step S125). By the process described with reference to Fig. 4, the inspection region determination unit 13 (determination unit 132) does not determine regions farther than the depth threshold value α as the inspection region, but can determine only regions closer to the depth threshold value α as the inspection region.

Referring again to Figure 3, the result output unit 14 outputs the result of the inspection area determination by the inspection area determination unit 13 (step S13).

Next, we will explain how the threshold determination unit 131 determines the depth threshold α.

In this embodiment, the number of pixels N of an object on an image required to inspect the object is set in advance, and the threshold determination unit 131 determines the depth threshold α based on that number of pixels N. Specifically, the threshold determination unit 131 determines the depth threshold α based on the size of the object in the image and the distribution of the depth of the area of the object having that size (hereinafter referred to as the "object size-depth distribution").

Figure 5 is a diagram explaining how to determine the number of pixels N of an object required for inspection.

As shown in Figure 5, the size of the object to be inspected is defined as x. x is the size in the direction parallel to the short side of the object to be inspected. Also, as shown in Figure 5, the size of the smallest deterioration that we want to detect through inspection is defined as y. The size of the deterioration y is the size in the same direction as the size x of the object.

If the number of pixels N of an object is x/y or greater (N≧x/y), then if there is degradation of size y, that degradation will span more than one pixel, making it possible to detect the degradation. For example, suppose the object's size x = 35 cm and the degradation size y = 7 cm. In this case, since 35/7 = 5, if the object's pixel count N is 5 or more pixels in the same direction as the object's size x was determined, the degradation to be inspected will span more than one pixel, making it possible to detect the degradation. On the other hand, if the object's pixel count N is less than 5 pixels, the degradation to be inspected will contain less information than one pixel, making it difficult to detect the degradation. Thus, if the object's pixel count (the size of the object on the image) is N or greater, the desired degradation can be detected through inspection. Therefore, it is clear that the inspection area can be determined to be the area on the image where the object's pixel count is N or greater.

Next, we will explain how to extract areas where N≧x/y (areas of an object where the ratio of the size x of the object to the size y of the deterioration to be inspected is equal to or greater than a predetermined value). In the following, we will explain the example of an object being a guardrail.

Figure 6A is an image of a guardrail captured by an in-vehicle camera. Figure 6B is an annotation image of the image shown in Figure 6A, in which the pixel area of the guardrail is assigned a "1" and pixels in other areas are assigned a "0". Figure 6C is an image showing the depth estimation results for only the pixel area of the guardrail using a depth estimation model. In Figures 6A to 6C, the horizontal direction on the paper is referred to as the x-direction, and the vertical direction on the paper is referred to as the y-direction. Note that an annotation image such as that shown in Figure 6B can be created, for example, by performing object recognition processing on an image input via the image input unit 11.

The guardrail is composed of a beam with a wavy cross-section and supports that support the beam. The size of the guardrail is defined by the number of pixels in the y-direction in the pixel area of the guardrail in the image. The threshold determination unit 131 defines the coordinates of the upper left pixel in the annotation image shown in Figure 6B as (1, 1), counts the number of pixels with a 1 input in the y-direction for each x-coordinate, and determines the counted number as the guardrail width at that x-coordinate (hereinafter referred to as the "GR width"). For example, in Figure 6B, the dashed line location has an x-coordinate of 100 and a GR width of 48. As can be seen from Figure 6A, the GR width decreases as the x-coordinate increases.

Next, the threshold determination unit 131 calculates the depth of the region of the object having the GR width determined based on the annotation image. Specifically, the threshold determination unit 131 defines the coordinates of the upper left pixel in the image shown in Figure 6C as (1, 1) and sums the depth values of each pixel in the pixel region of the guardrail in the y direction. The threshold determination unit 131 then divides the sum of the depth values for each x coordinate by the GR width at that x coordinate to calculate the depth of the region of the object having the GR width at that x coordinate. For example, in Figure 6C, the dashed line location has an x coordinate of 100, and the depth of the region of the object having the GR width is 183.

Figure 7 is a plot of the relationship between GR width and the depth of the area of the object (guardrail) having that GR width (object size-depth distribution), obtained from 4,400 sample images. For example, if the area of the guardrail that can be inspected is defined as the area where the GR width is N = 20 pixels or more, by setting a depth threshold α, the object size-depth distribution can be classified into four areas, as shown in Figure 7.

The upper right region is a region where the GR width is 20 pixels or more and the depth is greater than or equal to the depth threshold α (the region where the ratio (x/y) of the object size x to the degradation size y is greater than or equal to a predetermined value (N=20) and the depth is greater than or equal to the depth threshold α). This region is defined as TP. TP is a region where an image region with a GR width of 20 pixels or more is correctly extracted as a valid range for inspection.

The area in the lower right is an area where the GR width is less than 20 pixels and the depth is greater than or equal to the depth threshold α (an area where the ratio (x/y) of the object size x to the degradation size y is less than a predetermined value (N=20) and the depth is greater than or equal to the depth threshold α). This area is defined as FP. FP is an area where an image area with a GR width of less than 20 pixels is mistakenly extracted as a valid area for inspection.

The upper left area is an area where the GR width is 20 pixels or more and the depth is less than the depth threshold α (an area where the ratio (x/y) of the size x of the object to the size y of the deterioration to be inspected is greater than or equal to a predetermined value (N=20) and the depth is less than the depth threshold α). This area is defined as FN. FN is an area where image areas with a GR width of 20 pixels or more are mistakenly excluded as areas that are not valid for inspection.

The area in the lower left is an area where the GR width is less than 20 pixels and the depth is less than the depth threshold α (an area where the ratio (x/y) of the object size x to the degradation size y is less than a predetermined value (N=20) and the depth is less than the depth threshold α). This area is defined as TN. TN is an area where image areas with a GR width less than 20 pixels are correctly excluded as areas that are not valid for inspection.

The threshold determination unit 131 calculates the index (Accuracy) based on the above classification, for example, using formula (1). Note that TP, FP, FN, and TN in formula (1) are the numbers of data classified into the regions TP, FP, FN, and TN, respectively.
Accuracy=(TP+TN)/(TP+FP+FN+TN) Formula (1)

The threshold determination unit 131 determines the depth threshold α based on the index (Accuracy) calculated according to the above-described formula (1). Specifically, the threshold determination unit 131 determines the depth at which the index (Accuracy) calculated based on formula (1) is greatest as the depth threshold α.

Figure 8 shows an example of the index (Accuracy) calculated according to formula (1) based on the distribution shown in Figure 7. As shown in Figure 8, when the depth threshold α = 73, the index (Accuracy) is highest (classification accuracy is highest). Therefore, the threshold determination unit 131 determines the depth threshold α to be 73.

The index used to determine the depth threshold α is not limited to the above-described index (Accuracy). The threshold determination unit 131 may determine the depth threshold α using, for example, an index (Recall) calculated according to the following formula (2):
Recall=TP/(TP+FN) Formula (2)

When using the index (Recall) calculated according to equation (2), areas that may be suitable for the inspection area may be excluded from the decision area. In other words, by using the index (Recall), it is possible to determine a depth threshold α that reduces the likelihood of inappropriate areas being included in the inspection area.

Furthermore, the threshold value determination unit 131 may determine the depth threshold value α using an index (Specificity) calculated according to the following equation (3), for example.
Specificity=TN/(FP+TN) Formula (3)

When using the index (Specificity) calculated according to formula (3), areas that may not be included in the inspection area may be extracted as inspection areas. In other words, by using the index (Specificity), it is possible to determine a depth threshold α that will include a wider area in the inspection area.

The threshold determination unit 131 creates a size-depth distribution of the object as shown in Figure 7 based on the GR width calculated for each x-coordinate of the input image and the depth of the area having that GR width, and determines the depth threshold α using the above-mentioned indicators.

In this embodiment, the object is described as a guardrail, but the object is not limited to a guardrail. The threshold determination unit 131 sets TP, FP, FN, and TN for any object based on the size-depth distribution of the object, and determines the depth threshold α using the indices described above.

In addition, in the above example, the threshold determination unit 131 sets TP, FP, FN, and TN based on the size-depth distribution of the object, and determines the depth threshold α using the above-mentioned index, but this is not limited to this. For example, the threshold determination unit 131 may set an approximation line (approximation line or approximation curve) for the size-depth distribution of the object, and determine the depth threshold α.

Specifically, the threshold determination unit 131 sets an approximation line (an approximation straight line in FIG. 9) for the size-depth distribution of the object, as shown in FIG. 9. The threshold determination unit 131 then determines the depth corresponding to the object size on the approximation line where the ratio (x/y) of the object size x to the degradation size y is equal to or greater than a predetermined value (N=20) as the depth threshold α. In the example shown in FIG. 9, if N=20, then 47, which is the depth on the approximation line corresponding to a GR width of 20 pixels, is determined as the depth threshold α.

The threshold determination unit 131 may also set multiple approximation lines for the size-depth distribution of an object. For example, as shown in FIG. 10, the threshold determination unit 131 may set approximation lines for size-depth distributions of an object with different slopes. In the example shown in FIG. 10, if N = 20 in image 1, the depth corresponding to a GR width of 20 pixels is 80. Also, if N = 20 in image 2, the depth corresponding to a GR width of 20 pixels is 70.

In this case, the threshold determination unit 131 has a classification model 1311, as shown in FIG. 11.

Classification model 1311 is a model that classifies an input image according to the size-depth distribution of objects in the image. Classification model 1311 classifies an input image according to, for example, the gradient of the size-depth distribution of objects in the image. Classification model 1311 is constructed using a statistical discrimination model such as deep learning.

The threshold determination unit 131 sets an approximation line for the size-depth distribution of objects in the image for each classification of the input image by the classification model 1311. The threshold determination unit 131 then determines the depth threshold α to be the depth of the object region on the approximation line set for the image classified by the classification model 1311, where the ratio (x/y) of the object size x to the degradation size y is equal to or greater than a predetermined value. For example, if N = 20 and the input image is classified as image 1 by the classification model 1311, the threshold determination unit 131 sets the depth threshold α to 80, which is the depth corresponding to the region with a GR width of 20 pixels on the approximation line set for image 1. Also, for example, if N = 20 and the input image is classified as image 2 by the classification model 1311, the threshold determination unit 131 sets the depth threshold α to 70, which is the depth of the region with a GR width of 20 pixels on the approximation line set for image 2. By performing such processing, the inspection area can be determined with higher accuracy than when a single approximation formula is used.

Next, we will explain the hardware configuration of the image processing device 10 related to this embodiment.

Figure 12 is a diagram showing an example of the hardware configuration of an image processing device 10 according to this embodiment. Figure 12 shows an example of the hardware configuration of the image processing device 10 when the image processing device 10 is configured by a computer capable of executing program instructions. Here, the computer may be a general-purpose computer, a dedicated computer, a workstation, a PC (Personal Computer), an electronic notepad, etc. The program instructions may be program code, code segments, etc. for performing the required tasks.

As shown in FIG. 12, the image processing device 10 has a processor 21, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, a storage 24, an input unit 25, a display unit 26, and a communication interface (I/F) 27. Each component is connected to each other via a bus 29 so that they can communicate with each other. The processor 21 is specifically a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), SoC (System on a Chip), etc., and may be composed of multiple processors of the same or different types.

The processor 21 is a control unit that controls each component and performs various calculation processes. That is, the processor 21 reads a program from the ROM 22 or the storage 24 and executes the program using the RAM 23 as a working area. The processor 21 controls each component and performs various calculation processes in accordance with the program stored in the ROM 22 or the storage 24. In this embodiment, the ROM 22 or the storage 24 stores a program for operating a computer as the image processing device 10 according to the present disclosure. The program is read and executed by the processor 21 to realize each component of the image processing device 10, namely, the image input unit 11, the depth estimation unit 12, the inspection area determination unit 13, and the result output unit 14.

The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be provided in a form that can be downloaded from an external device via a network.

ROM 22 stores various programs and data. RAM 23 temporarily stores programs or data as a working area. Storage 24 is composed of an HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs and data including the operating system. ROM 22 or storage 24 stores, for example, frames output as cut-out frames.

The input unit 25 includes a pointing device such as a mouse and a keyboard, and is used to make various inputs.

The display unit 26 is, for example, a liquid crystal display, and displays various information. The display unit 26 may also function as the input unit 25 by adopting a touch panel system.

The communication interface 27 is an interface for communicating with other devices (e.g., a camera that photographs an object), and is, for example, an interface for a LAN.

A computer can be suitably used to function as each part of the image processing device 10 described above. Such a computer can be realized by storing a program describing the processing content that realizes the functions of each part of the image processing device 10 in the computer's memory, and having the computer's processor read and execute this program. In other words, the program can cause the computer to function as the image processing device 10 described above. The program can also be stored on a non-transitory storage medium. The program can also be provided via a network.

As such, the image processing device 10 according to this embodiment includes a depth estimation unit 12, an inspection area determination unit 13, and a result output unit 14 that outputs the determination results of the inspection area determination unit 13. The depth estimation unit 12 estimates the depth for each pixel constituting the image. The inspection area determination unit 13 determines the depth of the object area in the image where the ratio of the object size x to the degradation size y to be inspected is equal to or greater than a predetermined value as the depth threshold α. Then, based on the depth for each pixel estimated by the depth estimation unit 12, the inspection area determination unit 13 determines the area closer to the depth threshold α as the inspection area.

The depth of the object area where the ratio of the object size x to the deterioration size y is equal to or greater than a predetermined value is set as the depth threshold α, and the area closer to the depth threshold α is determined as the inspection area. This ensures that the size of the deterioration in the determined area is equal to or greater than the deterioration size y to be detected. Therefore, in an image of an object such as infrastructure equipment, the area that can be inspected for deterioration can be determined as the inspection area.

The following notes are further disclosed regarding the above embodiments.

[Additional note 1]
An image processing device that determines an inspection area in an image of an object to inspect for the presence or absence of deterioration of the object,
Memory and
a control unit connected to the memory;
Equipped with
The control unit
Estimating a depth, which is a distance in a depth direction in the image, for each pixel constituting the image;
A depth threshold is determined as a depth of a region of the object in the image where a ratio of a size of the object to a size of the deterioration to be inspected is equal to or greater than a predetermined value, and an area closer to the depth threshold is determined as the inspection area based on the estimated depth of each pixel;
an image processing device that outputs the result of determining the inspection area;

[Additional note 2]
In the image processing device according to claim 1,
The control unit determines the depth threshold based on a distribution of the size of the object in the image and the depth of a region of the object having that size.

[Additional note 3]
In the image processing device according to supplementary item 2,
In the distribution, let TP be the number of data in the region where the ratio is equal to or greater than the predetermined value and the depth is equal to or greater than the depth threshold, let FP be the number of data in the region where the ratio is less than the predetermined value and the depth is equal to or greater than the depth threshold, let FN be the number of data in the region where the ratio is equal to or greater than the predetermined value and the depth is less than the depth threshold, and let TN be the number of data in the region where the ratio is less than the predetermined value and the depth is less than the depth threshold.
The control unit determines the depth threshold based on an index obtained from (TP+TN)/(TP+FP+FN+TN), TP/(TP+FN), or TN/(FP+TN).

[Additional note 4]
In the image processing device according to supplementary item 2,
The control unit sets an approximation line for the distribution and determines the depth threshold as the depth corresponding to the size of the object on the approximation line at which the ratio is equal to or greater than the predetermined value.

[Additional note 5]
In the image processing device according to supplementary item 2,
The control unit is provided with a classification model that classifies an input image according to the distribution, sets an approximation line for the distribution for each classification of the input image by the classification model, and determines the depth threshold to be the depth corresponding to the size of the object at which the ratio is greater than or equal to the predetermined value on the approximation line set for the image classified by the classification model.

[Additional note 6]
1. An image processing method for an image processing device that determines an inspection area in an image of an object, in which the presence or absence of deterioration of the object is inspected, comprising:
Estimating a depth, which is a distance in a depth direction in the image, for each pixel constituting the image;
A depth threshold is determined as a depth of a region of the object in the image where a ratio of a size of the object to a size of the deterioration to be inspected is equal to or greater than a predetermined value, and an area closer to the depth threshold is determined as the inspection area based on the estimated depth of each pixel;
and outputting the result of determining the inspection area.

[Additional note 7]
A non-transitory storage medium storing a program executable by a computer, the non-transitory storage medium storing the program causing the computer to operate as the image processing device described in any one of appendix items 1 to 5.

The above-described embodiments have been described as representative examples, but it will be apparent to those skilled in the art that numerous modifications and substitutions are possible within the spirit and scope of the present disclosure. Therefore, the present invention should not be construed as being limited to the above-described embodiments, and various modifications and alterations are possible without departing from the scope of the claims. For example, multiple component blocks shown in the configuration diagrams of the embodiments may be combined into one, or one component block may be divided.

REFERENCE SIGNS LIST 10 Image processing device 11 Image input unit 12 Depth estimation unit 13 Inspection area determination unit 14 Result output unit 131 Threshold determination unit 132 Determination unit 1311 Classification model 21 Processor 22 ROM
23 RAM
24 Storage 25 Input unit 26 Display unit 27 Communication I/F
29 Bus

Claims (7)

An image processing device that determines an inspection area in an image of an object to inspect for the presence or absence of deterioration of the object,
a depth estimation unit that estimates a depth, which is a distance in a depth direction in the image, for each pixel that constitutes the image;
an inspection area determination unit that determines a depth of an area of the object in the image where the ratio of the size of the object to the size of the deterioration to be inspected is equal to or greater than a predetermined value as a depth threshold, and determines an area closer to the depth threshold as the inspection area based on the depth of each pixel estimated by the depth estimation unit;
a result output unit that outputs a result of the inspection area determination by the inspection area determination unit. 2. The image processing device according to claim 1,
The inspection area determination unit determines the depth threshold based on a distribution of the size of the object in the image and the depth of the area of the object having that size. 3. The image processing device according to claim 2,
In the distribution, a region where the ratio is equal to or greater than the predetermined value and the depth is equal to or greater than the depth threshold is defined as TP, a region where the ratio is less than the predetermined value and the depth is equal to or greater than the depth threshold is defined as FP, a region where the ratio is equal to or greater than the predetermined value and the depth is less than the depth threshold is defined as FN, and a region where the ratio is less than the predetermined value and the depth is less than the depth threshold is defined as TN.
An image processing device wherein the inspection area determination unit determines the depth threshold based on an index obtained from (TP+TN)/(TP+FP+FN+TN), TP/(TP+FN), or TN/(FP+TN). 3. The image processing device according to claim 2,
The inspection area determination unit sets an approximation line for the distribution and determines the depth threshold as the depth on the approximation line corresponding to the size of the object at which the ratio is greater than or equal to the predetermined value. 3. The image processing device according to claim 2,
The inspection area determination unit is provided with a classification model that classifies an input image according to the distribution, sets an approximation line to the distribution for each classification of the input image by the classification model, and determines the depth threshold to be the depth corresponding to the size of the object at which the ratio is greater than or equal to the predetermined value on the approximation line set for the image classified by the classification model. 1. An image processing method for an image processing device that determines an inspection area in an image of an object, in which the presence or absence of deterioration of the object is inspected, comprising:
a step of estimating a depth, which is a distance in a depth direction in the image, for each pixel constituting the image;
determining a depth threshold as a depth of a region of the object in the image where the ratio of the size of the object to the size of the deterioration to be inspected is equal to or greater than a predetermined value, and determining an area closer to the depth threshold as the inspection region based on the estimated depth of each pixel;
and outputting the result of determining the inspection area. A program that causes a computer to operate as the image processing device described in claim 1.