JP7505866B2 - Inspection support method, inspection support system, and inspection support program - Google Patents

Description

This disclosure relates to an inspection support method, an inspection support system, and an inspection support program.

When an inspector visually inspects an object (including finished products, machine parts, and intermediate products of the manufacturing process), determines the dimensions of defects, and records the information in a report, it is time-consuming and accuracy varies depending on the ability of the inspector. In addition, when performing 3D measurement with a laser, alignment takes time and the equipment is expensive, making it costly.

For this reason, in recent years, inspection support methods have been proposed that inspect the presence or absence of defects in an inspection object based on an image of the inspection object captured by a camera or IR camera. For example, Patent Document 1 discloses a method of capturing an image of an inspection object with an IR camera to obtain a two-dimensional image (actual IR image), and measuring three-dimensional distances on the two-dimensional image.

In this method, a simulated image is generated using a 3D CAD model, the simulated image is rearranged into a 2D image, a 3D distance scale matrix is generated, and the 3D distance between pixels in the 2D image is obtained. In rearranging the simulated image, a cost function based on the difference in grayscale values is evaluated to optimize the simulated image for the 2D image, and the relative positional relationship between the two is adjusted. In this method, the object to be inspected is temporarily heated, and an actual IR image is obtained during the heat dissipation process, thereby detecting the position and dimensions of three-dimensional defects, including the depth direction.

JP 2010-122218 A

The method described in Patent Document 1 requires the object to be inspected to be temporarily heated. In this case, it cannot be applied to objects for which heating is undesirable. Also, setting up the object to be inspected can be time-consuming. Furthermore, methods that use grayscale values (or binary values) can make it difficult to identify the object to be inspected.

On the other hand, in a two-dimensional real image taken by a normal visible light camera, rather than an IR real image, it is possible to evaluate the RGB color tones, which may make it easier to identify the object being inspected. Also, imaging the object being inspected with a normal visible light camera is advantageous in that it is not necessary to heat the object being inspected. However, in a two-dimensional real image taken by a visible light camera, the problem arises of how to derive the three-dimensional position and dimensions of defects in the object being inspected.

In view of the above, at least one embodiment of the present invention aims to provide an inspection support method capable of deriving the three-dimensional position and dimensions of a defect in an object to be inspected.

An inspection assistance method according to at least one embodiment of the present invention includes:
An image acquisition step of acquiring a two-dimensional actual image of an inspection object captured by a visible light camera ;
A shape identification step of identifying a shape of the inspection object included in the two-dimensional actual image;
a detection step of detecting a defect of the inspection object included in the two-dimensional actual image;
an image extraction step of extracting a required two-dimensional simulated image corresponding to the two-dimensional actual image from a previously acquired three-dimensional CAD model of the inspection object based on the identification result of the shape identification step;
a depiction step of adjusting the two-dimensional actual image including a defect image showing the defect detected in the detection step so as to match the two-dimensional simulated image, and depicting the adjusted defect image on the three-dimensional CAD model;
Equipped with.

According to this method, the defect image is depicted on a 3D CAD model of the object to be inspected, making it possible to derive the 3D position and dimensions of the defect in the object to be inspected from the dimensional data of the 3D CAD model.

In some embodiments, in the depiction step, the position and dimensions of the depicted defect image are adjusted by affine transforming the two-dimensional actual image including the defect image based on the result of comparing the two-dimensional actual image with the two-dimensional simulated image.

This method allows the position and dimensions of the defect image to be adjusted, making it possible to depict a more accurate defect image on the 3D CAD model.

In some embodiments, the shape identification step identifies the shape of the object to be inspected contained in the two-dimensional real image by detecting a plurality of reference portions of the object to be inspected in the two-dimensional real image.

In this method, a characteristic part of the object to be inspected (e.g. a corner) is set as a reference part, and the shape of the object to be inspected is identified by detecting this reference part from a two-dimensional actual image. In this case, it is possible to simplify the process and improve the accuracy of identification compared to identifying the shape of the object to be inspected by detecting a line indicating the outline of the object to be inspected.

In some embodiments, the image extraction step extracts the two-dimensional simulated image from the three-dimensional CAD model of the object to be inspected, the two-dimensional simulated image including portions corresponding to the multiple reference portions of the object to be inspected detected from the two-dimensional actual image.

In this method, characteristic parts of the object to be inspected (e.g., corners) are set as reference parts, and a two-dimensional simulated image is extracted from the three-dimensional CAD model based on multiple reference parts of the object to be inspected detected from the two-dimensional actual image. In this case, it is possible to simplify the process and improve the accuracy of extraction compared to when a two-dimensional simulated image is extracted based on lines showing the contour of the object to be inspected.

In some embodiments, the depiction step involves adjusting the position and dimensions of the defect image by affine transforming the two-dimensional actual image including the defect image based on a result of comparing a plurality of reference portions of the inspection object detected from the two-dimensional actual image with portions corresponding to the plurality of reference portions included in the two-dimensional simulated image, and depicting the defect image on the three-dimensional CAD model.

In this method, multiple reference parts of the inspection object detected from the 2D actual image are compared with parts corresponding to multiple reference parts contained in the 2D simulated image, and the position and size of the defect image are adjusted based on the differences in their positional relationship, shape, size, etc. In this case, it is possible to simplify the process and improve the accuracy of the adjustment compared to the case where the lines showing the contours of the inspection object are compared and the position and size of the defect image are adjusted.

In some embodiments, the shape identification step and the image extraction step are performed using a trained model that has been machine-learned to understand the relationship between the two-dimensional actual image of each of a plurality of objects, including the object to be inspected, and the three-dimensional CAD model or the two-dimensional simulated image of each of the plurality of objects.

This method makes it possible to more accurately identify the shape of the object being inspected and extract a 2D simulated image that corresponds to the actual 2D image.

In some embodiments, the detection step is performed using a trained model that has been machine-learned to understand the relationship between the two-dimensional actual images of each of a plurality of objects, including the object to be inspected, and images of defects that may occur in the plurality of objects.

This method makes it possible to more accurately detect defects in the object being inspected.

In some embodiments, the inspection support method further includes a report creation step of deriving the three-dimensional position and dimensions of the defect in the inspection object from the dimensional data of the three-dimensional CAD model, and creating a report including the results of deriving the position and dimensions.

According to this method, a report is created that includes the three-dimensional position and dimensions of the defect, reducing the workload of creating the report. In addition, by collecting information on the position and dimensions of defects for the same three-dimensional CAD model for multiple cases, it is also possible to obtain statistics (for example, if the defect is a dent, the locations that are likely to become dented can be identified from the statistical data).

An inspection support system according to at least one embodiment of the present invention includes:
an image acquisition unit that acquires a two-dimensional actual image of an inspection object captured by a visible light camera ;
A shape identification unit that identifies a shape of the inspection object included in the two-dimensional actual image;
a detection unit that detects defects of the inspection object included in the two-dimensional actual image;
A storage unit that stores a three-dimensional CAD model of the inspection object;
an image extraction unit that extracts a required two-dimensional simulated image corresponding to the two-dimensional actual image from the three-dimensional CAD model stored in the storage unit based on a classification result by the shape classification unit;
a depiction unit that adjusts the two-dimensional actual image including a defect image showing the defect detected by the detection unit so as to conform to the two-dimensional simulated image, and depicts the adjusted defect image on the three-dimensional CAD model;
Equipped with.

With this configuration, the defect image is depicted on a 3D CAD model of the object being inspected, making it possible to derive the 3D position and dimensions of the defect in the object being inspected from the dimensional data of the 3D CAD model.

An inspection assistance program according to at least one embodiment of the present invention includes:
On the computer,
an image acquisition step for acquiring a two-dimensional actual image of an inspection object captured by a visible light camera ;
a shape identification step of identifying a shape of the inspection object included in the two-dimensional actual image;
a detection step of detecting defects of the inspection object contained in the two-dimensional actual image;
a storing step of storing a three-dimensional CAD model of the inspection object;
an image extraction step of extracting, from the stored 3D CAD model, a required 2D simulated image corresponding to the 2D actual image based on a result of the shape identification step;
a depiction step of adjusting the two-dimensional actual image, including a defect image showing the defect detected in the detection step, so as to match the two-dimensional simulated image, and depicting the adjusted defect image on the three-dimensional CAD model;
Execute the command.

This program renders defect images on a 3D CAD model of the object being inspected, making it possible to derive the 3D position and dimensions of defects in the object being inspected from the dimensional data of the 3D CAD model.

At least one embodiment of the present invention provides an inspection support method capable of detecting the three-dimensional position and dimensions of defects in an object to be inspected.

1 is a block diagram illustrating a schematic configuration of an inspection support system according to an embodiment. 1 is a conceptual diagram for explaining a method of detecting a reference portion executed by an inspection support system according to an embodiment; 10 is a conceptual diagram for explaining a line detection method executed by an inspection support system according to a comparative example. FIG. 1 is a schematic diagram showing an example of a two-dimensional actual image acquired by an inspection support system according to an embodiment; 1 is a schematic diagram showing an example of a two-dimensional simulated image extracted by an inspection support system according to an embodiment; FIG. 2 is a schematic diagram showing an example of a three-dimensional CAD model in which a defect image is depicted. 4 is a flowchart illustrating an example of a process executed by the inspection support system according to an embodiment.

Hereinafter, some embodiments of the present invention will be described with reference to the accompanying drawings. However, the dimensions, materials, shapes, relative arrangements, etc. of components described as the embodiments or shown in the drawings are merely illustrative examples and are not intended to limit the scope of the present invention.
For example, expressions indicating that things are in an equal state, such as "identical,""equal," and "homogeneous," not only indicate a state of strict equality, but also indicate a state in which there is a tolerance or a difference to the extent that the same function is obtained.
On the other hand, the expressions "comprise,""include,""have,""includes," or "have" of one element are not exclusive expressions excluding the presence of other elements.

(Configuration of inspection support system)
The configuration of the inspection support system 1 according to an embodiment will be described below. Fig. 1 is a block diagram showing a schematic configuration of the inspection support system 1 according to an embodiment.

In the following embodiment, an inspection support system 1 configured as a single device (computer) is described. The inspection support system 1 is configured to acquire a two-dimensional actual image, which is an image of an inspection target 3 captured by a visible light camera 2.

The inspection support system 1 is not limited to this configuration, and may be composed of multiple devices (computers). For example, one or more communication terminals may communicate with a server device via a WAN (World Area Network) or a LAN (Local Area Network), and the functions of the inspection support system 1 may be realized through the cooperative relationship between them. The inspection support system 1 may also be configured to include a camera 2.

The inspection support system 1 is a system for supporting inspection work, such as detecting defects in an inspection object 3 and creating a report on the defects. The user of the inspection support system 1 may be an inspector who performs the inspection work, or a user who uses the inspection object 3. The defects detected by the inspection support system 1 are defects that can be determined from the appearance, and the types include, for example, cracks, dents, scratches, poor coating, oxidation, thinning, dirt, etc.

The inspection object 3 may be the entire product, or may be a part that constitutes the product (e.g., a machine part). The inspection object 3 may be a new product, a repaired part, or an existing facility. The inspection object 3 may be, for example, a gas turbine blade, a stator blade, a ring segment, a combustor, etc.

The configuration of the inspection support system 1 will be described in detail below. As shown in FIG. 1, the inspection support system 1 includes a communication unit 110 that communicates with other devices, a storage unit 120 that stores various data, an output unit 130 that outputs various information, an input unit 140 that accepts user input, and a control unit 150 that controls the entire device. These components are interconnected by a bus line 160.

The communication unit 110 is a communication interface equipped with a NIC (Network Interface Card controller) for wired or wireless communication. The communication unit 110 communicates with other devices (e.g., the camera 2 or a device equipped with the camera 2) via a network (not shown). Note that if the inspection support system 1 is configured to include the camera 2, the communication unit 110 may be omitted.

The storage unit 120 is composed of a RAM (Random Access Memory), a ROM (Read Only Memory), etc. The storage unit 120 stores programs for executing various control processes (e.g., an inspection support program, a trained model described below), and various data.

The various data include 3D CAD models of multiple objects. The multiple objects include inspection object 3. The 3D CAD model is 3D CAD data that shows the object as a mesh image in a 3D virtual space of any scale. The mesh image can be rotated, enlarged, and reduced, and the 3D CAD model is configured so that a 2D simulated image can be extracted from any viewpoint.

The storage unit 120 may also store acquired 2D actual images, extracted 2D simulated images, created reports, etc. The storage unit 120 may be configured with one storage device, may be configured with multiple storage devices, or may be an external storage device.

The output unit 130 is composed of an output device such as a display device, a speaker device, etc. The output unit 130 is an output interface for presenting various information to the user.

The input unit 140 is composed of input devices such as operation buttons, a keyboard, a pointing device, and a microphone. The input unit 140 is an input interface used by the user to input instructions.

The control unit 150 is composed of processors such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The control unit 150 controls the operation of the entire system by executing programs stored in the storage unit 120.

The functional configuration of the control unit 150 will be described below. The control unit 150 functions as an image acquisition unit 151, a shape identification unit 152, a detection unit 153, an image extraction unit 154, a depiction unit 155, and a report creation unit 156.

The image acquisition unit 151 acquires a two-dimensional actual image of the inspection target object 3 captured by the camera 2. Specifically, the image acquisition unit 151 acquires the two-dimensional actual image by receiving the two-dimensional actual image from another device via the communication unit 110.

The shape identification unit 152 identifies the two-dimensional shape of the inspection object in the two-dimensional actual image acquired by the image acquisition unit 151. The shape identification unit 152 may be configured to identify the shape of the inspection object 3 included in the two-dimensional actual image by detecting multiple reference parts of the inspection object 3 in the two-dimensional actual image.

Here, the method for detecting the reference portion will be described in comparison with the method for detecting lines in a comparative example. FIG. 2 is a conceptual diagram for explaining the method for detecting the reference portion executed by the inspection support system 1 according to one embodiment. FIG. 3 is a conceptual diagram for explaining the method for detecting lines executed by the inspection support system (not shown) according to the comparative example.

2 and 3 show an example in which the inspection object 3 has a triangular shape when viewed from the front. First, the inspection support system 1 is configured to detect a characteristic part of the inspection object 3 as a reference part. In one embodiment, the characteristic part is a corner. Note that the characteristic part may be any part that is useful for identifying the shape, and may be, for example, a landmark mark, trademark, keyhole, button, etc.

In one embodiment, the method for detecting the reference part uses machine learning (e.g., SSD (Single Shot MultiBox Detector) technology). SSD technology is known as a method for detecting objects in an image using deep learning. Specifically, as shown in FIG. 2, the inspection support system 1 uses a default box B0 to search the two-dimensional real image P1 and detects the corners of the inspection target object 3 as the reference part. The shape of the reference part is identified based on the RGB value indicating the red, green, and blue components of each pixel and the color tone compared to other pixels.

As a result, default boxes B1, B2, and B3 corresponding to the three corners are detected as reference parts. The inspection support system 1 identifies the inspection object 3 as triangular based on the positional relationship of these reference parts. In this case, even if there are unclear parts on the sides of the triangle, the inspection support system 1 can identify that the inspection object 3 is triangular if the reference parts are detected. Furthermore, the inspection support system 1 can identify the length of the sides of the triangle based on the relative positional relationship of the multiple detected reference parts.

In the line detection method according to the comparative example, lines showing the contour of the inspection object 3 are detected to identify the shape of the inspection object 3. In this example, as shown enlarged in FIG. 3, lines showing the contour of the inspection object 3 are detected based on grayscale values indicating the brightness of each pixel. Note that in the comparative example, it is also possible to detect lines by binarizing the image by defining a reference value instead of using grayscale values.

However, when detecting lines showing the contour of the inspection object 3 in the two-dimensional actual image P1, as in the comparative example, it becomes necessary to define the lines, which may lead to problems such as complicated processing and reduced accuracy. In addition, brightness varies depending on the shooting conditions such as the time of day, location, and lighting when the image is taken, so errors in line detection occur depending on the shooting conditions. In order to eliminate such errors, a system must be built taking into account various conditions.

In contrast, according to the reference portion detection method of the above embodiment, the reference portion is detected based on RGB values and color shades, so detection errors due to fluctuations in brightness are smaller than in the comparative example. In addition, the shape of the reference portion is identified by deep learning machine learning, so it is possible to speed up processing. Since there is no need to define lines, the amount of learning required for machine learning can also be reduced.

The shape identification unit 152 identifies the shape of the inspection object 3 included in the two-dimensional actual image by the above-mentioned method for detecting the reference part. In this case, it is not necessary to define the lines, and the process can be simplified because only the parts are compared. The shape identification unit 152 may also identify the shape of the inspection object 3 included in the two-dimensional actual image by using a trained model that has machine-learned the relationship between each of the two-dimensional actual images of multiple objects including the inspection object 3 and each of the three-dimensional CAD models of the multiple objects or their two-dimensional simulated images.

The detection unit 153 detects defects in the inspection object 3 contained in the two-dimensional actual image. The detection unit 153 may detect defects in the inspection object 3 using a trained model that has been machine-learned to determine the relationship between the two-dimensional actual images of each of the multiple objects including the inspection object 3 and images of defects that may occur in the multiple objects. For example, the detection unit 153 may be configured to analyze the RGB values of each pixel in the two-dimensional actual image and determine defects by pattern classification of the contrast and color tone.

Based on the identification result by the shape identification unit 152, the image extraction unit 154 refers to the required 3D CAD model stored in the storage unit 120, and extracts a 2D simulated image corresponding to the 2D actual image from the 3D CAD model. The image extraction unit 154 may extract a 2D simulated image including a portion corresponding to a plurality of reference portions of the inspection object 3 detected from the 2D actual image from the 3D CAD model of the inspection object 3. The image extraction unit 154 may also extract the 2D simulated image from the 3D CAD model using a trained model that has machine-learned the relationship between each of the 2D actual images of the plurality of objects including the inspection object 3 and each of the 3D CAD models of the plurality of objects or their 2D simulated images.

The depiction unit 155 adjusts the defect image showing the defect detected by the detection unit 153 so that it fits the two-dimensional simulated image, and depicts the adjusted defect image on the three-dimensional CAD model. This adjustment may be performed before depiction or when depicting, or may be performed on the defect image that has already been depicted after depiction once. The three-dimensional CAD with the defect image depicted may be displayed on the display device of the output unit 130.

The depiction unit 155 adjusts the position and dimensions of the depicted defect image by affine transforming the defect image included in the two-dimensional real image based on the result of comparing the two-dimensional real image with the two-dimensional simulated image. The depiction unit 155 may adjust the position and dimensions of the defect image by affine transforming the defect image included in the two-dimensional real image based on the result of comparing the multiple reference parts of the inspection object 3 detected from the two-dimensional real image with the parts corresponding to the multiple reference parts included in the two-dimensional simulated image. The depiction unit 155 may adjust the position and dimensions of the depicted defect image by affine transforming the two-dimensional real image including the defect image based on the result of comparing the two-dimensional real image with the two-dimensional simulated image. Since it is possible that the dimensions of the defect part have changed from the normal state in the defect image alone, the normal part around the defect part may also be included in the affine transformation.

For example, the depiction unit 155 may compare the lengths of the sides determined from the positional relationships of multiple reference parts between the two-dimensional actual image and the two-dimensional simulated image, and enlarge or reduce the defect image based on the similarity ratio. The depiction unit 155 may compare the positional relationships of multiple reference parts between the two-dimensional actual image and the two-dimensional simulated image, and adjust the aspect ratio of the defect image, translate it, linearly transform it, etc.

The report creation unit 156 derives the three-dimensional position and dimensions of defects in the inspection object 3 from the dimensional data of the three-dimensional CAD model, and creates a report including the derived position and dimensions. The created report may be stored in the memory unit 120, or may be transmitted to another device (e.g., a server device that manages the report) via the communication unit 110.

(Relationship between 2D actual image, 2D simulated image, and 3D CAD model)
Here, specific examples will be described of a two-dimensional actual image of the inspection object 3 captured by the camera 2, a two-dimensional simulated image extracted from a three-dimensional CAD model by the image extraction unit 154 based on the two-dimensional actual image, and a three-dimensional CAD model in which the depiction unit 155 depicts a defect image.

Figure 4 is a schematic diagram showing an example of a two-dimensional actual image acquired by the inspection support system 1 according to one embodiment. Figure 5 is a schematic diagram showing an example of a two-dimensional simulated image extracted by the inspection support system 1 according to one embodiment. Figure 6 is a schematic diagram showing an example of a three-dimensional CAD model in which a defect image is depicted.

The two-dimensional simulated image shown in FIG. 5 is an image extracted based on the two-dimensional actual image shown in FIG. 4. FIG. 6 shows a three-dimensional CAD model depicting the defect image included in the two-dimensional actual image shown in FIG. 4.

In the examples shown in Figures 4 to 6, the inspection object 3 is a gas turbine vane. As shown in Figure 4, the defect is a dent located in area A1 at the center of the blade in a two-dimensional actual image taken from the ventral side of the gas turbine vane. The detection unit 153 detects the image of this area A1 as a defect image. When the image acquisition unit 151 acquires the two-dimensional actual image shown in Figure 4, the image extraction unit 154 extracts the two-dimensional simulated image shown in Figure 5 from the three-dimensional CAD model based on the identification result by the shape identification unit 152.

Comparing FIG. 4 and FIG. 5, there are some differences in viewpoint and scale. Therefore, the depiction unit 155 adjusts the defect image of area A1 in FIG. 4 by performing affine transformation or the like so that it fits the two-dimensional simulated image in FIG. 5. The depiction unit 155 then depicts the adjusted defect image (area A2) on the three-dimensional CAD model as shown in FIG. 5. Note that on the three-dimensional CAD model, the defect image is displayed at the scale of the three-dimensional CAD model. Area A2 showing the defect image may be colored to distinguish it from the original image of the three-dimensional CAD model.

(Processing flow)
The following describes a specific example of the processing executed by the inspection support system 1. Fig. 7 is a flowchart showing an example of the processing executed by the inspection support system 1 according to an embodiment.

As shown in FIG. 7, the image acquisition unit 151 of the inspection support system 1 acquires a two-dimensional actual image of the inspection object 3 captured by the camera 2 (step S1). The shape identification unit 152 of the inspection support system 1 detects a reference portion of the inspection object 3 in the two-dimensional actual image and identifies the shape of the inspection object 3 (step S2). The detection unit 153 of the inspection support system 1 detects defects in the inspection object 3 (step S3). The detection of defects may be performed by comparison with a three-dimensional CAD model or may be performed using a trained model.

The control unit 150 of the inspection support system 1 then determines whether or not there is a defect in the inspection target 3 (step S4). If it is determined that there is no defect in the inspection target 3 (step S4; No), steps S5 to S7 are skipped, and the report creation unit 156 of the inspection support system 1 creates a report indicating that there is no defect (step S8).

On the other hand, if it is determined that the inspection object 3 has a defect (step S4; Yes), the image extraction unit 154 of the inspection support system 1 extracts a two-dimensional simulated image including portions corresponding to the multiple reference portions of the inspection object 3 detected from the two-dimensional actual image from the three-dimensional CAD model (step S5). A trained model may be used for this extraction.

The depiction unit 155 of the inspection support system 1 adapts the two-dimensional actual image (i.e., the defect image) showing the defect detected in step S3 so that it fits the two-dimensional simulated image (step S6). At this time, the defect image is adjusted by affine transformation or the like. The depiction unit 155 of the inspection support system 1 depicts the adjusted defect image on a three-dimensional CAD model (step S7). The report creation unit 156 of the inspection support system 1 derives the three-dimensional position and dimensions of the defect in the inspection object 3 from the dimensional data of the three-dimensional CAD model, and creates a report including the derived results (step S8). Note that, as shown by the dashed lines in FIG. 7, steps S1 to S4 and S6 are processes using two-dimensional information, and steps S5 and S7 are processes using three-dimensional information. Step S8 creates a report according to the application site and the inspection object, for example, by using both three-dimensional information and two-dimensional information or one of them, or by writing only the text if there is no defect.

As described above, for example, as shown in Figures 1 to 7, an inspection support method according to one embodiment includes an image acquisition step of acquiring a two-dimensional actual image of an inspection object 3, a shape identification step of identifying the shape of the inspection object 3 contained in the two-dimensional actual image, a detection step of detecting defects in the inspection object 3 contained in the two-dimensional actual image, an image extraction step of extracting a required two-dimensional simulated image corresponding to the two-dimensional actual image from a previously acquired three-dimensional CAD model of the inspection object 3 based on the identification result of the shape identification step, and a depiction step of adjusting the two-dimensional actual image including the defect image showing the defect detected in the detection step so as to fit the two-dimensional simulated image, and depicting the adjusted defect image on the three-dimensional CAD model.

According to this method, the defect image is depicted on a 3D CAD model of the inspection object 3, so that it is possible to derive the 3D position and dimensions of the defect in the inspection object 3 from the dimensional data of the 3D CAD model.

In some embodiments, in the depiction step, the position and dimensions of the depicted defect image are adjusted by affine transforming the two-dimensional actual image including the defect image based on the results of comparing the two-dimensional actual image and the two-dimensional simulated image.

This method allows the position and dimensions of the defect image to be adjusted, making it possible to depict a more accurate defect image on the 3D CAD model.

In some embodiments, the shape identification step identifies the shape of the object 3 contained in the two-dimensional real image by detecting multiple reference portions of the object 3 in the two-dimensional real image.

In this method, a characteristic part (e.g., a corner) of the inspection object 3 is set as a reference part, and the shape of the inspection object 3 is identified by detecting the reference part from a two-dimensional actual image. In this case, it is possible to simplify the process and improve the accuracy of identification compared to identifying the shape of the inspection object 3 by detecting a line indicating the outline of the inspection object 3.

In some embodiments, the image extraction step extracts a two-dimensional simulated image from a three-dimensional CAD model of the object to be inspected 3, the two-dimensional simulated image including portions corresponding to the multiple reference portions of the object to be inspected 3 detected from the two-dimensional actual image.

In this method, characteristic parts (e.g., corners) of the inspection object 3 are set as reference parts, and a two-dimensional simulated image is extracted from the three-dimensional CAD model based on multiple reference parts of the inspection object 3 detected from the two-dimensional actual image. In this case, it is possible to simplify the process and improve the accuracy of extraction compared to the case where a two-dimensional simulated image is extracted based on the lines showing the contour of the inspection object 3.

In some embodiments, the depiction step involves adjusting the position and dimensions of the defect image by affine transforming the two-dimensional actual image including the defect image based on the results of comparing multiple reference portions of the inspection object 3 detected from the two-dimensional actual image with portions corresponding to the multiple reference portions included in the two-dimensional simulated image, and depicting the defect image on the three-dimensional CAD model.

In this method, multiple reference parts of the inspection object 3 detected from the 2D actual image are compared with parts corresponding to multiple reference parts included in the 2D simulated image, and the position and size of the defect image are adjusted based on the differences in their positional relationship, shape, size, etc. In this case, it is possible to simplify the process and improve the accuracy of the adjustment compared to the case where the lines showing the contours of the inspection object 3 are compared to adjust the position and size of the defect image.

In some embodiments, the shape identification step and the image extraction step are performed using a trained model that has been machine-learned to understand the relationship between the two-dimensional actual images of each of the multiple objects, including the inspection target 3, and the three-dimensional CAD models or two-dimensional simulated images of each of the multiple objects.

This method makes it possible to more accurately identify the shape of the inspection target object 3 and extract a two-dimensional simulated image that corresponds to the actual two-dimensional image.

In some embodiments, the detection step is performed using a trained model that has been machine-learned to understand the relationship between two-dimensional actual images of each of the multiple objects, including the inspection target 3, and images of defects that may occur in the multiple objects.

This method makes it possible to more accurately detect defects in the inspection target object 3.

In some embodiments, the inspection support method further includes a report creation step of deriving the three-dimensional position and dimensions of the defect in the inspection object 3 from the dimensional data of the three-dimensional CAD model, and creating a report including the results of deriving the position and dimensions.

According to this method, a report is created that includes the three-dimensional position and dimensions of the defect, reducing the workload of creating the report. In addition, by collecting information on the position and dimensions of defects for the same three-dimensional CAD model for multiple cases, it is also possible to obtain statistics (for example, if the defect is a dent, the locations that are likely to become dented can be identified from the statistical data).

As shown in FIG. 1, the inspection support system 1 according to at least one embodiment of the present invention includes an image acquisition unit 151 that acquires a two-dimensional actual image of an inspection object 3, a shape identification unit 152 that identifies the shape of the inspection object 3 contained in the two-dimensional actual image, a detection unit 153 that detects defects in the inspection object 3 contained in the two-dimensional actual image, a storage unit 120 that stores a three-dimensional CAD model of the inspection object 3, an image extraction unit 154 that extracts a two-dimensional simulated image corresponding to the two-dimensional actual image from the three-dimensional CAD model stored in the storage unit 120 based on the identification result by the shape identification unit 152, and a depiction unit 155 that adjusts the two-dimensional actual image including the defect image showing the defect detected by the detection unit 153 so as to fit the two-dimensional simulated image and depicts the adjusted defect image on the three-dimensional CAD model.

With this configuration, the defect image is depicted on a 3D CAD model of the inspection object 3, making it possible to derive the 3D position and dimensions of the defect in the inspection object 3 from the dimensional data of the 3D CAD model.

The present invention is not limited to the above-described embodiments, but also includes variations of the above-described embodiments and appropriate combinations of these embodiments.

(Modification)
For example, the order of processes executed by the inspection support system 1 is not limited to the example shown in Fig. 7. For example, in Fig. 7, step S5 of extracting a two-dimensional simulated image may be executed before step S3 of detecting a defect. A part of the processes shown in Fig. 7 may be performed by a user. For example, the report creation in step S8 may be performed by a user, and the inspection support system 1 may not be provided with the report creation unit 156.

The inspection assistance system 1 according to the present disclosure may achieve its functions not by using the above-mentioned devices, but by having a computer execute a program (inspection assistance program). The inspection assistance program for achieving the functions of the inspection assistance system 1 may be stored in a computer-readable recording medium such as a Universal Serial Bus (USB) memory, a Compact Disc-Read Only Memory (CD-ROM), a Digital Versatile Disc (DVD), or a Hard Disc Drive (HDD), or may be downloaded to a computer via a network.

REFERENCE SIGNS LIST 1 Inspection support system 2 Camera 3 Inspection target object 110 Communication unit 120 Storage unit 130 Output unit 140 Input unit 150 Control unit 151 Image acquisition unit 152 Shape identification unit 153 Detection unit 154 Image extraction unit 155 Description unit 156 Report creation unit 160 Bus line

Claims (10)

An image acquisition step of acquiring a two-dimensional actual image of an inspection object captured by a visible light camera ;
A shape identification step of identifying a shape of the inspection object included in the two-dimensional actual image;
a detection step of detecting a defect of the inspection object included in the two-dimensional actual image;
an image extraction step of extracting a required two-dimensional simulated image corresponding to the two-dimensional actual image from a previously acquired three-dimensional CAD model of the inspection object based on the identification result of the shape identification step;
a depiction step of adjusting the two-dimensional actual image including a defect image showing the defect detected in the detection step so as to match the two-dimensional simulated image, and depicting the adjusted defect image on the three-dimensional CAD model;
An inspection support method comprising: The inspection support method according to claim 1, wherein in the depiction step, the position and size of the defect image to be depicted are adjusted by affine transforming the two-dimensional actual image including the defect image based on a result of comparing the two-dimensional actual image and the two-dimensional simulated image. The inspection support method according to claim 1 or 2, wherein the shape identification step identifies the shape of the object to be inspected contained in the two-dimensional actual image by detecting a plurality of reference portions of the object to be inspected in the two-dimensional actual image. The inspection support method according to any one of claims 1 to 3, wherein the image extraction step extracts the two-dimensional simulated image including portions corresponding to a plurality of reference portions of the object to be inspected detected from the two-dimensional actual image from the three-dimensional CAD model of the object to be inspected. 5. The inspection support method according to any one of claims 1 to 4, wherein the depiction step comprises adjusting the position and dimensions of the defect image by affine transforming the two-dimensional actual image including the defect image based on a result of comparing a plurality of reference parts of the inspection object detected from the two-dimensional actual image with parts corresponding to the plurality of reference parts included in the two-dimensional simulated image, and depicting the defect image on the three-dimensional CAD model. The inspection support method according to any one of claims 1 to 5, wherein the shape identification step and the image extraction step are performed using a trained model that has been machine-learned to understand the relationship between the two-dimensional actual image of each of a plurality of objects including the inspection object and the three-dimensional CAD model or the two-dimensional simulated image of each of the plurality of objects. The inspection support method according to any one of claims 1 to 6, wherein the detection step is performed using a trained model that has been machine-learned to understand the relationship between the two-dimensional actual images of each of a plurality of objects including the inspection object and the defect images that may occur in the plurality of objects. The inspection support method according to any one of claims 1 to 7, further comprising a report creation step of deriving three-dimensional positions and dimensions of the defects in the inspection object from dimensional data of the three-dimensional CAD model, and creating a report including the derivation results of the positions and dimensions. an image acquisition unit that acquires a two-dimensional actual image of an inspection object captured by a visible light camera ;
A shape identification unit that identifies a shape of the inspection object included in the two-dimensional actual image;
a detection unit that detects defects of the inspection object included in the two-dimensional actual image;
A storage unit that stores a three-dimensional CAD model of the inspection object;
an image extraction unit that extracts a required two-dimensional simulated image corresponding to the two-dimensional actual image from the three-dimensional CAD model stored in the storage unit based on a classification result by the shape classification unit;
a depiction unit that adjusts the two-dimensional actual image including a defect image showing the defect detected by the detection unit so as to conform to the two-dimensional simulated image, and depicts the adjusted defect image on the three-dimensional CAD model;
An inspection support system equipped with On the computer,
an image acquisition step for acquiring a two-dimensional actual image of an inspection object captured by a visible light camera ;
a shape identification step of identifying a shape of the inspection object included in the two-dimensional actual image;
a detection step of detecting defects of the inspection object contained in the two-dimensional actual image;
a storing step of storing a three-dimensional CAD model of the inspection object;
an image extraction step of extracting, from the stored 3D CAD model, a required 2D simulated image corresponding to the 2D actual image based on a result of the shape identification step;
a depiction step of adjusting the two-dimensional actual image including a defect image showing the defect detected in the detection step so as to match the two-dimensional simulated image, and depicting the adjusted defect image on the three-dimensional CAD model;
An inspection support program to carry out the above.

Images