JP7722964B2 - Visual Inspection System - Google Patents

Description

The present invention relates to a visual inspection system for structural welds.

Until now, inspection of welds in large welded structures, such as railway vehicle bodies, has been difficult to automate because there are more than 1,000 welds to inspect, and multiple check items, such as bead shape and dimensions and weld defects, have been carried out visually.
For this reason, it has been considered essential to provide training to visual inspectors of welds so that they can perform accurate inspections. However, due to the recent social issues of a declining birthrate and aging population, as well as the difficulty in securing workers, there is a demand for an automatic inspection mechanism for welds in large welded structures that does not affect the skills of inspectors.

Under these circumstances, an automatic inspection mechanism for welds that uses a non-contact three-dimensional shape measuring device, such as a 3D laser scanner, as exemplified by Patent Document 1, addresses the above issues by enabling automatic inspection of welds, making it possible to inspect welds without relying on skill.

Meanwhile, a mechanism such as that described in Patent Document 2, in which a 2D laser displacement meter is attached to a robot arm and the 2D laser displacement meter is scanned as the robot arm moves to acquire 3D shapes, also enables automatic inspection of welds, making it possible to inspect welds regardless of skill. Furthermore, by acquiring the coordinate system of the tip of the robot arm, it is also possible to determine the position within a structure of the weld for which shape information is being acquired.

JP 2013-68580 A Japanese Patent Application Laid-Open No. 2007-253221

The method of acquiring shape information about welds using a stationary 3D scanner, as described in Patent Document 1, had the following problems:

(1) Obtaining external shape information of welds using a 3D scanner for large welded structures over 10m requires high-density point cloud acquisition, which takes more than 12 hours to measure, so an inspection method with an even shorter lead time is needed.

(2) Depending on the location of the 3D scanner, the weld may be hidden by the laser irradiation, making it impossible to capture its shape.

Furthermore, the method of acquiring the weld shape by using a robot to scan with a 2D laser displacement sensor, as described in Patent Document 2, had the following problems:

(3) Many of the welds on railway vehicles are used to attach small parts, and many of these locations are inaccessible to robots for inspection, making it impossible to perform visual inspections of all welds.

The object of the present invention is to perform visual inspection of welds in structures quickly and reliably.

The present invention is an appearance inspection system that includes a camera unit placed within a structure, a position measurement unit that measures the position of the camera unit, a coordinate information acquisition unit that acquires the position of the camera unit within the structure and the camera direction, an image acquisition unit that acquires images from the camera unit, a weld coordinate comparison unit that creates a weld point cloud within the structure based on the position of the camera unit, the camera direction, and pixel coordinates of the image, and an image analysis unit that determines the pass/fail of welds based on the images from the camera unit and weld inspection specification information corresponding to the weld point cloud.

The present invention enables fast and reliable visual inspection of welds in structures.

1 is an overall view of a high-speed visual inspection system for structural welds. FIG. 2 shows a functional block diagram of a processing computer. 10 is a flowchart showing the operation of a processing computer. FIG. 10 is a diagram showing three-dimensional coordinates of a weld point cloud within a structure. FIG. 10 shows pixel coordinates of welds in an image corresponding to the camera position and shooting direction. FIG. 10 shows an image of a weld in a camera image. FIG. 10 is a conceptual diagram of the inside of a structure 7 where an actual appearance inspection is performed. FIG. 10 is a conceptual diagram of the inside of a structure 7 when data stored in a CAD system is used.

The following describes an embodiment of the present invention with reference to the drawings.

Figure 1A is an overall view of a high-speed visual inspection system for structural welds. The visual inspection system includes a weld inspection camera 1, position measurement scanners 2a, 2b, and 2c for acquiring three-dimensional position information and orientation of the weld inspection camera 1 within the structure 7, receivers 3a and 3b attached to the weld inspection camera 1, and a processing computer 4 for processing the position information and weld images.

Figure 1A also shows a worker 5 handling the image analysis camera, the weld 6 to be inspected, and the structure 7 to which the weld belongs.

The processing computer 4 is equipped with a CPU (Central Processing Unit), memory, a storage means (storage unit) such as a hard disk, and a network interface.

Figure 1B shows a functional block diagram of the processing computer 4. As shown in Figure 1B, the processing computer 4 receives signals from the weld inspection camera 1 and signals from the position measurement scanners 2a, 2b, and 2c. The processing computer 4 is composed of a coordinate information acquisition unit 41 that acquires the three-dimensional coordinates of the weld inspection camera 1, a weld coordinate comparison unit 42 that has weld coordinate data 46, an image acquisition unit 43 that acquires camera images, an image analysis unit 44, and a recording unit 45. The data is automatically processed by an algorithm inside the computer, and the processed data can be saved in the recording unit 45.

The multiple position measuring scanners 2a, 2b, and 2c installed on the structure 7 emit radio waves or infrared rays, allowing them to determine the distance (Dn) to the multiple receivers 3a and 3b attached to the weld inspection camera 1. The position measuring scanners 2a, 2b, and 2c and the receivers 3a and 3b are collectively referred to as the position measuring unit that measures the camera's position.

The coordinate information acquisition unit 41 receives signals from the position measurement scanners 2a, 2b, and 2c, and calculates and acquires three-dimensional position information 80 (Xc, Yc, Zc) for each of the receivers 3a and 3b in the structure 7. Then, by calculating a vector (Uc, Vc, Wc) from the obtained multiple pieces of three-dimensional position information 80 (Xc, Yc, Zc), the shooting direction 81 of the weld inspection camera 1 in the structure 7 is obtained.

Here, it has been explained that the position measuring scanners 2a, 2b, 2c calculate the distance (Dn) to the multiple receivers 3a, 3b. However, the position measuring scanners 2a, 2b, 2c may also send a time signal indicating the time it takes for radio waves reflected from the multiple receivers 3a, 3b to return to the processing computer 4, so that the coordinate information acquisition unit 41 can also calculate the distance (Dn) between the multiple receivers 3a, 3b and the position measuring scanners 2a, 2b, 2c, and calculate and acquire three-dimensional position information 80 (Xc, Yc, Zc) for each of the receivers 3a, 3b in the structure 7.

The weld coordinate comparison unit 42 is equipped with weld coordinate data 46, and by comparing the position of the weld inspection camera 1 in the structure 7 obtained from the above-mentioned three-dimensional position information 80 with the point cloud 82 based on the weld coordinate data 46 against the camera's shooting direction 81, the position is converted into pixel coordinates 85 of an image 84 displayed on a screen 83 within the weld inspection camera 1. The three-dimensional point cloud 82 and pixel coordinates 85 corresponding to each weld are used to match the weld point cloud 86 with the weld inspection specification 87 linked to the weld point cloud 86.

The image acquisition unit 43 acquires an image 84 from the weld inspection camera 1. The image acquisition unit 43 then stores a weld point cloud 86 corresponding to pixel coordinates 85 in the image 84, a data group of weld inspection specifications 87, and training data for image analysis corresponding to the weld inspection specifications 87.

The image analysis unit 44 uses image analysis techniques such as machine learning and deep learning to determine whether the welds in the image 84 captured by the weld inspection camera 1 meet the weld inspection specifications 87.

The recording unit 45 stores, as a record with one row in a table, the three-dimensional point cloud 82 corresponding to each weld in the weld coordinate matching unit 42, the weld point cloud 86 in pixel coordinates 85, the weld inspection specifications 87, and the pass/fail judgment data for the weld based on image analysis of the weld in the image analysis unit 44, using the image 84 as the primary key.

The processing computer 4 can be configured to display data related to the recording unit 45. One example of the display may show a 3D point cloud 82 within the structure 7 of the weld in the image 84, a weld point cloud 86 within pixel coordinates 85, weld inspection specifications 87, pass/fail judgment results, and inspection date and time. Other data may also be displayed as needed.

Figure 2 is a flowchart showing the operation of the processing computer 4. The actual operation of weld inspection and the data processing flow are explained below. To explain Figure 2, we will first explain Figures 6A and 6B. Figure 6A shows a conceptual diagram of the inside of structure 7, where the actual visual inspection is performed. Figure 6B is a conceptual diagram of the inside of structure 7 when data stored in CAD is used.

In Figures 6A and 6B, 80 represents the three-dimensional position information of the camera, 81 represents the camera's shooting direction, 82 represents the three-dimensional point cloud of the weld, 83 represents the screen, 84 represents the image on the screen, 85 represents the pixel coordinates of the weld in the screen coordinate system, 86 represents the point cloud of the weld in CAD, and 87 represents the weld inspection specifications.

The world coordinate system is a coordinate system of three-dimensional position coordinates within the structure 7, and the screen coordinate system is a coordinate system of two-dimensional pixel coordinates. The CAD coordinate system is a coordinate system of position coordinates stored in CAD. The camera coordinate system is a coordinate system in which the z direction is the shooting direction 81, and the x and y directions are the width and length directions of the captured image.

In step S1, the worker 5 points the weld inspection camera 1 toward the weld 6 of the structure 7, and the image acquisition unit 43 acquires an image 84.

In step S2, the coordinate information acquisition unit 41 acquires three-dimensional position information 80 (Xc, Yc, Zc) of the weld inspection camera 1 from the positional relationship between multiple position measurement scanners 2a, 2b, 2c and a single receiver 3a or receiver 3b. The three-dimensional position information of the weld inspection camera 1 may be the three-dimensional position information of either receiver 3a or 3b, or a calculated value such as averaging the position information may be used.

In step S3, the coordinate information acquisition unit 41 acquires the shooting direction 81 of the weld inspection camera 1 as a vector. Specifically, the vector (Uc, Vc, Wc) created based on the receiver's three-dimensional position information 80 (Xc, Yc, Zc) calculated from the positional relationships between the multiple position measurement scanners 2a, 2b, 2c and the multiple receivers 3a, 3b is set as the shooting direction 81 of the weld inspection camera 1.

In step S4, the weld coordinate matching unit 42 plots the values of the three-dimensional position information 80 (Xc, Yc, Zc) and shooting direction 81 (Uc, Vc, Wc) of the weld inspection camera 1 within the three-dimensional coordinate system of the three-dimensional point cloud 82 corresponding to each weld within the structure 7, and uses the camera's angle of view and the values of the width pixel size and height pixel size of the captured image to perform matrix calculations from the two-dimensional image pixel coordinates 85 to create a three-dimensional coordinate weld point cloud 86, which is output to the recording unit 45.

In step S5, the image acquisition unit 43 calls up from the recording unit 45 the data group of the weld inspection specifications 87 associated with the weld point cloud 86 corresponding to the pixel coordinates 85 created in step S4.

In step S6, the image analysis unit 44 compares the image analysis training data corresponding to the weld inspection specifications 87 with the weld at pixel coordinate 85 in the image 84.

In step S7, the image analysis unit 44 compares the image analysis training data with the weld and determines that the weld has a certain degree of similarity and passes if there is a certain degree of matching, and fails if there is not, and outputs the determination result and writes it to the recording unit 45.

This shows an example of a weld inspection operation and processing flow for a structure consisting of seven steps.

The processor in the processing computer 4 can be configured to call up a program recorded in a recording unit or the like and execute each step from S1 to S7.

When 11 welds on a box structure were inspected using the flow from steps S1 to S7, the 3D scanner took 22 minutes to measure, and an additional 3 minutes was required to match the point cloud between the 3D scan data obtained above and the 3D-CAD data, for a total of 25 minutes required to determine the pass/fail status of the welds.

Figure 3 shows the three-dimensional coordinates of the weld point cloud within the structure. Figure 4 shows the pixel coordinates of the welds in the image corresponding to the camera position and shooting direction. Figure 5 shows the image of the welds in the camera image.

In this embodiment, in step S4, the 3D point cloud of the weld shown in Figure 3 was converted into pixel coordinates within one field of view image of the camera shown in Figure 4, and the weld shown in the camera image was matched to the actual weld, and inspection was performed.

Then, using the YOLO program shown in the reference, the weld image in Figure 5 was compared with training data corresponding to the weld specifications to determine whether the weld passed or failed. As a result, when the field of view was set to 25, the same as the number of welds, the measurement time was 4 minutes, achieving a speed increase of more than 10 times. The reference here is "Redmon, Joseph & Divvala, Santosh & Girshick, Ross & Farhadi, Ali. (2016). You Only Look Once: Unified, Real-Time Object Detection. 779-788. 10.1109/CVPR.2016.91."

According to this embodiment, it is possible to perform high-speed visual inspection of welds within a structure using image analysis with a camera, without using a stationary 3D scanner or a robot and 2D laser displacement sensor.

In this example, weld inspection was performed using a weld inspection camera 1. However, instead of using a weld inspection camera 1, it is also effective to use a 3D shape measuring device, such as LIDAR, to determine the position of the weld to be inspected in the structure 7 and check whether the weld inspection specifications are met based on the 3D shape.

In this embodiment, the values of the three-dimensional position information 80 (Xc, Yc, Zc) and shooting direction 81 (Uc, Vc, Wc) held by the worker 5 are obtained from the relative positions of the position measurement scanners 2a, 2b, 2c and the receivers 3a, 3b. Alternatively, a beacon or stereo camera can be used as a position measurement unit to obtain three-dimensional position information and shooting direction.

As a result, when visually inspecting welds in a structure, each weld in the structure can be inspected quickly and reliably.

1...weld inspection camera, 2a, 2b, 2c...position measurement scanner, 3a, 3b...receiver, 4...processing computer, 5...worker, 6...weld to be inspected, 7...structure, 41...3D coordinate information acquisition device, 42...weld coordinate matching unit, 43...image acquisition unit, 44...image analysis unit, 45...recording unit, 46...weld coordinate data, 80...3D camera position information, 81...camera shooting direction, 82...weld point cloud in the camera coordinate system, 83...2D screen within the camera, 84...image captured by the camera, 85...pixel coordinates of the weld on the 2D screen within the camera, 86...weld point cloud corresponding to the weld on the 2D screen within the camera, 87...weld inspection specifications

Claims (8)

an imaging unit disposed within the structure;
a position measurement unit that measures the position of the imaging unit;
a coordinate information acquisition unit that acquires the position of the imaging unit within the structure and the imaging direction;
an image acquisition unit that acquires an image from the imaging unit;
a weld coordinate collating unit that creates a weld point cloud within the structure based on the position of the photographing unit, the photographing direction, and pixel coordinates of the image;
An appearance inspection system having an image analysis unit that determines whether a weld is acceptable or not based on the image from the photographing unit and weld inspection specification information corresponding to the weld point cloud. 2. The visual inspection system according to claim 1,
The position measurement unit
An appearance inspection system comprising a plurality of position measurement scanners arranged on the structure and a receiver arranged in the photographing unit. 2. The visual inspection system according to claim 1,
The visual inspection system, wherein the position measurement unit is a beacon or a stereo camera. 2. The visual inspection system according to claim 1,
The photographing unit is a camera for inspecting welded parts. 2. The visual inspection system according to claim 1,
The photographing unit is a visual inspection system that is a 3D measuring device. 2. The visual inspection system according to claim 1,
The image analysis unit
An appearance inspection system that compares the image from the photographing unit with training data for image analysis corresponding to the weld inspection specification information, and determines whether the image passes or fails based on similar trends. A visual inspection method for visually inspecting a welded portion in a structure using an image from an imaging unit, comprising:
creating a weld point cloud within the structure from the position of the photographing part, the photographing direction, and pixel coordinates of the image;
An appearance inspection method for determining whether a weld is acceptable or not based on the image from the photographing unit and weld inspection specification information corresponding to the weld point cloud. 8. The visual inspection method according to claim 7,
The visual inspection method uses information from a position measurement unit disposed on the structure to determine the position and direction of the photographing unit.