JP7088232B2 - Inspection equipment, inspection methods, and programs - Google Patents

Description

The present invention relates to a three-dimensional measurement technique for measuring a three-dimensional position from a photographed image of a measurement object, and in particular, a technique for determining whether or not the detection accuracy of the three-dimensional position is sufficient.

In recent years, many techniques for measuring the three-dimensional position of an object (subject) or the distance to the object from a photographed image have been used. As a method of three-dimensional measurement, there is an active stereo method in which a sine wave pattern or a pattern light to which a predetermined spatial coding is applied is projected to detect the position of an object by the principle of triangular measurement.

In order to obtain the distance or the three-dimensional position from the captured image regardless of the measurement principle, it is necessary that the object is photographed with sufficient brightness and that the luminance saturation does not occur. If the brightness is not sufficient or the brightness is saturated, accurate measurement cannot be performed. In order to deal with this problem, it is conceivable to take a plurality of images having different brightness, but this causes a problem that the processing time becomes long. When shooting is performed under conditions that do not cause brightness saturation, it is necessary to be able to determine whether the measurement can be performed with sufficient accuracy in the low brightness region.

Patent Document 1 discloses that data having a predetermined luminance amplitude or less is not used in a three-dimensional measurement using a phase shift method. Patent Document 1 states that by not using inaccurate data, it is possible to prevent an erroneous distance from being calculated in a low-luminance region.

Japanese Patent No. 5352997

However, when the distance to the region including a plurality of pixels is calculated as the average of the distances of the pixels included in the region, the distance accuracy cannot be obtained only from the luminance amplitude. The distance obtained in this way is also affected by, for example, the size of the region and the size of the offset component of the luminance. This is because the larger the region, the more stable the average value, and the larger the offset component (brighter), the larger the noise.

That is, it is not possible to determine whether or not the calculated distance has sufficient accuracy based only on the luminance amplitude in the captured image.

The present invention has been made in view of the above circumstances, and the present invention determines whether or not the obtained height is reliable in a three-dimensional measuring device that obtains the height of a measurement object based on a captured image. The purpose is to provide possible technologies.

The three-dimensional measuring device according to the first aspect of the present invention corresponds to an image acquisition means for acquiring an image of an object and a plurality of pixels included in a region set in the image based on the image. A height calculation means for obtaining the height of a subject in a pixel and obtaining a height for the region from the height of the subject for the plurality of pixels, a variation in the height of the subject for the plurality of pixels, and the plurality. A determination means for determining whether or not the height obtained for the region is reliable based on the number of pixels of the above is provided.

In this embodiment, the height calculation means can obtain the average height of the subjects of the plurality of pixels as the height for the region.

In this embodiment, the determination means is reliable in the height of the region when (the standard deviation of the subject height for the plurality of pixels) / (the number of the plurality of pixels) ^1/2 is equal to or less than the threshold value. It can be determined that it can be done. The standard deviation of the measurement result is generally not obtained unless the measurement is performed multiple times, but when the height of the region is obtained as the average height of the pixels in the region, the height for the region is calculated by the above formula. The standard deviation of the measurement result can be estimated. Therefore, the standard deviation of the height for the region can be obtained from the standard deviation of the height for each pixel in the region, and the certainty of the height for the region can be evaluated.

When the determination means determines that the height of the region is unreliable, the three-dimensional measuring device according to this embodiment expands the region and recalculates the height, or changes the imaging conditions. It is also preferable to further provide a control means that proposes to reacquire the image or perform these processes. Here, the change of the shooting condition may be any change as long as it can be changed so that a brighter image can be shot. Examples of changing the shooting conditions include changing the brightness of the illumination, changing the exposure time, and changing the F value of the optical system of the camera.

The three-dimensional measuring device in this embodiment is not particularly limited in the method of obtaining the height of the subject in each pixel. Examples of the height calculation method include a phase shift method, an optical cutting method, a coded optical projection method, and the like. In either method, the brightness value in the captured image affects the calculation accuracy.

The three-dimensional measuring device according to the second aspect of the present invention is
A projection means that projects light in a sine wave pattern,
An image acquisition means for acquiring an image of an object on which the light is projected, and an image acquisition means.
Based on a plurality of images taken with different phases of light projected on the object, the height of each of the plurality of pixels included in the region set in the image from the reference plane of the subject in the pixel. A height calculation means for obtaining the height and obtaining the average height of the plurality of pixels as the height for the region.
If σ / N ^1/2 ≤ _ET , where σ is the height variation for the plurality of pixels, N is the number of the plurality of pixels, and _ET is the height tolerance for the region. A determination means for determining that the height of the region is reliable, and otherwise unreliable.
To prepare for.

In this embodiment, the height calculation means obtains the phase of the light for each of the plurality of pixels based on the plurality of images, performs phase connection processing and phase connection error correction processing, and then performs the plurality of phases. The height of each of the pixels is obtained, and the determination means classifies the phase before the correction process into a plurality of classes, and the number of pixels included in the class having the largest number of included pixels is the plurality of pixels. It is also preferable to determine that the height for the region is unreliable even when it is less than a predetermined ratio of the total number of pixels.

The present invention can be regarded as a three-dimensional measuring device including at least a part of the above means. Further, the present invention can also be regarded as a three-dimensional measurement method including at least a part of the processing performed by the above means. Further, it can be regarded as a computer program for causing a computer to execute each step of these methods, or as a computer-readable storage medium in which the program is stored non-temporarily. Each of the above configurations and processes can be combined with each other to construct the present invention as long as there is no technical contradiction.

According to the present invention, it is possible to determine whether or not the obtained height is reliable in the three-dimensional measuring device that obtains the height of the object to be measured based on the captured image.

FIG. 1 is a diagram showing a configuration of a three-dimensional measuring device according to an embodiment. FIG. 2A is a diagram showing a pattern of projected light, and FIG. 2B is a diagram showing a measurement principle by a phase shift method. FIG. 3 is a flowchart showing the entire flow of the component floating inspection process, which is one of the substrate inspections according to the embodiment. FIG. 4 is a flowchart showing the details of the process of obtaining the height of the target area in the substrate inspection process. 5 (A) is a diagram showing a setting example of the distance detection region, FIG. 5 (B) is a histogram of the phase error before the phase connection error correction, and FIG. 5 (C) is a histogram of the phase error after the phase connection error correction. be. 6 (A) and 6 (B) are diagrams showing the experimental results for confirming the estimation accuracy of the standard deviation of the region height.

Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. The three-dimensional measurement system according to the present embodiment projects a pattern light onto the object to be measured to take an image, and measures the three-dimensional shape of the object to be measured by using the principle of triangular measurement.

(First Embodiment)
Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. This embodiment is a three-dimensional measuring device for inspecting a substrate. The three-dimensional measuring device according to the present embodiment projects a pattern light onto a substrate (measurement object) on which a component is attached to take an image, and measures the three-dimensional shape of the measurement object using the principle of triangular surveying. do. In this embodiment, the surface height of the board (distance from the reference plane parallel to the surface on which the board is installed) is measured, and whether or not the parts are normally mounted on the board is inspected (part floating inspection). ..

[Constitution]
FIG. 1 is a diagram showing a configuration of a three-dimensional measuring device 100 according to the present embodiment. The three-dimensional measuring device 100 is roughly composed of a projection unit 110, a camera 120, and an arithmetic unit 130.

The projection unit 110 projects a sine wave pattern light onto the object to be measured 150. The projection unit 110 includes a light source such as a halogen lamp or a xenon lamp, a pattern generation element such as a liquid crystal element for forming a pattern in the light emitted from the light source, and an optical system such as a microlens. The light projected by the projection unit 110 does not have to be visible light, and may be invisible light such as infrared light.

The pattern of light projected by the projection unit 110 will be described. In the present embodiment, since the three-dimensional measurement is performed by using the phase shift method, the projection unit 110 projects the periodically fluctuating striped pattern light. FIG. 2A shows an example of a pattern of light projected by the projection unit 110. The example shown in FIG. 2A shows a pattern light of vertical stripes, that is, a pattern light in which the brightness continuously changes in a sinusoidal manner in the horizontal direction and the brightness is constant in the vertical direction. The frequency of the sine wave (spatial frequency of the pattern light) is not particularly limited. In the present embodiment, the vertical stripe pattern light is projected a plurality of times while shifting the phase. At least three projections are required for phase calculation, and the larger the number of shots, the better, considering the measurement accuracy. In the present embodiment, the phase is shifted by 90 ° (quarter cycle) for ease of measurement, and four projections are performed. Hereinafter, these plurality of vertical stripe pattern lights will be referred to as a vertical stripe pattern light set. In this way, the height of the object to be measured 150 can be measured by calculating the phase of the change in brightness generated in the image from the plurality of images taken by projecting each of the vertical stripe pattern light sets onto the object to be measured 150. ..

FIG. 2B shows luminance values I1 to I4 at a certain point in a plurality of images (here, four images) taken with different phases of the sine waves of the projected light. From these luminance values, the phase shift θ at that point can be obtained (phase restoration). Based on this phase, the height of the point can be obtained using the principle of triangulation.

The camera 120 captures the measurement object 150 on which the pattern light is projected. The camera 120 takes a picture every time the projection unit 110 switches the phase of the pattern light based on the command from the control unit 131. In the present embodiment, the image data taken by the camera 120 is input to the image input unit 133.

The arithmetic unit 130 is a computer including a processor such as a CPU, a memory (storage device) such as RAM and ROM, an interface with an external device, an input device such as a keyboard and a mouse, and an output device such as a display and a speaker. The arithmetic unit 130 performs functions such as a control unit 131, an image input unit 133, a height calculation unit 134, an accuracy determination unit 135, an inspection unit 136, and a notification unit 137 by executing a program stored in the memory by the CPU. offer. Some or all of the above functions may be realized by a hardware circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). Details of each of these functional units will be described together with the following flowcharts.

[process]
[[Overall processing of board inspection]]
FIG. 3 is a flowchart showing the entire flow of the component float inspection process, which is one of the substrate inspections performed by the three-dimensional measuring device 100 according to the present embodiment. The flowchart shown here is an example of processing, and the processing content and processing order may be changed as appropriate. It is assumed that the calibration (calibration process) of the projection unit 110 and the camera 120 has been completed.

First, in step S102, a sine wave pattern whose phase is different for each projection timing is projected from the projection unit 110 onto the measurement object 150, and the camera 120 photographs the measurement object 150 in accordance with the projection of the pattern light. .. The projection unit 110 projects the sine wave pattern light of the specified phase at the specified timing onto the measurement object 150 according to the command from the control unit 132. The control unit 132 controls the camera 120 to take a picture each time the projection unit 110 changes the phase of the pattern light. The image data taken by the camera 120 is input to the image input unit 133 and temporarily stored. The plurality of image data captured in step S102 are collectively referred to as captured image data.

Next, in step S104, the user sets a plurality of distance detection areas in the captured image via the input unit (not shown). Alternatively, the three-dimensional measuring device 100 automatically sets a plurality of distance detection areas in the captured image by using a known technique. It is desirable that the distance detection area be an area in which the heights of the points are equal. FIG. 5A shows an example of setting the distance detection area. FIG. 5A shows an image 51 in which the substrate including the component 52 is photographed. Here, four distance detection areas 53a to 53d are set in the component 52. If the positional relationship between the substrate and the camera at the time of shooting is determined in advance, the distance detection area can be determined in advance.

As will be described later, one height is calculated for each distance detection area. From the heights of these four regions, it is determined whether the component 52 is horizontally installed at an appropriate position (height), that is, whether the component 52 is properly mounted on the substrate.

In step S106, the height calculation unit 134 obtains the height of each point (each pixel) included in the area for each of the distance detection areas 53a to 53d, and uses the average of the heights as the height of the area. decide.

In step S108, the inspection unit 136 determines whether or not the component 52 is normally attached from the height of each area. Specifically, if the maximum value of the distance difference in each region is within the reference value (allowable variation) and the average of the distances in each region is within the reference value (S108-YES), the inspection unit 136. Determines that the component 52 is normally attached (S110). It is assumed that these reference values (allowable variation in height and allowable range of height) are set in advance. The conditions for determining whether or not the parts are normally mounted are not limited to the conditions shown here.

On the other hand, in the determination of step S108, if the maximum value of the distance difference exceeds the reference value, it is found that the component 52 is not mounted horizontally, and if the average distance is outside the reference value, it is not mounted at the correct height. You can see that. Therefore, in this case (S108-NO), the inspection unit 136 determines that the mounting state of the component 52 is abnormal (S112).

[[Detailed processing of distance calculation processing]]
FIG. 4 is a flowchart showing a detailed flow of the distance calculation process in step S106. FIG. 4 is a process for obtaining the height of one distance detection area. Therefore, the process shown in FIG. 4 is executed for each of the distance detection areas 53a to 53d. Hereinafter, the distance detection area targeted for processing in FIG. 4 is referred to as a target area.

In step S202, the height calculation unit 134 calculates the phase for each pixel in the target region. The phase φ of each pixel can be converted to a height h by using the difference from the reference phase determined for each pixel by calibration. When finding the phase, an operation called phase connection is performed to convert the phase into a continuous quantity over a wide range. Not all phases are connected correctly by phase connection, and a phase connection error may occur. When a phase connection error occurs, 2πi / α (i is an integer and α is the connection magnification) is added to the phase after the phase connection as an error.

FIG. 5B shows a histogram of the phase error after the phase connection. The phase error is the difference between the phase of each pixel and the average phase in the target region. The error due to noise follows a normal distribution (Gaussian distribution), but the error due to a phase connection error takes a discrete value as described above. Therefore, the shape of the histogram is a mixture of α-type normal distributions whose centers are offset by 2πi / α. Here, since the probability that a phase connection error occurs is lower than the probability that a phase connection error does not occur, the distribution near zero phase error is the largest set as shown in FIG. 5 (B).

In step S204, the height calculation unit 134 classifies the distribution of FIG. 5B into a plurality of classes. By this processing, a plurality of distributions separated by dotted lines in FIG. 5B can be obtained.

In step S206, the height calculation unit 134 determines whether or not the number of pixels belonging to the class 54 having the largest number of pixels is equal to or greater than a predetermined ratio of the number of pixels in the entire target area. If this determination is YES, the process proceeds to step S208 to correct the phase connection error. Specifically, the height calculation unit 134 subtracts 2πi / α from the phase of each class (the integer i is determined for each class). As a result, the histogram of the phase error has one normal distribution 55 with a center of 0 as shown in FIG. 5 (C). On the other hand, if the determination in step S206 is NO, the phase connection error cannot be corrected correctly. Therefore, it is proposed to proceed to step S220 to change the shooting conditions and perform reshooting.

In general, the number of phases classified into a class without phase connection errors is sufficiently larger than the number of phases classified into other classes. However, depending on the conditions, there may not be a sufficient difference between the number of elements in the class without connection errors and the number of elements in other classes, and in such cases, the estimation of the class without connection errors may be incorrect. .. In order to prevent such an erroneous determination, the determination in step S206 has been introduced. If the condition of step S206 is not satisfied, it is determined that the correction of the connection error is impossible. The predetermined ratio in step S206 may be appropriately evaluated using an experiment, but it is considered that the number of elements in the class having a connection error does not exceed 35% of the total. Therefore, 35% can be used as a predetermined ratio.

In step S212, the height calculation unit 134 calculates the corrected phase standard deviation σ (that is, the standard deviation of the distribution 55 shown in FIG. 5C). Further, in step S214, the height calculation unit 134 acquires the number of pixels N (that is, the number of elements of the distribution 55) in the target region.

In step S216, the accuracy determination unit 135 determines whether or not σ / N ^1/2 is equal to or less than the tolerance _ET in the height measurement. The height measurement tolerance _ET is preset by the user and stored in memory. Since the target area is composed of pixels having the same (estimated) height, when the height of the target area is calculated as the average value of the heights of each pixel in the target area, the height of the target area is calculated. The standard error is σ / N ^1/2 . Therefore, by the determination in step S216, it can be determined whether or not the measurement error of the height of the region is within the permissible error. Here, the condition is that σ / N ^1/2 is equal to or less than the margin of error _ET , but the condition may be _ET / 2 or less or _ET / 3 or less.

If the determination in step S216 is YES, it can be estimated that the measurement error (deviation from the true value) of the height of the target region is within the permissible error. Therefore, in this case, the process proceeds to step S218, and the height calculation unit 134 determines the average height of each pixel in the target area as the height of the target area.

On the other hand, if the determination in step S216 is NO, there is a high possibility that the estimation error of the height of the target region will be larger than the permissible error. Therefore, in this case, the process proceeds to step S220, and the control unit 131 controls the notification unit 137 to propose to change the shooting conditions and perform the remeasurement. Examples of changing the shooting conditions include changing the brightness of the illumination, changing the exposure time, and changing the F value of the optical system of the camera. In step S220, the control unit 131 may automatically change the shooting conditions and automatically perform remeasurement instead of proposing a change in the shooting conditions.

[Evaluation results]
6 (A) and 6 (B) are diagrams showing the evaluation results by the experiment. FIG. 6A shows the evaluation results for the region with relatively favorable conditions, and FIG. 6B shows the evaluation results for the region with worse conditions than FIG. 6A.

In this experiment, phase estimation was tried 50 times. A plane region (an example is shown as region 61 in FIG. 6 (A) and region 65 in FIG. 6 (B)) for which the average height is to be obtained is set, and the average phase in the region is set for each experimental data. And standard deviation estimation was performed. The standard deviation was obtained for the average phase of 50 times, and the standard deviation of the average phase was compared with the estimated value in each trial. We also evaluated the relationship between the area size (number of pixels) and the standard deviation by changing the area of the area.

In FIGS. 6 (A) and 6 (B), the circle 62 indicates the standard deviation of the average phase of 50 times. The vertical line 63 (actually a plot of 50 points) shows the standard deviation of the average phase estimated from each experimental data (standard deviation of phase in region and number of pixels in region). Curve 64 shows the theoretical value of the standard deviation of the mean phase.

From FIGS. 6 (A) and 6 (B), it can be seen that the standard deviation can be estimated with an accuracy of about 0.001 radian even when the conditions are relatively bad. Further, it can be seen that as the region size (the number of pixels in the region) increases, the error in the average phase of the region decreases in the order of N- ^{1 / 2} with respect to the region size N.

[Advantageous effect of this embodiment]
Originally, the measurement error cannot be obtained unless a plurality of measurements are performed. In the present embodiment, since the height of the region is obtained as the average of the heights of a plurality of pixels in the region, the standard error of the height of the region is obtained from the standard deviation of the height in the region and the number of pixels in the region. Can be estimated. Therefore, the height measurement error can be obtained by one measurement (more accurately, one set of pattern light projection and photographing).

In the substrate inspection, the luminance amplitude due to the sine wave pattern of illumination becomes small in the region where the reflectance is low, and due to factors such as a decrease in the SN ratio and a phase connection error, it is not possible to perform height measurement with sufficient accuracy on a pixel-by-pixel basis. On the other hand, in the case of inspecting the floating of parts in the substrate inspection, the inspection is possible if the variation accuracy of the average height in the area unit is sufficiently stable, not in the pixel unit. In the present embodiment, it is possible to determine whether or not the variation in the height measurement in each region is within the allowable range. That is, the height can be measured with the required accuracy even in a low amplitude region where the reflectance is low. In addition, if sufficient accuracy cannot be obtained, it can be proposed to change the shooting conditions and perform remeasurement.

(Modification example)
In the above description, the height of the component is obtained by using the phase shift method, but the specific method for determining the height of the component is not limited to the phase shift method. As other methods that can be used, a light cutting method that projects slit light or spot light, a coded light projection method that projects spatially coded pattern light, and the like can also be used. Regardless of the specific method, the error of the average height in the region can be obtained from the height variation and the region size at each point in the region.

Further, the three-dimensional measuring device according to the present invention does not necessarily have to be incorporated into a substrate inspection device or an inspection device for use. The measurement result of the three-dimensional measurement can be used for any purpose, and the present invention can be regarded as a three-dimensional measuring device or a three-dimensional measuring method.

Further, although the height of the object surface is obtained in the above embodiment, a position along an arbitrary axis, a position in a certain projection plane, a three-dimensional position, or the like may be measured. Since the variation of the measured value of each pixel in the target area follows a normal distribution, the size of the target area should be smaller than the error due to noise when the measurement target is a three-dimensional position. Must be set to.

100: 3D measurement device 110: Projection unit 120: Camera 130: Arithmetic unit 150: Measurement object 131: Control unit 133: Image input unit 134: 3D position calculation unit 135: Accuracy determination unit 136: Inspection unit 137: Notification Department

Claims (10)

An inspection device that inspects whether an object is normally attached to a board.
An image acquisition means for acquiring an image of the object and
Based on the image, the height of the subject in the pixel is obtained for each of the plurality of pixels included in the region set in the image, and the height of the subject from the plurality of pixels is the height of the region. Height calculation means to find
The height obtained for the region when the value obtained by dividing the variation in the subject height for the plurality of pixels included in the region by the value corresponding to the number of the plurality of pixels is equal to or less than a predetermined threshold value. A determination means that determines that the image is reliable, and if it is not, it is determined that the image is unreliable.
Equipped with
The height calculation means obtains the height of each of the plurality of regions of the object, and obtains the height of the region.
When the heights of the plurality of regions are determined to be reliable by the determination means, whether or not the object is normally attached to the substrate is determined based on the heights of the plurality of regions. Further equipped with inspection means for determination,
An inspection device characterized by that. The inspection means is the object if the maximum value of the distance difference between the heights of the plurality of regions is within the first reference value and the average of the heights of the plurality of regions is within the second reference value. Is determined to be normally attached to the board,
The inspection device according to claim 1. The height calculation means obtains the average of the heights of the subjects of the plurality of pixels as the height for the region.
The inspection device according to claim 1 or 2. The variation in the subject height for the plurality of pixels is the standard deviation of the subject height for the plurality of pixels.
The inspection device according to any one of claims 1 to 3. When the determination means determines that the height of the area is unreliable, the area is enlarged to recalculate the height, or the shooting conditions are changed and the image is reacquired. Further equipped with a control means that proposes to perform the processing of
The inspection device according to any one of claims 1 to 4. The height of the subject in the pixel is obtained for each of the pixels by any of the phase shift method, the optical cutting method, and the coded light projection method.
The inspection device according to any one of claims 1 to 5. An inspection device that inspects whether an object is normally attached to a board.
A projection means that projects light in a sine wave pattern,
An image acquisition means for acquiring an image of an object on which the light is projected, and an image acquisition means.
Based on a plurality of images taken with different phases of light projected on the object, the height of each of the plurality of pixels included in the region set in the image from the reference plane of the subject in the pixel. A height calculation means for obtaining the height and obtaining the average height of the plurality of pixels as the height for the region.
If σ / N ^1/2 ≤ _ET , where σ is the height variation for the plurality of pixels, N is the number of the plurality of pixels, and _ET is the height tolerance for the region. A determination means for determining that the height of the region is reliable, and otherwise unreliable.
Equipped with
The height calculation means obtains the height of each of the plurality of regions of the object, and obtains the height of the region.
When the heights of the plurality of regions are determined to be reliable by the determination means, whether or not the object is normally attached to the substrate is determined based on the heights of the plurality of regions. Further equipped with inspection means for determination,
An inspection device characterized by that. The height calculation means obtains the phase of the light for each of the plurality of pixels based on the plurality of images, performs phase connection processing and phase connection error correction processing, and then for each of the plurality of pixels. The height of the
The determination means classifies the phase before the correction process into a plurality of classes, and the number of pixels included in the class having the largest number of pixels included may be less than a predetermined ratio of the total number of the plurality of pixels. , Judge that the height for the area is unreliable,
The inspection device according to claim 7. It is an inspection method to inspect whether the object is normally attached to the board.
An image acquisition step to acquire an image of an object, and
Based on the image, the height of the subject in the pixel is obtained for each of the plurality of pixels included in the region set in the image, and the height of the subject from the plurality of pixels is the height of the region. The height calculation step to find
The height obtained for the region when the value obtained by dividing the variation in the subject height for the plurality of pixels included in the region by the value corresponding to the number of the plurality of pixels is equal to or less than a predetermined threshold value. Is determined to be reliable, and if not, it is determined to be unreliable.
Including
In the height calculation step, for each of the plurality of regions of the object, the height of the region is obtained.
If the heights of the plurality of regions are determined to be reliable in the determination step, whether or not the object is normally attached to the substrate is determined based on the heights of the plurality of regions. Including additional inspection steps to determine,
Inspection methods. A program for causing a computer to perform each step of the method according to claim 9.

Images