JP6969619B2 - Calibration method - Google Patents

Description

The present invention relates to a calibration method .

In the stereo method, which is one of the three-dimensional measurement technologies, pixel blocks that have a correlation with the pixel blocks in the image captured by one camera are specified in the image captured by the other camera (corresponding points are extracted). Distance data is calculated using the principle of triangulation from the misalignment, which is the relative amount of deviation in both images. For the purpose of improving the calculation accuracy of distance data, a technique using a pattern having many feature points (for example, a random pattern having an irregular pattern) is known in order to increase the density of corresponding point extraction. ..

For example, Patent Document 1 discloses a technique for cutting (removing) low-frequency components contained in a random pattern for the purpose of improving uniformity and dispersibility.

However, in the prior art, since the random pattern is generated without considering the imaging distance (distance between the camera and the random pattern at the time of imaging), there is a problem that the feature points cannot be extracted stably. be.

In order to solve the above-mentioned problems and achieve the object, the present invention includes a step of printing a pattern including a second image generated by synthesizing a plurality of first images each containing different spatial frequencies. It is characterized by including a step of capturing the printed pattern with a stereo camera and a step of calibrating the stereo camera based on the captured image obtained by capturing the pattern .

According to the present invention, feature points can be stably extracted even if the imaging distance of the pattern changes.

FIG. 1 is a diagram showing an example of the configuration of the system of the embodiment. FIG. 2 is a schematic diagram showing an example of the relationship between the appearance of a predetermined pattern image included in a captured image and the imaging distance. FIG. 3 is a diagram showing an example of the hardware configuration of the pattern projection device. FIG. 4 is a diagram showing an example of the function of the pattern projection device. FIG. 5 is a diagram for explaining an example of a method for generating a first image. FIG. 6 is a flowchart showing an example of processing by the acquisition unit. FIG. 7 is a diagram for explaining an example of a method for generating a second image. FIG. 8 is a flowchart showing an operation example of the pattern projection device. FIG. 9 is a diagram showing an example of the configuration of the modified example system. FIG. 10 is a schematic view showing a state in which an image-receiving tool of a modified example is imaged. FIG. 11 is a schematic view showing a state in which an image-receiving tool of a modified example is imaged. FIG. 12 is a diagram showing an example of the function of the calculation unit of the modified example. FIG. 13 is a flowchart showing an operation example of the calculation unit of the modified example.

Hereinafter, embodiments of the calibration method according to the present invention will be described in detail with reference to the accompanying drawings. Hereinafter, as an example of the image processing apparatus to which the present invention is applied, a pattern projection apparatus, which is one aspect of the image projection apparatus, will be described as an example, but the present invention is not limited thereto.

FIG. 1 is a diagram showing an example of the configuration of the system 1 of the present embodiment. As shown in FIG. 1, the system 1 of the present embodiment includes a pattern projection device 10 and a three-dimensional measuring device 20.

The pattern projection device 10 includes a second feature point including a large number of feature points on the surface of the image-imposed tool 30 (an example of an object) used for determining the correction parameters for correcting the coordinate values of the captured image described later. Projects light based on an image (described later). As a result, a projection image representing the second image described later is formed on the surface (projection surface) of the image-imaging tool 30. From a different point of view, it can be considered that the pattern projection device 10 displays the second image described later on the object to be imaged (object) 30. The detailed configuration of the pattern projection device 10 will be described later.

The three-dimensional measuring device 20 uses a stereo camera composed of a pair of cameras, and has a pixel block in an image obtained by imaging with one camera (may be referred to as an “captured image” in the following description). Correlated pixel blocks are specified in the captured image obtained by imaging with the other camera, and distance data is calculated from the misalignment, which is the relative deviation amount in both captured images, using the principle of triangular measurement. In the present specification, "imaging" refers to converting an image of a subject (imaging target) imaged by an optical system such as a lens into an electric signal.

In the present embodiment, the three-dimensional measuring device 20 has an imaging unit 21, a calculation unit 22, and a measuring unit 23. The image pickup unit 21 takes an image of the device to be imaged 30 on which the second image described later is displayed. In this example, the image pickup unit 21 is composed of a stereo camera.

The calculation unit 22 was obtained by imaging by the image pickup unit 21 based on the image captured by the image pickup tool 30 and the distance (known distance) between the image pickup unit 21 and the image pickup tool 30. Calculate the correction parameters for correcting the captured image. As a method for calculating this correction parameter, various known techniques can be used. For example, the methods disclosed in Japanese Patent No. 4109077 and Japanese Patent No. 4501239 can be used. In the present embodiment, the calculation unit 22 has a correction parameter (first correction parameter) for correcting an captured image obtained by imaging by one of the cameras (first camera) constituting the stereo camera, and the other. The correction parameters (second correction parameters) for correcting the captured image obtained by the imaging by the camera (second camera) of the above are calculated respectively.

The measuring unit 23 measures the distance data based on the corrected image obtained by correcting the captured image obtained by the imaging by the imaging unit 21 using the correction parameters calculated by the calculation unit 22. As a method for calculating the distance data, various known techniques can be used. For example, matching processing by SAD (Sum of Absolute Difference), NCC (Normalized Cross Correlation), or the like is performed to calculate the parallax value, and the parallax value is converted to a distance as needed. In the present embodiment, the measuring unit 23 has the first corrected image obtained by correcting the captured image obtained by the first camera and the second corrected image by the second camera. The parallax is calculated from the second corrected image obtained by correcting the captured image obtained in the above using the second correction parameter, and the reference of the first corrected image and the second corrected image. It is possible to generate and output a parallax image (depth map) showing the depth value (depth) of each pixel value of the image.

Next, the configuration of the pattern projection device 10 will be described. Prior to the description of the specific contents, the outline of the features of the pattern projection device 10 of the present embodiment will be described. In FIG. 2, the imaging unit 21 of the above-mentioned three-dimensional measuring device 20 displays an arbitrary pattern image (a pattern image including a large number of feature points, referred to as a “predetermined pattern image” for convenience of explanation) by the pattern projection device 10. The relationship between the appearance of a predetermined pattern image included in the captured image obtained by imaging the image-imaging tool 30 and the imaging distance (distance between the imaging unit 21 and the imaging tool 30 at the time of imaging). It is a schematic diagram which shows an example. In the example of FIG. 2, the predetermined pattern image displayed on the image-imaging tool 30 can accurately recognize the predetermined pattern image when the imaging distance is a medium distance. That is, the spatial frequency corresponding to the predetermined pattern image here is assumed to indicate a value that can accurately recognize the predetermined pattern image when the imaging distance is a medium distance.

When the imaging distance becomes x times, the size of the object reflected in the captured image appears to be 1 / x times, that is, the spatial frequency is equivalent to x times the original spatial frequency, so it is included in the predetermined pattern image. The pattern (pattern) to be displayed becomes visible as a finer pattern. Therefore, in the example of FIG. 2, when the imaging distance becomes larger than the medium distance (at a long distance), the patterns included in the predetermined pattern image appear as finer patterns, and each pattern cannot be recognized. It will be crushed and the contrast will decrease. On the contrary, when the imaging distance is smaller than the medium distance (at a short distance), only a part of the predetermined pattern image is visible, and one pattern (for example, a black rectangle) has several tens of pixels (1 pixel is 1). When it covers (representing the size of a pixel), the feature (typically, the change in shade) disappears in the center of the pattern, and the feature point cannot be extracted. Therefore, when the distance (imaging distance) between the image pickup tool 30 on which the predetermined pattern image is displayed and the image pickup unit 21 is larger than the medium distance (long distance) or smaller than the medium distance (short distance). In the case), there arises a problem that the feature points cannot be stably extracted, and the calibration (correction) of the captured image and the calculation of the distance data cannot be performed with high accuracy. That is, there arises a problem that the parameter calculation by the calculation unit 22 and the distance measurement by the measurement unit 23 cannot be performed accurately.

Therefore, in the present embodiment, the pattern projection device 10 includes a spatial frequency component (pattern) suitable for each distance in one pattern image. That is, when it is assumed that the imaging distance is smaller than the medium distance, the pattern image should include a spatial frequency component smaller than that of the medium distance (a spatial frequency component suitable for a distance smaller than the medium distance). .. On the contrary, when assuming that the imaging distance is larger than the medium distance, the pattern image should include the spatial frequency component larger than that of the medium distance (the spatial frequency component suitable for the distance larger than the medium distance). do. As a result, even if the imaging distance changes, a spatial frequency pattern suitable for the changed distance appears as a feature, so that feature points can be stably extracted. Since the spatial frequency suitable for the distance depends on the lens angle of view, the extraction range in feature point extraction, etc., the one that is advantageous for feature point extraction at each distance in the three-dimensional measuring device 20 is appropriately selected. Will be done. Hereinafter, a specific configuration of the pattern projection device 10 of the present embodiment will be described.

FIG. 3 is a diagram showing an example of the hardware configuration of the pattern projection device 10. As shown in FIG. 3, the pattern projection device 10 includes a CPU 101, a ROM 102, a RAM 103, a projection unit 104, and an I / F unit 105.

The CPU 101 comprehensively controls the operation of the pattern projection device 10.

The ROM 102 is a non-volatile memory for storing various data such as programs.

The RAM 103 is a volatile memory that functions as a work area (work area) for processing various operations executed by the CPU 101.

Under the control of the CPU 101, the projection unit 104 projects light based on a projection image (in this example, a second image described later) onto the projection target. In the example of FIG. 3, the projection unit 104 includes a light source 106, an image forming unit 107, and a lens unit 108. The light emitted from the light source 106 is supplied to the image forming unit 107. The image forming unit 107 has a function of converting the light supplied from the light source 106 into light based on a projection image (a second image described later) and outputting it to the lens unit 108. For example, the image forming unit 107 may be a transmissive liquid crystal element, a reflective liquid crystal element, a DMD (Digital Micro mirror Device), or the like. The lens unit 108 projects the light output from the image forming unit 107 toward the projection object. The lens unit 108 is composed of, for example, a plurality of lenses, prisms, mirrors, and other optical elements.

The I / F unit 105 is an interface for connecting to an external device.

FIG. 4 is a diagram showing an example of the functions of the pattern projection device 10 of the present embodiment. As shown in FIG. 4, the pattern projection device 10 includes an acquisition unit 110, a generation unit 120, and an output control unit 130. For convenience of explanation, FIG. 4 mainly illustrates the functions according to the present invention, but the functions of the pattern projection device 10 are not limited to these.

Further, in the present embodiment, the functions (acquisition unit 110, generation unit 120, output control unit 130) of the pattern projection device 10 are realized by the CPU 101 executing a program stored in the ROM 102 or the like. It is not limited to. For example, at least a part of the functions of the pattern projection device 10 may be realized by a dedicated hardware circuit (semiconductor integrated circuit or the like).

The acquisition unit 110 acquires a plurality of first images corresponding to a plurality of spatial frequencies and one-to-one. More specifically, each of the plurality of first images contains at least a component (pattern) of the corresponding spatial frequency. In the present embodiment, the acquisition unit 110 generates an original image by randomly determining the pixel value of each of the plurality of pixels included in the image area having a size different from the size of the first image, and the original image. By changing the size of the image to the same size as the first image, the first image corresponding to the spatial frequency of one of the plurality of spatial frequencies is generated. In this example, the size of the image area is smaller than the size of the first image, and the acquisition unit 110 expands the size of the generated original image to the same size as the first image, so that the spatial frequencies are among the plurality of spatial frequencies. A first image corresponding to one spatial frequency can be generated.

As an example, a case where a first image corresponding to a certain spatial frequency is generated will be described as an example. In this example, it is assumed that the size of the first image is 6 pixels in the horizontal direction (horizontal direction) and 4 pixels in the vertical direction (vertical direction). Not limited. As shown in FIG. 5, the acquisition unit 110 is an image area having a size of 1/2 of the size of the desired first image (in the example of FIG. 5, the size is 3 pixels in the horizontal direction and 2 pixels in the vertical direction). To secure. Then, the original image is generated by determining the pixel value of each of the plurality of pixels (3 × 2 = 6 pixels) included in the image area by a random number or the like. In the example of FIG. 5, it is assumed that the image has a gray scale of 256 gradations, and the pixel value of each pixel is set to either 0 (minimum gradation value) or 255 (maximum gradation value). The pixel value is not limited to the two values of "0" indicating the minimum gradation and "255" indicating the maximum gradation, and may be in the form of selecting an arbitrary value from 256 steps. .. Further, the same processing can be performed regardless of whether the pixel is a color image having a plurality of colors or a monochrome image showing white or black.

Then, the acquisition unit 110 doubles the size of each of the horizontal direction and the vertical direction in order to make the size of the original image the size of the desired first image. As a result, a pattern image showing a spatial frequency (corresponding to a certain spatial frequency) that is 1/2 of the original image can be generated as the first image. In this example, interpolation is performed by the nearest neighbor method (nearest neighbor method), but the interpolation process is not limited to this, and interpolation processing such as linear enlargement method (bilinear method) or cubic interpolation method (bicubic method) is performed. There may be. In the example of FIG. 5, since the sizes of the horizontal direction and the vertical direction are doubled, the enlarged image can be treated as a pattern image (first image) showing a spatial frequency of 1/2 of the original image. ..

FIG. 6 is a flowchart showing an example of processing by the acquisition unit 110. As shown in FIG. 6, first, the acquisition unit 110 secures an image area smaller than the size (desired size) of the first image (step S1). Next, the acquisition unit 110 randomly determines the pixel value of each of the plurality of pixels included in the image area by a random number or the like (step S2) to generate an original image. Next, the acquisition unit 110 enlarges the size of the generated original image to the same size as the first image (step S3). Thereby, a pattern image showing one of the plurality of spatial frequencies (spatial frequency of 1/2 of the original image) can be generated as the first image.

In this example, the pattern (pattern) represented by the first image is an irregular pattern, but the pattern is not limited to this, and the pattern represented by the first image may be a regular pattern. For example, the acquisition unit 110 generates an original image by determining the pixel value of each of a plurality of pixels included in an image area having a size different from the size of the first image according to a predetermined rule, and the size of the original image. The first image can also be generated by changing to the same size as the first image. A predetermined rule can be set arbitrarily. For example, a column composed of a set of pixels showing the maximum gradation value and a column composed of a set of pixels showing the minimum gradation value are alternately arranged. As described above, the pixel value of each pixel in the image region may be determined. For example, the pixel showing the maximum gradation value and the pixel showing the minimum gradation value are in the row direction and the column direction. The pixel value of each pixel in the image region may be determined so as to be arranged alternately in each.

Further, in the present embodiment, the acquisition unit 110 generates a plurality of first images corresponding to a plurality of spatial frequencies and one-to-one, but the acquisition unit 110 is not limited to this, and the acquisition unit 110 is an external server device. A plurality of first images generated in advance may be acquired from (or an external storage device), a memory inside the pattern projection device 10, or the like. The method of generating a plurality of first images corresponding to a plurality of spatial frequencies and one-to-one is arbitrary. For example, the plurality of spatial frequencies correspond to a desired plurality of distances and one-to-one. May be good. That is, when the imaging distance is one distance, the spatial frequency corresponding to each distance is set (for example, an experiment) so that the spatial frequency component corresponding to the one distance appears as a feature (the observer can accurately recognize it). It may be in a form that is preset). The appearance of the image displayed on the object to be imaged (object) 30 changes not only with the imaging distance but also with the focal length of the camera and the degree of blurring, so the spatial frequency should be determined in consideration of these factors. It is desirable to select.

Returning to FIG. 4, the explanation will be continued. The generation unit 120 generates a second image by synthesizing a plurality of first images acquired by the acquisition unit 110. In the present embodiment, the generation unit 120 obtains and sets the average of the brightness values of the pixels in each of the plurality of first images for each of the plurality of pixels included in each of the plurality of first images. , Generate a second image. For example, as shown in FIG. 7, by the acquisition unit 110, the first image showing the spatial frequency f = 1/64, the first image showing the spatial frequency f = 1/16, and the spatial frequency f = 1/4. It is assumed that each of the first images showing the above is generated. In this example, since the size of each of the plurality of first images generated by the acquisition unit 110 is the same, the three first images are for each of the plurality of pixels included in each of the three first images. The average of the brightness values of the pixels in each of the above is calculated. Then, by setting the pixel value of each pixel in the image area of the same size as the first image to the average value of the brightness values of the pixel in each of the three first images, the second image (composite). Image) can be generated.

FIG. 8 is a flowchart showing an operation example of the pattern projection device 10 when generating the second image. As shown in FIG. 8, first acquisition unit 110 generates a plurality of first images corresponding to a plurality of spatial frequencies in one-to-one (step S 10). The specific contents are as described above. Then, generation unit 120, a plurality of first image generated in step S 10 generates a second image by synthesizing (step S 11). The specific contents are as described above.

Returning to FIG. 4, the explanation will be continued. The output control unit 130 controls the output based on the second image generated by the generation unit 120. In the present embodiment, the output control unit 130 controls the projection unit 104 so as to project the light based on the second image generated by the generation unit 120 onto the image-receiving tool 30.

In the present embodiment, the pattern projection device has been described as an example of the image processing device to which the present invention is applied, but the type of the image processing device to which the present invention is applied can be arbitrarily changed. In short, the image processing apparatus to which the present invention is applied may be in a form having at least a function corresponding to the above-mentioned acquisition unit 110 and a function corresponding to the above-mentioned generation unit 120.

Further, the form of the output based on the second image generated by the generation unit 120 is a form according to the type of the image processing apparatus to which the present invention is applied. For example, when the image processing device to which the present invention is applied is a printing device which is one aspect of the image forming device, the output control unit 130 records the second image generated by the generation unit 120 on a recording medium (paper or paper). The printing engine unit may be controlled so as to be formed on a metal plate or the like). For example, when the recording medium is made of an aluminum plate, the aluminum plate on which the second image is printed may be used as the image-receiving tool 30. Further, for example, when the recording medium is composed of paper or the like, even in the form of creating the image-imaging tool 30 by attaching the paper on which the second image is printed to the surface of the plate-shaped member. good.

Further, for example, the image processing device to which the present invention is applied may be a device that creates a plate-shaped mold (for example, a light-shielding mold) based on the second image generated by itself. In this case, a light source is arranged on the side opposite to the object (for example, the object to be imaged 30) with the device in between, and the light emitted from the light source toward the mold and the object produces a projected image according to the mold. It can also be in the form formed on the surface of the object. Further, it may be a plate that emits light by itself. It may be something like so-called electronic paper.

As described above, in the present embodiment, as the pattern image to be displayed on the image-imaging tool 30, a second image obtained by synthesizing a plurality of first images corresponding to a plurality of spatial frequencies and one-to-one is generated. .. As a result, even if the imaging distance of the pattern image changes, a spatial frequency pattern suitable for the changed distance appears as a feature, so that an advantageous effect that feature points can be stably extracted can be achieved.

Although the embodiments according to the present invention have been described above, the present invention is not limited to the above-described embodiments as they are, and at the implementation stage, the components can be modified and embodied within a range that does not deviate from the gist thereof. In addition, various inventions can be formed by an appropriate combination of the plurality of components disclosed in the above-described embodiment. For example, some components may be removed from all the components shown in the embodiments.

For example, as in the configuration described in Japanese Patent Application No. 2014-208897, while changing the distance (imaging distance) between the three-dimensional measuring device 20 and the image-receiving tool 30 on which the above-mentioned second image is displayed. An image taken by taking an image and taking an image at that distance for each of a plurality of distances (an image taken by the image-receiving tool 30 on which the second image is displayed) and the distance (known value). It is also possible to calculate the correction parameter based on. In short, the calculation unit 22 is an image captured by the image pickup unit 21 based on the captured image in which the object is reflected and the distance (known distance) between the image pickup unit 21 and the object. It may be in the form of calculating the correction parameter for correcting the above. For example, as shown in FIG. 9, the imaging unit 21 has a pair of first cameras 24a and second cameras 24b arranged with a predetermined distance (baseline length) B in the horizontal direction. Each of the first camera 24a and the second camera 24b has an image pickup lens 25 and an image pickup element (for example, a CCD image sensor) 26, and an image of a subject imaged by the image pickup lens 25 is on the image pickup surface of the image pickup element 26. It is imaged.

First, as shown in FIG. 10, the first camera 24a and the second camera 24b are set to a distance X1 (for example, 5 m) between the image pickup tool 30 on which the second image is displayed and the image pickup unit 21. Next, as shown in FIG. 11, the first camera is set to a distance X2 (for example, 1 m) between the image-receiving tool 30 on which the second image is displayed and the image pickup unit 21. The correction parameters can be calculated based on the images taken by the 24a and the second camera 24b and the images obtained at the respective distances.

FIG. 12 is a diagram showing an example of the function of the calculation unit 22 of this modification. As shown in FIG. 12, the calculation unit 22 includes a first acquisition unit 31, a first calculation unit 32, a second acquisition unit 33, a second calculation unit 34, and a third calculation unit 35.

The first acquisition unit 31 uses the first camera 24a to take an image of the image-receiving tool 30 (the image-imaging tool 30 on which the second image is displayed) separated from the image pickup unit 21 by a first distance X1 (5 m in this example). The captured image obtained by the above (may be referred to as "first captured image" in the following description) and the captured image obtained by capturing the imaged tool 30 with the second camera 24b (hereinafter referred to as "first captured image"). In the description, it is referred to as a "second captured image").

The first calculation unit 32 is based on the first distance X1 (5 m in this example), the first captured image, and the second captured image, and the second image is obtained for at least one of the first captured image and the second captured image. For each of a plurality of feature points corresponding to one-to-one with a plurality of feature points in the first image region indicating the image region (the central region of the captured image in this modification) in which the image-imaging tool 30 in which is displayed is reflected. First, the first correspondence data associated with the target coordinate values for obtaining the ideal parallax is calculated. When the image-receiving tool 30 is imaged by each of the first camera 24a and the second camera 24b, the position of the first camera 24a and the position of the second camera 24b are different, so that the image is reflected in the first image. There is a difference in appearance (there is a discrepancy) between the image-imaging tool 30 and the image-imaging tool 30 reflected in the second captured image. That is, the positions of the first captured image and the positions of the second captured image corresponding to the same feature points in the second image displayed on the image-receiving tool 30 are separated (shifted) by the visual difference.

The first calculation unit 32 has, for each of the plurality of feature points included (reflected) in the above-mentioned first image region of the first captured image, the coordinate values of the first captured image corresponding to the feature points and the second. The coordinate values of the captured image are specified and the ideal deviation amount is obtained. As a method for specifying each of the coordinate values of the first captured image and the coordinate values of the second captured image corresponding to the same feature point (corresponding point search method), various known techniques (for example, SAD (Sum Of)) Absolute Difference) and POC (phase-limited correlation, etc.) can be used. After that, the first calculation unit 32 sets an ideal coordinate value (ideal deviation) for each of the plurality of feature points included in the first image region. The position to obtain, hereinafter sometimes referred to as "target coordinate value") is obtained.

As described above, the first calculation unit 32 puts each of the first captured image and the second captured image in the above-mentioned first image region based on the first distance X1, the first captured image, and the second captured image. First correspondence data is calculated in which the target coordinate values for obtaining the ideal parallax are associated with each of the plurality of feature points included and the plurality of coordinate values corresponding to one-to-one (coordinate values of the captured image before correction). do. In this example, the first calculation unit 32 calculates the first corresponding data corresponding to the first captured image and the first corresponding data corresponding to the second captured image, but the present invention is not limited to this. For example, the first calculation unit 32 may be in a form of calculating only the first corresponding data corresponding to one of the first captured image and the second captured image.

In short, the first calculation unit 32 displays a second image for at least one of the first captured image and the second captured image based on the first distance X1, the first captured image, and the second captured image. The ideal misalignment for each of the plurality of feature points in the first image region in which the object (in this example, the image-imaging tool 30 on which the second image is displayed) is reflected and the plurality of coordinate values corresponding to one-to-one. It may be in the form of calculating the first correspondence data associated with the target coordinate values for obtaining.

The description of FIG. 12 will be continued. The second acquisition unit 33 uses the first camera 24a to take an image of the image-to-image tool 30 (the image-to-image tool 30 on which the second image is displayed) separated from the image pickup unit 21 by a second distance X2 (1 m in this example). The third captured image obtained by the above and the fourth captured image obtained by imaging the tool 30 to be imaged with the second camera 24b are acquired. In this modification, the above-mentioned first acquisition unit 31 and second acquisition unit 33 are described as separate bodies, but the present invention is not limited to this, and for example, the first acquisition unit 31 and the second acquisition unit 33 have the same constituent elements. It may be. That is, one of the first acquisition unit 31 and the second acquisition unit 33 may also have the function of the other.

The second calculation unit 34 is based on the second distance X2 (1 m in this example), the third captured image, and the fourth captured image, and the second image is obtained for at least one of the third captured image and the fourth captured image. A second image region (in this modification, other than the central region of the captured image) indicating an image region other than the portion corresponding to the above-mentioned first image region in the image region in which the image-receiving tool 30 in which is displayed is reflected. The second correspondence data in which the target coordinate values for obtaining the ideal parallax are associated with each of the plurality of feature points included in the image area) and the plurality of coordinate values corresponding to one-to-one is calculated.

The calculation method of the second correspondence data is basically the same as the calculation method of the first correspondence data described above. The second calculation unit 34 of this modification is included in the above-mentioned second image region for each of the third captured image and the fourth captured image based on the second distance X2, the third captured image, and the fourth captured image. Second correspondence data is generated in which the target coordinate values for obtaining the ideal parallax are associated with each of the plurality of coordinate values (coordinate values of the captured image before correction) corresponding to a plurality of feature points and one-to-one.

In this example, the second calculation unit 34 generates the second corresponding data corresponding to the third captured image and the second corresponding data corresponding to the fourth captured image, but the present invention is not limited to this, for example. The second calculation unit 34 may be in a form of generating only the second corresponding data corresponding to one of the third captured image and the fourth captured image. In short, the second calculation unit 34 displays the second image for at least one of the third captured image and the fourth captured image based on the second distance X2, the third captured image, and the fourth captured image. A plurality of image regions in the second image region showing an image region other than the portion corresponding to the first image region in the image region in which the object (in this example, the image-imaging tool 30 on which the second image is displayed) is reflected. The second correspondence data may be calculated in which the target coordinate values for obtaining the ideal parallax are associated with each of the plurality of coordinate values corresponding to the feature points on a one-to-one basis.

The description of FIG. 12 will be continued. The third calculation unit 35 calculates a correction parameter for correcting the captured image obtained by imaging by the first camera 24a or the second camera 24b based on the above-mentioned first correspondence data and the above-mentioned second correspondence data. do. More specifically, the third calculation unit 35 is the minimum from a plurality of sets in which the coordinate values and the target coordinate values included in each of the above-mentioned first correspondence data and the above-mentioned second correspondence data are associated with each other. By the square method, the coefficient of the correction formula representing the relationship between the coordinate value of the captured image obtained by the image pickup by the first camera 24a or the second camera 24b and the target coordinate value is calculated as the correction parameter. The correction formula can be expressed by, for example, the following formula 1. In Equation 1, x indicates the coordinate value in the left-right direction of the captured image before correction, and y indicates the coordinate value in the vertical direction of the captured image before correction. Further, x'indicates a target coordinate value in the left-right direction of the captured image, and y'indicates a target coordinate value in the vertical direction of the captured image. The correction formula is not limited to the affine transformation formula as in the following formula 1, and may be in any other form.

In this modification, the third calculation unit 35 includes a plurality of data sets (a set of a coordinate value before correction and a target coordinate value) included in the first corresponding data corresponding to the first captured image described above, and the above-mentioned. From a plurality of data sets included in the second corresponding data corresponding to the third captured image, the relationship between the coordinate value of the captured image obtained by imaging by the first camera 24a by the least squares method and the target coordinate value is shown. The coefficients a to f (which can be considered as the first correction parameter) of the correction formula (which may be referred to as the "first correction formula" in the following description) are calculated. Further, the third calculation unit 35 includes a plurality of data sets included in the first corresponding data corresponding to the above-mentioned second captured image and a plurality of data included in the second corresponding data corresponding to the above-mentioned fourth captured image. From the set, a correction formula that expresses the relationship between the coordinate value of the captured image obtained by imaging with the second camera 24b by the least squares method and the target coordinate value (when referred to as the "second correction formula" in the following description). There are) coefficients a to f (which can be considered to be the second correction parameter).

As described above, the third calculation unit 35 has the first correction formula expressing the relationship between the coordinate value of the captured image obtained by the image pickup by the first camera 24a and the target coordinate value, and the image pickup by the second camera 24b. A second correction formula representing the relationship between the coordinate values of the obtained captured image and the target value is obtained. Then, the measuring unit 23 obtains the first corrected image by correcting the coordinate values of the captured image obtained by the imaging by the first camera 24a by using the first correction formula. Further, the measuring unit 23 obtains the second corrected image by correcting the coordinate value of the captured image obtained by the image captured by the second camera 24b by using the second correction formula. Then, the parallax is calculated from the first corrected image and the second corrected image, and the parallax image (depth) showing the depth value (depth) of each pixel of the reference image of the first corrected image and the second corrected image is shown. Map) can be generated and output.

Also in this modification, by generating a second image so as to include the spatial frequency component corresponding to the distance X1 and the spatial frequency component corresponding to the distance X2, at any distance (X1, X2). Even when imaging is performed, feature points can be stably extracted. Further, in this modification, since the number of the image-receiving tool 30 used for calibration is only one, there is an advantage that the configuration can be simplified.

FIG. 13 is a flowchart showing an operation example of the calculation unit 22 of the modification.

First, the first acquisition unit 31 uses the first camera 24a to attach the image-imaging tool 30 (image-imaging tool 30 on which the second image is displayed) separated from the image pickup unit 21 by a first distance X1 (5 m in this example). The first image taken by taking an image and the second image obtained by taking an image of the tool 30 to be imaged by the second camera 24b are acquired (step S21). Next, the first calculation unit 32 describes the above-mentioned first correspondence data based on the first captured image and the second captured image acquired in step S21 and the known first distance X1 (5 m in this example). Is calculated (step S22). The calculation method of the first correspondence data is as described above.

Next, the second acquisition unit 33 uses the first camera 24a to move the image-imaging tool 30 (image-imaging tool 30 on which the second image is displayed) separated from the image pickup unit 21 by a second distance X2 (1 m in this example). The third captured image obtained by imaging with the second camera 24b and the fourth captured image obtained by imaging the tool 30 to be imaged with the second camera 24b are acquired (step S23). Next, the second calculation unit 34 will use the above-mentioned second correspondence data based on the third captured image and the fourth captured image acquired in step S23 and the known second distance X2 (1 m in this example). Is calculated (step S24). The calculation method of the second correspondence data is as described above.

Next, the third calculation unit 35 calculates the correction parameter based on the first correspondence data calculated in step S22 and the second correspondence data calculated in step S4 (step S25). As described above, the third calculation unit 35 is obtained by imaging with the first camera 11a based on the first corresponding data corresponding to the first captured image and the second corresponding data corresponding to the third captured image. The coefficient of the first correction formula representing the relationship between the coordinate value of the captured image and the target coordinate value is calculated as a correction parameter. Further, the third calculation unit 35 is the image captured by the second camera 11b based on the first corresponding data corresponding to the second captured image and the second corresponding data corresponding to the fourth captured image. The coefficient of the second correction formula representing the relationship between the coordinate value and the target coordinate value is calculated as a correction parameter.

Then, the measuring unit 23 corrects (calibrates) the captured image by using the correction parameter calculated in step S25 (step S26). As described above, the measuring unit 23 obtains the first corrected image by correcting the coordinate values of the captured image obtained by the imaging by the first camera 24a by using the first correction formula. Further, the measuring unit 23 obtains the second corrected image by correcting the coordinate value of the captured image obtained by the image captured by the second camera 24b by using the second correction formula.

The above calibration method (method for correcting the coordinate values of the captured image) is a calibration method for calibrating a stereo camera using a calibration pattern (including a chart, etc.), and the calibration pattern includes a plurality of spatial frequencies and one. Using a calibration pattern based on a second image generated by synthesizing a plurality of first images corresponding to one-to-one, it can be considered that the first imaging step, the second imaging step, and the calibration step are included. can. The first imaging step is a step of obtaining an image including a calibration pattern (corresponding to the first captured image and the second captured image in the above example) by imaging at the first distance. The second imaging step is a step of obtaining an image including a calibration pattern (corresponding to the third captured image and the fourth captured image in the above example) by imaging at a second distance different from the first distance. .. The calibration step is a step of calibrating the stereo camera based on the image obtained at the first distance and the image obtained at the second distance.

Further, the pattern projection device 10 in the above-described embodiment and modification is not only used for calibrating the stereo camera, but can also be used for calculating the parallax when the stereo camera is operating. For example, it can also be used in a system provided with a stereo camera that utilizes pattern irradiation, such as Japanese Patent Application Laid-Open No. 2007-101276. That is, it is a system that calculates parallax based on an irradiated pattern, and as a pattern, a second image generated by synthesizing a plurality of first images corresponding to a plurality of spatial frequencies and one-to-one. It may be a system that calculates the parallax based on the pattern using the pattern.

(program)
The program executed by the pattern projection device 10 of the above-described embodiment is a file in an installable format or an executable format, and is a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versaille Disk), or a USB. It may be configured to be recorded and provided on a computer-readable recording medium such as (Universal Serial Bus), or may be configured to be provided or distributed via a network such as the Internet. Further, various programs may be configured to be provided by incorporating them into a ROM or the like in advance.

1 System 10 Pattern projection device 20 Three-dimensional measuring device 21 Imaging unit 22 Calculation unit 23 Measuring unit 30 Imaging tool 101 CPU
102 ROM
103 RAM
104 Projection unit 105 I / F unit 106 Light source 107 Image forming unit 108 Lens unit 110 Acquisition unit 120 Generation unit 130 Output control unit

Japanese Unexamined Patent Publication No. 2011-118328

Claims (8)

A step of printing a pattern containing a second image, which is generated by synthesizing a plurality of first images, each containing a different spatial frequency.
The step of capturing the printed pattern with a stereo camera and
The step of calibrating the stereo camera based on the captured image obtained by capturing the pattern, and
A calibration method characterized by including. The second image is generated by finding and setting the average of the luminance values of the pixels in each of the plurality of first images for each of the plurality of pixels included in each of the plurality of first images. Be done,
The calibration method according to claim 1, wherein the calibration method is characterized by the above. In the step of calibrating the stereo camera, the stereo camera is calibrated based on the captured image obtained by capturing the pattern and the distance between the stereo camera and the pattern.
The calibration method according to claim 1 or 2, wherein the calibration method is characterized by the above. The plurality of first images are images corresponding to a plurality of distances between the stereo camera and the pattern in a one-to-one manner.
The calibration method according to any one of claims 1 to 3, wherein the calibration method is characterized by that. The first image contains an irregular pattern.
The calibration method according to any one of claims 1 to 4, wherein the calibration method is characterized by that. The printing is performed on paper.
The calibration method according to any one of claims 1 to 5 , wherein the calibration method is characterized by that. The printing is performed on a metal plate.
The calibration method according to any one of claims 1 to 5 , wherein the calibration method is characterized by that. In the calibration step, a correction parameter for correcting an image captured by the stereo camera is calculated.
The calibration method according to any one of claims 1 to 7 , wherein the calibration method is characterized by that.

Images