JP7712802B2 - Distance Measuring Device - Google Patents

Description

The present invention relates to a distance measuring device.

Conventionally, there are known distance measurement methods and devices that use multiple cameras, such as stereo cameras, to measure the distance from a given position to a subject, or the three-dimensional position of the subject, with high accuracy. In a stereo camera, a correlation value is calculated between a reference image block containing an image of a subject captured by one camera and a comparison image block containing an image of the same subject captured by the other camera, the degree of match between the images is determined based on the magnitude of the correlation value, and distance information to the subject is calculated using the principle of triangulation, with the difference in image positions on the image sensors of each camera when a match is determined as parallax. In this case, the degree of match between the images is determined to be the same on the premise that the images from the two cameras are the same, so the two cameras need to have characteristics as similar as possible.

International Publication No. 2011/010438 (Patent Document 1) discloses a "parallax detection device (3) that calculates the parallax occurring between a plurality of optical systems, the parallax detection device including a PSF identification unit (5) that corrects at least one of a plurality of images obtained from each of the plurality of optical systems so that the point spread of the plurality of optical systems becomes the same as the point spread of a predetermined optical system, and a parallax calculation unit (9) that calculates the parallax occurring between the plurality of optical systems using the image corrected by the PSF identification unit (5)."

In recent years, the recognition range of in-vehicle cameras has expanded to include a wider angle of view and a wider distance range, and in order to ensure safety over a wider area, there has been an increase in the use of wide-angle lens cameras with large distortion, and the number of cameras is also on the rise.

Therefore, in order to minimize the number of cameras that work together, it is important to consider stereo cameras (multiple viewpoints) with wide-angle, high-distortion lenses and sensors with different pixel pitches.

In addition, technology has been developed that utilizes optical systems and the projection method of lenses and mirrors to meet the demands for a wide angle of view and a wide distance range. JP 2021-012075 A (Patent Document 2) discloses a stereo camera 101 in which "first light R1 from a subject is reflected by an upper hyperbolic mirror 102, then further reflected by a lower hyperbolic mirror 103 (inner hyperbolic mirror 103A), and then incident on an image sensor 105 via an imaging optical system 104, and second light R2 from the subject, which is different from the first light R1, is reflected by a lower hyperbolic mirror 103 (outer hyperbolic mirror 103B) and incident on an image sensor 105 via an imaging optical system 104, and the first light R1 and the second light R2 are incident on different points on the image sensor 105 from different directions, thereby making it possible to calculate the distance to the subject, etc."

International Publication No. 2011/010438 JP 2021-012075 A

In distance measurement devices, if the image sensor sizes and pixel pitches of the image sensors of the two cameras used for distance measurement using the stereo camera principle are different, there is a problem that the pixel pitches (sampling pitch/sampling frequency) of the images from the two cameras will be different. In addition, if the focal lengths of the lenses of the two cameras are different, the size of the image formed on the sensor itself will be different, which also poses the problem that images with different sampling pitches must be compared to find the parallax.

At this time, the images on the two image sensors are first interpolated so that they have the same sampling pitch, but when converting an image with a coarse pixel pitch to an image with a fine pixel pitch, the size of the original image increases, making it pointless to use image sensors with a coarse pixel pitch, which would otherwise require less data processing. In addition, when converting an image with a fine pixel pitch to an image with a coarse pixel pitch, the size of the parallax image becomes smaller, not only reducing the image density of the parallax image, but also creating the problem that high-resolution images required for other recognition processes cannot be used.

In addition, according to the sampling theorem, an image with a coarse pixel pitch (low sampling frequency) only has signal components up to half the frequency of the low sampling frequency determined by the original coarse pixel pitch, while an image with a fine pixel pitch (high sampling frequency) has signal components up to half the frequency of the high sampling frequency determined by the fine pixel pitch. This creates the problem that even if the two images are converted to the same pixel pitch by interpolation processing, they do not become exactly the same image.

Furthermore, in stereo cameras that use optical systems with special projection characteristics, such as wide-angle lenses with large distortion or curved mirrors, the images are calculated as lattice points in a new image space through geometric transformation operations such as enlarging, reducing, and rotating the image, which are called geometric correction, resampling, or resampling.

The luminance values on the image newly generated by resampling are not necessarily located on the lattice points of the newly generated image space that are geometrically transformed from the original sampling points, but are found by interpolation between the geometrically transformed points of the original sampling points and the luminance values on the original sampling points. Since the amount of geometric transformation such as distortion differs depending on the point on the image, the sampling frequency conversion in sampling and resampling is not uniform in space but has a distribution on the imaging surface.

In a stereo camera, the block positions of the reference image and the search image are different, and although geometric correction is performed, there is a problem that even when the images are of the same subject, there are signal differences based on the sampling theorem, which reduces parallax performance such as parallax accuracy and parallax rate.

The object of the present invention is to provide a distance measuring device that can perform parallax detection with high accuracy even when using image sensors with different pixel pitches, lenses with different focal lengths, or lenses with distortion.

The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings.

The following is a brief overview of the most representative aspects of the present invention.

A distance measuring device according to one embodiment is a distance measuring device that measures distances by determining the degree of coincidence between a first image and a second image on an imaging element captured of the same object , and has a first correction processing unit that performs a process of adding interpolation points by resampling at least one of the first image and the second image , and a second correction processing unit that performs geometric correction processing on at least one of the resampled image data of the first image and the image data of the second image.

Problems, configurations, and advantages other than those described above will become clear from the description of the embodiment of the invention below.

The present invention provides a distance measuring device capable of performing disparity detection with high accuracy.

FIG. 1 is a diagram for explaining the principle of distance measurement in a stereo camera. FIG. 2A is a diagram for explaining a problem that occurs when imaging elements having different pixel pitches are used in a stereo camera. FIG. 2B is a diagram for explaining a problem that occurs when imaging elements having different pixel pitches are used in a stereo camera. FIG. 3A is a diagram illustrating an example of the configuration of a distance measuring device according to the first embodiment. FIG. 3B is a diagram illustrating an example of an image correction method after resampling in the first embodiment. FIG. 3C is a diagram illustrating another example of an image correction method after resampling in the first embodiment. FIG. 4A is a diagram for explaining calculation of the sum of absolute differences (SAD), which is the correlation value for each partial image region. FIG. 4B is a diagram showing the transition of the SAD when the image block is moved by one pixel at a time. FIG. 4C is a diagram illustrating sub-pixel disparity for equiangular line fitting. FIG. 5 is a diagram illustrating an example of the configuration of the distance measuring device according to the first embodiment. FIG. 6 is a flowchart showing the distance measuring method according to the first embodiment. FIG. 7 is a diagram showing an example of the configuration of a distance measurement device in the case where two lenses having different focal lengths are used in a stereo camera in the second embodiment. FIG. 8 is a diagram illustrating an example of the configuration of a distance measuring device according to the second embodiment. FIG. 9 is a diagram for explaining the influence of lens distortion. FIG. 10 is a diagram for explaining the influence of lens distortion. FIG. 11 is a diagram illustrating stereo matching in the third embodiment. FIG. 12 is a diagram illustrating an example of the configuration of a distance measuring device according to the third embodiment.

The following describes embodiments of the present invention with reference to the drawings. The embodiments and examples are merely examples for explaining the present invention, and have been omitted or simplified as appropriate for clarity of explanation. The present invention can also be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

The position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings.

When there are multiple components with the same or similar functions, they may be described using the same reference numerals with different subscripts. Also, when there is no need to distinguish between these multiple components, the subscripts may be omitted.

In all drawings used to explain the embodiments, the same components or parts are generally given the same reference numerals, and repeated explanations may be omitted. The drawings may be shown more diagrammatically than the actual embodiment in order to make the explanation clearer, but they are merely examples and do not limit the interpretation of the present invention.

The distance measurement device according to the first embodiment of the present invention is a distance measurement device 6 having a stereo camera SC composed of image pickup elements 30a and 30b with different pixel pitches, as shown in FIG. 3A described later.

First, an example of calculating the distance to a subject using a stereo distance measurement method will be described with reference to Figure 1. Figure 1 is a diagram illustrating an example of calculating the distance to a subject 4 using a stereo distance measurement method when two cameras 2, camera 2a and camera 2b, are used. Cameras 2a and 2b, which constitute the stereo camera SC of distance measurement device (also called distance measurement device) 6r, are arranged side by side with a distance B (baseline length) between them and their optical axes 3a and 3b parallel to each other.

The light beam emitted from the subject 4 passes through the optical center of the lens 5a of the camera 2a and the lens 5b of the camera 2b, and forms an image on the image sensor 1a of the camera 2a and the image sensor 1b of the camera 2b. The optical axis 3a and the optical axis 3b represent the optical axis 3 of the lens 5a and the lens 5b. At this time, the subject 4 is formed on the image sensor 1a at a position distance La from the optical axis 3a, and on the image sensor 1b at a position distance Lb from the optical axis 3b, and a parallax L (= La - Lb) is generated between the cameras 2a and 2b. This parallax L changes depending on the magnitude of the distance D from the cameras 2a and 2b of the distance measuring device 6r to the subject 4. If the focal lengths of the cameras 2a and 2b are both focal length f, the distance D to the subject 4 is expressed by the following (Equation 1).

D=f(B/L) (Formula 1)
The image sensors 1a and 1b are usually composed of light receiving elements arranged at points on a grid with equal pitch and semiconductor devices such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). Therefore, the luminance signal of the subject image is sampled at points that are discretized at equal intervals on a two-dimensional plane.

Next, using Figures 2A and 2B, we will explain the issues that arise when imaging elements with different pixel pitches are used in a stereo camera. In Figures 2A and 2B, we will explain the case where imaging elements with different pixel pitches are imaging element 10a with a coarse pixel pitch and imaging element 10b with a finer pixel pitch than imaging element 10a. Figure 2A is a diagram showing the state in which the same image is formed on a two-dimensional plane when imaging elements with different pixel pitches are used, and these are sampled and resampled. Figure 2B is a diagram explaining the state of Figure 2A by showing it one-dimensionally for simplicity.

In FIG. 2A, two-dimensional plane 22 shows a state in which an image formed on the two-dimensional plane of image sensor 10a, which has a coarse pixel pitch, has been sampled. Two-dimensional plane 21 shows a state in which an image formed on the two-dimensional plane of image sensor 10b, which has a finer pixel pitch than image sensor 10a, has been sampled. It is assumed that the same image is formed on the two-dimensional plane of image sensor 10a and the two-dimensional plane of image sensor 10b. Two-dimensional plane 21r shows a state in which sampled two-dimensional plane 21 has been resampled. Two-dimensional plane 24 shows a state in which sampled two-dimensional plane 22 has been resampled. Through resampling, the coarse pixel pitch of two-dimensional plane 22 is converted to match the fine pixel pitch of two-dimensional plane 21r (21), and two-dimensional plane 24 with a fine pixel pitch is constructed.

Figure 2B shows a one-dimensional sampling state 26 of an image sensor 10a with a coarse pixel pitch, and a one-dimensional sampling state 25 of an image sensor 10b with a fine pixel pitch. Also shown are a resampling state 25r obtained by resampling sampling state 25, and a resampling state 28 obtained by resampling sampling state 26. In resampling state 28, interpolation points indicated by black circles are added by resampling.

In sampling states 25 and 26 in FIG. 2B, as shown by the dotted line as "image on image sensor", the image formed on the surface of image sensor 10a, 10b by the lens (for example, lenses 5a, 5b in FIG. 1) has a continuous light distribution, but these are sampled according to the pixel pitch of image sensor 10a, 10b. According to the sampling theorem, the sampled signal is assumed to be composed only of frequency components less than half the sampling frequency, so image sensor 10b with fine pixel pitch has higher spatial frequency components and maintains a shape close to the original continuous light distribution. On the other hand, image sensor 10a with coarse pixel pitch cannot have high frequency components, so the signal deteriorates from the original continuous light distribution shape (signal deterioration occurs due to sampling).

In stereo matching, image blocks are compared using a template matching technique. However, since the pixel pitches of the image sensor 10a and the image sensor 10b are different, it is necessary to convert them to the same pixel pitch by an interpolation process or a resampling process. In a distance measurement device system in which image sensors (10a, 10b) with different pixel pitches are mixed, the reason for adopting an image sensor (10b) with a fine pixel pitch is that an image with a fine pixel pitch is usually required, so the sampling frequency of the image sensor (10b) with a fine pixel pitch is often used for the resampling. However, as described above, the signal of the image sensor (10a) with a coarse pixel pitch itself only has signal components with a frequency half the sampling frequency determined by the coarse pixel pitch, so the image (28) after resampling of the image sensor (10a) with a coarse pixel pitch is a degraded image with few frequency components.

The resampling process can be performed on all images at once, or can be performed each time a reference image is selected and a search image is selected during stereo matching. The latter method requires less storage space.

In template matching, the correlation value between two images is calculated to determine whether the two images match, so if the errors caused by the conversion in the resampling process are different, the correlation value will decrease and the disparity performance will deteriorate.

Next, a method of correcting an image after resampling in the first embodiment will be described with reference to Figures 3A, 3B, and 3C. Figure 3A is a diagram showing an example of the configuration of a distance measuring device in the first embodiment.
Fig. 3B is a diagram showing an example of an image correction method after resampling in the embodiment 1. Fig. 3C is a diagram showing another example of an image correction method after resampling in the embodiment 1.

The distance measurement device 6 shown in FIG. 3A differs from the distance measurement device 6r in FIG. 1 in that the distance measurement device 6 employs a stereo camera composed of image pickup elements 30a and 30b with different pixel pitches. The other configurations of the distance measurement device 6 shown in FIG. 3A are the same as the other configurations of the distance measurement device 6r in FIG. 1.

As shown in FIG. 3A, the cameras 2a and 2b constituting the stereo camera SC of the distance measurement device (also called the distance measuring device) 6 are arranged side by side at a distance B (baseline length) from each other so that the optical axes 3a and 3b are parallel. A light ray emitted from the subject 4 passes through the optical center of the lens 5a of the camera 2a and the lens 5b of the camera 2b, and forms an image on the image sensor 30a of the camera 2a and the image sensor 30b of the camera 2b. The optical axis 3a and the optical axis 3b represent the optical axis 3 of the lens 5a and the lens 5b. At this time, the subject 4 is imaged on the image sensor 30a at a position distance La from the optical axis 3a, and on the image sensor 30b at a position distance Lb from the optical axis 3b, and a parallax L (=La-Lb) is generated between the cameras 2a and 2b. This parallax L changes depending on the magnitude of the distance D from the cameras 2a and 2b of the distance measurement device 6 to the subject 4. If the focal length of both cameras 2a and 2b is f, then the distance D to the subject 4 is expressed by the formula 1 described above.

In the resampling (also called resampling) described in Figures 2A and 2B, since it is known what frequency components of the image signal are degraded by the resampling, in this embodiment, a correction process is performed to correct the difference in signal degradation due to resampling. In the example shown in Figure 3B, a correction process is performed to restore the signal drop of the image sensor 30a with a coarse pixel pitch by super-resolution processing. Also, in the example shown in Figure 3C, a correction process is performed to block high frequency components such as decimation filter (DEF) processing of image data from the image sensor 30b with a fine pixel pitch.

Figure 3B shows a one-dimensional sampling state 32 of an image sensor 30a with a coarse pixel pitch, and a one-dimensional sampling state 31 of an image sensor 30b with a fine pixel pitch. Then, resampling is performed on sampling state 32 and sampling state 31. The sampling frequency used here is the sampling frequency of the image sensor (30b) with a fine pixel pitch. Resampling state 33 obtained by resampling sampling state 32 is shown. Resampling state 31r obtained by resampling sampling state 31 is shown. This aligns the pixel pitch between resampling state 33 and resampling state 31r.

In resampling state 33, interpolation points indicated by black circles are added by resampling. On the other hand, resampling state 31r obtained by resampling sampling state 31 is basically the same.

Then, super-resolution processing is performed on the image data in the resampling state 33. In the super-resolution processing, a correction process is performed to restore the image data due to the signal drop of the image sensor 30a with a coarse pixel pitch, and the resampling state 33 is converted to the image data state 34 after the super-resolution processing. This corrects the error between the resampling state 31r and the image data state 34 after the super-resolution processing.

In FIG. 3C, like FIG. 3B, one-dimensional sampling state 32 of image sensor 30a with a coarse pixel pitch, one-dimensional sampling state 31 of image sensor 30b with a fine pixel pitch, and resampling state 33 obtained by resampling sampling state 32 are shown. Resampling state 31r obtained by resampling sampling state 31 is also shown. The sampling frequency used here is the sampling frequency of the image sensor (30b) with a fine pixel pitch. This aligns the pixel pitch between resampling state 33 and resampling state 31r.

After resampling, decimation filter (DEF) processing is performed on the image data in the resampling state 31r. In the decimation filter (DEF) processing, a correction process is performed to block the high frequency components of the image data from the image sensor 30b, which has a fine pixel pitch, and the resampling state 31r is converted to the image data correction state 36 after decimation filter (DEF) processing. This corrects the error between the image data correction state 36 after decimation filter (DEF) processing and the resampling state 33.

In other words, after resampling, correction processing using super-resolution processing (restoration processing for signal degradation) and correction processing using decimation filtering (blocking of high frequency components) are performed to correct the difference in signal degradation between the left and right images (31r and 33) that occurs during resampling. This allows the left and right image blocks compared during stereo matching processing to be corrected for errors in the left and right image blocks caused by left and right sampling of the block image portion and interpolation by resampling, and allows the parallax L to be calculated accurately by stereo matching processing, thereby providing a distance measuring device 6 that can obtain the distance D to the subject 4 with high accuracy.

After the correction process using super-resolution processing (restoration of signal loss) and the correction process using decimation filtering (blocking of high-frequency components), the distance measurement device 6 calculates the parallax L as described in FIG. 3A.

In the following description, image data after correction processing using super-resolution processing (restoration processing of signal loss) or correction processing using decimation filter processing (cutting off high-frequency components) (for example, states 31r and 34 in FIG. 3B, and states 36 and 33 in FIG. 3C) will be referred to as the image after resampling, or the image after resampling processing. In addition, correction processing using super-resolution processing (restoration processing of signal loss) or correction processing using decimation filter processing (cutting off high-frequency components) will be referred to as resampling processing.

As mentioned above, the resampling process may be performed on all images at once, or may be performed each time a reference image and a search image are selected during stereo matching. Below, as a representative example, we will explain an example in which the stereo matching process is performed after the resampling process is performed on the entire screen at once.

The stereo matching process of this embodiment will be described with reference to Figures 4A, 4B, and 4C. Figure 4A is a diagram for explaining the calculation of the SAD, which is a correlation value for each partial image region. Figure 4B is a diagram showing the transition of the SAD when the image block is moved by one pixel at a time. Figure 4C is a diagram for explaining the subpixel disparity of equiangular straight line fitting.

As shown in FIG. 4A, the SAD (Sum of Absolute Difference) which is the correlation value for each partial image area (block area, template image) is calculated for the resampled image from the image sensor 30a and the resampled image from the image sensor 30b. The SAD is shown in Equation 2. One block area is, for example, an image area of 8 pixels x 8 pixels.

The distance measurement device 6 then uses the calculated correlation value to calculate the image shift on the image at the pixel pitch of the image after resampling of the image sensor 30a and the image sensor 30b, that is, the parallax L. Note that SAD is an example of a correlation value, and the distance measurement device 6 can also use commonly known Sum of Squared Difference (SSD) or Normalized Cross-Correlation (NCC) as a correlation value.

First, a partial region of the reference image REF is selected from the resampled image from the image sensor 30a. Next, a partial region of the comparison image CMP is selected from the resampled image from the image sensor 30b, and correlation values such as SAD are calculated for each partial image region (REF and CMP). Next, the partial region of the comparison image CMP is shifted from the resampled image from the image sensor 30b by one pixel pitch of the pixel pitch after resampling, for example, to the right, and correlation values such as SAD of the partial image region (REF and CMP) after the shift are calculated again. Figure 4B shows the transition of SAD when the image block is moved by one pixel at a time. The pixel shift amount (shift amount) SG at which the SAD is minimum becomes the parallax (L) in one pixel unit. Here, SG is the shift amount (integer disparity) at which the SAD is smallest, S(0) is the correlation value (SAD) at the shift amount SG at which the SAD is smallest, and the SADs at the adjacent shift amounts are S(-1) and S(1).

In the case of Figures 4A and 4B, the detection accuracy of disparity (L) is one pixel, so disparity cannot be obtained with decimal point pixel accuracy (hereafter referred to as sub-pixel disparity).

A method of estimating sub-pixel level disparity has been proposed as a method for determining distance measurement accuracy (ranging accuracy), i.e., disparity detection resolution, not in units of one pixel, but with higher accuracy. For example, in a sub-pixel disparity estimation method called equiangular linear fitting, as shown in FIG. 4C, the actual disparity is estimated at the sub-pixel level by first-order linear interpolation by assuming that the absolute value of the slope of the SAD transition is the same on the left and right with reference to the actual disparity. The sub-pixel disparity calculation formula for equiangular linear fitting, i.e., the interpolation formula, is shown in (Formula 3). Here, δ is the sub-pixel disparity, SG is the shift amount (integer disparity) at which the SAD is minimum, S(0) is the correlation value (SAD) at the shift amount SG at which the SAD is minimum, and S(-1) and S(1) are the SADs at the adjacent shift amounts.

By multiplying the parallax calculated by (Equation 3) by the pixel pitch size, the parallax L in (Equation 1) is obtained, and the distance D to the subject 4 is calculated.

A configuration example of the distance measurement device 6 will be described with reference to FIG. 5. FIG. 5 is a diagram showing a configuration example of the distance measurement device according to the first embodiment. Note that the distance measurement device 6 in FIG. 5 describes an example in which the correction process (decimation filter process) described in FIG. 3C is performed on the search image (comparison image) for each selection of the reference image and the search image (comparison image) during stereo matching processing.

The distance measurement device 6 includes a first camera 2a, a second camera 2b, a first resampling circuit 40a, a second resampling circuit 40b, and a parallax calculation circuit 50.

The first camera 2a includes a first optical system including a lens 5a, and a first imaging element 30a. The second camera 2b includes a second optical system including a lens 5b, and a second imaging element 30b. The pixel pitch of the first imaging element 30a is coarser than the pixel pitch of the second imaging element 30b. In other words, the pixel pitch of the second imaging element 30b is finer than the pixel pitch of the first imaging element 30a. In this example, the focal lengths of the first camera 2a and the second camera 2b are the same, and both are focal length f.

The first resampling circuit 40a is provided to resample the image formed on the two-dimensional plane of the image sensor 30a after sampling by the image sensor 30a (see resampling state 33 in FIG. 3C). The second resampling circuit 40b is provided to resample the image formed on the two-dimensional plane of the image sensor 30b after sampling by the image sensor 30b (see resampling state 31r in FIG. 3C). The sampling frequency of the image sensor (30b) with a fine pixel pitch is used here. This aligns the pixel pitch of the image (33) from the first image sensor 30a and the pixel pitch of the image (31r) from the second image sensor 30b to be the same pixel pitch. Here, the image from the first image sensor 30a and the image from the second image sensor 30b with the same pixel pitch are referred to as the image after resampling.

The parallax calculation circuit 50 includes a reference position detection circuit (also called a standard image position detection circuit) 51, a comparison image position detection circuit 52, a comparison image correction circuit 53, and a parallax detection circuit 54. The image (33) from the first imaging element 30a and the image (31r) from the second imaging element 30b, which have the same pixel pitch, are input to the reference position detection circuit 51 and the comparison image position detection circuit 52, respectively. The reference position detection circuit 51 detects or selects a partial area of the reference image REF from the resampled image (31r) from the first imaging element 30a. The comparison image position detection circuit 52 detects or selects a partial area of the comparison image CMP from the resampled image (33) from the imaging element 30b.

The comparison image correction circuit 53 calculates the amount of correction resulting from the difference between the position of the reference image REF and the position of the comparison image CMP, and corrects the comparison image CMP. Here, a correction process using a decimation filter (blocking of high frequency components) is performed on the comparison image CMP.

The parallax detection circuit 54 detects the parallax L between the reference image REF and the corrected comparison image CMP. This allows the distance measurement device 6 to calculate the accurate distance D to the subject 4 from the detected parallax L using (Equation 1).

When performing the correction process (restoration process for signal degradation) using the super-resolution process in FIG. 3B, the comparison image correction circuit 53 is deleted from FIG. 5, and instead, a reference image correction circuit is provided between the output of the reference position detection circuit 51 and one input of the disparity detection circuit 54. In this case, the output of the comparison image position detection circuit 52 is connected to the other input of the disparity detection circuit 54.

If super-resolution correction (restoration of signal loss) and decimation filter correction (cutting off high frequency components) are performed on all images at once before stereo matching, as shown in Figures 3B and 3C, this can be done as follows.

In the case of FIG. 3B, a correction circuit that performs correction processing using super-resolution processing is provided between the first resampling circuit 40a and the reference position detection circuit 51.

In the case of FIG. 3C, a correction circuit that performs correction processing using decimation filtering is provided between the second resampling circuit 40b and the comparison image position detection circuit 52.

An example of a distance measurement method of the distance measurement device 6 in FIG. 5 will be described with reference to FIG. 6. FIG. 6 is a flow diagram showing a distance measurement method according to the first embodiment.

Step S1:
An image of the subject 4 is acquired using a first imaging element 30a of the first camera 2a. Also, an image of the subject 4 is acquired using a second imaging element 30b of the second camera 2b. The pixel pitch of the first imaging element 30a is coarser than the pixel pitch of the second imaging element 30b.

Step S2:
A reference image block position is set in the image of the subject 4 captured by the first imaging element 30a of the first camera 2a. The reference image block position is set by the reference position detection circuit 51, for example.

Step S3:
In order to align the left and right pixel pitches, resampling is performed using the first resampling circuit 40a and the second resampling circuit 40b, whereby the pixel pitch of the image (33) from the first imaging element 30a and the pixel pitch of the image (31r) from the second imaging element 30b are aligned to be the same pixel pitch.

Step S4:
In each resampled image, the reference image position and the comparison image position are detected. The reference position detection circuit 51 detects a partial area of the reference image REF from the resampled image (33) from the first imaging element 30a. The comparison image position detection circuit 52 detects a partial area of the comparison image CMP from the resampled image (31r) from the imaging element 30b.

Step S5:
The comparison image correction circuit 53 calculates a correction amount due to the difference between the reference image position and the comparison image position detected in step S4. Here, the correction amount is calculated for the resampled image (31r) from the imaging element 30b, for performing correction processing by decimation filter processing (cutting off high frequency components).

Step S6:
The comparison image correction circuit 53 performs correction processing by decimation filtering on the resampled image (31r) from the imaging element 30b using the correction amount calculated in step S5.

Step S7:
The parallax detection circuit 54 detects the parallax L between the reference image REF and the corrected comparison image CMP.

Step S8:
The parallax detection circuit 54 determines whether the block position of the reference image REF used in step S7 is the final reference image block position. If the block position of the reference image REF used in step S7 is the final reference image block position (Yes), the flow of the distance measurement method ends. On the other hand, if the block position of the reference image REF used in step S7 is not the final reference image block position (No), the process proceeds to step S2. After that, steps S2 to S7 are repeatedly executed until the block position of the reference image REF becomes the final reference image block position.

According to the first embodiment, one or more of the following effects can be obtained.

1) Even if imaging elements with different pixel pitches are used in a stereo camera, the pixel pitch of the left and right images can be made the same by resampling.

2) After resampling, correction processing using super-resolution processing (restoration processing of signal degradation) or correction processing using decimation filtering (blocking of high frequency components) is performed, making it possible to correct the difference in signal degradation between the left and right images (31 and 33) that occurs during resampling.

3) By using 2) above, it is possible to correct errors in the left and right image blocks compared during stereo matching processing, which are caused by left and right sampling of the block image portions and interpolation by resampling.

4) Even when imaging elements with different pixel pitches are used in a stereo camera, the parallax L can be accurately calculated by stereo matching processing, and a distance measuring device 6 can be provided that can obtain the distance D to the subject 4 with high accuracy.

The second embodiment (Example 2) of the present invention relates to a distance measurement device using different lenses. Fig. 7 is a diagram showing an example of the configuration of a distance measurement device in Example 2 when two lenses with different focal lengths are used in a stereo camera. Fig. 8 is a diagram showing an example of the configuration of a distance measurement device in Example 2.

Figure 7 shows an example of the configuration of a distance measurement device 6a in which the two cameras 200a and 200b that make up the stereo camera SC have lenses 70a and 70b with different focal lengths. Here, a stereo camera SC is shown that uses a camera 200a with a lens 70a with a long focal length fa and a camera 200b with a lens 70b with a short focal length fb.

The size of the image (Sa) of the same subject 4 formed on image sensor 1a through lens 70a is different from the size of the image (Sb) formed on image sensor 1b through lens 70b (Sb<Sa). These are sampled as image data by image sensor 1a and image sensor 1b, which have the same pixel pitch.

These images cannot be used for stereo matching as is, so they must be enlarged or reduced to have the same pixel pitch and image size. When converting to have the same image size, there are many sampling points in camera 200b, which has a larger original image, and few sampling points in camera 200a, which has a smaller original image, resulting in a difference in the effective sampling frequency and a difference in image characteristics due to resampling.

Therefore, similar to the first embodiment, by implementing a correction process that compensates for differences in image characteristics due to resampling, it is possible to improve parallax performance, such as parallax accuracy and parallax rate.

8, the distance measurement device 6a has a first camera 200a and a second camera 200b as a stereo camera SC. The first camera 200a includes a first optical system including a lens 70a with a long focal length fa, and an image sensor 1a. The second camera 200b includes a second optical system including a lens 70b with a short focal length fb (fb<fa), and an image sensor 1b.

Since the image size (Sa) of the image formed on the image sensor 1a is different from the image size (Sb) of the image formed on the image sensor 1b, the first resampling circuit 41a and the second resampling circuit 41b are configured to enlarge or reduce the image so that the images have the same size with the same pixel pitch, and then perform resampling.

The comparison image correction circuit 53a performs correction processing on the comparison image to compensate for differences in image characteristics caused by resampling.

The third embodiment (Example 3) of the present invention will be described in detail below with reference to the drawings. The third embodiment (Example 3) of the present invention relates to a distance measurement device using a lens with distortion. FIG. 9 is a diagram for explaining the influence of lens distortion. FIG. 10 is a diagram for explaining the influence of lens distortion. FIG. 11 is a diagram for explaining stereo matching in Example 3.

Figure 9 shows the dot pattern DA (shown by white circles) that should be captured when a distorted lens is used, and the actual dot pattern DB (shown by black circles). The square lattice shape created by the actual dot pattern DB has small distortion in the center CEN of the image, but the square shape becomes more distorted as it approaches the edge PER of the image. Also, at the edge PER of the image, the area of the rectangle formed by the four lattice points is smaller than the area of the rectangle in the center CEN. In other words, a phenomenon occurs in which the image of the same subject 4 is smaller when it is imaged on the periphery of the image sensor than when it is imaged in the center of the image sensor (size of image of subject 4 imaged in the center of the image sensor > size of image of subject 4 imaged on the periphery of the image sensor).

The error caused by this phenomenon will be explained using Figure 10. Figure 10 is a one-dimensional schematic representation of a highly distorted image that is located at an angle of view away from the optical axis and is formed on the peripheral portion PER of the image sensor, and an image that is formed on the central portion CEN of the image sensor near the optical axis. It is desirable that the image that should be captured (referred to as the original image) be the same on both the peripheral portion PER and the central portion CEN, but the peripheral portion PER is significantly affected by lens distortion, and is actually formed as a small image (referred to as the actual image) (see state 61). The central portion CEN is less affected by lens distortion, so the original image and the actual image are formed in almost the same way (see state 62).

As shown in states 63 and 64, the pixel pitch of the image sensor is constant, so the small image in the peripheral area PER and the large image in the central area CEN are sampled at the same sampling frequency. Even if the images are from the same subject 4, the number of sampling points in the peripheral area PER, where distortion is large, is small (3 points in state 63), while the number of sampling points in the central area CEN, where distortion is small, is large (7 points in state 64).

When correcting lens distortion, it is necessary to perform image enlargement/reduction correction according to the amount of distortion (in this case, enlargement correction). The peripheral area PER, which has a large amount of distortion, is reduced by a large amount, so it needs to be enlarged by a large amount (see state 65), while the central area CEN, which has almost no distortion, is enlarged to almost the same size (see state 66). However, the number of sampling points for the image from the same subject 4 is low for the peripheral area PER (3 points) and high for the central area CEN (7 points), so resampling is required by interpolation processing to increase the number of pixel points in the peripheral area (see states 67 and 68). At this time, the image after conversion of the peripheral area PER is significantly degraded.

In stereo matching using lenses with large distortion or optical systems using special projection methods, problems can arise even when cameras use lenses with the same focal length and image sensors with the same pixel pitch, because the positions of the reference image block and the search image block are different, and distortion correction differs at each position.

The stereo matching in the third embodiment will be explained with reference to FIG. 11. As an example, FIG. 11 explains stereo matching in the case of using a stereo camera including two image sensors (1a, 1b) as in the first and second embodiments. Note that it will be obvious to those skilled in the art that the third embodiment can also be applied to the case of a single monocular camera including one image sensor.

As shown in FIG. 11, when the image positions of the reference block (partial region of the reference image) REF and the search block (partial region of the comparison image) CMP (CMP1) are the same (i.e., when the pixel shift amount is 0), the lens distortions of the reference block REF and the search block CMP1 are the same, so the same resampling process is performed. However, as the image shift amount increases, the difference between the position of the reference block REF and the positions of the search blocks CMP2 and CMP3 increases, and the resampling process of the reference block REF and the resampling process of the search block become different. Therefore, during stereo matching processing, the image position of the reference image block REF and the image position of the comparison image block CMP being searched are detected, and only the resampling error caused by the image position difference is corrected, so that the difference in the correction amount due to resampling can be minimized.

In other words, a distance measuring device using a distorted lens is provided with a detection circuit that detects the image position of the reference image block REF and the image position of the comparison image block CMP being searched for, and a correction circuit that corrects only the resampling error caused by the image position difference.

Next, a configuration example of a distance measurement device using a monocular camera with a distorted lens in Example 3 will be described with reference to FIG. 12. FIG. 12 is a diagram showing a configuration example of a distance measurement device in Example 3.

As shown in FIG. 12, the distance measurement device 6b includes a camera 300, a viewpoint separation circuit 90, a first viewpoint image capture unit resampling circuit 41c, a second viewpoint image capture unit resampling circuit 41d, and a parallax calculation circuit 50. The parallax calculation circuit 50 includes a reference position detection circuit (reference image position detection circuit) 51, a comparison image position detection circuit 52, a comparison image correction circuit 53b, a parallax detection circuit 54, and a reference image correction circuit 55.

The camera 300 includes an optical system that includes a distorted lens 80, and an image sensor 1c. The image formed on the two-dimensional plane of the image sensor 1c via the lens 80 is sampled by the image sensor 1c.

The viewpoint separation circuit 90 selects a first viewpoint shooting area and a second viewpoint shooting area in the image sampled by the image sensor 1c. The viewpoint separation circuit 90 supplies the first viewpoint image data corresponding to the first viewpoint shooting area to a resampling circuit 41c of the first viewpoint shooting area. The viewpoint separation circuit 90 also supplies the second viewpoint image data corresponding to the second viewpoint shooting area to a resampling circuit 41d of the second viewpoint shooting area.

The resampling circuit 41c for the first viewpoint shooting area detects the position of the first viewpoint image data on the imaging element 1c, and performs resampling processing of the first viewpoint image data based on that position. In order to correct distortion of the lens 80, the resampling circuit 41c performs geometric transformations such as enlargement, reduction, and rotation, and interpolation processing based on the position of the first viewpoint image data on the imaging element 1c during the resampling processing.

The resampling circuit 41d for the second viewpoint shooting area detects the position of the second viewpoint image data on the imaging element 1c, and performs resampling processing of the second viewpoint image data based on that position. In order to correct distortion of the lens 80, the resampling circuit 41d performs geometric transformations such as enlargement, reduction, and rotation, and interpolation processing based on the position of the second viewpoint image data on the imaging element 1c during resampling processing. This makes the pixel pitch of the first viewpoint image data after resampling the same as that of the second viewpoint image data after resampling. The first viewpoint image data after resampling is supplied to the reference position detection circuit 51, and the second viewpoint image data after resampling is supplied to the comparison position detection circuit 52.

The reference position detection circuit 51 detects or selects a partial area of the reference image REF from the first viewpoint image data after resampling. The comparison image position detection circuit 52 detects or selects a partial area of the comparison image CMP from the second viewpoint image data after resampling.

The reference image correction circuit 55 calculates the amount of correction due to the position of the reference image REF on the first viewpoint image data after resampling, and corrects the reference image REF. In other words, the reference image correction circuit 55 performs a correction process to correct the signal degradation of the image of the first viewpoint image data that occurs due to the resampling process.

The comparison image correction circuit 53b calculates the amount of correction due to the position of the comparison image CMP on the second viewpoint image data after resampling, and corrects the comparison image CMP. In other words, the comparison image correction circuit 53b performs a correction process to correct the signal degradation of the image of the second viewpoint image data that occurs due to the resampling process.

The parallax detection circuit 54 detects the parallax L from the reference image REF corrected by the reference image correction circuit 55 and the comparison image CMP corrected by the comparison image correction circuit 53b. This allows the distance measurement device 6b, which uses a distorted lens 80, to calculate the accurate distance D to the subject 4 from the detected parallax L using (Equation 1).

The reference image correction circuit 55 can be omitted. In this case, the comparison image correction circuit 53b is configured to calculate a correction amount resulting from the difference between the position of the reference image REF on the first viewpoint image data after resampling and the position of the comparison image CMP on the second viewpoint image data after resampling, and to perform a correction process to correct the comparison image CMP. The disparity detection circuit 54 detects the disparity L from the reference image REF and the comparison image CMP corrected by the comparison image correction circuit 53b.

As a result, the distance measurement device 6b using the distorted lens 80 can calculate the accurate distance D to the subject 4 from the detected parallax L using (Equation 1).

The configuration examples of the distance measuring device in Examples 1, 2, and 3 can be summarized as follows:

1) A distance measuring device (6, 6a, 6b) that detects image positions on an image sensor of at least two images (REF, CMP) of an identical object (subject 4) by judging the degree of coincidence between the two images, and performs distance measurement (measurement of distance),
The image on the imaging element is sampled at the pixel pitch of the imaging element (30a, 30b, 1a, 1b, 1c),
When performing a coincidence determination of the images (REF, CMP) at the time of detecting the image positions, geometric correction (Example 1: super-resolution processing, decimation filter processing, Example 2: enlargement conversion and reduction conversion, Example 3: geometric conversion such as enlargement, reduction, and rotation) and resampling processing (interpolation processing) are performed,
The image signal degradation occurring during the resampling process is corrected.

2) In the distance measuring device of 1) above (see Example 3),
The two images (reference image REF and comparison image CMP) for determining the degree of coincidence of the above images are stored on one image sensor (1c).

3) In the distance measuring device of 1) above (see Example 1),
2. The distance measuring device of claim 1,
The two images (reference image REF and comparison image CMP) for which the degree of coincidence of the above images is to be determined are stored on two image pickup devices (30a, 30b) having different pixel pitches.

4) In the distance measuring device of 1) above (see Example 2),
The two images (reference image REF and comparison image CMP) for determining the degree of coincidence of the above images are formed by an optical system consisting of lenses (70a, 70b) having different focal lengths.

5) In the distance measuring device of 1) above (Examples 1, 2, and 3),
When performing a coincidence determination of the images (reference image REF, comparison image CMP) at the time of the image position detection, a geometric correction and a resampling process are performed on the image data on the imaging element required for determining at least one of the images;
A distance measuring device that compensates for image signal degradation that occurs during the resampling process.

6) In the distance measuring device of 1) above (Example 3),
Detecting the positions of the two images (reference image REF and comparison image CMP) on the image sensor for which the degree of match is to be determined;
Image signal degradation caused by geometric correction and resampling processing at each of the two image positions is corrected.

In other words, the distance measurement device of the present invention has a geometric correction unit that performs, during or before the corresponding point search of the stereo camera, a geometric correction unit that makes the image of the same object (subject 4) the same size, and a resampling processing unit that allows comparison at the same pixel pitch, and the resampling processing unit performs a resampling process and a correction process that corrects the difference in signal degradation that occurs due to resampling of the reference image and the search image.

Furthermore, when searching for corresponding points in images in disparity detection, image data from the imaging element required for evaluating the degree of match is extracted for at least one of the images of the reference image and the search image, and geometric correction and resampling processing are performed to correct the difference in signal degradation due to resampling of the images of the reference image and the search image.

Furthermore, when performing the corresponding point search, the positions of the two images, the reference image and the search image, on the imaging element are detected, and the signal degradation of the images caused by the geometric correction and resampling process at each of the two image positions is corrected.

This will give you the following effects:

1) It is possible to provide a stereo camera using different types of cameras, or a stereo camera consisting of an optical system with special projection characteristics using a wide-angle lens, curved mirror, etc.

2) It is possible to eliminate problems caused by image zooming due to different camera lenses and image sensors with different pixel pitches, improving the performance of matching image search in stereo cameras and improving the accuracy of disparity detection when using different cameras.

3) In stereo cameras with cameras with different sizes and pixel pitches of image sensors (sensors), when the left and right are converted to the same image space by geometric correction to perform stereo matching, the sampling frequency of the sensor is different, so the characteristics of the converted image are different. In addition, in stereo cameras with optical systems such as high distortion lenses, special projection lenses, and hyperbolic mirrors, when pixel positions are converted by geometric transformation (distortion correction/geometric correction) to perform stereo matching, the original sampling points on the sensor do not exist at equal intervals on the new pixel positions in a lattice, but exist at uneven intervals, so the characteristics of the image change depending on the image position after the geometric transformation. In the distance measurement device of the present invention, the data on the sensor before geometric correction is taken from the image block after geometric correction used in stereo matching, and geometric interpolation is performed so that the block images have the same image size and pixel pitch on the left and right, and the difference in the original sampling frequency is corrected and matching is evaluated. In addition, the pixel positions of the reference block and the pixel positions of the search block are detected, and the sampling difference due to the difference in the positions on the images where the two stereo matching is performed is corrected.

The invention made by the inventor has been specifically described above based on examples, but it goes without saying that the present invention is not limited to the above embodiments and examples, and various modifications are possible.

1a, 1b, 1c, 30a, 30b: imaging element, 2a, 2b, 200a, 200b, 300: camera, 3a, 3b: optical axis, 4: subject, 5a, 5b, 70a, 70b, 80: lens, 6, 6a, 6b: distance measuring device, 40a, 40b, 41a, 41b: resampling circuit, 40c: first viewpoint imaging section resampling circuit, 40d: second viewpoint imaging section resampling circuit, 50: parallax calculation circuit, 51: reference position detection circuit, 52: comparison image position detection circuit, 53, 53a: comparison image correction circuit, 54: parallax detection circuit, 55: reference image correction circuit, 90: viewpoint separation circuit

Claims (7)

A distance measuring device that measures distance by determining a degree of coincidence between a first image and a second image on an imaging element, the first image and the second image being captured of the same object ,
a first correction processing unit that performs a process of adding an interpolation point by resampling at least one of the first image and the second image ;
a second correction processing unit that performs a geometric correction process on at least one of the resampled first image data and the second image data;
A distance measuring device comprising : 2. A distance measuring device according to claim 1,
2. A distance measuring device according to claim 1, wherein the first image and the second image for which the degree of coincidence is to be determined are located on a single image sensor. 2. A distance measuring device according to claim 1,
13. A distance measuring device , comprising: said first image and said second image, the degree of coincidence of which is to be determined, located on two image pickup devices each having a different pixel pitch. 2. A distance measuring device according to claim 1,
13. A distance measuring device, comprising: an optical system including lenses each having a different focal length, the first image and the second image for which the degree of coincidence is to be determined. 2. A distance measuring device according to claim 1,
The geometric correction process includes a super-resolution process for restoring a signal drop, or a decimation filter process for blocking high frequency components of image data;
A distance measuring device characterized in that the geometric correction process is performed on one of the resampled image data of the first image and the image data of the second image, and the image data of one of the images after the geometric correction process is aligned with the image data of the other image. 2. A distance measuring device according to claim 1,
the first image and the second image for which the degree of coincidence is to be determined are formed by an optical system including a lens having distortion;
The first correction processing unit includes:
a detection circuit for detecting the respective positions on the imaging element of the first image and the second image for which the degree of coincidence is to be determined;
A distance measuring device characterized by performing a process of performing a geometric transformation on the image data of the first image and the image data of the second image based on the position detected by the detection circuit and a process of adding the interpolation points . The present invention includes a camera, a viewpoint separation circuit, a first viewpoint image capture unit resampling circuit, a second viewpoint image capture unit resampling circuit, and a parallax calculation circuit,
the parallax calculation circuit includes a reference position detection circuit, a comparison image position detection circuit, a comparison image correction circuit, a parallax detection circuit, and a reference image correction circuit;
the camera includes an optical system including a lens with distortion and an image sensor, an image formed on a two-dimensional plane of the image sensor via the lens is sampled by the image sensor,
the viewpoint separation circuit selects a first viewpoint shooting area and a second viewpoint shooting area in the image sampled by the image sensor, supplies first viewpoint image data corresponding to the first viewpoint shooting area to the first viewpoint shooting unit resampling circuit, and supplies second viewpoint image data corresponding to the second viewpoint shooting area to the second viewpoint shooting unit resampling circuit;
the first viewpoint image capture unit resampling circuit detects a position of the first viewpoint image data on the imaging element and performs a resampling process on the first viewpoint image data based on the position; and during the resampling process, the first viewpoint image capture unit resampling circuit performs a geometric transformation and an interpolation process based on the position of the first viewpoint image data on the imaging element in order to correct distortion of the lens;
the second viewpoint image capturing unit resampling circuit detects a position of the second viewpoint image data on the imaging element, and performs a resampling process of the second viewpoint image data based on the position; and the second viewpoint image capturing unit resampling circuit performs a geometric conversion and an interpolation process during the resampling process based on the position of the second viewpoint image data on the imaging element in order to correct distortion of the lens;
a pixel pitch of the first viewpoint image data after resampling is made the same as that of the second viewpoint image data after resampling, the first viewpoint image data after resampling is supplied to the reference position detection circuit, and the second viewpoint image data after resampling is supplied to the comparison image position detection circuit;
the reference position detection circuit detects or selects a partial area of a reference image from the first viewpoint image data after resampling;
the comparison image position detection circuit detects or selects a partial area of the comparison image from the second viewpoint image data after resampling;
the reference image correction circuit calculates a correction amount attributable to a position of the reference image on the first viewpoint image data after resampling, and performs a correction process to correct a signal degradation of the image of the first viewpoint image data caused by the resampling process by correcting the reference image;
the comparison image correction circuit calculates a correction amount attributable to a position of the comparison image on the second viewpoint image data after resampling, and performs a correction process to correct a signal degradation of the image of the second viewpoint image data caused by the resampling process by correcting the comparison image;
a disparity detection circuit for detecting a disparity between the reference image corrected by the reference image correction circuit and the comparison image corrected by the comparison image correction circuit;