JP7083014B2 - Stereo camera - Google Patents

Description

The present invention relates to a stereo camera.

The prior art of Patent Document 1 stores correction information for correcting distortion of images of two imaging units, and uses the correction information of the image to geometrically correct two images to make two images. After calculating the disparity between them, the distance is calculated. Of the two image pickup units, the transmittance of the first optical element on the image side used for recognition is higher than that of the other second optical element, and the distortion of the first optical element is smaller than that of the second optical element. By satisfying at least one of the sensitivity characteristics of the first image sensor being higher than that of the second image sensor and the noise of the first image sensor being smaller than that of the second image sensor, the recognition performance is improved. Reduce variability.

The prior art of Patent Document 2 creates correction information for the images of the two imaging units stored in the prior art of Patent Document 1. Information that a chart is installed in front of a stereo camera, a chart in which patterns of multiple feature points are drawn is taken, the position of the feature points on the image is detected, and distortion of the image is corrected. To create. Images are taken at different distances from the chart, manufacturing errors and equipment errors are estimated based on the differences in the positions of their feature points, and deviations in the image correction information due to manufacturing errors and equipment errors are corrected to correct the image correction information. Improve the accuracy of.

The prior art of Patent Document 3 performs geometric correction processing on two images, extracts features from the two images, associates the same features on the two images, and obtains these features and their correspondence. Based on this, the deviation of the camera parameters is corrected.

Japanese Unexamined Patent Publication No. 2014-72592 International Publication No. 2014/181581 Japanese Unexamined Patent Publication No. 2004-354257

In a camera with a relatively narrow angle of view of about 40 ° or less, the intersection of the main ray and the optical axis (center of the entrance pupil) does not depend on the angle of incidence of the main ray and is located at a fixed place. In the stereo camera, the distance is measured on the premise of a pinhole camera model in which the center of the entrance pupil does not move, regardless of the angle of incidence of the main ray.

However, in a wide-angle camera, the intersection of the main ray and the optical axis (center of the entrance pupil) differs depending on the incident angle of the main ray. Specifically, as the angle of incidence of the main ray increases, the center of the entrance pupil moves forward. Therefore, the incident angle of the main ray from the same object is different from that of the pinhole camera model, and the position is deviated from the pinhole camera model around the image. Since the deviation around the image differs depending on the distance of the object, it cannot be corrected only by giving a certain amount of correction, and an error occurs in the calculated distance.

The prior art of Patent Document 2 creates image correction information using a pattern of feature points on a chart installed at a location relatively close to the camera. Since the prior art of Patent Document 1 uses the correction information of this image, the deviation on the image is small at a position close to the camera, and the deviation on the image is large at a position far from the camera. Since the parallax of a distant object is small, if a parallax error occurs there, the distance error becomes large.

Since the prior art of Patent Document 3 does not consider the movement of the center of the entrance pupil for each angle of incidence of the main ray, it is not possible to correct the deviation on the image that differs depending on the distance, and a distance error occurs.

An object of the present invention is to provide a stereo camera capable of reducing a distance error caused by the movement of the center of the entrance pupil for each angle of incidence of the main ray.

In order to achieve the above object, the stereo camera of the present invention includes a first imaging unit that captures a first image of an object, a second imaging unit that captures a second image of the object, and a second imaging unit. A geometric correction unit that geometrically corrects the first image and the second image, and a parallax calculation unit that calculates the misparity from the geometrically corrected first image and the second image. The incident pupil is provided based on the three-dimensional position of the region on the object with respect to the first imaging unit and the amount of movement of the incident pupil center of the first imaging unit according to the incident angle of the main light beam. The first position of the region of the object in the first image when the center does not move is calculated, the three-dimensional position of the region on the object with respect to the second image pickup unit, and the incident angle of the main light beam. The second position of the region on the object in the second image when it is assumed that the center of the incident pupil does not move based on the amount of movement of the center of the incident pupil of the second image pickup unit according to the above. Is calculated.

According to the present invention, it is possible to reduce the distance error caused by the movement of the center of the entrance pupil for each angle of incidence of the main ray. Issues, configurations and effects other than those described above will be clarified by the following description of the embodiments.

It is a figure which shows the structure of the stereo camera by one Embodiment of this invention. It is a figure which shows the operation of the stereo camera shown in FIG. It is a figure which shows an example of the operation of the parallax correction in consideration of the movement of the center of the entrance pupil shown in FIG. It is a figure which shows the reference image and the reference image. It is a figure which shows an example of the deviation of the image correction information from the pinhole camera model for each distance and incident angle in one Embodiment of this invention. It is a figure which shows an example of the deviation distribution of the image correction information with the pinhole camera model on the image at a certain distance in one Embodiment of this invention. It is a figure which shows an example of the deviation of the image correction information from the pinhole camera model for each distance and incident angle when image correction information is created at a close position. It is a figure which shows an example of the deviation distribution of the image correction information with the pinhole camera model on the image at infinity when the image correction information is created at a close position. It is a figure which shows an example of the distance error when image correction information is created at a close position. It is a figure which shows an example of the distance error when the image correction information of this invention is used. It is a figure which shows the incident angle and the image deviation of the main ray of the model which the center of an entrance pupil moves, and the pinhole camera model. It is a figure which shows the modification of the operation of the parallax correction in consideration of the movement of the center of the entrance pupil shown in FIG. It is a figure which shows an example of the table of the entrance pupil center movement information storage part. It is a figure which shows another example of the table of the entrance pupil center movement information storage part. It is a figure which shows another example of the table of the entrance pupil center movement information storage part.

FIG. 1 shows the configuration of the stereo camera 1 according to the embodiment of the present invention. The stereo camera 1 is a wide-angle camera having an angle of view of 40 ° or more.

The stereo camera 1 according to the embodiment of the present invention includes an image pickup system unit 100a, an image pickup system unit 100b, a calculation unit 110, a screen audio output unit 130, and a control unit 140.

The image pickup system unit 100a of a camera or the like includes an optical element unit 101a and an image pickup element unit 102a.

The optical element unit 101a such as a lens refracts light to form an image on the image pickup element unit 102a.

The image sensor unit 102a such as an image sensor receives an image of light refracted by the optical element unit 101a and generates an image according to the intensity of the light.

The image pickup system unit 100b of a camera or the like includes an optical element unit 101b and an image pickup element unit 102b. Further, the design values of the focal lengths of the image pickup system unit 100a and the image pickup system unit 100b are the same. The directions of the optical axes of the image pickup system unit 100a and the image pickup system unit 100b are substantially the same.

The optical element unit 101b such as a lens refracts light to form an image on the image pickup element unit 102b.

The image sensor unit 102b such as an image sensor receives an image of light refracted by the optical element unit 101b and generates an image according to the intensity of the light.

The image captured by the imaging system unit 100a is referred to as a reference image because it is a reference image when creating a parallax image. In other words, the image pickup system unit 100a (first image pickup unit) captures a reference image (first image) of the object. Further, the image captured by the imaging system unit 100b is an image in which a region matching the region extracted from the reference image is searched for when creating a parallax image, and is referred to as a reference image. In other words, the image pickup system unit 100b (second image pickup unit) captures a reference image (second image) of the object.

The arithmetic unit 110 composed of a CPU (central processing unit, a central arithmetic processing device), a memory, and the like includes an image pickup image storage unit 111, a geometric correction image storage unit 112, a parallax image storage unit 113, a geometric correction information storage unit 114, and a geometry. Correction change information storage unit 115, incident pupil center movement information storage unit 116, synchronization signal generation unit 117, reference image acquisition unit 118a, reference image acquisition unit 118b, geometric correction unit 119, parallax calculation unit 120, parallax correction unit 121 A recognition unit 122 and a geometric calibration unit 123 are provided.

The image pickup image storage unit 111 such as a memory stores the images output from the image pickup system unit 100a and the image pickup system unit 100b.

The geometrically corrected image storage unit 112 such as a memory has a reference image (reference image after geometric correction) and a reference image (reference image after geometric correction) in which the captured reference image (reference image after imaging) and the reference image (reference image after imaging) are geometrically corrected. Stores the reference image (reference image after geometric correction).

The parallax image storage unit 113 such as a memory stores the parallax image.

The geometric correction information storage unit 114 such as a memory stores the two-dimensional coordinates (geometric correction information) on the image after imaging with distortion corresponding to each pixel on the image without distortion in the reference image and the reference image. .. This geometric correction information is used by the geometric correction unit 119 for processing to correct lens distortion and optical axis deviation of the reference image and the reference image after imaging. That is, the geometric correction unit 119 geometrically corrects the reference image (first image) and the reference image (second image) using the geometric correction information.

The geometric correction information is obtained from the pinhole camera model only by the value of the positional deviation on the image when an object at a certain distance is projected between the case where the center of the entrance pupil moves and the case where the center of the entrance pupil moves (pinhole camera model). It is out of alignment.

In other words, the geometric correction information storage unit 114 determines the position of the reference image (first image) of the object when the center of the entrance pupil, which indicates the intersection of the main light and the optical axis, moves according to the angle of incidence, and the incident. The geometric correction information having an error by the difference between the position of the reference image of the object and the position of the reference image when it is assumed that the center of the pupil does not move according to the angle of incidence is stored. Further, the geometric correction information storage unit 114 determines the position of the reference image (second image) of the object when the center of the entrance pupil indicating the intersection of the main light and the optical axis moves according to the angle of incidence, and the center of the entrance pupil. Stores geometric correction information that has an error of the difference between the position of the reference image of the object and the position of the reference image when it is assumed that does not move according to the angle of incidence.

That is, as shown in FIG. 5, when an object at infinity is projected, it matches the value of the geometric correction information of the pinhole camera model, the distance becomes short, and the incident angle of the main ray becomes large, the pinhole camera There is a discrepancy with the value of the geometric correction information assuming the model.

The geometric correction change information storage unit 115 such as a memory moves the reference image and the reference image horizontally so that the geometric correction unit 119 further moves the geometric correction information in the horizontal and vertical directions when the geometric correction unit 119 performs the geometric correction processing of the image. The amount and the amount of vertical movement are stored. Here, the initial values of the horizontal movement amount and the vertical movement amount of the reference image and the reference image are zero. The horizontal movement amount and vertical movement amount of the reference image and the reference image are calculated by the geometric calibration unit 123, and are for correcting the vertical and horizontal deviations caused by mechanical deviations such as aging, thermal changes, and shock vibrations. Is.

The entrance pupil center movement information storage unit 116 such as a memory stores the value of the movement amount of the entrance pupil center to each entrance angle with respect to the incident angle zero in the image pickup system unit 100a and the image pickup system unit 100b. In other words, as shown in FIG. 13, the entrance pupil center movement information storage unit 116 has the respective incident angles (θ1, θ2, ...) And the corresponding movement amount of the entrance pupil center (ΔL1, ΔL2, ...). (・ ・) And are stored in the table T1301.

The synchronization signal generation unit 117 generates and transmits a synchronization signal.

The reference image acquisition unit 118a acquires the image generated by the image sensor unit 102a after transmitting the synchronization signal and the exposure time information to the image sensor unit 102a in accordance with the synchronization signal of the synchronization signal generation unit 117. The image is stored in the captured image storage unit 111.

The reference image acquisition unit 118b acquires the image generated by the image sensor unit 102b after transmitting the synchronization signal and the exposure time information to the image sensor unit 102b in accordance with the synchronization signal of the synchronization signal generation unit 117. The image is stored in the captured image storage unit 111.

The geometric correction unit 119 reads the geometric correction information from the geometric correction information storage unit 114, and reads the horizontal movement amount and the vertical movement amount of the reference image and the reference image from the geometric correction change information storage unit 115. Horizontal movement amount and verticality of the reference image and the reference image to the two-dimensional coordinates (geometric correction information) on the distorted image corresponding to each pixel on the image without distortion in the reference image and the reference image which are geometric correction information, respectively. Add up the amount of movement. Geometric correction processing is performed based on the reference image and reference image after imaging and the added geometric correction information, and a distortion-free image is calculated. The reference image and the reference image after these geometric corrections are stored in the geometric correction image storage unit 112.

The parallax calculation unit 120 reads the reference image and the reference image after the geometry correction from the geometry correction image storage unit 112, and has the same height on the reference image corresponding to a region (template image) of a predetermined size extracted from the reference image. Explore the area in. The difference between the position of the region on the reference image that matches the template image and the position of the template image on the reference image, that is, the parallax is calculated. A parallax image is calculated by calculating the parallax for each region.

In other words, the parallax calculation unit 120 calculates the parallax from the geometrically corrected reference image (first image) and reference image (second image).

The parallax correction unit 121 captures the parallax image from the parallax image storage unit 113, and the center of the entrance pupil from the entrance pupil center movement information storage unit 116 to each entrance angle with reference to the incident angle zero in the image pickup system unit 100a and the image pickup system unit 100b. Read the amount of movement. The parallax correction unit 121 calculates the distance of each region on the image using the parallax image, the focal lengths of the image pickup system unit 100a and the image pickup system unit 100b, the pixel pitch, and the baseline length.

The positions on the reference image and the reference image of each region are the positions after geometric correction when the center of the entrance pupil moves. Therefore, when the center of the entrance pupil does not move using the amount of movement of the center of the entrance pupil to each angle of incidence with reference to the position on the reference image of each region, the parallax and the distance, the position of the optical axis, the parallax image, and the angle of incidence of zero. After calculating the position of each region on the reference image and the reference image after geometric correction, the difference is calculated and used as the corrected parallax of each region.

In other words, the parallax correction unit 121 includes the amount of movement of the center of the entrance pupil according to the incident angle of the main light of the object and the parallax calculated by the parallax calculation unit (when the center of the entrance pupil moves according to the angle of incidence). Based on the parallax), the parallax (parallax of the pinhole camera model) when it is assumed that the center of the entrance pupil does not move according to the angle of incidence is calculated and used as the corrected parallax. This makes it possible to reduce the parallax error due to the movement of the center of the entrance pupil.

The parallax correction unit 121 stores the corrected parallax image in the parallax image storage means. The parallax calculated by the parallax calculation unit 120 is the parallax when the center of the entrance pupil moves. The parallax correction unit 121 corrects the parallax when the center of the entrance pupil moves by the above process to the parallax of the pinhole camera model (when the center of the entrance pupil does not move), which is a prerequisite for the distance measurement of the stereo camera 1. do.

The recognition unit 122 reads a parallax image from the parallax image storage unit 113, and takes an image based on the parallax, the focal length (base line length) between the image pickup system unit 100a and the image pickup system unit 100b, the focal length, and the size of one pixel. The distance from the stereo camera 1 to the object on the image is calculated in the optical axis direction of the system unit 100a and the imaging system unit 100b. A distance image is calculated by calculating the distance for each area.

Next, the recognition unit 122 reads the reference image after the geometry correction from the geometry correction image storage unit 112, and uses the reference image and the distance image after the geometry correction to display the object and the reference image on the reference image. The position of the object is recognized, and the three-dimensional relative position and relative speed of the object with respect to the stereo camera 1 are calculated. Here, the three-dimensional relative position coordinate system with respect to the stereo camera 1 has the center of the incident pupil of the image pickup system unit 100a as the origin, the x coordinate in the right direction, the y coordinate in the downward direction, and the optical axis direction with respect to the image pickup system unit 100a. Take the z coordinate to. Further, the recognition unit 122 calculates the time until the collision based on the relative position and the relative speed of the stereo camera 1 and the object, and determines whether or not the collision occurs within a predetermined time. The relative position, relative speed, collision determination result and collision time between the stereo camera 1 and the object are sent to the screen audio output unit 130 and the control unit 140.

The geometric calibration unit 123 reads the reference image and the reference image after the geometry correction from the geometry correction image storage unit 112, and reads the difference image from the difference image storage unit 113, and based on these information, on the reference image after the geometry correction. The horizontal movement amount and vertical movement amount of the reference image so that the optical axis (vanishing point) position of the The horizontal movement amount and the vertical movement amount of the reference image are calculated. The results are stored in the geometric correction change information storage unit 115.

The screen audio output unit 130 such as a monitor and a speaker displays a reference image, a parallax image, and a distance image on the screen. In addition, a frame or marker is displayed at the position of the object. At this time, the color of the frame or marker of the object, which is the determination that the collision determination result from the recognition unit 122 collides, is different from that of the non-collision object. When there is an object for which the collision determination result from the recognition unit 122 is determined to collide, the screen sound output unit 130 outputs a warning sound.

A control unit 140 such as a CPU generates a control signal based on the relative position, relative speed, collision time, and collision determination result between the stereo camera 1 and the object, and outputs the control signal to the outside of the stereo camera 1.

The operation procedure of the stereo camera 1 according to the embodiment of the present invention shown in FIG. 1 will be described with reference to FIGS. 2 and 3. The subject of each step is the same in principle unless otherwise specified.

Step 201:
The synchronization signal generation unit 117 generates a synchronization signal and sends it to the reference image acquisition unit 118a and the reference image acquisition unit 118b. Immediately after receiving the synchronization signal from the synchronization signal generation unit 117, the reference image acquisition unit 118a sends information on the synchronization signal and the exposure time to the image pickup element unit 102a. Immediately after receiving the synchronization signal and the exposure time information from the reference image acquisition unit 118a, the image sensor unit 102a receives an image of the light refracted by the optical element unit 101a for the exposure time, and adjusts the intensity of the light. The corresponding image is generated, and the image is sent to the reference image acquisition unit 118a. The reference image acquisition unit 118a receives an image from the image sensor unit 102a and stores the image in the image pickup image storage unit 111.

Immediately after receiving the synchronization signal from the synchronization signal generation unit 117, the reference image acquisition unit 118b sends the synchronization signal and the exposure time information to the image pickup element unit 102b. Immediately after receiving the synchronization signal and the exposure time information from the reference image capture unit 118b, the image sensor unit 102b receives an image of the light refracted by the optical element unit 101b for the exposure time, and adjusts the intensity of the light. The corresponding image is generated, and the image is sent to the reference image acquisition unit 118b. The reference image acquisition unit 118b receives an image from the image sensor unit 102b and stores the image in the image pickup image storage unit 111.

Step 202:
The geometry correction unit 119 reads the reference image and the reference image after imaging from the image capture image storage unit 111. The geometric correction information of the reference image and the reference image is read from the geometric correction information storage unit 114, and the horizontal movement amount and the vertical movement amount of the reference image and the reference image are read from the geometric correction change information storage unit 115.

Using the formulas (1) and (2), the coordinates (Fx (X2, Y2), Fy (X2, Y2)) on the image after imaging corresponding to each pixel on the reference image after geometric correction, the reference. Based on the horizontal movement amount ΔX2 and the vertical movement amount ΔX2 of the image, the positions (X1, Y1) of the reference image before imaging corresponding to the pixels (X2, Y2) of the reference image after the geometric correction are calculated.

The luminance values of the pixels (X2, Y2) of the reference image after geometric correction are calculated by performing two-dimensional linear interpolation of the luminance values of the four peripheral pixels of the reference image positions (X1, Y1). The above procedure is performed for each pixel of the reference image after the geometry correction, and the brightness value of the reference image after the geometry correction is calculated. Further, the above procedure is also performed for the reference image to calculate the brightness value of the geometrically corrected image of the reference image. The reference image and the reference image after the geometry correction are stored in the geometry correction image storage unit 112.

Step 203:
The parallax calculation unit 120 reads the geometry-corrected image of the reference image and the reference image from the geometry-corrected image storage unit 112. As shown in FIG. 4, an image 403 (template image) of a region of a predetermined size of the reference image 401 after geometric correction is extracted. The reference image 402 after the geometric correction searches for an image in the area where the same object as the template image 403 is shown by the following template matching.

An image 404 (searched image) in a region of a predetermined size of the reference image 402 at the same height as the template image 403 is extracted, and the absolute value of the difference between the brightness value of the template image 403 and the brightness value of the searched image 404 is absolute value. (SAD, Sum of Absolute Difference) is calculated. The SAD is calculated for each searched image 404 on the reference image 402 at the same height as the template image 403, and the searched image 405 having the smallest SAD value is searched.

Using the SAD of the searched image 405 and the SAD of the searched image adjacent to the left and right one pixel from the searched image 405, equiangular straight line fitting is performed, and the searched image 405 on the reference image that most closely matches the template image 403. Calculate the subpixel. By adding subpixels to the difference in position between the template image 403 and the image to be searched 405, the parallax of the template image 403 on the reference image 401 after geometric correction is calculated.

Next, it is determined whether the parallax is valid or invalid by the following two determination methods. Using each searched image 404 on the reference image 402 at the same height as the template image 403, if the minimum value of the obtained SAD is equal to or greater than the threshold value, it is assumed that the template 403 and the searched image 405 do not match. It is determined that the parallax in the region is invalid, and if the minimum value of SAD is less than the threshold value, it is determined to be valid.

Further, in the SAD representing the degree of coincidence of the patterns, the difference between the SAD and the adjacent SAD is calculated. When the place where the value of these differences changes from negative to positive is detected, the place where the SAD of those places is the smallest and the place where the SAD is the second smallest are detected, and the difference between those SADs is less than the threshold value. , It is determined that the parallax in the area is invalid because there is a pattern similar to the template image and there is a possibility of mismatching, and if not, it is determined to be valid. Here, when there is only one place where the difference between adjacent SADs changes from negative to positive, it is determined that the parallax is valid. The area determined to be valid by both of the two determination methods is valid, and if it is determined to be invalid by either determination method, the area is invalidated.

Such processing is performed for all areas on the reference image after geometric correction, and the parallax of the entire reference image 401 is calculated. The parallax image calculated in this way is stored in the parallax image storage unit 113.

Step 204:
The parallax correction unit 121 captures the parallax image from the parallax image storage unit 113, and the center of the entrance pupil from the entrance pupil center movement information storage unit 116 to each entrance angle with reference to the incident angle zero in the image pickup system unit 100a and the image pickup system unit 100b. Read the amount of movement. The parallax correction unit 121 calculates the distance of each region on the image using the parallax image, the focal lengths of the image pickup system unit 100a and the image pickup system unit 100b, the pixel pitch, and the baseline length. The positions on the reference image and the reference image of each region are the positions after geometric correction when the center of the entrance pupil moves.

Therefore, when the center of the entrance pupil does not move using the amount of movement of the center of the entrance pupil to each angle of incidence with reference to the position on the reference image of each region, the parallax and the distance, the position of the optical axis, the parallax image, and the angle of incidence of zero. After calculating the position of each region on the reference image and the reference image after geometric correction, the difference is calculated and used as the corrected parallax of each region. The corrected parallax image is stored in the parallax image storage means. The operation procedure of this step is shown in FIG.

Step 205:
The recognition unit 122 reads the parallax image from the parallax image storage unit 113 and the geometry-corrected image from the geometry-corrected image storage unit 112.

Using the mathematical formula (3), the distance L in the optical axis direction from the stereo camera 1 in the region on the parallax image is calculated. Here, f is a design value of the focal lengths of the image pickup system unit 100a and the image pickup system unit 100b, B is the distance (base line length) between the main points of the image pickup system unit 100a and the image pickup system unit 100b, d is the parallax, and c is the image pickup. This is the pixel pitch of the element unit 102a and the image pickup element unit 102b.

This process is performed for all areas of the parallax image, the distance in the optical axis direction to the stereo camera 1 in the entire parallax image is calculated, and a distance image is created.

Next, the recognition unit 122 calculates the position of the vanishing point on the processed image of the reference image, determines an object such as a car or a pedestrian, calculates the relative position and relative speed of the object with respect to the stereo camera 1, and the object. And the stereo camera 1 are determined to collide with each other.

First, the recognition unit 122 calculates the position of the vanishing point on the reference image according to the following procedure. The white lines on both sides at the boundary of the lane on the reference image are detected, and the inclination of the white lines on the reference image is calculated. Assuming that the white lines on both sides are straight lines, the position on the reference image of the point where the white lines on both sides intersect is calculated from the calculated inclination. This is the position of the vanishing point.

Next, the recognition unit 122 detects an object such as a car or a pedestrian according to the following procedure. In a distance image, a region 1 in which pixels having a distance within a predetermined range are connected is obtained. As an example of a predetermined range, since the width is 5 m such as 5 to 10 m, 7.5 to 12.5 m, 10 to 15 m, etc., a plurality of ranges having overlapping ranges are set every 2.5 m. The lengths in the vertical and horizontal directions on the reference image of each region 1 in which the pixels whose distances are within a predetermined range are connected are obtained. The length in the vertical direction on the reference image of each area 1 and the value obtained by multiplying the distance by the pixel pitch are divided by the focal length to calculate the three-dimensional vertical length of each area 1. Similarly, the lateral length on the reference image of each region 1 and the value obtained by multiplying the distance by the pixel pitch are divided by the focal length to calculate the three-dimensional lateral length of each region 1. Using the formula (4), the vertical position Vg on the reference image with respect to the ground of each region 1 is approximately calculated. Here, Vv is the height of the vanishing point, Hi is the mounting height of the stereo camera 1, and Lr is the average distance of the region 1. Further, the calculation formula is based on the assumption that the optical axes of the image pickup system unit 100a and the image pickup system unit 100b are generally in the horizontal direction.

The three-dimensional vertical and horizontal lengths of the region 1 are within the predetermined range of the automobile, and the vertical position of the lower limit of the region 1 on the reference image and the region 1 calculated by the mathematical formula (2). When the difference in the vertical positions on the reference image of the ground is within the threshold value, it is determined that the object in the region 1 is an automobile. Similarly, the three-dimensional vertical and horizontal lengths of the area 1 are within the predetermined range of the pedestrian, and the position of the lower limit of the area 1 in the vertical direction on the reference image is calculated by the formula (2). When the difference in the vertical position on the reference image of the ground of the area 1 is within the threshold value, it is determined that the object of the area 1 is a pedestrian. These processes are performed for all areas 1 to determine whether they are automobiles or pedestrians.

Next, the relative position and relative speed of the object with respect to the stereo camera 1 are calculated by the following procedure. For the area 1 determined to be a car or a pedestrian, the relative positions (Xo, Yo, Zo) of the object with respect to the stereo camera 1 are calculated using the mathematical formulas (5) to (7). Here, (Uo, Vo) is a position on the reference image with respect to the center of the region 1 determined to be a car or a pedestrian.

The processes of steps 202 to 208 are repeated in a predetermined cycle. If the difference in position on the reference image of the area 1 detected in step 205 of the previous process and this process is within the threshold value, it is determined that they are the same object, and the stereo camera 1 calculated by this process is used. The relative speed (Vx) of the object with respect to the stereo camera 1 is divided by the value obtained by subtracting the relative position calculated in step 205 of the previous processing from the relative position of the object by the time interval of the processing cycle of steps 202 to 208. , Vy, Vz).

Finally, the collision determination between the object and the stereo camera 1 is performed by the following procedure. When the relative speed Vz of the object with respect to the stereo camera 1 is 0 or more, it is determined that the object does not collide with the object in the region 1 determined to be a car or a pedestrian. When the relative velocity Vz of the object with respect to the stereo camera 1 is negative, the relative position Zo of the object with respect to the stereo camera 1 calculated in this process is divided by the absolute value of the relative velocity Vz of the object with respect to the stereo camera 1. Calculate the time until collision (collision time). Further, the relative position Xo of the object is added to the value obtained by multiplying the relative velocity Vx of the object with respect to the stereo camera 1 by the collision time to calculate the relative position Xo of the object with respect to the stereo camera 1 at the time of collision.

Therefore, the relative velocity Vz of the object with respect to the stereo camera 1 is negative, the collision time is within the threshold value, and the absolute value of the relative position Xo of the object with respect to the stereo camera 1 at the time of collision is within the threshold value. In this case, it is determined that the vehicle collides with an object in the region 1 determined to be a car or a pedestrian. In other cases, it is determined that there is no collision. The recognition unit 122 displays the positions of the four corners on the reference image regarding the region 1 determined to be a car or a pedestrian, the relative position and relative speed of the object with respect to the stereo camera 1, the collision determination result, and the collision time with the screen audio output unit 130 and the collision time. It is sent to the control unit 140.

Step 206:
The geometric calibration unit 123 reads the reference image and the reference image after the geometric correction from the geometric correction image storage unit 112, and reads the parallax image from the parallax image storage unit 113. The left and right white lines on the road on the reference image after geometric correction are detected, and the approximate straight line of the left and right white lines is calculated. Calculate the position of the intersection of the approximate straight lines on the left and right white lines, and assume that the position of this intersection is the disappearance point and coincide with the position of the optical axis on the reference image, and the design value of the position of the intersection and the position of the optical axis The difference is calculated and used as the correction amount in the horizontal direction and the vertical direction of the reference image. These correction amounts are used as the horizontal movement amount and the vertical movement amount of the reference image.

The geometric calibration unit 123 extracts the parallax corresponding to each position in the vertical direction on the image of the left and right white lines, and calculates an approximate straight line between the position in the vertical direction on the image of the left and right white lines and the parallax. From this approximate straight line, the parallax of the white line corresponding to the design value of the position in the vertical direction of the optical axis of the reference image is calculated. Originally, the parallax of the white line corresponding to the design value of the position in the vertical direction of the optical axis of the reference image is zero, so the parallax of the white line corresponds to the horizontal correction amount of the reference image, and this correction amount is used as the reference image. The amount of horizontal movement of.

Similarly for the reference image, the geometric calibration unit 123 detects the left and right white lines on the road on the reference image after the geometric correction, and calculates an approximate straight line between the left and right white lines. Calculate the position of the intersection of the approximate straight lines on the left and right white lines, and assume that the position of this intersection is the vanishing point and coincide with the position of the optical axis on the reference image. The difference is calculated and used as the horizontal correction amount of the reference image. These correction amounts are used as the vertical movement amount of the reference image.

The horizontal movement amount and vertical movement amount of the reference image and the reference image calculated by the above processing are sent to the geometric correction change information storage unit 115, and the geometric correction change information storage unit 115 causes the horizontal movement amount and vertical movement of the reference image and the reference image. Store the quantity.

Step 207:
The screen sound output unit 130 determines the positions of the four corners on the reference image regarding the area 1 determined to be a vehicle or a pedestrian by the recognition unit 122, the relative position and speed of the object with respect to the stereo camera 1, the collision determination result, and the collision time. receive. The reference image after the geometric correction is read from the geometric correction image storage unit 112. A reference image is displayed on the screen, and the area 1 determined to be a car or a pedestrian is displayed as a frame. Further, the color of the frame of the area 1 which is the judgment result that the collision determination result collides is changed with the color of the frame of the object area 1 of the judgment result that the collision does not occur, and the color is displayed on the screen. If there is a judgment result that the collision judgment result collides in the area 1, a warning sound is output.

Step 208:
The control unit 140 receives from the recognition unit 122 the positions of the four corners on the reference image regarding the region 1 determined to be a vehicle or a pedestrian, the relative positions and relative velocities of the object with respect to the stereo camera 1, the collision determination result, and the collision time. When there is a determination result that the collision determination result collides in the region 1 determined to be a car or a pedestrian, a control signal for avoiding the collision is generated and output to the outside of the stereo camera 1.

The operation procedure of parallax correction in consideration of the movement of the center of the entrance pupil in step 204 of the operation procedure of the stereo camera 1 according to the embodiment of the present invention will be described with reference to FIG.

Step 301:
The parallax correction unit 121 receives a parallax image from the parallax image storage unit 113, and each incident angle (θ1, θ2, .... The amount of movement of the center of the entrance pupil (ΔL1, ΔL2, ...) Up to (・ ・) is read.

Step 302:
The parallax correction unit 121 uses the mathematical formula (3) to obtain a focal length f, a pixel pitch c, and a baseline length B common to the parallax image, the image pickup system unit 100a, and the image pickup system unit 100b, and each of them on the reference image. The distance L of the region is calculated.

Step 303:
The parallax correction unit 121 uses the equations (8) to (10) to determine the position (U, V) and the distance L on the reference image of each region, and the focal length common to the imaging system unit 100a and the imaging system unit 100b. Based on f and the pixel pitch c, the three-dimensional position (X, Y, Z) with the center of the entrance pupil of the image pickup system unit 100a corresponding to each region when the entrance pupil moves is calculated as the origin. Here, (U0, V0) is the optical axis position on the reference image.

Step 304:
The parallax correction unit 121 uses the mathematical formula (11) to determine the position (U, V) on the reference image of each region, the focal length f common to the image pickup system unit 100a and the image pickup system unit 100b, and the pixel pitch c. In addition, the incident angle θ of each region on the reference image is calculated.

Step 305:
The parallax correction unit 121 is a step among the amount of movement (ΔL1, ΔL2, ...) Of the center of the entrance pupil to each entrance angle (θ1, θ2, ...) With respect to the incident angle zero in the imaging system unit 100a. The amount of movement of the center of the entrance pupil (for example, ΔL1 and ΔL2) corresponding to the two entrance angles (for example, θ1 and θ2) close to the incident angle θ in a certain region calculated in 304 is extracted, and linear interpolation is performed. The amount of movement ΔL of the center of the entrance pupil in the region is calculated. The above processing is performed for each area.

In other words, the parallax correction unit 121 responds to the incident angle of the main ray of the object based on the respective incident angles stored in the entrance pupil center movement information storage unit 116 and the corresponding movement amount of the entrance pupil center. Calculate the amount of movement of the center of the entrance pupil. This makes it possible to easily estimate the amount of movement of the center of the entrance pupil.

Step 306:
The parallax correction unit 121 uses mathematical formulas (12) and (13) to calculate the three-dimensional positions (X, Y, Z) of each region when the center of the entrance pupil calculated in step 303 moves, in step 305. Based on the movement amount ΔL of the center of the entrance pupil, the position (U', V') on the reference image of each region when the center of the entrance pupil does not move is calculated.

In other words, the parallax correction unit 121 is an image pickup system according to the three-dimensional position (X, Y, Z) of the image pickup system unit 100a (first image pickup unit) of the object and the incident angle of the main ray of the object. Based on the amount of movement ΔL of the center of the entrance pupil of the unit 100a, the position (U', V') of the object in the reference image when the center of the entrance pupil does not move is calculated.

Step 307:
The parallax correction unit 121 subtracts the parallax d from the horizontal position U on the reference image of each region, and calculates the horizontal position Uref on the reference image of each region. Using the formulas (8) to (10), the positions (Uref, Vref) and the distance L on the reference image of each region, the focal length f common to the image pickup system unit 100a and the image pickup system unit 100b, and the pixel pitch c are obtained. Based on this, the three-dimensional positions (Xref, Yref, Zref) with the center of the entrance pupil of the image pickup system unit 100b corresponding to each region when the entrance pupil moves are calculated as the origin. Here, the vertical position of each region on the reference image is the same as that of the reference image (Vref = V).

Step 308:
Using the mathematical formula (11), the parallax correction unit 121 is based on the positions (Uref, Vref) on the reference image of each region, the focal length f common to the image pickup system unit 100a and the image pickup system unit 100b, and the pixel pitch c. In addition, the incident angle θref of each region on the reference image is calculated.

Step 309:
The parallax correction unit 121 is the incident angle of a certain region calculated in step 308 of the amount of movement (θ1, θ2, ...) of the center of the entrance pupil to each incident angle with respect to the incident angle of zero in the imaging system unit 100b. The amount of movement of the center of the entrance pupil (for example, ΔL1 and ΔL2) corresponding to two angles of incidence (for example, θ1 and θ2) close to θref is extracted, linear interpolation is performed, and the amount of movement of the center of the entrance pupil in a certain region is ΔLref. Is calculated. The above processing is performed for each area.

Step 310:
The parallax correction unit 121 uses mathematical formulas (12) and (13) to calculate the three-dimensional positions (Xref, Yref, Zref) of each region when the center of the entrance pupil moves, which was calculated in step 307, in step 309. Based on the movement amount ΔLref of the center of the entrance pupil, the positions (U'ref, V'ref) on the reference image of each region when the center of the entrance pupil does not move are calculated.

In other words, the misalignment correction unit 121 is an image pickup system according to the three-dimensional positions (Xref, Yref, Zref) of the image pickup system section 100b (second image pickup section) of the object and the incident angle of the main ray of the object. Based on the amount of movement ΔLref of the center of the entrance pupil of the unit 100b, the position (U'ref, V'ref) of the object in the reference image (second image) assuming that the center of the entrance pupil does not move is determined. calculate.

Step 311:
The difference (| U'-U'ref |) in the horizontal position on the reference image and the reference image of each region when the center of the entrance pupil calculated in steps 306 and 310 does not move is calculated, and these are incident. The parallax of each region when the center of the pupil does not move.

In other words, the parallax correction unit 121 assumes that the position (U', V') of the object in the reference image (first image) when the center of the entrance pupil does not move and the center of the entrance pupil do not move. The horizontal difference from the position (U'ref, V'ref) of the object in the reference image (second image) is defined as the corrected parallax.

Here, the problems of the present embodiment will be described in detail with reference to FIG. One of the prerequisites for distance measurement of a stereo camera is that the intersection (center of the entrance pupil) between the main ray and the optical axis 1101 does not move for each incident angle of the main ray (main ray 1105, 1108, etc. described later) (pinhole camera). There is a model). However, in a wide-angle camera, the position of the center of the entrance pupil moves forward as the angle of incidence of the main ray increases.

For example, as shown in FIG. 11, when the distance to the object 1102 is short, the incident angle of the main ray 1106 incident on the position 1104 of the entrance pupil center when the center of the entrance pupil moves from the object 1102 and the object. Since the angle of incidence of the main ray 1105 incident on the position 1103 of the center of the entrance pupil when the center of the entrance pupil does not move from 1102 is significantly different from that of the position 1112 on the image 1110 when the center of the entrance pupil moves and does not move. The position 1111 is greatly displaced.

On the other hand, when the distance to the object 1107 is long, the angle of incidence of the main ray 1109 incident on the position 1107 of the center of the entrance pupil when the center of the entrance pupil moves from the object 1107 and the center of the entrance pupil move from the object 1107. Since the difference in the angle of incidence of the main ray 1108 incident on the position 1103 of the center of the entrance pupil is small, the deviation between the position 1114 and the position 1115 on the image 1113 when the center of the entrance pupil moves and when it does not move is small.

As described above, in the wide-angle camera in which the center of the entrance pupil moves, the model of the optical system is different from the pinhole camera model, so that the image cannot be geometrically corrected so that the parallax can be calculated accurately at any distance.

In the prior art of Patent Document 2, a chart installed at a relatively short distance is photographed, and geometric correction information of an image is created based on a pattern on the chart. Therefore, as shown in FIG. 7, when the incident angle is large at a short distance, the geometric correction information of the image deviates greatly from the pinhole camera model, and as shown in FIG. 8, at the edge region of the image, at infinity. The deviation of the geometric correction information of the image from the pinhole camera model becomes large. As a result, as shown in FIG. 9, an accurate distance can be calculated at a short distance, but a distance error becomes large at a long distance.

As shown in FIG. 5, the stereo camera 1 according to the embodiment of the present invention shown in FIG. 1 has geometric correction information in which the deviation from the geometric correction information assuming a pinhole camera model becomes zero at infinity. It is stored in the storage unit 114, and in step 202 of the operation procedure (FIG. 2), the geometric correction unit 119 corrects the distortion of the reference image and the reference image by using the geometric correction information having the above-mentioned characteristics, so that it is infinite. There is no positional deviation on the reference image and reference image with the pinhole camera model at a distance, and the misalignment error is reduced. At a short distance, the positional deviation on the reference image and the reference image from the pinhole camera model becomes large, but the parallax is large, so that the distance error is suppressed to be smaller than that of the prior art as shown in FIG. Here, FIG. 6 shows the deviation between the geometric correction information and the pinhole camera model at infinity.

In step 204 (FIG. 3) of the operation procedure (FIG. 2) of the stereo camera 1 according to the embodiment of the present invention shown in FIG. 1, the parallax correction unit 121 measures the parallax when the center of the entrance pupil moves. Since the parallax is corrected to the case where the camera does not move (pinhole camera model), the parallax error due to the movement of the center of the entrance pupil is reduced. Further, in step 205, the recognition unit 122 can calculate the distance based on the corrected parallax image and accurately calculate the collision time to the object, so that the collision prevention function can operate normally.

In step 206 of the operation procedure (FIG. 2) of the stereo camera 1 according to the embodiment of the present invention shown in FIG. 1, the geometry is geometrically corrected by using the parallax corrected in step 204, the reference image after geometric correction, and the reference image. The horizontal movement amount (horizontal correction amount) of the reference image and the reference image calculated by the calibration unit 123 is also accurate. Further, in step 202, the geometric correction unit 119 geometrically corrects the image using the horizontal movement amount of the reference image and the reference image accurately obtained in step 206, so that the image distortion is accurately corrected. In step 203, the disparity image is accurately calculated using the reference image and the reference image that have been accurately geometrically corrected in step 202.

As described above, according to the present embodiment, it is possible to reduce the distance error caused by the movement of the center of the entrance pupil for each angle of incidence of the main ray.

The stereo camera 1 of the present invention is not limited to the embodiment as described above, and can be variously modified and applied. Hereinafter, a modified example of the stereo camera 1 of the present invention will be described.

(Modification 1-1)
In the stereo camera 1 according to the embodiment of the present invention shown in FIG. 1, the entrance pupil center movement information storage unit 116 moves the center of the entrance pupil for each incident angle θ of the main rays of the image pickup system unit 100a and the image pickup system unit 100b. Instead of storing _ΔL , as shown in FIG. 14, the coefficients (an, an _-1 ) of the polynomial of the incident angle θ of the main rays of the image pickup system unit 100a and the image pickup system unit 100b and the movement amount ΔL of the center of the entrance pupil , ..., a ₀ ) is stored in the table T1401.

In step 301 of the operation procedure (FIG. 3), the parallax correction unit 121 reads from the entrance pupil center movement information storage unit 116 the coefficient of the polymorphism of the incident angle of the main ray and the movement amount of the entrance pupil center, and steps 304 and 308. In, the parallax correction unit 121 calculates the amount of movement of the center of the entrance pupil from the angle of incidence in each region by using the polymorphism of the angle of incidence of the main ray and the amount of movement of the center of the entrance pupil. In other words, the parallax correction unit 121 calculates the amount of movement of the center of the entrance pupil according to the incident angle of the main ray of the object by using a polynomial. This makes it possible to easily estimate the amount of movement of the center of the entrance pupil.

Even in this way, the parallax when the center of the entrance pupil moves can be corrected to the parallax when the center of the entrance pupil does not move, so that the parallax error due to the movement of the center of the entrance pupil can be reduced.

(Modification 1-2)
In the stereo camera 1 according to the embodiment of the present invention shown in FIG. 1, the entrance pupil center movement information storage unit 116 moves the center of the entrance pupil for each incident angle θ of the main rays of the image pickup system unit 100a and the image pickup system unit 100b. Instead of storing ΔL, as shown in FIG. 15, the image pickup system unit 100a and the image pickup system unit 100a and the image pickup system unit when the center of the entrance pupil moves by a predetermined distance L and by an incident angle θ of a predetermined main ray. The correction amount Cx of the geometric correction information of 100b is stored in the table T1501.

In other words, the entrance pupil center movement information storage unit 116 stores the correction amount Cx of the geometric correction information corresponding to the combination of each distance L of the object and each incident angle θ.

FIG. 12 shows the operation procedure of parallax correction in consideration of the movement of the center of the entrance pupil in step 204 of the operation procedure of the stereo camera 1 (FIG. 2) according to the embodiment of the present invention shown in FIG. Even so, the parallax when the center of the entrance pupil moves can be corrected to the parallax when the center of the entrance pupil does not move, so that the parallax error due to the movement of the center of the entrance pupil can be reduced.

The operation procedure of parallax correction in consideration of the movement of the center of the entrance pupil in step 204 of the operation procedure of the stereo camera 1 (FIG. 2) according to the embodiment of the present invention shown in FIG. 1 will be described with reference to FIG. Here, the description of steps 302 to 304, step 307, step 308, and step 311 already described is omitted.

Step 1201:
The parallax correction unit 121 does not move the parallax image from the parallax image storage unit 113 from the case where the center of the entrance pupil moves from the entrance pupil center movement information storage unit 116 by a predetermined distance L and by an incident angle θ of a predetermined main ray. The correction amount of the geometric correction information of the image pickup system unit 100a and the image pickup system unit 100b for the case is read.

Step 1202:
The disparity correction unit 121 has steps 302 and steps of the correction amount of the geometric correction information of the imaging system unit 100a from the case where the center of the incident pupil moves at each predetermined distance and at each incident angle of the predetermined main light beam to the case where the center of the incident pupil moves. By searching for the correction amounts of the four geometric correction information close to the distance and incident angle of a certain area on the reference image calculated in 304 and interpolating those values, the distance and incident angle of a certain area on the reference image are searched. Calculates the correction amount of the geometric correction information in. The position where the position of a certain area on the reference image is changed in the optical axis direction by the amount of correction is calculated. The above processing is carried out for all areas on the reference image.

In other words, the parallax correction unit 121 is based on the correction amount Cx of the geometric correction information corresponding to the combination of each distance L of the object stored in the entrance pupil center movement information storage unit 116 and each incident angle θ. Therefore, the position (first position) of the object in the reference image (first image) when it is assumed that the center of the entrance pupil does not move is calculated.

Step 1203:
The parallax correction unit 121 has steps 302 and steps of the correction amount of the geometric correction information of the imaging system unit 100b from the case where the center of the incident pupil moves at each predetermined distance and at each incident angle of the predetermined main light beam to the case where the center of the incident pupil moves. By searching for the correction amounts of the four geometric correction information close to the distance and incident angle of a certain region on the image calculated in 308 and interpolating those values, the distance and incident angle of a certain region on the reference image are obtained. Calculate the correction amount of the geometric correction information. The position where the position of a certain area on the reference image is changed in the optical axis direction by the amount of correction is calculated. The above processing is performed for all areas on the reference image.

In other words, the parallax correction unit 121 is based on the correction amount Cx of the geometric correction information corresponding to the combination of each distance L of the object stored in the entrance pupil center movement information storage unit 116 and each incident angle θ. Therefore, the position (second position) of the object in the reference image (second image) when it is assumed that the center of the entrance pupil does not move is calculated.

(Modification 1-3)
In step 206 of the operation procedure (FIG. 2) of the stereo camera 1 according to the embodiment of the present invention shown in FIG. 1, the geometric calibration unit 123 stores the reference image after the geometric correction from the geometric correction image storage unit 112 as a parallax image. The parallax image is read from the unit 113, and the geometric calibration unit 123 extracts the parallax corresponding to each position in the vertical direction on the image of the left and right white lines, and approximates the position in the vertical direction on the image of the left and right white lines and the parallax. Calculate a straight line. From this approximate straight line, the parallax of the white line corresponding to the design value of the position in the vertical direction of the optical axis of the reference image is calculated.

Originally, the deviation of the white line corresponding to the design value of the position in the vertical direction of the optical axis of the reference image is zero, so the deviation of the white line corresponds to the correction amount in the horizontal direction of the reference image, and this correction amount is used as the reference image. As the horizontal movement amount of the reference image, the horizontal movement amount of the reference image is sent to the geometric correction change information storage unit 115, and the geometric correction change information storage unit 115 can accurately correct the misalignment even if the horizontal movement amount of the reference image is stored. Here, instead of the horizontal movement amount of the reference image, the sign of the correction amount may be reversed to correct the horizontal movement amount of the reference image.

It should be noted that the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the described configurations. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

Further, each of the above configurations, functions and the like may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be placed in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

The embodiment of the present invention may have the following aspects.

(1) On the reference image and the reference image when the intersection of the main ray and the optical axis (center of the incident pupil) moves for each incident angle and the two imaging units that capture the reference image and the reference image. A storage unit having geometric correction information (correction amount of each pixel) of the reference image and the reference image having an error by the difference between the position and the position on the reference image and the reference image when not moving. , An image correction unit that geometrically corrects the reference image and the reference image using the geometric correction information of the reference image and the reference image, and a parallax image generation unit that generates a disparity image. Equipped with a stereo camera.

(2) A parallax correction unit that corrects the parallax image by calculating the amount of movement of the center of the entrance pupil for each angle of incidence and the image shift between the case where the center of the entrance pupil moves and the case where the center of the entrance pupil does not move, using the above-mentioned parallax image. The stereo camera of (1).

(3) The stereo camera of (2) including the reference image and a geometric calibration unit that corrects the amount of horizontal movement of the reference image or the reference image by using the parallax image corrected by the parallax correction unit.

According to the above (1) to (3), by using the image correction information in which the positions on the image of the pinhole camera model match at infinity, the center of the entrance pupil moves for each incident angle of the main light beam. The image shift is reduced at infinity to reduce the distance error at a distance where a large error occurs even with a small parallax error.

In addition, the amount of movement of the center of the entrance pupil for each angle of incidence of the main ray and the parallax image are used to calculate and correct the parallax deviation for each distance, thereby moving the center of the entrance pupil for each angle of incidence of the main ray. Reduce parallax.

1 ... Stereo camera 100a ... Image pickup system unit 100b ... Image pickup system section 101a ... Optical element section 101b ... Optical element section 102a ... Image pickup element section 102b ... Image pickup element section 110 ... Calculation section 111 ... Picture pickup image storage section 112 ... Geometric correction image storage Unit 113 ... Disparity image storage unit 114 ... Geometric correction information storage unit 115 ... Geometric correction change information storage unit 116 ... Entrance pupil center movement information storage unit 1117 ... Synchronous signal generation unit 118a ... Reference image acquisition unit 118b ... Reference image acquisition unit Unit 119 ... Geometric correction unit 120 ... Misalignment calculation unit 121 ... Disparity correction unit 122 ... Recognition unit 123 ... Geometric calibration unit 130 ... Screen audio output unit 140 ... Control unit 401 ... Reference image 402 ... Reference image 403 ... Reference image template image 404 ... Searched image of reference image 405 ... Searched image that best matches the template image 1101 ... Optical axis 1102 ... Object 1103 ... Entrance pupil center 1104 that does not move ... Entrance pupil center that moves 1105 ... When the entrance pupil center does not move 1106 of the main ray from the object to the center of the entrance pupil 1106 ... The main ray from the object to the center of the entrance pupil when the center of the entrance pupil moves 1107 ... The center of the entrance pupil moving 1108 ... The object when the center of the entrance pupil does not move Main ray from 1109 to the center of the entrance pupil 1109 ... The main ray from the object to the center of the entrance pupil when the center of the entrance pupil moves 1110 ... Image 1111 ... The position of the object on the image when the center of the entrance pupil does not move 1112 ... Position of the object on the image when the center of the entrance pupil moves 1113 ... Image 1114 ... Position of the object on the image when the center of the entrance pupil does not move 1115 ... Object on the image when the center of the entrance pupil moves Position of

Claims (5)

A first image pickup unit that captures a first image of an object,
A second image pickup unit that captures a second image of the object, and a second image pickup unit.
A geometric correction unit that geometrically corrects the first image and the second image, and
A parallax calculation unit for calculating parallax from the geometrically corrected first image and the second image is provided.
The center of the entrance pupil is based on the three-dimensional position of the region on the object with respect to the first image pickup unit and the amount of movement of the center of the entrance pupil of the first image pickup unit according to the incident angle of the main ray. The first position of the area of the object in the first image when it does not move is calculated.
The center of the entrance pupil is based on the three-dimensional position of the region on the object with respect to the second image pickup unit and the amount of movement of the center of the entrance pupil of the second image pickup unit according to the incident angle of the main ray. A stereo camera comprising calculating a second position of a region on the object in the second image when it is assumed that it does not move. The stereo camera according to claim 1.
The geometric correction unit is
The position of the first image of the object when the center of the incident pupil moves according to the angle of incidence, and the object of the object when it is assumed that the center of the incident pupil does not move according to the angle of incidence. The geometric correction information of the first image, which has an error by the difference from the position of the first image and is used for the process of geometrically correcting the lens distortion, and the incident pupil center according to the incident angle. The difference between the position of the second image of the object when moving and the position of the second image of the object when it is assumed that the center of the incident pupil does not move according to the angle of incidence. The first image and the second image are geometrically corrected by using the geometric correction information of the second image, which has an error only and is used for the process of geometrically correcting the lens distortion . A stereo camera that features that. The stereo camera according to claim 1.
Based on each of the incident angles stored in the storage unit and the corresponding movement amount of the entrance pupil center, the movement amount of the entrance pupil center according to the entrance angle of the main ray of the object is calculated. A stereo camera that features that. The stereo camera according to claim 1.
A stereo camera characterized in that the difference in the horizontal direction between the first position and the second position is the corrected parallax. The stereo camera according to claim 1.
A stereo camera characterized by calculating the amount of movement of the center of the entrance pupil according to the angle of incidence of the main ray of the object by using a polynomial that derives the amount of movement of the center of the entrance pupil from the angle of incidence. ..

Images