JP6983740B2 - Stereo camera system and distance measurement method - Google Patents

Description

The present invention relates to a stereo camera system and a distance measuring method.

Development and improvement of automatic driving, automatic braking, automatic tracking, etc. are being promoted as user support functions in automobiles. In order to realize these user support functions, technology for detecting objects existing around the vehicle is commonly used. In particular, for autonomous driving, technology for detecting objects around the entire vehicle (360 ° circumference) is used. Is important to establish.

As a technique for detecting an object around the entire circumference of an automobile, a method using LIDAR (Laser Imaging Detection and Ranging), which detects the distance from the reflection return time of a laser pulse to an object, is known.

Further, a method of measuring the distance to an object using a stereo camera is also known. For example, in Patent Document 1, a plurality of stereo cameras are arranged on a vehicle body, and a stereo camera is used based on a plurality of obtained images. The construction machine to be calibrated is disclosed.

Japanese Unexamined Patent Publication No. 2014-215039

In the case of the method using LIDAR, since the image cannot be acquired only by LIDAR, it is necessary to use a camera capable of taking an image together in order to recognize what the detected object is. In addition, when LIDAR is used in the surrounding automobiles, the laser beams may interfere with each other.

In the construction machine described in Patent Document 1, a plurality of stereo cameras merely measure the distance measurement range of each of them individually. For example, by combining adjacent stereo cameras, the distance measurement accuracy may be improved. It is not used for various purposes.

The present invention has been made in view of such a situation, and an object of the present invention is to enable accurate distance measurement according to the distance of an object.

The present application includes a plurality of means for solving at least a part of the above problems, and examples thereof are as follows. In order to solve the above problems, the stereo camera system according to one aspect of the present invention captures an object existing in its own angle of view range, and has a plurality of range-finding cameras for measuring the object and the range-finding camera. The pair is a virtual stereo camera, and the virtual stereo camera is provided with an integrated unit for measuring the distance of the object based on the images captured by each of the pairs of the distance measuring cameras, and the virtual stereo camera is compared with the distance measuring camera. Therefore, it is characterized in that the distance measurement accuracy of the object existing farther than a predetermined distance is high.

According to the present invention, it is possible to measure the distance with high accuracy according to the distance of the object.

Issues, configurations and effects other than those described above will be clarified by the following description of the embodiments.

It is a figure which shows the structural example of the stereo camera system which is 1st Embodiment which concerns on this invention. It is a block diagram which shows the detailed configuration example of a stereo camera system. It is a figure which shows the example of the positional relationship between the 1st image pickup part and the 2nd image pickup part in a distance measuring camera. It is a figure which shows the example of the positional relationship of two distance measuring cameras which are regarded as a stereo camera. It is a figure which shows the distance measurement accuracy with respect to the distance of a single body of a stereo camera and a distance measurement camera. It is a figure which shows the distance measurement accuracy with respect to the distance of a single body of a stereo camera and a distance measurement camera. It is a figure which shows the distance measurement accuracy with respect to the distance of a single body of a stereo camera and a distance measurement camera. It is a flowchart explaining an example of operation by a stereo camera system. It is a figure which shows the configuration example of the vehicle-mounted stereo camera system which is the 2nd Embodiment which concerns on this invention. It is a figure which shows the angle of view range of each distance measuring camera in an in-vehicle stereo camera system. It is a figure which shows the range which adopts the distance measurement result of each distance measurement camera alone in an in-vehicle stereo camera system. It is a figure which shows the structural example of the stereo camera which can be adopted as a distance measuring camera. It is sectional drawing which shows the relationship between the upper side hyperboloid mirror, the outer peripheral side hyperboloid mirror, and the inner peripheral side hyperboloid mirror. It is a figure which shows an example of the parameter of each hyperboloid mirror.

Hereinafter, a plurality of embodiments according to the present invention will be described with reference to the drawings. In addition, in all the drawings for explaining each embodiment, in principle, the same members are designated by the same reference numerals, and the repeated description thereof will be omitted. Further, in the following embodiments, the constituent elements (including element steps and the like) are not necessarily essential unless otherwise specified or clearly considered to be essential in principle. Needless to say. In addition, when saying "consisting of A", "consisting of A", "having A", and "including A", other elements are excluded unless it is clearly stated that it is only that element. It goes without saying that it is not something to do. Similarly, in the following embodiments, when the shape, positional relationship, etc. of the constituent elements are referred to, the shape is substantially the same, except when it is clearly stated or when it is considered that it is not clearly the case in principle. Etc., etc. shall be included.

<Structure example of the stereo camera system according to the first embodiment of the present invention>
FIG. 1 shows a configuration example of a stereo camera system 1 according to a first embodiment of the present invention. The stereo camera system 1 includes distance measuring cameras 10a and 10b, and an integrated unit 20.

The range-finding camera 10a is, for example, a stereo having a first image pickup unit 11a (FIG. 2) and a second image pickup unit 13a (FIG. 2) arranged in a direction perpendicular to the road surface (direction perpendicular to the drawing). It consists of a camera. The ranging camera 10a captures the angle of view range AOVa and outputs the image obtained as the imaging result to the integration unit 20. Further, the distance measuring camera 10a has a function of measuring a distance (measuring the distance) of an object existing in the angle of view range AOVa as a single unit, and outputs the distance measuring result to the integrated unit 20. The ranging camera 10a can be realized by, for example, a TOF (time of flight) camera in addition to the stereo camera.

Similarly, the ranging camera 10b comprises, for example, a stereo camera having a first imaging unit 11b (FIG. 2) and a second imaging unit 13b (FIG. 2) arranged in a direction perpendicular to the road surface. The ranging camera 10b captures the angle of view range AOVb and outputs the image obtained as the imaging result to the integration unit 20. Further, the ranging camera 10b has a function of measuring an object existing in the angle of view range AOVb as a single unit, and outputs the ranging result to the integrated unit 20. The ranging camera 10b can be realized by, for example, a TOF camera in addition to a stereo camera.

Hereinafter, when it is not necessary to distinguish the distance measuring cameras 10a and 10b individually, the distance measuring camera 10 is simply referred to as a distance measuring camera 10.

The integration unit 20 regards a pair of ranging cameras 10 (distance measuring cameras 10a and 10b) as one stereo camera Sub and measures an object. The stereo camera Sub corresponds to the virtual stereo camera of the present invention.

That is, the integrated unit 20 acquires images captured by each of the distance measuring camera 10a and the distance measuring camera 10b, and performs stereoscopic matching processing using the acquired images to obtain the angle of view range AOVa of the distance measuring camera 10a and the distance measuring. The distance is measured for an object existing in the overlapping region Rab where the angle of view range AOVb of the camera 10b overlaps. As a matter of course, the stereo camera Sab cannot measure an object that does not exist in the overlapping region Rab.

Since the baseline length L2 (FIG. 4) of the stereo camera Sub is longer than the baseline length L1 (FIG. 4) of the distance measuring camera 10 alone, the distance measuring accuracy for an object existing farther than a predetermined distance can be improved. It can be raised more than the distance measuring camera 10 alone.

Therefore, the integrated unit 20 sets a predetermined distance at the boundary where the stereo camera Sab has higher distance measurement accuracy than the distance measurement camera 10 alone as the distance threshold THL, and among the objects existing in the overlapping region Rab, the distance measurement For objects farther than the distance threshold THL from the camera 10, the distance measurement result by the stereo camera Sab is adopted, and for objects existing in the regions Ra and Rb closer to the distance threshold THL, the measurement by the distance measurement camera 10 alone is adopted. Adopt the distance result. Further, the integration unit 20 adopts the distance measurement result of the distance measurement camera 10 alone for the object that cannot be distance-measured by the stereo camera Sub.

Further, the integration unit 20 performs self-calibration of the stereo camera Sub based on the difference between the accurate distance and the distance measurement result by measuring the distance of the object whose exact distance is known.

Although it is different in the case of FIG. 1, the range-finding camera 10b is arranged in the angle-of-view range AOVa of the range-finding camera 10a, and the range-finding camera 10b captured by the range-finding camera 10a is reflected in the integrated unit 20. The baseline length of the stereo camera Sab and its inclination may be determined and corrected based on the existing image.

Next, FIG. 2 is a block diagram showing a detailed configuration example of the stereo camera system 1. In the configuration example of FIG. 2, the distance measuring camera 10 is a stereo camera having a first imaging unit 11 and a second imaging unit 13.

The distance measuring camera 10 has a first image pickup unit 11, a distortion correction unit 12, a second image pickup unit 13, a distortion correction unit 14, and a distance calculation unit 15.

Hereinafter, the first image pickup unit 11 of the distance measuring camera 10a will be referred to as a first image pickup unit 11a by adding a to the reference numeral. The same applies to other components. The same applies to the ranging camera 10b.

The first imaging unit 11 has, for example, a condensing unit 31 (FIG. 3) made of a wide-angle lens, a fisheye lens, or the like, and a high-resolution image sensor 32 (FIG. 3) made of CMOS (Complementary Metal Oxide Semiconductor) or the like. In the first image pickup unit 11, the light of the object is focused on the high-resolution image sensor 32 by the light-collecting unit 31, and the first image is generated by photoelectric conversion by the high-resolution image sensor 32. The first image pickup unit 11 outputs the first image to the distortion correction unit 12. The distortion correction unit 12 corrects the image distortion occurring in the first image and outputs it to the distance calculation unit 15.

The second imaging unit 13 has, for example, a condensing unit 33 (FIG. 3) made of a wide-angle lens, a fisheye lens, or the like, and a low-resolution image sensor 34 (FIG. 3) made of CMOS or the like. In the second image pickup unit 13, the light of the object is focused on the low-resolution image sensor 34 by the light-collecting unit 33, and the second image is generated by photoelectric conversion by the low-resolution image sensor 34. The second image pickup unit 13 outputs the second image to the distortion correction unit 14. The distortion correction unit 14 corrects the image distortion occurring in the second image and outputs it to the distance calculation unit 15.

FIG. 3 shows an example of the positional relationship between the first image pickup unit 11 and the second image pickup unit 13 in the distance measuring camera 10. As shown in the figure, the first image pickup unit 11 and the second image pickup unit 13 are arranged apart by a baseline length L1 (for example, several centimeters) in a direction perpendicular to the road surface. Since the first image pickup unit 11 and the second image pickup unit 13 have a common angle of view range in the direction horizontal to the road surface, it is possible to suppress the occurrence of occlusion.

The high-resolution image sensor 32 of the first image pickup unit 11 and the low-resolution image sensor 34 of the second image pickup unit 13 have different resolutions. Therefore, the first image obtained from the high-resolution image sensor 32 has a higher resolution than the second image obtained from the low-resolution image sensor 34. Since the low-resolution image sensor 34 is cheaper than the high-resolution image sensor 32, the production cost can be reduced accordingly. However, the resolutions of the high-resolution image sensor 32 and the low-resolution image sensor 34 may be unified.

Return to FIG. The distance calculation unit 15 calculates the distance to the object shown in both the first image and the second image by the stereo matching process based on the first image and the second image corrected for the image distortion, and determines the distance. Output to unit 23.

Since the range-finding camera 10b is configured in the same manner as the range-finding camera 10a, the description thereof will be omitted.

FIG. 4 shows an example of the positional relationship between the first imaging unit 11a and the second imaging unit 13a of the ranging camera 10a and the first imaging unit 11b and the second imaging unit 13b of the ranging camera 10b.

The range-finding camera 10a and the range-finding camera 10b forming the stereo camera Sub are arranged in a direction horizontal to the road surface, for example, a few meters apart. Therefore, the baseline length L2 of the stereo camera Sab is also several meters, which is longer than the baseline length L1 of the distance measuring camera 10 alone. Therefore, the stereo camera Sab can realize high distance measurement accuracy as compared with a single distance measurement camera 10.

Return to FIG. The integration unit 20 is composed of, for example, a CPU (Central Processing Unit), a memory, a communication I / F, and the like, and realizes each functional block of the image acquisition unit 21, the distance calculation unit 22, and the distance determination unit 23.

The image acquisition unit 21 acquires a high-resolution first image a from the distortion correction unit 12a of the range-finding camera 10a, acquires a high-resolution first image b from the distortion correction unit 12b of the range-finding camera 10b, and calculates the distance. Output to unit 22. The distance calculation unit 22 calculates the distance to the object shown in both the first images a and b by stereo matching processing based on the two first images a and b, and outputs the distance to the distance determination unit 23. ..

The image acquisition unit 21 acquires low-resolution second images a and b from the distance measuring cameras 10a and 10b, and the distance calculation unit 22 calculates the distance to the object based on the second images a and b. You may try to do it. However, the stereo matching process can be performed with higher accuracy by using the high-resolution first images a and b than by using the low-resolution second images a and b, so that the obtained measurement can be obtained. The distance result can also be obtained with higher accuracy.

The distance determination unit 23 adopted the distance measurement result of the distance measurement camera 10 alone for the object whose measurement result was less than the distance threshold THL (for example, 10 m), and the measurement result was the distance threshold THL or more. For the object, it is determined that the distance measurement result by the stereo camera Sab, that is, the calculation result by the distance calculation unit 22 is adopted. Further, the distance determination unit 23 determines that the distance measurement result of the distance measurement camera 10 alone is adopted even for an object that cannot be distance-measured by the stereo camera Sub. Further, in the distance determination unit 23, for some reason (when stereo matching cannot be performed, in the case of failure, etc.), one of the distance measuring camera 10 and the stereo camera Sub cannot obtain the distance measuring result, and the other is the distance measuring result. If is obtained, it is determined that the other distance measurement result is adopted. Then, the distance determination unit 23 outputs the distance measurement result corresponding to the determination result to a predetermined device (not shown) or the like.

Regarding the distance of the object existing in the overlapping region Rab that can be measured by both the distance measuring camera 10 alone and the stereo camera Sab, the distance measurement result by the distance measurement camera 10 alone and the distance measurement result by the stereo camera Sab are obtained. A predetermined calculated value using the above (for example, a weighted average value) may be adopted.

The distance measurement result of the object output to the predetermined device can be displayed on a display as a 3D image, supplied to an ECU (Electronic Control Unit) of an automobile equipped with a stereo camera system 1, or sent to a predetermined server. You may send it.

Next, FIGS. 5 to 7 show the distance measurement accuracy with respect to the distance to the object by the stereo camera Sab and the distance measuring camera 10 alone. The horizontal axis of each figure shows the actual distance to the object to be distanced, and the vertical axis shows the distance measurement accuracy. Here, the distance measurement accuracy indicates the error of the distance measurement result with respect to the actual distance as a percentage, and in each figure, the smaller the value, the higher the distance measurement accuracy.

In FIG. 5, for an object existing closer than a predetermined distance (distance threshold THL) indicated by a broken line parallel to the vertical axis, the stereo camera Sub cannot measure the distance by occlusion and exists farther than the predetermined distance. As for the object to be used, the stereo camera Sab shows the case where the distance measuring accuracy is higher than that of the distance measuring camera 10 alone. In such a case, the predetermined distance is set as the distance threshold THL, and the distance measurement result by the distance measuring camera 10 alone is adopted for the object existing closer than the distance threshold THL, and the object existing farther than the distance threshold THL is adopted. For, the distance measurement result by the stereo camera Sab is adopted.

In FIG. 6, for an object existing closer than a predetermined distance (distance threshold THL) indicated by a broken line parallel to the vertical axis, the distance measuring camera 10 alone has higher distance measuring accuracy than the stereo camera Sab. For an object that exists farther than the distance threshold THL, the case where the stereo camera Sub has higher distance measurement accuracy than the distance measurement camera 10 alone is shown. Even in such a case, a predetermined distance is set as the distance threshold THL, and for an object existing closer than the distance threshold THL, the distance measurement result by the distance measuring camera 10 alone is adopted, and the object exists farther than the distance threshold THL. For the object, the distance measurement result by the stereo camera Sab is adopted.

FIG. 7 shows the distance measurement accuracy before and after self-calibration by the integrated unit 20. After self-calibration, the predetermined distance indicated by the broken line parallel to the vertical axis, in which the superiority or inferiority of the distance measurement accuracy of the distance measurement camera 10 alone and the distance measurement accuracy of the stereo camera Sub is reversed, changes. Later, the distance threshold THL will also be changed. In the figure, only the stereo camera Sub is self-calibrated, but the range-finding camera 10 may be self-calibrated as a single unit.

<Operation of stereo camera system 1>
Next, FIG. 8 is a flowchart illustrating an example of operation by the stereo camera system 1.

This operation is started, for example, in response to a predetermined operation from the user.

First, the distance measuring cameras 10a and 10b measure the distance of an object existing in their own angle of view range as a single unit, and output the distance measuring result to the integrated unit 20 (step S1).

Specifically, in the distance measuring camera 10a, the first image pickup unit 11a captures the angle of view range AOVa, the first image a obtained as a result is output to the distortion correction unit 12a, and the distortion correction unit 12a is the first image. The image distortion generated in a is corrected and output to the distance calculation unit 15a. At the same time, the second image pickup unit 13a images the angle of view range AOVa, outputs the second image a obtained as a result to the distortion correction unit 14a, and the distortion correction unit 14a produces an image in the second image a. The distortion is corrected and output to the distance calculation unit 15a. Further, the distance calculation unit 15a is the distance to the object shown in both the first image a and the second image a by the stereo matching process based on the first image a and the second image a corrected for the image distortion. Is calculated and output to the distance determination unit 23.

The range-finding camera 10b also performs the same operation as the range-finding camera 10a described above at the same time, but the description thereof will be omitted.

Next, the integrated unit 20 regards the ranging cameras 10a and 10b as one stereo camera Sub and measures the distance of the object (step S2).

Specifically, the image acquisition unit 21 of the integration unit 20 receives the high-resolution first image a from the distortion correction unit 12a of the distance measuring camera 10a and the high-resolution first image from the distortion correction unit 12b of the distance measuring camera 10b. b is acquired and output to the distance calculation unit 22. Then, the distance calculation unit 22 is reflected in both the first images a and b by the stereo matching process based on the high-resolution first images a and b corrected for the image distortion, that is, the overlapping region Rab (FIG. The distance to the object existing in 1) is calculated and output to the distance determination unit 23.

Next, the distance determination unit 23 determines that the distance measurement result by the stereo camera Sub is adopted for the object whose measurement result is the distance threshold THL or more, and the distance measurement is performed for the object whose measurement result is less than the distance threshold THL. It is determined that the camera 10a or 10b adopts the distance measurement result calculated by itself. Then, the distance determination unit 23 outputs the distance measurement result corresponding to the determination result to a predetermined device (not shown) or the like (step S3).

After that, the process returns to step S1, and for example, steps S1 to S3 are repeatedly executed until an operation instructing the end of the operation is input from the user.

As described above, according to the stereo camera system 1, it is possible to accurately measure an object existing in the angle of view range of the two distance measuring cameras 10 according to the distance.

<Structure example of an in-vehicle stereo camera system according to a second embodiment of the present invention>
Next, FIG. 9 shows a configuration example of the vehicle-mounted stereo camera system 2 according to the second embodiment of the present invention. Of the components of the in-vehicle stereo camera system 2, the components common to the components of the stereo camera system 1 (FIG. 2) are designated by the same reference numerals, and the description thereof will be omitted as appropriate.

The in-vehicle stereo camera system 2 is mounted on the automobile 40. The in-vehicle stereo camera system 2 is for detecting an object existing all around the automobile 40.

The in-vehicle stereo camera system 2 includes distance measuring cameras 10a, 10b, 10c, 10d, and integrated units 20ab, 20bc, 20cd, 20ad.

The ranging cameras 10a to 10d are arranged at the four corners of the automobile 40. That is, for example, the ranging camera 10a is arranged in the right front corner of the automobile 40 toward the right front, the ranging camera 10b is arranged in the right rear corner of the automobile 40 toward the right rear, and the ranging camera 10c is arranged in the automobile 40. The distance measuring camera 10d is arranged in the left rear corner of the 40 toward the left rear direction, and the ranging camera 10d is arranged in the left front corner of the automobile 40 toward the left front direction. However, the arrangement of the distance measuring cameras 10a to 10d is not limited to the above-mentioned example, and may be arranged in a portion protruding from the vehicle body such as a door mirror.

The integrated unit 20ab, 20bc, 20cd, and 20ad are configured in the same manner as the integrated unit 20 (FIG. 2), respectively.

The integrated unit 20ab is connected to the ranging cameras 10a and 10b, and acquires the ranging results as a single unit from each of the ranging cameras 10a and 10b. Further, the integrated unit 20ab regards the distance measuring cameras 10a and 10b as stereo cameras and measures the object. Then, the integrated unit 20ab outputs the distance measurement result of each of the distance measurement cameras 10a and 10b as a single unit or the distance measurement result by the stereo camera Sab to the traveling control ECU 50.

The integrated unit 20bc is connected to the ranging cameras 10b and 10c, and obtains the ranging results as a single unit from each of the ranging cameras 10b and 10c. Further, the integrated unit 20bc regards the distance measuring cameras 10b and 10c as stereo cameras Sbc and measures the object. Then, the integrated unit 20bc outputs the distance measurement result of each of the distance measurement cameras 10b and 10c as a single unit or the distance measurement result by the stereo camera Sbc to the traveling control ECU 50.

The integrated unit 20cd is connected to the ranging cameras 10c and 10d, and obtains the ranging results as a single unit from each of the ranging cameras 10c and 10d. Further, the integrated unit 20cd regards the distance measuring cameras 10c and 10d as stereo cameras Scd and measures the object. Then, the integrated unit 20cd outputs the distance measurement result of each of the distance measurement cameras 10c and 10d as a single unit or the distance measurement result by the stereo camera Scd to the traveling control ECU 50.

The integrated unit 20ad is connected to the ranging cameras 10a and 10d, and acquires the ranging results as a single unit from each of the ranging cameras 10a and 10d. Further, the integrated unit 20ad regards the distance measuring cameras 10a and 10d as stereo cameras Sad and measures the distance of the object. Then, the integrated unit 20ad outputs the distance measurement result of each of the distance measurement cameras 10a and 10d as a single unit or the distance measurement result by the stereo camera Sad to the traveling control ECU 50.

Therefore, in the vehicle-mounted stereo camera system 2, four stereo cameras are realized by the four range-finding cameras 10.

The travel control ECU 50 included in the automobile 40 controls various functions in the automobile 40. For example, the travel control ECU 50 detects objects existing in the entire surroundings based on the distance measurement result input from each integrated unit 20, and realizes an automatic driving function, an automatic braking function, an automatic tracking function, and the like.

The integrated portions 20ab, 20bc, 20cd, and 20ad may be integrally formed with the distance measuring camera 10 to be connected. Further, the traveling control ECU 50 may be made to execute the operation by the integrated unit 20ab, 20bc, 20cd, 20ad.

Next, FIG. 10 shows the angle of view range of each ranging camera 10 in the in-vehicle stereo camera system 2.

Each distance measuring camera 10 in the in-vehicle stereo camera system 2 has a wider angle of view than each distance measuring camera 10 in the stereo camera system 1 (FIG. 1), and has an angle of view range of, for example, about 240 °.

When paying attention to the range-finding camera 10a arranged in the right front corner of the automobile 40, the angle-of-view range AOVa includes an overlapping region Rab overlapping with the angle-of-view range AOVb of the range-finding camera 10b arranged in the right rear corner of the automobile 40. And an overlapping region Rad that overlaps with the angle of view range AOVd of the range-finding camera 10d arranged in the left front corner of the automobile 40, and an occlusion region Ba that does not overlap the angle of view ranges AOVb and AOVd.

The object existing in the overlapping region Rab can be measured by the distance measuring camera 10a alone, and can be measured by the stereo camera Sab including the distance measuring cameras 10a and 10b. The object existing in the overlapping region Rad can be measured by the distance measuring camera 10a alone, and can be measured by the stereo camera Sad including the distance measuring cameras 10a and 10d. The object existing in the occlusion area Ba can be measured only by the distance measuring camera 10a alone. Therefore, as shown in FIG. 11, the general unit 20ab sets the distance threshold value THL that determines the range Ra for adopting the distance measurement result by the distance measurement camera 10a alone to the farthest distance from the distance measurement camera 10a to the occlusion region Ba. You can set it to a large value.

Then, the integrated unit 20ab determines that the distance measurement result by the stereo camera Sab is adopted for the object whose measurement result is the distance threshold value THL or more, and the distance measurement camera 10a is used for the object whose measurement result is less than the distance threshold value THL. It may be determined that the distance measurement result of the single unit is to be adopted, and the distance measurement result corresponding to the determination result may be output to the traveling control ECU 50.

The case of paying attention to the range-finding cameras 10b to 10d is the same as the case of paying attention to the range-finding camera 10a described above, and thus the description thereof will be omitted.

As described above, according to the in-vehicle stereo camera system 2, it is possible to accurately measure an object existing around the entire circumference of the automobile 40 according to the distance.

<Modification example of ranging camera 10>
FIG. 12 shows a configuration example of a stereo camera 101 that can be adopted as a distance measuring camera 10.

The stereo camera 101 includes an upper hyperboloid mirror 102, a lower hyperboloid mirror 103, an imaging optical system 104, an image sensor 105, a drive control unit 118, and an image processing unit 119.

The upper hyperboloid mirror 102 and the lower hyperboloid mirror 103 are mirrors having a convex hyperboloid. The upper hyperboloid mirror 102 and the lower hyperboloid mirror 103 have a common central axis, and are arranged vertically at predetermined intervals in the directions in which the vertices face each other.

The upper hyperboloid mirror 102 has an opening 106 at the apex portion.

The lower hyperboloid mirror 103 includes an outer peripheral hyperboloid mirror 107 having a common central axis and different curvatures, and an inner peripheral hyperboloid mirror 108. There may be a slight step at the boundary between the outer peripheral side hyperboloid mirror 107 and the inner peripheral side hyperboloid mirror 108.

The imaging optical system 104 is composed of one or more lenses and is arranged in the aperture 106. The imaging optical system 104 converges the light reflected by the lower hyperboloid mirror 103 and causes it to enter the image sensor 105.

The image sensor 105 is composed of, for example, CMOS, and under the control of the drive control unit 118, generates an output image 117 based on the light incident through the imaging optical system 104 and outputs the output image 117 to the drive control unit 118.

The drive control unit 118 controls the drive of the image sensor 105. Further, the drive control unit 118 supplies the output image 117 output by the image sensor 105 to the image processing unit 119. The image processing unit 119 generates the three-dimensional position information 120 of the subject 109 based on the supplied output image 117.

Here, the general properties of the hyperboloid mirror forming the upper hyperboloid mirror 102, the outer peripheral side hyperboloid mirror 107, and the inner peripheral side hyperboloid mirror 108 will be described.

The hyperboloids of the upper hyperboloid mirror 102, the outer peripheral side hyperboloid mirror 107, and the inner peripheral side hyperboloid mirror 108 each have a conical constant κ smaller than -1 in the quadric surface represented by the following equation (1). Corresponds to the case.

Here, z (r) in the equation (1) is the sag amount of the surface in the optical axis direction with the apex on the optical axis as the origin. c is the curvature on the optical axis (curvature on the axis). r is the radial coordinate from the optical axis.

In general, hyperboloids have two focal points. The coordinates of the two focal points are expressed by the following equation (2) with respect to the surface vertices.

The case where ± in Eq. (2) is + represents the coordinates of the focal point inside the hyperboloid. When ± in equation (2) is −, it represents the coordinates of the focal point outside the bicurved surface. Hereinafter, the focal point inside the hyperboloid is referred to as a first focal point, and the focal point outside the hyperboloid is referred to as a second focal point.

The hyperboloid mirror has a property of condensing the reflected light to the second focal point when the light ray directed to the first focal point is reflected by the hyperboloidal surface. On the contrary, the hyperboloid mirror has a property of reflecting the light emitted from the second focal point as if it were emitted from the first focal point.

Next, FIG. 13 is a cross-sectional view showing the relationship between the upper hyperboloid mirror 102, the outer peripheral side hyperboloid mirror 107, and the inner peripheral side hyperboloid mirror 108.

The relationship between the upper hyperboloid mirror 102, the outer peripheral hyperboloid mirror 107, and the inner peripheral hyperboloid mirror 108 is that the second focus 1022 of the upper hyperboloid mirror 102 is the first focus 1081 of the inner peripheral hyperboloid mirror 108. It is designed to be a close match. Further, this coincidence point does not have to substantially coincide with the first focal point 1071 of the outer peripheral side hyperboloid mirror 107, but is set to be located in the vicinity.

Further, the second focal point 1072 of the outer peripheral side hyperboloidal mirror 107 is configured to substantially coincide with the second focal point 1082 of the inner peripheral side hyperboloidal mirror 108. An imaging optical system 104 is arranged at this coincidence point. This coincidence point does not have to substantially coincide with the first focal point 1021 of the upper hyperboloid mirror 102, but is set to be located in the vicinity.

As a result, in the distance measuring camera 10 shown in FIG. 12, among the light 110 from the subject 109, the light 112 directed to the first focal point (upper viewpoint) 1021 of the upper hyperboloid mirror 102 is the upper hyperboloid mirror 102. Is reflected by and focused on the second focal point 1022 of the upper hyperboloid mirror 102.

The second focus 1022 of the upper hyperboloid mirror 102 substantially coincides with the first focus 1081 of the inner peripheral side hyperboloid mirror 108. Therefore, the light focused on the second focal point 1022 of the upper double-curved mirror 102 can be regarded as the light directed to the first focal point 1081 of the inner-circumferential double-curved mirror 108. It is reflected again by the mirror 108 and focused on the second focal point 1082 of the inner peripheral side double-curved mirror 108.

On the other hand, of the light 110 from the subject 109, the light 114 directed to the first focal point (lower viewpoint) 1071 of the outer peripheral side hyperboloid mirror 107 is reflected by the outer peripheral side hyperboloid mirror 107, and the outer peripheral side hyperboloid mirror 107 It is focused and reflected toward the second focal point 1072.

The second focal point 1072 of the outer peripheral side hyperboloid mirror 107 and the second focal point 1082 of the inner peripheral side hyperboloid mirror 108 substantially coincide with each other, and the imaging optical system 104 is arranged at this coincidence point. Therefore, the reflected light reflected by the upper double-curved mirror 102 and further reflected by the inner peripheral-side double-curved mirror 108 and the reflected light reflected by the outer peripheral-side double-curved mirror 107 are simultaneously imaged by the imaging optical system 104. It is incident on the sensor 105. As a result, in the image sensor 105, the image 115 of the subject 109 seen from the upper viewpoint (first focus) 1021 is projected on the inner peripheral side, and at the same time, the subject seen from the lower viewpoint (first focus) 1071 is projected on the outer peripheral side. It is possible to generate an output image 117 in which the image 116 of 109 is captured.

The output image 117 thus obtained is supplied to the image processing unit 119 via the drive control unit 118. The image processing unit 119 separates the output image 117 into an inner peripheral region in which the image 115 is captured and an outer peripheral region in which the image 116 is captured, and performs projective conversion processing on each of them, and the resulting 2 is obtained. Stereo matching processing is performed on the image to generate the three-dimensional position information 120 of the subject 109.

The generated three-dimensional position information 120 is supplied to the integration unit 20 (FIG. 1).

Some components of the stereo camera 101 described above, for example, the drive control unit 118 and the image processing unit 119, are executed independently of the stereo camera 101, for example, in the travel control ECU 50 (FIG. 9) of the automobile 40. You may let me.

<Other relationships between the upper hyperboloid mirror 102, the outer peripheral hyperboloid mirror 107, and the inner peripheral hyperboloid mirror 108>
In the stereo camera 101, the absolute value of the conical constant κ of the inner peripheral side hyperboloid mirror 108 is set to a value larger than the absolute value of the conical constant κ of the outer peripheral side hyperboloid mirror 107. As a result, in the output image 117, the sizes of the image 115 of the subject 109 seen from the upper viewpoint 1021 and the image 116 of the subject 109 seen from the lower viewpoint 1071 are aligned.

By matching the sizes of the image 115 on the inner peripheral side and the image 116 on the outer peripheral side in the output image 117, the resolutions of the image 115 and the image 116 can be matched, and the accuracy of the stereo matching process can be improved.

Further, in the stereo camera 101, a hyperboloid mirror having a larger diameter than the lower hyperboloid mirror 103 is adopted as the upper hyperboloid mirror 102. By making the diameter of the upper hyperboloid mirror 102 larger than that of the lower hyperboloid mirror 103, the light rays having the upward angle of view of the lower viewpoint 1071 can be received by the upper viewpoint 1021, and the upper and lower angles of view can be viewed stereoscopically. It is possible to secure the widest range as much as possible.

If the diameters of the upper hyperboloid mirror 102 and the lower hyperboloid mirror 103 are substantially the same, the lower hyperboloid mirror 103 is separated into an outer peripheral side hyperboloid mirror 107 and an inner peripheral side hyperboloid mirror 108. Therefore, the viewing range of the lower viewpoint 1071 is inevitably narrowed.

By the way, in the stereo camera 101, the upper hyperboloid mirror 102 having a large diameter and including the imaging optical system 104 is arranged on the upper side, and the lower hyperboloid mirror 103 having a smaller diameter is arranged on the lower side. Even if the vertical positional relationship is reversed, there is no practical problem.

However, when the stereo camera 101 is arranged at the four corners of the automobile 40, there is a problem that the angle of view range that can be viewed in stereo is not so wide. In particular, in order to use the stereo camera 101 for ambient sensing applications for fully automatic driving, it is essential to take a picture of the vicinity of the automobile, and it is necessary to secure a downward angle of view range.

In the stereo camera 101, the light beam having a downward angle of view incident on the upper viewpoint 1021 can cover a wider range than the angle of view range incident on the lower viewpoint 1071. This makes it possible to take a picture of the vicinity of the automobile, although the monocular view is obtained from the upper viewpoint 1021. Images in the vicinity of the automobile are mainly required to detect white lines such as the center line drawn on the road surface, but since it is known that the white lines are drawn on the road surface, the distance should be measured even with monocular vision. Is possible. Therefore, it is effective to arrange the larger upper hyperboloid mirror 102 in which the imaging optical system 104 is arranged on the upper side.

Next, FIG. 14 shows an example of the radius of curvature r, the conical constant κ, the apex position, and the diameter of the upper hyperboloid mirror 102, the outer peripheral hyperboloid mirror 107, and the inner peripheral hyperboloid mirror 108.

In the example of the figure, the radius of curvature r of the upper hyperboloid mirror 102 is 15 mm, the conical constant κ is -3.1, the apex position is 0 mm, the diameter is 34 mm, and the radius of curvature r of the outer peripheral hyperboloid mirror 107 is -17 mm. The conical constant κ is -2.94, the apex position is -18.5056 mm, the diameter is 25 mm, the radius of curvature r of the inner peripheral side hyperboloid mirror 108 is -22 mm, the conical constant κ is -130, and the apex position is -18.5086 mm. , The diameter is 9 mm. However, FIG. 14 is only an example, and an arbitrary upper hyperboloid mirror 102, an outer peripheral side hyperboloid mirror 107, and an inner peripheral side hyperboloid mirror 108 can be adopted for the stereo camera 101.

Although each embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and includes various modifications. For example, an example of the above-described embodiment has been described in detail for the sake of clarity of the present invention, and the present invention is not limited to those having all the configurations described here. Further, it is possible to replace a part of the configuration of one example of a certain embodiment with the configuration of another example. It is also possible to add the configuration of another example to the configuration of one example of one embodiment. Further, it is also possible to add / delete / replace a part of the configuration of one example of each embodiment with another configuration. Further, each of the above configurations, functions, processing units, processing means and the like may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. In addition, the control lines and information lines in the figure indicate what is considered necessary for explanation, and do not necessarily indicate all. You can think of almost all configurations as interconnected.

1 ... Stereo camera system, 2 ... In-vehicle stereo camera system, 10 ... Distance measuring camera, 11 ... First image pickup unit, 12 ... Distortion correction unit, 13 ... Second image pickup unit , 14 ... distortion correction unit, 15 ... distance calculation unit, 20 ... integration unit, 21 ... image acquisition unit, 22 ... distance calculation unit, 23 ... distance determination unit 23, 31 ... Condensing unit, 32 ... Low resolution image sensor, 33 ... Condensing unit, 34 ... High resolution image sensor, 40 ... Automobile, 50 ... Travel control ECU, 101 ... Stereo camera, 102 ... upper double curved mirror, 103 ... lower double curved mirror, 104 ... imaging optical system, 105 ... image sensor, 106 ... aperture, 107 ... outer circumference Side double curved mirror, 108 ... Inner peripheral side double curved mirror, 109 ... Subject, 116 ... Image, 117 ... Output image, 118 ... Drive control unit, 119 ... Image processing unit , 120 ... 3D position information

Claims (15)

A plurality of ranging cameras that capture an object existing in its own angle of view and measure the object, and
A pair of the distance measuring cameras is a virtual stereo camera, and an integrated unit for measuring the distance of the object based on the images captured by each of the pairs of the distance measuring cameras is provided.
The ranging camera is a stereo camera that simultaneously captures a first image and a second image having different resolutions.
The integrated unit uses the pair of the stereo cameras forming the range-finding camera as the virtual stereo camera, and uses the object based on the first image having the higher resolution captured by each of the pairs of the stereo cameras. Measure the distance and
The virtual stereo camera is a stereo camera system characterized in that the distance measurement accuracy of the object existing at a distance longer than a predetermined distance is higher than that of the distance measurement camera. The stereo camera system according to claim 1.
The integrated part is
A distance determination unit that determines whether to adopt the distance measurement result of the distance measurement camera alone or the distance measurement result of the virtual stereo camera.
A stereo camera system characterized by having. A plurality of ranging cameras that capture an object existing in its own angle of view and measure the object, and
A pair of the distance measuring cameras is a virtual stereo camera, and an integrated unit for measuring the distance of the object based on the images captured by each of the pairs of the distance measuring cameras is provided.
The integrated part is
It has a distance determination unit for determining whether to adopt the distance measurement result by the distance measurement camera alone or the distance measurement result by the virtual stereo camera.
Based on the difference between the distance measurement result of an object whose exact distance is known and the accurate distance, the virtual stereo camera is self-calibrated.
The distance determination unit
With a predetermined distance as the distance threshold value, it is determined that the distance measurement result by the virtual stereo camera is adopted for the object existing farther than the distance threshold value, and the object existing closer than the distance threshold value is determined to be adopted. It was determined that the distance measurement result of the distance measurement camera alone would be adopted.
Depending on the results of the self-calibration, change the distance threshold value,
The virtual stereo camera is a stereo camera system characterized in that the distance measurement accuracy of the object existing at a distance longer than the predetermined distance is higher than that of the distance measurement camera. The stereo camera system according to claim 2 or 3.
The distance determination unit is a stereo camera system characterized in that it determines that an object existing in an area including at least an occlusion area of the virtual stereo camera is to adopt the distance measurement result of the distance measurement camera alone. The stereo camera system according to claim 2.
The distance determination unit determines that the predetermined distance is set as the distance threshold value and adopts the distance measurement result by the virtual stereo camera for the object existing farther than the distance threshold value, and exists closer than the distance threshold value. A stereo camera system characterized in that it is determined that a distance measurement result by a single distance measuring camera is adopted for the object. The stereo camera system according to claim 5.
The integrated unit self-calibrates the virtual stereo camera based on the difference between the distance measurement result of an object whose exact distance is known and the accurate distance.
The distance determination unit is a stereo camera system characterized in that the distance threshold value is changed according to the result of the self-calibration. The stereo camera system according to claim 2 or 3.
The stereo camera system is characterized in that, when one of the distance measurement result by the distance measurement camera alone or the distance measurement result by the virtual stereo camera cannot be obtained, the distance determination unit determines to adopt the other. The stereo camera system according to any one of claims 1 to 7.
The integrated unit is based on an image captured by one of the pair of ranging cameras and the other of the pair of ranging cameras is captured, and the distance measuring result of the pair of ranging cameras with respect to the other. , A stereo camera system characterized in that the baseline length of the virtual stereo camera and its inclination are determined. The stereo camera system according to claim 1.
A stereo camera system characterized in that the baseline length of the stereo camera forming the ranging camera is shorter than the baseline length of the virtual stereo camera. The stereo camera system according to claim 1.
A stereo camera system characterized in that the baseline length of the stereo camera forming the ranging camera and the baseline length of the virtual stereo camera are different in direction. The stereo camera system according to claim 10.
A stereo camera system characterized in that the baseline length of the stereo camera forming the ranging camera and the baseline length of the virtual stereo camera are orthogonal to each other. The stereo camera system according to claim 11.
The baseline length of the stereo camera forming the distance measuring camera is in the direction perpendicular to the road surface.
A stereo camera system characterized in that the baseline length of the virtual stereo camera is in a direction horizontal to the road surface. The stereo camera system according to claim 1.
The stereo camera forming the distance measuring camera is
The first and second convex hyperboloid mirrors, which have a common central axis and are arranged vertically with their vertices facing each other,
An imaging optical system arranged in an opening provided at the apex portion of the first convex hyperboloid mirror, and
The light of the subject is reflected by the first convex double-curved mirror, then reflected by the second convex double-curved mirror, and then incident through the imaging optical system, and the light of the subject. Is reflected by the second convex double-curved mirror, and then two images corresponding to each other are simultaneously projected when the subject is imaged from two different viewpoints based on the light incident through the imaging optical system. An image sensor that generates an output image and
A stereo camera system characterized by having. It is a method of measuring the distance of a stereo camera system.
An object existing in its own angle of view range is imaged, and the pair of ranging cameras that measure the object is set as a virtual stereo camera, and the object is captured based on the images captured by each pair of the ranging cameras. Including the distance measuring step to measure the distance,
The ranging camera is a stereo camera that simultaneously captures a first image and a second image having different resolutions.
In the distance measuring step, the pair of the stereo cameras forming the distance measuring camera is the virtual stereo camera, and the object is based on the first image of the higher resolution imaged by each of the pairs of the stereo cameras. To measure the distance,
The virtual stereo camera is a distance measuring method characterized in that the distance measuring accuracy of the object existing at a distance longer than a predetermined distance is higher than that of the distance measuring camera. It is a method of measuring the distance of a stereo camera system.
An object existing in its own angle of view range is imaged, and the pair of ranging cameras that measure the object is set as a virtual stereo camera, and the object is captured based on the images captured by each pair of the ranging cameras. The distance measurement step to measure the distance and the distance measurement step
With a predetermined distance as the distance threshold value, it is determined that the distance measurement result by the virtual stereo camera is adopted for the object existing farther than the distance threshold value, and the object existing closer than the distance threshold value is determined to be adopted. A determination step for determining that the distance measurement result of the distance measurement camera alone is adopted, and
The virtual stereo camera is self-calibrated based on the difference between the distance measurement result of an object whose exact distance is known and the accurate distance, and the distance threshold is changed according to the self-calibration result. Including change steps and
The virtual stereo camera has higher distance measurement accuracy of the object existing at a distance than the predetermined distance as compared with the distance measuring camera.
A distance measuring method characterized by that.

Images