JP6251099B2 - Distance calculation device - Google Patents

Description

  The present invention relates to a distance calculation device that calculates a distance to an object using images taken by a plurality of cameras.

  Calculate the distance to the object using the principle of triangulation based on the parallax between the pixel block in one image taken by the in-vehicle stereo camera and the pixel block in the other image correlated with the pixel block Has been done. In order to calculate the distance accurately, it is desirable that there is no positional shift other than parallax in the stereo camera image, but the positional shift caused by the positional shift caused by the mounting accuracy of the stereo camera, vibration, etc. There is a gap. Particularly, the translational deviation in the horizontal direction appears as a parallax error, which causes a distance calculation error. In view of this, there has been proposed an apparatus that corrects an error caused by translational displacement in the horizontal direction of a stereo camera and calculates a distance to an object.

  For example, Patent Document 1 calculates a first vanishing point from the intersection of approximate straight lines extending in the distance direction in one image plane obtained by a stereo camera, and similarly from the other image plane. A second vanishing point is calculated. Then, the vanishing point parallax is calculated from the shift amount between the first vanishing point and the second vanishing point, and the calculated distance is corrected using the calculated vanishing point parallax.

JP 2003-83742 A

  In Patent Document 1, white lines extending in the distance direction exist on the left and right sides of the road, and pairs of white lines on the left and right are often parallel to each other. Therefore, approximate straight lines calculated from the white lines extending in the distance direction are parallel to each other. Assuming that the vanishing point parallax is calculated. However, when the road surface is wavy, there is a possibility that the approximate straight line pair is not parallel to each other. If the approximate straight line pair includes those that are not parallel to each other, the calculation accuracy of the vanishing point parallax may be lowered and the calculation accuracy of the distance may be reduced.

  In view of the above circumstances, the present invention has as its main object to provide a distance calculation device capable of improving the distance calculation accuracy.

  In order to solve the above problems, the present invention is a distance calculation device based on a plurality of image capturing devices that capture a surrounding environment including a road and a pair of images captured by the plurality of image capturing devices. Based on the parallax calculation means for calculating the parallax, the boundary line extraction means for extracting the edge points constituting the left and right boundary lines of the road in the image, and the edge points extracted by the boundary line extraction means, Approximate straight line calculating means for calculating an approximate straight line that is a line segment of a predetermined length that approximates the left and right boundary lines, and the approximate straight line calculating means for each vanishing point parallax while changing the value of the vanishing point parallax in the pair of images. Of the right and left approximate straight lines calculated by the above, the foot of a perpendicular line dropped from the first end of one of the approximate straight lines to the straight line including the other approximate straight line, and the vicinity of the other corresponding to the first end A degree of coincidence calculating means for calculating the degree of coincidence with the second end of the straight line, and the vanishing point parallax corresponding to the highest degree of coincidence calculated by the degree of coincidence calculating means A vanishing point parallax determining unit that determines the vanishing point parallax in the image, a parallax calculated by the parallax calculating unit, and a vanishing point parallax determined by the vanishing point parallax determining unit are calculated. And a distance calculating means.

  According to the present invention, the parallax is calculated based on a pair of images captured by a plurality of image capturing devices. In addition, edge points constituting the left and right boundary lines of the road are extracted from the image. Based on the extracted edge points, an approximate straight line that is a line segment of a predetermined length that approximates the left and right boundary lines is calculated.

  Here, if the vanishing point parallax is wrong, the left and right approximate straight lines that are not originally parallel may become parallel, or the originally parallel left and right approximate straight lines may not be parallel. Therefore, for each vanishing point parallax, while the value of the vanishing point parallax is changed, the foot of a perpendicular line dropped from the first end of one of the calculated approximate straight lines to the straight line including the other approximate straight line And the degree of coincidence between the second end of the other approximate line corresponding to the first end. Then, the vanishing point parallax corresponding to the highest matching degree among the calculated matching degrees is determined as the vanishing point parallax in the pair of images.

  When the left and right approximate lines are truly parallel in real space and the vanishing point parallax is a true value, the vertical leg that is lowered from the first end of one of the left and right approximate lines to the other, and the other Coincides with the second end of the approximate straight line, and the degree of coincidence is the highest. Therefore, the vanishing point parallax corresponding to the highest matching degree among the calculated matching degrees is selected as the correct vanishing point parallax and determined as the vanishing point parallax in the pair of images. Thereby, the calculation accuracy of vanishing point parallax can be improved. Further, the distance to the object is calculated based on the calculated parallax and the determined vanishing point parallax. Accordingly, the vanishing point parallax calculation accuracy can be improved, and the distance calculation accuracy can be improved.

The block diagram which shows the structure of the distance calculation apparatus which concerns on this embodiment. The figure which shows the parallax of a pair of image. (A) The figure which shows image coordinates. (B) An expression for converting image coordinates to real space coordinates. The figure which shows the predetermined area | region which extracts a boundary line. (A) The figure which shows the edge point which comprises a boundary line. (B) The figure which shows the edge point used for calculation of an approximate expression. (A) A figure showing a plurality of line segment pairs. (B) An approximate expression of the right line segment. (C) An approximate expression of the left line segment. The figure which shows the change of the parallelism of a line segment pair accompanying the change of vanishing point parallax. The figure which shows the coefficient of the approximate straight line which approximates each line segment on the left side. The figure which shows the perpendicular drawn from one line segment to the other line segment, and coincidence. The figure which shows a response | compatibility with the calculated coincidence and vanishing point parallax. (A) The figure which shows the length of a perpendicular. (B) An expression for calculating a deviation between the interval between line segment pairs and the width of the road. The figure which shows the vanishing point parallax calculated from the image image | photographed at the some time. The time chart which shows the time change of the calculated vanishing point parallax. The flowchart which shows the process sequence which calculates vanishing point parallax.

  Hereinafter, an embodiment embodying a distance calculation device will be described with reference to the drawings. First, with reference to FIG. 1, the structure of the distance calculation apparatus 10 which concerns on this embodiment is demonstrated. The distance calculation device 10 includes a pair of image capturing devices 20 and an image processing device 30.

  The pair of image capturing devices 20 is a stereo camera with a built-in image sensor such as a CCD or CMOS sensor, or an infrared camera. One is a main camera 21 and the other is a sub camera 22. The main camera 21 and the sub camera 22 are attached to the right end and the left end of the rearview mirror with a predetermined camera base line length in the vehicle width direction, respectively. Take a picture.

  The main camera 21 outputs a reference image signal, and the sub camera 22 outputs a comparison image signal. The main camera 21 and the sub camera 22 are synchronized with each other, capture the surrounding environment at the same timing, and output a pair of analog image data to the image processing device 30.

  The image processing apparatus 30 includes a microcomputer configured by a CPU, a RAM, a ROM, an I / O, and the like, an A / D conversion circuit, an image parallelizing unit 40, a boundary line extracting unit 50, a parallax calculating unit 60, a correction. The functions of the quantity calculation unit 70 and the distance calculation unit 80 are realized.

  The image collimator 40 converts the pair of analog image data input from the image capturing device 20 into digital image data having a predetermined luminance gradation by an A / D conversion circuit. Further, the image collimating unit 40 corrects the pair of digital image data with the posture of the main camera 21 and the sub camera 22 and the distortion coefficient of the lens, and parallelizes the pair of digital images. Thereby, the epipo line (horizontal line) in a pair of digital images, that is, a stereo image is coincident, and the epipo line is parallel to the u axis (see FIG. 3A). The coincidence of epipolines in a stereo image is necessary to ensure the accuracy of stereo matching. The stereo image corrected by the image collimator 40 is output to the parallax calculator 60 and the boundary line extractor 50.

  The parallax calculation unit 60 calculates the parallax d based on the stereo image. The parallax d indicates the position of the object in the image (standard image) of the main camera 21 and the object in the image (reference image) of the sub camera 22 when the same object is photographed by the main camera 21 and the sub camera 22. It is the difference from the position. The disparity calculating unit 60 calculates one disparity d for each pixel block of 4 × 4 pixels in the reference image, for example.

  As shown in FIG. 2, the parallax d increases as the distance from the image capturing device 20 increases, and decreases as the distance increases. Theoretically, the vanishing point parallax d0 that is the parallax at the vanishing point of the stereo image is zero. That is, theoretically, the positions of the vanishing points coincide in the stereo image.

  The image capturing device 20 is configured such that the vanishing point parallax d0 becomes 0 in the parallelized stereo image when initially mounted on the vehicle. However, the position of the image capturing device 20 may be shifted due to vibrations of the vehicle and the vanishing point parallax d0 may not be zero. When the vanishing point parallax d0 is small, the stereo image matching rate is not affected, but the distance to the object calculated using the triangulation principle is affected. For this reason, when calculating the distance to the object using the principle of triangulation, it is necessary to correct the parallax d by the vanishing point parallax d0 (see FIG. 3B). A detailed method for calculating the vanishing point parallax d0 will be described later.

  The distance calculation unit 80 calculates the distance from the vehicle to the object based on the parallax d calculated by the parallax calculation unit 60 and the vanishing point parallax d0 determined by the vanishing point parallax determination unit 73 described later. Specifically, as shown in FIG. 3A, when the coordinate axes u and v are set in the reference image, the distance calculation unit 80 determines the coordinates (u, v, d) of the object in the reference image as shown in FIG. As shown in b), the vanishing point parallax d0 is used to convert to real space coordinates (X, Y, Z). Then, the distance calculation unit 80 calculates the distance in the real space from the coordinates (X, Y, Z) of the object in the real space.

  Next, a method for calculating the vanishing point parallax d0 will be described. In order to obtain the vanishing point parallax d0 with high accuracy, it is necessary to find a set of straight lines parallel to each other in the real space. Therefore, the boundary line extraction unit 50 and the correction amount calculation unit 70 extract edge points constituting the left and right boundary lines, and calculate a plurality of approximate straight line pairs based on the extracted edge points. Further, while changing the vanishing point parallax d0, the line segment pair having the highest parallelism is selected from the plurality of calculated line segment pairs for each vanishing point parallax d0. The degree of coincidence M between the leg of the perpendicular line drawn from one end A of the selected line segment pair to the straight line including the other line segment and the end B of the other line segment corresponding to the end A is calculated. Then, the vanishing point parallax d0 corresponding to the line segment pair having the highest matching degree M is set as the vanishing point parallax d0 in the stereo image. This will be described in detail below.

  The boundary line extraction unit 50 acquires, for example, one point for each pixel block of 16 × 64 pixels within a predetermined region of the reference image (see FIG. 4), and performs a three-dimensional Hough transform based on the acquired points. To calculate the road surface formula. In addition, the boundary line extraction unit 50 extracts edge points constituting the left and right boundary lines of the road in the reference image. Specifically, the boundary line extraction unit 50 searches for an up-edge point in the horizontal direction from the left end in the reference image. And the boundary line extraction part 50 detects the straight line showing the white line pair on a road by two-dimensional Hough transformation based on the searched edge point. The straight line representing the white line pair is a straight line having the largest number of edge points from each of the positive and negative straight lines in the reference image.

  Further, as shown in FIG. 4, a predetermined area near the image capturing device 20 is set as an edge extraction area. This predetermined area includes a line segment representing the right broken white line detected by the two-dimensional Hough transform and a straight line representing the left detected white line. In order to extract edge points with high accuracy, the boundary line extraction unit 50 has a higher edge strength for pixels having edges in directions perpendicular to the approximate inclination of the left and right white lines detected by the two-dimensional Hough transform. The filter that becomes stronger is applied to the luminance data of the reference image, and the left and right edge points are respectively extracted (hatched portions in FIG. 5A). Further, the boundary line extraction unit 50 extracts edge points having edge strengths equal to or higher than a threshold value from among the left and right edge points calculated by filtering as edge points constituting the left and right boundary lines of the road (FIG. 5). (Outlined part of (b)). Thus, by searching for the edge point in the direction perpendicular to the white line, the outline of the white line is detected with higher accuracy than when searching for the edge point in the horizontal direction of the image.

  The correction amount calculation unit 70 includes an approximate straight line calculation unit 71, a coincidence calculation unit 72, a vanishing point parallax determination unit 73, and a reliability calculation unit 74, and calculates a correction amount of the parallax d, that is, the vanishing point parallax d0.

  The reliability calculation unit 74 calculates a higher reliability for the image data of the pixel that has captured the vicinity of the image capturing device 20. The reliability corresponds to the probability that the image data of a certain pixel is a true value. In general, the probability of a true value is higher as the pixel image data is captured in the vicinity of the image capturing device 20, and the probability of a true value is lower as the pixel image data is captured far away.

  The approximate straight line calculation unit 71 calculates the approximate straight line of the left and right white lines based on the left and right edge points whose edge intensity extracted by the boundary line extraction unit 50 is equal to or greater than the threshold value. The approximate straight line calculation unit 71 first calculates an approximate straight line (first straight line) that approximates a line segment that represents a white broken line on the right side. Specifically, the approximate line calculation unit 71 weights the right edge point extracted by the boundary line extraction unit 50 with the reliability calculated by the reliability calculation unit 74. Then, the approximate straight line calculation unit 71 calculates an approximate straight line formula (see FIG. 6B) that approximates the right line segment by the least square method for the weighted right edge point. αrx, αry, βrx, and βry are coefficients.

  Next, the approximate straight line calculation unit 71 cuts out a plurality of line segments having the same length as the right line segment from the left edge point extracted by the boundary line extraction unit 50, as shown in FIG. In actual roads, there are few long straight roads, and there are many roads that are curved or undulating. If the white line of a curved or wavy road is approximated by a long straight line, there is a risk that straight lines parallel to each other may not be found in real space. Therefore, the approximate straight line calculation unit 71 cuts out a plurality of short line segments having the same length as the right line segment from the edge points constituting the long left straight line.

  Then, the approximate straight line calculation unit 71 weights the left edge point extracted by the boundary line extraction unit 50 with reliability, and the left edge point weighted with reliability uses a least square method to obtain a plurality of line segments. An approximate straight line (second straight line) equation (see FIG. 6C) is calculated. Specifically, the approximate straight line calculation unit 71 cuts out a line segment having the same length as the right line segment while shifting the left edge point extracted by the boundary line extraction unit 50 pixel by pixel from the near side to the back direction. Then, an approximate straight line equation that approximates each line segment is calculated. In the present embodiment, it is assumed that 80 line segments are cut out on the left side. In this case, 80 line segment pairs are formed by the left line segment and the right line segment. In this case, 80 coefficients of the approximate straight line on the left side, αlx, αly, βlx, and βly are each calculated.

  When both the left and right white lines are long straight lines, a predetermined length line segment is cut out from the edge points constituting one of the white lines, and an approximate straight line is calculated. Then, a plurality of line segments of a predetermined length are cut out from the edge points constituting the other white line, and approximate lines for each line segment are calculated.

  Here, when the vanishing point parallax d0 is changed to an incorrect value in an approximate straight line approximating a pair of line segments parallel to each other in real space (see FIG. 7A), the line segment pairs are not parallel (FIG. 7 ( b)). Conversely, in an approximate straight line that approximates a pair of line segments that are not parallel in real space, the line segment pair may be parallel by changing the vanishing point parallax d0 to an incorrect value. When the line segment pair is parallel with the wrong vanishing point parallax d0, the position of the line segment pair in the traveling direction shifts, so it is lowered from the end near one line segment to a straight line including the other line segment. The leg of the vertical line is shifted from the end on the near side of the other line segment. On the other hand, when the line segment pair is parallel with the true vanishing point parallax d0, the leg of the perpendicular line dropped from the end near one line segment to the straight line including the other line segment, and the other There exists a line segment pair whose end on the vicinity side of the line segment coincides. Such a line segment pair is a line segment pair that is truly parallel in real space. Then, by changing the value of the vanishing point parallax d0, a pair of line segments closest to a truly parallel line segment pair in the real space is searched, and the vanishing point parallax d0 corresponding to the found line segment pair is determined as stereo. The vanishing point parallax d0 of the image is determined.

  Therefore, the approximate straight line calculation unit 71 calculates an approximate straight line on the right line segment and calculates an approximate straight line on the left side of the plurality of line segments. Further, the approximate straight line calculation unit 71 smoothes the calculated coefficients of the plurality of approximate straight lines on the left side, reduces singular points, and brings the approximate straight line coefficients close to the true value. Specifically, smoothing is performed by arranging coefficients in order from the line segment on the front side. FIG. 8A is a diagram in which the coefficients αlx of a plurality of approximate straight lines on the left are arranged in order from the front side. The approximate line calculation unit 71 smoothes the coefficient αlx in order from the smallest line segment pair number.

  Any smoothing method may be used. For example, linear interpolation is performed on the coefficient αlx shown in FIG. 8A, and smoothing is performed by performing a moving average on the linearly interpolated coefficient αlx. FIG. 8B shows the smoothed coefficient αlx. By smoothing in this way, a coefficient αlx that smoothly changes corresponding to the smooth slope of the actual road is obtained. Similarly, the approximate line calculation unit 71 smoothes the coefficients αly, βlx, βly as well. Thus, an approximate straight line of 80 line segment pairs is calculated.

  The degree of coincidence calculation unit 72 changes the value of the vanishing point parallax d0 from −3 to 3 in steps of 0.05 [pix], and for each vanishing point parallax d0, the line segment pair calculated by the approximate line calculation unit 71 The pair of line segments with the highest parallelism is determined from among the above. Specifically, for each vanishing point parallax d0, the degree-of-match calculation unit 72 determines the line segment pair that satisfies αlxd0 + βlx = αrxd0 + βrx and αlyd0 + βly = αryd0 + βry (parallel condition) as the line segment pair with the highest parallelism. On straight roads, multiple line pairs satisfy the parallel condition. When a plurality of line segment pairs satisfy the parallel condition, the plurality of line segment pairs are determined as the line segment pair having the highest parallelism.

  Then, the matching degree calculation unit 72 calculates the matching degree M of the determined line segment pair for each vanishing point parallax d0. As shown in FIG. 9A, when the left and right line segment pairs are parallel and the vanishing point parallax d0 is a true value, the left line segment from the end A (first end) of the right line segment is included. There is a line segment pair in which the leg of the perpendicular line drawn down toward the straight line coincides with the end B (second end) of the left line segment. The end A and the end B correspond to an end near the right line segment and an end near the left line segment, respectively. In this case, the length Li of the perpendicular line and the width W of the road are the same regardless of where the perpendicular line is drawn from the right line segment toward the approximate straight line on the left line segment.

  On the other hand, as shown in FIGS. 9B and 9C, when the value of the vanishing point parallax d0 is incorrect, the perpendicular is drawn from the end A of the right line segment toward the straight line including the left line segment. Is shifted from the end B of the left line segment. Therefore, the coincidence calculation unit 72 digitizes the amount of deviation between the leg of the perpendicular drawn from the end A and the end B, and sets this as the coincidence M. The smaller the deviation amount, the higher the matching degree M. It should be noted that since the data at the end on the near side has higher reliability than the data at the end on the far side of the line segment, it is preferable to use the end on the near side of the line segment for calculation of the matching degree M.

  The vanishing point parallax determining unit 73 selects the vanishing point parallax d0 corresponding to the highest matching degree M among the matching degrees M calculated by the matching degree calculating unit 72, that is, the line segment closest to the truly parallel line segment pair. The vanishing point parallax d0 corresponding to the pair is determined as the vanishing point parallax d0 in the stereo image. FIG. 10A is a diagram in which the degree of coincidence M of the pair of line segments selected for each vanishing point parallax d0 is plotted. FIG. 10B shows an enlarged view of the region where the degree of coincidence M is high in FIG. As shown in FIG. 10B, the coincidence degree M is highest when the vanishing point parallax d0 is 0.25. Therefore, in this case, the vanishing point parallax determining unit 73 determines the vanishing point parallax d0 in the stereo image as 0.25.

  Further, the vanishing point parallax determining unit 73 determines a vanishing point parallax d0 for each of the captured stereo images at a plurality of different time points that are temporally continuous, and based on the determined plurality of vanishing point parallaxes d0. The vanishing point parallax d0 used by the distance calculation unit 80 is statistically determined. The time interval for determining the vanishing point parallax d0 is set according to the processing capability of the image processing device 30.

  Specifically, for the vanishing point parallax d0 determined for each of the plurality of stereo images, the vanishing point parallax determining unit 73 determines the interval between the left and right line segment pairs corresponding to the vanishing point parallax d0, the width W of the road, and the like. Are grouped by, for example, the K-means method using the width W0 of the road with the smallest deviation (see FIG. 12). And a vanishing point parallax determination part determines to the vanishing point parallax d0 which the distance calculation part 80 uses based on the result statistics for every group. Specifically, the vanishing point parallax determination unit 73 obtains the variance of each group, excludes the group having a large variance as an outlier, and determines the average value of the group having a small variance as the vanishing point parallax d0 used by the distance calculation unit 80. To do.

  In the following, a method for calculating the width W0 of the road where the deviation between the vanishing point parallax d0 determined for each stereo image and the interval between the left and right line segment pairs corresponding to the vanishing point parallax d0 is the smallest will be described. The vanishing point parallax determining unit 73 performs parallel processing for each line segment pair based on the deviation between the length Li of the perpendicular line drawn from the right approximate line toward the straight line including the left approximate line and the width W of the road. The degree E is calculated. First, the vanishing point parallax determination unit 73 calculates the length Li of the perpendicular. Specifically, as shown in FIG. 11A, an intersection Pli (ul, vl, dl) between a vertical plane passing through a point Pri (ur, vr, dr) on the right approximate line and the left approximate line is represented. And the distance between the point Pri and the point Pli is calculated. This distance is the perpendicular length Li.

  The vanishing point parallax determination unit 73 lowers the perpendicular to the approximate straight line on the left side while shifting the position of the point Pri one pixel at a time between the vicinity end A and the far end on the right side of the line segment, The length Li of each perpendicular is calculated. Therefore, the length Li of the perpendicular is calculated for the number of pixels included in the right line segment. Note that a perpendicular line may be dropped to the approximate straight line on the left side while shifting the position of the point Pri by a predetermined number of pixels in the depth direction.

  Next, the vanishing point parallax determination unit 73 calculates the parallelism E using the formula shown in FIG. Specifically, the vanishing point parallax determination unit 73 calculates the square value of the deviation between the length Li of each perpendicular and the width W of the road, and calculates the square value of each deviation by the reliability calculation unit 74. Weighting is performed by the reciprocal of the reliability Vi. The vanishing point parallax determining unit 73 calculates the parallelism E by adding all the square values of the deviations weighted by the reciprocal of the reliability Vi.

  Pixel image data obtained by photographing the vicinity of the image photographing device 20 has higher information reliability. Therefore, in the vicinity of the image capturing device 20, the square value of the deviation is close to 0 when the line segment pair is parallel in the real space, but in the distance, the deviation of the deviation is even when the line segment pair is parallel in the real space. The square value may be larger than zero. Therefore, even for the same deviation, the square value of each deviation is weighted by the reciprocal of the reliability Vi so that the parallelism E in the vicinity is lower than that in the distance.

  The parallelism E becomes 0 when the line segment pair is parallel in the real space, and becomes larger as the line segment pair is separated from the parallel in the real space. That is, the parallelism E is an index representing the parallelism of the line segment pair, and the higher the value, the higher the parallelism.

  Further, the vanishing point parallax determining unit 73 performs partial differentiation on the equation of parallelism E shown in FIG. The width W of the road when the partial differential is 0 is the width W0 of the road with the smallest deviation. The width W0 of the road is a value close to the actual width of the road on which the vehicle was traveling when each stereo image was acquired. By calculating using the road width W0 calculated in this way, the vanishing point parallax d0 used for distance calculation can be calculated with high accuracy.

  FIG. 13 shows the time change of the vanishing point parallax d0 determined in this way. It can be seen that the value of the vanishing point parallax d0 converges as time passes. The vanishing point parallax d0 after convergence is a highly accurate value.

  Note that the boundary line extraction unit 50, the parallax calculation unit 60, the approximate straight line calculation unit 71, the coincidence degree calculation unit 72, the vanishing point parallax determination unit 73, the reliability calculation unit 74, and the distance calculation unit 80 are each a boundary line extraction unit. , Parallax calculation means, approximate straight line calculation means, coincidence calculation means, vanishing point parallax determination means, reliability calculation means, and distance calculation means.

  Next, a processing procedure for calculating the vanishing point parallax d0 will be described with reference to the flowchart of FIG. This process is executed by the image processing apparatus 30 at predetermined time intervals.

  First, a pair of analog images photographed by the image photographing device 20 is digitized and a parallel stereo image is acquired (S10). Subsequently, the parallax d at each point on the image coordinates is calculated based on the stereo image acquired in S10 (S11).

  Subsequently, edge points representing the outlines of the left and right white lines, which are the left and right boundary lines of the road, are extracted in a predetermined region near the image capturing device 20 (S12). Subsequently, the edge points constituting the line segment of the predetermined length on the right extracted in S12 are weighted with reliability. Then, the weighted edge point is approximated by the least square method, and an approximate straight line that approximates the right line segment is calculated (S13). Similarly, a plurality of straight lines approximating a predetermined length of the line segment are calculated while shifting by a predetermined number of pixels on the near side and the far side of the right line segment.

  Subsequently, with respect to the left edge point extracted in S12, a plurality of line segments having the same length as the right line segment are cut out while shifting one pixel at a time from the front side. Then, the edge points belonging to each cut line segment are weighted with reliability, the weighted edge points are approximated by the least square method, and an approximate straight line approximating each line segment is calculated (S14). Note that an approximate straight line of a predetermined length on the left side may be calculated in S13, and an approximate straight line on the right side may be calculated in S14.

  Subsequently, the coefficients αlx, αly, βlx, βly of the left approximate straight line calculated in S14 are smoothed. Further, the coefficients αrx, αry, βrx, βry of the right approximate line calculated in S13 and the coefficients of the line approximating a plurality of line segments on the near side and the far side of the right line segment are smoothed (S15). . In each vanishing point parallax d0, the smoothed coefficients are used as the coefficients of the left and right approximate straight lines.

  Subsequently, in each vanishing point parallax d0, a line segment pair that is most parallel in the real space is determined from a plurality of line segment pairs (S16). Subsequently, in each vanishing point parallax d0, the coincidence degree M of each line segment pair determined in S16 is calculated (S17). Then, the vanishing point parallax d0 corresponding to the highest matching degree M among the matching degrees M calculated for each vanishing point parallax d0 is determined as the vanishing point parallax d0 of the stereo image acquired in S10 (S18).

  Subsequently, based on the plurality of vanishing point parallaxes d0 determined in S18 and the vanishing point parallax d0 determined in S18 for the plurality of stereo images acquired at predetermined time intervals before the stereo image is acquired in S10. The vanishing point parallax d0 used for calculating the distance is determined (S19). Specifically, for a plurality of vanishing point parallaxes d0 determined at different points in time, statistical processing such as a K-means method is performed using the road width W0 that minimizes the parallelism E, and the vanishing point parallax d0 used for distance calculation is performed. To decide.

  This process is complete | finished above. Next, until the vanishing point parallax d0 is updated by performing this process, the vanishing point parallax d0 calculated in the current process is used to calculate the distance to the object from the equation in FIG.

  According to this embodiment described above, the following effects are obtained.

  Edge points that constitute the left and right boundary lines of the road are extracted, and an approximate straight line that is a line segment of a predetermined length that approximates the left and right boundary lines is calculated based on the extracted edge points. Then, while the value of the vanishing point parallax d0 is changed, for each vanishing point parallax d0, a perpendicular line dropped from the end A of one of the calculated approximate straight lines to the straight line including the other approximated line. The degree of coincidence M between the foot and the end B of the other approximate line is calculated. Then, the vanishing point parallax d0 corresponding to the highest matching degree M among the calculated matching degrees M is determined as the vanishing point parallax d0 in the stereo image. Thereby, the calculation accuracy of the vanishing point parallax d0 can be improved. Further, the distance to the object is calculated based on the calculated parallax d and the determined vanishing point parallax d0. Therefore, it is possible to improve the calculation accuracy of the vanishing point parallax d0, and consequently improve the calculation accuracy of the distance.

  A stereo image is acquired at a plurality of different time points, and the vanishing point parallax d0 is determined for each stereo image. And the vanishing point parallax d0 used for distance calculation is statistically determined based on the vanishing point parallax d0 determined for each stereo image. Thereby, outliers are excluded, and the calculation accuracy of the vanishing point parallax d0 can be improved.

  An approximate straight line is calculated on the basis of the weighted edge points so that the reliability of the image data of the pixels captured in the vicinity of the image capturing device 20 is higher. Thereby, the approximate straight line can be calculated with high accuracy, and as a result, the calculation accuracy of the vanishing point parallax d0 can be improved.

  A plurality of line segment pairs of an approximate straight line on the left side and an approximate straight line on the right side are calculated so that a line along the shape of the road can be approximated even on a road with a curve or undulation. Then, in each vanishing point parallax d0, the most parallel line segment pair is determined from the calculated plurality of line segment pairs, and the matching degree M is calculated for the determined line segment pair. Therefore, regardless of the shape of the road, it is possible to find a highly parallel line segment pair from a plurality of line segment pairs and calculate the vanishing point parallax d0 with high accuracy.

  -By shifting one pixel at a time, a number of line segments are cut out from the edge points that make up the white line of the left straight line, and an approximate straight line that approximates each line segment is calculated, so that even if the road curve or undulation is large, it is parallel You can find high-quality line segment pairs. As a result, the vanishing point parallax d0 can be calculated with high accuracy.

  -The square value of the deviation between the length Li of the vertical line drawn from one line segment to the straight line including the other line segment and the road width W is the minimum road width W0. The vanishing point parallax d0 determined for the group is grouped. Then, the vanishing point parallax d0 used for calculating the distance is determined based on the result of the statistics of the vanishing point parallax d0 for each group. Thereby, the vanishing point parallax d0 used for calculating the distance can be statistically calculated with high accuracy.

  Even if the square value of the deviation between the length Li of the perpendicular line and the width W of the road is the same value, the parallelism E in the vicinity of the image capturing device 20 is weighted by the reciprocal of the reliability so that the parallelism E in the vicinity is lower than in the distance. The parallelism E is calculated from the square value of the deviation. Thereby, the parallelism E can be calculated with high accuracy, and consequently the calculation accuracy of the vanishing point parallax d0 by statistics can be improved.

(Other embodiments)
-When traveling on a long straight road, the left and right white lines may be approximated by one straight line, and the degree of coincidence E may be calculated.

  -When a plurality of line segments of a predetermined length are cut out from a long straight line, they may be cut out while shifting by a predetermined number of pixels in the depth direction.

  The vanishing point parallax d0 determined for the stereo image acquired at one time point may be used for calculating the distance until the vanishing point parallax d0 is determined for the stereo image acquired at the next time point. . That is, the vanishing point parallax d0 may not be determined by performing statistical processing.

  Although there is a possibility that the calculation accuracy of the vanishing point parallax d0 may decrease, an approximate straight line may be calculated from edge points that are not weighted with reliability.

  Although the calculation accuracy of the vanishing point parallax d0 may be reduced, the parallelism E may be calculated by adding the square value of the deviation that is not weighted by the reciprocal of the reliability.

  If the standard width of the target road is known as the road width W0, the standard width may be used. As the road width W0, a result generally derived from a certain white line recognition may be used.

  DESCRIPTION OF SYMBOLS 20 ... Image pick-up device, 30 ... Image processing device, 50 ... Boundary line extraction part, 60 ... Parallax calculation part, 71 ... Approximate straight line calculation part, 72 ... Concordance degree calculation part, 73 ... Vanishing point parallax determination part, 74 ... Trust Degree calculation unit, 80... Distance calculation unit.

Claims (7)

A plurality of image capturing devices (20) for capturing a surrounding environment including a road;
Parallax calculating means (60) for calculating parallax based on a pair of images captured by the plurality of image capturing devices;
In the image, boundary line extraction means (50) for extracting edge points constituting the left and right boundary lines of the road;
Approximate straight line calculating means (71) for calculating an approximate straight line that is a line segment of a predetermined length that approximates the left and right boundary lines based on the edge points extracted by the boundary line extracting means;
While changing the value of the vanishing point parallax in the pair of images, for each vanishing point parallax, from the first end of one of the approximate lines calculated by the approximate line calculation unit to the other of the left and right approximate lines A degree of coincidence calculation means (72) for calculating a degree of coincidence between a foot of a perpendicular line drawn down to a straight line including an approximate straight line and a second end of the other approximate straight line corresponding to the first end;
Vanishing point parallax determining means (73) for determining, as the vanishing point parallax in the pair of images, the vanishing point parallax corresponding to the highest matching degree of the matching degrees calculated by the matching degree calculating means;
Distance calculating means (80) for calculating the distance to the object based on the parallax calculated by the parallax calculating means and the vanishing point parallax determined by the vanishing point parallax determining means. Distance calculation device to do. The approximate straight line calculating means calculates a first straight line that is an approximate straight line of the predetermined length line from an edge point constituting one of the left and right boundary lines for each vanishing point parallax, and the left and right boundary lines A plurality of second straight lines that are approximate straight lines of the predetermined length from the edge points constituting the other of
The degree-of-match calculation means determines, for each vanishing point parallax, the line segment pair having the highest parallelism from the line segment pairs of the first straight line and the second straight line, and determines the most parallel determined The distance calculation device according to claim 1, wherein the degree of coincidence of the pair of line segments that is high is calculated.   The distance calculating device according to claim 2, wherein the approximate straight line calculating unit calculates a plurality of the second straight lines from edge points constituting the other of the left and right boundary lines while shifting by a predetermined number of pixels in the depth direction. A reliability calculation means (74) for calculating the reliability of the image data of a pixel obtained by imaging the vicinity of the image capturing device;
The distance calculation device according to claim 1, wherein the approximate line calculation unit calculates the approximate line based on the edge points weighted with the reliability calculated by the reliability calculation unit. The plurality of image capturing devices capture the pair of images at a plurality of different time points,
The vanishing point parallax determining unit statistically determines the vanishing point parallax used by the distance calculating unit based on the vanishing point parallax determined for each of the plurality of images. The distance calculation apparatus in any one.   The vanishing point parallax determining means groups the vanishing point parallax determined for each of the plurality of images using the width of the road, and based on a result of statistics for each group, the distance calculating means The distance calculation apparatus according to claim 5, wherein the vanishing point parallax to be used is determined. A reliability calculation means for calculating the reliability higher as the image data of the pixel that has captured the vicinity of the image capturing device;
The vanishing point parallax determining means is a deviation between an interval between the left and right approximate straight lines corresponding to the vanishing point parallax determined for each of the plurality of images and the width of the road, and the reliability calculating means The distance calculation device according to claim 6, wherein the deviation weighted by the reciprocal of the reliability calculated by using the width of the road that is the smallest.