JP6489645B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus that recognizes three-dimensional position information of a subject based on an image obtained by capturing a traveling direction of a vehicle using a stereo camera.

  An image processing apparatus for recognizing a positional relationship between a subject and a host vehicle based on a pair of images obtained by capturing a traveling direction of the host vehicle with a stereo camera including a pair of cameras as a vehicle system for recognizing an environment outside the vehicle. There is something to prepare. In this way, in the system for recognizing a subject existing in the traveling direction of the host vehicle, three-dimensional position information (position information in real space) of the subject including information on the distance to the subject is generated. Specifically, a pair of image data with different viewpoints is captured by the stereo camera, the parallax of the subject between the pair of image data is calculated, and the three-dimensional position information of the subject is generated based on the parallax.

  The three-dimensional position information is a point in space when a point directly below the center of a pair of cameras is the origin, the X axis is the direction connecting the pair of cameras, the Y axis is the vertical direction, and the Z axis is the front and rear direction ( X, Y, Z). The three-dimensional position information is generated based on the triangulation method. Specifically, the pixel on the image is represented by coordinates (i, j), the parallax is dp, the distance between the pair of cameras is CD, 1 When the viewing angle per pixel is PW, the mounting height of a pair of cameras is CH, and the i-coordinate and j-coordinate on the image at the infinity point in front of the camera are IV and JV, respectively, [Equation 1] It is obtained by coordinate transformation represented by ~ [Equation 3].

X = CD / 2 + Z * PW * (i-IV) ... [Formula 1]
Y = CH + Z × PW × (j−JV) [Formula 2]
Z = CD / {PW × (dp−DP)} [Formula 3]
However, “DP” in [Expression 3] is a parallax offset value, which is determined when the stereo camera is attached to the vehicle, and is treated as a constant thereafter.

  The parallax offset value DP is a value determined with reference to a state in which the stereo camera is attached to the host vehicle in a predetermined manner. If the value changes from the predetermined state, the value of the distance Z cannot be correctly calculated by the above [Equation 3].

  Therefore, as a technique for correcting an error in the parallax offset value DP caused by such a change in the mounting state of the stereo camera over time, the present applicant previously proposed a correction technique described in Patent Document 1. doing. In Patent Document 1, a vanishing point is calculated in each captured image of a pair of cameras based on a captured image of a white line such as a continuous line or a broken line that is linearly marked on the road surface, and correction is performed based on the result. According to the technique described in Patent Document 1, the parallax offset value DP can be corrected when the host vehicle is traveling.

JP 2003-83742 A

  For example, when a pair of cameras constituting a stereo camera are arranged so as to capture the front of the host vehicle through the front window of the vehicle, the image distortion caused by the transmission through the front window The upper and lower parts may not be uniform. The reason why the distortion of the image changes in the vertical direction of the image is, for example, that the distance at which the light passes through the front window is different between the upper and lower sides of the image, or that the front window is a curved surface.

  Thus, when the distortion of the image changes in the vertical direction of the image, an error occurs in the calculation of the vanishing point of the white line on the image as in the technique described in Patent Document 1, and correction of the parallax offset value DP is performed. An error may occur.

  An object of the present invention is to solve the above-described problem, and to correct a parallax offset value accurately in an image processing apparatus that recognizes three-dimensional position information of a subject from an image captured using a stereo camera. To do.

  An image processing apparatus according to an aspect of the present invention is an image processing apparatus that recognizes three-dimensional position information of a subject based on a pair of image data obtained by capturing the traveling direction of the host vehicle using a stereo camera. On the pair of image data, a lane detection processing unit for detecting a shape of a pair of white lines indicating a vehicle lane of the host vehicle, the pair of image data, and a pair of white lines detected by the lane detection unit A parallax offset value correction processing unit that corrects a calculation formula for calculating the distance from the parallax based on the data, and the parallax offset value correction processing unit is at two points with different distances in the traveling direction of the host vehicle. For both a short distance point and a long distance point, obtaining the parallax of the white line on the pair of image data and the number of separated pixels of the pair of white lines, and the white at the short distance point Based on the parallax and the number of pixels separated from each other, a step of calculating a distance between the short distance white lines, which is a separation distance between a pair of white lines in the real space at the short distance point, , A step of obtaining a distance between long distance white lines, which is a separation distance between a pair of white lines in the real space at the long distance point, and a long distance predicted when the distance between the short distance white lines is considered to be equal at the long distance point A step of calculating a long-distance estimated parallax that is a parallax at a point; a step of calculating an error of a calculation formula for calculating the distance from the parallax based on a comparison result of the long-distance white line parallax and the long-distance estimated parallax; And correcting the calculation formula for calculating the distance from the parallax based on the error value.

  According to the present invention, the parallax offset value can be accurately corrected in an image processing apparatus that recognizes the three-dimensional position information of a subject from an image captured using a stereo camera.

It is a block diagram which shows the structure of an image processing apparatus. It is a figure which shows the white line on a pair of image imaged with the stereo camera. It is a figure which shows the white line on real space. It is a flowchart which shows operation | movement of an image processing apparatus.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. In the drawings used for the following description, the scale of each component is made different in order to make each component recognizable on the drawing. It is not limited only to the quantity of the component described in (1), the shape of the component, the ratio of the size of the component, and the relative positional relationship of each component.

  An image processing apparatus 1 according to this embodiment shown in FIG. 1 is mounted on a vehicle on which a stereo camera 2 is mounted, and the relative position of the object (subject) shown in the image captured by the stereo camera 2 with respect to the vehicle. Is calculated.

  The stereo camera 2 includes a pair of first camera 2a and second camera 2b having different viewpoints arranged at a certain distance in the vehicle width direction so that the traveling direction of the vehicle is in the field of view. The first camera 2a and the second camera 2b have parallel optical axes and capture moving images at the same frame rate.

  The stereo camera 2 includes an A / D converter that converts a captured image into digital data, an image correction unit that performs image correction such as noise removal and luminance value correction on the image, and the like. The stereo camera 2 transmits the captured image to the image processing apparatus 1 as digital data.

  The image processing apparatus 1 includes a computer having a CPU, ROM, RAM, input / output interface, and the like connected to a bus. The image processing apparatus 1 may be incorporated in a vehicle control system 3 including a computer that controls the operation of the vehicle, or may be a computer different from the vehicle control system 3.

  The image processing apparatus 1 can communicate with the vehicle control unit 3. The vehicle control system 3 is based on information input from a sensor unit 5 including a vehicle speed sensor, a steering angle sensor, an acceleration sensor, an angular velocity sensor, and the like provided in the vehicle, and information input from the image processing device 1. Recognize the situation including the surrounding environment of the vehicle and execute the operation related to driving support.

  The operation related to the driving support includes a warning operation for outputting a lane departure warning and a collision warning to the driver, a semi-automatic or automatic driving assist operation such as automatic braking and automatic steering, and the like. Since driving assistance using a stereo camera is a known technique, detailed description thereof is omitted.

  The image processing apparatus 1 includes a memory 1a, a three-dimensional position information generation processing unit 1b, a lane detection processing unit 1c, and a parallax offset value correction processing unit 1e.

  The memory 1 a stores image data input from the stereo camera 2. Hereinafter, an image captured by the first camera 2a is referred to as first image data, and an image captured by the second camera 2b is referred to as second image data.

  Each of the three-dimensional position information generation processing unit 1b, the lane detection processing unit 1c, and the parallax offset value correction processing unit 1e includes a configuration for executing an image processing function of the image processing apparatus 1 described below.

  The three-dimensional position information generation processing unit 1b generates three-dimensional position information of the subject based on a pair of image data with different viewpoints captured by the stereo camera 2.

  The three-dimensional position information generation processing executed by the three-dimensional position information generation processing unit 1b detects a corresponding point between the first image data and the second image data by pattern matching, and a coordinate shift between the detected corresponding points. Is calculated as the parallax dp, and the position information of the corresponding point in the real space is generated as the three-dimensional position information by the principle of triangulation using the parallax dp.

  In calculating the coordinate shift as the parallax dp as described above, one of the first image data and the second image data is determined in advance as a “reference image” and the other as a “comparison image”. The comparison image is generated as an image having a larger number of pixels in the horizontal direction than the reference image in order to allow the calculation of the parallax dp for the object located at the horizontal end on the reference image.

  Here, the three-dimensional position information is based on a point just below the center of the first camera 2a and the second camera 2b, which is a pair of cameras, with the X axis in the direction connecting the pair of cameras, the Y axis in the vertical direction, and the front-back direction. Is information expressed as a point (X, Y, Z) in space when the Z axis is taken.

  Each value of X, Y, and Z as the three-dimensional position information is expressed by the pixel coordinates (i, j) when the axis parallel to the horizontal direction in the image is the i axis and the axis parallel to the vertical direction is the j axis. ), The distance between the pair of cameras (baseline length) is CD, the viewing angle per pixel is PW, the mounting height of the pair of cameras is CH, the i-coordinate on the reference image at the infinity point in front of the camera, When the j coordinate is IV and JV, respectively, it is obtained by coordinate conversion represented by the following [Expression 1] to [Expression 3].

X = CD / 2 + Z × PW × (i-IV) [Formula 1]
Y = CH + Z × PW × (j−JV) [Formula 2]
Z = CD / {PW × (dp−DP)} [Formula 3]
“DP” in [Expression 3] is also referred to as a vanishing point parallax, an infinite correspondence point, or the like, but in short, the parallax dp between corresponding points between the reference image and the comparison image, and in the real space up to the corresponding point Is a value determined so as to satisfy the above [Equation 3]. Hereinafter, this “DP” is referred to as “parallax offset value DP”.

  The lane detection processing unit 1c performs lane detection processing for detecting a white line formed on the road surface on which the host vehicle travels in the image data captured by the stereo camera 2. Here, the white line is a linear road marking formed on the road surface along the left and right boundaries of the vehicle lane to indicate the vehicle lane where the host vehicle is traveling. In this embodiment, the white line is referred to as a white line, but the color of the white line is not limited to white. The white line may be a broken line.

  The lane detection processing unit 1c detects the shape of the white line existing on both sides of the vehicle lane where the host vehicle is traveling for both the first image data and the second image data.

  The parallax offset value correction processing unit 1e performs the correction on the calculation formula for calculating the distance Z from the parallax dp (that is, the previous [Formula 3]) based on the white line information detected by the lane detection processing unit 1c. An offset value correction process is executed. As described above, the parallax offset value DP in [Equation 3] due to a change with time in the camera mounting state causes an error from the optimum value corresponding to the actual camera mounting state. The offset value correction process corrects this.

  FIG. 2 is a diagram showing a white line in image data captured by the stereo camera 2, and FIG. 3 is a diagram showing a white line on the XY plane which is an actual coordinate. FIG. 4 is a flowchart of the parallax offset value correction process.

  The parallax offset value correction process shown in FIG. 4 is executed in synchronization with the frame rate of the stereo camera 2. That is, the parallax offset value correction process is executed for all frames of an image captured by the stereo camera 2.

  As shown in the flowchart of FIG. 4, in the parallax offset value correction process, first, in step S10, it is determined whether or not the parallax offset value correction process can be executed. In the parallax offset value correction process, as will be described later, the distance between the pair of white lines in which the white line indicates the vehicle lane of the host vehicle in the vehicle width direction is constant within a predetermined range in front of the host vehicle, and the road surface is It must be approximately horizontal. Hereinafter, the distance between the pair of white lines in the vehicle width direction is referred to as a distance between the white lines.

  Therefore, in step S10, when the host vehicle is traveling straight at an equal speed for a predetermined time or more by the sensor unit 5, it is considered that the vehicle traffic zone in which the host vehicle travels is flat and straight. Whether or not the host vehicle is traveling straight at a constant speed is determined based on output information from a steering angle sensor, an acceleration sensor, or the like included in the sensor unit 5. For example, when the rudder angle is within a predetermined range near neutral and the acceleration value in the vehicle longitudinal direction is within a predetermined range near zero, the image processing apparatus 1 causes the host vehicle to travel straight at a constant speed. Is determined. In Step S10, it is determined whether the distance between the white lines of the pair of white lines detected by the lane detection processing unit 1c is constant for a predetermined time or more.

  In step S10, the parallax offset value correction processing unit 1e performs parallax offset value correction processing when the host vehicle is traveling straight at a constant speed and the distance between the pair of white lines in the vehicle width direction is constant for a predetermined time or more. Is determined to be possible.

  The determination in step S10 is for determining that it is possible to execute the parallax offset value correction process that the thickness value of the white line detected by the lane detection processing unit 1c is constant for a predetermined time or more. It may be added as a requirement. This is because there is a variation in the value of the thickness of the white line, which means that the detection accuracy of the shape of the white line by the lane detection processing unit 1c is low due to snow or rain, or the actual white line is unclear This is because it can be considered.

  If it is determined in step S10 that the parallax offset value correction process cannot be executed (NO in step S20), the parallax offset value correction process ends. In this case, the value of the parallax offset value DP is not changed.

  On the other hand, when it is determined in step S10 that the parallax offset value correction process can be executed (YES in step S20), the process proceeds to step S30.

  In step S30, the parallax offset value correction processing unit 1e acquires information on the white line detected by the lane detection processing unit 1c from the first image data and the second image data which are a pair of image data.

  FIG. 2 shows the second image data superimposed on the first image data captured by the stereo camera 2 for explanation. In FIG. 2, the white line that appears in the first image data is indicated by a solid line, and the white line that appears in the second image data is indicated by a two-dot chain line. In the illustrated example, the second camera 2b that captures the second image data is located on the left side in the vehicle traveling direction with respect to the first camera 2a, but the first camera 2a and the second camera 2b The positional relationship may be reversed.

  In FIG. 2, reference numeral 10RL is a left white line located on the left side in the traveling direction with respect to the own vehicle in the first image data, and reference numeral 10RR is located on the right side in the traveling direction with respect to the own vehicle in the first image data. The white line on the right. In FIG. 2, reference numeral 10LL is a left white line located on the left side in the traveling direction with respect to the host vehicle in the second image data, and reference numeral 10LR is on the right side in the traveling direction with respect to the host vehicle in the second image data. The right white line is located.

  In step S30, the parallax offset value correction processing unit 1e acquires the parallax and the distance between the white lines for the white lines at two points with different Z-axis coordinates ahead of the host vehicle. More specifically, in step S30, the short distance white line parallax md [pixel], which is the parallax of the white line at the short distance point ahead of the host vehicle on the image data, and the separation between the white lines at the short distance point on the image data. A short distance white line pixel m [pixel] which is a distance, a long distance white line parallax nd [pixel] which is a parallax of a white line far from the short distance point ahead of the host vehicle on the image data, and an image The parallax offset value correction processing unit 1e obtains, from the lane detection processing unit 1c, the long-distance white-line pixel n [pixel], which is the distance between the white lines at the long-distance point on the data.

  As shown in FIG. 3, when the Z coordinate of the short distance point is a [mm] and the Z coordinate of the long distance point is b [mm], the relationship of a <b is established. In this embodiment, as an example, a = 15000 [mm] and b = 40000 [mm].

Therefore, the short distance white line parallax md is a white line parallax at a coordinate ja corresponding to a short distance point in a pair of image data, as shown in FIG. In this embodiment, as an example, the short distance white line parallax md is an average value of the parallax mdl [pixel] of the left white line and the parallax mdr [pixel] of the right white line at the coordinate ja. That is, md is obtained by the following equation.
md = (mdl + mdr) / 2
The short distance white line parallax md may be a value obtained by averaging the parallax of the white line detected at the j coordinate of several pixels in the vicinity of the coordinate ja.

  As shown in FIG. 2, the short distance white line pixel m is the number of pixels between a pair of white lines at a coordinate ja corresponding to a short distance point in the reference image data. The short-distance white line pixel m may be a value obtained by averaging the number of pixels between white lines detected at the j coordinate of several pixels in the vicinity of the coordinate ja.

Further, the long-distance white line parallax nd is the parallax of the white line at the coordinate jb corresponding to the long-distance point in the pair of image data, as shown in FIG. In this embodiment, as an example, the long distance white line parallax nd is an average value of the parallax ndl [pixel] of the left white line and the parallax ndr [pixel] of the right white line at the coordinate jb. That is, nd is obtained by the following equation.
nd = (ndl + ndr) / 2
The long-distance white line parallax nd may be a value obtained by averaging the parallax of the white line detected at the j coordinate of several pixels in the vicinity of the coordinate jb.

  As shown in FIG. 2, the long distance white line pixel n is the number of pixels between a pair of white lines at a coordinate jb corresponding to a long distance point in the reference image data. The long-distance white line pixel n may be a value obtained by averaging the number of pixels between white lines detected at the j coordinate of several pixels in the vicinity of the coordinate jb.

Next, in step S40, the parallax offset value correction processing unit 1e calculates a short distance white line distance A [mm] at a short distance point calculated from the pair of image data. Specifically, the parallax offset value correction processing unit 1e converts the short-distance white line pixel m acquired in step S30 into the short-distance white line distance A using the short-distance white line parallax md using the following equation.
A = CD · m / md

Next, in step S50, the parallax offset value correction processing unit 1e obtains the distance B [mm] between the long distance white lines at the long distance points calculated from the pair of image data. Specifically, the parallax offset value correction processing unit 1e converts the long-distance white line pixel n acquired in step S30 into the short-distance white line distance B using the short-distance white line parallax nd using the following equation.
B = CD · n / nd

Next, in step S60, the parallax offset value correction processing unit 1e estimates the long-distance estimated parallax f [pixel] that is a parallax at a long-distance point that is predicted when the short-distance white line distance A is also equal at the long-distance point. Is calculated. Specifically, the parallax offset value correction processing unit 1e uses the long-distance white line pixel n acquired in step S30 and the short-distance white line distance A calculated in step S40 to calculate the estimated parallax f by the following equation. calculate.
f = CD · n / A

Next, in step S70, the parallax offset value correction processing unit 1e determines that the long-distance white line parallax nd, which is the parallax of the white line at the long distance point acquired from the pair of image data, and the short-distance white line distance A are at the long distance point. And parallax error e [pixel] which is a difference from the long-range estimated parallax f predicted when the two are also equal.
The parallax error e indicates an error of the long-distance white line parallax nd with respect to the long-distance estimated parallax f when the long-distance estimated parallax f is used as a reference. Specifically, the parallax offset value correction processing unit 1e calculates the parallax error e by the following equation.
e = f-nd

Next, in step S80, the parallax offset value correction processing unit 1e corrects the parallax offset value by the following equation using the value of the parallax error e. In the following equation, the current parallax offset value is indicated as DP, and the corrected parallax offset value is indicated as DP ′.
DP ′ = DP−e

  The image processing apparatus 1 stores the corrected parallax offset value DP ′ in the memory 1a as a new parallax offset value DP.

  In the present embodiment, when the absolute value of the parallax error e calculated in step S70 is greater than or equal to a predetermined value, the absolute value of the parallax error e is set as the predetermined value. In other words, in the present embodiment, the width in which the parallax offset value DP can be changed by one parallax offset value correction process is limited to a predetermined value or less. This is to prevent the parallax offset value DP from abruptly deviating from the true value when a large error is included in the process of calculating the parallax error e.

  As described above, the image processing apparatus 1 according to the present embodiment recognizes the three-dimensional position information of the subject based on a pair of image data obtained by capturing the traveling direction of the host vehicle using the stereo camera 2. A lane detection processing unit 1c that detects a shape of a pair of white lines indicating a vehicle lane of the host vehicle on a pair of image data, the pair of image data, and the lane detection unit 1c. And a parallax offset value correction processing unit 1e for correcting a calculation formula for calculating the distance from the parallax between the pair of image data to the subject based on the shape data of the pair of white lines.

  Then, the parallax offset value correction processing unit 1e of the present embodiment has white line parallax (md on the pair of image data (md) for both the short-distance point and the long-distance point which are two points having different distances in the traveling direction of the host vehicle. , Nd) and the number of separated pixels (m, n) between the pair of white lines are acquired from the lane detection processing unit 1c. Then, the parallax offset value correction processing unit 1e calculates a short-distance white line distance A, which is a separation distance between a pair of white lines in the real space at the short distance point, based on the parallax md of the white line at the short distance point and the number m of separated pixels. To do.

  Then, the parallax offset value correction processing unit 1e has a long-distance estimated parallax f that is a parallax at a long-distance point predicted when the short-distance white-line distance A is considered to be equal even at a long-distance point, and a pair of images. Based on the comparison result between the distance B between the long distance white lines, which is the distance between the pair of white lines in the real space at the long distance point obtained from the data, an error in the calculation formula for calculating the distance to the subject (disparity of the parallax offset value DP) Error) is calculated, and the calculation formula (parallax offset value DP) is corrected based on the error value.

  Since the image processing apparatus 1 according to the present embodiment corrects the parallax offset value DP while the host vehicle is traveling, it corrects an error in the parallax offset value DP caused by a change in the mounting state of the stereo camera 2 over time. can do.

  In the present embodiment, the parallax offset value DP is corrected using information on the distance between the pair of white lines obtained using the number of pixels in the i-axis direction of the image data. For example, when the stereo camera 2 is arranged inside the front window of the host vehicle, the state of image distortion due to transmission through the front window may change between the upper part and the lower part of the image. The i-axis direction is less susceptible to distortion changes than the j-axis direction. This is because the vehicle front window tilt angle in the vehicle front-rear direction is generally larger than the vehicle front width tilt angle of the vehicle.

  As described above, in this embodiment, by using the information on the distance between the pair of white lines that is not easily affected by the distortion caused by the transmission through the front window, for example, the vanishing point of the pair of white lines is calculated as compared with the conventional technique. Thus, the parallax offset value DP can be corrected more accurately.

  The present invention is not limited to the above-described embodiment, and can be appropriately changed without departing from the gist or concept of the invention that can be read from the claims and the entire specification, and an image processing apparatus with such a change is also applicable. Moreover, it is included in the technical scope of the present invention.

1 image processing device,
1a memory,
1b 3D position information generation processing unit,
1c lane detection processing unit,
1e Parallax offset value correction processing unit,
2 Stereo camera,
2a first camera,
2b Second camera,
3 Vehicle control system,
5 Sensor part,
10L left white line (on real coordinates),
10R right white line (on real coordinates),
10RL Left white line (in the first image data),
10RR Right white line (in the first image data),
10LL Left white line (in the second image data),
10LR Right white line (in the second image data),
A Distance between short distance white lines,
a Distance to a short distance point,
B Distance between long distance white lines,
b Distance to long distance point,
CD baseline length,
DP parallax offset value,
DP 'corrected parallax offset value,
dp parallax,
e Parallax error,
f estimated parallax,
m pixels between short distance white lines,
md short distance white line parallax,
n Long distance white line pixels,
nd Far white line parallax.

Claims (2)

An image processing apparatus that recognizes three-dimensional position information of a subject based on a pair of image data obtained by imaging the traveling direction of the host vehicle using a stereo camera,
On the pair of image data, a lane detection processing unit that detects the shape of a pair of white lines indicating a vehicle traffic zone of the host vehicle;
A parallax offset value correction processing unit that corrects a calculation formula for calculating a distance to the subject based on the pair of image data and shape data of a pair of white lines detected by the lane detection unit,
The parallax offset value correction processing unit
Obtaining a parallax of white lines on a pair of image data and a number of separated pixels of a pair of white lines for both a short-distance point and a long-distance point which are two points having different distances in the traveling direction of the host vehicle;
Based on the parallax of the white line at the short distance point and the number of spaced pixels, calculating a distance between the short distance white lines, which is a distance between a pair of white lines in the real space at the short distance point;
Obtaining a distance between long-distance white lines, which is a separation distance between a pair of white lines in the real space at the long-distance point, based on the parallax of white lines at the long-distance point and the number of separated pixels;
Calculating a long-distance estimated parallax that is a parallax at a long-distance point predicted when the short-distance white line distance is also considered to be equal at a long-distance point;
Calculating an error of a calculation formula for calculating the distance from the parallax based on a comparison result of the long-distance white line parallax and the long-distance estimated parallax;
And correcting the calculation formula based on the error value. The parallax offset value correction processing unit, when the vehicle is traveling straight at a constant speed in a predetermined period in the past, and the distance in the vehicle width direction of the pair of white lines is constant in the past predetermined period, The image processing apparatus according to claim 1, wherein a step of correcting the calculation formula is executed.

Images