JP4037671B2 - Stereo image processing device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a stereo image processing apparatus that generates a distance image by processing a pair of images captured by a stereo camera.
[0002]
[Prior art]
In general, as a three-dimensional measurement technique based on images, a correlation between a pair of images obtained by capturing an object from different positions with a stereo camera composed of two cameras is obtained, and the mounting position and focal length of the stereo camera are determined from the parallax for the same object. Image processing based on a so-called stereo method is known in which a distance is obtained based on the principle of triangulation using camera parameters.
[0003]
As image processing by this stereo method, the present applicant previously measured the three-dimensional position of an object outside the vehicle by processing an image picked up by a stereo camera mounted on the vehicle in Japanese Patent Laid-Open No. 5-11409. The technology to do is proposed. In this technique, stereo matching for searching for a corresponding position by calculating a city block distance between a plurality of small areas is processed at high speed by a hardware circuit for a pair of images captured in stereo, and pixel shift of the corresponding small area is performed. The quantity is output as distance information.
[0004]
Furthermore, in order to perform accurate stereo matching without being affected by variations between a pair of images captured in stereo or low-frequency noise, the applicant of the present application disclosed in Japanese Patent Application Laid-Open No. 11-234701 has a main image average value processing unit. To calculate the average luminance value of the small area of the main image, the sub-image average value processing unit calculates the luminance average value of the small area of the sub-image one pixel at a time with respect to the main image, and the average value difference unit A value obtained by subtracting the average value from the luminance value of each pixel in the small area with respect to the image and the main image is calculated, and the value obtained by subtracting the average value from the luminance value of the main image in the stereo matching unit and the luminance value of the sub-image A technique for average value difference matching is proposed in which the absolute value of the difference from the value obtained by subtracting the average value is summed to calculate the city block distance. In this average value difference matching, only the high-frequency component of the image is targeted for matching, low-frequency noise is removed, the accuracy of the two images is slightly out of balance, and the effects of gain changes due to aging of cameras and analog circuit components are affected. On the other hand, accurate distance information can be obtained without causing mismatching.
[0005]
[Problems to be solved by the invention]
However, in the average value difference matching previously proposed by the present applicant, the luminance average value in a small area is calculated so as not to be affected by variations between a pair of stereo images and low-frequency noise, and the luminance average value and the pixel are calculated. Stereo matching is performed by calculating the city block distance based on the difference from the brightness of the image. For this reason, it cannot be denied that the ability to prevent mismatching is insufficient for an image on which random high frequency noise is superimposed, which has been a new problem.
[0006]
The present invention has been made in view of the above circumstances, and provides a stereo image processing apparatus capable of performing accurate stereo matching without being affected by noise, not only for low-frequency noise but also for images with high frequency noise. It is intended to provide.
[0007]
[Means for Solving the Problems]
In order to achieve the above object, the invention described in claim 1 corresponds to a pair of images composed of a reference image and a comparative image captured by a stereo camera by calculating a city block distance between small areas of each other image. In a stereo image processing apparatus that searches for a small area and generates a distance image in which perspective information based on a pixel shift amount generated according to a distance to an object in a corresponding small area is converted into a numerical value, Means for averaging the image data of each pixel and calculating it as a first average value; means for averaging the image data of each pixel in the small area of the comparison image and calculating as a second average value; Means for subtracting the first average value from the image data of each pixel in the small area of the image and calculating as a first data group; and the second from the image data of each pixel in the small area of the comparative image. The average value of Min, and comparing means for calculating, the image data of each pixel in the small area of the reference image, a decision value that is set based on the image data mean value of the entire image of the reference image as a second data group And determining whether the data is valid, selecting valid data of the first data group and the second data group, each valid data of the first data group and the second data group And a means for summing up absolute values of differences from each valid data and calculating the city block distance.
[0008]
The invention according to claim 2 is the invention according to claim 1, wherein the determination value is obtained by multiplying or subtracting the first average value by a coefficient value based on an image data average value of the entire image of the reference image. It is characterized by setting.
The invention described in claim 3 is characterized in that, in the invention described in claim 2, the coefficient value is set to be larger as the average luminance of the whole image of the reference image is lower.
[0009]
That is, according to the first aspect of the present invention, the first average obtained by averaging the image data of each pixel in the small area from the image data of each pixel in the small area of the reference image with respect to the pair of images captured by the stereo camera. A first data group obtained by subtracting a value, and a second data group obtained by subtracting a second average value obtained by averaging the image data of each pixel in the small area from the image data of each pixel in the small area of the comparison image The image data of each pixel in the small area of the reference image is compared with a determination value set based on the average value of the image data of the entire image of the reference image to determine whether it is valid or not. Valid data is selected from the data group and the second data group. Then, by calculating the city block distance by summing the absolute values of the differences between the valid data of the first data group and the valid data of the second data group, there is not only low frequency noise but also high frequency noise. Even for images, accurate stereo matching is possible without being affected by noise.
[0010]
At this time, as in the invention described in claim 2, the determination values for the first data group and the second data group are set as the first average value, and the coefficient value based on the image data average value of the entire image of the reference image is set as the first average value. It is desirable to set by multiplication or subtraction . The coefficient value is desirably set larger as the average luminance of the entire reference image is lower.
[0011]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described below with reference to the drawings. 1 to 5 relate to an embodiment of the present invention, FIG. 1 is a basic configuration diagram of a stereo image processing apparatus, FIG. 2 is a block diagram of a stereo processing unit, and FIG. 3 is a small area of a main image and a sub image. FIG. 4 is an explanatory diagram showing data, FIG. 4 is an explanatory diagram of a coefficient value table, and FIG. 5 is a timing chart of image processing.
[0012]
FIG. 1 shows a three-dimensional distance information obtained by processing a pair of images obtained by stereo imaging of a target landscape, for example, mounted on a moving body such as an automobile or a helicopter, and based on the obtained distance information. 1 shows a basic configuration of a stereo image processing apparatus used for a device for recognizing an image, a stereo camera 10 composed of a set of two cameras, and an image input unit 20 that performs input processing on an image captured by the stereo camera 10 , An original image memory 25 that stores the captured image processed by the image input unit 20 as an original image, a stereo processing unit 30 that stereo-processes the original image to acquire distance distribution information (distance image), and a distance at which the distance image is stored The image memory 35 includes a recognition processing unit 40 that reads a distance image and performs various recognition processes.
[0013]
The two cameras 10a and 10b constituting the stereo camera 10 are CCD cameras that are synchronized with each other and have a variable shutter speed, and the main camera that captures a reference image for stereo processing of one CCD camera 10a. The other CCD camera 10b is arranged as a sub-camera for capturing a comparative image in stereo processing so that the vertical axes of the imaging surfaces are parallel to each other with a predetermined baseline length.
[0014]
The image input unit 20 includes an amplifier and an A / D converter for processing two analog image pickup signals from the CCD cameras 10a and 10b, and various functional circuits for image processing configured by, for example, a highly integrated FPGA. Further, an image adjustment circuit for electrically correcting a slight shift of mechanical optical positions of the CCD cameras 10a and 10b, lens distortion, and the like is provided. Images taken from the CCD cameras 10a and 10b are obtained. The image data is converted into digital image data having a predetermined luminance gradation, and an error in the mechanical attachment positions of the CCD cameras 10a and 10b, lens distortion, and the like are corrected by image adjustment and stored in the original image memory 25.
[0015]
The stereo processing unit 30 calculates the city block distance for each of the small areas of each image for the two images of the main image (reference image) and the sub-image (comparison image) stored in the original image memory 25, and calculates each other. 3D image obtained by performing stereo matching to identify the corresponding small area by obtaining the correlation of, and digitizing the perspective information from the pixel shift (= parallax) generated according to the distance to the object to the object Information (distance image) is acquired, and the function related to the matching sequence is similarly configured by a highly integrated FPGA or the like.
[0016]
The recognition processing unit 40 includes a RISC processor for performing various recognition processes at high speed based on distance distribution information obtained from a distance image. For example, when mounted on a vehicle, the recognition processing unit 40 travels such as a road shape and the presence of a preceding vehicle. Recognizes the environment, determines the risk of collision, issues a warning to the driver, automatically activates the brake to stop, or automatically adjusts the traveling speed to keep the distance between the preceding vehicle safe Outputs vehicle control data such as increase / decrease.
[0017]
The stereo matching in the stereo processing unit 30 is basically the same as the process described in detail in Japanese Patent Application Laid-Open No. 11-234701 previously submitted by the applicant of the present invention, but in Japanese Patent Application Laid-Open No. 11-234701. In the processing, average value difference matching is performed in which a luminance average value of a small region that is matched in advance with the original image is taken, and a city block distance is calculated by a value obtained by subtracting the luminance average value from the luminance value of each pixel in the small region. On the other hand, in the present invention, average value difference matching with a pixel selection function for selecting pixels in a small area to be used for matching based on the luminance average value of the small area of the reference image is performed.
[0018]
This average value difference matching with a pixel selection function has the advantage of average value difference matching, that is, only high frequency components of an image are targeted for matching and has an action equivalent to a high-pass filter, so that low frequency noise can be removed. In addition to the advantage of being able to reduce the effects of slight brightness imbalance between two images and the effects of gain changes due to aging of cameras and analog circuit components, pixels can also be selected for high-frequency noise By doing so, mismatching does not occur, and accurate distance information can be obtained.
[0019]
Specifically, the functional configuration of the stereo processing unit 30 is calculated by a main image average value processing unit 50 that calculates a luminance average value of a small area of the main image, as shown in FIG. Main average dual-port memory 55 for storing the luminance average value data of the small area of the main image, and a sub image average value processing unit for calculating the luminance average value of the small area of the sub image while shifting the luminance average value by one pixel from the main image 60, the sub average value dual port memory 65 for storing the luminance average value data of the small area of the sub image calculated by the sub image average value processing unit 60, the sub image and the main image, each pixel in the small area An average value difference unit 70 for calculating a value obtained by subtracting the luminance average value from the luminance value, and determining whether a pixel in the small area is valid or invalid based on the luminance average value of the small area of the main image. The effective pixel determination unit 80, the data selection unit 90 that selectively outputs the average value difference data based on the determination result of the effective pixel determination unit 80, the value obtained by subtracting the luminance average value from the luminance value of the main original image, and the sub A stereo matching unit 100 that calculates the city block distance by summing the absolute values of the difference between the luminance value of the original image and the luminance average value, and evaluates the minimum and maximum values of the city block distance to evaluate the city block It is composed of a city block distance check unit 110 and the like for checking whether the minimum value of the distance really indicates a match between the small areas of the two images, and further, a singular point for eliminating abnormal distance data A removal filter unit 120 is provided as an accessory.
[0020]
In this embodiment, the size of the original image is 500 × 200 pixels, the processing range is 420 × 160, and matching is performed in a small region of 8 × 4 pixels. , Luminance data Mi, j (i = 1 to 8, j = 1 to 4) of each pixel in a small area of 8 × 4 pixels in the main image, and 8 × 4 pixels in the sub image. Average luminance values AM and SM shown in the following equations (1) and (2) are obtained for the luminance data Si, j (i = 1 to 8, j = 1 to 4) of each pixel in the small area of . Here, Σ is the sum of i = 1-8 and j = 1-4.
AM = ΣMi, j / 32 (1)
SM = ΣSi, j / 32 (2)
[0021]
Then, as shown in the following equation (3), a value obtained by subtracting the luminance average value AM from the luminance data Mi, j of the main original image and a value obtained by subtracting the luminance average value SM from the luminance data Si, j of the sub image The city block distance CB is calculated by summing up the absolute values of the differences. However, the pixels to be added in the calculation of the city block distance are pixels that satisfy the validity determination condition of Mi, j> (AM × k) or Mi, j> (AM−k) as described below. , K are coefficient values used for validity determination.
CB = Σ│ (Mi, j-AM)-(Si, j-SM) │ ... (3)
[0022]
The luminance average value of the small area is calculated every four lines of the original image, and the main image average value processing unit 50 adds an adder to a 4-byte addition block 51 that combines three 2-byte adders. And two adder blocks 52 and 53 (additions 1 and 2 in the figure), which are a combination of a D flip-flop and a shift register, are connected in parallel. The main average value is output to the dual port memory 55.
[0023]
On the other hand, the sub-image average value processing unit 60 calculates a luminance average value while shifting a small area of 8 × 4 pixels horizontally by one pixel at a time in the horizontal scanning direction. The adder block 61 is connected to nine adders (additions 1 to 9 in the figure) connected in parallel and an adder averaging block 62 composed of eight multiplexers. The addition average block 62 processes the adder outputs of every two additions 1 to 8 in a three-stage configuration with seven multiplexers, and further adds the multiplexer output of the three-stage configuration and the addition 9 in the final stage multiplexer. Data selection is performed on the output of the device, and the average value is output.
[0024]
The average value difference unit 70 is composed of four average value difference blocks 71, 72, 73, 74 (average value differences 1 to 4 in the figure) corresponding to the four lines of the original image, each for the main image. And a sub-image differentiator. In the main image difference unit, the process of subtracting the average value data of the main average value dual port memory 55 from the main image data of the original image memory 25 is sequentially performed for 8 bytes in width through the multiplexer and the 8-stage shift register. In the sub image subtractor, the average value data in the sub average value dual port memory 65 is subtracted from the sub image data in the original image memory 25.
[0025]
Similarly, the valid pixel determination unit 80 sets four validity determination units 81 to 84 (valid determinations 1 to 4 in the figure) corresponding to four lines of four original images, and coefficient value k for validity determination. And a determination value calculation unit 85. Each of the validity determination units 81 to 84 includes a multiplier (or a differentiator) and a comparator, and the determination value calculation unit 85 sets the luminance average value for each small area from the main average value dual port memory 55. The judgment value obtained by multiplying or subtracting the coefficient value k and the pixel brightness value of the main image from the original image memory 25 are compared by a comparator, and pixels brighter (higher brightness) than the judgment value are validated. Dark (low brightness) pixels are invalidated, and pixels effective for the calculation of the city block distance are extracted. As a result, it is possible to eliminate dark pixels that are greatly affected by noise and prevent mismatching, and it is also possible to apply to infrared images and the like.
[0026]
In the present embodiment, the coefficient value k set by the determination value calculation unit 85 is determined based on the average luminance value obtained by obtaining the average luminance value of all the images for the main image in the original image memory 25. For example, when the luminance level is expressed in 256 gradations, a coefficient value table is created by dividing the luminance level into four stages of 0 to 63, 64 to 127, 128 to 191 and 192 to 255 as shown in FIG. At each stage, a multiplication constant multiple value (for example, 1 time, 0.75 time, 0.5 time, 0.25 time) or a subtraction constant value (for example, 128, 64, 32, 0) Is stored in advance, and the coefficient value k is set by referring to this coefficient value table. The coefficient value k is set larger as the average luminance of the entire main image is lower.
[0027]
The data selection unit 90 is composed of a total of eight selection units 91 to 98 (selections 1 to 8 in the figure), two for each of the four average value difference blocks 71, 72, 73, and 74. And a multiplexer. This multiplexer selectively outputs the output data from each block and “0” based on the determination output (comparator output) from the valid pixel determination unit 80, and clears the output data from each block when it is invalid. When valid, output data from each block is passed through.
[0028]
The stereo matching unit 100 has a configuration in which a horizontal addition block 102 is connected to a vertical addition block 101 in which four differentiators, four absolute value arithmetic units, and three adders are connected in a pyramid shape. . The vertical addition block 101 has a four-stage configuration, and the first stage is composed of four differentiators that take the difference between the main image data and the sub-image data of each average value difference block 71, 72, 73, 74. The second stage is composed of four absolute value calculators. The output data of the second stage absolute value calculation is added every two by the third stage adder (two), and the output data of the third stage two adders is the final stage of the fourth stage. To the adder. The adder output at the final stage of the vertical addition block 101 is input to a horizontal addition block 102 that is a combination of an adder and a D-flip flop, and a calculation result of the city block distance is output from the horizontal addition block 102.
[0029]
In the stereo image processing apparatus having the above configuration, as shown in the time chart of FIG. 5, a distance image is generated at a processing cycle of every 100 ms.
[0030]
That is, image adjustment is performed by sampling an image at the beginning of the processing cycle, and a distance image of a necessary area is generated in the remaining processing time. In the example of FIG. 5, stereo processing is performed for 40 lines (line 00 to line 27), that is, 420 horizontal pixels × 160 vertical pixels in one cycle, and in this stereo processing, the head of processing for one line is performed. Thus, the process of obtaining the average value of the luminance data of the small area of 8 × 4 pixels for the four lines of the original image of 420 × 160 pixels is performed. In this averaging process, the luminance average value of each small area of the main original image for 40 bytes is stored in the main average value dual port memory 55, and the sub average value dual port memory 65 stores the luminance average value of each small area of the sub original image. The average luminance value is stored for 413 bytes (420-7 = bytes).
[0031]
For the main original image, the 4-byte addition block 51 of the main image average value processing unit 50 adds 4 bytes of the main original image, and the addition result is added to the addition block 52 (addition block 53) in the next stage. It is added to the previous 4 bytes of data latched in the D flip-flop. By repeating this eight times by pipeline processing, an operation of 8 × 4 = 32 bytes is performed on a small area, and an average value is obtained by bit shifting with a shift register. The two addition blocks 52 and 53 are used for bank switching in order to prevent the processing flow from becoming non-uniform due to a calculation delay, and the calculation for 32 bytes is alternately performed up to the horizontal 320 pixels of the main original image. Then, 40 bytes (320/8 = 40 bytes) of average value data (first average value) is output to the main average value dual port memory 55 through the multiplexer 54.
[0032]
In the sub-original image, the luminance average value of a small area of 8 × 4 pixels in the horizontal direction is calculated with respect to the main original image up to 100 pixel shift while shifting one pixel at a time in the horizontal scanning direction. The 4-byte addition block 61 adds the vertical 4 bytes of the sub-original image, and the addition 1 adder of the next-stage addition averaging block 62 adds the horizontal 8 bytes from the first pixel to the eighth pixel. Adder 2 adds 8 bytes in the horizontal direction from the 2nd pixel to the 9th pixel, and Adder 3 adds 8 bytes in the horizontal direction from the 3rd pixel to the 10th pixel. The addition of 4 bytes from the 4th pixel to the 11th pixel is performed by the adder of addition 4, and the addition of 8 bytes is performed from the 5th pixel to the 12th pixel by the adder of addition 5. , Adder 6 adds 8 bytes from the 6th pixel to the 13th pixel Addition is performed, addition of 8 bytes in the horizontal direction from the 7th pixel to the 14th pixel is performed in the adder of addition 7, and 8 bytes in the horizontal direction from the 8th pixel to the 15th pixel is performed in the adder of addition 8 Addition is performed, and an adder for addition 9 performs addition for the horizontal 8 bytes from the 9th pixel to the 16th pixel.
[0033]
Next, the adder of addition 1 to the adder of addition 9 are sequentially used so that the addition of 8 bytes in the horizontal direction from the 10th pixel to the 17th pixel is executed again by the adder of addition 1. The summed average value is output through a multi-stage connected multiplexer. This average value calculation is executed for the central 420 pixels of the sub-original image, and the average value data (second average value) for 413 bytes (420-7 = 413 bytes) is output to the sub-average value dual port memory 65. . In parallel with this, the determination value calculation unit 85 of the effective pixel determination unit 80 obtains an average luminance value for all images in the original image memory 25, sets a coefficient value k for the average luminance value, and determines an effective determination. Output to sections 81-84.
[0034]
Then, stereo processing (area00 to area27) is started for 40 small regions in the main image, and city block processing for calculating the city block distance is started. In this city block processing, the difference calculation for one line of the original image is performed in the average value difference blocks 71, 72, 73, 74, respectively, and the average value is subtracted from the luminance data for each pixel in the small area of the main image. Individual data of the difference data group (first data group) and the difference data group (second data group) obtained by subtracting the average value from the luminance data for each pixel in the small area of the sub-image are sequentially input to the data selection unit 90. In parallel with this, the validity determination units 81 to 84 compare the pixel luminance value with the value obtained by multiplying or subtracting the coefficient value k from the luminance average value of the small area of the main image, and determine whether the corresponding pixel is valid or invalid. And the determination result is output to the data selection unit 90. In the data selection unit 90, when the determination result is valid, the data group from the average value difference blocks 71, 72, 73, 74 is sequentially output to the stereo matching unit 100 as it is, and when the determination result is invalid, the stereo matching unit “0” is output to 100.
[0035]
This process is sequentially performed up to 100 pixel shifts for each pixel shift (dpx00 to dpx64), and in the vertical addition block 101 of the stereo matching unit 100, the average value difference blocks 71, 72, 73, and 74 are output at the first stage, respectively. The sub-image difference data is subtracted from the main image difference data, the absolute value calculation is performed by the second-stage absolute value calculator, and the second-stage absolute value is calculated by each of the two third-stage adders. Two simultaneous input data from the arithmetic unit are added and the output of the previous stage is added at the fourth stage.
[0036]
Then, the calculation result for the vertical 4 bytes in the vertical addition block 101 is input to the horizontal addition block 102, and the process proceeds to the horizontal 8 bytes (row addition processing: row00H to row7H). In this horizontal addition block 102, the data for the vertical 4 bytes latched in the D-flip-flop is sequentially added to obtain a total of 8 × 4 = 32 pixels, and the calculated value of the city block distance is output.
[0037]
The city block distance output from the stereo matching unit 100 is evaluated by the city block distance check unit 110 for the minimum value and the maximum value, and the minimum value of the city block distance really indicates a match between the small areas of the two images. It is checked whether it is a thing. Then, the pixel deviation amount that satisfies the check condition and the city block distance is minimized is stored in the distance image memory 35 as effective distance data, and a singular point for eliminating local abnormal data with a delay of two lines Removal filter processing is performed.
[0038]
【The invention's effect】
As described above, according to the present invention, low-frequency noise is removed by matching only the high-frequency component of the image, and mismatching does not occur by selecting an effective pixel for high-frequency noise. Accurate distance information can be obtained.
[Brief description of the drawings]
FIG. 1 is a basic configuration diagram of a stereo image processing apparatus. FIG. 2 is a block diagram of a stereo processing unit. FIG. 3 is an explanatory diagram showing data of small areas of a main image and a sub image. FIG. 5 is a timing chart of image processing.
DESCRIPTION OF SYMBOLS 10 Stereo camera 30 Stereo processing part 50 Main image averaging process part 60 Sub image averaging process part 70 Average value difference part 80 Effective pixel determination part 90 Data selection part 100 Stereo matching part

Claims (3)

For a pair of images composed of a reference image and a comparison image captured by a stereo camera, a corresponding small region is searched by calculating a city block distance between the small regions of each other image, and the target region is detected in the corresponding small region. In a stereo image processing apparatus for generating a distance image in which perspective information based on a pixel shift amount generated according to a distance is quantified,
Means for averaging the image data of each pixel in the small area of the reference image and calculating as a first average value;
Means for averaging the image data of each pixel in the small area of the comparison image and calculating as a second average value;
Means for subtracting the first average value from the image data of each pixel in the small area of the reference image and calculating as a first data group;
Means for subtracting the second average value from the image data of each pixel in the small area of the comparison image and calculating as a second data group;
The image data of each pixel in the small area of the reference image is compared with a determination value set based on an average value of image data of the entire image of the reference image to determine whether it is valid, and the first Means for selecting valid data of the data group and the second data group;
A stereo image comprising means for summing up absolute values of differences between the effective data of the first data group and the effective data of the second data group and calculating the city block distance. Processing equipment. 2. The stereo image processing apparatus according to claim 1, wherein the determination value is set by multiplying or subtracting the first average value by a coefficient value based on an image data average value of the entire image of the reference image. . 3. The stereo image processing apparatus according to claim 2, wherein the coefficient value is set to be larger as the average luminance of the entire image of the reference image is lower.

Images