JP7616973B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing device and an image processing method that uses multiple cameras to recognize obstacles outside the vehicle.

To improve vehicle driving safety, there are systems that use on-board forward monitoring sensors to detect obstacles in front of the vehicle, and if there is a risk of the vehicle colliding with the obstacle, they will warn the driver and automatically apply the brakes.

Sensors used in such systems for monitoring the forward movement include millimeter-wave radar, laser radar, and cameras. Camera types include monocular cameras and stereo cameras that use multiple cameras. Stereo cameras can measure the distance to an object photographed by using the parallax of the overlapping area photographed by two cameras spaced at a specified distance. This makes it possible to accurately grasp the risk of collision with an object ahead.

A stereo camera calculates the disparity between images captured by two cameras and converts the disparity into distance. A characteristic of stereo cameras is that the disparity becomes smaller as the measurement distance becomes farther. A known method of calculating disparity is to check the correspondence between the left and right images using block matching.

However, the block matching method of disparity calculation has the problem that the effective disparity is reduced in images with little texture, resulting in a decrease in disparity accuracy. Recently, therefore, a method of calculating disparity using a neural network has been developed as an alternative to the block matching method.

For example, the abstract of Patent Document 1 states that the problem is to "provide a technology capable of correcting disparity values in mismatched regions in a distance image," and states that the solution is to "provide a distance image generating device 110 mounted on a vehicle 10, which uses left and right captured images taken by a stereo camera 122 as a reference image and a comparison image to generate a distance image showing the distance to an object present in the reference image, a comparison image generating unit 112 which generates a comparison image to be compared with the distance image using a neural network, a correction area detecting unit 113 which detects a correction area in the distance image corresponding to an area in the reference image where the image feature amount is equal to or less than a predetermined threshold, and a correction unit 114 which corrects the distance information of each pixel in the correction area according to information on the corresponding part of the correction area in the comparison image." That is, Patent Document 1 proposes a method of applying a parallax calculation using a neural network to a part of an image (a part with few features such as the road surface).

JP 2020-143957 A

The neural network method of Patent Document 1 overcomes the problems with the block matching method and can calculate effective parallax with high accuracy even from images with little texture, but it has the problem of requiring a huge amount of calculations.

The present invention aims to use an image processing device that uses multiple cameras to calculate parallax using a neural network with a reduced amount of calculation, and to use that parallax as a base to increase the effective parallax of the entire image by block matching, thereby generating a highly accurate parallax image over the entire input image.

To solve the above problem, the present invention includes an image reduction unit that reduces two input images to generate two reduced images, a first parallax image generation unit that calculates the parallax between the two reduced images by neural network processing and generates a reduced parallax image, and a second parallax image generation unit that calculates the parallax between the two input images and generates a parallax image by calculating an area in one of the input images that matches a partial area in the other input image, and the second parallax image generation unit sets a search range when calculating the matching area for each area of the input images to which each pixel of the reduced parallax image corresponds, based on the parallax value of each pixel of the reduced parallax image.

According to the present invention, the effective parallax is calculated from the reduced image by a neural network, and the parallax of the pre-reduced image at a position corresponding to the effective parallax is calculated by a block matching method so that it is the same as or close to the effective parallax, so that the amount of calculations by the neural network can be reduced while increasing the effective parallax of the pre-reduced image.

Functional block diagram of an image processing apparatus according to a first embodiment 1 is a flowchart of image processing in the image processing apparatus according to the first embodiment. A diagram explaining a disparity calculation method using the block matching method. FIG. 1 is a diagram for explaining an example of a method for determining a search range for block matching in the first embodiment; FIG. 11 is a diagram for explaining another example of a method for determining a search range for block matching in the first embodiment; An example of an image with large feature values Functional block diagram of an image processing apparatus according to a second embodiment 11 is a flowchart of a road surface height estimation process of the image processing device according to the second embodiment. Functional block diagram of an image processing apparatus according to a third embodiment Flowchart of first parallax image generating process according to the fourth embodiment Flowchart of second parallax image generating process according to the fifth embodiment

The following describes an embodiment of the present invention with reference to the drawings. Note that the following description shows specific examples of the contents of the present invention, and the present invention is not limited to these descriptions. Various changes and modifications are possible by those skilled in the art within the scope of the technical ideas disclosed in this specification. In addition, in all drawings used to explain the present invention, parts having the same functions are given the same reference numerals, and repeated explanations may be omitted.

1 is a functional block diagram of an image processing device 100 according to a first embodiment of the present invention. As shown in the figure, the image processing device 100 includes a right camera 1R, a left camera 1L, an image reduction unit 2, a first parallax image generation unit 3, a search range determination unit 4, a second parallax image generation unit 5, and a recognition processing unit 6. The right camera 1R and the left camera 1L do not necessarily need to be the left and right cameras of a stereo camera, and a pair of monocular cameras arranged on the left and right may be used as the right camera 1R and the left camera 1L, respectively. In addition, the image reduction unit 2 to the recognition processing unit 6 are functional units realized by a calculation unit such as a CPU, a storage device such as a semiconductor memory, and a computer equipped with hardware such as a communication device, by the calculation unit executing a predetermined program in the storage device. In the following, the details of each unit will be described while appropriately omitting such well-known technologies in the computer field.

As shown in Fig. 1, images captured by the right camera 1R and the left camera 1L are transferred to the image reducing unit 2 as a right image P _R and a left image P _L , respectively. The image reducing unit 2 reduces the input left and right images to generate a right reduced image S _R and a left reduced image S _L. The reason for reducing the images is to reduce the amount of calculation in the first parallax image generating unit 3, and the reduction ratio is determined according to the processing time allowed by the system for the first parallax image generating unit 3. The first parallax image generating unit 3 receives the right reduced image S _R and the left reduced image S _L as input, and generates a first parallax image D1 (reduced parallax image) and a feature map M by neural network processing.

The search range determination unit 4 uses the first parallax image D1 and the feature map M to determine the search range of the second parallax image generation unit 5. The second parallax image generation unit 5 generates the second parallax image D2 from two images, the right image P _R and the left image P _L, by a block matching method, and the search range of the block matching follows the search range determination unit 4. The second parallax image D2 is a parallax image with a higher density than the first parallax image D1 because the parallax is generated from the image before being reduced. The recognition processing unit 6 uses the second parallax image D2 and the right image P _R to recognize a vehicle, a pedestrian, an obstacle, and the like ahead. Then, the recognition result of the recognition processing unit 6 is input to an ECU (not shown) that controls the drive system, braking system, steering system, and the like of the vehicle, so that an alarm can be issued to the driver or automatic braking can be applied as necessary.

Figure 2 is a flowchart outlining the image processing performed by the image processing device 100 of this embodiment.

First, in step S10, the image reducing unit 2 reduces the right image P _R and the left image P _L input from the left and right cameras to generate a right reduced image S _R and a left reduced image S _L.

Next, in step S11, the first parallax image generating unit 3 generates a first parallax image D1 and a feature map M by neural network processing using the right reduced image S _{4 R} and the left reduced image S 4 _L as input.

In step S12, the search range determination unit 4 determines the search range of the second parallax image generation unit 5 using the first parallax image D1 or using the first parallax image D1 and the feature map M. Details of the method of using the first parallax image D1 are described in FIG. 4, and details of the method of using the first parallax image D1 and the feature map M are described in FIG. 5.

In step S13, the second parallax image generating unit 5 generates a second parallax image D2 from the two images, the right image P _R and the left image P _L, by a block matching method using the search range determined by the search range determining unit 4. Details of the block matching method will be described with reference to FIG.

In step S14, the recognition processing unit 6 uses the second parallax image D2 and the right image _PR to recognize a forward vehicle, a pedestrian, an obstacle, and the like.

FIG. 3 illustrates a parallax calculation process by a general block matching method. In this example, the right image P _R captured by the right camera 1R is used as a reference image, and a reference block image P _B , such as 16 pixels x 16 pixels (the size is not limited to this example), is defined. Then, in the left image P _L captured by the left camera 1L, a reference image P _ref with a predetermined search width r (for example, 272 pixels) is selected based on the same vertical position (Y coordinate) and horizontal position (X coordinate) as the reference block image P _B. Then, the difference between the reference block image P _B and the reference image P _ref is calculated. This difference calculation is called SAD, and is performed according to the following formula 1.

In addition, in formula 1, I is an image block (e.g., 16×16 pixels) in the reference image _Pref , T is an image block in the base block image P _B , and i, j are coordinates in the image block. In order to calculate one parallax, the reference position of the reference image _Pref is shifted one pixel at a time from the left end and compared with all images in the search width r. Therefore, if the base block image P _B has a width of 16 pixels and the search width r is 272 pixels, 256 comparison calculations of the block images are performed to search for the position with the smallest SAD value.

In the actual scene seen from the front window of the vehicle in FIG. 3, there is a vehicle V ahead running on the road. When there is a right image P _R captured by the right camera 1R and a left image P _L captured by the left camera 1L, a part V1 of the vehicle ahead V is captured at the position of the base block image P _B in the right image P _R , and at the position of the reference block image P _B ' in the left image P _L. As a result, the SAD value of the base block image P _B and the reference block image P _B ' is minimized at the position of the parallax d. This parallax d is a large value when the vehicle ahead V is close to the image processing device 100, and a small value when it is far away. The parallax calculated in this way is calculated for the entire image. Using this parallax d, the distance to the image processing device 100 can be measured according to the principle of triangulation. From the parallax d, the distance Z can be calculated by the following formula 2.

In Equation 2, f is the focal length of the left and right cameras, and B is the distance (baseline length) between the right camera 1R and the left camera 1L.

<Example of detailed operations of the search range determination unit 4 and the second parallax image generation unit 5>
FIG. 4 shows an example of detailed operations of the search range determination unit 4 and the second parallax image generation unit 5, and corresponds to the flow of determining the search range for block matching based on the first parallax image D1 in step S12 in FIG. 2.

As shown in FIG. 4, the right image P _R and the left image P _L are assigned coordinates X from left to right in the horizontal direction and Y from top to bottom in the vertical direction. The right image P _R and the left image P _L are reduced by the image reduction unit 2 to become a right reduced image S _R and a left reduced image S _L. Here, the reduction ratio is set to 1/4 as an example. Next, the first parallax image generation unit 3 generates a first parallax image D1 by neural network processing using the right reduced image S _R and the left reduced image S _L as inputs. In this example, since the reduction ratio of the image is 1/4, the pixel of the parallax value A of the first parallax image D1 corresponds to the positions of four pixels of the right image P _R, namely, X coordinate = 0 and Y coordinate = 0, X coordinate = 0 and Y coordinate = 1, X coordinate = 1 and Y coordinate = 0, and X coordinate = 1 and Y coordinate = 1.

The search range determination unit 4 determines the range of block matching of the second parallax image generation unit 5 by the first parallax image D1. For example, when the second parallax image generation unit 5 generates parallax of four _pixels of X coordinate=0 and Y coordinate=0, X coordinate=0 and Y coordinate=1, X coordinate=1 and Y coordinate=0, and X coordinate=1 and Y coordinate=1 in the right image PR, the search range is determined based on the parallax value A of X coordinate=0 and Y coordinate=0 in the first parallax image D1 corresponding to the four pixels. Specifically, the start position s of the search range is a value obtained by subtracting a fixed value from the parallax value A by a subtracter 41, and the end position e of the search range is a value obtained by adding a fixed value to the parallax value A by an adder 42. The fixed value is, for example, 10. As a result, the four pixels (A1, A2, A3, A4) in the second parallax image D2 corresponding to the position of the parallax value A have a parallax value that is close to or equal to the parallax value A, which is the effective parallax value, and are therefore highly likely to be effective parallax values, since the search range of the block matching is limited to the vicinity of the effective parallax value. Also, the number of block matching operations required to calculate the parallax of one pixel is only 20. In a general block matching method (FIG. 3), the search range is fixed and, for example, 256 SAD operations are required, but in this method, the amount of calculations for block matching is 1/8 or less.

<Another Example of Detailed Operation of Search Range Determination Unit 4 and Second Parallax Image Generation Unit 5>
Fig. 5 shows another example of detailed operations of the search range determination unit 4 and the second parallax image generation unit 5, and corresponds to the flow of determining the search range for block matching based on the first parallax image D1 and the feature map M in step S12 in Fig. 2. Differences from Fig. 4 will be described below.

The first parallax image generator 3 calculates the parallax by neural network calculation. The neural network performs convolution calculation processing, and as a result, a feature map M is generated. Parts of an image where there is a large change in pixel value tend to have large feature values. For example, as shown in FIG. 6, image region R1 is an image of only the road surface, so there are few feature values, and the distance in this region is almost the same, so the parallax value is also an approximate value. On the other hand, image region R2 includes the road surface, part of the forward vehicle V, and traffic signs, and multiple objects overlap, resulting in a large feature value. The forward vehicle V, road surface, and traffic signs in this image region R2 are all at different distances, so the parallax also differs, so it is necessary to prevent the parallax in this image region R2 from becoming an approximate value.

In FIG. 4, the four pixels (pixel values A1, A2, A3, and A4) in the second parallax image D2 that correspond to pixel value A in the first parallax image D1 are all approximations of the parallax value A. Therefore, if there is a boundary between a distant object and a nearby object among these four pixels, there is a possibility that the parallax is not an approximation of the parallax value A.

5, a subtractor 43 and an adder 44 for the feature map M, which have the same functions as the subtractor 41 and the adder 42 for the first parallax image D1, are provided, so that the search range is made wider in the part where the feature amount indicated by the feature map M is large , and the correct parallax can be calculated. Specifically, when the second parallax image generating unit 5 processes the part corresponding to pixel A of the first parallax image D1, the value of pixel i of the feature map M is subtracted from the start position s of the search range and added to the end position e. When the part corresponding to pixel B of the first parallax image D1 is processed, pixel v of the feature map M is used. In this way, the search range can be expanded in the part where the feature amount is large , and the correct parallax can be calculated.

According to the image processing device of this embodiment described above, the effective parallax is calculated from the reduced image by a neural network, and the parallax of the image before reduction at a position corresponding to the effective parallax is calculated by a block matching method so that it is the same as or close to the effective parallax, so that the effective parallax of the image before reduction can be increased while reducing the amount of calculations of the neural network.

Next, an image processing device according to a second embodiment will be described with reference to Figures 7 and 8. Note that a duplicate description of the points common to the first embodiment will be omitted.

Figure 7 is a functional block diagram of an image processing device 100 according to a second embodiment. The difference from the first embodiment is that the first parallax image D1 is used when the recognition processing unit 6 recognizes the road surface. When the recognition processing unit 6 detects a forward vehicle V or an obstacle, it is necessary to determine whether they are present on the road surface. For example, when the first parallax image generation and the second parallax image generation are processed by hardware and the recognition processing is processed by software, the road surface recognition processing (software processing) can be processed in parallel with the generation processing (hardware processing) of the second parallax image D2, and the processing speed of the entire system can be increased.

Figure 8 shows a flowchart when the first parallax image D1 is used by the recognition processing unit 6 to recognize the height of the road surface. First, after generating the first parallax image D1 (step S20), the recognition processing unit 6 performs a process of recognizing the height of the road surface using the first parallax image D1 (step S21), and in parallel with this, the second parallax image generation unit 5 generates the second parallax image D2 (step S22). After that, the recognition processing unit 6 uses the second parallax image D2 to detect a vehicle V ahead on the road and obstacles (step S23). In this case, the processing speed of the entire system can be increased compared to when the recognition processing unit 6 performs all processing using the second parallax image D2.

Next, an image processing device according to a third embodiment will be described with reference to FIG. 9. Note that a duplicate description of points common to the above embodiments will be omitted.

FIG. 9 is a functional block diagram of an image processing device 100 according to a third embodiment. The difference from the first embodiment is that a parallax accuracy reduction area detection unit 7 is added. The parallax accuracy reduction area detection unit 7 determines an image area where the first parallax image generation unit 3 may not generate a correct parallax, and the first parallax image generation unit 3 invalidates a part of the first parallax image D1 and the feature map M corresponding to the determined area. For example, when a wiper is reflected in a part of the right image P _R and the left image P _L during rainy weather, an accurate parallax cannot be calculated from the image of the wiper part. The first parallax image generation unit 3 is notified to output an invalid parallax for such a part. The parallax accuracy reduction area detection unit 7 may detect an image area where the brightness value of a part of the image is extremely dark compared to other parts.

Next, an image processing device according to a fourth embodiment will be described with reference to FIG. 10. Note that a duplicate description of points common to the above embodiments will be omitted.

Figure 10 shows a flowchart of a method for changing the reduction ratio of the image reduction unit 2 depending on the processing position of the image. For example, in the case of an image of an actual landscape such as that shown in Figure 3, the lower part of the image of the road surface is closer to the camera than the upper part of the image of the road surface, so the road width is wider and objects are photographed larger. Therefore, it is considered that parallax can be calculated without any problem even if the reduction ratio is smaller in the lower part of the image.

In the flowchart of FIG. 10, the captured image is divided into two parts, top and bottom, and the image is reduced by changing the reduction ratio according to the value of the Y coordinate of the divided image, and the first parallax image generating unit 3 performs processing. Specifically, the Y coordinates of the left and right images are checked (step S40), and if the Y coordinate is equal to or greater than a predetermined value (at the lower part of the image where it is highly likely that a nearby object is being photographed) the reduction ratio is set to 1/8 (step S41). On the other hand, if the Y coordinate is smaller than the predetermined value (at the upper part of the image where it is highly likely that a distant object is being photographed) the reduction ratio is set to 1/4 (step S42). Then, the image reducing unit 2 generates a reduced image at the reduction ratio set in steps S41 and 42 (step S43), and the first parallax image generating unit 3 generates the first parallax image D1 and the feature map M using the left and right reduced images with different reduction ratios for the top and bottom. In this way, the amount of calculation can be reduced by the amount of reduction in the reduction ratio of the image.

Next, an image processing device according to a fifth embodiment will be described with reference to FIG. 11. Note that a duplicate description of points common to the above embodiments will be omitted.

Figure 11 is a flowchart for changing the fixed value that determines the search range according to the processing position of the image. As described above, the lower side of the road surface image captures the vicinity, and the upper side of the road surface image captures the distant area. As shown in Equation 2, the parallax is inversely proportional to the distance, so a difference of 1 in the parallax value in the near area is a difference of several tens of centimeters in distance, but a difference of 1 in the parallax value in the distant area is a difference of several meters in distance. Therefore, in order to bring the accuracy of the second parallax image D2 closer to that of the first parallax image D1, it is desirable to change the search range according to the distance. Therefore, when processing the image of the road surface, if the Y coordinate is smaller than a predetermined Y coordinate, the fixed value is changed to 5 to determine the search range, and the second parallax image generation unit 5 generates the second parallax image D2. Specifically, the Y coordinates of the left and right images are checked (step S50), and if the Y coordinate is equal to or greater than a predetermined value (at the lower side of the image where it is highly likely that the vicinity is being captured), the fixed value is set to 10 (step S51). On the other hand, if the Y coordinate is smaller than the predetermined value (at the top of the image where it is highly likely that a distant object is being photographed), a fixed value of 5 is set (step S52). Then, the search range determination unit 4 determines the search range using the fixed values set in steps S51 and S52 (step S53), and the second parallax image generation unit 5 generates the second parallax image D2 using a search region with different widths above and below. By doing this, the amount of calculations can be reduced by the amount that the width of the search range is reduced.

The present invention is not limited to the above-described embodiments, but includes various modified examples. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described. It is also possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

100: Image processing device 1L: Left camera 1R: Right camera 2: Image reduction unit 3: First parallax image generation unit 4: Search range determination unit 41, 43: Subtractor 42, 44: Adder 5: Second parallax image generation unit 51: SAD calculator 6: Recognition processing unit 7: Parallax accuracy reduction area detection unit P _L : Left image P _R : Right image P _B : Base reference block image P _B ': Reference block image S _R : Right reduced image S _L : Left reduced image D1: First parallax image D2: Second parallax image M: Feature map

Claims (9)

an image reducing unit that reduces two input images to generate two reduced images;
a first parallax image generating unit that obtains a parallax between the two reduced images by neural network processing and generates a reduced parallax image;
a second parallax image generating unit that obtains a region in one of the input images that matches a partial region of the other input image, thereby obtaining a parallax of the two input images and generating a parallax image;
the second parallax image generating unit sets a search range for determining the matching area based on a parallax value of each pixel of the reduced parallax image for each area of the input image to which each pixel of the reduced parallax image corresponds, and
The image processing device according to claim 1, further comprising: a step of: expanding the search range for pixels having a large feature amount in the feature map output by the first parallax image generating unit . an image reducing unit that reduces two input images to generate two reduced images;
a first parallax image generating unit that obtains a parallax between the two reduced images by neural network processing and generates a reduced parallax image;
a second parallax image generating unit that obtains a region in one of the input images that matches a partial region of the other input image, thereby obtaining a parallax of the two input images and generating a parallax image;
In the image processing device, the second parallax image generating unit sets a search range for determining the matching area based on a parallax value of each pixel of the reduced parallax image for each area of the input image to which each pixel of the reduced parallax image corresponds ,
13. An image processing device comprising: an input image of a road surface divided into upper and lower parts for processing, the input image being reduced at different reduction rates from the upper image and the lower image. 3. The image processing device according to claim 1,
The image processing device according to claim 1, wherein the second parallax image generating unit generates a parallax having the same value as or an approximate value to the reduced parallax image at a corresponding position. 3. The image processing device according to claim 1,
The image processing device, wherein the second parallax image generating unit performs matching processing within the search range. 3. The image processing device according to claim 1,
2. An image processing device comprising: a step of: performing a process of recognizing a height of a road surface using the reduced parallax image; and recognizing an object on the road surface based on a result of the process performed by the second parallax image generating unit. 3. The image processing device according to claim 1,
an image processing device having a disparity accuracy reduction area detection unit that determines from the reduced image that there is an area that cannot be calculated as part of the disparity calculated by the first disparity image generation unit, and indicates that the disparity of the image area detected by the disparity accuracy reduction area detection unit is invalid. 3. The image processing device according to claim 1,
13. An image processing device comprising: a first parallax image generating unit configured to generate a second parallax image for generating a second parallax image on a first image side of a road surface; a first step of reducing two input images to generate two reduced images;
a second step of calculating a parallax between the two reduced images by neural network processing and generating a reduced parallax image;
a third step of obtaining a region in one of the input images that matches a partial region of the other of the input images, thereby obtaining a disparity between the two input images and generating a disparity image;
the third step is a method for processing an image, comprising the steps of: setting a search range for determining the matching region based on a disparity value of each pixel of the reduced disparity image for each region of the input image to which each pixel of the reduced disparity image corresponds ; and widening the search range for pixels having a large feature amount in a feature map . a first step of reducing two input images to generate two reduced images;
a second step of calculating a parallax between the two reduced images by neural network processing and generating a reduced parallax image;
a third step of obtaining a region in one of the input images that matches a partial region of the other of the input images, thereby obtaining a disparity between the two input images and generating a disparity image;
In the third step, for each region of the input image to which each pixel of the reduced parallax image corresponds, a search range is set based on the parallax value of each pixel of the reduced parallax image when determining the matching region, and when the road surface portion of the input image is divided into upper and lower parts and processed, the reduction ratio of the upper image and the reduction ratio of the lower image are different.

Images