JP7818478B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing device and image processing method that calculates the three-dimensional position of an object based on a pair of images captured synchronously by a stereo camera.

In recent years, cars equipped with advanced driver assistance systems and autonomous driving systems are becoming increasingly common. These systems estimate the three-dimensional positions of captured objects around the vehicle, such as other vehicles, pedestrians, obstacles, road centerlines, lane markings, etc., and control the vehicle's driving, braking, steering, etc. based on these estimation results.

A stereo camera is known as one type of external recognition device used in these systems. With a stereo camera, multiple cameras placed in different locations capture images of the same object from different viewpoints in a synchronized manner, and the three-dimensional position of the captured object is estimated by calculating the distance to the object based on the difference in how the same object appears in the multiple captured images, known as parallax.

Disparity calculations use techniques such as block matching. This method uses one of a pair of images captured synchronously by a pair of cameras as a reference image, extracts a specified small area from the reference image, and searches for the same small area in the other reference image. Cost functions such as SAD (Sum of Absolute Difference) are used to determine whether the small areas are identical. However, these disparity calculation methods require that the optical characteristics of each camera are identical.

While searches during block matching are typically performed in pixel units, Patent Document 1 describes a technology for improving ranging resolution through post-processing. The abstract of this document states that the problem is to "eliminate the reduction in ranging resolution at long distances caused by pixel-based parallax obtained by processing stereo images, thereby improving ranging resolution from short to long distances." It also states that the solution is to "perform stereo matching on a pair of images captured by the stereo camera 10, in which the stereo processing unit 30 calculates the city-block distance for each small region of each image and determines the correlation between them to identify corresponding small regions. A distance image is generated in which pixel deviations (parallax) that occur depending on the distance to the object are visualized as distance data. Furthermore, the recognition processing unit 40 performs stereo matching using data from the reference image and comparison image to determine parallax of less than one pixel (sub-pixel components), and interpolates the pixel-based parallax obtained from the distance image with a resolution of less than one pixel. This eliminates the reduction in ranging resolution at long distances and ensures ranging accuracy from short to long distances."

Japanese Patent Application Laid-Open No. 2000-283753

Ensuring distance measurement accuracy using the technology in Patent Document 1 requires the assumption that the optical characteristics of the left and right cameras of the stereo camera are identical. However, in reality, the optical characteristics of each camera are not necessarily identical, so if parallax is calculated under the assumption that the optical characteristics of each camera are identical, the parallax calculation result will contain errors.

One of the reasons why the optical characteristics of each camera differ is the difference in the diffusion characteristic F of each camera's lens. As shown in Figure 1, in camera 2, which has lens 21 (simply shown as a single lens, but may be a combination of multiple lenses) and image sensor 22 (e.g., a CMOS image sensor), light emitted from an ideal point light source S is projected through lens 21 onto image sensor 22, generating image P.

In this case, since a point light source S, which has no size, is being imaged, ideally it would be expected that the light source would be imaged only in one pixel where the point light source S is imaged on the image sensor 22, as in the "ideal model" image Pi. However, in reality, the light from the point light source S is diffused around the imaging position due to the diffusion characteristic F of the lens 21, so the light resulting from the point light source S is imaged over a wide area, with brightness decreasing as it moves toward the periphery, as in the "actual model" image Pr.

Therefore, if the diffusion characteristics F of the left and right cameras of a stereo camera are different, even if a point light source S of the same brightness is captured from the same position, each camera will capture a different image. As a result, when calculating parallax using stereo cameras with different diffusion characteristics F, block matching is performed based on the different diffusion components for each image, so the parallax of the light source is not calculated accurately and the three-dimensional position of the light source cannot be accurately estimated. In particular, when the light source is far away, due to the characteristics of stereo cameras, even a small parallax error can lead to a large ranging error. Therefore, in processes that measure the distance to a distant object and use this information for subsequent control, such as a vehicle-following function using an in-vehicle stereo camera, parallax errors caused by differences in the diffusion characteristics F of each camera can be a major problem.

One possible solution to this problem would be to pair lenses so that the left and right cameras of a stereo camera have the same diffusion characteristic F, but because there is variation in the diffusion characteristic F even between lenses of the same design value and the same lot, pairing lenses with matching diffusion characteristics requires a great deal of effort, as it is necessary to measure the diffusion characteristic F for each lens and then find and pair lenses with similar characteristics.

The present invention was made in consideration of these problems, and aims to provide an image processing device and image processing method that can accurately calculate parallax even when the diffusion characteristics of each camera that makes up a stereo camera differ.

To achieve the above objective, the present invention comprises a light source area identification unit that identifies the area where the light source is captured in each of multiple images simultaneously captured by multiple cameras with overlapping fields of view, and a disparity calculation unit that weights pixels in the light source area according to their luminance values and uses each weighted pixel to perform disparity matching of the light source area to determine the disparity.

The image processing device and image processing method of the present invention can accurately calculate parallax even if the diffusion characteristics of each camera constituting a stereo camera differ. Issues, configurations, and advantages other than those described above will be made clear in the following examples.

Illustrative diagram of lens diffusion characteristics 1 is a schematic diagram of an image processing apparatus according to a first embodiment of the present invention; Functional block diagram of an image processing unit according to the first embodiment An explanatory diagram of dynamic calculation of a light source determination threshold An explanatory diagram of dynamic calculation of a light source determination threshold Functional block diagram of a parallax calculation unit according to the first embodiment 1 is an example of weights set by the weight setting unit of the first embodiment. An example of luminance values in light source and non-light source areas 1 is a functional block diagram of a disparity matching unit according to a first embodiment; 10 is another example of a functional block diagram of the parallax matching unit according to the first embodiment; An explanatory diagram of the luminance subtraction process that reflects weights Functional block diagram of a light source detection unit according to a second embodiment Example of object identification result used in Example 2 Functional block diagram of a disparity calculation unit according to a second embodiment Example of integration weight calculation process Functional block diagram of an image processing unit according to a third embodiment 10 is a functional block diagram of a reliability calculation unit according to a third embodiment. Another example of a functional block diagram of the reliability calculation unit according to the third embodiment Functional block diagram of a parallax calculation unit according to a fourth embodiment

An embodiment of the image processing device of the present invention will be described below with reference to the drawings. Note that, although the present invention will be described below using an image processing device mounted on a vehicle as an example, the image processing device of the present invention is not limited to being mounted on a vehicle.

The image processing device 1 according to the first embodiment of the present invention will be described using Figures 2 to 10.

First, the schematic configuration of the image processing device 1 of Example 1 will be described using Figure 2. As shown here, the image processing device 1 of this example includes a stereo camera 11, a memory 12, an arithmetic processing unit 13, an image processing unit 14, and an external output unit 15. Each component is connected to each other via a communication line 16 so that they can communicate with each other.

The stereo camera 11 is a camera system that uses a left camera 2L and a right camera 2R to capture left and right images synchronously. Hereinafter, any one of the synchronously captured left and right images will be referred to as the base image P0, and the other as the reference image P1. In this embodiment, the left camera 2L and right camera 2R are positioned so that their line of sight are parallel and their fields of view overlap.

Memory 12 is a storage device such as a semiconductor memory that temporarily stores the base image P0 and reference image P1, and stores programs executed by the calculation processing unit 13 and image processing unit 14.

The calculation processing unit 13 is a calculation device such as a CPU (Central Processing Unit) that implements the calculation processing described below in accordance with the instructions of the program stored in the memory 12.

The image processing unit 14 is a computing device such as a GPU (Graphics Processing Unit) that calculates the parallax p based on the base image P0 and the reference image P1. The specific processing performed by the image processing unit 14 to calculate the parallax p will be described later.

The external output unit 15 is an interface that transmits the parallax p, etc. calculated by the image processing unit 14 to an external device (for example, an ECU (Electronic Control Unit)). The ECU is a control device that controls the vehicle's drive system, braking system, steering system, etc., and uses the parallax p, etc. obtained from the external output unit 15 for object detection, driving path detection, etc., which are functions of advanced driver assistance systems and autonomous driving systems. These functions may be incorporated as part of the image processing device 1.

<Image processing unit 14>
Next, the image processing unit 14 will be described in detail using the functional block diagram of Fig. 3. As shown in the figure, the image processing unit 14 has a light source region specifying unit 141 and a parallax calculation unit 142. Details of each unit will be described below in order.

<<Light source area identification unit 141>>
The light source region identification unit 141 receives a pair of images captured by the stereo camera 11 as a base image P0 and a reference image P1, and identifies the region in the received images where the light source is captured (hereinafter referred to as the light source region R). Here, the light source in this embodiment is a term that encompasses not only self-luminous automobile headlights, brake lights, tail lights, street lights, traffic lights, etc., but also objects that emit stronger light toward the camera 2 than the surrounding environment, such as mirrors that reflect strong light.

One method for identifying the light source region R is to refer to the luminance values of pixels in the image and extract the region where the luminance value is equal to or greater than a threshold as the light source region R. In this case, the brighter the light source, the stronger the diffuse component, and the darker the periphery, the stronger the influence of the diffuse component on parallax matching. Therefore, as shown in Figure 4A, it is desirable to design the light source region identification unit 141 so that the light source determination threshold increases as the periphery determined from the entire image becomes brighter.

Figure 4B also shows a simple diagram of the relationship between ambient brightness and the diffuse light component that affects parallax matching. In this diagram, the further to the right on the horizontal axis the brighter the periphery, and the further up the vertical axis the brighter the light source. As is clear from this diagram, even if the light source has the same brightness and generates the same amount of diffuse light, if the ambient light is bright, the diffuse light component will be blended into the ambient light and become smaller, resulting in a relatively small impact on parallax matching. On the other hand, if the ambient light is dark, the impact of the diffuse light component will be relatively large, even if the light source is relatively dark. In other words, by determining a threshold for determining a light source based on ambient brightness, as in Figure 4A, it is possible to efficiently extract only the imaged area of the light source (light source area R) that affects parallax matching.

<<Parallax calculation unit 142>>
The parallax calculation unit 142 weights each pixel in the light source region R identified by the light source region identification unit 141, and calculates the final parallax p by performing parallax matching processing using each weighted pixel.

As shown in the functional block diagram in Figure 5, the disparity calculation unit 142 has a weight setting unit 142a and a disparity matching unit 142b. Details of each unit will be explained below.

The weight setting unit 142a sets a weight for each pixel of the image based on the base image P0 and reference image P1 captured by the stereo camera 11, and the light source region R identified by the light source region identification unit 141.

The weights set by the weight setting unit 142a will be explained using Figure 6. As shown in this example, in the light source region R0 of the reference image P0, the portion A1 where the light source itself is photographed has a particularly high brightness value, while the diffuse light component A2 that causes parallax error is distributed around it at low brightness. For this reason, for this reference image P0, a method of simply setting weights according to brightness values is conceivable, as shown in the graph at the bottom left of Figure 6. Alternatively, weights may be set taking into account not only brightness values but also brightness gradients. In the graph at the bottom left of Figure 6, the weights change discretely, but they may also be modeled so that they change smoothly in accordance with smoothly changing brightness values.

Here, we will explain the weights that should be set for each region in an image P captured at night, using as an example a light source region R that includes a light source and a non-light source region R' that includes objects that are not light sources (for example, a forest of columnar objects). As shown in Figure 7, the forest of columnar objects in the non-light source region R' are low-brightness pixels simply due to the low amount of light. However, since these are pixels that we want to actively use in parallax matching, we want to set a large weight. On the other hand, the diffuse light component of the light source region R is a pixel with a low brightness similar to that of the columnar object imaging region in the non-light source region R', but since it is thought to become noise during parallax matching, we want to set a small weight. In other words, even for pixels with similar low brightness, the optimal weights are different between the light source region R and the non-light source region R'.

Comparing the characteristics of the light source region R and the non-light source region R', which can be useful when distinguishing which category a low-brightness pixel falls into, reveals the following differences between the two. In other words, in the non-light source region R', the brightness at the outer edge of the columnar object changes suddenly depending on whether or not the object is present, so even if the overall brightness value is low, it tends to contain a large brightness gradient. On the other hand, the diffuse light component in the light source region R has a low brightness value and a gentle brightness gradient.

Therefore, taking into account the differences in the characteristics of the two, by designing weights that are small only when the luminance value is low and the luminance gradient is small (i.e., only when it is considered to correspond to the diffuse light component within the light source region R), it is possible to avoid the risk of assigning inappropriately small weights to low-luminance areas other than the diffuse light component (for example, objects within the non-light source region R').

To explain more specifically the weight w set for each pixel in this embodiment, if the luminance value at a certain image coordinate (i, j) is L(i, j) and the luminance gradient is L'(i, j), then the weight w defined by the following equation 1 can be assigned to the image coordinate (i, j). Here, a and b in equation 1 are coefficients that act as scale factors that match the scale of the luminance value and luminance gradient.

The disparity matching unit 142b performs disparity matching while weighting each pixel based on the weight set by the weight setting unit 142a, and outputs the disparity p. The method for reflecting the weighting is explained below.

Figure 8 shows a functional block diagram of the parallax matching unit 142b, which weights each pixel on the brightness difference calculated prior to calculating the parallax. In this example, after calculating the brightness difference between the base image P0 and the reference image P1, a weight is applied to each brightness difference. The weighted brightness differences are then used to calculate the parallax through parallax matching. However, this method does not take into account the weights that should be set for each pixel on the reference image P1 side, and therefore cannot fully eliminate the influence of pixels with low weights (diffuse light components).

As an example of a method for improving the problem shown in Figure 8, Figure 9 shows a functional block diagram of the parallax matching unit 142b, which performs image transformation on both the base image P0 and the reference image P1 in advance to weight each pixel in both images. In this diagram, the base image P0 and the reference image P1, which are luminance images, are transformed in advance so that the luminance of pixels with low weights has less of an effect on the difference calculation that follows. For example, in areas with low weights, if the luminance values of the corresponding pixels are transformed to all 0 in the left and right images (base image P0 and reference image P1), the difference calculated during parallax matching within that area will always be 0, and differences in other areas will be given priority.

An example of the actual conversion procedure is shown in Figure 10. The top graph in the figure is a graph showing the horizontal brightness change in the light source region R0 of the reference image P0. In a method where the brightness of this top graph is simply multiplied by a weighting coefficient such as that in Equation 1, an image is generated by multiplying each brightness by the weight set for the region, and then parallax matching is performed. This method is simple, but as shown in the middle graph in Figure 10, the image changes suddenly between areas with strong and weak weights, and new edges are generated in areas that did not originally exist. This can lead to the mistaken belief that an object that did not originally exist has been detected, which can lead to incorrect matching.

Therefore, as shown in the lower graph in Figure 10, a luminance image is generated by subtracting a predetermined amount from the luminance of the entire image, and this is then used for parallax matching. This allows for parallax matching that minimizes the impact of weight-change areas. With this method, matching is performed after a certain luminance value is uniformly subtracted from the entire area identified as the light source. Because luminance is subtracted from bright areas as well, a constant luminance gradient is maintained in high-luminance areas, as shown in the lower graph. The specific luminance value to be subtracted can be set so that the luminance values of all pixels with weights below a certain value are set to 0, or so that the luminance of the light source area is 20% of the initial value. These values are also affected by the diffusion characteristic F of the lens 21 used, so they must be designed based on the parallax accuracy required for the stereo camera 11.

When the image processing device 1 of the embodiment described above captures an image of a light source and calculates parallax, it calculates the parallax after removing the effects of diffuse light components around the light source, thereby minimizing parallax errors caused by differences in the characteristics of the left and right lenses and achieving highly accurate distance measurement.

Next, an image processing device 1 according to a second embodiment of the present invention will be described using Figures 11 to 14. Note that overlapping explanations of points common to the first embodiment will be omitted.

Example 2 is a modification of Example 1, and is an example in which the object identification result I of an object that can be a light source is acquired externally. According to this example, the light source area on which the image processing of the present invention should be performed can be identified using information other than image brightness values, and image processing can be performed without overlooking important light sources.

Figure 11 shows a functional block diagram of the light source region identification unit 141 in this embodiment. As shown here, the light source region identification unit 141 in this embodiment receives the base image P0, the reference image P1, and the object identification result I as input, and outputs the light source region R. It has a monitoring region setting unit 141a and a brightness change determination unit 141b.

The object identification result I is a combination of the area in the image where a three-dimensional object is detected and a label indicating the type of three-dimensional object, and is generated by an object identification unit (not shown) using pattern matching or the like. An example of the object identification result I is shown in Figure 12. For example, if a vehicle detection result Ob1 is obtained as the object identification result I, there is a possibility that the brake lights or tail lights are on, so the light source area identification unit 141 treats the area indicated by the vehicle detection result Ob1 as a possible light source object. Specifically, the monitoring area setting unit 141a sets the area where the vehicle's brake lights are likely to be located as the monitoring area. The relevant area is then tracked over time, and the brightness change determination unit 141b determines the moment when the brightness value changes significantly. If the brightness value changes in the brighter direction, it is determined that this is the timing for the brake lights to be turned on, and the relevant area is output as a light source area.

Similarly, when signal detection result Ob2 is obtained as object identification result I, the monitoring area setting unit 141a sets the position of the signal lamp as a possible light source object in the monitoring area, as described above, and then the brightness change determination unit 141b monitors the brightness change of the area, and if it becomes brighter, outputs it as a light source area.

On the other hand, if other detection results Ob3 are obtained for objects that cannot be light sources, such as roadside trees, the monitoring area setting unit 141a does not need to set a specific monitoring area.

Next, Figure 13 shows a functional block diagram of the disparity calculation unit 142 of this embodiment. As shown here, the disparity calculation unit 142 of this embodiment includes a type parameter setting unit 142c and a weight merging unit 142d in addition to the weight setting unit 142a and disparity matching unit 142b described in Figure 5 of Example 1, etc.

The type parameter setting unit 142c receives the object identification result I described above and sets a type parameter separate from the weighting used in Example 1. The type parameter is a weight that determines the importance of the weighting process of the present invention based on the importance of distance measurement for the target object. When strong weighting is applied, the diffuse light component is more effectively removed, improving distance measurement accuracy. On the other hand, the number of pixels not used for parallax matching increases, resulting in a decrease in detection stability. The opposite is true when weak weighting is applied. These effects are a trade-off, so it is important to use the weighting appropriately depending on the scene.

As shown in Figure 12, if the type of object being imaged is known in advance based on the object identification result I obtained from an external source, for example, if the preceding vehicle has its brake lights on, it is likely to be subject to emergency braking by the vehicle itself, and both the urgency and the required accuracy of distance measurement are high. Therefore, the diffused light component should be removed as much as possible using the present invention to perform highly accurate distance measurement. However, if the vehicle has its tail lights on, the urgency is low and the lamps are dimmer than the brake lights, so the amount of error due to the influence of diffused light is smaller, so the amount of diffused light component removed can be smaller than for brake lights. Furthermore, if the vehicle is a traffic light or street light, the required accuracy of distance measurement is low.

By changing the weighting depending on the type in this way, when distance measurement accuracy is required, this process is performed strongly to prioritize accuracy, and when this is not the case, this process is weakened to ensure stability. The type parameter is output to subsequent processing as a parameter indicating these tendencies.

The weights and type parameters calculated by the weight setting unit 142a and type parameter setting unit 142c are integrated by the weight merging unit 142d into weights to be used for disparity matching. For illustrative purposes, Figure 14 shows an example of integrated weight calculation when the exact same light source is determined to be the brake light of a preceding vehicle, and an example of integrated weight calculation when it is determined to be a traffic light.

As shown in the left diagram of Figure 14, when the light source is determined to be a brake light, the weighting of the light source portion and the diffused light portion is changed more steeply to reduce the influence of diffused light as much as possible and improve distance measurement accuracy. This is because, in order to avoid a rear-end collision with a leading vehicle that is suddenly decelerating using control from an advanced driver assistance system or autonomous driving system, it is necessary to accurately estimate the distance to the leading vehicle's brake light.

On the other hand, as shown in the right image of Figure 14, when a light source is determined to be a signal, the distance measurement accuracy is not improved in the light source area, and the stability of signal detection is maintained by increasing the information used for disparity matching. This is because there is no particular adverse effect even if the distance to the signal cannot be accurately estimated.

According to this embodiment, the light source area for which weighting processing should be performed is identified using information other than image brightness values, and by performing weighting according to the type of light source, it is possible to perform parallax matching that appropriately removes diffuse light components only for light sources that require distance measurement accuracy.

Next, an image processing device 1 according to a third embodiment of the present invention will be described using Figures 15 to 17. Note that overlapping explanations of points common to the above embodiments will be omitted.

Example 3 is a variation of Example 1, in which a reliability calculation unit 143 that calculates a reliability map M for each parallax is added to the image processing unit 14 of Example 1. According to this example, a reliability map M can be output for each output parallax p, and when using the parallax p in subsequent processing, it is possible to selectively perform high-precision ranging using only parallax p with high reliability indicated by the reliability map M, or dense point cloud processing using points with low reliability.

Figure 15 shows a configuration diagram of the image processing unit 14 in this embodiment. As shown here, the image processing unit 14 in this embodiment is configured by adding a reliability calculation unit 143 to the rear end of the image processing unit 14 in embodiment 1 (see Figure 3). As a result, after the disparity calculation unit 142 calculates the disparity p using the method described above, the reliability calculation unit 143, which calculates the reliability of each disparity, can calculate the reliability map M.

An example of a functional block diagram of the reliability calculation unit 143 is shown in Figure 16. The reliability calculation unit 143 shown here has a normal disparity matching unit 143a and a disparity comparison unit 143b.

The normal disparity matching unit 143a performs simple disparity calculation without weighting. While methods for calculating disparity include the block matching mentioned above, the same method as the disparity matching unit 142b of the disparity calculation unit 142 is used, with the only difference being the presence or absence of weighting. Hereinafter, the disparity p calculated here will be referred to as the normal disparity pn, and the weighted disparity pw, which is the output of the disparity matching unit 142b, to distinguish them.

Next, the parallax comparison unit 143b compares the normal parallax pn with the weighted parallax pw for each pixel. Pixels with matching weights are determined to be unaffected by the diffuse light component, and therefore the reliability of the pixel can be determined to be high. On the other hand, if the normal parallax pn and weighted parallax pw differ, it is determined to be affected by the diffuse light component, and therefore the reliability of the pixel can be determined to be low. Note that areas where the normal parallax pn and weighted parallax pw differ are areas where parallax error has been removed by removing the diffuse light component according to the present invention.

However, as described in Example 2, the weighted parallax pw output by the parallax matching unit 142b is equivalent to performing parallax matching by removing pixels containing diffuse light components, and the number of pixels used for parallax matching is smaller than usual, resulting in a lower reliability. Additionally, since the larger the difference at this time, the greater the influence of the diffuse light components is considered to be, and therefore the reliability is set lower, taking into account the possibility that the influence has not been completely removed by weighting. The reliability map M set based on the above ideas is used as the final output of the reliability calculation unit 143.

Alternatively, the reliability may be calculated directly from the diffusion characteristics F of the left and right lenses, without comparing the parallax p using the parallax comparison unit 143b. Figure 17 shows a functional block diagram of the reliability calculation unit 143 in this case. The reliability calculation unit 143 shown here has a diffusion estimation unit 143c and a diffusion characteristic-based reliability generation unit 143d. The method of calculating the reliability using the reliability calculation unit 143 in Figure 17 will be described in detail below.

First, during a preliminary camera assembly process, an actual point light source S is captured and the diffusion characteristic F of the lens 21 is measured.

Next, the diffusion estimation unit 143c estimates the diffusion in each image region of the reference image P0. This diffusion is a process that estimates the degree of blur in a small region of the image by looking at the frequency components, brightness gradient, etc. of that region, and ultimately compares this with the diffusion characteristic F of the lens 21 to determine whether the small region contains components similar to the diffusion components caused by imaging a light source.

Then, the diffusion characteristic reliability generation unit 143d determines that in an area having a component similar to the diffusion characteristic F of the lens 21, it is not a texture due to the presence of an object, but rather a diffused light component captured pseudo-by diffusion from a light source that is affecting the parallax calculation.
In such a region, the reliability is determined to be low. Therefore, the reliability generation unit 143d is designed so that the reliability is lowered as the region becomes more similar to the diffusion characteristic F of the lens 21.

It is not necessary to select only one of the reliability calculation methods shown in Figures 16 and 17; they may be used simultaneously.

According to this embodiment, it is possible to extract disparities p whose reliability has been reduced due to diffuse light components, and to output a reliability map M that indicates the reliability of each disparity p. This makes it possible to perform processing using the reliability map M in subsequent processing in the ECU, etc.

Next, an image processing device 1 according to a fourth embodiment of the present invention will be described using Figure 18. Note that overlapping explanations of points common to the above embodiments will be omitted.

Example 4 is a variation of Example 1, in which the magnitude of the influence of a light source is determined based on the brightness of the surrounding area. According to this example, in scenes where the influence of diffuse light components is significant, such as a strong light source at night, the weighting gradient is increased, making it possible to appropriately remove the influence.

A functional block diagram of the disparity calculation unit 142 in this embodiment is shown in Figure 18. As shown here, the disparity calculation unit 142 in this embodiment has a brightness parameter setting unit 142e instead of the type parameter setting unit 142c in Embodiment 2.

The brightness parameter setting unit 142e in this embodiment receives the reference image P0 as input and outputs a setting value for the brightness parameter. Here, the brightness parameter is a weight that determines the degree of importance of the weighting process of the present invention based on the importance of distance measurement for the target object. The brightness of the surroundings is then determined from the luminance value of the entire screen of the reference image P0, and if it is determined that the surroundings are dark, the weight merging unit 142d applies strong weighting and strongly removes diffuse light components. On the other hand, if it is determined that the surroundings are bright, the diffuse light components are considered to be sufficiently small compared to other light, so the weight merging unit 142d weakens the weighting to prevent a loss of information. The effect of the strength of weighting in the weight merging unit 142d is the same as for the light source type in Example 2.

Here we have described an example in which the surrounding brightness is determined from an image, but if time information can be obtained from an external source, the brightness parameters can be determined from information on morning, noon, and night, or if GPS or map information is available, the brightness parameters can be determined from information such as whether the vehicle is inside or outside a tunnel.

According to this embodiment, weighting is performed according to the degree of influence of diffuse light components due to changes in ambient brightness, making it possible to perform parallax matching that appropriately removes diffuse light components at night, when removal of diffuse light components is particularly necessary.

The present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

Furthermore, each of the above components may be configured in part or in whole as hardware, or may be configured to be realized by a program being executed by a processor. Furthermore, the control lines and information lines shown are those considered necessary for the explanation, and do not necessarily represent all control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

1 Image processing device 11 Stereo camera 12 Memory 13 Arithmetic processing unit 14 Image processing unit 141 Light source area identification unit 141a Monitoring area setting unit 141b Brightness change determination unit 142 Parallax calculation unit 142a Weight setting unit 142b Parallax matching unit 142c Type parameter setting unit 142d Weight merging unit 142e Brightness parameter setting unit 143 Reliability calculation unit 143a Normal parallax matching unit 143b Parallax comparison unit 143c Diffusion degree estimation unit 143d Reliability generation unit based on diffusion characteristics 15 External output unit 16 Communication line 2 Camera 21 Lens 22 Imaging element 2L Left camera 2R Right camera S Point light source F Diffusion characteristics Pi Ideal image of point light source Pr Actual image of point light source P0 Reference image P1 Reference image p Parallax R Light source area R' Non-light source area I Object identification result

Claims (15)

a light source area specifying unit that specifies an area where a light source is captured in each of a plurality of images simultaneously acquired by a plurality of cameras having overlapping fields of view;
a parallax calculation unit that weights pixels in a light source area according to the luminance values of the pixels, and calculates parallax by performing parallax matching of the light source area using the weighted pixels;
An image processing device comprising: 2. The image processing device according to claim 1,
The image processing device according to claim 1, wherein the parallax calculation unit weights pixels with high luminance values so that they have a greater influence on the parallax matching than pixels with low luminance values. 2. The image processing device according to claim 1,
The image processing device is characterized in that the parallax calculation unit performs parallax matching processing after uniformly subtracting luminance values from the entire light source area. 2. The image processing device according to claim 1,
The image processing device, wherein the parallax calculation unit weights the luminance values and luminance gradients so that the higher the luminance values and luminance gradients, the greater the influence they have on parallax matching. 2. The image processing device according to claim 1,
The image processing device, wherein the parallax calculation unit changes the degree of change in weight depending on the type of light source. 2. The image processing device according to claim 1,
The image processing device is characterized in that the light source area specifying unit detects an area in the captured image where the luminance value is equal to or greater than a threshold value. 7. The image processing device according to claim 6,
10. An image processing device, comprising: an image processing unit configured to dynamically change the threshold value so that the threshold value increases as the entire image becomes brighter. 2. The image processing device according to claim 1,
The light source area identification unit acquires from an external device an object identification result that identifies an object as having the potential to emit light, monitors the luminance value of the area identified by the object identification result in time series, and determines that a light source attached to a preceding vehicle has emitted light when the luminance value becomes large. 2. The image processing device according to claim 1,
The image processing device further comprises a reliability map calculation unit that calculates a reliability map indicating reliability corresponding to each of the parallaxes obtained by the parallax calculation unit. 10. The image processing device according to claim 9,
The image processing device is characterized in that the reliability map calculation unit calculates disparity without using the weight, compares it with the disparity calculation result using the weight, and calculates a reliability map in which the greater the difference, the lower the reliability. 10. The image processing device according to claim 9,
The reliability map calculation unit calculates a reliability map in which the reliability decreases as the difference in diffusion characteristics between multiple cameras measured in advance increases or the brightness of the target light source increases. 2. The image processing device according to claim 1,
The image processing device, wherein the parallax calculation unit includes a brightness parameter setting unit that sets the degree of change in weight to be greater as the surrounding brightness information becomes darker. 13. The image processing device according to claim 12,
The image processing device is characterized in that the brightness parameter setting unit determines the brightness parameter using time information or map information obtained from an external source. 13. The image processing device according to claim 12,
The image processing device is characterized in that the brightness parameter setting unit determines the brightness parameter according to a luminance value of a reference image. a light source area specifying step of specifying an area where a light source is captured in each of a plurality of images simultaneously acquired by a plurality of cameras having overlapping fields of view;
a parallax calculation step of weighting pixels in a light source area according to luminance values of the pixels, and using the weighted pixels to perform parallax matching of the light source area to obtain parallax;
An image processing method comprising: