JP7652897B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing device and an image processing method that recognize the outside environment of a vehicle based on images captured outside the vehicle.

In recent years, automobiles equipped with driving assistance functions that assist with acceleration/deceleration and steering operations according to the external environment are becoming more common, and the need for driving assistance functions when driving at night is also increasing. To provide appropriate driving assistance when driving at night, it is necessary to accurately estimate the distance to the preceding vehicle and the speed of the preceding vehicle, regardless of the brightness of the brake lights of the preceding vehicle. For this reason, recent image processing devices are required to have improved functions for accurately detecting light sources with low to high brightness, and technologies such as HDR (High Dynamic Range) that expand the dynamic range when capturing images are being used.

However, when the dynamic range during imaging is widened, a flare (halation) phenomenon occurs more easily when imaging a high-brightness light source (for example, a brake light during braking), making the area around the light source bright and blurry. When flare occurs, there is an issue that three-dimensional objects cannot be detected correctly in the flare area.

Therefore, the image processing device in the abstract of Patent Document 1 "provides an image processing device capable of capturing an image of a high-brightness target without causing halation using a single camera, even at night, etc.", and "includes a camera and control unit that are mounted on a vehicle and capture images of the area around the vehicle, and the control unit extracts the vehicle from the image captured by the camera, and calculates an area for correcting brightness for the extracted vehicle and its vicinity based on the brightness in the image of the area around the vehicle captured by the camera. It then calculates parts of the calculated area for brightness correction that cause halation, such as the headlight part, the part of the headlight reflecting the road surface, and the flare part of the headlight, and corrects the calculated part causing halation to a brightness that does not cause halation by multiplying it by a predetermined reduction rate, and outputs the corrected halation area by superimposing it on the image captured by the camera."

JP 2010-073009 A

However, in the image processing device of Patent Document 1, brightness correction is performed only on the area causing halation, so that the change in brightness is discontinuous inside and outside the boundary of the correction area, making it impossible to accurately calculate disparity information (distance information for each point on the image) near the boundary, and resulting in the problem that the distance and speed of objects near the boundary (such as a preceding vehicle) cannot be accurately estimated.

Therefore, the present invention aims to provide an image processing device that can accurately calculate disparity information around a high-brightness light source even if flare occurs when imaging a three-dimensional object equipped with a high-brightness light source, and can accurately estimate the position, speed, shape, type, etc. of the three-dimensional object.

In order to solve the above problem, the image recognition device of the present invention is an image processing device that includes a camera that captures a first image with a first exposure amount and captures a second image with a second exposure amount that is less than the first exposure amount, a three-dimensional object extraction unit that extracts a first area in which a three-dimensional object exists from the first image and extracts a second area in which the three-dimensional object exists from the second image, a three-dimensional object information detection unit that detects first three-dimensional object information from the first area and detects second three-dimensional object information from the second area, and a three-dimensional object information integration unit that integrates the first three-dimensional object information and the second three-dimensional object information.

According to the image processing device of the present invention, even if flare occurs when capturing an image of a three-dimensional object having a high-brightness light source, the disparity information around the high-brightness light source can be accurately calculated, and the position, speed, shape, type, etc. of the three-dimensional object can be accurately estimated.

FIG. 2 is a hardware configuration diagram of the stereo camera device according to the embodiment. FIG. 1 is a functional block diagram of a stereo camera device according to an embodiment. 4 is a process flowchart of the stereo camera device according to the embodiment. 13 is a flowchart of a three-dimensional object recognition process according to a comparative example. 4 is a flowchart of a three-dimensional object recognition process according to the present invention. 13 shows an example of a procedure for calculating the amount of flare light. FIG. 4 is a functional block diagram showing details of an object recognition unit. FIG. 4 is a functional block diagram showing details of a normal shutter three-dimensional object processing unit. FIG. 4 is a functional block diagram showing details of a low exposure shutter three-dimensional object processing unit.

Below, we will explain a stereo camera device, which is one embodiment of the image processing device of the present invention, using figures.

<Hardware configuration diagram of the stereo camera device 10>
FIG. 1 is a hardware configuration diagram showing an outline of a stereo camera device 10 according to the present embodiment.
The stereo camera device 10 is an in-vehicle device that recognizes the vehicle exterior environment based on captured images of the vehicle exterior, such as white lines on the road, pedestrians, other vehicles, other three-dimensional objects, traffic signals, signs, lit lamps, etc. Then, the stereo camera device 10 determines a control policy, such as acceleration/deceleration support and steering support for the vehicle, according to the recognized vehicle exterior environment.

1, the stereo camera device 10 includes cameras 11 (left camera 11L, right camera 11R), an image input interface 12, an image processing unit 13, an arithmetic processing unit 14, a memory unit 15, a CAN interface 16, and an abnormality monitoring unit 17. The components from the image input interface 12 to the abnormality monitoring unit 17 are configured as a single or multiple computer units interconnected via an internal bus, and various functions such as the image processing unit 13 and the arithmetic processing unit 14 are realized by the arithmetic device (CPU, etc.) of the computer unit executing a predetermined program, but the present invention will be described below while omitting such well-known technologies.

The camera 11 is a stereo camera consisting of a left camera 11L and a right camera 11R, which are installed on the vehicle so as to capture an image of the area in front of the vehicle. The imaging element of the left camera 11L alternately captures a first left image _P1L with a normal exposure (hereinafter referred to as "normal shutter") and a second left image _P2L with an exposure lower than normal (hereinafter referred to as "low exposure shutter"). In parallel with the imaging by the left camera 11L, the imaging element of the right camera 11R alternately captures a first right image _P1R with a normal shutter and a second right image _P2R with a low exposure shutter.

Here, the "exposure amount" in this embodiment is a physical quantity obtained by multiplying the exposure time of the image sensor by a gain that amplifies the output of the image sensor. The exposure amount of the "normal shutter" is, for example, an exposure time of 20 ms multiplied by a gain 16 times the low-exposure shutter gain G described below, and the exposure amount of the "low-exposure shutter" is, for example, an exposure time of 0.1 ms multiplied by a predetermined low-exposure shutter gain G, but each exposure amount is not limited to the examples.

In the above, it has been explained that each camera alternately captures the first image P1 ( _P1L , _P1R ) with a normal shutter and the second image P2 ( _P2L , _P2R ) with a low exposure shutter. However, under certain conditions, capturing the second image P2 with a low exposure shutter may be omitted. For example, the second image P2 with a low exposure shutter may be captured only when the luminance value of the first image P1 with a normal shutter or an illuminance sensor determines that the external light is dark, or when the lamp size W1 (see FIG. 5B) in the first image P1 described below is determined to be equal to or greater than a predetermined threshold value _Wth .

The image input interface 12 is an interface that controls the imaging of the camera 11 and captures the captured images P ( _P1L , _P2L , _P1R , _P2R ). The images P captured through this interface are transmitted to the image processing unit 13 and the like through the internal bus.

The image processing unit 13 compares the left image P _L (P1 _L , P2 _L ) captured by the left camera 11L with the right image P _R (P1 _R , P2 _R ) captured by the right camera 11R, and performs corrections such as correction of device-specific deviations due to the imaging element and noise interpolation on each image, and then stores the corrected image P in the memory unit 15.

Furthermore, the image processing unit 13 calculates corresponding points between the left and right images with the same exposure amount, calculates parallax information I (distance information for each point on the image), and stores it in the storage unit 15. Specifically, the first parallax information I1 is calculated by comparing the first left image _P1L and the first right image _P1R, which are captured at the same timing, and the second parallax information I2 is calculated by comparing the second left image _P2L and the second right image _P2R, which are captured at the same timing.

The calculation processing unit 14 uses the image P and parallax information I stored in the memory unit 15 to recognize various objects necessary for perceiving the environment around the vehicle. The various objects recognized here include people, other vehicles, other obstacles, traffic lights, signs, tail lights and headlights of other vehicles, etc. These recognition results and some of the intermediate calculation results are recorded in the memory unit 15. Furthermore, the calculation processing unit 14 uses the object recognition results to determine a control policy for the vehicle.

The memory unit 15 is a storage device such as a semiconductor memory, and stores the corrected image P and disparity information I which are the output of the image processing unit 13, as well as the object recognition results and vehicle control policies which are the output of the calculation processing unit 14.

The CAN interface 16 is an interface that transmits the object recognition results obtained by the arithmetic processing unit 14 and the vehicle's control policy to an in-vehicle network CAN (Controller Area Network).
In addition, a control system (ECU, etc.) that controls the vehicle's drive system, braking system, steering system, etc. is connected to the in-vehicle network CAN. By using the control system, the stereo camera device 10 can perform driving assistance such as automatic braking and steering avoidance in accordance with the environment outside the vehicle.

The abnormality monitoring unit 17 monitors whether each part within the stereo camera device 10 is operating abnormally, whether any errors have occurred during data transfer, etc., and is a mechanism for preventing abnormal operation.

<Functional block diagram of stereo camera device 10>
Next, a basic processing flow of the stereo camera device 10 will be described with reference to the functional block diagram of FIG.

As described above, the left camera 11L alternately captures the first left image _P1L with a normal shutter and the second left image _P2L with a low exposure shutter, and the right camera 11R alternately captures the first right image _P1R with a normal shutter and the second right image _P2R with a low exposure shutter.

The image correction unit 13a of the image processing unit 13 performs image correction processing on each of the images P ( _P1L , _P2L , _P1R , _P2R ) to absorb the inherent characteristics of the imaging element. Then, the corrected image P is stored in the image buffer 15a in the storage unit 15.

Thereafter, the parallax calculation unit 13b of the image processing unit 13 compares the left and right images captured at the same time (the first left image _P1L and the first right image _P1R , or the second left image _P2L and the second right image _P2R ) and calculates parallax information I for each exposure amount. This calculation makes it clear which points on the left image _PL correspond to which points on the right image _PR , so that the distance to the object on each image can be obtained according to the principle of triangulation. The parallax information I (I1, I2) obtained here for each exposure amount is then stored in the parallax buffer 15b of the memory unit 15.

Furthermore, the object recognition unit 14a of the calculation processing unit 14 performs object recognition processing to extract an area where a three-dimensional object exists, using the image P and the parallax information I stored in the memory unit 15. The three-dimensional objects to be recognized include, for example, people, cars, other three-dimensional objects, signs, traffic lights, tail lights, etc., and when recognizing, the recognition dictionary 15c is referred to as necessary.

After completing object recognition, the vehicle control unit 14b of the calculation processing unit 14 determines a control policy for the vehicle by taking into consideration the object recognition result output by the object recognition unit 14a and the state of the vehicle (speed, steering angle, etc.). For example, if there is a possibility of a collision with a preceding vehicle, a control signal is generated to warn the occupants and prompt them to take action to avoid the collision, or to control the vehicle's braking and steering angle to avoid the preceding vehicle, and the control signal is output to the control system (ECU, etc.) via the CAN interface 16 and the in-vehicle network CAN.

<Processing flow chart of image processing unit and calculation processing unit>
Next, the processing performed by the image processing unit 13 and the calculation processing unit 14 will be described with reference to the flowchart of FIG.

First, in step S1, the image correction unit 13a of the image processing unit 13 processes the right image P _R. Specifically, the image correction unit 13a performs device-specific deviation correction, noise correction, and the like on the right image P _R (P1 _R , P2 _R ) captured by the right camera 11R, and then stores the image in the image buffer 15a of the storage unit 15.

Next, in step S2, the image correction unit 13a of the image processing unit 13 processes the left image P _L. Specifically, the image correction unit 13a performs device-specific deviation correction, noise correction, and the like on the left image P _L (P1 _L , P2 _L ) captured by the left camera 11L, and then stores the image in the image buffer 15a of the storage unit 15. Note that the order of steps S1 and S2 may be reversed.

Then, in step S3, the parallax calculation unit 13b of the image processing unit 13 compares the corrected left image P _L and right image P _R stored in the image buffer 15 a to calculate the parallax, and stores the obtained parallax information I in the parallax buffer 15b of the memory unit 15.

Next, in step S4, the object recognition unit 14a of the calculation processing unit 14 performs object recognition using one of the left image P _L or the right image P _R and the parallax information I. In this step, a three-dimensional object such as a preceding vehicle is also recognized, and the details of the three-dimensional object recognition method of this embodiment will be described later.

Finally, in step S5, the vehicle control unit 14b of the calculation processing unit 14 determines a vehicle control policy based on the object recognition result and outputs the result to the in-vehicle network CAN.

Comparative Example of Three-Dimensional Object Recognition Method
Here, the details of the three-dimensional object recognition method performed by the image processing device of the comparative example at the timing of step S4 in Fig. 3 will be described using the flowchart of Fig. 4. Note that in the image processing device of the comparative example, the second image P2 (second left image _P2L , second right image _P2R ) of the low exposure shutter is not captured, so in step S3 of Fig. 3, only the first parallax information I1 based on the first image P1 of the normal shutter is calculated.

First, in step S41, a three-dimensional object in the first image P1 is extracted based on the parallax distribution in the first parallax information I1 calculated in step S3 in Fig. 3, and the shape of the three-dimensional object and the distance to the three-dimensional object are estimated. In addition, a history of the distance to the three-dimensional object is recorded, and the speed of the three-dimensional object is estimated from the history information.

Next, in step S42, the recognition dictionary 15c is referenced to determine the type of three-dimensional object detected in step S41.

Finally, in step S43, the three-dimensional object information (shape, distance, speed) estimated in step S41 and the three-dimensional object type determined in step S42 are output to the downstream vehicle control unit 14b.

The image processing device of the comparative example determines three-dimensional object information (shape, distance, speed) and three-dimensional object type using the above method, but if flare occurs when capturing an image of a high-brightness light source, the disparity information I calculated in step S3 becomes inaccurate around the high-brightness light source, and the three-dimensional object information around the high-brightness light source detected in step S41 also becomes inaccurate. As a result, in the image processing device of the comparative example, the three-dimensional object type around the high-brightness light source determined in step S42 is also unreliable, and driving assistance control based on unreliable information may not be appropriate.

<Three-dimensional object recognition method according to the present embodiment>
Next, the details of the three-dimensional object recognition method performed by the stereo camera device 10 of this embodiment at the timing of step S4 in Fig. 3 will be described using the flowchart of Fig. 5A. Note that in this embodiment, in addition to the first image P1 ( _P1L , _P1R ) captured with a normal shutter, the second image P2 ( _P2L , _P2R ) captured with a low exposure shutter is also captured, so that in step S3 in Fig. 3, both the first parallax information I1 based on the first image P1 and the second parallax information I2 based on the second image P2 are calculated.

First, in step S41, the object recognition unit 14a detects information about a three-dimensional object based on each of the parallax distributions in the two types of parallax information I calculated in step S3 of FIG. 3. Specifically, the area of the three-dimensional object in the first image P1 is extracted based on the parallax distribution in the first parallax information I1, and the shape of the three-dimensional object and the distance to the three-dimensional object are estimated. In addition, a history of the distance to the three-dimensional object in the first image P1 is recorded, and the speed of the three-dimensional object is estimated from the history information. Similarly, the area of the three-dimensional object in the second image P2 is extracted based on the parallax distribution in the second parallax information I2, and the shape of the three-dimensional object and the distance to the three-dimensional object are estimated. In addition, a history of the distance to the three-dimensional object in the second image P2 is recorded, and the speed of the three-dimensional object is estimated from the history information.

In addition, in step S41, the object recognition unit 14a also detects light points in the image P. Here, Fig. 5B(a) is an example of a light point group detected from the first image P1 with a normal shutter, and Fig. 5B(b) is an example of a light point group detected from the second image P2 with a low-exposure shutter.

Next, in step S4a, if the object recognition unit 14a extracts multiple light spot groups in step S41, it determines whether the pair is a lamp pair from the same vehicle based on the distance and position of each light spot group. If it determines that the pair is a lamp pair from the same vehicle, it outputs coordinate information for the lamp pair and proceeds to step S4b. On the other hand, if it determines that the pair is not a lamp pair from the same vehicle, it proceeds to step S42.

In step S4b, the object recognition unit 14a uses the position information of the lamp pairs between the multiple shutters to link three-dimensional object information that is estimated to be the same object. Here, the first three-dimensional object information of the lamp pair L1 illustrated in FIG. 5B(a) and the second three-dimensional object information of the lamp pair L2 illustrated in FIG. 5B(b) are linked.

In step S4c, the object recognition unit 14a determines whether the amount of flare light of the lamp pair is equal to or greater than a threshold. If the amount of flare light of the lamp pair is equal to or greater than the threshold, the process proceeds to step S4d. If the amount of flare light is less than the threshold, the process proceeds to step S42.

Here, the flare light amount is determined in step S4c, for example, as follows. First, a brightness threshold for extracting lamps for each exposure amount is determined in advance, and the area of pixels exceeding the brightness threshold is extracted as the lamp (see Figures 5B (a) and (b)). Next, the lamp sizes W1 and W2 are obtained from the width of each lamp. Then, the difference between the lamp size W1 of the normal shutter (first image P1) and the lamp size W2 of the low exposure shutter (second image P2) is calculated as the flare light amount, and it is determined whether this flare light amount is equal to or greater than the threshold. The method of calculating the difference can be, for example, a method in which the area obtained by subtracting the circular area of the lamp size W2 from the circular area of the lamp size W1 is defined as the flare light amount, as shown in Figure 5B (c).

Although the above description has been given on the assumption that the process of Fig. 5A is performed by selecting either the left image P _L or the right image P _R , the flare light amount may be calculated for each of the _left and right images when the process of Fig. 5A is performed for both the left image _P L and the right image P R. In this way, even if one of the left and right cameras is out of order, the desired control can be continued using the image captured by the normal camera.

In step S4d, the object recognition unit 14a calculates integrated three-dimensional object information by integrating the first three-dimensional object information of the normal shutter (first image P1) and the second three-dimensional object information of the low-exposure shutter (second image P2) linked in step S4b. Specifically, a weighted average of the first three-dimensional object information and the second three-dimensional object information is taken using an integration coefficient set individually for each piece of information (position, speed, shape, etc.) of the three-dimensional object. For example, when integrating the distance to a three-dimensional object, which is one type of three-dimensional object information, if the integration coefficient set for the distance is 0.3, the distance D1 to the three-dimensional object indicated by the first three-dimensional object information and the distance D2 to the three-dimensional object indicated by the second three-dimensional object information are integrated by the following formula 1.

Distance D after integration = distance D1 x (1 - 0.3) + distance D2 x 0.3 (Formula 1)
In addition, since the first image P1 captured with a normal shutter and the second image P2 captured with a low exposure shutter are captured alternately, there is a slight time difference between the capture of the two images. Therefore, it is more preferable to correct each piece of information in consideration of the time difference between the capture timing of the first image P1 and the second image P2 and the relative relationship between the target object (the preceding vehicle) and the own vehicle, and then integrate the information.

The integration coefficient does not need to be a fixed value, and may be increased or decreased in proportion to the distance to the three-dimensional object, for example. The integration coefficient may be set to a value according to the amount of flare light by investigating the correlation between the amount of flare light and the accuracy of each piece of information on the object in advance. Furthermore, when calculating the amount of flare light using both the left and right cameras, the variation in the amount of flare light may also be reflected in the integration coefficient. When the amount of flare light is large, the weight of the information with higher accuracy, either the normal exposure or the low exposure shutter, is set to be large, thereby improving the accuracy after integration. The integration coefficient may be set individually for each piece of information (position, speed, shape, etc.) that is further subdivided. For example, for the shape, the height and width are set individually. When the amount of flare light is large or small, the integration coefficient may be set so that the results of either shutter are not used at all during integration.

In step S42, if the three-dimensional object information is integrated in step S4d, the object recognition unit 14a determines the type of the three-dimensional object using the integrated three-dimensional object information, and if not, determines the type of the three-dimensional object using the first three-dimensional object information obtained from the normal shutter (first image P1).

Finally, in step S43, the object recognition unit 14a outputs the first three-dimensional object information estimated in step S41 or the integrated three-dimensional object information integrated in step S4d, and the three-dimensional object type determined in step S42 to the downstream vehicle control unit 14b.

In the stereo camera device 10 of this embodiment described above, the first three-dimensional object information based on the first image P1, which is easily affected by flare, is integrated with the second three-dimensional object information based on the second image P2, which is less affected by flare, thereby making it possible to increase the reliability of the three-dimensional object information, and thereby to further increase the reliability of the discrimination of the type of three-dimensional object and the reliability of driving assistance.

<Specific configuration of object recognition unit 14a>
Next, a more specific configuration of the object recognition unit 14a in FIG. 2 will be described with reference to the functional block diagrams of FIGS. 6A to 6C.

As shown in FIG. 6A, the memory unit 15 outputs a first image P1 and first parallax information I1 captured with a normal shutter, and a second image P2 and second parallax information I2 captured with a low exposure shutter to the object recognition unit 14a.

The normal shutter three-dimensional object processing unit 21 in the object recognition unit 14a receives the first normal shutter image P1 and the first parallax information I1. FIG. 6B shows an example of the configuration of the normal shutter three-dimensional object processing unit 21, in which the parallax reference unit 21a, distance estimation unit 21b, speed estimation unit 21c, and shape estimation unit 21d perform three-dimensional object detection in step S41 using the first image P1 and the first parallax information I1, and output the first three-dimensional object information. The lamp detection unit 21e also performs lamp detection in step S41 for the first image P1, and outputs the first lamp information.

Similarly, the low-exposure shutter second image P2 and the second parallax information I2 are input to the low-exposure shutter three-dimensional object processing unit 22 in the object recognition unit 14a. Figure 6C shows an example of the configuration of the low-exposure shutter three-dimensional object processing unit 22, in which the parallax reference unit 22a, distance estimation unit 22b, speed estimation unit 22c, and shape estimation unit 22d shown here perform three-dimensional object detection in step S41 using the second image P2 and the second parallax information I2, and output second three-dimensional object information. In addition, the lamp detection unit 22e performs lamp detection in step S41 for the second image P2, and outputs second lamp information.

The lamp pair detection unit 23 performs lamp pair detection of step S4a for the first image P1 based on the first lamp information from the normal shutter three-dimensional object processing unit 21 (see FIG. 5B (a)). The lamp pair detection unit 24 performs lamp pair detection of step S4a for the second image P2 based on the second lamp information from the low-exposure shutter three-dimensional object processing unit 22 (see FIG. 5B (b)).

The lamp detection result linking unit 25 performs the linking process of step S4b for the three-dimensional object information of each lamp pair detected by the lamp pair detection unit 23 and the lamp pair detection unit 24.

The flare determination unit 26 calculates the amount of flare light using the method illustrated in Figure 5B (c).

The three-dimensional object information integration unit 27 performs the processes of steps S4c and S4d, and if the amount of flare light calculated by the flare determination unit 26 is equal to or greater than a predetermined threshold, integrates the first three-dimensional object information from the normal shutter three-dimensional object processing unit 21 and the second three-dimensional object information from the low-exposure shutter three-dimensional object processing unit 22 under a predetermined rule. On the other hand, if the amount of flare light calculated by the flare determination unit 26 is less than the predetermined threshold, the first three-dimensional object information from the normal shutter three-dimensional object processing unit 21 is output as is.

The type determination unit 28 performs the processing of step S42 and determines the type of three-dimensional object based on the object information output by the three-dimensional object information integration unit 27 and the recognition dictionary 15c.

The detection result output unit 29 carries out step S43 and outputs the three-dimensional object information (position, speed, shape, etc.) which is the output of the three-dimensional object information integration unit 27 and the three-dimensional object type which is the output of the type determination unit 28 to the vehicle control unit 14b.

According to the stereo camera device 10 of the present embodiment described above, even if flare occurs when capturing an image of a three-dimensional object with a high-brightness light source, the disparity information around the high-brightness light source can be accurately calculated, and the position, speed, shape, type, etc. of the three-dimensional object can be accurately estimated.

Although the above describes the case where the image processing device of the present invention is a stereo camera device, the image processing device of the present invention may also be a monocular camera device that captures images alternately with a normal shutter and a low exposure shutter. In this case, for example, by adopting a method of assuming the actual size of the imaged object and determining the distance from the size on the image, it is possible to obtain three-dimensional object information similar to that of a stereo camera device from only the output of the monocular camera, and therefore the above-mentioned three-dimensional object information integration method of the present invention can be used.

10...Stereo camera device, 11...Camera, 11L...Left camera, 11R...Right camera, 12...Image input interface, 13...Image processing unit, 13a...Image correction unit, 13b...Parallax calculation unit, 14...Calculation processing unit, 14a...Object recognition unit, 21...Normal shutter three-dimensional object processing unit, 22...Low exposure shutter three-dimensional object processing unit, 23, 24...Lamp pair detection unit, 25...Lamp detection result linking unit, 26...Flare determination unit, 27...Three-dimensional object information integration unit, 28...Type determination unit, 29...Detection result output unit, 14b...Vehicle control unit, 15...Memory unit, 15a...Image buffer, 15b...Parallax buffer, 15c...Recognition dictionary, 16...CAN interface, 17...Abnormality monitoring unit, CAN...In-vehicle network

Claims (8)

a camera that captures a first image with a first exposure amount and captures a second image with a second exposure amount that is less than the first exposure amount;
a three-dimensional object extraction unit that extracts a first area in which a three-dimensional object exists from the first image and extracts a second area in which the three-dimensional object exists from the second image;
a three-dimensional object information detection unit that detects first three-dimensional object information from the first area and detects second three-dimensional object information from the second area;
A three-dimensional object information integration unit that calculates integrated three-dimensional object information by integrating the first three-dimensional object information and the second three-dimensional object information;
a flare determination unit that determines a difference between a lamp size of a light source captured in the first image and a lamp size of a light source captured in the second image as a flare light amount captured in the first image;
An image processing device comprising: 2. The image processing device according to claim 1,
The image processing device, characterized in that the first three-dimensional object information and the second three-dimensional object information include at least one type of information of a distance to the three-dimensional object, a shape of the three-dimensional object, or a speed of the three-dimensional object. 2. The image processing device according to claim 1,
The image processing device, characterized in that the camera captures the first image and the second image alternately. 2. The image processing device according to claim 1 ,
The image processing device according to the present invention, wherein the three-dimensional object information integration unit calculates the integrated three-dimensional object information by the following formula:
Integrated three-dimensional object information=First three-dimensional object information×(1−integration coefficient)+Second three-dimensional object information×integration coefficient 5. The image processing device according to claim 4 ,
4. An image processing apparatus comprising: an image processing unit configured to process an image of a subject; 6. The image processing device according to claim 5 ,
The three-dimensional object information integration unit
When the amount of flare light is equal to or greater than a predetermined threshold, the integrated three-dimensional object information is output.
When the amount of flare light is less than the threshold value, the image processing device outputs the first three-dimensional object information. 2. The image processing device according to claim 1,
The camera includes:
a left camera that captures a first left image of the first exposure amount and a second left image of the second exposure amount;
a right camera configured to capture a first right image having the first exposure amount and a second right image having the second exposure amount;
having
The three-dimensional object extraction unit is
a parallax calculation unit that compares the first left image and the first right image to calculate first parallax information and compares the second left image and the second right image to calculate second parallax information;
A first three-dimensional object processing unit that extracts the first region in which the three-dimensional object exists from the first image based on the first parallax information;
A second three-dimensional object processing unit that extracts the second region in which the three-dimensional object exists from the second image based on the second parallax information;
13. An image processing device comprising: capturing a first image at a first exposure;
capturing a second image with a second exposure amount less than the first exposure amount;
Extracting a first region in which a three-dimensional object exists from the first image;
extracting a second region in which the three-dimensional object exists from the second image;
Detecting first three-dimensional object information from the first area;
Detecting second three-dimensional object information from the second area;
A step of calculating integrated three-dimensional object information by integrating the first three-dimensional object information and the second three-dimensional object information;
determining a difference between a lamp size of a light source captured in the first image and a lamp size of a light source captured in the second image as a flare light amount captured in the first image;
An image processing method comprising:

Images