JP7675355B2 - external world recognition device - Google Patents

Description

The present invention relates to an external environment recognition device such as an in-vehicle stereo camera, and more particularly to an external environment recognition device that accurately performs a stop determination for driving assistance control using an image captured by an in-vehicle camera.

There are known driving assistance controls such as adaptive cruise control (ACC) and automatic emergency braking (AEB). These controls recognize the surrounding environment (external world) from images captured by a camera installed in the windshield with the subject in front of the vehicle, and control the vehicle.

In the above driving assistance control, if the camera's visibility is obstructed by heavy rain or fogging of the windshield, etc., and the image deteriorates, non-detection or erroneous detection occurs, resulting in operations such as no braking in AEB, erroneous acceleration during ACC, erroneous braking in AEB, and erroneous braking during ACC. Therefore, a HALT function is required that temporarily stops the driving assistance control before it causes such operations and notifies the user that it has been temporarily stopped. However, if the driving assistance control is temporarily stopped, it is not possible to reduce the burden of driving operations on the driver, so it is desirable to accurately determine the situation in which the driving assistance control should be temporarily stopped.

Patent document 1 describes that in order to accurately determine a situation in which driving assistance control should be temporarily stopped, the difference in edge or brightness is calculated from the first and second captured image data while the wipers are operating, and if the difference is greater than or equal to a specified value, the stop determination process is not performed.

JP 2015-088047 A

In Patent Document 1, the stop determination process uses the number of edges, the number of distance data, and the brightness in the current frame, but the stop determination (image deterioration determination) is performed using only the information in the current frame, and it is not possible to determine whether the environment is originally one in which brightness and edges are difficult to produce, or whether brightness and edges are not produced due to image deterioration caused by heavy rain, etc. In that case, when raindrops or snow adhere to the vehicle and slight deterioration of the image occurs in an environment in which brightness and edges are originally difficult to produce, such as a snowy road, the system will erroneously determine that the vehicle is HALT and stop the function (driving assistance control).

Therefore, the present invention aims to provide an external environment recognition device that judges the device to be HALT and halts its functions when the image degradation is severe, such as in heavy rain, and does not judge the device to be HALT and halt its functions when the image degradation is minor, such as in light rain.

In order to solve the above problems, one representative external environment recognition device of the present invention is characterized by having a camera mounted in a vehicle cabin, an image feature calculation unit that calculates image features from an image acquired by the camera, a recognition unit that recognizes objects outside the vehicle from an image acquired by the camera, a memory that stores a first image feature in a first image acquired at a timing determined in a predetermined cycle, and a function stop determination unit that determines whether to temporarily suspend the function of the recognition unit based on the first image feature or a comparison result between the first image feature and a second image feature in a current image.

According to the present invention, by making a judgment using time series data, it is possible to judge whether the environment is originally one in which image features such as parallax are difficult to appear, or whether image features such as parallax are not appearing due to image deterioration caused by heavy rain, etc., and therefore, when the image deterioration is severe, such as heavy rain, it is judged as HALT and the function is stopped, and when the image deterioration is light rain, the HALT judgment is not made and the function is not stopped. Therefore, when the image deterioration is severe, such as heavy rain, while when the image deterioration is light rain and the image deterioration is not severe enough to HALT, the function (driving assistance control) is not stopped and driving assistance is continued, so that the burden of the driver's driving operation can be continued to be reduced.

Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments.

FIG. 1 is a configuration diagram of an in-vehicle stereo camera according to an embodiment of the present invention. 1A is a diagram illustrating a left camera image, a right camera image, and a parallax image in a normal state; 1A is an explanatory diagram of a left camera image, a right camera image, and a parallax image when raindrops are attached; FIG. FIG. 4 is an explanatory diagram of a wiper position before the wiper is operated. FIG. 11 is an explanatory diagram of the wiper position immediately after wiping with the wiper. FIG. 11 is an explanatory diagram of a camera image taken before the wipers are operated. FIG. 11 is an explanatory diagram of a camera image taken immediately after wiping with a wiper. 5 is a flowchart of a function stop determination unit of the in-vehicle stereo camera. An image of the behavior of this system's stoppage judgment during heavy rain, and an explanatory diagram of the time series change in the disparity number. An image of the behavior of this system's shutdown judgment during heavy rain, and an explanatory diagram of the results of deterioration judgment in each frame. This is an image of the behavior of this system's shutdown judgment during heavy rain, and an explanatory diagram of the HALT occurrence counter. This is an image of the behavior of the system's stop judgment during light rain, and an explanatory diagram of the time series change in the disparity number. This is an image of the behavior of the system's shutdown judgment during light rain, and an explanatory diagram of the results of deterioration judgment in each frame. This is an image of the behavior of the system's stop judgment during light rain, and an explanatory diagram of the HALT occurrence counter.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will now be described with reference to the drawings.

FIG. 1 is a representative diagram of this embodiment, and is a configuration diagram of an in-vehicle stereo camera in this embodiment. The in-vehicle stereo camera 1 is mounted on a vehicle and has a function as an external recognition device that recognizes objects outside the vehicle from images acquired (captured) by (multiple) cameras attached in the vehicle cabin, and realizes driving assistance functions such as ACC and AEB. The in-vehicle stereo camera 1 is composed of cameras 2A and 2B (left camera 2A and right camera 2B) as a pair of left and right imaging units, an image feature amount calculation unit 3, a recognition unit 4, a memory 5 that stores a first image feature amount, and a function stop determination unit 6. The in-vehicle stereo camera 1 includes hardware such as a camera, a CPU, a memory, and an electronic circuit, and software that cooperates with the hardware to provide driving assistance for the vehicle.

The cameras 2A and 2B in Fig. 1 capture images. The cameras 2A and 2B are fixed to the vehicle so as to be spaced apart in the vehicle width direction, and capture images of the front through the windshield of the vehicle, so that they can capture the same area.

The recognition unit 4 in FIG. 1 detects and recognizes objects such as vehicles and pedestrians in the surrounding environment (here, ahead) from images acquired by the cameras 2A and 2B, and realizes driving assistance functions such as ACC and AEB.

The image feature amount calculation unit 3 receives an image captured by the left camera 2A (hereinafter also referred to as a left camera image) and an image captured by the right camera 2B (hereinafter also referred to as a right camera image) as input, and calculates image feature amounts by analyzing pixel values (luminance values) in the images. The image feature amount may be calculated using the pixel values themselves, or a local region may be defined from a plurality of pixels, and the image feature amount may be calculated using an average luminance, weighted average luminance, representative luminance, or the like within the local region. In this example, the image feature amount calculation unit 3 calculates the number of parallaxes and the number of edges as image feature amounts.

The number of edges in an image can be calculated from (the image from) the left camera 2A or the right camera 2B in the current frame using a known technique.

Parallax is calculated by detecting corresponding points in the images of the left and right cameras 2A and 2B by pattern matching, and calculating the coordinate shift between the detected corresponding points. By using this parallax, the distance to the corresponding point in real space can be calculated by the principle of triangulation. FIG. 2 shows an image of parallax calculation in the image feature amount calculation unit 3. In FIG. 2, a parallax image (c) is calculated by pattern matching (a) the left camera image and (b) the right camera image. Parallax is calculated in parts where edges appear, such as people, white lines, and preceding vehicles, but parallax cannot be calculated in parts where edges are difficult to appear, such as the sky, and pattern matching is difficult. The number of valid parallaxes calculated in the parts where edges appear in the entire parallax image is called the parallax number.

Fig. 3 shows an example of the case where raindrops are attached to the example of Fig. 2. When raindrops are attached to the left camera (specifically, the windshield in the field of view of the left camera) as in the left camera image (a) of Fig. 3, the captured image of the surrounding environment is distorted or hidden by the raindrops, making pattern matching with the right camera image (b) difficult, and the overall number of parallaxes is reduced as in the parallax image (c).

The memory 5 stores the first image feature amount of the first image acquired by the image feature amount calculation unit 3. The first image for calculating the first image feature amount is acquired at a timing determined by a predetermined cycle defined by the wiper wiping. Figures 4A and 4B and Figures 5A and 5B show the wiper positions and camera images to explain the state immediately after the wiper wiping. In Figures 4A and 4B, 2A and 2B are left and right cameras, 10 is the windshield, and 11A and 11B are left and right wipers.

FIG. 4A is an image before the wipers are operated. In rainy weather, raindrops adhere to the windshield in front of the camera, and the image obtained from the camera is as shown in FIG. 5A. On the other hand, FIG. 5B is an image immediately after the wipers are wiped. As shown in FIG. 4B, the wipers 11A and 11B both pass in front of the left and right cameras to wipe off the raindrops adhering to the windshield, and a clear image as shown in FIG. 5B can be obtained. This is defined as immediately after the wipers are wiped. For example, in a typical vehicle, the wipers 11A and 11B operate (rotate) in one direction (clockwise in the example shown in FIG. 4A and B) on the windshield from the position before the wipers are operated (normal or stored position) as shown in FIG. 4A. Then, after reaching the position furthest from the position before the wipers are operated (positions AA and BB in FIG. 4B), they operate (rotate) in the opposite direction (counterclockwise in the example shown in FIG. 4A and B) on the windshield, and return to the position before the wipers are operated (normal or stored position) as shown in FIG. 4A. By repeatedly performing such a series of operations once or a plurality of times, raindrops adhering to the windshield are wiped away. In such a vehicle, the wipers 11A and 11B pass in front of the left camera 2A and the right camera 2B twice (twice in one direction and the other direction) during the series of operations. Depending on the installation position of the camera and the wiping range (operation range) of the wipers, the wipers 11A and 11B may pass in front of the left camera 2A twice (a total of four times for the wipers 11A and 11B), or the wipers 11A and 11B may pass in front of the right camera 2B twice (a total of four times for the wipers 11A and 11B). In general, it is considered that the wiper wiping is completed during the series of operations when the wipers 11A and 11B pass in front of the left camera 2A and the right camera 2B twice in one direction and the other direction (not once in one direction). Therefore, in the series of operations described above, the time immediately after wipers 11A and 11B have completely passed in front of the left camera 2A and the right camera 2B for the second time (in other words, immediately after passing in one direction and then in the other direction) is defined as immediately after wiper wiping.

In addition, in this embodiment, the timing of the wiper wiping operation can be the reference for the timing of function stop determination (image deterioration determination). Therefore, the period from immediately after the wiper wiping in one series of operations to immediately after the wiper wiping in the next series of operations is defined as the operation cycle (drive cycle) of the wipers 11A and 11B. In other words, the period from when the wipers 11A and 11B move (rotate) on the windshield from the position before the wiper operation to when they return to the position before the wiper operation can also be defined as the operation cycle of the wipers 11A and 11B. In other words, a predetermined timing in the series of operations of the wiper wiping operation can be set as the reference timing, and the period from a predetermined timing in one series of operations to a predetermined timing in the next series of operations can also be defined as the operation cycle of the wipers 11A and 11B.

Whether or not it is immediately after wiping with the wiper can be determined from an image captured by the camera, a wiper motor signal, a wiper drive cycle, a signal from a raindrop sensor, etc. The image feature amount calculation unit 3 overwrites and saves the image feature amount of the image immediately after wiping with the wiper in the memory 5 as the image feature amount of the first image.

The function stop determination unit 6 in FIG. 1 compares the image feature amount in the current frame (image) acquired by the image feature amount calculation unit 3 as a second image feature amount with the first image feature amount stored in the memory 5 to determine whether or not to stop the function (the driving assistance function of the recognition unit 4).

Next, a specific operation of the function stop determination unit 6 of the above-mentioned vehicle-mounted stereo camera 1 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing a procedure of the function stop determination unit 6 of the vehicle-mounted stereo camera 1 according to this embodiment.

First, in S101, it is determined whether the wipers are in operation, and if so, the process proceeds to S102. If the wipers are not in operation, the function stop determination is terminated. In order to determine whether the wipers are in operation, a wiper switch signal or the like is acquired.

In S102, the first image feature immediately after wiping stored in memory 5 is compared with the second image feature which is the number of features of the current frame to obtain a reduction rate d. When the first image feature is f1 and the second image feature is f2, the reduction rate d [%] can be obtained by the following formula (1).
[Equation 1]
d = (f1 - f2) ÷ f1 × 100...(1)

In S103, it is determined whether the decrease rate d is greater than a reference value dTH. If it is greater than the reference value dTH, the current frame is determined to be degraded, and the counter value of the HALT occurrence counter is incremented in S104. If it is equal to or less than the reference value dTH, the current frame is determined to be non-degraded, and the counter value of the HALT occurrence counter is decremented in S105.

In S106, it is determined whether or not to halt (HALT) the camera function based on the counter value of the HALT occurrence counter calculated in S104 and S105. If the counter value is equal to or greater than the reference value, the process proceeds to S107, where the camera function is halted (HALT). In this state, the driving support function performed by the recognition unit 4 is temporarily stopped.

Next, the specific behavior of the above flow will be described with reference to FIGS. 7A, 7B, and 7C, and FIGS. 8A, 8B, and 8C.

7A, 7B, and 7C are conceptual diagrams of behavior in a heavy rain scene. FIG. 7A is a graph of the time series change in the number of parallaxes. In the case of heavy rain, the image is clearly reflected immediately after the wiper is wiped (▲) (see FIG. 5B), so the number of parallaxes reaches a peak. After that, until the next wiper wipe, raindrops gradually adhere to the windshield in front of the camera (see FIG. 5A), so the number of parallaxes decreases. The number of parallaxes increases again immediately after the next wiper wipe, and the same behavior is repeated thereafter. FIG. 7B shows the results of the degradation judgment in each frame, and represents the result of the judgment in S103. In the graph of FIG. 7B, the case where degradation is judged in S104 is represented as 1, and the case where no degradation is judged in S105 is represented as 0. In the case of heavy rain, the rate of decrease in the number of parallaxes becomes large, so the behavior is a periodic repetition of degradation (1) and no degradation (0). FIG. 7C represents the HALT occurrence counters in S104 and S105. In the case of FIG. 7B, the counter value repeatedly increases and decreases by periodically repeating degradation (the rate of decrease in the number of disparities periodically exceeds the reference value multiple times) and non-degradation (the rate of decrease in the number of disparities periodically falls below the reference value multiple times). In this example, the increase value is made larger compared to the decrease value in each frame. Here, the counter value is compared with the reference value (S106), and if the counter value exceeds the reference value, the camera function is temporarily suspended (HALT) (S107). This temporarily stops the driving assistance function performed by the recognition unit 4 before it fails to operate or malfunctions.

8A, 8B, and 8C are conceptual diagrams of the behavior of light rain in an environment where parallax (or edges) are difficult to produce, such as snowfall. FIG. 8A is a graph of the time series change in the number of parallaxes. In the case of light rain, the image is clear immediately after wiping with the wiper (▲) as in the case of heavy rain (see FIG. 5B), so the number of parallaxes reaches a peak and then decreases, but the rate of decrease is small. FIG. 8B shows the results of the degradation judgment for each frame, and represents the result of the judgment in S103. In the case of light rain, the rate of decrease in the number of parallaxes is not large, so it is judged to be non-degraded (0). FIG. 8C shows the HALT occurrence counter in S104 and S105. Since it is non-degraded, the counter value decreases (specifically, since the counter value is 0 or more, it does not change and remains flat). Here, the counter value is compared with a reference value (S106), and the counter value is below the reference value. This allows the driving assistance function performed by the recognition unit 4 to be continued without stopping, and the burden of the driver's driving operation to be reduced.

In this way, the function stop determination unit 6 calculates the rate of reduction in the number of parallaxes by comparing the current frame with the frame immediately after the wiper has been wiped (when the wiper has completely passed in front of the left and right cameras and the image is clearly captured) as a reference frame, and can determine whether the environment is one in which parallax is unlikely to occur to begin with, or whether parallax is not occurring due to image degradation caused by heavy rain or the like (in other words, it is possible to distinguish between environments in which parallax is unlikely to occur to begin with, such as normal rain and accumulated snow).

In the above embodiment, the HALT occurrence counter is used to determine whether the function has stopped, but another method of capturing a periodic change in the number of disparities as shown in Fig. 7A may be used. For example, the function may be determined based on a periodic change (periodicity) in the rise interval or fall interval of the degradation determination result for each frame as shown in Fig. 7B.

In the above embodiment, the timing for determining whether the function has stopped is determined by comparing the current frame with the time immediately after the wiper has been wiped (when the wiper has completely passed in front of the left and right cameras and an image is clearly captured) as a reference, and calculating the reduction rate of the image feature amount of the current frame relative to the image feature amount of the image immediately after the wiper has been wiped, but the timing for determining whether the function has stopped is not limited to the above embodiment.

For example, the function stop may be determined by comparing the current frame with a predetermined time (or a predetermined number of frames) after the wiper is wiped as a reference and calculating a decrease rate of the image feature. Alternatively, the function stop may be determined by comparing the current frame with a frame immediately before the wiper is wiped (an image state in which the parallax is less likely to occur immediately before the wiper is wiped) as a reference and calculating an increase rate of the image feature of the current frame relative to the image feature of the image immediately before the wiper is wiped. Alternatively, the function stop may be determined by comparing the current frame with a frame immediately before the wiper is wiped as a reference (a frame immediately before the wiper is wiped or a frame immediately after the wiper is wiped) as a reference and calculating a change rate (decrease rate or increase rate) of the image feature. The predetermined time (or the predetermined number of frames) used in the determination here can be specified based on the wiper operation cycle (for example, the period from immediately after a wiper is wiped to immediately after the next wiper is wiped), etc.

In other words, the function stop determination unit 6 of this embodiment is able to determine function stop by identifying two different points (two times) in the wiper's operating cycle and calculating the rate of change (rate of decrease or rate of increase) in the image features between the two different points (two times) in the wiper's operating cycle.

Furthermore, in the above embodiment, the function stoppage is determined using the rate of change of the first and second image features, but the function stoppage may also be determined based on, for example, the results of a comparison between the absolute values of the first and second image features.

As described above, in the conventional technology, it is not possible to determine whether the environment is originally one in which image features such as parallax are difficult to produce, or whether image features such as parallax are not produced due to image degradation caused by heavy rain, etc. In such a case, when an environment in which image features such as parallax are originally difficult to produce, such as a snowy road, causes minor degradation of the image due to raindrops or snow adhering to the image, the system will erroneously determine that the system is HALT and stop the function (driving assistance control).

When image characteristics such as parallax are no longer visible due to image degradation (distortion of the entire screen) caused by heavy rain, etc., the HALT is activated and the function (driving assistance control) is temporarily stopped, and the user does not feel uncomfortable. On the other hand, when image characteristics such as parallax are originally difficult to appear on snowy roads, etc., and image characteristics such as parallax are no longer visible due to slight degradation of the image caused by raindrops (distortion of a part of the screen), the HALT is activated and the function (driving assistance control) is temporarily stopped, the user feels uncomfortable. In other words, when slight degradation not enough to cause HALT occurs in an image in a place where image characteristics such as parallax are difficult to appear (snowy roads or pitch black roads), if HALT is accidentally activated and the function (driving assistance control) is temporarily stopped, the user feels uncomfortable.

For this reason, it is important to determine whether the environment is one in which image features such as parallax are difficult to bring out in the first place, or whether image features such as parallax are not coming out due to image degradation caused by heavy rain, etc., and to accurately determine the situation in which the function (driving assistance control) should be temporarily stopped. In cases where the image degradation is severe, such as heavy rain, it is important to make a HALT judgment and stop the function, and in cases where the image degradation is only mild, such as light rain, it is important to not make a HALT judgment and not stop the function.

The in-vehicle stereo camera (external environment recognition device) 1 of the present embodiment described above includes cameras (left camera 2A, right camera 2B) mounted in the vehicle cabin, an image feature amount calculation unit 3 that calculates image feature amounts (parallax number, edge number, etc.) from images acquired by the cameras, a recognition unit 4 that recognizes objects outside the vehicle from images acquired by the cameras, a memory 5 that stores a first image feature amount in a first image acquired at a timing determined in a predetermined cycle (a timing specified by the operation of wipers 11A and 11B), and a function stop determination unit 6 that determines whether to temporarily suspend the function of the recognition unit 4 (HALT determination) based on the first image feature amount or a comparison result between the first image feature amount and a second image feature amount in a current image.

More specifically, the in-vehicle stereo camera (external environment recognition device) 1 of this embodiment saves the image feature amount immediately after wiping with the wiper, calculates the decrease rate with respect to the current image feature amount, and temporarily stops the function of the recognition unit 4 if the decrease rate is greater than a reference value and if the decrease rate periodically becomes greater than the reference value multiple times.

According to this embodiment, by making a judgment using time series data (for example, two time series frames based on the time immediately after wiping the wiper), it is possible to judge whether the environment is originally one in which image features such as parallax are difficult to appear, or whether image features such as parallax are not appearing due to image deterioration caused by heavy rain, etc., so that when the image deterioration is severe, such as heavy rain, it is judged as HALT and the function is stopped, and when the image deterioration is light rain, the HALT judgment is not made and the function is not stopped. Therefore, when the image deterioration is severe, such as heavy rain, the function (driving assistance control) is temporarily stopped, while when the image deterioration is light rain, which is not enough to HALT, the function (driving assistance control) is not stopped, and the driving assistance can be continued to reduce the burden of the driver's driving operation.

The present invention is not limited to the above-mentioned embodiment, and various modifications are included. For example, the above-mentioned embodiment has been described in detail to explain the present invention in an easy-to-understand manner, and is not necessarily limited to those having all the configurations described. In addition, it is possible to replace a part of the configuration of a certain embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of a certain embodiment. In addition, it is possible to add, delete, or replace other configurations with respect to a part of the configuration of each embodiment. In addition, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be realized by hardware, for example, by designing them as integrated circuits. In addition, each of the above-mentioned configurations, functions, etc. may be realized by software by interpreting and executing a program that realizes each function by a processor. Information such as a program, table, file, etc. that realizes each function can be placed in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

1...In-vehicle stereo camera (external environment recognition device)
2A, 2B... Camera (imaging unit) (2A: left camera, 2B: right camera)
3: Image feature amount calculation unit 4: Recognition unit 5: Memory 6: Function stop determination unit 10: Windshield 11A, 11B: Wiper

Claims (6)

A camera installed inside the vehicle,
an image feature amount calculation unit that calculates an image feature amount from the image captured by the camera;
A recognition unit that recognizes an object outside the vehicle from the image acquired by the camera;
a memory that stores a first image feature amount in a first image acquired at a timing determined in a predetermined cycle;
a function stop determination unit that determines whether or not to suspend a function of the recognition unit based on the first image feature amount or a comparison result between the first image feature amount and a second image feature amount in a current image;
having
the memory for storing the first image feature amount stores an image feature amount captured at a timing defined by the operation of a wiper;
the function stop determination unit determines whether or not to suspend the function of the recognition unit based on a rate of change in the image feature amount between two time series points in an operation cycle of the wiper;
The function stop determination unit calculates a rate of change of the second image feature relative to the first image feature from the first image feature immediately before wiping by the wiper stored in the memory and the second image feature in the current image, and stops the function of the recognition unit if the rate of change is greater than a reference value, and does not stop the function of the recognition unit if the rate of change is equal to or less than the reference value . The external environment recognition device according to claim 1,
The external environment recognition device is characterized in that the first image is acquired at a timing defined by the operation of a wiper. The external environment recognition device according to claim 1,
The external environment recognition device, wherein the image feature calculation unit sets the number of parallaxes calculated from two or more cameras in the current frame as the image feature. The external environment recognition device according to claim 1,
The external environment recognition device, wherein the image feature calculation unit sets the number of edges calculated from one camera in the current frame as the image feature. The external environment recognition device according to claim 1 ,
The external environment recognition device, wherein the function stop determination unit stops the function of the recognition unit when the rate of change periodically becomes greater than a reference value multiple times. The external environment recognition device according to claim 1,
The function stop determination unit determines whether to temporarily suspend the function of the recognition unit based on a comparison result between absolute values of the first image feature amount and the second image feature amount. The external environment recognition device.

Images