JP7602636B2 - External world recognition device and external world recognition method - Google Patents

Description

The present invention relates to an external environment recognition device and an external environment recognition method used during vehicle control such as following a preceding vehicle.

In recent years, automobiles equipped with Adaptive Cruise Control (ACC), which allows the vehicle to follow the vehicle ahead while maintaining a safe distance according to the vehicle speed, are becoming more common. Prior art documents related to ACC include, for example, Patent Document 1.

The abstract of Patent Document 1 states that the problem is to "provide an object detection method and object detection device that quickly determine whether or not a vehicle will cut in," and states that the solution is to "provide a method and device for detecting whether or not a vehicle will cut in." The abstract states that the problem is to "provide an object detection method and device for detecting whether or not a vehicle will cut in." and a solution is "The object detection device 100 includes stereo cameras 10, 11 that capture images of the area around the vehicle 50, and a controller 20. The controller 20 extracts a plurality of feature points from the images and extracts a plurality of feature points that constitute a moving object from the plurality of feature points. The controller 20 calculates the variance of the motion vectors of the plurality of feature points that constitute a moving object from a plurality of images captured at different times. If the variance of the components of the motion vector in the road width direction is equal to or greater than a threshold value, the controller 20 determines that the moving object is likely to cut in to the lane in which the vehicle is traveling."

Furthermore, paragraph 0014 of Patent Document 1 explains, "The optical flow calculation unit 22 calculates an optical flow from feature points of a past image and feature points of a current image. Optical flow is a vector that represents the movement of an object in a stereo image. The optical flow calculation unit 22 detects, as related feature points, current feature points that correspond to the same object in real space as an object corresponding to a past feature point. The optical flow calculation unit 22 calculates a combination of mutually related past feature points and current feature points as an optical flow."

In this way, the object detection device of Patent Document 1 recognizes other vehicles that cut in front of the vehicle by utilizing optical flow in stereo images.

JP 2020-3971 A

Paragraph 0030 of Patent Document 1 states that "When the cut-in determination unit 28 determines that there is a possibility that another vehicle will cut in, the vehicle control unit 30 brakes the vehicle so that the occupants do not feel uncomfortable. Also, the vehicle control unit 30 may brake the vehicle so that the occupants do not feel uncomfortable even when the cut-in determination unit 28 determines that another vehicle will cut in." In other words, in Patent Document 1, when there is a possibility that another vehicle will cut in or when it is determined that another vehicle will cut in, the vehicle is braked regardless of the manner of the cut-in. However, even if there is an approaching vehicle, if it is not possible to accurately determine whether or when contact is likely to occur based on the contour shape or speed vector of the approaching vehicle, it is clear that the control will cause the occupants to feel uncomfortable.

Furthermore, in paragraph 0004 of Patent Document 1, it is stated that "when speed control is performed using preceding vehicle following control or the like, if the cut-in determination is delayed, sudden deceleration may occur, which may cause passengers to feel uncomfortable.
" However, there is no specific mention of how to achieve control to follow the preceding vehicle without causing discomfort to the occupants.

Therefore, the present invention aims to provide an external environment recognition device and an external environment recognition method that, when a cutting-in vehicle cuts in between a preceding vehicle and the vehicle itself, recognizes the cutting-in vehicle as a vehicle based on side information of the cutting-in vehicle. It also aims to provide an external environment recognition device and an external environment recognition method that brakes the vehicle itself if there is a possibility of contact with the cutting-in vehicle, but if there is no possibility of contact, switches the vehicle to be followed in the preceding vehicle following control from the previous preceding vehicle to the cutting-in vehicle, thereby allowing the vehicle to continue stable driving.

An external environment recognition device comprising: a side extraction unit that extracts side information of the other vehicle from the image based on the parallax between left and right images capturing a portion of the other vehicle approaching the host vehicle; a rear search unit that searches for the rear position of the other vehicle based on the side information; a shape fitting unit that fits a rectangular parallelepiped shape that approximates the other vehicle in the image based on the side information and the rear position; and an other vehicle recognition unit that uses the rectangular parallelepiped shape to recognize the other vehicle as a vehicle ahead of the host vehicle.

According to the present invention, when a vehicle cuts in between the preceding vehicle and the vehicle itself, the vehicle can be recognized as a vehicle based on the side information of the vehicle. If there is a possibility of contact with the vehicle, the vehicle itself is braked, whereas if there is no possibility of contact, the vehicle to be followed in the preceding vehicle following control is switched from the previous preceding vehicle to the vehicle that cuts in, allowing the vehicle to continue driving stably.

FIG. 1 is a front view of a vehicle equipped with an external environment recognition device according to an embodiment of the present invention; Functional block diagram of an external environment recognition device according to an embodiment. A bird's-eye view of a scene where a vehicle cuts in between the vehicle ahead and the vehicle in question The environment ahead of the vehicle when a car cuts in between the vehicle ahead and the vehicle itself An example of UD space after voting for disparity values FIG. 5 is an explanatory diagram of three-dimensional object candidates extracted from the UD space in FIG. FIG. 5B is an explanatory diagram of a detection frame set in an image based on the three-dimensional object candidate of FIG. 5A; Functional block diagram of the preceding vehicle identification unit Functional block diagram of side surface extraction unit An explanatory diagram of a line segment parallel to the vehicle path extracted from the UD space in FIG. Illustration of the procedure for extracting side candidate lines Functional block diagram of right-angle fitting Illustration of side end fitting procedure Illustration of fitting procedure for rear side An explanatory diagram of the side-based rear search procedure when capturing an image of a part of a cutting-in vehicle An explanatory diagram of the side-based rear search procedure when capturing an image of the entire cutting-in vehicle An illustration of how the vehicle's outline appears on the image Illustration of the back-based side-searching procedure Detailed shape fitting procedure diagram Diagram of the procedure for tracking feature points within a contour and estimating speed Functional block diagram of the cutting-in vehicle determination unit Functional block diagram of the collision tracking determination unit An overhead view of an example of an environment in which collision tracking determination is performed. Processing flow chart of an external environment recognition device according to an embodiment

Below, one embodiment of the external environment recognition device of the present invention is explained using drawings.

<External world recognition device 100>
FIG. 1 is a front view of a vehicle V0 to which an external environment recognition device 100 according to an embodiment of the present invention is attached, and FIG. 2 is a functional block diagram of the external environment recognition device 100 of FIG. 1.

As shown in the front view of Figure 1, the external environment recognition device 100 is attached to the upper inner surface of the windshield of the host vehicle V0 and is a device for recognizing the environment in front of the host vehicle V0 during vehicle control such as adaptive cruise control (ACC).

As shown in the functional block diagram of FIG. 2, the external environment recognition device 100 includes a stereo camera unit 1, a parallax voting unit 2, a back surface extraction unit 3, a preceding vehicle identification unit 4, a contact tracing determination unit 5, a side surface extraction unit 6, a right-angle shape fitting unit 7, and an intrusion vehicle determination unit 8. In addition, each unit other than the stereo camera unit 1 in the configuration of the external environment recognition device 100 is specifically a computer unit equipped with hardware such as a calculation device such as a CPU, a storage device such as a semiconductor memory, and a communication device. The calculation device executes a program stored in the storage device to realize each of the above-mentioned functions, and the details of each unit will be described in sequence below while appropriately omitting such well-known technologies in the computer field.

<Stereo camera unit 1>
The stereo camera unit 1 is composed of a left camera 1L and a right camera 1R facing the front of the vehicle, and an image processing unit. In the image processing unit, a known stereo matching process is performed on the left image PL and the right image PR captured by the image pickup elements of each camera to generate a parallax image (information in which the parallax value D of the left and right images is arranged in two dimensions), and this is output as three-dimensional information in front of the vehicle. In this embodiment, the stereo camera unit 1 is used to generate the three-dimensional information, but instead of the stereo camera unit 1, other types of front sensors such as a monocular camera or Lidar may be used to generate the three-dimensional information.

<Disparity Voting Unit 2>
The disparity voting unit 2 votes for the disparity value D, which is the result of stereo matching in the stereo camera unit 1, in a disparity voting space (hereinafter referred to as "UD space") with the vertical axis representing the disparity value D and the horizontal axis representing the image horizontal coordinate U. By applying the principle of triangulation, the disparity value D can be converted into the depth distance to the captured object, so that the vertical axis of the UD space shown in FIG. 4 and other figures indicates not only the magnitude of the disparity value D but also the near or far depth distance.

For example, as illustrated in the overhead view of Figure 3A, if a cutting-in vehicle V2 cuts in from the left lane between the host vehicle V0, which is undergoing control to follow the preceding vehicle, and the preceding vehicle V1, which is the current vehicle to be followed, the environment in front of the host vehicle V0 will appear as illustrated in Figure 3B.

Figure 4 shows a UD space in which the disparity value D of the disparity image output by the stereo camera unit 1 in the environment of Figure 3B is voted. The disparity voting unit 2 completes the UD space as shown in Figure 4 by scanning the disparity value D vertically, one pixel row at a time (or multiple pixel rows at a time) from one end of the disparity image, and voting for the disparity value D for each pixel row in the UD space. If the disparity image includes a three-dimensional object, the disparity value D will be approximately the same value at the same depth distance of the three-dimensional object, so that in the vertical axis direction of each row in the UD space, the votes for the disparity value D will be concentrated near a certain value. In Figure 4, the vote concentration of the disparity value D is displayed with high luminance (dark white). In the UD space of Fig. 4 generated under the environment of Fig. 3B, vote concentrations of the parallax value D corresponding to the depth of each of the positions on the horizontal axis (image horizontal coordinate U) corresponding to the back of the preceding vehicle V1, the side of the cutting-in vehicle V2, and the back of the cutting-in vehicle V2 are generated, resulting in high luminance areas (dark white areas) being expressed in three places in the UD space. Note that, as shown in Fig. 3B, a curb is installed on the right side of the host vehicle V0, so that a vote concentration of the parallax value D caused by the curb (a tilted light white area) is also expressed on the right side of the UD space of Fig. 4.

<Back surface extraction unit 3>
The back surface extraction unit 3 detects a three-dimensional object based on the UD space (FIG. 4) obtained by the parallax voting unit 2. For a wide three-dimensional object that moves parallel to the host vehicle V0, such as a preceding vehicle V1 or an oncoming vehicle (not shown), the vote concentrations of the parallax value D are aligned horizontally in the UD space, so the back surface extraction unit 3 can detect a three-dimensional object candidate (back surface) by scanning the UD space horizontally and extracting a high-brightness area that is long in the horizontal direction. Below, an overview of the processing by the back surface extraction unit 3 will be described with reference to FIGS. 5A and 5B.

Fig. 5A is a diagram in which the back surface extraction unit 3 horizontally scans the UD space in Fig. 4 to extract regions where vote concentrations (dark white) of the disparity value D are continuous in the horizontal direction as solid object candidates. Note that the UD space in Fig. 5A originally also includes the luminance information shown in Fig. 4, but in order to emphasize the subject of explanation in Fig. 5A, the luminance information is omitted in Fig. 5A and only information not in Fig. 4 is displayed.
This treatment is the same in the UD space shown in FIG. 8, which will be described later.

When the three-dimensional object candidate shown in FIG. 5A is extracted, the back surface extraction unit 3 can correctly extract a three-dimensional object such as the back surface of the preceding vehicle V1 moving parallel to the vehicle V0 as shown in the detection frame in the image in FIG. 5B. On the other hand, for a three-dimensional object such as the cutting-in vehicle V2 moving in a direction not parallel to the vehicle V0, the back surface extraction unit 3 may erroneously detect the back surface and side surface of the cutting-in vehicle V2 in the image as independent three-dimensional objects. In addition, if the traveling direction (appearance) of the cutting-in vehicle V2 gradually changes in each frame continuously captured by the stereo camera unit 1, the width and ratio of the two three-dimensional object candidates (see FIG. 5A) caused by the cutting-in vehicle V2 gradually change, so that the back surface extraction unit 3 extracts two three-dimensional objects with unstable size, position, attitude, speed, etc.

Here, the back surface extraction unit 3 tracks the detection frame (dotted line in FIG. 5B) of the three-dimensional object extracted in each frame of the continuous captured images in a time series, and if it is tracked as a stable horizontal position and depth, it is determined to be a stable three-dimensional object. Then, only objects that can be tracked stably as a three-dimensional object in a time series are used for subsequent processing, and objects that are not stable in a time series are excluded as noise factors. For this reason, the detection frame for an unstable object, such as the side of the cutting-in vehicle V2, is not stable in a time series and is not used as a three-dimensional object. Also, as for the back surface, the width is unstable to a certain extent, and it may not be detected, or even if it can be tracked to a certain extent, the position, attitude, speed, etc. are unstable, and it is often not the correct target for warning or control. In order to suppress false warnings and erroneous control, it is necessary to design it so that control is not performed on objects with unstable positions and speeds.

In this way, when extracting a three-dimensional object using the back surface extraction unit 3, it was difficult to accurately recognize the cutting-in vehicle V2, whose back surface and part of its side surface are captured simultaneously, and in particular, because the cutting-in vehicle V2 cuts in at a steep angle from close range during traffic jams, etc., and therefore cannot be captured in its entirety.

<Preceding vehicle identification unit 4>
The preceding vehicle identification unit 4 selects three-dimensional objects that can be stably extracted in time series from among the three-dimensional object candidates extracted by the back surface extraction unit 3 (for example, the three-dimensional object candidate caused by the preceding vehicle V1 in FIG. 5A), and further performs a process of determining whether or not one of these is a vehicle based on its size and appearance in the image. To perform this process, the preceding vehicle identification unit 4 has a tracking unit 41, a vehicle back surface candidate selection unit 42, a vehicle identification unit 43, and a position and speed estimation unit 44, all of which are shown in FIG. 6. Each unit will be described in turn below.

First, the tracking unit 41 performs a tracking process on three-dimensional objects whose positions and sizes are stable over time in continuously captured images, and simply calculates their positions and speeds.

Next, the vehicle back surface candidate selection unit 42 selects from the three-dimensional objects being tracked those that have a size and speed that could possibly be a vehicle, and selects vehicle back surface candidates.

The vehicle identification unit 43 performs an identification process on the selected three-dimensional object candidates that are the size of the back of a vehicle, and identifies whether they are cars or not. Specifically, the identification process is performed for each frame, and a three-dimensional object that has a high identification score for several consecutive frames is finally identified as a vehicle.

The position and speed estimation unit 44 estimates with high accuracy the position and speed of a three-dimensional object identified as a vehicle.

As for other vehicles traveling ahead in the same lane as the vehicle V0, such as the preceding vehicle V1 in Fig. 3B, or other vehicles traveling in adjacent lanes that are traveling at a certain distance and whose sides are not easily visible, the rear of the vehicle can be captured stably, and therefore the preceding vehicle identification unit 4 can identify the vehicle as a vehicle. In addition, as for the cutting-in vehicle V2 at a long distance, even if there is some recognition delay, the possibility of a collision is low, so it was also possible to deal with the cutting-in vehicle V2 using the output of the preceding vehicle identification unit 4.

However, when it comes to a cutting-in vehicle V2 that cuts in at a steep angle from close range during a traffic jam, the back surface extraction unit 3 may not be able to extract the vehicle as a three-dimensional object reliably and may not be able to identify it properly, so the preceding vehicle identification method based on the back surface of the preceding vehicle identification unit 4 often does not work well. In order to accurately recognize the cutting-in vehicle V2 under such circumstances, the external environment recognition device 100 of this embodiment adds a side surface extraction unit 6 and a right-angle shape fitting unit 7, which will be described below.

<Side surface extraction unit 6>
The side surface extraction unit 6 searches for side surface candidates of the cutting-in vehicle V2 by using the UD space exemplified in Fig. 4. To execute this process, the side surface extraction unit 6 has a side surface candidate line search unit 61, a vehicle candidate determination unit 62, a tracking unit 63, and a side surface speed analysis unit 64, all of which are shown in Fig. 7. Each unit will be described below in order.

First, the side candidate line search unit 61 searches for a line segment extending along the vehicle path R0 (a line segment within a predetermined angle with respect to the vehicle path R0 on the overhead view) as shown in Fig. 8 from the high-luminance area in the UD space in Fig. 4. In the example of Fig. 8, a line segment caused by the side of the cutting-in vehicle V2 is searched for on the left side of the vehicle path R0, and a line segment caused by a curb is searched for on the right side of the vehicle path R0.

In this search process, the search target is a line segment with an inclination of about ±50° with respect to the vehicle's path R0 so that the side of the cutting-in vehicle V2 that cuts into the vehicle's path R0 can be searched for. Note that this angle of "50°" does not refer to the angle in the UD space of FIG. 4 or the angle on the actual image as in FIG. 3B, but refers to the actual relative posture of the vehicle on the overhead view (see FIG. 3A). The reason for limiting the angle of the line segment to be searched is that even the process of searching the back side can handle a certain degree of inclination of the back side. In addition, for vehicles that jump out completely from the side and proceed in a direction at an angle of 90° with respect to the vehicle V0 on the overhead view, separate measures have been taken to deal with head-on accidents, so they are not searched for in this embodiment. Note that, in the following, it is assumed that the cutting-in vehicle V2, including motorcycles, is detected, and line segments that exist on the side of the vehicle are searched for, for example, with a length of 1.5 m to 15 m. In the UD space, many three-dimensional objects exist other than the vehicle on the side, such as road shoulders and building walls, so multiple sides can be searched. However, when considering the safety of the vehicle V0, it is sufficient to consider only the nearest object.

For this reason, the side candidate line search unit 61 detects the approximate lateral position where the nearby cutting-in vehicle V2 is likely to cut in from the left and right sides of the UD space in Figure 4 using the procedure shown in Figure 9.

First, as shown in Fig. 9 (1), many lines parallel to the vehicle's path R0 on the overhead view are set in UD space (Fig. 4). Next, as shown in Fig. 9 (2), a histogram is generated that accumulates the number of votes for the disparity value D on each line. At the position where a three-dimensional object with a side (cutting-in vehicle V2) is located (near the dashed-dotted arrow in Fig. 9 (1)), the accumulated number of votes on the line is greater than elsewhere even if the angle of the side is slightly off, so in the histogram, a peak (maximum value) of the accumulated number of votes appears at the position on the horizontal axis (image horizontal coordinate U) that corresponds to this line.

In FIG. 9 (3), a peak with a cumulative number of votes equal to or greater than a threshold is searched for outward from the vehicle's path R0 (center). For example, in an environment in which the side of the cutting-in vehicle V2, the road shoulder, and a wall are present, in that order, on the left side of the vehicle's path R0, three peaks will be lined up on the left side of the histogram. However, since the process here is for searching for a three-dimensional object that comes into contact with the vehicle V0 or a three-dimensional object that directly affects the vehicle behavior of the vehicle V0, once a peak close to the center of the histogram (a peak caused by the side of the cutting-in vehicle V2) is extracted, the search process outward from that point can be stopped, and it can be determined that the peak is highly likely to be a three-dimensional object with a side, and a detailed analysis of this peak can be performed.

Next, in FIG. 9 (4), a line segment (dashed line arrow in FIG. 9 (1)) that corresponds to the approximate position of the line segment extending along the vehicle's path R0 in UD space is identified from the position of the histogram peak selected in FIG. 9 (3). After that, in FIG. 9 (5), a number of line segments with varying positions and angles are set around the line segment selected in FIG. 9 (4) as a reference, and a histogram is generated for each line segment by accumulating the number of votes for the disparity value D, as in FIG. 9 (2). Then, the line with the largest accumulated number of votes on the line (for example, the dashed line arrow in FIG. 9 (5)) is extracted as a candidate side line.

Next, the vehicle candidate determination unit 62 provisionally determines the length of the side based on the extracted side candidate line. For example, if the vote value on the side candidate line is greater than or equal to a certain threshold three times in a row, the first vote is set as the starting point of the line, and if the vote value falls below the threshold twice along the way, the line is considered to be continuing, and the length is checked by tracing the line, and if it falls below the threshold three times in a row, the point before the three consecutive votes is registered as the end point of the line. If the length of the line obtained in this way is 1.5 m or more, which corresponds to the side length of a two-wheeled vehicle, and 15 m or less, which corresponds to the side length of a large truck, the line is extracted as a vehicle candidate. On the other hand, if the length is outside this range, it is excluded from the vehicle side candidate line as noise, road shoulder, wall, etc. Furthermore, if it is determined to be a vehicle side candidate line, it is assumed that the length is 1 m or more and the effective parallax is about 40% or more, and it is confirmed that the vote amount on the line changes depending on the depth, but 40% of the vote amount is present assuming a height of 1 m. It is acceptable for there to be some areas that fall below the threshold, but if the average exceeds the 40% threshold, the vehicle is considered a candidate. In situations where there are not enough votes, the target object is determined to be a low object from the road surface, such as a road shoulder, and is not a vehicle, including motorcycles.

The tracking unit 63 performs tracking of the side candidate lines selected by the vehicle candidate determination unit 62. Once tracking of the side candidate lines has begun, processing is performed to reduce interruptions in tracking due to tracking failures by including line segments that do not meet the length or height criteria set by the vehicle candidate determination unit 62 as tracking targets.

The side speed analysis unit 64 calculates the speed (depth direction speed, lateral movement direction speed) of the side candidate line tracked by the tracking unit 63. Furthermore, the side speed analysis unit 64 performs stabilization in a time series and determines for each frame whether the extracted side candidate line is approaching the host vehicle V0 and whether it is an approaching side even when the lateral movement of the host vehicle behavior is taken into account.

<Right-angle fitting portion 7>
The right-angle fitting unit 7 precisely extracts the contour shape of the cutting-in vehicle V2 while reducing the computational processing load by fitting a right-angle shape only to the side candidate lines extracted by the side extraction unit 6. The right-angle fitting unit 7 also calculates the position and speed of the cutting-in vehicle V2, and determines whether the cutting-in vehicle V2 should be monitored in order to continue safe and comfortable driving of the host vehicle V0. Here, the cutting-in vehicle V2 that is determined to be a monitoring target mainly includes two types: (1) a cutting-in vehicle V2 that may come into contact with the host vehicle V0 unless the host vehicle V0 takes measures such as emergency braking or emergency avoidance, and (2) a cutting-in vehicle V2 that may approach to a distance less than a safe distance unless the host vehicle V0 decelerates. 10, the right-angle shape fitting unit 7 has a side leading edge fitting unit 71, a side trailing edge fitting unit 72, a side-based back surface searching unit 73, a detection base line switching unit 74, a back surface-based side surface searching unit 75, a detailed shape fitting unit 76, a feature point tracking/speed estimation unit 77, and a vehicle identification unit 78. Each unit will be described in turn below.

Side end fitting part 71
The side surface extraction unit 6 roughly extracts the side surface length of the cutting-in vehicle V2, but the side surface tip fitting unit 71 searches for the tip of the cutting-in vehicle V2 in detail. To this end, the side surface tip fitting unit 71 executes a method of finding the tip based on distance using parallax, or a method of finding the tip based on texture.

First, a method of re-searching the tip of the cutting-in vehicle V2 based on the distance using parallax will be described with reference to FIG. 11. In this method, as shown in FIG. 11, the depth of the tip of the cutting-in vehicle V2 on the original image is measured using a parallax image, based on the side tip position (initial value) extracted in the UD space. In this case, the difference in depth between the vehicle body and the road surface in the parallax image is used to identify the tip position of the vehicle body. As shown in the upper diagram of FIG. 11, when checking the depth on a dashed line that crosses near the bumper of the cutting-in vehicle V2 (e.g., corresponding to a height of 50 cm from the bottom end of the cutting-in vehicle V2) on the original image, a parallax value D corresponding to the depth distance to the side of the vehicle body is calculated on the left side of the dashed line, whereas a parallax value D corresponding to the depth distance to the road surface that is farther away than the side of the vehicle body is calculated on the right side of the dashed line, so that there is a large difference in the parallax value D (depth distance) between the left and right of a certain position. By extracting the boundary where this difference occurs as a more accurate position of the vehicle front end, it is possible to search for the accurate front end of the cutting-in vehicle V2.

On the other hand, when the above method cannot be used, for example, when only a small portion of the cutting-in vehicle V2 is included in the imaging range and the texture of the cutting-in vehicle V2 is insufficient, or when it is nighttime and the texture of the cutting-in vehicle V2 is dark and cannot be imaged clearly, normal stereo matching cannot be performed, and invalid parallax occurs frequently, so fitting of the tip position described above may not be possible. In such cases, not only parallax values but also texture information can be used effectively. Specifically, a method of correcting the tip of the vehicle body by utilizing a method of identifying the boundary position from the difference between the road surface texture and the vehicle body texture may be used.

A method for separating by texture may be a method for extracting boundaries based on differences in brightness or color information, a method for dividing similar textures into small regions, or a method for utilizing semantic segmentation. A method for improving stability by using both parallax and brightness may also be used.

Side rear end fitting portion 72
In the side rear end fitting unit 72, the rear end position is fitted again to the line segment of the side of the cutting-in vehicle V2 that has already been extracted. Even if the cutting-in vehicle V2 cuts in front of the host vehicle V0, it basically cuts in while running parallel to the host vehicle V0. Therefore, if the rear end of the cutting-in vehicle V2 enters the image captured by the stereo camera unit 1 of the host vehicle V0 to some extent, the parallax of at least the rear end (rear) part of the vehicle should be visible as parallax vote values (high brightness areas) that are aligned almost horizontally in the UD space. Of course, if the angle of the cut-in is too sharp, this is not the case and the voting line may be slightly inclined, but basically the vehicle rear parallax is aligned horizontally in the UD space.

As shown in the lower diagram of Figure 12, an analysis is performed to accumulate voting values horizontally for the rear end of the candidate line on the side of the vehicle that was extracted in advance and create a histogram. When created as a histogram, the rear part where voting values accumulate horizontally becomes the peak of the accumulated value. For this reason, this peak part is extracted as the detailed rear position. As a result after fitting, the extension of the candidate line segment on the side and the intersection of the histogram peak are extracted as white square points, and this is used as the rear end fitting result. If the cut-in angle is sharp and the histogram peaks are relatively dispersed, the current correction value can be treated as provisional, with emphasis placed on correction in subsequent processing.

Side-based back surface search unit 73
The side-based rear search unit 73 searches for a rear line segment starting from the position of the rear end of the side surface in the UD space extracted by the side rear end fitting unit 72. For example, FIG. 13A is an explanatory diagram of the side-based rear search procedure when a part of the cutting-in vehicle V2 is imaged, and FIG. 13B is an explanatory diagram of the side-based rear search procedure when the entire cutting-in vehicle V2 is imaged. As shown in these figures, the angle is changed to search, assuming some angular deviation of the side line segment centered on the position that is perpendicular to the inclination of the side surface in the real space (on the bird's-eye view). Line scanning is performed by changing the angle and vertical position centered on the rear end position of the side surface of each view, which is the starting point for starting the search. In line scanning, a histogram is generated by accumulating the vote values on the line in the UD space. Therefore, the accumulated vote values are highest on the line drawn along the actual rear surface of the vehicle, so the vehicle rear position can be searched for by identifying the line with the highest accumulated value from the histogram. Since the side rear end position itself may contain an error, by searching while changing the position and angle using this position as a reference, the error in the side rear end position itself can be absorbed, and even if there is a deviation in the angle of the side, it is possible to correctly and stably search for the line segment of the back surface in the UD space. Also, it is possible to confirm that the back surface is almost not within the screen by checking that the side rear end position is at the edge of the screen and that the cumulative value of the histogram does not become high even after searching.

Detection base line switching unit 74
The detection base line switching unit 74 determines whether it is more stable to perform detection processing of the cutting-in vehicle V2 shown in the image based on the back or the side. For more stable detection of the cutting-in vehicle V2, it is better to use a longer line segment on the image as the base line, and by using the longer base line as a clue to search for the other line segment that is difficult to see in the perpendicular direction, the vehicle's outline shape (length and position of the back and side) can be stably detected. Therefore, the detection base line switching unit 74 determines whether to use the back or side as the base.

Here, the left diagram of FIG. 14 is shown as an example of a situation in which it is more stable to detect the contour of the cutting-in vehicle V2 based on the side, and the right diagram of FIG. 14 is shown as an example of a situation in which it is more stable to detect the contour of the cutting-in vehicle V2 based on the back. As shown in the upper left diagram of FIG. 14, when the cutting-in vehicle V2 cuts in just before the vehicle V0, the length of the side of the cutting-in vehicle V2 is longer than the length of the back in the image captured by the stereo camera unit 1 (lower left diagram of FIG. 14). Therefore, it is considered that the contour detection of the cutting-in vehicle V2 is stable if the side is used as the base. On the other hand, as shown in the upper right diagram of FIG. 14, when the cutting-in vehicle V2 cuts in from a distance, the length of the back of the cutting-in vehicle V2 is longer than the length of the side in the image captured by the stereo camera unit 1 (lower right diagram of FIG. 14). Therefore, it is considered that the contour detection of the cutting-in vehicle V2 is stable if the back is used as the base.

As shown in FIG. 14, if the exact contour shape (length of the back and length of the side) of the cutting-in vehicle V2 is known on the image captured by the stereo camera unit 1 of the vehicle V0 (see the lower left or lower right diagram of FIG. 14), it is possible to calculate which of the side and the back is longer on the image. However, before the back and the side are identified, a process is required to select either the back or the side as the base line. For this reason, the detection base line switching unit 74 estimates the rough center of gravity position of the cutting-in vehicle V2 at the time when only the back of the cutting-in vehicle V2 is identified or when only the side of the cutting-in vehicle V2 is identified, and selects which of the estimated center of gravity and the angle of the stereo camera unit 1 is used to extract the contour shape. In this case, the center of gravity position may be estimated using a standard vehicle size that serves as a reference from the extraction results of the side and back, and their relative positions. Note that this only roughly determines whether the back or side should be used as the base to stabilize the overall processing, so if the back and side are roughly the same length, the results will not differ significantly regardless of which is used as the base for the search.

Contour detection performance is significantly degraded when trying to extract the back of the vehicle V2 based on the side when the side of the vehicle V2 is not clearly visible, as in FIG. 15. In this case, the back is searched for based on the unstable side extraction results, making it difficult to ultimately obtain a stable contour shape (length of the back and side). Conversely, for a vehicle V2 whose entire back is not captured in the image as in FIG. 13A, or a vehicle V2 whose side appears longer than the back due to a sharp cut-in angle as in FIG. 13B, using the back as a base results in the extraction of the side from the unstable back, and as a result, the searched side is likely to be unstable and unreliable.

For this reason, the detection base line switching unit 74 may estimate the approximate center of gravity position of the cutting-in vehicle V2 under various conditions, such as assuming that the cutting-in vehicle V2 is about the same size as a typical passenger car or that the cutting-in vehicle V2 is traveling parallel to the host vehicle V0, and then select whether the back or side is to be used as the base line based on the angle calculation result between the center of gravity position and the installation position of the stereo camera unit 1 of the host vehicle V0. Alternatively, the detection base line switching unit 74 may switch the base line for the detection process using the extracted length of the side or back, and the angle between the center of gravity and the host vehicle V0.

In addition, if the cutting-in vehicle V2, which has begun to be tracked, gradually moves in front of the host vehicle and eventually becomes the preceding vehicle V1 of the host vehicle V0, the detection base line switching unit 74 may switch from side-based detection processing to rear-based detection processing at a certain point in time.

Back-based side search unit 75
The rear surface-based side surface search unit 75 uses the searched rear surface inner end position to search for the side surface line segment in the UD space. As shown in the lower diagram of FIG. 15, the side surface is searched for by changing the angle in the UD space, assuming some angle deviation of the rear surface line segment, centered on the position that is perpendicular to the inclination of the rear surface of the cutting-in vehicle V2 on the bird's-eye view. Furthermore, considering the position deviation of the rear surface inner end, the lateral position is also changed and searched for. Line scanning is performed by changing the angle and position with the angle at the center being a right angle to the rear surface, centered on the rear surface inner end in the lower diagram of FIG. 15, which is the reference position for starting the search. In line scanning, a line is drawn in the UD space, and the voting values on the line are accumulated to generate a histogram. Therefore, the line drawn to trace the vehicle side portion with a high voting value has the highest cumulative value, and the vehicle body side position can be searched for. Since the rear surface inner position itself may have an error, by changing the position and angle and searching, it is possible to absorb the deviation of the rear surface inner position and stably search the side surface line segment correctly in the UD space.

Detailed shape fitting section 76
In determining a collision with the cutting-in vehicle V2, it is important to accurately estimate the rear corner position of the cutting-in vehicle V2. This is because, although a collision or contact with the cutting-in vehicle V2 may occur at the rear, rear corner, side, or front corner, and while a collision or contact with the rear can be avoided by using a conventional technology for avoiding a rear-end collision with the preceding vehicle V1, a collision or contact with the rear corner, side, or front corner cannot be avoided by the conventional technology, so that accurately estimating the positions of the side and front corners based on an accurately estimated rear corner position is effective in avoiding a collision or contact with the rear corner, side, or front corner of the cutting-in vehicle V2.

Although the general position of the side can be estimated by any of the above methods, if the cutting in vehicle V2 is a vehicle with rounded corners, or if the vehicle has a body shape or color that makes it difficult to calculate parallax, it may not be possible to accurately estimate the length of the side line segment (i.e., the positions of the front and rear corners). Therefore, the detailed shape fitting unit 76 assumes that the cutting in vehicle V2 has a rectangular outline on the overhead view, fits the vehicle's outline, and also accurately extracts the positions of the rear corners.

Figure 16 is a diagram illustrating an example of the procedure for fitting an outline frame by the detailed shape fitting unit 76. First, as shown in Figure 16(a), the parallax of the side imaged by the stereo camera unit 1 is used to calculate the normal at each point on the side, and the parallax of the back imaged by the stereo camera unit 1 is used to calculate the normal at each point on the back. This allows a large number of normals to be drawn on the side and back of the cutting-in vehicle V2. Next, as shown in Figure 16(b), a vertical line on the overhead view to the back normal and a vertical line on the overhead view to the side normal are calculated, pairs of side normals and back normals are selected in sequence, and the intersections of each pair are calculated. Then, a point where the intersections are concentrated near the rear corner of the cutting-in vehicle V2 is calculated, and this part is extracted as the rear end corner. With this method, the position of the rear corner can be accurately extracted while removing the influence of missing parallax and errors in the length of the line segment as noise.

Alternatively, instead of Fig. 16(b), the position of the rear corner may be accurately extracted by executing the process of Fig. 16(c). That is, as shown in Fig. 16(c), the multiple normals obtained in Fig. 16(a) and a rectangle prepared in advance (for example, a rectangle corresponding to the outline of a typical passenger vehicle) are set, and the rectangle is rotated so that the total distance at which the normals intersect with the rectangle is the shortest (so that the total squared error is minimized), and the angle at which the rectangle best fits the cutting-in vehicle V2 is estimated. This makes it possible to accurately estimate the rear end corner and the leading edge.

Feature point tracking/speed estimation unit 77
The detection process described above has made it possible to estimate the contour shape (length and position of the back and side) of the cutting-in vehicle V2. However, in a situation where only a part of the side or back of the cutting-in vehicle V2 is captured in the image, after fitting of the contour shape by the above method, the vehicle size is often detected as being larger as the cutting-in vehicle V2 enters the screen. For this reason, in a speed estimation method using the center of gravity or a speed estimation method assuming that the entire vehicle body is visible, there is a possibility that the error in the speed estimated for the cutting-in vehicle V2 is large. Therefore, in order to stably estimate the speed of the cutting-in vehicle V2 that has entered the image captured by the stereo camera unit 1, a method of tracking characteristic parts of the vehicle body in the image (such as corners of parts) is effective, and in order to extract the accurate vehicle speed, it is considered that it is effective to use the original image, which is the image with the highest resolution. A general UD space often has a lower resolution than the original image, and it may be difficult to calculate an accurate speed in terms of resolution. However, if the resolution is the same as that of the original image, there is no effect of the resolution. However, to calculate the speed more accurately, it is better to use the corners of the image, etc., as this allows for a stable calculation of the speed regardless of whether the part captured on the image is part of the image or not.

As a result, the feature point tracking/speed estimation unit 77 tracks feature points between successive image frames for the cutting-in vehicle V2 within a rectangular parallelepiped equivalent to the rectangle fitted by the detailed shape fitting unit 76, as shown in Fig. 17. In particular, by preferentially using feature points that can be stably tracked across multiple frames, noise components that could only be tracked in one frame are removed, and from among the tracked feature points, those with a moving distance that deviates significantly are removed as noise components, and only the feature points that could be stably tracked are used to calculate the speed of the cutting-in vehicle V2.

Because the vehicle's contour shape and the three-dimensional positions of the feature points can be measured from the parallax images of the stereo camera unit 1, in FIG. 17 the movement directions (depth direction, lateral direction) of the three-dimensional points on the vehicle body contour shape are expressed as velocity vectors. In this way, because multiple feature points are calculated, multiple velocity vectors are calculated with each position on the surface of the vehicle body contour shape as the origin. Assuming the vehicle body is a rigid body, even if the origin is different in three-dimensional space, the same speed will be obtained in the same direction. Although the obtained velocity vector contains noise components, a stable velocity vector is calculated by removing outliers and using the average value.

In addition, the speed vector is calculated by tracking feature points of the background and white lines in the image in the same way, and the relative speed vector, absolute speed vector (the direction in which the vehicle is moving relative to stationary objects), position within the lane, etc., required to determine whether the cutting in vehicle V2 poses a risk of collision or contact with the host vehicle V0, such as whether the cutting in vehicle V2 is approaching a white line, whether it is approaching across a white line, or whether the other vehicle is approaching the host vehicle regardless of the behavior of the host vehicle V0, are calculated.

Vehicle identification unit 78
When the feature point tracking/speed estimation unit 77 completes the estimation of the speed vector, it can be determined whether the three-dimensional object candidate is a rectangular parallelepiped and whether it is approaching the host vehicle V0, so that the three-dimensional object candidate can be identified with a certain degree of accuracy as the cutting-in vehicle V2. However, in order to further improve the detection accuracy of the cutting-in vehicle V2, the vehicle identification unit 78 is configured to identify whether the cutting-in vehicle V2 is a four-wheeled vehicle or a two-wheeled vehicle based on information such as the rear width and side length of the cutting-in vehicle V2. This prevents a stationary object from being erroneously detected as a cutting-in vehicle candidate in an environment where errors are likely to occur in the host vehicle behavior and the movement of other vehicle candidates, such as a situation in which the host vehicle V0 approaches a wall while traveling around a curve.

<Cut-in vehicle determination unit 8>
The cutting-in vehicle determination unit 8 utilizes the information obtained in the right-angle shape fitting unit 7 in a time series manner to comprehensively determine whether the cutting-in vehicle V2 should be subject to control with respect to the host vehicle V0. To execute the above processes, the cutting-in vehicle determination unit 8 has a white line information unit 81, a same lane merging determination unit 82, a complex congested environment determination unit 83, a preceding vehicle candidate determination unit 84, and a contact possibility determination unit 85, all of which are shown in Fig. 18. Each unit will be described in turn below.

White line information unit 81
First, the white line information unit 81 observes the results of white line recognition as one piece of information for making a comprehensive judgment. Specifically, the white line information unit 81 grasps the left and right white lines of the lane in which the host vehicle V0 is traveling, and also grasps whether the host vehicle's behavior is stable within the left and right white lines. Furthermore, if there is a vehicle traveling parallel to the vehicle, it grasps whether the vehicle traveling parallel to the vehicle is traveling within the white lines while complying with traffic rules. Then, if both the host vehicle and the other vehicle are traveling stably within the white lines, the same lane merging judgment unit 82 is activated. On the other hand, if not, the congested complex environment judgment unit 83 is activated.

Same lane merging determination unit 82
The same lane merging determination unit 82 uses the position and presence of white lines to determine whether a vehicle will cut in to the lane in which the vehicle V0 is traveling. The same lane merging determination unit 82 extracts, as candidates for cutting in vehicles, vehicles that are likely to cut in to the lane in which the vehicle V0 is traveling, that are already approaching the lane by slightly straddling the white line, or vehicles that are likely to come into contact within a predetermined time. Conversely, for vehicles that do not cut in to the white line, even if they are approaching the vehicle V0 to a certain extent, it is determined that the vehicle has no intention of cutting in. However, if the lateral position of the vehicle or another vehicle within the white line is changing quickly, the possibility of contact may be determined by predicting the degree of change even if the vehicle does not deviate from the lane.

Congested and complex environment determination unit 83
The complex congested environment determination unit 83 determines that the driving environment is one in which the rule of driving within the white lines is not observed and the white lines are blurred or do not exist in the first place, and does not use the results of white line recognition even if they are available. This is because if neither other vehicles nor the subject vehicle V0 observe the rule of driving within the white lines, there is no point in using the results of white line recognition. In such a driving environment, it is basically determined whether control should be performed based on the behavior of the subject vehicle V0 and the behavior of the opposing vehicle.

- Leading vehicle candidate determination unit 84
When the host vehicle V0 performs adaptive cruise control (ACC), the preceding vehicle candidate determination unit 84 determines whether a nearby vehicle can be a candidate for the preceding vehicle V1 to be followed, and if so, The vehicle is set as a candidate for the preceding vehicle V1. In particular, as shown in FIG. 3A and FIG. 3B, when the host vehicle V0 is automatically following the preceding vehicle V1, the vehicle V2 that cuts in at a lower speed than the preceding vehicle V1 is set as a candidate for the preceding vehicle V1. When the preceding vehicle V1 cuts in on the path of the vehicle, it is important to quickly switch the vehicle to be followed from the preceding vehicle V1 to the cutting-in vehicle V2 in order to continue safe control of following the preceding vehicle. If the vehicle switching is delayed, there is a possibility that the vehicle V0 will not decelerate and will collide with the low-speed cutting-in vehicle V2. Even if the vehicle V0 avoids a collision by decelerating to a certain extent, the vehicle V0 may be overly close to the cutting-in vehicle V2. If the vehicle V0 approaches the obstacle, it may cause anxiety to the occupants of the vehicle V0, and the driver may take action such as braking or evading the obstacle even though the vehicle is in automatic following driving mode.

Therefore, for example, when the preceding vehicle candidate determination unit 84 determines that the driving area is one in which driving within the white lines is required, it uses as one of the conditions for determining whether the preceding vehicle is a candidate that the cutting in vehicle V2 crosses the white lines and cuts in to the own lane, but conversely, when the driving area is determined to be one in which driving within the white lines is not required, it does not use that condition.

・Contact possibility determination unit 85
The contact possibility determination unit 85 performs contact determination using the contour shapes of the vehicle V0 and the other vehicle. In a conventional contact determination method that uses only the rear surface of the other vehicle, the possibility of contact with the tip of the other vehicle is low. In the past, it was difficult to determine the possibility of side-to-side contact, etc. Therefore, the contact possibility determination unit 85 of this embodiment accurately estimates the outline shape of the other vehicle and uses this outline shape to The shape of the vehicle V0 is assumed to be known, and a collision judgment between the contour shapes is performed. In areas where the rule of driving within the white lines is observed, a warning or control is issued to a vehicle that cuts through the white lines and approaches. On the other hand, in areas where the rule of driving within the white lines is not observed, the system does not take into account whether the vehicle is cutting in between the white lines, and does not issue warnings or control until the vehicle is close to contact. In general, the host vehicle V0 and another vehicle are often driven so close to coming into contact with each other that it is difficult to determine whether the driving is intentional or whether the scene poses an unintentional risk of collision. For this reason, no warnings or controls will be issued until contact is imminent.

<Contact tracing determination unit 5>
The contact tracing determination unit 5 performs determination of warning and control for contact, collision, etc. with the vehicle V0 for objects that are identified as vehicles and whose position and speed change in a manner that is likely to occur for a vehicle. The objects are also used as tracking targets while the preceding vehicle automatic tracking (ACC) is in use. To execute this process, the contact tracing determination unit 5 has a collision determination unit 51, a preceding vehicle tracking unit 52, a tracking target change unit 53, and an integrated determination unit 54, as shown in FIG. 19. Each unit will be described in order below.

20 shows a situation where a preceding vehicle V1 and a vehicle V0 are present in the same lane. First, the collision determination unit 51 and the preceding vehicle tracking unit 52 will be described with reference to this diagram.
20 is moving slower than the host vehicle V0 or is stopped, there is a possibility that the host vehicle V0 will collide with the preceding vehicle V1. If the distance between the preceding vehicle V1 and the host vehicle V0 is sufficiently large, the driver can apply the brakes to slow down or stop the vehicle, and a warning is sufficient as a collision prevention measure, and emergency braking is not necessary.

Therefore, the collision judgment unit 51 uses the relative speed and distance from the position and speed of the vehicle recognized by the preceding vehicle identification unit 4 to judge how many seconds it will take for the vehicle to come into contact with the vehicle's own vehicle V0, and decides to issue an alarm if the situation is such that the rear-end collision can be avoided by the driver applying the brakes, but on the other hand, decides to issue an emergency brake if the timing is such that it is difficult to avoid the rear-end collision through the driver's operation.

Next, the preceding vehicle tracking unit 52 determines the control amount of the accelerator, brake, etc. of the host vehicle V0 so that the host vehicle V0 can smoothly follow the preceding vehicle V1 based on the relative speed and distance between the preceding vehicle V1 and the host vehicle V0 when the driver is performing ACC, so that the distance between the preceding vehicle V1 and the host vehicle V0 is maintained at a constant distance set by the driver and below the upper limit speed.

In the tracking target change unit 53, when the preceding vehicle candidate determination unit 84 determines that a cutting-in vehicle V2 has cut in between the preceding vehicle V1 and the vehicle itself V0 while following the preceding vehicle, the tracking target change unit 53 determines the timing to change the vehicle to be followed.

On the other hand, when a vehicle approaches from a close distance, only the side of the vehicle is captured in the image of the stereo camera unit 1, or even if the rear of the vehicle is captured, only a small part of the rear of the vehicle is shown in the image. In such cases, in scenes where the side of the vehicle is widely visible, conventional methods tend to be unstable in detecting the approaching vehicle and estimating its speed, and it is often determined that control cannot be used.

For this reason, in this embodiment, a side-based cutting-in vehicle detection method beginning with side extraction unit 6 is used to stably detect the vehicle body contour and estimate the speed of vehicles whose sides appear large. Based on the information obtained by these methods, and using the judgment results from cutting-in vehicle judgment unit 8, the integrated judgment unit 54 determines the timing of warnings and the content of control, including emergency braking and avoidance measures for cutting-in vehicle V2 at close range.

In addition, in the integrated judgment unit 54, when the driver has permitted control to avoid obstacles within the lane or the roadway, based on the information obtained above, an integrated judgment may be made as to whether to simply apply the brakes, or whether avoidance while steering and braking should be performed after checking the availability of three-dimensional objects within the lane or the availability of three-dimensional objects on the roadway including the shoulder. In addition to avoidance within the white lines, when shoulder detection and sensing of the entire circumference of the vehicle are possible, the integrated judgment unit may also perform evacuation to the shoulder. When a vehicle traveling slightly diagonally ahead of the vehicle V0 is on the right front and there are no surrounding vehicles, a judgment is made to perform evasive steering while crossing the white line to the left, and evacuation is performed by simultaneously sensing vehicles traveling in the left lane and the shoulder of the road.

Vehicle control unit 9
The vehicle control unit 9 performs sound and vehicle speed control, emergency braking, steering, etc., in accordance with the warning and control content determined above.

<Fig. 21 Processing flow>
Hereinafter, a processing procedure performed by the external environment recognition device 100 having the above-described configuration will be described with reference to the flowchart of FIG.

First, in step S1, the left and right cameras of the stereo camera unit 1 capture a left image PL and a right image PR of the area in front of the host vehicle V0 (see Figure 3B).

Next, in step S2, the image processing unit of the stereo camera unit 1 performs stereo matching after parallelizing the left and right images and performing sensitivity correction, to generate a disparity image.

In step S3, the disparity voting unit 2 uses the disparity image obtained in step S2 to vote for a disparity value D in the UD space (see FIG. 4). In this disparity value voting, when there is disparity with the same depth and the same horizontal coordinate on the image, that is, when there is a three-dimensional object standing vertically on the road surface reflected in the image, the disparity votes tend to gather in the same place in the UD space. In other words, by using this UD space, disparity vote values are accumulated in places where three-dimensional objects exist. Therefore, by searching for places in the UD space with high vote values, the presence of a three-dimensional object can be detected. The surrounding environment is sensed using this UD space.

In step S4, the back surface extraction unit 3 horizontally scans the UD space to search for three-dimensional object candidates in order to extract the back surface of the three-dimensional object using the results of voting in the UD space (see FIG. 5A). Since the search is performed horizontally, a three-dimensional object whose back surface extends horizontally as viewed from the camera can be detected stably. In particular, since the vehicles are often facing the same direction, the back surface of the vehicle can be detected by this process, and the vehicle back surface candidates extracted by this process are used for emergency braking of the preceding vehicle V1 and following the preceding vehicle V1. In this process, three-dimensional objects such as two-wheeled vehicles, pedestrians, and utility poles are also extracted, and three-dimensional objects with a certain width or more are extracted once, and then, for running objects such as pedestrians, four-wheeled vehicles, and two-wheeled vehicles, the type identification process is performed.

In step S5, the preceding vehicle identification unit 4 performs an identification process on the three-dimensional object candidates extracted in step S4 that are within the vehicle's expected width, and identifies whether they are vehicles or not. Furthermore, if a lane recognition result is obtained, this result is used in combination to determine whether the object is a vehicle in an adjacent lane or the preceding vehicle V1. Furthermore, for objects within the expected width for a two-wheeled vehicle among the three-dimensional object candidates, a determination is simultaneously made as to whether they are two-wheeled vehicles or not. Similarly, this is also performed on pedestrians, and by identifying the type of moving object for which collisions or contact are of concern, particularly while driving a vehicle, reliability is improved.

Conventional three-dimensional object detection and vehicle detection using steps S4 and S5 described above are basically methods for searching for walls facing the vehicle V0, and are simplified by capturing three-dimensional objects. This method itself is very efficient from the viewpoint of capturing three-dimensional objects in the vehicle's path R0 and applying emergency braking, or following the preceding vehicle V1. However, it has drawbacks, such as being insufficient for handling walls that come into contact with the side of the vehicle V0, or the side of a cutting-in vehicle, and making it difficult to accurately capture the inclination and position of three-dimensional objects.

The basic function of the past was sufficient when three-dimensional objects such as vehicles and pedestrians were relatively high and were controlled only when they entered the vehicle's path R0, but in recent years, this method has been insufficient when considering not only the incorporation of basic functions but also control of the driving environment around the path. For this reason, in the present invention, in order to perform contact judgment with a cutting-in vehicle V2, which is difficult to handle in the conventional processing (steps S4 and S5) that searches for the rear, and to perform switching judgment of the preceding vehicle to be followed when cutting in, the following steps S6 to S8 are executed in parallel with the above steps S4 and S5.

In the case of a cutting-in vehicle V2 traveling close by in an adjacent lane, the side of the vehicle is largely visible from the vehicle's own vehicle V0, so it is often more stable to extract the side of the vehicle rather than the back. In particular, there are cases in which part or all of the back of the vehicle is not visible from the camera, and the back extraction process (step S4), which searches laterally, cannot stably and accurately extract the side of the cutting-in vehicle V2.

Therefore, in step S6, the side surface extraction unit 6 performs a side surface search and extracts the side surface of the cutting-in vehicle V2 (see Figures 8 and 9). Furthermore, since the cutting-in vehicle V2 is the search target in this step, the side surface extraction unit 6 determines whether it is a three-dimensional object with a certain height, a certain depth length, and whether it is approaching the vehicle's own vehicle, thereby judging whether it is a candidate for a cutting-in vehicle.

In step S7, if the candidate vehicle is actually the vehicle V2, the right-angle fitting unit 7 uses the fact that a back line segment exists at a right angle to the side line segment to perform a refit process (see Figs. 11 and 12) of the leading edge and trailing edge positions of the side line segment, and further a back search process (see Figs. 13A and 13B) to search for a more stable right-angle shape. This makes it possible to capture a stable outline shape for vehicles that are missing only the side or part of the back relative to the vehicle V0, or vehicles whose side appears larger than the back, which was previously difficult to do, and makes it possible to obtain information for accurately determining contact, switching of the preceding vehicle to be followed, etc.

Furthermore, regarding the speed of the vehicle, the movement of the characteristic points within this contour shape is searched for on the image to further improve the accuracy (see FIG. 17). This is especially necessary when there is a difference in resolution between the parallax image and the original image. Image information processed for post-processing, such as parallax images, often has a lower resolution than the original image. For this reason, accuracy is high if the original image with a high resolution is used for the speed calculation. For this reason, the original image is used, and the relative speed is calculated by calculating the flow within the extracted contour shape, and by taking into account the contour shape, the speed in the depth direction using parallax and the speed in the lateral direction are calculated simultaneously. Furthermore, the speed of stationary objects in the background is calculated in the same way, and the moving speed of the cutting-in vehicle V2 relative to the stationary objects (the absolute speed is also calculated).
In step S8, the cutting-in vehicle determination unit 8 uses the relative position, speed, absolute position, speed, white line position, vehicle behavior, etc. of the cutting-in vehicle candidate to determine whether the cutting-in vehicle V2 will affect the driving of the host vehicle V0. If there is a possibility of contact, etc., depending on the angle of contact and taking into account the contour shapes of each vehicle, it determines whether the host vehicle V0 should apply emergency braking, or whether it should avoid the vehicle within the lane by steering, or whether it should consider avoidance candidates including the shoulder of the road. Also, if the host vehicle is following the preceding vehicle, it determines when to switch the target vehicle to follow, even if there is no risk of contact or collision.

In step S9, the contact tracing determination unit 5 uses the preceding vehicle identification information, the vehicle's behavior, and information on the vehicle determined to be the cutting-in vehicle V2 to determine a collision or contact with the vehicle V0. By using the contour information of the vehicle V0 and the contour information of the other vehicle while taking into account the vehicle's behavior, it is possible to accurately determine a contact accident, including not only a head-on collision, but also contact, collisions with the vehicle corners and sides, etc. It is also possible to suppress false alarms and erroneous control.

In addition, even if the vehicle V0 travels close enough to come into contact with the preceding vehicle V1 and switches to the new preceding vehicle V1, the timing of switching to the preceding vehicle V1 is appropriately adjusted so that the automatic following driving of the vehicle V0 itself does not become a speed that makes the driver feel uneasy. By switching the preceding following vehicle to the cutting-in vehicle V2 at a safe timing, efforts are made to ensure a safe vehicle distance. Even if the driving of the other vehicle itself is like that, the vehicle V0 moves as quickly and accurately as possible to a safe relative distance and position, making it possible to provide safety and security to the driver. By performing a collision and tracking judgment that uses the outline shapes, which is simple but takes advantage of the right-angle shapes of the vehicle V0 and the other vehicle, it is possible to stably control the vehicle V0 appropriately against the cutting-in vehicle V2 that passes close to the side of the vehicle V0, which was previously difficult.

In step S10, the vehicle control unit 9 issues an alarm, performs control, or switches the target vehicle to be followed based on the judgment result of step S9.

As described above, according to the external environment recognition device of this embodiment, when a vehicle cuts in between the preceding vehicle and the vehicle itself, the vehicle can be recognized as a vehicle based on the side information of the vehicle. Also, if there is a possibility of contact with the vehicle, the vehicle itself is braked, whereas if there is no possibility of contact, the vehicle to be followed in the preceding vehicle following control is switched from the previous preceding vehicle to the vehicle that cuts in, allowing the vehicle to continue driving stably.

100...external environment recognition device, 1...stereo camera unit, 1L...left camera, 1R...right camera, 2...parallax voting unit, 3...back surface extraction unit, 4...preceding vehicle identification unit, 41...tracking unit, 42...vehicle back surface candidate selection unit, 43...vehicle identification unit, 44...position and speed estimation unit, 5...contact tracing determination unit, 51...collision determination unit, 52...preceding vehicle tracking unit, 53...tracking target change unit, 54...integrated determination unit, 6...side surface extraction unit, 61...side surface candidate line search unit, 62...vehicle candidate determination unit, 63...tracking unit, 64... Side speed analysis unit, 7...right angle shape fitting unit, 71...side leading edge fitting unit, 72...side trailing edge fitting unit, 73...side-based rear search unit, 74...detection base line switching unit, 75...rear-based side search unit, 76...detailed shape fitting unit, 77...feature point tracking/speed estimation unit, 78...vehicle identification unit, 8...cutting in vehicle determination unit, 81...white line information unit, 82...same lane merging determination unit, 83...congested complex environment determination unit, 84...preceding vehicle candidate determination unit, 85...contact possibility determination unit, 9...vehicle control unit, V0...own vehicle, V1...preceding vehicle, V2...cutting in vehicle

Claims (10)

Based on the parallax between left and right images of a part of a vehicle approaching the own vehicle,
a side surface extraction unit for extracting side surface information from the image;
a rear surface search unit that searches for a rear surface position of the other vehicle based on the side surface information;
Based on the side information and the rear position, a rectangular object that is close to the other vehicle in the image is
a shape fitting unit that fits a body shape;
Using the rectangular parallelepiped shape, the vehicle is recognized as a vehicle ahead of the host vehicle.
In an external environment recognition device having a recognition unit ,
The other vehicle recognition unit,
Based on an estimated value of a relative posture between the forward vehicle and the host vehicle, or a length of a side and a rear of the forward vehicle in the image,
An external environment recognition device characterized by switching between searching for a side based on the side of the vehicle in front and searching for a side based on the rear of the vehicle in front. Based on the parallax between left and right images of a part of a vehicle approaching the own vehicle,
a side surface extraction unit for extracting side surface information from the image;
a rear surface search unit that searches for a rear surface position of the other vehicle based on the side surface information;
Based on the side information and the rear position, a rectangular object that is close to the other vehicle in the image is
a shape fitting unit that fits a body shape;
Using the rectangular parallelepiped shape, the vehicle is recognized as a vehicle ahead of the host vehicle.
In an external environment recognition device having a recognition unit ,
The side information includes a position and a posture of the side of the other vehicle, and
is the length in the depth direction,
The side extraction unit extracts side information of the other vehicle by voting the disparity of the left and right images in a disparity voting space having two axes, a disparity value and a horizontal image coordinate. Based on the parallax between left and right images of a part of a vehicle approaching the own vehicle,
a side surface extraction unit for extracting side surface information from the image;
a rear surface search unit that searches for a rear surface position of the other vehicle based on the side surface information;
Based on the side information and the rear position, a rectangular object that is close to the other vehicle in the image is
A shape fitting unit that fits a body shape;
Using the rectangular parallelepiped shape, the vehicle is recognized as a vehicle ahead of the host vehicle.
In an external environment recognition device having a recognition unit ,
The side information includes a position and a posture of the side of the other vehicle, and
is the length in the depth direction,
The side surface extraction unit is
By analyzing the image laterally and identifying locations of different textures, or
The disparity between the left and right images is analyzed in the horizontal direction, and a position where the disparity value changes suddenly is identified.
An external environment recognition device characterized by extracting the position of the side tip of the vehicle ahead with high accuracy. Based on the parallax between left and right images of a part of a vehicle approaching the own vehicle,
a side surface extraction unit for extracting side surface information from the image;
a rear surface search unit that searches for a rear surface position of the other vehicle based on the side surface information;
Based on the side information and the rear position, a rectangular object that is close to the other vehicle in the image is
a shape fitting unit that fits a body shape;
Using the rectangular parallelepiped shape, the vehicle is recognized as a vehicle ahead of the host vehicle.
In an external environment recognition device having a recognition unit ,
The side information includes a position and a posture of the side of the other vehicle, and
is the length in the depth direction,
The side surface extraction unit is
On the assumption that the rear surface of the vehicle in front is perpendicular to the side surface of the vehicle in front, the parallax of the rear surface is voted toward the side surface, and the position with the most votes is identified as the rear end of the side surface.
An external environment recognition device characterized by extracting the position of the rear end of the side of the vehicle ahead with high accuracy. The external environment recognition device according to claim 2 ,
The side surface extraction unit is
On the assumption that the rear surface of the vehicle in front is perpendicular to the side surface of the vehicle in front, in the disparity voting space,
An external environment recognition device that searches for disparity between rear surface positions arranged in a straight line and extracts the line with the most accumulated votes as a rear surface candidate line. Based on the parallax between left and right images of a part of a vehicle approaching the own vehicle,
a side surface extraction unit for extracting side surface information from the image;
a rear surface search unit that searches for a rear surface position of the other vehicle based on the side surface information;
Based on the side information and the rear position, a rectangular object that is close to the other vehicle in the image is
a shape fitting unit that fits a body shape;
Using the rectangular parallelepiped shape, the vehicle is recognized as a vehicle ahead of the host vehicle.
In an external environment recognition device having a recognition unit ,
The other vehicle recognition unit accurately calculates the rear angle of the vehicle ahead by calculating the intersection of a straight line perpendicular to the normal of the side of the vehicle ahead and a straight line perpendicular to the normal of the rear of the vehicle ahead. Based on the parallax between left and right images of a part of a vehicle approaching the own vehicle,
a side surface extraction unit for extracting side surface information from the image;
a rear surface search unit that searches for a rear surface position of the other vehicle based on the side surface information;
Based on the side information and the rear position, a rectangular object that is close to the other vehicle in the image is
a shape fitting unit that fits a body shape;
Using the rectangular parallelepiped shape, the vehicle is recognized as a vehicle ahead of the host vehicle.
In an external environment recognition device having a recognition unit ,
The other vehicle recognition unit uses a plurality of frames of the image,
Calculating a relative velocity vector of the forward vehicle by tracking the forward vehicle within the rectangular parallelepiped shape;
An external environment recognition device characterized in that an absolute speed of the vehicle ahead is estimated by calculating a speed vector of a stationary object outside the rectangular parallelepiped shape. An external environment recognition method for recognizing a vehicle that cuts in ahead of a vehicle based on three-dimensional information of a front area of the vehicle acquired from an on-vehicle front sensor during a preceding vehicle following control, comprising:
a voting step of voting the depth distance indicated by the three-dimensional information in a voting space having two axes of the depth distance and the image abscissa;
a line segment searching step of searching, within the voting space, for a line segment within a predetermined angle with respect to a path of the host vehicle on the overhead view;
a side surface extraction step of extracting a side surface of the cutting-in vehicle from the line segment within the predetermined angle that has been searched for;
a back surface extraction step of extracting a back surface of the cutting-in vehicle based on the extracted side surface;
a cutting-in vehicle recognition step of recognizing the cutting-in vehicle based on the extracted side and rear faces;
An external environment recognition method comprising: The method for recognizing an external environment according to claim 8 ,
The side surface extraction step includes extracting, from among the line segments within the predetermined angle, a line segment that is closest to the host vehicle as the side surface of the cutting-in vehicle. The method for recognizing an external environment according to claim 8 ,
a contact determination step for determining a possibility of contact with the cutting-in vehicle;
a braking command step of commanding the host vehicle to brake if there is a possibility of contact;
a step of changing the vehicle to be followed if there is no possibility of braking;
The external environment recognition method further comprises:

Images