JP7356372B2 - Three-dimensional object detection device, in-vehicle system, and three-dimensional object detection method - Google Patents

Description

The present invention relates to a three-dimensional object detection device, a vehicle-mounted system, and a three-dimensional object detection method.

BACKGROUND ART There is a known technique for detecting three-dimensional objects such as other vehicles around a vehicle based on differences between overhead images (also called bird's-eye images) at different timings (see, for example, Patent Document 1 and Patent Document 2). Such technology is applied to a parking support system, etc., which detects three-dimensional objects such as other nearby vehicles as obstacles when a vehicle is parked, and issues a warning, as in Patent Document 1.

Japanese Patent Application Publication No. 2008-227646 International Publication No. 2014/017521

Shadows of structures such as buildings, signs, and traffic lights often appear on road surfaces, and these shadows are observed as they move relative to each other from a moving vehicle. When such a shadow exists between a moving vehicle and a surrounding three-dimensional object, there is a problem in that the three-dimensional object detection accuracy deteriorates.

An object of the present invention is to provide a three-dimensional object detection device, an in-vehicle system, and a three-dimensional object detection method that can improve the detection accuracy of three-dimensional objects existing around a running vehicle.

The present invention provides an overhead view conversion processing unit that converts a first photographed image and a second photographed image taken by a camera at different timings while the vehicle is running into a first bird's-eye view image and a second bird's-eye view image; a difference image generation unit that generates a difference image between the first bird's-eye view image and the second bird's-eye view image in which the photographing positions of the images are aligned, and masking areas other than the three-dimensional object candidate area in which the three-dimensional object is reflected in the difference image. a mask difference image generation unit that generates a mask image and masks the difference image with the mask image to generate a mask difference image, and identifies a ground line in the vicinity of the three-dimensional object in the difference image based on the mask difference image. a nearby ground line specifying unit that specifies the width of the three-dimensional object; a width specifying unit that specifies the width of the three-dimensional object; and a distant ground line of the three-dimensional object in the difference image based on the width of the three-dimensional object and the nearby ground line. and a position specifying section that specifies the position of the three-dimensional object in the difference image based on the nearby ground line and the far ground line, and the width specifying section is configured to detect the mask difference. An end point of the three-dimensional object is determined based on an image, and an end point of the three-dimensional object is determined based on the farthest non-masking area boundary, which is a boundary of a non-masking area located farthest from the imaging position, in the mask image, and the mask difference image. identifying an intersection with a straight line extending in the width direction of the three-dimensional object through an end point of the three-dimensional object, and specifying the width of the three-dimensional object based on a distance between the end point and the intersection point. do.

According to the present invention, it is possible to improve the detection accuracy of three-dimensional objects existing around a running vehicle.

1 is a diagram showing the configuration of an in-vehicle system according to an embodiment of the present invention. It is a figure showing the functional composition of camera ECU. It is a figure showing the positional relationship of the vehicle and other vehicles in the same embodiment. It is a flowchart of three-dimensional object detection processing. 3A and 3B are diagrams for explaining a difference image generation operation, in which (A) shows an example of a first bird's-eye view image and a second bird's-eye view image, and (B) shows an example of a difference image. It is a flowchart of mask image generation processing. FIG. 3 is a diagram for explaining the collapse of a vertical contour line, in which (A) shows an example of a photographed image, and (B) shows an example of an overhead image. FIG. 3 is a diagram schematically showing a label image. FIG. 3 is a diagram schematically showing a lookup table. 2A and 2B are diagrams for explaining generation of a difference histogram, in which (A) shows an example of a difference image, and (B) shows an example of a difference histogram. 2A and 2B are diagrams for explaining generation of an edge strength histogram, in which (A) shows an example of an edge image, and (B) shows an example of an edge strength histogram Rb. 2A and 2B are diagrams for explaining generation of a mask image, in which (A) shows an example of a label image, and (B) shows an example of a mask image. FIG. 6 is an explanatory diagram of the operation of generating a mask difference image. It is a flowchart of nearby ground line identification processing. FIG. 3 is a diagram for explaining generation of a mask difference histogram, in which (A) shows an example of a mask difference image, and (B) shows an example of a mask difference histogram obtained from the mask difference image in (A). FIG. 3 is an explanatory diagram of a nearby ground line specifying operation. It is a flowchart of vehicle width identification processing. FIG. 3 is an explanatory diagram of various parameters set in a mask image in vehicle width identification processing. FIG. 2 is an explanatory diagram of an operation for specifying an end point of another vehicle based on a differential mask image, in which (A) shows an example of a differential mask image, (B) shows an example of a differential mask image for end point identification, and (C) shows an example of an end point identification operation. An example of the identification difference histogram is shown, and (D) shows an example of the end point of another vehicle in the difference mask image. FIG. 7 is a diagram illustrating examples of camera images, overhead images, and mask images for a plurality of other vehicles with different vehicle widths. FIG. 3 is an explanatory diagram of vehicle width specifying conditions. FIG. 3 is a diagram showing the relationship between a nearby ground line, a far ground line, and a vertical contour line in an overhead image. FIG. 3 is a schematic explanatory diagram of three-dimensional object region specifying processing. It is a flowchart of three-dimensional object area identification processing. 2A and 2B are diagrams for explaining generation of a difference neighborhood histogram, in which (A) shows an example of a difference image, and (B) shows an example of a difference neighborhood histogram. FIG. 3 is a diagram for explaining generation of an edge strength neighborhood histogram, in which (A) shows an example of an edge image, and (B) shows an example of an edge intensity neighborhood histogram Rbn. FIG. 3 is a diagram for explaining generation of a differential far-field histogram, in which (A) shows an example of a differential image, and (B) shows an example of a differential far-field histogram. FIG. 3 is a diagram for explaining generation of an edge strength far histogram, in which (A) shows an example of an edge image, and (B) shows an example of an edge strength far histogram. FIG. 3 is an explanatory diagram of a nearby mask label image and a far mask label image. FIG. 7 is an explanatory diagram of another vehicle region specifying operation when three-dimensional object detection is performed limited to a nearby region. FIG. 7 is an explanatory diagram of a grouping operation of each intersection point in a neighboring ground line. FIG. 6 is an explanatory diagram of the final determination operation of a single other vehicle area.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a diagram showing the configuration of an in-vehicle system 1 according to this embodiment.
The in-vehicle system 1 is a system mounted on a vehicle 2, and includes an imaging section 3, a camera ECU 6, a vehicle control unit 8, an HMI section 9, and a CAN 10. The CAN 10 is a type of in-vehicle network that connects the camera ECU 6, the vehicle control unit 8, and the HMI section 9.

The photographing unit 3 outputs a camera image 5 obtained by photographing the entire circumference (360 degree range) of the vehicle 2 to the camera ECU 6. The photographing unit 3 includes a front camera that photographs the front of the vehicle 2, a rear camera that photographs the rear DB (Fig. 3), a right side camera that photographs the right side, and a left side camera that photographs the left side. A plurality of cameras 4 are provided, and a camera image 5 is output from each camera 4. In this embodiment, it is assumed that the vehicle 2 is traveling forward, and the forward direction of the vehicle 2 and the traveling direction B match.

The camera ECU 6 is a device that has a function of controlling the photographing operation of the photographing section 3 and a function of performing appropriate signal processing on each camera image 5. The camera ECU 6 of this embodiment detects the position of another vehicle A (FIG. 3), which is a three-dimensional object running parallel to the vehicle 2, for each camera image 5 of each camera 4, and transmits the position information of the other vehicle A through the CAN 10. It also functions as an other vehicle detection device (three-dimensional object detection device) that transmits data to the vehicle control unit 8. Details of the camera ECU 6 will be described later.

The vehicle control unit 8 is a unit equipped with a processor that executes various controls related to the running of the vehicle 2, and has a function of controlling various parts of the vehicle 2 such as a steering mechanism and a drive mechanism for this control. The vehicle control unit 8 also includes one or more sensors that detect various types of vehicle information (at least traveling speed) required for these controls.

The vehicle control unit 8 also has a function to control the driving of the vehicle 2 in place of the driver's driving operation (so-called automatic driving control function), and when the vehicle 2 is running, it can detect the position of three-dimensional objects existing around the vehicle 2. are acquired sequentially, and the driving is controlled to ensure an appropriate distance to the three-dimensional object. For example, when the vehicle 2 may approach another vehicle A, which is an example of a three-dimensional object, such as when changing lanes, merging, or dividing, the vehicle control unit 8 acquires the position of the other vehicle A, and Based on the position of the other vehicle A, driving is controlled to ensure an appropriate inter-vehicle distance between the other vehicle A and the other vehicle A.

Note that the vehicle control unit 8 may include a driving support function that supports the driver's driving operation in place of the automatic driving control function or in combination with the automatic driving control function. The driving support function sequentially acquires the positions of three-dimensional objects that exist around the vehicle 2 while the vehicle 2 is running, and guides the driver to the three-dimensional objects and issues various warnings based on the position of the three-dimensional objects. It supports the driver's driving operations by notifying the driver.

The HMI unit 9 includes an input device and an output device that serve as a user interface, and the output device includes a display device 9A that displays various information and a speaker that outputs various sounds. The display device 9A displays the camera images 5 of each camera 4, an overhead view of the vehicle 2, etc., and also displays information regarding the other vehicle A detected by the camera ECU 6 (the position and size of the other vehicle A). etc.) are also displayed in an appropriate manner.

The camera ECU 6 described above includes a CPU 12 that is an example of a processor, a memory 14 such as a ROM or RAM that stores various information such as a program 13, and a CANI/F 16 as a communication circuit module that communicates with the vehicle control unit 8 via a CAN 10. It is equipped with a so-called computer. The camera ECU 6 of this embodiment functions as the above-mentioned other vehicle detection device when the CPU 12 executes a program stored in the memory 14.

FIG. 2 is a diagram showing the functional configuration of the camera ECU 6. As shown in FIG.
The camera ECU 6 of this embodiment acquires a first photographed image and a second photographed image taken at different timings from a camera video 5 taken with the same camera 4, and uses these first photographed images and second photographed images. The position of the other vehicle A reflected in the camera image 5 of the camera 4 is detected based on the difference image G that is the difference between the first bird's-eye view image F1 and the second bird's-eye view image F2 obtained by bird's-eye conversion. The camera ECU 6 detects the position of the other vehicle A around the entire circumference of the vehicle 2 by performing such detection for each camera image 5 of each camera 4.
For such a detection operation, the camera ECU 6 includes a vehicle information acquisition section 20, a preprocessing section 22, and a three-dimensional object position specifying section 24, as shown in FIG.

The vehicle information acquisition section 20 acquires vehicle information from the vehicle control unit 8. This vehicle information includes at least the traveling speed of the vehicle 2.

The preprocessing unit 22 executes processing to obtain a difference image G from the camera image 5, and includes a camera image acquisition unit 30, a brightness conversion processing unit 32, an overhead view conversion processing unit 34, and a difference image generation unit. 36.

The camera image acquisition section 30 controls the image capturing section 3 so that the image capturing section 3 continues to capture images for a predetermined period of time or more, and acquires the camera image 5 obtained by the image capturing.
The brightness conversion processing unit 32 converts each frame (captured image) constituting the camera image 5 into a brightness image. Each frame corresponds to a photographed image that is a still image, and the brightness image is an image obtained by converting each pixel value of this photographed image into the brightness value of that pixel.
The bird's-eye view conversion processing unit 34 converts each brightness image into a bird's-eye view image. The bird's-eye view image is an image viewed directly below from a virtual viewpoint set above the vehicle 2. The bird's-eye view conversion processing unit 34 generates a bird's-eye view image by projectively converting the luminance image (also referred to as viewpoint conversion).
The difference image generation unit 36 generates, from among the overhead images sequentially generated by the overhead conversion processing unit 34, two first photographed images having different photographing timings, and a first overhead image F1 obtained from the second photographed image. and a second bird's-eye view image F2 (see FIG. 5(A)), and a difference image G between the two is generated. The difference image G is created by converting the pixel value (luminance value) of each pixel of the first photographed image E1 into a difference between the pixel value (luminance value) of the corresponding pixel of the second photographed image E2, and This is an image in which the pixel value of each pixel is binarized using a threshold value. When generating the difference image G, the difference image generation unit 36 aligns the photographing positions O of the first bird's-eye view image F1 and the second bird's-eye view image F2 with respect to the first bird's-eye view image F1, and calculates each pixel of both images. A difference image G is generated by calculating the amount of difference between the values.

The three-dimensional object position specifying unit 24 specifies the position of another vehicle A, which is an example of a three-dimensional object, based on the difference image G. It includes a width specifying section 52, a distant ground line specifying section 53, and a position specifying section 54.

The mask difference image generation unit 50 generates a mask image 90 for masking the remaining area excluding the other vehicle candidate area 60 in the difference image G, and masks the difference image G with the mask image 90 to detect other vehicle candidates. A mask difference image Gm (see FIG. 13) in which areas other than the area 60 are masked is generated.
The other vehicle candidate area 60 is an area where there is a high probability that the other vehicle A is shown in the difference image G, and is a candidate area for the other vehicle area H. The other vehicle area H is an area (three-dimensional object area) in which it is determined that the other vehicle A is shown in the difference image G.

The nearby ground line identifying unit 51 identifies the nearby ground line L1 in the difference image G based on the mask difference image Gm.
Here, the grounding line refers to a line where the other vehicle A existing in another lane adjacent to the lane in which the vehicle 2 is traveling touches the ground, and the contour line 63 on both left and right sides of the other vehicle A in a plan view (Fig. Corresponds to 3).
In this embodiment, as shown in FIG. 3, of the left and right contour lines 63 of the other vehicle A, the line closer to the vehicle 2 is referred to as a nearby grounding line L1, and the line farther from the vehicle 2 is referred to as a far grounding line L1. It is called line L2.
By setting these nearby grounding line L1 and distant grounding line L2 in the difference image G, the position of the other vehicle area H in the lateral direction Ch as seen from the vehicle 2 is specified in this difference image G. . Note that the lateral direction Ch refers to a direction perpendicular to the traveling direction B of the vehicle 2.

The vehicle width identifying unit 52 identifies the vehicle width Vw (FIG. 3) of the other vehicle A based on the mask image 90 (FIG. 13).
The far grounding line identification unit 53 identifies the far grounding line L2 in the difference image G based on the nearby grounding line L1 and the vehicle width Vw.

The position specifying unit 54 specifies a single other vehicle area K in the difference image G based on the nearby ground line L1, the distant ground line L2, the front end VF, and the rear end VB of the other vehicle A (FIG. 3). The single other vehicle area K is an area where one (single) other vehicle A occupies (is visible) in the difference image G.
To explain in detail, when a plurality of other vehicles A are traveling in parallel, one area including these plurality of other vehicles A may be specified as an other vehicle area H. The single other vehicle area K corresponds to an area where the other vehicle area H is divided for each other vehicle A, and the position specifying unit 54 ultimately uses the single other vehicle area K instead of the other vehicle area H. be specific.
Further, the position specifying unit 54 specifies the position of the other vehicle A in real space based on the position of the single other vehicle area K in the difference image G, and sequentially transmits the position to the vehicle control unit 8. Any known or well-known appropriate method can be used to convert the position of the single other vehicle area K to the position of the other vehicle A in real space.

Further, the position specifying unit 54 transmits data for displaying information on the position of the other vehicle A in real space to the HMI unit 9, and displays the information on the display device 9A. Such a display mode may include, for example, a mode in which the outline of the single other vehicle area K is displayed superimposed on an image of the vehicle 2 viewed from above. In the present embodiment, the vehicle width Vw of the other vehicle A is not a preset default value but an actual value determined by the vehicle width identifying unit 52 based on the mask image 90. The position (the area occupied by the other vehicle A) can be displayed more accurately.

Next, the operation of detecting another vehicle A by the camera ECU 6 will be explained.
In this operation description, a case will be exemplified in which the camera ECU 6 detects another vehicle A shown in the camera image 5 of the rear camera. Furthermore, in this operation description, as shown in FIG. It is assumed that the vehicle is traveling in the same direction as the vehicle No. 2 (that is, traveling parallel to the vehicle No. 2). Further, it is assumed that any other vehicle A is located further back DB than the vehicle 2 and within the range of the viewing angle α of the rear camera. In addition, in FIG. 3, the area indicated by a rectangular line indicates a detection area 72 in which the presence of another vehicle A is detected in automatic driving control or driving support by the vehicle control unit 8.

FIG. 4 is a flowchart of the three-dimensional object detection process.
The three-dimensional object detection process is continuously and repeatedly executed by the camera ECU 6 for each camera image 5 of each camera 4, at least while the vehicle 2 is traveling, in order to detect the presence of other vehicles A in the surroundings.

In the three-dimensional object detection process, first, the vehicle information acquisition section 20 acquires vehicle information (including at least the traveling speed) (step Sa1), and the camera image acquisition section 30 acquires the camera image 5 (step Sa2).
Next, the brightness conversion processing unit 32 sequentially converts each frame (photographed image) of the camera image 5 into a brightness image (step Sa3), and the bird's-eye view conversion processing unit 34 sequentially converts the bird's-eye view image F based on each brightness image. Generate (step Sa4).

Next, the difference image generation unit 36 generates, from among the bird's-eye images sequentially generated by the bird's-eye view conversion processing unit 34, two first captured images with different shooting timings, and a first bird's-eye view obtained from the second captured image. The image F1 and the second bird's-eye view image F2 are extracted, and a difference image G between the two is generated (step Sa5).

FIG. 5 is a diagram for explaining the difference image generation operation, and FIG. 5(A) shows an example of the first bird's-eye view image F1 and the second bird's-eye view image F2, and FIG. 5(B) shows an example of the difference image G. shows.
Note that in the following description, the most recently photographed image will be referred to as the first photographed image, and the photographed image photographed earlier will be referred to as the second photographed image.

Since the respective shooting positions O of the first photographed image and the second photographed image are shifted in the traveling direction B, the first bird's-eye view image F1 and the second bird's-eye view image F2 are also taken as shown in FIG. 5(A). A shift DE occurs at position O due to movement of vehicle 2. In order to correct the deviation DE of the photographing position O and generate the difference image G, the difference image generation unit 36 changes the photographing position O of both the first bird's-eye image F1 and the second bird's-eye image F2 to either one (Fig. 5(A), a difference image G between the two is generated with the first bird's-eye view image F1) aligned as a reference.

Specifically, the difference image generation unit 36 calculates the traveling distance of the vehicle 2 based on the traveling speed of the vehicle 2 and the time difference ta between the photographing timings of the first photographed image and the second photographed image. Then, the difference image generation unit 36 moves each pixel of either the first bird's-eye view image F1 or the second bird's-eye view image F2 (in FIG. It is shifted along the traveling direction B by the number of pixels corresponding to the distance. As a result, the photographing positions O of the first bird's-eye image F1 and the second bird's-eye image F2 are aligned based on the other one.

When an arbitrary stationary object such as a white line 74, which is a type of road marking, is shown in each of the first bird's-eye view image F1 and the second bird's-eye view image F2, the photographing positions O of both images are aligned, so that the image shown in FIG. ), the positions of the stationary objects are aligned.
On the other hand, when the other vehicle A, which is a moving object, appears in each of the first bird's-eye view image F1 and the second bird's-eye view image F2, the photographing positions O of both are aligned, as shown in FIG. 5(A). As such, a shift occurs in the position of the other vehicle A reflected in each image.
As a result, in the difference image G showing the difference between the two, as shown in FIG. The pixel value (amount of difference) in the area where the other vehicle A is reflected becomes relatively large. Therefore, by appropriately setting the threshold value of the luminance value used for binarizing the difference image G, an image excluding stationary objects can be obtained as the difference image G. Then, in this difference image G, the other vehicle area H in which the other vehicle A is reflected can be specified based on the area of high luminance value.

However, as shown in FIG. The pixel value of the area corresponding to 76 also increases. Therefore, when the position specifying unit 54 simply extracts an area with a large pixel value in the difference image G as the other vehicle area H, the area corresponding to the shadow 76 is included in the other vehicle area H, and the area corresponding to the shadow 76 is included in the other vehicle area H. accuracy becomes worse.

Therefore, in this embodiment, before the position specifying section 54 specifies the other vehicle area H based on the differential image G, the mask differential image generating section 50 first uses the differential image G as shown in FIG. A mask difference image Gm is generated by masking the remaining area excluding the other vehicle candidate area 60, that is, the area corresponding to noise such as the shadow 76 (step Sa6: mask difference image generation process). Then, based on this mask difference image Gm, the nearby grounding line identification unit 51 identifies the aforementioned nearby grounding line L1, which is one of the lines that partition the other vehicle area H (step Sa7: nearby grounding line identification process). . Thereby, the nearby ground line L1 can be specified with high accuracy.

FIG. 6 is a flowchart of the above-described mask image generation process.
In the mask image generation process, first, the mask difference image generation unit 50 generates the mask image 90 (step Sb1). The mask image 90 is an image that masks the masking area 62 in the difference image G. The masking area 62 is the remaining area in the difference image G excluding the other vehicle candidate area 60. The mask difference image generation unit 50 identifies this other vehicle candidate area 60 based on the vertical contour line P of the other vehicle A shown in the difference image G.

FIG. 7 is a diagram for explaining the collapse of the vertical contour line P, in which (A) shows an example of a photographed image M, and (B) shows an example of an overhead image F.
As shown in FIG. 7A, the vertical contour line P includes the contour line 63 of the other vehicle A shown in the photographed image M (frame of the camera image 5) and the body parts of the other vehicle A (such as the door). ), the outline of a pattern drawn on another vehicle A, etc., are lines that extend in the vertical direction (perpendicular to the ground) in real space. Due to the projective transformation (viewpoint transformation) of the photographed image M, the vertical contour line P of the photographed image M changes from the angle of the camera 4 (rear camera in this operation description) in the bird's-eye view image F, as shown in FIG. 7(B). It is converted into radiation Q extending radially from the imaging position O (so-called collapse of the vertical contour line P). In other words, in the bird's-eye view image F, the area (the area indicated by hatching in FIG. 7(B)) defined by the radiation Q that includes this vertical contour line P in its line segment indicates the area where the other vehicle A is present. The area becomes the other vehicle candidate area 60. Note that the direction of the ray Q whose line segment includes the vertical contour line P is also called the direction in which the three-dimensional object collapses due to projective transformation (viewpoint transformation).

In the difference image G that is the difference between the two bird's-eye views F, the vertical contour line P is a line segment of the radiation Q, similarly to the bird's-eye view image F. In the difference image G, each pixel forming the radiation Q including the vertical contour line P has a pixel value (difference amount) larger than other pixels. Based on this, the mask difference image generation unit 50 identifies the other vehicle candidate area 60 based on each pixel value of the difference image G. Furthermore, the mask difference image generation unit 50 efficiently identifies the other vehicle candidate area 60 using the label image 91 and the lookup table 92.

FIG. 8 is a diagram schematically showing the label image 91.
The label image 91 is an image of a plurality of rays Q extending radially at equal intervals from the imaging position O, each of which is identified by a label number, and each ray Q includes the vertical contour line P in the difference image G. It is a candidate for radiation Q. In this embodiment, the label image 91 includes 100 radiations Q with label numbers from "1" to "100".
This label image 91 has the number of pixels corresponding to the difference image G, and as shown in FIG. ) are associated.

FIG. 9 is a diagram schematically showing the lookup table 92.
The lookup table 92 specifies, for each pixel of the label image 91, either a pixel value of "255" corresponding to non-masking (white) or "0" corresponding to masking (black). By setting the pixel value of each pixel of the label image 91 based on the specification of the lookup table 92, a mask image 90 in which each pixel is in a non-masking state (white) or a masking state (black) is obtained. It will be done.

As shown in FIG. 9, in the lookup table 92, the pixel value of the radiation Q is specified for each label number of the radiation Q. This pixel value is determined based on the pixel value for each radiation Q in the difference image G.

In the mask image generation process (FIG. 6: step Sb1), the mask difference image generation unit 50 first sets all pixel values of the label image 91 to "255" (non-masking state) or "0" (masking state). ) and initialize it (step Sb1A).
Next, the mask difference image generation unit 50 creates the lookup table 92 based on the pixel values of each radiation Q in the difference image G (step Sb1B). Specifically, the mask difference image generation unit 50 generates a flag (“0” or “255 ”) to be determined.

FIG. 10 is a diagram for explaining the generation of the difference histogram Ra. FIG. 10(A) shows an example of the difference image G, and FIG. 10(B) shows the difference image G shown in FIG. 10(A). An example of the obtained difference histogram Ra is shown.
As shown in FIG. 10(B), the difference histogram Ra has the label number as the horizontal axis, and the cumulative value of the presence or absence of pixel values for each radiation Q in the difference image G shown in FIG. This is a graph in which the vertical axis is the directional difference amount cumulative value. The radial direction difference amount cumulative value increases when the ray Q includes the vertical contour line P, so in this difference histogram Ra, each ray whose radial direction difference amount cumulative value exceeds the predetermined first threshold Th1 By specifying Q, the range Ua of the radiation Q that becomes the other vehicle candidate area 60 can be specified.
In addition, by specifying the radiation Q including the vertical contour line P based on the cumulative value of the difference amount in the radiation direction for each radiation Q, image processing such as contour extraction processing is performed on the difference image G, and the vertical contour line P is Compared to methods that detect radiation Q, radiation Q can be identified faster and with higher precision.

FIG. 11 is a diagram for explaining the generation of the edge strength histogram Rb, FIG. 11(A) shows an example of the edge image E, and FIG. 11(B) is obtained from the edge image E of FIG. 11(A). An example of an edge strength histogram Rb is shown.
As shown in FIG. 11(B), the edge strength histogram Rb has the label number as the horizontal axis, and is the cumulative value of the presence or absence of pixel values for each radiation Q in the edge image E shown in FIG. 11(A) (hereinafter referred to as " This is a graph in which the vertical axis is the cumulative value of edge intensity in the radial direction.
The edge image E is the bird's-eye view image of the first bird's-eye image F1 and the second bird's-eye image F2, which is captured at a later timing (i.e., the most recent one) (the first bird's-eye image F1 in this embodiment). This is an image in which contour components of an object (including the pattern of the object, etc.) reflected in the image are extracted. In the edge image E, the mask difference image generation unit 50 converts the pixel value of each pixel in the bird's-eye view image that has a large brightness difference (greater than a predetermined value) from surrounding pixels to a value ( intensity value). Therefore, the edge strength histogram Rb is a graph showing the magnitude of the edge component of the three-dimensional object included in the radiation Q for each label of the radiation Q.

When creating the lookup table 92 in step Sb1B of FIG. A radiation Q whose radiation direction edge strength cumulative value exceeds a predetermined second threshold Th2 is identified in the histogram Rb. Then, the mask difference image generation unit 50 sets pixel values of "non-masking state" to these radiations Q in the lookup table 92, and sets pixel values of "masking state" to the other radiations Q.
Next, the mask difference image generation unit 50 generates the mask image 90 by setting each pixel value of the label image 91 based on the lookup table 92 (step Sb1C).

FIG. 12 is a diagram for explaining the generation of the mask image 90. FIG. 12(A) shows an example of the label image 91, and FIG. 12(B) shows an example of the mask image 90.
By applying the lookup table 92 to the label image 91 shown in FIG. 12(A), as shown in FIG. 12(B), the area corresponding to the other vehicle candidate area 60 becomes a non-masking area in a non-masking state. 64, and a mask image 90 is obtained in which the area other than the non-masking area 64 is the masking area 62.

Then, in the subsequent step Sb2, the mask difference image generation unit 50 superimposes the mask image 90 on the difference image G, thereby generating a mask difference in which areas other than the other vehicle candidate area 60 are masked, as shown in FIG. Generate image Gm.

Then, when the mask difference image Gm is generated, as described above, the nearby ground line identification unit 51 identifies the nearby ground line L1 based on this mask difference image Gm (FIG. 4: Step Sa7: Nearby ground line identification process).

FIG. 14 is a flowchart of the nearby ground line identification process.
In order to find the nearby ground line L1, the nearby ground line specifying unit 51 first generates a mask difference histogram Rc (step Sc1).

FIG. 15 is a diagram for explaining the generation of the mask difference histogram Rc, FIG. 15(A) shows an example of the mask difference image Gm, and FIG. 15(B) is a diagram obtained from the mask difference image Gm of FIG. 15(A). An example of the obtained mask difference histogram Rc is shown.
As shown in FIG. 15(B), the mask difference histogram Rc has the position in the lateral direction Ch perpendicular to the traveling direction B of the vehicle 2 (hereinafter referred to as "lateral position") as the horizontal axis. The horizontal direction of the mask difference image Gm shown in FIG. This is a graph whose vertical axis is the cumulative value. In the mask difference image Gm, since areas other than the other vehicle candidate area 60 are masked, the nearby grounding line L1 of the other vehicle A can be specified based on the distribution of the running direction difference amount cumulative value in the lateral direction Ch.

Specifically, as shown in FIG. 14, the nearby grounding line specifying unit 51 sets a third threshold Th3 of the cumulative running direction difference amount at which it is assumed that another vehicle A exists at the lateral position ( Step Sc2). The third threshold Th3 is set to an intermediate value (=(Ave+Min)/2) between the average value Ave of the cumulative running direction difference amount and the minimum value Min of the cumulative running direction difference amount.

Next, the nearby ground line identification unit 51 determines, based on the range Uc of lateral positions in which the running direction difference amount cumulative value exceeds the third threshold Th3 for a predetermined number of times or more in the mask difference histogram Rc. Identify the nearby ground line L1.
Specifically, as shown in FIG. 16, the nearby ground line identification unit 51 sets determination points X at equal intervals at i locations (i is an integer greater than or equal to 1) on the horizontal axis of the mask difference histogram Rc. Each determination point X may correspond to an interval (column of the graph) on the horizontal axis of the mask difference histogram Rc.
Then, as shown in FIG. 14, the nearby ground line identification unit 51 sequentially determines whether or not a predetermined nearby ground line determination condition is satisfied starting from the determination point X closest to the imaging position O (step Sc3). If not satisfied (step Sc3: No), the next determination point X is determined (step Sc4). If the nearby ground line determination condition is satisfied (step Sc3: Yes), the nearby ground line identification unit 51 identifies the determination point X as the position of the nearby ground line L1 (step Sc5).

The above-mentioned nearby grounding line judgment condition is that the running direction difference amount cumulative value of the judgment point The condition is that the value is greater than or equal to the third threshold Th3.
By determining the nearby grounding line determination conditions in order from the determination point X closest to the photographing position O, as shown in FIG. 16, the cumulative value of the running direction difference amount of all the determination points For the range Uc exceeding the range Uc, the immediately preceding determination point X as seen from the photographing position O is determined, and this determination point X is specified as the nearby ground line L1. As a result, the nearby ground line L1 is not set at a position where the other vehicle A enters (in a range exceeding the third threshold value Th3), and the nearby ground line L1 is set at a more accurate position.

In this way, the position of the nearby ground line L1 in the differential image G is specified based not on the differential image G but on the masked differential image Gm in which noise such as the shadow 76 has been masked, so the identified position is extremely will be accurate.

When the nearby ground line L1 is specified, the vehicle width specifying unit 52 executes a vehicle width specifying process as shown in FIG. 4 (step Sa8). In this vehicle width identification process, the vehicle width identification unit 52 identifies the vehicle width Vw of the other vehicle A based on the mask image 90.

FIG. 17 is a flowchart of the vehicle width identification process, and FIG. 18 is an explanatory diagram of various parameters set on the mask image 90 in the vehicle width identification process.
In the vehicle width identification process, the vehicle width identification unit 52 first determines whether the position of the other vehicle A is within the vehicle width identification condition range (step Se1). The vehicle width specifying condition is a condition indicating whether the vehicle width Vw of the other vehicle A shown in the difference image G can be specified from the mask image 90. Note that this vehicle width specifying condition will be described later.

If the position of the other vehicle A is not within the vehicle width specifying condition range (step Se1: No), the vehicle width specifying unit 52 ends the process, and if the position of the other vehicle A is within the vehicle width specifying condition range ( Step Se1: Yes), in order to identify the vehicle width Vw of the other vehicle A, the following process is executed.

That is, the vehicle width specifying unit 52 specifies the farthest non-masking area boundary Nf of the non-masking area boundaries N in the mask image 90 (step Se2).
The non-masking area boundary N is the boundary between the non-masking area 64 and the masking area 62, and corresponds to the edge of the non-masking area 64. The non-masking area 64 corresponds to an area other than the masking area 62 in the difference image G, that is, an other vehicle candidate area 60 that is a candidate in which the other vehicle A is shown. The farthest non-masking area boundary Nf is the farthest non-masking area boundary N in the horizontal direction Ch when viewed from the imaging position O in the mask image 90.

When parts on the side of the vehicle body, outlines of patterns, or the shadow of the vehicle 2 are reflected in the camera image 5, the mask image 90 includes one or more ( In the illustrated example, one masking area 62A is generated. This masking area 62A occurs within the non-masking area 64 for one vehicle (corresponding to the other vehicle candidate area 60 in the difference image G), so normally one non-masking area 64 would be multiple (two in the illustrated example). ). Therefore, specifying the farthest non-masking area boundary Nf means specifying the non-masking area boundary N of the non-masking area 64 divided by the masking area 62A.

In step Se2, the vehicle width specifying unit 52 specifies the farthest non-masking area boundary Nf in the following manner.
That is, as shown in FIG. 18, the vehicle width specifying unit 52 sets a plurality of detection points Dt on the first horizontal line Lc1 extending linearly from the photographing position O in the horizontal direction Ch. Next, the vehicle width specifying unit 52 scans in a far direction Cvf (corresponding to the rear DB of the vehicle 2 in this embodiment) that moves away from the photographing position O along the longitudinal direction from each detection point Dt, and detects the farthest non-masking area boundary. Identify point Nfd. The farthest non-masking area boundary point Nfd is a point that constitutes the farthest non-masking area boundary Nf, and is specified by the point that changes from the masking area 62 to the non-masking area 64 during the scanning process by the vehicle width specifying unit 52. .
Next, the vehicle width specifying unit 52 obtains an approximate straight line Lapp from the farthest non-masking area boundary point Nfd at each detection point Dt using an appropriate linear approximation method. This approximate straight line Lapp becomes the farthest non-masking area boundary Nf.

After identifying the non-masking area boundary Nf using the approximate straight line Lapp, the vehicle width identifying unit 52 identifies the position of the end point V of the other vehicle A in the mask image 90, as shown in FIG. 17 (step Se3). The end point V of the other vehicle A is the front end VF or the rear end VB of the other vehicle A shown in the camera image 5. In this operation example, since the other vehicle A is reflected in the camera image 5 of the rear camera, the leading end VF of the other vehicle A is reflected in the camera image 5 and the difference image G obtained from the camera image 5. Therefore, in step Se3, the position of the tip VF is specified as the end point V of the other vehicle A, as shown in FIG.

In this embodiment, the vehicle width identifying unit 52 identifies the position of the end point V of the other vehicle A in the mask image 90 based on the mask difference image Gm.
Specifically, as shown in FIG. 19A, the vehicle width identification unit 52 superimposes the nearby ground line L1 on the mask difference image Gm, and in the mask difference image Gm, the vehicle 2 By further masking the region closer to the photographing position O, a mask difference image Gmt for end point identification shown in FIG. 19(B) is generated.
Next, as shown in FIG. 19C, the vehicle width identifying unit 52 generates an end point identifying difference histogram Rgmt from the end point identifying mask difference image Gmt. The end point identification difference histogram Rgmt divides the end point identification mask difference image Gmt in the vertical direction (in this embodiment, the direction of the rear DB of the vehicle 2) into strip-shaped small regions at predetermined intervals, and performs end point identification for each region. It is a graph in which the presence or absence of pixel values is accumulated in the horizontal direction Ch of the mask difference image Gmt. Hereinafter, this cumulative value of pixel values will be referred to as a horizontal difference cumulative value.

In the end point identification mask difference image Gmt, in addition to the area other than the other vehicle candidate area 60, the area closer to the photographing position O of the vehicle 2 than the nearby grounding line L1 is masked, so the lateral direction difference amount cumulative value Based on the distribution of , it is possible to specify the position of the end of the other vehicle A that is closer to the photographing position O of the vehicle 2 (in this embodiment, the tip VF of the other vehicle A).

Specifically, the vehicle width identifying unit 52 uses the same method as the method in which the nearby grounding line identifying unit 51 identifies the nearby grounding line L1 based on the mask difference histogram Rc to identify other vehicles based on the end point identifying difference histogram Rgmt. Identify the edge near A.
That is, the vehicle width specifying unit 52 scans each section of the end point specifying difference histogram Rgmt in order from the determination point X closest to the photographing position O. Then, as shown in FIG. 19(B), if the cumulative value of the horizontal difference amount in each section in the end point identification difference histogram Rgmt exceeds the end point determination threshold value Thgmt consecutively for a predetermined number of times, the vehicle As shown in FIG. 19C, the width specifying unit 52 specifies the first section of the continuous section (that is, the section closest to the photographing position O among the continuous sections) Kgmt. Then, the vehicle width specifying unit 52 superimposes an end line Lgmt extending in the lateral direction Ch through a position corresponding to the section Pgmt in the mask difference image Gm. This end line Lgmt indicates the position of the end (tip VF) of the other vehicle A in the mask difference image Gm. Therefore, the vehicle width specifying unit 52 specifies the end point V in the mask difference image Gm by finding the intersection between the end line Lgmt and the nearby ground line L1.
Then, the vehicle width specifying unit 52 specifies, in the mask image 90, a position corresponding to the position of the end point V of the other vehicle A in the mask difference image Gm as the position of the end point V in the mask image 90.

Next, as shown in FIG. 18, the vehicle width specifying unit 52 sets a second horizontal line Lc2 that passes through the end point V and extends linearly in the horizontal direction Ch on the mask image 90 (FIG. 17: Step Se4). Then, the vehicle width specifying unit 52 specifies the intersection Vm of the second horizontal line Lc2 and the approximate straight line Lapp (the farthest non-masking area boundary Nf) in the mask image 90 (FIG. 17: Step Se5).

Here, the second horizontal line Lc2 indicates a line extending in the vehicle width direction of the other vehicle A shown in the difference image G, and furthermore, the intersection Vm of the second horizontal line Lc2 and the approximate straight line Lapp is the tip of the other vehicle A. VF or rear end VB is shown.
Therefore, the end point V specified in step Se3 and the intersection point Vm specified in step Se5 correspond to both ends of the leading end side of the other vehicle A in the vehicle width direction. Therefore, the vehicle width identifying unit 52 identifies the vehicle width Vw in the real space by converting the distance between the end point V and the intersection Vm in the mask image 90 into a distance in the real space (step Se6).

In this way, the vehicle width specifying unit 52 specifies the farthest non-masking area boundary Nf in the mask image 90, and uses the farthest non-masking area boundary Nf to specify the vehicle width Vw.
As a result, a masking area 62A is generated in the mask image 90 due to parts on the side of the vehicle body, outlines of patterns, or the shadow of the vehicle 2, and even if one non-masking area 64 is originally divided into multiple parts, The non-masking area boundary N of the original non-masking area 64 is specified by the farthest non-masking area boundary Nf, and the vehicle width Vw is accurately determined.

FIG. 20 is a diagram showing examples of camera images 5, overhead images F, and mask images 90 for a plurality of other vehicles A having different vehicle widths Vw. In addition, in the figure, as examples of other vehicles A, a regular car (regular four-wheeled vehicle), a large bus (large passenger vehicle), and a motorcycle (two-wheeled vehicle) are shown.
As shown in the figure, the vehicle width Vw is appropriately specified for each other vehicle A by the vehicle width specifying process.

By the way, in this vehicle width specifying process, the vehicle width Vw is specified based on the end point V of the front end side or the rear end side of the other vehicle A in the mask image 90, so that the front end side or the rear end side is , and is not reflected in the camera image 5, the vehicle width Vw cannot be specified. In this case, execution of the vehicle width identification process is wasteful.
Therefore, in the present embodiment, the condition that the front end side or the rear end side of the other vehicle A is reflected in the camera image 5 is set in advance as the vehicle width specifying condition, and in step Sa1 at the start of the vehicle width specifying process, By determining whether the width specific condition is satisfied, unnecessary processing is prevented.

The conditions under which the front end side or the rear end side of the other vehicle A is reflected in the camera image 5 can be defined by the range in which the other vehicle A is located within the entire 360-degree circumference around the vehicle 2 . For example, as shown in FIG. 21, the range in which the front end or rear end of the other vehicle A is not visible in the camera image 5 (overhead image F in the example of FIG. 21) is that the other vehicle A is located approximately to the side of the vehicle 2. A predetermined range β is set, and the remaining range excluding this predetermined range β is set as the vehicle width specifying condition.
Then, in step Se1, the vehicle width specifying unit 52 determines whether the other vehicle A is located in the remaining range excluding the predetermined range β, and only when the other vehicle A is located within the range, the vehicle width specifying unit 52 Processing after Se2 is executed. This prevents the vehicle width identification process from being performed in vain. The vehicle width specifying unit 52 may specify the position of the other vehicle A with respect to the vehicle 2, for example, based on the camera images 5 of each camera 4 or the bird's-eye view image F, or may specify the position of the other vehicle A with respect to the vehicle 2 based on a separate object detection method such as sonar. The identification may be made using the detection results of the sensor.

When the vehicle width Vw is specified, as shown in FIG. 4, the far grounding line specifying unit 53 specifies the far grounding line L2 in the difference image G (step Sa9). Specifically, in the difference image G, the far ground line identification unit 53 identifies a line that is spaced apart from the nearby ground line L1 by the vehicle width Vw and parallel to the nearby ground line L1 as the far ground line L2.
Since the position of the far ground line L2 is determined using the vehicle width Vw obtained from the mask image 90, it can be determined more accurately than in the case where the vehicle width Vw is set to a predetermined fixed value.

Next, the position specifying unit 54 specifies the other vehicle area H in the difference image G (step Sa10: three-dimensional object area specifying process).
The other vehicle area H includes the nearby ground line L1 and the distant ground line L2 specified in the nearby ground line identification process (step Sa7) and the distant ground line identification process (step Sa9), and the tip VF of the other vehicle A. , and the rear end VB.

The leading end VF and rear end VB of the other vehicle A can be determined from the intersections of the vertical contour line P with the nearby grounding line L1 and the distant grounding line L2, as described above.
That is, as shown in FIG. 22, in the overhead image F, the intersections of the vertical contour line P1 near the rear end and the vertical contour P2 near the tip of the other vehicle A with the ground line L1, A position L1VB near the rear end and a position L1VF near the tip of the nearby ground line L1 can be determined. Similarly, the rear end far side position on the far ground line L2 is determined by the intersections of the vertical contour line P1 near the rear end of the other vehicle A, the vertical contour line P2 near the tip, and the far ground line L2. L2VB and the tip far side position L2VF can be determined.
Note that the rear end near side position L1VB and the rear end far side position L2VB indicate both ends in the vehicle width direction on the rear end side of the other vehicle A, and the front end near side position L1VF and the front end far side position L2VF indicate the ends of the rear end side of the other vehicle A. Both ends of the front end of A in the vehicle width direction are shown.

However, in the bird's-eye view image F, as mentioned above, the other vehicle A appears to be leaning down in the direction of the radiation Q, so the other vehicle A appears longer in the driving direction B than it actually is due to the influence of the roof portion Ar of the other vehicle A, etc. Region H will be detected, and an error will occur in the position of other vehicle A.

In the three-dimensional object area specifying process of this embodiment, in order to eliminate such errors, the position specifying unit 54 specifies the other vehicle area H in the following manner.
That is, as shown in FIG. 23, the position specifying unit 54 determines the intersection LV with the radiation Q including the vertical contour line P of the other vehicle A for each of the nearby grounding line L1 and the distant grounding line L2. Next, the position specifying unit 54 specifies a provisional first other vehicle area H1 from the intersection LV on the nearby grounding line L1, and also specifies a provisional second other vehicle area H2 from the intersection LV on the distant grounding line L2. Then, the position specifying unit 54 specifies the front end VF and the rear end VB of the other vehicle A based on the area where the provisional first other vehicle area H1 and the provisional second other vehicle area H2 overlap. Then, the other vehicle area H is accurately specified by the nearby ground line L1, the distant ground line L2, the front end VF, and the rear end VB.
Here, if the accuracy of the far grounding line L2 is poor, the provisional first other vehicle area H1 and the provisional second other vehicle area H2 may not overlap, and the other vehicle area H may not be specified. In contrast, in the present embodiment, the distant ground line L2 is accurately identified using the nearby ground line L1 specified based on the mask difference image Gm and the vehicle width Vw specified based on the mask image 90. Therefore, the other vehicle area H can be reliably specified.

The three-dimensional object region specifying process will be described in more detail below.

FIG. 24 is a flowchart of the three-dimensional object region specifying process.
In order to specify the provisional first other vehicle area H1 and the provisional second other vehicle area H2, the position specifying unit 54 first uses the difference neighborhood histogram Ran, the edge strength neighborhood histogram Rbn, the difference far distance histogram Raf, and the edge An intensity far-field histogram Rbf is generated (step Sd1).

FIG. 25 is a diagram for explaining the generation of the difference neighborhood histogram Ran, FIG. 25(A) shows an example of the difference image G, and FIG. 25(B) is obtained from the difference image G in FIG. 25(A). An example of a differential neighborhood histogram Ran is shown. Further, FIG. 26 is a diagram for explaining the generation of the edge strength neighborhood histogram Rbn. FIG. 26(A) shows an example of the edge image E, and FIG. 26(B) An example of the obtained edge strength neighborhood histogram Rbn is shown.
The difference neighborhood histogram Ran is, as shown in FIG. 25B, the difference histogram Ra described above for the neighborhood region Jn in the difference image G shown in FIG. 25A. Further, the edge strength neighborhood histogram Rbn, as shown in FIG. 26(B), is obtained by obtaining the above-mentioned edge strength histogram Rb for the neighborhood region Jn in the edge image E shown in FIG. 26(A).
The nearby area Jn is an area sandwiched between the nearby ground line L1 and the far ground line L2 in the difference image G.

FIG. 27 is a diagram for explaining the generation of the differential far-field histogram Raf, FIG. 27(A) shows an example of the differential image G, and FIG. 27(B) is obtained from the differential image G in FIG. 27(A). An example of a differential far-field histogram Raf is shown. Further, FIG. 28 is a diagram for explaining the generation of the edge strength far histogram Rbf, and FIG. 28(A) shows an example of the edge image E, and FIG. 28(B) is a diagram obtained from the edge image E of FIG. 28(A). An example of the obtained edge strength far histogram Rbf is shown.
The differential far-field histogram Raf, as shown in FIG. 27(B), is obtained from the above-described differential histogram Ra for the far-field area Jf in the differential image G shown in FIG. 27(A). Further, the edge strength far histogram Rbf is obtained by obtaining the above-mentioned edge strength histogram Rb for the far region Jf in the edge image E shown in FIG. 28(A), as shown in FIG. 28(B).
The far area Jf is an area in the difference image G that is further away than the nearby ground line L1 when viewed from the imaging position O.

The neighborhood mask label image 91n is used to generate the differential neighborhood histogram Ran and the edge intensity neighborhood histogram Rbn, and the far mask label image 91f is used to generate the differential distant histogram Raf and the edge intensity far histogram Rbf.

FIG. 29 is an explanatory diagram of the near mask label image 91n and the far mask label image 91f.
The neighborhood mask label image 91n is an image that invalidates pixel values other than the neighborhood area Jn in the difference image G so that they are excluded from the cumulative value count. The neighborhood mask label image 91n is obtained by superimposing a neighborhood mask image 90n that masks areas other than the neighborhood Jn in the difference image G on the label image 91 described above, as shown in FIG.
The position specifying unit 54 obtains the difference histogram Ra and the edge strength histogram Rb using the neighborhood mask label image 91n, so that only the pixel values of the neighborhood area Jn are subject to cumulative value addition, and the difference neighborhood histogram Ran and An edge strength neighborhood histogram Rbn is obtained.

The far-field mask label image 91f is an image in which pixel values other than the far-field area Jf in the differential image G are invalidated and excluded from the cumulative value count, and the far-field mask label image 91f is an image in which pixel values other than the far-field area Jf in the differential image G are masked. It is obtained by superimposing the area mask image 90f on the label image 91.
The position specifying unit 54 obtains the difference histogram Ra and the edge strength histogram Rb using the far-field mask label image 91f, so that only the pixel values in the far-field area Jf are subject to cumulative value addition, and the difference far-field histogram Raf and An edge strength far-field histogram Rbf is determined.

Returning to FIG. 24 above, the position specifying unit 54 next performs three-dimensional object determination in the nearby region Jn based on the difference neighborhood histogram Ran and the edge intensity neighborhood histogram Rbn (step Sd2).
Specifically, the position specifying unit 54 determines that in the differential neighborhood histogram Ran, the radial direction difference amount cumulative value is equal to or greater than the fourth threshold Th4 (FIG. 25), and in the edge strength neighborhood histogram Rbn, the radial direction edge strength cumulative value is the fourth threshold value Th4 (FIG. 25) or more. 5. A radiation Q having a label number equal to or higher than the threshold value Th5 (FIG. 26) is identified.
Then, as shown in FIG. 30, the position specifying unit 54 specifies, in the difference image G, an intersection LV between each of the specified radiation lines Q and the nearby ground line L1. These intersection points LV are determined only for a predetermined detection area set in the difference image G. This detection area is, for example, the detection area 72 (FIGS. 3 and 5).
These intersection points LV specify a provisional first other vehicle area H1 when three-dimensional object detection is performed limited to the nearby area Jn. Note that the vehicle width Vw specified in the vehicle width specifying process (FIG. 4: Step Sa9) is used as the width in the lateral direction Ch of the provisional first other vehicle area H1.

Returning to FIG. 24, the position specifying unit 54 performs three-dimensional object determination in the far area Jf based on the differential far-field histogram Raf and the edge strength far-field histogram Rbf (step Sd3). Thereby, the provisional second other vehicle area H2 in the case where three-dimensional object detection is performed limited to the far area Jf is specified.

Here, as described above, if a plurality of other vehicles A are traveling in parallel, two or more other vehicles A may be included in the provisional first other vehicle area H1 and the provisional second other vehicle area H2. There is sex.
Therefore, the position specifying unit 54 groups each intersection LV on the nearby ground line L1 and each intersection LV on the distant ground line L2 for each other vehicle A as follows (step Sd4, Sd5).

Taking the intersection LV of the nearby ground line L1 as an example, as shown in FIG. If the distance W between the LVs is less than or equal to the sixth predetermined threshold Th6, the two intersection points LV are classified into the same group 97, and if the distance W exceeds the sixth predetermined threshold Th6, the photographing position O The intersection point LV that is further away from is classified into a new group 97. As a result, groups 97 are divided at locations where the interval between the intersection points LV is wider than the sixth threshold Th6, that is, between the two other vehicles A, and the intersection points LV are grouped for each other vehicle A.
Then, the position specifying unit 54 specifies the single other vehicle area K1 for each group 97 using the intersection LV belonging to the group 97, so that the provisional first other vehicle area H1 is determined for each single other vehicle A. It will be divided.

Returning to FIG. 24, the position specifying unit 54 determines the final location based on the single other vehicle area K1 specified for the nearby grounding line L1 and the single other vehicle area K2 specified for the distant grounding line L2. A single other vehicle area K is determined (step Sd6).
That is, as shown in FIG. 32, among the front end VF and rear end VB of the other vehicle A in the single other vehicle area K1 and the single other vehicle area K2, the single other vehicle area K1 and the single other vehicle area The area in which both other vehicle areas K2 overlap is specified as the final tip VF and rear end VB of the single other vehicle area K.
Then, by using these front end VF, rear end VB, near ground line L1, and far ground line L2, errors caused by tilting due to projective transformation are eliminated, and the accurate position of the rectangular single other vehicle area K is determined. It will be specified.

According to the embodiment described above, the following effects are achieved.

The camera ECU 6 (three-dimensional object detection device) of the present embodiment becomes a candidate when another vehicle A is seen in the difference image G of the first bird's-eye view image F1 and the second bird's-eye view image F2 in which the photographing positions O are aligned with each other. A mask difference image Gm is generated in which areas other than the other vehicle candidate area 60 are masked, and the position of the other vehicle A in the difference image G is specified based on this mask difference image Gm.
As a result, compared to the case where the position of the other vehicle A is specified based on the difference image G, even when the shadow 76 exists between the running vehicle 2 and other vehicles A around it, the position of the other vehicle A can be detected. The location can be determined with high accuracy.
Therefore, in the vehicle control unit 8, based on the accurate position of the other vehicle A, more accurate automatic driving control can be realized in scenes where the vehicle approaches another vehicle when changing lanes, merging, or branching.
In addition, the camera ECU 6 determines the end point V of the other vehicle A based on the mask difference image Gm, and determines the end point V of the other vehicle A based on the distance between the end point V of the other vehicle A and the non-masking area boundary N in the mask image 90. A vehicle width specifying section 52 is provided to specify the vehicle width Vw of A. Thereby, the position of the other vehicle A can be specified more accurately than when the position of the other vehicle A is specified using a fixed value appropriately determined in advance as the vehicle width Vw by the camera ECU 6.
Moreover, since the vehicle width specifying unit 52 specifies the vehicle width Vw based on the mask image 90 used for generating the mask difference image Gm, the vehicle width Vw can be efficiently specified using the existing mask image 90.
Furthermore, the vehicle width specifying unit 52 specifies the intersection Vm between the farthest non-masking area boundary Nf and the second horizontal line Lc2 passing through the end point V of the other vehicle A in the mask image 90, and connects the end point V and the intersection Vm. The vehicle width Vw of the other vehicle A is specified based on the distance between the two vehicles. As a result, a masking area 62A (FIG. 18) is generated in the mask image 90 due to parts on the side of the vehicle body, outlines of patterns, or the shadow of the vehicle 2, and a single non-masking area 64 is divided into multiple parts. Even in this case, the non-masking area boundary N of the original non-masking area 64 is specified by the farthest non-masking area boundary Nf, and the vehicle width Vw can be accurately determined.

In the mask image 90, the vehicle width specifying unit 52 of the present embodiment detects the mask from each of a plurality of detection points Dt set on a first horizontal line Lc1 extending linearly from the photographing position O in the vehicle width direction of the other vehicle A. The image 90 is scanned in the far direction Cvf along the vertical direction to identify the farthest non-masking area boundary point Nfd where the masking area 62 changes to the non-masking area 64. Then, the vehicle width specifying unit 52 specifies the farthest non-masking area boundary Nf by obtaining an approximate straight line from each of the farthest non-masking area boundary points Nfd of the detection point Dt using a linear approximation method.
Thereby, the farthest non-masking area boundary Nf can be reliably specified.

The position specifying unit 54 of this embodiment locates the nearby ground line L1 based on the intersection points LV of the radiation Q including the vertical contour line P of the other vehicle A and each of the nearby ground line L1 and the distant ground line L2. A provisional first other vehicle area H1 where the other vehicle A is located and a provisional second other vehicle area H2 where the other vehicle A is located at the far grounding line L2 are specified, and these provisional first other vehicle areas H1 and the provisional second other vehicle area H1 are identified. The other vehicle area H in the difference image G is specified based on the range in which the two other vehicle areas H2 overlap.
Thereby, the influence of collapse due to projective transformation can be eliminated, and the other vehicle area H can be accurately specified.
Furthermore, if the accuracy of the distant grounding line L2 is poor, the provisional first other vehicle area H1 and the provisional second other vehicle area H2 may not overlap, and the other vehicle area H may not be specified. In contrast, in the present embodiment, the distant ground line L2 is accurately identified using the nearby ground line L1 specified based on the mask difference image Gm and the vehicle width Vw specified based on the mask image 90. Therefore, the other vehicle area H can be reliably specified.

The nearby ground line identifying unit 51 of the present embodiment generates a difference image based on the position on the horizontal axis where the running direction difference amount cumulative value exceeds the third threshold Th3 in the mask difference histogram Rc generated based on the mask difference image Gm. The nearby grounding line L1 of the other vehicle A at G is specified.
Thereby, the nearby ground line L1 is accurately specified using the mask difference image Gm.

The embodiment described above is merely an example of one aspect of the present invention, and can be arbitrarily modified and applied without departing from the spirit of the present invention.

In the embodiment described above, the camera ECU 6 functions as a three-dimensional object detection device, but the present invention is not limited to this, and any device provided in the in-vehicle system 1 may function as a three-dimensional object detection device.

In addition, in the embodiment described above, the functional blocks shown in FIG. The components can be further classified into more components depending on the processing content. It is also possible to classify one component so that it performs more processes.

In the embodiments described above, unless otherwise specified, directions such as horizontal and vertical directions and various shapes are referred to as peripheral directions and approximate shapes (so-called equivalent range).

1 In-vehicle system 2 Vehicle 3 Photographing unit 4 Camera 6 Camera ECU (three-dimensional object detection device, computer)
9A Display device 24 Three-dimensional object position identification unit 30 Camera image acquisition unit 34 Overhead view conversion processing unit 36 Difference image generation unit 50 Mask difference image generation unit 51 Nearby grounding line identification unit 52 Vehicle width identification unit (width identification unit)
53 Distant ground line identification unit 54 Position identification unit 60 Other vehicle candidate area 62, 62A Masking area 64 Masking area A Other vehicle (three-dimensional object)
B Traveling direction Ch Lateral direction Cvf Far direction Dt Detection point E Edge image F Overhead image G Difference image Gm Mask difference image H Other vehicle area H1 Temporary first other vehicle area H2 Temporary second other vehicle area L1 Nearby grounding line L2 Distance grounding Line Lapp Approximate straight line M Photographed image N Masking area boundary Nf Masking area boundary Nfd Masking area boundary point O Imaging position P Vertical contour Q Radiation Rc Mask difference histogram Th3 Third threshold V End point Vm Intersection Vw Vehicle width

Claims (6)

an overhead view conversion processing unit that converts each of a first photographed image and a second photographed image taken by a camera at different timings while the vehicle is running into a first bird's-eye view image and a second bird's-eye view image;
a difference image generation unit that generates a difference image between the first bird's-eye view image and the second bird's-eye view image whose shooting positions are aligned with each other;
a mask difference image generation unit that generates a mask image for masking areas other than the candidate three-dimensional object candidate area in which the three-dimensional object is shown in the difference image, and generates a mask difference image by masking the difference image with the mask image;
a nearby ground line identification unit that identifies a nearby ground line of the three-dimensional object in the difference image based on the mask difference image;
a width specifying unit that specifies the width of the three-dimensional object;
a far ground line identification unit that identifies a far ground line of the three-dimensional object in the difference image based on the width of the three-dimensional object and the nearby ground line;
a position specifying unit that specifies the position of the three-dimensional object in the difference image based on the nearby ground line and the distant ground line,
The width specifying section is
determining end points of the three-dimensional object based on the mask difference image;
In the mask image, the three-dimensional object passes through the farthest non-masking area boundary, which is the boundary of the non-masking area located farthest from the imaging position, and the end point of the three-dimensional object determined based on the mask difference image. A three-dimensional object detection device, characterized in that the width of the three-dimensional object is specified based on the distance between the end point and the intersection point. The width specifying section is
In the mask image, scanning from each of a plurality of detection points set on a straight line extending from the photographing position in the width direction of the three-dimensional object in a far direction away from the photographing position along the vertical direction of the mask image, Identify the farthest non-masking area boundary point that changes from the masking area to the non-masking area,
The three-dimensional object according to claim 1, wherein the farthest non-masking area boundary is identified by obtaining an approximate straight line from each of the farthest non-masking area boundary points of the detection points using a first-order approximation method. Detection device. The position specifying unit is
The three-dimensional object is located at the nearby ground line based on the intersection points of the radiation extending from the imaging position and including the vertical contour line of the three-dimensional object in the difference image and each of the nearby ground line and the distant ground line. and a second area where the three-dimensional object is located in the far ground line, and based on the range where these first areas and second areas overlap, identify the three-dimensional object in the difference image. The three-dimensional object detection device according to claim 1 or 2, wherein the three-dimensional object detection device specifies a three-dimensional object area in which a three-dimensional object is reflected. The nearby grounding line identification unit is
In the mask difference histogram, the horizontal axis is the horizontal direction perpendicular to the running direction of the vehicle, and the vertical axis is the running direction difference cumulative value obtained by accumulating the pixel values of each pixel of the mask difference image along the running direction. The three-dimensional object detection device according to any one of claims 1 to 3, wherein the nearby grounding line is specified based on a position on the horizontal axis at which a running direction difference amount cumulative value exceeds a third threshold value. A three-dimensional object detection device according to any one of claims 1 to 4,
a display device that displays the position of the other vehicle specified by the three-dimensional object detection device;
An in-vehicle system characterized by being equipped with. The computer is
a first step of converting each of a first photographed image and a second photographed image taken by a camera at different timings while the vehicle is running into a first bird's-eye view image and a second bird's-eye view image;
a second step of generating a difference image between the first bird's-eye view image and the second bird's-eye view image whose shooting positions are aligned with each other;
a third step of generating a mask image for masking an area other than the candidate three-dimensional object candidate area in which the three-dimensional object is shown in the difference image, and masking the difference image with the mask image to generate a mask difference image;
a fourth step of identifying a ground line near the three-dimensional object in the difference image based on the mask difference image;
a fifth step of specifying the width of the three-dimensional object;
a sixth step of identifying a far ground line of the three-dimensional object in the difference image based on the width of the three-dimensional object and the nearby ground line;
a seventh step of specifying the position of the three-dimensional object in the difference image based on the nearby ground line and the distant ground line,
In the fifth step,
determining end points of the three-dimensional object based on the mask difference image;
In the mask image, the three-dimensional object passes through the farthest non-masking area boundary, which is the boundary of the non-masking area located farthest from the imaging position, and the end point of the three-dimensional object determined based on the mask difference image. A three-dimensional object detection method, comprising: identifying an intersection with a straight line extending in the width direction, and determining the width of the three-dimensional object based on the distance between the end point and the intersection.

Images