JP7561664B2 - OBJECT DETECTION DEVICE, OBJECT DETECTION METHOD, AND OBJECT DETECTION PROGRAM - Google Patents

Description

The present invention relates to an object detection device, an object detection method, and an object detection program for detecting objects around a vehicle's driving path.

Conventionally, for vehicles such as automobiles, there have been considered technologies that detect obstacles around the vehicle, judge the risk of colliding with the obstacle and issue a warning to the driver, show the driver the distance to the obstacle, and automatically apply the brakes to stop the vehicle depending on the distance. One of the means for detecting the obstacles is a distance detection device that uses image recognition technology based on images captured by a camera mounted on the vehicle.

In Patent Document 1, an ultrasonic sensor detects the distance to an object. Feature points of the object are extracted from two camera images, and the direction (pitch) is estimated from the position of the feature points on the images. The three-dimensional position of the feature points is estimated from the distance to the object, the direction of the feature points, and the distance traveled by the vehicle.

Patent No. 6753383

However, in the conventional technology, the process is carried out after feature points have been identified, and there is no description of the relative positions of the two camera images used to extract feature points, nor is there any mention of the shooting positions of the two images, raising concerns about accuracy.

The present invention aims to provide an object detection device, an object detection method, and an object detection program that can quickly collect a group of feature points with high accuracy in estimating three-dimensional positions.

The object detection device of the present invention is an object detection device that detects objects around a vehicle, and includes an imaging unit that photographs the area around the vehicle, a distance detection unit that detects the distance from the mounting position of the imaging unit to a specific object, an acquisition unit that acquires a pair of images from multiple images captured by the imaging unit in accordance with the movement of the vehicle based on acquisition conditions that depend on the distance to the object detected by the distance detection unit, and an estimation unit that extracts a feature point of the object based on the amount of displacement on the images of an identical feature point extracted from the pair of images, and estimates the three-dimensional position of the feature point, wherein the acquisition condition of the acquisition unit is a predetermined distance, and the acquisition unit acquires an image for which the condition of less than or equal to the predetermined distance is satisfied as at least the first image of the pair of images .

According to the present invention, the imaging unit captures images of the surroundings of the vehicle. The distance detection unit detects the distance from the mounting position of the imaging unit to a specific object.

The acquisition unit acquires a pair of images from multiple images captured by the image capture unit in response to the movement of the vehicle, based on acquisition conditions that depend on the distance to the object detected by the distance detection unit, and the estimation unit extracts a feature point of the object based on the amount of displacement on the image of the same feature point extracted from the pair of images, and estimates the three-dimensional position of the feature point.

This allows for the rapid collection of feature points with high accuracy in estimating three-dimensional positions.

If at least the first image is taken at a distance less than a predetermined distance, the positions of the feature points to be extracted can be obtained with high accuracy.

The present invention is characterized in that the estimation unit identifies the boundary between the road surface area and the object area based on the direction of the optical axis of the imaging unit and the distance to the object, and preferentially extracts feature points from the object area rather than the road surface area.

By selecting and discarding feature points, it is possible to obtain a sufficient number of feature points for map creation.

The present invention is characterized in that after the pair of images is acquired from the images captured by the imaging unit, the images are acquired sequentially according to the movement of the vehicle, and the estimation unit estimates the three-dimensional positions of the feature points, thereby creating a feature point cloud map.

By calculating the displacement of feature points on the image, the three-dimensional position of the feature points can be estimated.

The present invention is characterized in that the shooting positions of the shooting unit that captures one image and the other image of the pair of images are at positions that are apart by a predetermined distance or more.

If the shooting positions of the imaging units that capture one image and the other image are separated by a specified distance or more, the accuracy of the amount of displacement of the feature points on the images can be improved.

In the present invention, the photographing positions of the photographing unit that photographs one image and the other image of the pair of images are positions where the lines of sight to the feature points of both images form an angle of a predetermined value or more.

If the shooting positions of the shooting units that capture one image and the other image are such that the shooting optical axes of both images form an angle of a specified value or more, the accuracy of the displacement of the feature points on the images can be improved.

The object detection method according to the present invention is an object detection method for detecting objects around a vehicle, comprising:
The system is characterized in that the surroundings of the vehicle are photographed by an imaging unit, the distance from the mounting position of the imaging unit to a specific object is detected, and an image in which the detected distance to the specific object satisfies a condition that the distance is less than or equal to a predetermined distance is obtained as at least the first image of a pair of images, a feature point of the object is extracted based on the amount of displacement on the image of the same feature point extracted from the pair of images, and the three-dimensional position of the feature point is estimated.

According to the present invention, it is possible to quickly collect a group of feature points with high estimation accuracy of three-dimensional positions. Furthermore, if at least the first image is an image taken at a predetermined distance or less, it is possible to obtain the positions of the feature points to be extracted with high accuracy.

The object detection program according to the present invention is characterized in that it causes a computer to operate as the acquisition unit and the estimation unit of the object detection device.

The present invention makes it possible to quickly collect a group of feature points with high accuracy in estimating three-dimensional positions.

1 is a side view of a vehicle equipped with an object detection device for extracting feature points of an object and creating a high-precision map that serves as the basis for autonomous driving according to the present embodiment. FIG. 2 is a control block diagram in which processing for object detection in the object detection device according to the present embodiment is functionally blocked. FIG. FIG. 2 is a front view showing an example of an image captured by an in-vehicle camera. 1 is an explanatory diagram showing a schematic diagram of a process for estimating three-dimensional features from images (two images taken at different positions) captured by an on-board camera; 4 is a control flowchart for creating a high-precision map, focusing mainly on image acquisition control of the object detection device 12. 6A is a plan view showing the positional relationship between the vehicle and an object, and FIG. 6B is a characteristic diagram showing a three-dimensional coordinate system in FIG.

Figure 1 is a side view of a vehicle 14 equipped with an object detection device 12 for extracting feature points of an object 10 and creating a high-precision map that serves as the basis for autonomous driving, according to this embodiment.

The object 10 is placed on the road surface 20 on which the vehicle 14 is traveling, and is positioned as an obstacle to the vehicle 14.

The vehicle 14 is equipped with an on-board camera 16 and an ultrasonic sensor 18. FIG. 1 illustrates the on-board camera 16 and ultrasonic sensor 18 that monitor the rear of the vehicle 14. The vehicle 14 is also equipped with an on-board camera 16 and ultrasonic sensor 18 (not shown) that monitor the periphery (360° surroundings) of the vehicle 14.

In this embodiment, the purpose is to create a parking environment map for parking the vehicle 14 (for example, a map of a parking lot such as a piloti space of a building), and it is assumed that the vehicle will reverse to park in a specified parking area and then move forward to exit the specified parking area.

Based on the above assumptions, in this embodiment, an onboard camera 16 and an ultrasonic sensor 18 mounted on the rear of the vehicle 14 are used to detect an object 10 (a standing object with which the vehicle 14 may collide) located behind the vehicle 14.

The onboard camera 16 captures an image with a predetermined angle of view (see FIG. 3) that includes an object 10 present behind the vehicle 14 and the road surface 20 leading to the object. Meanwhile, the ultrasonic sensor 18 measures the distance D to the object 10.

When capturing an image of the rear of the vehicle 14, the vehicle-mounted camera 16 is usually mounted with its optical axis A facing the road surface 20 at a predetermined depression angle θ.

Thus, as will be described in detail later, when extracting feature point P (see FIG. 3), it is necessary to take into account the position of the road surface that corresponds to distance D on the image captured by the on-board camera 16 from the distance D from the vehicle 14 to the object 10 detected by the ultrasonic sensor 18.

As shown in FIG. 3, the object detection device 12 mounted on the vehicle 14 extracts a feature point P from an image captured by the on-board camera 16, and estimates the three-dimensional position of the feature point from at least two images captured by the on-board camera 16 at different positions of the vehicle 14 (see FIG. 4).

The object detection device 12 is connected to a map creation device 22, and creates a so-called high-precision map based on the three-dimensional position information of each feature point estimated by the object detection device 12. Note that the map creation device 22 may be provided on the cloud instead of being mounted on the vehicle 14.

Figure 2 is a control block diagram in which the processing for object detection in the object detection device 12 according to this embodiment is divided into blocks according to function.

Note that each block in the control block diagram of FIG. 2 is classified by function and does not limit the hardware configuration of the object detection device 12. In addition, some or all of the processing may be executed as an object detection program.

The vehicle-mounted camera 16 is connected to the image information acquisition unit 24. The ultrasonic sensor 18 is connected to the distance comparison unit 26. The distance comparison unit 26 is connected to the image information acquisition unit 24 and notifies the image information acquisition unit 24 of the time when the image is acquired.

In addition, a travel distance sensor 28 that detects the amount of movement of the vehicle 14 is connected to the image information acquisition unit 24. The travel distance sensor 28 can be, for example, a sensor that detects the wheel speed that is originally attached to the vehicle 14.

In the distance comparison unit 26, a threshold value is set for the distance D to the object 10 detected by the ultrasonic sensor 18, and as an image from which feature point P (see the square frame in Figure 3) is extracted, at least an image in which the distance D from the vehicle 14 to the object 10 is within the threshold value (equal to or less than the predetermined minimum distance X shown in Figure 4) is set as the first captured image to be applied to the object detection process.

Conventionally, when extracting feature points and estimating three-dimensional positions, the three-dimensional position is estimated based on two images taken at different positions, regardless of the distance to the object. However, this means that if the distance to the feature point is long, the accuracy of the position on the images decreases according to the distance. If the accuracy of the position on the images is low, errors also occur in the estimated value of the three-dimensional position.

Therefore, in this embodiment, at least the captured image from which the feature point P is initially extracted is within a predetermined minimum distance X (X is a threshold value) from the object 10, and the distance comparison unit 26 instructs the image information acquisition unit 24 to acquire an image, thereby maintaining the positional accuracy of the feature point P on the image.

The specific distance X may be set within the range detectable by the ultrasonic sensor 18 (e.g., 30 cm to 5 m), and it is preferable for X to be approximately 4 m to 5 m.

The image information acquisition unit 24 is connected to a feature point extraction unit 30. The feature point extraction unit 30 extracts feature points P from the captured image. The feature point extraction unit 30 is connected to a feature point selection unit 32, and sends the extracted feature points P to the feature point selection unit 32.

The road surface boundary estimation unit 34 is connected to the feature point selection unit 32. The road surface boundary estimation unit 34 acquires distance information from the ultrasonic sensor 18. The road surface boundary estimation unit 34 estimates the boundary between the object 10 and the road surface based on the distance D from the vehicle 14 to the object 10 detected by the ultrasonic sensor 18.

An example of estimation will be described with reference to Figure 6. Figure 6 (A) is a plan view showing the positional relationship between the vehicle and the object, and (B) shows the three-dimensional coordinate system in Figure 6 (A).

When the distance to the object is D and the azimuth angle of the reflection point of the ultrasonic sensor 16 is φ, the position (x _v , y _v , z _v ) in the vehicle coordinate system of the boundary point Ps between the object and the road surface is expressed by the following equation (1).

... (1)

When the depression angle of the vehicle-mounted camera 16 is θ, the camera height from the road surface is h, and the focal length is f, the following relationships (2) and (3) hold between the position ( _xv , _yv , _zv ) of the road surface boundary point in the vehicle coordinate system, the position ( _xc , _yc , _zc ) in the camera coordinate system, and the position ( _xi , _yi ) in the image coordinate system.

Since the azimuth angle φ of the reflection point is unknown, the radiation angle Φ of the ultrasonic sensor 18 is taken into consideration, and the position (x i , y i ) in the image coordinate system is calculated using the above equations (1) to (3) for various azimuth angles φ (for example, −0.5×π+k×0.01×Φ, where k=0, ₁ , _{... 100} ) that satisfy the following equation (4), thereby making it possible to identify the road boundary line K on the image.

...(4)

The feature point selection unit 32 receives information about the feature points P from the feature point extraction unit 30, and selects from among multiple feature points P present in the captured image based on the road boundary lines from the road boundary estimation unit 34.

In other words, since the area below the road boundary line (see dotted line K in Figure 3) is a location where no object 10 exists, by lowering (or removing) the priority of the feature point P, it is possible to select important feature points P that depend on the object 10.

The feature point selection unit 32 is connected to the three-dimensional position estimation unit 36. The three-dimensional position estimation unit receives the image information acquired by the image information acquisition unit 24, and estimates at least the three-dimensional positions of the feature points P shared by the two images (each of the feature points P selected by the feature point selection unit 32).

The three-dimensional position estimation unit 36 is connected to the map creation device 22 and sends the three-dimensional position (estimated value) of each estimated feature point P to the map creation device 22. The map creation device 22 creates a map based on the three-dimensional position (estimated value) of the newly received feature point P. The created map is used, for example, as a high-precision map of the parking environment that is the basis for autonomous driving.

Figure 4 is an explanatory diagram that shows a schematic diagram of the process for estimating the three-dimensional shape of feature point P from images (two images taken at different positions) captured by the vehicle-mounted camera 16.

In FIG. 4(A), the three-dimensional position of characteristic point P is estimated while the vehicle 14 is turning around characteristic point P.

First, the vehicle-mounted camera 16 captures the nth image 50A at a first position, and then moves to a second position to capture the (n+1)th image 50B. The positions of the feature points P1 and P2 on the two images 50A and 50B are different. Based on this amount of displacement, the three-dimensional position of the feature point P can be estimated.

The technology for estimating the three-dimensional position of feature point P from the distance to the object on which feature point P exists, the direction of feature point P, and the distance traveled by vehicle 14 is a publicly known technology (e.g., Patent Publication No. 6753383), so a detailed explanation will be omitted here.

It is preferable that the angle (included angle) β between the lines of sight of the nth image 50A and the n+1th image 50B is 1° or more. In this embodiment, it is essential that the first image 50A (n=1) is captured at a distance D to the object 10 that is equal to or less than the threshold value X.

On the other hand, FIG. 4(B) is an explanatory diagram that shows a schematic diagram of the process for estimating the three-dimensional location of the characteristic point P when two images 50A and 50B are captured while the vehicle 14 is moving away from the characteristic point P.

The principle is the same as in FIG. 4(A), but for example, when the vehicle 14 is moving away, there is little displacement of the positions of the feature points P1 and P2 on the two images 50A and 50B. Therefore, the interval for capturing the captured images can be determined on the premise that when the nth image 50A is captured and then the n+1th image 50B is captured, the angle (included angle) between the lines of sight of both images is 1° or more. Note that, even in FIG. 4(B), it is essential that the first image 50A (n=1) is captured when the distance D to the object 10 is equal to or less than the threshold value X.

The operation of this embodiment is explained below.

Figure 5 is a control flowchart for creating a high-precision map, focusing mainly on image acquisition control by the object detection device 12.

In step 100, as an initial setting, flag F is reset (to 0) and the count value n of the number of shots is set to 1, and then the process proceeds to step 102.

In step 102, the image captured by the vehicle-mounted camera 16 is requested, and then in step 104, it is determined whether the image has been acquired, and the process waits in this step 104 until a positive determination is made.

If the answer is yes in step 104, the process proceeds to step 106, where the state of flag F (F=0 or not) is checked.

If the determination in step 106 is positive (F=0), it is determined that the first image has not been acquired, and the process proceeds to step 108 to determine whether the distance D to the object 10 is equal to or less than a predetermined value (equal to or less than the threshold value X in FIG. 4).

If the result of this step 108 is negative, it is determined that the distance does not match the distance for acquiring the first image (less than or equal to threshold value X), and the process returns to step 102 and the above steps are repeated.

If the result of step 108 is positive, the process proceeds to step 110, where it is determined that the distance matches that of the first image capture, and feature point P (see the square frame in Figure 3) of the image captured in step 104 is extracted.

In the next step 112, the road boundary line is identified based on the distance D and the orientation (depression angle θ) of the vehicle-mounted camera 16.

In the next step 114, feature points P are selected based on the road boundary line (see dotted line K in FIG. 3) identified in step 112. Specifically, feature points P that are in the area below the road boundary line are excluded.

In the next step 116, flag F is set (1), then the process proceeds to step 118 where the image is stored as the nth image, and the process proceeds to step 120.

In step 120, it is determined whether image acquisition has ended. If the determination in step 120 is negative, the process proceeds to step 122 to acquire the next image, incrementing the count value n of the number of shots (n←n+1), and proceeding to step 124.

In step 124, the amount of movement of the vehicle 14 (on-board camera 16) is read from the movement distance sensor 28, and the process proceeds to step 126 to determine whether or not it has moved a predetermined distance or more. Here, the determination is made based on the amount of movement in distance, but it is preferable to determine whether or not the angle (included angle) β between the lines of sight of the two positions of the on-board camera 16 is 1° or more, as shown in FIG. 4 (B). This is because when the vehicle 14 moves away from the characteristic point P, as shown in FIG. 4 (B), the amount of displacement of the characteristic points P1 and P2 may be small even if the movement distance increases.

If the result of step 126 is negative, the process returns to step 124 and monitors the amount of movement. If the result of step 126 is positive, the process returns to step 102 and the above process is repeated.

On the other hand, if the result of step 120 is positive, it is determined that a sufficient number of images have been acquired, and the process proceeds to step 128, where an estimation process is performed for the three-dimensional position of each feature point P, and then the process proceeds to step 130, where a feature point map is created, and the routine ends.

As described above, according to this embodiment, when the images required for estimating the three-dimensional position of feature point P are acquired from the vehicle-mounted camera 16, the first (first) image is one in which the distance between the vehicle 14 and the object 10 on which feature point P is located is within a predetermined value (distance X or less). If the distance between the vehicle 14 and the object 10 is distance X or more, the position accuracy of the extracted feature point P is low, and as a result, the estimation accuracy of the three-dimensional position is low. In this embodiment, by setting the distance between the vehicle 14 and the object 10 to be X or less, the position accuracy of feature point P can be set to a predetermined accuracy standard.

In addition, when determining the amount of movement of the vehicle 14 (on-board camera 16) when acquiring the nth image and the n+1th image, the angle (included angle) β between the lines of sight of the on-board camera 16 at both positions is set to 1° or more, so that, as shown in Figure 4 (B), even if the movement is away from the feature point P, the amount of displacement of the captured feature points P1 and P2 can be sufficiently ensured.

Furthermore, based on the depression angle θ of the vehicle-mounted camera 16, the range of the road surface boundary line corresponding to the distance D to the object 10 can be identified, and the feature points P in the area below this road surface boundary line (dotted line K in Figure 3) can be excluded, and the feature points P necessary and sufficient for recognizing the object 10 can be extracted.

In this embodiment, an example is shown in which a high-precision map of the parking area for the vehicle 14 (for example, a parking lot in a piloti space of a building) is created, but this is not limited to parking lots, and does not exclude general main roads, etc. However, creating a map of a driving route in which there are objects around that may be obstacles to the driving of the vehicle 14 can be more effective.

10 Object 12 Object detection device 14 Vehicle 16 In-vehicle camera (photographing unit)
18 Ultrasonic sensor (distance detection unit)
20 Road surface 22 Map creation device 24 Image information acquisition unit (acquisition unit)
26 Distance comparison unit 28 Travel distance sensor 30 Feature point extraction unit 32 Feature point selection unit 34 Road surface boundary estimation unit 36 Three-dimensional position estimation unit (estimation unit)
50A, 50B Image K Dotted line (road boundary line)
P Feature point P1, P2 Feature points (on image)
A Optical axis X Distance (Threshold value for first shot)
θ Depression angle β Including angle D Distance (between vehicle and object (feature point))

Claims (7)

An object detection device that detects objects around a vehicle,
An imaging unit that images the surroundings of the vehicle;
a distance detection unit that detects a distance from an attachment position of the photographing unit to a specific object;
an acquisition unit that acquires a pair of images from a plurality of images captured by the image capture unit in response to movement of the vehicle, based on an acquisition condition that depends on a distance to an object detected by the distance detection unit;
an estimation unit that extracts a feature point of the object based on a displacement amount of the same feature point on the pair of images and estimates a three-dimensional position of the feature point,
An object detection device, wherein the acquisition condition of the acquisition unit is a predetermined distance, and the acquisition unit acquires an image in which the condition of being less than or equal to the predetermined distance is satisfied as at least a first image of the pair of images . The estimation unit,
The object detection device according to claim 1 , further comprising: a detecting unit that detects an object from a road surface area based on a direction of the optical axis of the photographing unit and a distance to the object; and a detecting unit that detects an object from the road surface area based on the .... 3. The object detection device according to claim 1, wherein after the pair of images is acquired from the images captured by the photographing unit, the images are acquired sequentially in accordance with the movement of the vehicle, and the estimation unit estimates the three-dimensional positions of the feature points, thereby creating a feature point cloud map . 3. The object detection device according to claim 1, wherein the image capturing positions of the image capturing unit that captures one image and the other image of the pair of images are at positions spaced apart by a predetermined distance or more. An object detection device as described in any one of claims 1 to 4, wherein the shooting positions of the shooting unit that captures one image and the other image of the pair of images are positions where the shooting optical axes of both images form an angle of a predetermined value or more . 1. An object detection method for detecting objects around a vehicle, comprising:
The camera captures the area around the vehicle,
Detects the distance from the installation position of the imaging unit to a specific object,
An image in which a condition that the distance to the detected specific object is equal to or less than a predetermined distance is satisfied is acquired as at least a first image of a pair of images;
extracting a feature point of the object based on an amount of displacement of the same feature point on the pair of images, and estimating a three-dimensional position of the feature point;
Object detection methods. Computer,
The object detection device according to any one of claims 1 to 5 is operated as the acquisition unit and the estimation unit.
Object detection program.