JP7632164B2 - Object detection method and object detection device - Google Patents

Description

The present invention relates to an object detection method and an object detection device.

Conventionally, there has been known an object detection method that extracts candidate points (feature points) of an object from images captured by a stereo camera, and detects the object by clustering the extracted candidate points (Patent Document 1). The object detection method disclosed in Patent Document 1 acquires the coordinates of candidate points of an object from two images captured by a stereo camera, votes for each candidate point of the object on a voting surface based on the acquired coordinates, and performs clustering by grouping the candidate points of the object into adjacent groups on the voting surface. The clustering described in Patent Document 1 uses a shortest distance method that uses width and depth as feature quantities.

JP 2006-53756 A

However, the accuracy of object position detection by a stereo camera decreases as the distance between the stereo camera and the object increases. Therefore, the object detection method disclosed in Patent Document 1 may not be able to cluster candidate points of detected objects in areas where the accuracy of position detection by the stereo camera decreases, or may perform erroneous clustering, resulting in failure to accurately detect the object.

The present invention has been made in consideration of the above problems, and aims to provide an object detection method and an object detection device that can perform clustering with high accuracy even in areas where the accuracy of position detection by a stereo camera decreases, and can accurately detect objects.

In an object detection method according to one aspect of the present invention, two images are obtained by capturing images from different positions in a predetermined direction from the vehicle, and multiple feature points are extracted from each of the two images. From the positions of the multiple feature points on the two images, the distance in the predetermined direction from the vehicle to the multiple feature points and the distance between the multiple feature points in real space are calculated. A first weight is assigned to a first possibility indicating that the multiple feature points calculated based on at least the distance between the multiple feature points in real space are feature points of the same object, and a second weight is assigned to a second possibility indicating that the multiple feature points calculated based on the distance between the multiple feature points in the image and the moving speed of each feature point are feature points of the same object, based on the distance in the predetermined direction from the vehicle to the multiple feature points. The multiple feature points are clustered using the first possibility with the first weight assigned and the second possibility with the second weight assigned.

According to the present invention, it is possible to perform clustering with high accuracy even in areas where the accuracy of position detection by a stereo camera decreases, and objects can be detected accurately.

FIG. 1 is a block diagram showing a configuration of an object detection device according to an embodiment. FIG. 2 is a schematic diagram showing a theoretical error that occurs when the stereo camera 10 measures the position of an object. FIG. 3A is a schematic diagram showing first weights and second weights that are set based on distances in the z direction from the host vehicle to multiple feature points (first example). FIG. 3B is a schematic diagram showing first weights and second weights that are set based on distances in the z direction from the host vehicle to multiple feature points (second example). FIG. 3C is a schematic diagram showing first weights and second weights that are set based on distances in the z direction from the host vehicle to multiple feature points (third example). FIG. 3D is a schematic diagram showing first weights and second weights that are set based on distances in the z direction from the host vehicle to multiple feature points (fourth example). FIG. 3E is a schematic diagram showing first weights and second weights that are set based on the distances in the x direction from the host vehicle to multiple feature points. FIG. 4A is a schematic diagram showing filtering according to an embodiment (part 1). FIG. 4B is a schematic diagram showing filtering according to the embodiment (part 2). FIG. 5 is a diagram illustrating a match cost matrix according to an embodiment. FIG. 6A is a schematic diagram showing a clustering result based on a first possibility. FIG. 6B is a schematic diagram showing the clustering result based on the second possibility. FIG. 7 is a flowchart illustrating an example of the operation of the object detection device according to the embodiment.

The embodiment will be described with reference to the drawings. In the description of the drawings, the same parts are given the same reference numerals and the description will be omitted.

[Configuration of Object Detection Device]
A configuration of an object detection device 1 according to an embodiment will be described with reference to Fig. 1. The object detection device 1 includes a stereo camera 10 and a control unit 20. The object detection device 1 can be used for object detection in, for example, Advanced Driver-Assistance Systems (ADAS). In this embodiment, the object detection device 1 is mounted on a vehicle and detects an object present in front of the vehicle.

The stereo camera 10 includes a camera 11 and a camera 12. The cameras 11 and 12 are disposed at different positions within the housing of the stereo camera 10. The stereo camera 10 is disposed at the front of the vehicle interior, and the cameras 11 and 12 capture images of the front of the vehicle from different positions to obtain two images. Specifically, when the stereo camera 10 is installed on the vehicle, the camera 11 is disposed on the right side of the vehicle, and the camera 12 is disposed on the left side of the vehicle. The cameras 11 and 12 capture images at the same time. The camera 11 captures a first image of the front of the vehicle from the right side of the vehicle, and the camera 12 captures a second image of the front of the vehicle from the left side of the vehicle. The stereo camera 10 repeatedly captures images of the front of the vehicle at a predetermined cycle, obtains two images of the front of the vehicle captured at the same time, and transmits the obtained images to the control unit 20.

In this embodiment, the stereo camera 10 (camera 11 and camera 12) is a camera that captures the view in front of the vehicle, but is not limited to this. The stereo camera 10 may be a camera that captures the side of the vehicle. In addition, the cameras 11 and 12 are arranged at different positions in the left-right direction of the vehicle, but if the stereo camera 10 is a camera that captures the side of the vehicle, it is arranged at different positions in the front-rear direction of the vehicle. In other words, the stereo camera 10 may be a camera that captures a predetermined direction from the vehicle, and the cameras 11 and 12 may be arranged at different positions in a direction having an angle with respect to the predetermined direction that is the imaging direction of the stereo camera 10. In the following, an example will be described in which the predetermined direction that is the imaging direction of the stereo camera 10 is the front-rear direction, and the different positions at which the cameras 11 and 12 are arranged are different positions in the left-right direction of the vehicle.

In the present embodiment, the stereo camera 10 is exemplified as a stereo camera having two cameras, camera 11 and camera 12, arranged at different positions, but is not limited thereto. For example, the stereo camera 10 may have cameras arranged at three or more different positions. Also, the stereo camera 10 may have only one camera and may be a so-called motion stereo camera that acquires two images taken by one camera at different times while the vehicle is moving, that is, multiple images taken by one camera at different positions. In other words, the stereo camera 10 is not limited in the number of cameras, and may be a camera that acquires multiple images taken at different positions. Hereinafter, in the present embodiment, for the sake of simplicity, the stereo camera 10 will be described as a stereo camera having two cameras, camera 11 and camera 12, arranged at different positions.

The control unit 20 detects an object in front of the vehicle by extracting multiple feature points from each of the two images captured by the stereo camera 10 and clustering the extracted feature points. In this embodiment, the object may be a moving object including another vehicle, a motorcycle, a bicycle, or a pedestrian, or a stationary object including a parked vehicle. The control unit 20 is a general-purpose microcomputer equipped with a CPU (central processing unit), memory (storage unit) such as RAM and ROM, and an input/output unit. A computer program for functioning as an object detection device is installed in the microcomputer. By executing the computer program, the microcomputer functions as multiple information processing circuits (21, 22, 23, 24, 25, 26) equipped in the object detection device. In this embodiment, an example is shown in which the multiple information processing circuits (21, 22, 23, 24, 25, 26) equipped in the object detection device are realized by software, but it is also possible to configure the information processing circuit by preparing dedicated hardware for executing each information processing. In addition, the multiple information processing circuits may be configured by individual hardware.

The control unit 20 includes multiple information processing circuits, including a feature point extraction unit 21, a distance image generation unit 22, an optical flow calculation unit 23, a vehicle behavior measurement unit 24, a three-dimensional flow calculation unit 25, and an object detection unit 26.

The feature point extraction unit 21 extracts multiple feature points, which are pixels having characteristics that can be distinguished from surrounding pixels, from each of the images acquired by the camera 11 and the camera 12. A well-known method can be applied to extract the feature points, and for example, the method described in the non-patent document "Jianbo Shi and Carlo Tomasi, "Good Features to Track," 1994 IEEE Conference on Computer Vision and Pattern Recognition (CVPR'94), 1994, pp.593 - 600." is used. The feature point extraction unit 21 outputs the multiple extracted feature points to the distance image generation unit 22 and the optical flow calculation unit 23. Note that, hereinafter, unless otherwise specified, it is assumed that there are multiple feature points.

The distance image generating unit 22 calculates the distances from the vehicle to the multiple feature points in the forward/rearward direction of the vehicle and the distances between the multiple feature points in real space, and generates a distance image that displays the relative positions of the multiple feature points with respect to the vehicle. Specifically, the distance image generating unit 22 calculates the three-dimensional coordinates (relative coordinates) of the multiple feature points with respect to the vehicle from the parallax (shift amount) for the same feature point among the feature points of the first image and the feature points of the second image. Next, the distance image generating unit 22 calculates the distances from the vehicle to the multiple feature points in the forward/rearward direction of the vehicle and the distances between the multiple feature points in real space from the three-dimensional coordinates of the multiple feature points. The distance image generating unit 22 may also calculate the distances from the vehicle to the multiple feature points in the forward/rearward direction of the vehicle from the parallax between the feature points extracted from the first image acquired by the camera 11 and the feature points extracted from the second image acquired by the camera 12. The distance image generating unit 22 also calculates the angle that the feature point makes with respect to the vehicle's straight-ahead direction (predetermined direction) from the three-dimensional coordinates of the multiple feature points relative to the vehicle. Next, the distance image generating unit 22 places each feature point on the projection plane based on the three-dimensional coordinates of the multiple feature points, and generates a distance image that shows the distance from the vehicle to the multiple feature points in the forward/rearward direction of the vehicle. The distance image generating unit 22 outputs the generated distance image to the three-dimensional information evaluating unit 28 and the evaluation method determining unit 29.

The optical flow calculation unit 23 calculates an optical flow from the positions of feature points in a past image of either the camera 11 or the camera 12 extracted by the feature point extraction unit 21 and the positions of feature points in a current image. The optical flow represents the two-dimensional movement (movement speed) of feature points on an image as a vector. The optical flow calculation unit 23 detects current feature points corresponding to the same object in real space as an object corresponding to a past feature point as related feature points. The optical flow calculation unit 23 calculates a combination of past feature points and current feature points that are related to each other as an optical flow. This allows the optical flow calculation unit 23 to calculate the distance between multiple feature points in an image and the movement speed of each feature point. A known method can be applied to calculate the optical flow, for example, the method described in the non-patent document "Bruce D. Lucasand Takeo Kanade. An Iterative Image Registration Technique with an Application to Stereo Vision. International Joint Conference on Artificial Intelligence, pages 674-679, 1981" is used. The optical flow calculation unit 23 outputs the calculated optical flow to the three-dimensional flow calculation unit 25 and the two-dimensional information evaluation unit 27.

The vehicle behavior measurement unit 24 measures the behavior of the vehicle. Specifically, the vehicle behavior measurement unit 24 measures the behavior of the vehicle by performing odometry and dead reckoning. The behavior of the vehicle refers to the change in the relative position, relative attitude, and speed of the vehicle at a specified time. The vehicle behavior measurement unit 24 can measure the behavior of the vehicle at a specified time by measuring the relative position, relative attitude, and speed of the vehicle with respect to a specified reference point based on information obtained from a wheel speed sensor that detects the wheel speed of each wheel of the vehicle and a steering angle sensor that detects the steering angle of the steering wheel. The vehicle behavior measurement unit 24 may obtain the absolute position and absolute attitude of the vehicle in the earth coordinates from a receiving device (not shown) that receives GPS (Global Positioning System) signals, and measure the behavior of the vehicle from the amount of change in each at a specified time.

The three-dimensional flow calculation unit 25 calculates the three-dimensional movement (flow) of the feature points in real space. Specifically, first, the three-dimensional flow calculation unit 25 converts the optical flow calculated by the optical flow calculation unit 23 into a movement in real space. As described above, the optical flow is a two-dimensional movement of the feature points on the image. Therefore, the three-dimensional flow calculation unit 25 converts the two-dimensional movement of the multiple feature points on the image into a two-dimensional movement (velocity) in real space. Next, the three-dimensional flow calculation unit 25 adds the forward/rearward movement relative to the host vehicle to the calculated two-dimensional movement in real space of the multiple feature points. The forward/rearward movement of the multiple feature points relative to the host vehicle can be calculated from the change per unit time in the forward/rearward distance from the host vehicle to the multiple feature points calculated by the distance image generation unit 22. Note that the three-dimensional movement of the multiple feature points calculated up to this point is a relative movement relative to the host vehicle. Therefore, the three-dimensional flow calculation unit 25 corrects the relative movements of the multiple feature points with respect to the vehicle using the vehicle behavior calculated by the vehicle behavior measurement unit 24, and calculates three-dimensional movements (absolute values) in real space. The three-dimensional flow calculation unit 25 outputs the calculated three-dimensional movements (three-dimensional flows) of the feature points in real space to the three-dimensional information evaluation unit 28.

The object detection unit 26 clusters the feature points and detects the clustered feature points as one object. Specifically, the object detection unit 26 sets a first weight to be assigned to a first possibility calculated based on at least the distance between the feature points in real space, the first possibility indicating the possibility that the feature points are feature points of the same object. The object detection unit 26 also sets a second weight to be assigned to a second possibility calculated based on the distance between the feature points in the image and the moving speed of each feature point, the second possibility indicating the possibility that the feature points are feature points of the same object. Specifically, the object detection unit 26 sets the first weight and the second weight according to the distance from the vehicle to the feature points in the front-rear direction of the vehicle and the angle that the feature points make with respect to the straight direction of the vehicle. The object detection unit 26 then clusters the feature points to detect an object using the first possibility with the first weight assigned and the second possibility with the second weight assigned.

The first possibility is calculated based on at least the distance between multiple feature points in real space. Therefore, the first possibility takes into account the three-dimensional positional relationship between the feature points. Therefore, the first possibility is effective in areas where the accuracy of position detection by the stereo camera 10 is good. The second possibility is calculated based on the distance between multiple feature points in the image and the moving speed of each feature point. Therefore, the second possibility takes into account the two-dimensional positional relationship between the feature points and the moving speed of the feature points in two dimensions. In other words, the second possibility is effective in areas where the accuracy of position detection by the stereo camera 10 decreases by not taking into account the distance between the feature points in the front-rear direction (depth direction) of the vehicle.

2, a theoretical error that occurs when calculating the distance in the front-rear direction of the vehicle 2 from the vehicle 2 to the multiple feature points (i1, i2, i3, i4) and the left-right position of each feature point of the vehicle 2 from two images acquired by the stereo camera 10 is described. In this embodiment, the distance image generating unit 22 calculates the distance in the front-rear direction of the vehicle 2 from the multiple feature points (i1, i2, i3, i4) and the three-dimensional coordinates (relative coordinates) of the multiple feature points relative to the vehicle 2, but the same is true when the stereo camera 10 calculates the distance to an object. Here, the description is based on the premise that the stereo camera 10 calculates the distance in the front-rear direction of the vehicle 2 from the vehicle 2 to the multiple feature points (i1, i2, i3, i4) and the left-right position (x direction) of each feature point of the vehicle 2. The theoretical error that occurs when the stereo camera 10 calculates the distance in the front-rear direction of the vehicle 2 from the vehicle 2 to the feature point i1 is tez1. In addition, the theoretical error that occurs when the stereo camera 10 calculates the distance in the longitudinal direction of the vehicle 2 from the vehicle to the feature point i2 that is farther away than the feature point i1 is tez2. In this way, when the stereo camera 10 calculates the distance in the longitudinal direction of the vehicle 2 from the vehicle 2 to the feature point, the greater the distance in the longitudinal direction of the vehicle 2 from the stereo camera 10 to the feature point, the greater the theoretical error. In addition, in the case of feature points that have an angle with respect to the straight-ahead direction (z direction) of the vehicle, such as feature points i3 and i4, the greater the angle that the feature point makes with respect to the straight-ahead direction of the vehicle (θ1<θ2), the greater the theoretical error when calculating the position in the left-right direction (x direction) of the vehicle. Therefore, the theoretical error at the x-direction position (x1) of the feature point i3 is tex1, whereas the theoretical error at the x-direction position (x2) of the feature point i4 is tex2. Therefore, when the feature point i4 is on the broken line l, the feature point i4 is calculated as the same x coordinate.

The object detection unit 26 includes a two-dimensional information evaluation unit 27, a three-dimensional information evaluation unit 28, an evaluation method determination unit 29, and a clustering unit 30. The specific processing of each element constituting the object detection unit 26 will be described below.

The evaluation method determination unit 29 sets a first weight to be assigned (multiplied) to the first possibility and a second weight to be assigned (multiplied) to the second possibility based on the distance from the host vehicle to the multiple feature points in the fore-and-aft direction (z direction) of the host vehicle. Here, with reference to Figures 3A to 3D, the process in which the evaluation method determination unit 29 sets the first weight w1 and the second weight w2 based on the distance from the host vehicle to the multiple feature points in the fore-and-aft direction (z direction) of the host vehicle will be specifically described.

(First Example)
FIG. 3A shows a first example in which the evaluation method determination unit 29 sets the first weight w1 and the second weight w2. In the first example, the evaluation method determination unit 29 judges whether or not the distance in the front-rear direction (z direction) of the host vehicle from the host vehicle to the multiple feature points is equal to or greater than a predetermined distance zc. Then, when the evaluation method determination unit 29 judges that the distance in the front-rear direction of the host vehicle from the host vehicle to the multiple feature points is equal to or greater than the predetermined distance zc, the evaluation method determination unit 29 sets the first weight w1 to 0.0 and the second weight w2 to 1.0. Note that 1.0 is the maximum value of the weight that can be set. That is, when the distance in the front-rear direction of the host vehicle from the host vehicle to the multiple feature points is equal to or greater than the predetermined distance zc, the evaluation method determination unit 29 sets the weights to be determined to be equal to or greater than the predetermined distance zc by using only the second possibility. In addition, when the evaluation method determination unit 29 determines that the distance from the vehicle to the multiple feature points in the longitudinal direction of the vehicle is less than the predetermined distance zc, the evaluation method determination unit 29 sets the first weight w1 to 1.0 and the second weight w2 to 0.0. That is, when the distance from the vehicle to the multiple feature points in the longitudinal direction of the vehicle is less than the predetermined distance zc, the evaluation method determination unit 29 sets the first weight w1 to 1.0 and the second weight w2 to 0.0 to determine whether the multiple feature points are feature points of the same object using only the first possibility. The predetermined distance zc is a distance (e.g., 50 m) at which the accuracy of position detection by the stereo camera 10 decreases. In this way, in the first example, the evaluation method determination unit 29 discontinuously changes the first weight w1 and the second weight w2.

(Second Example)
3B shows a second example in which the evaluation method determination unit 29 sets the first weight w1 and the second weight w2. The difference from the first example is that when the evaluation method determination unit 29 determines that the distance from the vehicle to the multiple feature points in the front-rear direction of the vehicle is equal to or greater than a predetermined distance zc, the evaluation method determination unit 29 sets the first weight w1 to 0.1 and the second weight w2 to 0.9. This allows the first possibility to be taken into consideration even in an area where the accuracy of position detection of the stereo camera 10 decreases, and it is possible to prevent feature points that are far apart in real space and are not feature points of the same object from being determined to be feature points of the same object.

(Third Example)
3C shows a third example in which the evaluation method determination unit 29 sets the first weight w1 and the second weight w2. In the third example, when the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle is less than a predetermined distance z1, the evaluation method determination unit 29 sets the first weight w1 to 1.0 and the second weight w2 to 0.0. When the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle is equal to or greater than the predetermined distance z1 and less than the predetermined distance z2, the evaluation method determination unit 29 continuously changes the first weight w1 and the second weight w2 based on the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle. More specifically, the farther the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle, the more the first weight w1 is decreased from 1.0 and the second weight w2 is increased from 0.0. When the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle is equal to or greater than a predetermined distance z2, the evaluation method determination unit 29 sets the first weight w1 to 0.0 and the second weight w2 to 1.0. The predetermined distance z1 is a distance at which the theoretical error when calculating the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle is 2 m. The predetermined distance z1 is, for example, 30 m. The predetermined distance z2 is a distance at which the theoretical error when calculating the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle is 10 m. The predetermined distance z2 is, for example, 70 m. In this way, in the third example, the evaluation method determination unit 29 continuously changes the first weight w1 and the second weight w2 between the predetermined distance z1 and the predetermined distance z2.

(Fourth Example)
3D shows a fourth example in which the evaluation method determination unit 29 sets the first weight w1 and the second weight w2. In the fourth example, when the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle is less than a predetermined distance z3, the evaluation method determination unit 29 sets the first weight w1 to 1.0 and the second weight w2 to 0.0. The evaluation method determination unit 29 sets the value obtained by dividing the theoretical error in the longitudinal direction of the host vehicle at the predetermined distance z3 by the theoretical error in the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle as the first weight w1. The second weight w2 is set to a value obtained by subtracting the first weight w1 from 1.0.

The evaluation method determination unit 29 sets the first weight w1 to the first possibility and the second weight w2 to the second possibility based on the angle that the feature point makes with respect to the straight-line direction of the vehicle. Here, referring to FIG. 3E, the process in which the evaluation method determination unit 29 sets the first weight w1 and the second weight w2 based on the angle that the feature point makes with respect to the straight-line direction of the vehicle will be specifically described. As described with reference to FIG. 2, in the case of a feature point having an angle with respect to the straight-line direction (z direction) of the vehicle, the greater the angle that the feature point makes with the straight-line direction of the vehicle, the greater the theoretical error in calculating the position in the left-right direction (x direction) of the vehicle. Therefore, the evaluation method determination unit 29 reduces the first weight w1 and increases the second weight w2 in the region r1 where the angle that the feature point makes with respect to the straight-line direction of the vehicle is larger than in the region r0. Similarly, in region r3 where the angle that the feature point makes with respect to the straight-ahead direction of the vehicle is larger, the first weight w1 is decreased and the second weight w2 is increased compared to region r2. As described above, the evaluation method determination unit 29 sets the first weight w1 assigned to the first possibility and the second weight w2 assigned to the second possibility based on the distance from the vehicle to the feature point in the front-rear direction of the vehicle and the angle that the feature point makes with respect to the straight-ahead direction of the vehicle.

The three-dimensional information evaluation unit 28 filters the feature points, and calculates a first possibility indicating the possibility that the first weighted feature points are feature points of the same object, based on the distance between the feature points in real space after filtering and the moving speed of each feature point in real space. Here, the filtering of feature points will be specifically described with reference to Figures 4A and 4B. The filtering of feature points is a process that prohibits clustering of feature points whose distance between them in real space is longer than a predetermined allowable distance as feature points of the same object, taking into account theoretical errors that occur when calculating the distance from the host vehicle to the feature points in the longitudinal direction of the host vehicle.

Figure 4A shows feature point i, feature point j, theoretical error te that occurs when calculating the distance from the vehicle to feature point i and the distance from the vehicle to feature point j in the front-rear direction (z direction) of the vehicle, and a predetermined allowable distance pd. Note that in Figures 4A and 4B, for convenience, the theoretical errors te of feature point i and feature point j are the same value. The three-dimensional information evaluation unit 28 judges whether the distance d1 between feature point i and feature point j in the real space taking into account the theoretical error te is within a predetermined allowable distance pd. The distance d1 between multiple feature points in the real space taking into account the theoretical error is the distance at which the distance between feature point i and feature point j is the shortest when taking into account the theoretical error te. In Figure 4A, the distance d1 between multiple feature points in the real space taking into account the theoretical error is longer than the predetermined allowable distance pd. Therefore, the three-dimensional information evaluation unit 28 prohibits clustering of feature point i and feature point j as feature points of the same object. More specifically, the calculation of possibility 1, which indicates the possibility that feature point i and feature point j are feature points of the same object, is not performed. In contrast to FIG. 4A, the distance d1 between multiple feature points in real space in FIG. 4B, which takes into account the theoretical error te between feature point i and feature point j, is within a predetermined allowable distance pd. Therefore, the three-dimensional information evaluation unit 28 calculates the first possibility, which indicates the possibility that feature point i and feature point j are feature points of the same object.

The three-dimensional information evaluation unit 28 calculates a first possibility indicating the possibility that the first weighted feature points are feature points of the same object, based on the distance between the feature points in real space after filtering and the moving speed of each feature point in real space. Specifically, the first possibility is calculated by the following formula (1). Note that the first possibility f3d(i, j) in the following formula (1) is calculated as a smaller value the more likely it is that feature point i and feature point j are feature points of the same object.

Xi, Yi, Zi are the coordinates of feature point i in real space, and vXi, vYi, vZi are the velocities of feature point i in the x-, y-, and z-directions in real space. Xj, Yj, Zj are the coordinates of feature point j in real space, and vXj, vYj, vZj are the velocities of feature point j in the x-, y-, and z-directions in real space. a to f are normalization parameters for stabilizing the theoretical error included in the evaluation formula. The three-dimensional information evaluation unit 28 outputs the result of applying a first weight to the first possibility calculated by formula (1) to the clustering unit 30.

The two-dimensional information evaluation unit 27 calculates a second possibility indicating the possibility that the multiple feature points with the second weighting are feature points of the same object based on the distance between the multiple feature points and the moving speed of each feature point. Specifically, the second possibility is calculated by the following formula (2). Note that the second possibility f2d(i, j) in the following formula (2) is calculated as a smaller value the more likely it is that feature point i and feature point j are feature points of the same object.

xi, yi are the coordinates of feature point i in the image, and vxi, vyi are the movement speeds of feature point i in the x and y directions in the image. xj, yj are the coordinates of feature point j in the image, and vxj, vyj are the movement speeds of feature point i in the x and y directions in the image. a to d are normalization parameters for stabilizing the theoretical error included in the evaluation formula. The two-dimensional information evaluation unit 28 outputs the result of applying a second weight to the second possibility calculated by formula (2) to the clustering unit 30.

The clustering unit 30 adds the value obtained by multiplying the first possibility by the first weight w1 and the value obtained by multiplying the second possibility by the second weight w2, and calculates the matching cost. The clustering unit 30 creates a matching cost matrix for each combination of feature points shown in FIG. 5. In FIG. 5, each feature point is indicated by "A" to "F", and the matching cost for each combination of feature points is indicated. For example, the matching cost of feature point "A" and feature point "B" is 16. The clustering unit 30 performs clustering by linking the combinations in ascending order of cost, and detects the clustered feature points as one object by combining feature points whose matching cost is equal to or less than a predetermined threshold. Hereinafter, a plurality of feature points clustered as one object will be referred to as a "cluster".

As described above, the first possibility (f3d(i,j)) and the second possibility (f2d(i,j)) are calculated as values that are smaller the more likely that feature point i and feature point j are feature points of the same object, and therefore the matching cost is smaller the more likely that feature point i and feature point j are feature points of the same object. However, for example, the first possibility may be the reciprocal (1/f3d(i,j)) and the second possibility may be the reciprocal (1/f2d(i,j)), and the first possibility and the second possibility may be values that are larger the more likely that feature point i and feature point j are feature points of the same object. Also, the sign of the first possibility may be a negative sign (-f3d(i,j)) and the sign of the second possibility may be a negative sign (-f2d(i,j)), and the first possibility and the second possibility may be values that are larger the more likely that feature point i and feature point j are feature points of the same object. Alternatively, the inverse of the value obtained by weighting and adding the first possibility (f3d(i, j)) and the second possibility (f2d(i, j)) may be used as the cost, so that the more likely it is that feature point i and feature point j are feature points of the same object, the higher the cost. In this case, the more likely it is that feature point i and feature point j are feature points of the same object, the higher the matching cost becomes. Therefore, the clustering unit 30 performs clustering by linking combinations in descending order of cost, and combines feature points whose matching cost is equal to or greater than a predetermined threshold. In this way, it is possible to detect the clustered feature points as a single object. Therefore, it is possible to appropriately change whether the first possibility and the second possibility are set to a small value or a large value as the more likely it is that feature point i and feature point j are feature points of the same object. In other words, it is sufficient that the first possibility and the second possibility are parameters that change depending on the possibility that feature point i and feature point j are feature points of the same object, and it is sufficient that the matching cost is a value obtained by weighting and adding the first possibility and the second possibility, and the value itself is not limited to being positive or negative.

Here, referring to FIG. 6A and FIG. 6B, the process of the clustering unit 30 clustering the feature points to detect an object will be described. FIG. 6A shows a cluster c1 in a case where a plurality of feature points ig are present in an area where the accuracy of position detection by the stereo camera 10 is good. The distance image di in FIG. 6A shows the distance in the front-rear direction of the host vehicle from the host vehicle to the plurality of feature points ig, and the distance between the feature points in real space. The plurality of feature points ig shown in the distance image di are present in an area where the accuracy of position detection by the stereo camera 10 is good. Therefore, the clustering unit 30 performs clustering c1 based on a matching cost matrix in which the first possibility is reflected more than the second possibility. This makes it possible to cluster the plurality of feature points ig to detect the object 3. FIG. 6B shows a cluster c2 in a case where a plurality of feature points ip are present in an area where the accuracy of position detection by the stereo camera 10 is reduced. The distance image di in FIG. 6B shows the distance in the front-rear direction of the host vehicle from the host vehicle to the feature points, and the distance between the feature points in real space. The multiple feature points ip shown in the distance image di are present in an area where the accuracy of position detection by the stereo camera 10 is reduced. Therefore, the clustering unit 30 performs clustering c2 based on a matching cost matrix in which the second possibility is reflected more than the first possibility. This makes it possible to detect the object 4 by clustering c2 the multiple feature points ip.

[Object detection method]
Next, an example of the processing of the object detection device shown in Fig. 1 will be described with reference to Fig. 7. The operation of the object detection device shown in the flowchart of Fig. 7 starts at the same time that the ignition switch or power switch of the vehicle is turned ON, and ends when the ignition switch or power switch is turned OFF.

In step S10, the feature point extraction unit 21 acquires two images of the area in front of the vehicle captured by the stereo camera 10 from different positions, and the process proceeds to step S20. In step S20, the feature point extraction unit 21 extracts multiple feature points, each of which is a pixel that has a characteristic that can be distinguished from the surrounding pixels, and the process proceeds to step S30.

In step S30, the distance image generating unit 22 calculates the three-dimensional coordinates (relative coordinates) of the multiple feature points relative to the vehicle from the parallax (shift amount) for the same feature points among the feature points in the first image and the feature points in the second image, and calculates the distances in the front-rear direction of the vehicle from the multiple feature points to the multiple feature points from the three-dimensional coordinates of the multiple feature points. Then, the process proceeds to step S40, where the distance image generating unit 22 calculates the distances between the multiple feature points in real space from the three-dimensional coordinates of the multiple feature points relative to the vehicle, and the process proceeds to step S50.

In step S50, the optical flow calculation unit 23 detects, as related feature points, current feature points that correspond to the same object in real space as the object corresponding to the past feature point, and calculates a combination of the mutually related past feature points and current feature points as an optical flow. The optical flow calculation unit 23 calculates the distance between multiple feature points in the image from the optical flow, and the process proceeds to step S60. In step S60, the optical flow calculation unit 23 calculates the movement speed of each feature point in the image from the optical flow, and the process proceeds to step S70.

In step S70, the three-dimensional information evaluation unit 28 filters the feature points. Specifically, the three-dimensional information evaluation unit 28 determines whether the distance between the multiple feature points in real space, taking into account the theoretical error that occurs when calculating the distance from the vehicle to the multiple feature points in the longitudinal direction of the vehicle, is within a predetermined allowable distance. Then, the three-dimensional information evaluation unit 28 prohibits feature points whose distance between the multiple feature points in real space, taking into account the theoretical error, is longer than the predetermined allowable distance from being clustered as feature points of the same object, and the process proceeds to step S80.

In step S80, the evaluation method determination unit 29 sets a first weight to be assigned to the first possibility and a second weight to be assigned to the second possibility based on the longitudinal distance from the vehicle to the multiple feature points, and the process proceeds to step S90.

In step S90, the three-dimensional information evaluation unit 28 calculates a first possibility indicating that the multiple feature points to which a first weight has been applied (multiplied) are feature points of the same object based on the distance between the multiple feature points in real space after filtering and the moving speed of each feature point in real space. The two-dimensional information evaluation unit 27 calculates a second possibility indicating that the multiple feature points to which a second weight has been applied (multiplied) are feature points of the same object based on the distance between the multiple feature points and the moving speed of each feature point. Then, the clustering unit 30 adds the first possibility and the second possibility to calculate a matching cost. The clustering unit 30 creates a matching cost matrix for each combination of feature points. Then, the process proceeds to step S100, where the clustering unit 30 performs clustering by linking the combinations in ascending order of cost, and detects the clustered feature points as one object by combining feature points whose matching cost is equal to or less than a predetermined threshold.

[Action and Effect]
As described above, according to this embodiment, the following advantageous effects can be obtained.

A first weight is assigned to a first possibility calculated based on at least the distance between the multiple feature points in real space, the first possibility indicating the possibility that the multiple feature points are feature points of the same object, and a second weight is assigned to a second possibility calculated based on the distance between the multiple feature points in the image and the moving speed of each feature point, the second possibility indicating the possibility that the multiple feature points are feature points of the same object, the first weight is assigned to the first possibility calculated based on the distance between the multiple feature points in the image and the moving speed of each feature point, and a second weight is assigned to the second possibility indicating the possibility that the multiple feature points are feature points of the same object, the second weight is assigned to the second possibility calculated based on the distance between the multiple feature points in the forward/rearward direction (predetermined direction) of the vehicle, and the multiple feature points are clustered to detect an object using the first possibility with the first weight and the second possibility with the second weight.

The first possibility is that multiple feature points calculated based on the distances between multiple feature points in real space are feature points of the same object, and this possibility is effective in areas where the accuracy of position detection by the stereo camera 10 is good. The second possibility is that multiple feature points calculated based on the distances between multiple feature points in an image and the moving speed of each feature point are feature points of the same object, and this possibility is effective in areas where the accuracy of position detection by the stereo camera 10 decreases. Therefore, by setting the first weight and the second weight based on the distances from the host vehicle to the multiple feature points in the forward/rearward direction of the host vehicle, clustering can be performed with high accuracy even in areas where the accuracy of position detection by the stereo camera decreases, and objects can be detected accurately.

The system determines whether the distance between multiple feature points in real space, taking into account theoretical errors that arise when calculating the longitudinal distance from the vehicle to the multiple feature points from the positions of the multiple feature points on two images, is within a predetermined allowable distance. Then, feature points whose distance between multiple feature points in real space taking into account theoretical errors is longer than the predetermined allowable distance are prohibited from being clustered as feature points of the same object. In this way, feature points whose distance between multiple feature points in real space taking into account theoretical errors is longer than the predetermined allowable distance are determined not to be feature points of the same object, and clustering is prohibited, thereby preventing erroneous clustering. In other words, objects can be detected with high accuracy.

It is determined whether the distance in the longitudinal direction of the vehicle from the vehicle to the multiple feature points is equal to or greater than a predetermined distance, and if the distance in the longitudinal direction is equal to or greater than the predetermined distance, the first weight is set to 0, and if the distance in the longitudinal direction is less than the predetermined distance, the second weight is set to 0. As a result, if the distance in the longitudinal direction of the vehicle from the vehicle to the multiple feature points is equal to or greater than a predetermined distance, it is possible to determine whether the multiple feature points are feature points of the same object using the second possibility that is effective in an area where the accuracy of position detection by the stereo camera 10 is reduced. Also, if the distance in the longitudinal direction of the vehicle from the vehicle to the multiple feature points is less than a predetermined distance, it is possible to determine whether the multiple feature points are feature points of the same object using the first possibility that is effective in an area where the accuracy of position detection by the stereo camera 10 is good. Therefore, it is possible to more accurately determine the possibility that the multiple feature points are feature points of the same object.

The first weight and the second weight are continuously changed based on the distance from the vehicle to the multiple feature points in the forward/backward direction of the vehicle and the angle the feature points make with respect to the straight-ahead direction of the vehicle (predetermined direction). This makes it possible to determine whether multiple feature points are feature points of the same object using both the first possibility and the second possibility while changing the first weight and the second weight in response to changes in the distance from the vehicle to the multiple feature points in the forward/backward direction of the vehicle and the angle the feature points make with respect to the straight-ahead direction of the vehicle. In other words, clustering can be performed with higher accuracy.

The first weight decreases as the distance from the host vehicle to the multiple feature points increases in the longitudinal direction of the host vehicle, and the second weight increases as the distance from the host vehicle to the multiple feature points increases in the longitudinal direction of the host vehicle. As a result, the influence of the first possibility when calculating the possibility that the multiple feature points are feature points of the same object can be reduced and the influence of the second possibility can be increased as the distance from the host vehicle to the multiple feature points increases in the longitudinal direction of the host vehicle. Therefore, the influence of the first possibility and the influence of the second possibility can be appropriately changed according to the distance from the host vehicle to the multiple feature points in the longitudinal direction of the host vehicle, and the possibility that the multiple feature points are feature points of the same object can be calculated. In other words, clustering can be performed with higher accuracy.

The first weight decreases as the angle of the feature point relative to the straight-line direction of the vehicle increases, and the second weight increases as the angle of the feature point relative to the straight-line direction of the vehicle increases. In position detection by the stereo camera 10, in the case of a feature point having an angle relative to the straight-line direction of the vehicle, the theoretical error in calculating the left-right position of the vehicle increases as the angle of the feature point relative to the straight-line direction of the vehicle increases. Therefore, the weights of the first possibility and the second possibility can be changed in accordance with the characteristic that the theoretical error in calculating the left-right position of the vehicle increases as the angle of the feature point relative to the straight-line direction of the vehicle of the stereo camera 10 increases. As a result, the influence of the first possibility and the influence of the second possibility can be appropriately changed according to the angle of the feature point relative to the straight-line direction of the vehicle, and the possibility that multiple feature points are feature points of the same object can be calculated. In other words, clustering can be performed with higher accuracy.

As described above, an embodiment of the present invention has been described, but the descriptions and drawings that form part of this disclosure should not be understood as limiting this invention. Various alternative embodiments, examples, and operating techniques will become apparent to those skilled in the art from this disclosure.

REFERENCE SIGNS LIST 1 Object detection device 10 Stereo camera 20 Control unit 21 Feature point extraction unit 22 Distance image generation unit 23 Optical flow calculation unit 24 Vehicle behavior measurement unit 25 Three-dimensional flow calculation unit 26 Object detection unit 27 Two-dimensional information evaluation unit 28 Three-dimensional information evaluation unit 29 Evaluation method determination unit 30 Clustering unit

Claims (7)

Obtaining two images by capturing images from different positions in a predetermined direction from the vehicle;
Extracting a plurality of feature points from each of the two images;
calculating distances from the vehicle to the plurality of feature points in the predetermined direction from the positions of the plurality of feature points on the two images, and distances between the plurality of feature points in real space;
a first weighting applied to a first possibility calculated based on at least a distance between the plurality of feature points in the real space, the first weighting applied to the first possibility indicating a possibility that the plurality of feature points are feature points of a same object, and a second weighting applied to the second possibility calculated based on a distance between the plurality of feature points in the image and a moving speed of each of the feature points, the second weighting applied to the second possibility indicating a possibility that the plurality of feature points are feature points of a same object ... calculated based on a distance between the plurality of feature points in the image and a moving speed of each of the feature points, the second weighting applied to the second possibility indicating a possibility that the plurality of feature points are feature points of a same object,
An object detection method, comprising: detecting an object by clustering the plurality of feature points using the first possibility with the first weighting and the second possibility with the second weighting. determining whether or not a distance between the plurality of feature points in the real space, taking into consideration a theoretical error that occurs when calculating a distance from the host vehicle to the plurality of feature points in the predetermined direction, from the positions of the plurality of feature points on the two images, is within a predetermined allowable distance;
2. The object detection method according to claim 1, wherein feature points whose distance between the plurality of feature points in the real space taking into account the theoretical error is longer than the predetermined allowable distance are prohibited from being clustered as feature points of the same object. determining whether a distance from the vehicle to the plurality of characteristic points in the predetermined direction is equal to or greater than a predetermined distance;
If the distance in the predetermined direction is equal to or greater than a predetermined distance, the first weight is set to 0;
The object detection method according to claim 1 , wherein the second weight is set to 0 when the distance in the predetermined direction is less than a predetermined distance. 2. The object detection method according to claim 1, wherein the first weights and the second weights are continuously changed based on distances from the host vehicle to the plurality of feature points in the predetermined direction. the first weight decreases as the distance in the predetermined direction increases,
The object detection method according to claim 4 , wherein the second weight increases as the distance in the predetermined direction increases. Calculating angles that the plurality of feature points make with respect to the predetermined direction; and the first weight is decreased as the angle increases.
The object detection method according to claim 4 , wherein the second weight increases as the angle increases. Two cameras that capture two images from different positions toward a predetermined direction from the vehicle;
A control unit,
The control unit is
Extracting a plurality of feature points from each of the two images;
calculating distances from the vehicle to the plurality of feature points in the predetermined direction from the positions of the plurality of feature points on the two images, and distances between the plurality of feature points in real space;
a first weighting applied to a first possibility calculated based on at least a distance between the plurality of feature points in the real space, the first weighting applied to the first possibility indicating a possibility that the plurality of feature points are feature points of a same object, and a second weighting applied to the second possibility calculated based on a distance between the plurality of feature points in the image and a moving speed of each of the feature points, the second weighting applied to the second possibility indicating a possibility that the plurality of feature points are feature points of a same object ... calculated based on a distance between the plurality of feature points in the image and a moving speed of each of the feature points, the second weighting applied to the second possibility indicating a possibility that the plurality of feature points are feature points of a same object,
An object detection device comprising: an object detection device that detects an object by clustering the plurality of feature points using the first possibility with the first weighting and the second possibility with the second weighting.

Images