JP7074571B2 - Target position measurement device and target position measurement program - Google Patents

Description

The present disclosure relates to a technique for measuring three-dimensional information to be measured.

As a technique for measuring three-dimensional information of a vehicle, there is a technique using a stereo camera or LiDAR (Laser Imaging Detection and Ranking). However, in the technique using a stereo camera, the distance measurement accuracy decreases as the distance from the stereo camera increases. Further, the technique using a stereo camera or LiDAR cannot cope with bad weather such as fog and rain because it is highly dependent on the weather.

Japanese Unexamined Patent Publication No. 2010-249613

By the way, as a technique disclosed in Patent Document 1, there is a technique of using a millimeter wave radar and a monocular camera in combination. Therefore, in the technique disclosed in Patent Document 1, the distance measurement accuracy is improved even when the distance is long from the radar device. Further, since the technique disclosed in Patent Document 1 has a small dependence on weather, it can cope with bad weather such as fog and rain. However, the technique disclosed in Patent Document 1 only detects the distance from the radar device to the front of the vehicle and the rectangle bordering the front of the vehicle, and cannot measure the three-dimensional information of the vehicle.

Therefore, in order to solve the above problems, the present disclosure aims to measure three-dimensional information to be measured by using a millimeter-wave radar or the like and a monocular camera or the like in combination.

Millimeter-wave radar and the like only measure two-dimensional information such as the width and depth of the measurement target. Monocular cameras and the like only measure two-dimensional information such as the width and height of the measurement target. In order to solve the above-mentioned problems, a millimeter-wave radar or the like and a monocular camera or the like are used in combination to complement each other's lacking dimensional information and measure three-dimensional information such as the width, depth and height of the measurement target. .. Then, not only the outer edge contour of the measurement target but also the in-plane contour of the measurement target is detected.

Specifically, in the present disclosure, a measurement target acquisition unit that acquires radar reflection point information and an image for a measurement target, a reflection point plotting unit that plots the radar reflection point on the image, and the above-mentioned on the image. A quadrangular pyramid detecting unit that detects a quadrangular pyramid that borders the measurement target on the image by performing edge analysis in the vicinity of the radar reflection point, and in each plane of the quadrangular pyramid on the image. By executing the edge analysis in, the target contour detection unit that detects the contour of the measurement target on the image and the position conversion that converts the position of the contour of the measurement target on the image to the position in the real space. It is a target position measuring device characterized by having a unit and a unit.

According to this configuration, by using a millimeter-wave radar or the like and a monocular camera or the like together, it is possible to measure three-dimensional information such as the width, depth and height of the measurement target. Then, by detecting not only the outer edge contour of the measurement target but also the in-plane contour of the measurement target, it is possible to recognize the surrounding environment corresponding to automatic operation such as rubbing and slipping. Instead of a radar that only measures two-dimensional information such as the width and depth of the measurement target, a radar that can measure three-dimensional information such as the width, depth and height of the measurement target may be used. Also, instead of a monocular camera that only measures two-dimensional information such as the width and height of the measurement target, a stereo camera that can measure three-dimensional information such as the width, height and depth of the measurement target can also be used. good.

Further, in the present disclosure, the target contour detection unit divides each surface of the quadrangular pyramid into a plurality of grids, performs edge analysis in each grid, and performs edge analysis in each grid of the contour of the measurement target in each grid. A part is detected, a part of the contour of the measurement target is converted into a representative point in each grid, and the position conversion unit converts the position of the representative point into a position in real space in each grid. It is a target position measuring device characterized by performing.

According to this configuration, the technique of the present disclosure is applied to static information such as a three-dimensional environment map generated by LiDAR or the like by converting the outer edge / in-plane contour of the measurement target into a contour point cloud of the measurement target. Dynamic information such as vehicles and pedestrians generated by can be plotted. Then, even if the outer edge / in-plane contour of the measurement target has a complicated shape in the entire measurement target, the outer edge / in-plane contour of the measurement target can be easily converted into the contour point cloud of the measurement target in each grid. can. Further, even if the outer edge / in-plane contour of the measurement target has a complicated shape in the entire measurement target, the outer edge / in-plane contour of the measurement target can be easily detected in each grid.

Further, in the present disclosure, the target contour detection unit performs edge analysis in each surface of the quadrangular pyramid, detects the contour of the measurement target in each surface of the quadrangular frustum, and detects the contour of the measurement target. Each surface of is divided into a plurality of grids, a part of the contour of the measurement target is converted into a representative point in each grid, and the position conversion unit converts the position of the representative point in each grid into a real space. It is a target position measuring device characterized by converting to an internal position.

According to this configuration, the technique of the present disclosure is applied to static information such as a three-dimensional environment map generated by LiDAR or the like by converting the outer edge / in-plane contour of the measurement target into a contour point cloud of the measurement target. Dynamic information such as vehicles and pedestrians generated by can be plotted. Then, even if the outer edge / in-plane contour of the measurement target has a complicated shape in the entire measurement target, the outer edge / in-plane contour of the measurement target can be easily converted into the contour point cloud of the measurement target in each grid. can. Further, if the outer edge / in-plane contour of the measurement target includes a linear portion in a part of the measurement target, the outer edge / in-plane contour of the measurement target can be detected at high speed in each surface of the quadrangular pyramid.

Further, in the present disclosure, the target contour detection unit determines the number of grids that divide each surface of the quadrangular pyramid according to the degree of danger of the measurement target as seen from the own vehicle equipped with the radar device and the image pickup device. It is a target position measuring device characterized by being set.

According to this configuration, if the risk of the measurement target is high, the contour point cloud of the measurement target can be detected in detail by increasing the number of grids that divide each surface of the quadrangular pyramid, resulting in friction and friction. It is possible to recognize the surrounding environment that can handle automatic driving such as slipping through. On the other hand, if the risk of the measurement target is low, the contour point cloud of the measurement target can be detected at high speed by reducing the number of grids that divide each surface of the quadrangular pyramid, and the measurement has a low risk. It is possible not to wastefully detect the contour point cloud of the measurement target for the target.

Further, in the present disclosure, the target contour detection unit determines the number of grids that divide each surface of the quadrangular pyramid according to the distance between the own vehicle and the measurement target measured by the radar device. It is a target position measuring device characterized by being set.

According to this configuration, the degree of danger of the measurement target can be determined based on the distance between the own vehicle and the measurement target, and the detection accuracy of the contour point cloud of the measurement target can be set.

Further, in the present disclosure, the target contour detection unit is a grid that divides each surface of the quadrangular pyramid according to the difference in traveling speed between the own vehicle and the measurement target measured by the radar device. It is a target position measuring device characterized in that the number is set.

According to this configuration, the degree of danger of the measurement target can be determined based on the difference in traveling speed between the own vehicle and the measurement target, and the detection accuracy of the contour point cloud of the measurement target can be set.

Further, in the present disclosure, the target contour detection unit is a grid that divides each surface of the quadrangular pyramid according to the difference in the traveling direction between the own vehicle and the measurement target measured by the radar device. It is a target position measuring device characterized in that the number is set.

According to this configuration, the degree of danger of the measurement target can be determined based on the difference in the traveling direction between the own vehicle and the measurement target, and the detection accuracy of the contour point cloud of the measurement target can be set.

Further, in the present disclosure, the target contour detection unit sets the number of grids that divide each surface of the quadrangular pyramid according to the approach time of the own vehicle to the measurement target measured by the radar device. It is a target position measuring device characterized by performing.

According to this configuration, the degree of danger of the measurement target can be determined based on the approach time of the own vehicle to the measurement target, and the detection accuracy of the contour point cloud of the measurement target can be set.

Further, the present disclosure includes a measurement target acquisition step for acquiring radar reflection point information and an image for a measurement target, a reflection point plot step for plotting the radar reflection point on the image, and the radar reflection point on the image. A quadrangular pyramid detection step for detecting a quadrangular pyramid that borders the measurement target on the image by performing an edge analysis in the vicinity of the image, and an edge in each surface of the quadrangular pyramid on the image. A target contour detection step of detecting the contour of the measurement target on the image by executing an analysis, and a position conversion step of converting the position of the contour of the measurement target on the image to a position in the real space. This is a target position measurement program for causing a computer to execute the above in order.

According to this configuration, by using a millimeter-wave radar or the like and a monocular camera or the like together, it is possible to measure three-dimensional information such as the width, depth and height of the measurement target. Then, by detecting not only the outer edge contour of the measurement target but also the in-plane contour of the measurement target, it is possible to recognize the surrounding environment corresponding to automatic operation such as rubbing and slipping. Instead of a radar that only measures two-dimensional information such as the width and depth of the measurement target, a radar that can measure three-dimensional information such as the width, depth and height of the measurement target may be used. Also, instead of a monocular camera that only measures two-dimensional information such as the width and height of the measurement target, a stereo camera that can measure three-dimensional information such as the width, height and depth of the measurement target can also be used. good.

As described above, in the present disclosure, three-dimensional information to be measured can be measured by using a millimeter-wave radar or the like and a monocular camera or the like in combination.

It is a block diagram which shows the structure of the object position measurement system of this disclosure. It is a flowchart which shows the processing procedure of the target position measuring apparatus of this disclosure. It is a figure which shows the processing content of the measurement target acquisition of this disclosure. It is a figure which shows the processing content of the reflection point plot and the quadrangular pyramid detection of this disclosure. It is a figure which shows the processing content of the 1st target contour detection of this disclosure. It is a figure which shows the processing content of the 2nd target contour detection of this disclosure. It is a figure which shows the processing content of the position conversion of this disclosure. It is a figure which shows the detection result of the contour point cloud of one vehicle of this disclosure. It is a figure which shows the detection result of the contour point cloud of one pedestrian of this disclosure. It is a figure which shows the image of the three vehicles of this disclosure. It is a figure which shows the detection result of the contour point cloud of three vehicles of this disclosure. It is a figure which shows the detection result of the prior art and the contour point cloud of this disclosure. It is a figure which shows the detection result of the contour point cloud with various grid divisions of this disclosure. It is a figure which shows the setting method of the 1st grid division number of this disclosure. It is a figure which shows the setting method of the 2nd grid division number of this disclosure. It is a figure which shows the setting method of the 3rd grid division number of this disclosure. It is a figure which shows the setting method of the 4th grid division number of this disclosure.

Embodiments of the present disclosure will be described with reference to the accompanying drawings. The embodiments described below are examples of the embodiments of the present disclosure, and the present disclosure is not limited to the following embodiments.

(Overview of target position measurement system)
FIG. 1 shows a block diagram showing the configuration of the target position measurement system of the present disclosure. FIG. 2 shows a flowchart showing the processing procedure of the target position measuring device of the present disclosure. The target position measurement system S includes a radar device 1, an image pickup device 2, and a target position measurement device 3. The target position measurement device 3 is a computer in which the target position measurement program shown in FIG. 2 is installed, and is a measurement target acquisition unit 31, a reflection point plot unit 32, a quadrangular pyramid base detection unit 33, a target contour detection unit 34, and a position. It is composed of a conversion unit 35.

The radar device 1 is a MIMO (Multiple Input-Multiple Input) millimeter-wave radar or the like installed inside a bumper of the own vehicle. The image pickup device 2 is a monocular camera or the like installed on the upper part of the windshield of the own vehicle or inside the bumper of the own vehicle. The target position measuring device 3 measures three-dimensional information of the target position by using the radar device 1 and the imaging device 2 together. Then, not only the outer edge contour of the measurement target but also the in-plane contour of the measurement target is detected.

FIG. 3 shows the processing content of the measurement target acquisition of the present disclosure. The measurement target acquisition unit 31 acquires information and an image of the radar reflection point for the measurement target (step S1).

Information on the radar cross section from the measurement target is acquired by the beamformer method, the MUSIC (Multiple Signal Classification) method, or the like. The measurement target acquisition unit 31 has acquired information on a plurality of white lines L. Vehicle C1 is detected as a cluster of a plurality of radar reflection points in the adjacent lane on the left side when viewed from the antenna A of the radar device 1, and vehicle C2 is detected as a cluster of a plurality of radar reflection points in the same lane. , Vehicle C3 is detected as a cluster of a plurality of radar reflection points in the adjacent lane on the right side.

The image to be measured is acquired by a monocular camera or the like. When viewed from the sensor of the image pickup device 2, a plurality of white lines L are detected in the forward direction, vehicle C1 is detected in the adjacent lane on the left side, vehicle C2 is detected in the same lane, and vehicle C2 is detected in the same lane. Vehicle C3 is detected in the adjacent lane.

Since the radar device 1 cannot afford to install the plurality of antennas A apart from each other in the height direction, the radar device 1 only measures two-dimensional information such as the width X [m] and the depth Z [m] of the measurement target. Since the image pickup device 2 cannot afford to install a plurality of sensors separated in the width direction, it only measures two-dimensional information such as the width x [pix] and the height y [pix] of the measurement target. Therefore, the target position measuring device 3 uses the radar device 1 and the imaging device 2 in combination to complement each other's lacking dimensional information, and the width X [m], depth Z [m], and height of the measurement target. Three-dimensional information such as Y [m] is measured. Then, not only the outer edge contour of the measurement target but also the in-plane contour of the measurement target is detected.

(Processing content of quadrangular pyramid detection)
FIG. 4 shows the processing contents of the reflection point plot and the quadrangular pyramid detection of the present disclosure. The reflection point plotting unit 32 plots radar reflection points on the image (step S2).

In the radar reflection point information from the vehicle C1, a plurality of radar reflection points P1, P2, P3 are detected on the front surface of the vehicle C1, and a plurality of radar reflection points P4, P5, P6, are detected on the side surface of the vehicle C1. P7 is detected. In the image of the vehicle C1, a plurality of radar reflection points P1 to P7 are plotted in the vicinity of the vehicle C1. Here, in contrast to the formula for converting the position on the image shown in FIG. 7 to the position in the real space, a plurality of formulas for converting the position in the real space, which is the reverse processing, to the position on the image are used. Radar reflection points P1 to P7 are plotted on the image of vehicle C1.

In the image of the vehicle C1, the reflection point plotting unit 32 assumes that a plurality of radar reflection points P1 to P7 are on the road surface. This is because the heights of the plurality of radar reflection points P1 to P7 are unknown around the installation height of the radar device 1. Therefore, it is assumed that the heights of the plurality of radar reflection points P1 to P7 are equal to the height of the road surface (close to the installation height of the radar device 1).

The quadrangular pyramid detection unit 33 detects the quadrangular frustum that borders the measurement target on the image by performing edge analysis in the vicinity of the radar reflection point on the image (step S3).

In the first process, the quadrangular frustum detection unit 33 borders the vehicle C1 on the image based on a plurality of radar reflection points P1 to P7 assuming that the reflection point plotting unit 32 is on the road surface. The sides R1 and R2 on the bottom surface of the surface are detected. Specifically, the quadrangular pyramid detection unit 33 calculates approximate straight lines of a plurality of radar reflection points P1 to P3 by the minimum square method or the like, and calculates approximate straight lines of a plurality of radar reflection points P1 to P3 of the quadrangular pyramid stand R. It is detected as the side R1 on the bottom surface. Then, the quadrangular pyramid detection unit 33 calculates the approximate straight lines of the plurality of radar reflection points P4 to P7 by the least squares method or the like, and the approximate straight lines of the plurality of radar reflection points P4 to P7 are the sides of the bottom surface of the quadrangular pyramid base R. Detected as R2.

As described above, in the image of the vehicle C1, the vehicle C1 is bordered by assuming that the heights of the plurality of radar reflection points P1 to P7 are equal to the height of the road surface (close to the installation height of the radar device 1). The sides R1 and R2 on the bottom surface of the square pyramid R can be easily detected. Unlike the edge analysis of other positions of the vehicle C1, the lower edge analysis of the vehicle C1 is not executed because the space between the lower edge of the vehicle C1 and the road surface is about the radius of the tire. This is because the height of the vehicle C1 cannot be measured with high accuracy (see the third equation in FIG. 7).

In the second process, the quadrangular pyramid detection unit 33 and the front surface of the quadrangular pyramid R are based on a straight line extending in the height direction on the image from the intersection of the two sides R1 and R2 on the bottom surface of the quadrangular pyramid R. The side R3 at the boundary with the side surface is detected.

In the third process, the quadrangular pyramid detection unit 33 performs edge analysis of the quadrangular pyramid R at a predetermined distance in the height direction on the image from the sides R1 and R2 of the bottom surface of the quadrangular pyramid R. The sides R4 and R5 on the upper surface are detected.

Specifically, the square cone detection unit 33 widens the range y _E [pix] of a predetermined distance in the height direction on the image on which the edge analysis is performed as the distance from the radar device 1 to the vehicle C1 becomes shorter. The farther the distance from the radar device 1 to the vehicle C1, the narrower the range y _E [pix] of the predetermined distance in the height direction on the image on which the edge analysis is performed. Here, the minimum heights of the edge analysis ranges E4 and E5 for detecting the sides R4 and R5 are set based on, for example, the minimum height of the current light vehicle and the third coordinate conversion formula shown in FIG. The maximum heights of the edge analysis ranges E4 and E5 are set based on the maximum height regulation stipulated by law and the third coordinate conversion formula shown in FIG.

Then, the quadrangular frustum detection unit 33 can easily determine the sides R4 and R5 of the upper surface of the quadrangular frustum R by applying the Canny method or the like for detecting a long edge in the width direction only in the edge analysis ranges E4 and E5. Can be detected. The quadrangular frustum detection unit 33 detects the sides R4 and R5 of the upper surface of the quadrangular frustum R by measuring the brightness gradient in the height direction at various width coordinate positions only in the edge analysis ranges E4 and E5. You can also do it.

In the fourth process, the quadrangular pyramid detection unit 33 performs edge analysis at a predetermined distance in the width direction on the image from the side R3 of the boundary between the front surface and the side surface of the quadrangular pyramid R, thereby performing the quadrangular pyramid base. Of the four sides of the front surface of R, the side R6 facing the side R3 of the boundary between the front surface and the side surface of the quadrangular pyramid R is detected.

Specifically, the square cone detection unit 33 widens the range x _E [pix] of a predetermined distance in the width direction on the image on which the edge analysis is performed as the distance from the radar device 1 to the vehicle C1 becomes shorter. The farther the distance from the radar device 1 to the vehicle C1, the narrower the range x _E [pix] of the predetermined distance in the width direction on the image on which the edge analysis is performed. Here, the minimum width coordinates of the edge analysis range E6 for detecting the side R6 are set based on, for example, the minimum width of the current light vehicle and the second coordinate conversion formula shown in FIG. 7, and the edge analysis range E6. The maximum width coordinates are set based on the maximum width regulation stipulated by law and the second coordinate conversion formula shown in FIG. 7.

Then, the quadrangular frustum detection unit 33 and the side R3 of the boundary between the front surface and the side surface of the quadrangular frustum R by applying the Canny method or the like for detecting a long edge in the height direction only in the edge analysis range E6. The opposite sides R6 can be easily detected. The quadrangular pyramid detection unit 33 may detect the other side R6 in front of the quadrangular frustum R by measuring the brightness gradient in the width direction at various height coordinate positions only in the edge analysis range E6. can.

In the final process, the quadrangular pyramid detection unit 33 performs edge analysis at a predetermined distance in the depth direction on the image from the frontmost radar reflection point P7 of the vehicle C1 to perform edge analysis on the four sides of the side surface of the quadrangular frustum R. Of these, the side R7 facing the side R3 at the boundary between the front surface and the side surface of the quadrangular pyramid R is detected.

Specifically, the square cone detection unit 33 widens the range z _E [pix] of a predetermined distance in the depth direction on the image on which the edge analysis is performed as the distance from the radar device 1 to the vehicle C1 becomes shorter. The farther the distance from the radar device 1 to the vehicle C1, the narrower the range z _E [pix] of the predetermined distance in the depth direction on the image on which the edge analysis is performed. Here, the minimum depth and the maximum depth of the edge analysis range E7 for detecting the side R7 are set so that the side R7 can be reliably detected.

Then, the quadrangular pyramid detection unit 33 measures the brightness gradient in the depth direction at various height coordinate positions only in the edge analysis range E7, so that the side R3 of the boundary between the front surface and the side surface of the quadrangular frustum R The opposite sides R7 can be reliably detected. The quadrangular frustum detection unit 33 may also detect the other side R7 on the side surface of the quadrangular frustum R by applying the Canny method or the like for detecting a long edge in the height direction only in the edge analysis range E7. can.

(Processing content of target contour detection)
The target contour detection unit 34 detects the contour of the measurement target on the image by performing edge analysis in each surface of the quadrangular pyramid on the image (step S4). Specifically, the target contour detection unit 34 converts the contour line segment of the measurement target into a contour point group of the measurement target.

FIG. 5 shows the processing content of the first target contour detection of the present disclosure. First, the target contour detection unit 34 divides each surface of the quadrangular pyramid R into a plurality of grids. From the left column in the upper part of FIG. 5 to the middle column in the upper part of FIG. 5, the front surface and the side surface of the quadrangular pyramid R are each divided into 5 × 5 grids. The number of grid divisions will be described later with reference to FIGS. 13 to 17.

Next, the target contour detection unit 34 executes edge analysis in each grid and detects a part of the contour of the vehicle C1 in each grid. In the upper right column of FIG. 5, in the upper left 1 grid of the front of the quadrangular pyramid R, the edge analysis is executed after the background removal and the enhancement processing are executed, and the upper left contour of the front of the vehicle C1 is It has been detected. The edge analysis may be executed by applying the Canny method or the like, may be executed by measuring the luminance gradient, or may be executed so as to detect curved edges at the subpixel level. good.

Next, the target contour detection unit 34 converts a part of the contour of the vehicle C1 into a representative point in each grid. In the lower left column of FIG. 5, in the upper left 1 grid on the front of the quadrangular pyramid R, the upper left contour on the front of the vehicle C1 is converted into a representative point. As a representative point, a part of the contour of the vehicle C1 may be the center of gravity of a polygon or the center of a line segment. Further, the number of representative points may be singular or plural in each grid.

In the description of FIG. 5, simple serial processing is performed for edge analysis, contour detection, and representative point conversion in each grid, but high-speed parallel processing may be performed.

In this way, even if the outer edge / in-plane contour of the measurement target has a complicated shape in the entire measurement target, since it has a simple shape in each grid, the outer edge / in-plane contour of the measurement target in each grid. Can be easily converted into a contour point cloud to be measured. Furthermore, even if the outer edge / in-plane contour of the measurement target has a complicated shape in the entire measurement target, since it has a simple shape in each grid, the outer edge / in-plane contour of the measurement target can be easily obtained in each grid. Can be detected.

FIG. 6 shows the processing content of the second target contour detection of the present disclosure. First, the target contour detection unit 34 executes edge analysis in each surface of the quadrangular pyramid R, and detects the contour of the vehicle C1 in each surface of the quadrangular frustum R. From the upper left column of FIG. 6 to the upper right column of FIG. 6, edge analysis is executed on the front and side surfaces of the quadrangular pyramid R after background removal and enhancement processing are performed, and the front of the vehicle C1. And the contour of the side surface is detected. The edge analysis may be executed by applying the Canny method or the like, may be executed by measuring the luminance gradient, or may be executed so as to detect curved edges at the subpixel level. good.

Next, the target contour detection unit 34 divides each surface of the quadrangular pyramid R into a plurality of grids. In the lower left column of FIG. 6, the front surface and the side surface of the quadrangular pyramid R are each divided into 5 × 5 grids. The number of grid divisions will be described later with reference to FIGS. 13 to 17.

Next, the target contour detection unit 34 converts a part of the contour of the vehicle C1 into a representative point in each grid. In the lower right column of FIG. 6, in the front and side grids of the quadrangular pyramid R, the contours of the front and side surfaces of the vehicle C1 are converted into representative points. As a representative point, a part of the contour of the vehicle C1 may be the center of gravity of a polygon or the center of a line segment. Further, the number of representative points may be singular or plural in each grid.

In the description of FIG. 6, a simple serial process may be executed or a high-speed parallel process may be executed for the representative point conversion in each grid.

In this way, even if the outer edge / in-plane contour of the measurement target has a complicated shape in the entire measurement target, since it has a simple shape in each grid, the outer edge / in-plane contour of the measurement target in each grid. Can be easily converted into a contour point cloud to be measured. Furthermore, if the outer edge / in-plane contour of the measurement target includes a linear part in a part of the measurement target, a rough edge analysis in each surface of the quadrangular frustum is sufficient, and the measurement target is in each surface of the quadrangular frustum. It is possible to detect the outer edge and in-plane contour of the surface at high speed.

The processing content of the position conversion of the present disclosure is shown in FIG. The position conversion unit 35 converts the position of the contour of the measurement target on the image into a position in the real space (step S5). Specifically, the position conversion unit 35 converts the position of the representative point in each grid to the position in the real space.

The coordinates of the intersection of the sides R1 and R6 are set to (x ₁ [pix], y ₁ [pix]) on the image, and (X ₁ [m], Y ₁ [m], Z ₁ [m] in the real space. ). The coordinates of the intersection of the sides R2 and R7 are set to (x ₂ [pix], y ₂ [pix]) on the image, and (X ₂ [m], Y ₂ [m], Z ₂ [m] in the real space. ). The coordinates of the intersections of the sides R1, R2, and R3 are set to (x ₃ [pix], y ₃ [pix]) on the image, and (X ₃ [m], Y ₃ [m], Z ₃ [ m]). The coordinates of the intersections of the sides R3, R4, and R5 are set to (x ₄ [pix], y ₄ [pix]) on the image, and (X ₄ [m], Y ₄ [m], Z ₄ [ m]). The coordinates of the intersection of the sides R4 and R6 are set to (x ₅ [pix], y ₅ [pix]) on the image, and (X ₅ [m], Y ₅ [m], Z ₅ [m] in the real space. ). The coordinates of the intersection of the sides R5 and R7 are set to (x ₆ [pix], y ₆ [pix]) on the image, and (X ₆ [m], Y ₆ [m], Z ₆ [m] in the real space. ). The coordinates of the representative points in each grid are (x _E [pix], y _E [pix]) on the image, and (X _E [m], Y _E [m], _ZE [m] in the real space. ]). The coordinates of the vanishing point V of the image (for example, the intersection of the extension lines of the sides R2 and R5, or the intersection of the extension lines of the plurality of white lines L) are set on the image (x _V [pix], y _V [pix]). And.

x ₁ to x ₆ and x _E are represented by x, y ₁ to y ₆ and y _E are represented by y, X ₁ to X ₆ and X _E are represented by X, and Y ₁ to Y ₆ and Y _E are represented. Is represented by Y, Z ₁ to Z ₆ and _ZE are represented by Z. The coordinates (x [pix], y [pix]) on the image are converted into the coordinates (X [m], Y [m], Z [m]) in the real space by the following mathematical formula. The coordinates (X _E [m], Y _E [m], _ZE [m]) of the representative points in each grid in the real space may be measured by the following mathematical formulas, and each side R1 It may be measured by interpolation of coordinates on the image of the intersection of ~ R7 and in real space.
Z [m] = (focal length [mm] x number of pixels in the height direction [pix] x sensor installation height [cm]) / ((y [pix] -y _V [pix]) x sensor size [mm] ] × 100)
X [m] = (x [pix] -x _V [pix]) x resolution [m / pix] at Z [m]
Y [m] = (y [pix] -y _V [pix]) x resolution [m / fix] at Z [m]

(Summary of target position measurement system)
As described above, by using the radar device 1 and the image pickup device 2 together, it is possible to measure three-dimensional information such as the width X [m], the depth Z [m], and the height Y [m] of the measurement target. can. Then, by detecting not only the outer edge contour of the measurement target but also the in-plane contour of the measurement target, it is possible to recognize the surrounding environment that can cope with automatic operation such as rubbing and slipping.

Further, by converting the outer edge / in-plane contour of the measurement target into the contour point cloud of the measurement target, static information such as a three-dimensional environment map generated by LiDAR or the like is generated by the technique of the present disclosure. Dynamic information such as vehicles and pedestrians can be plotted. Then, by converting the outer edge / in-plane contour of the measurement target into the contour point cloud of the measurement target, it becomes easier to machine-learn the correspondence between the contour point cloud and the target type, and information on the target type and distance / velocity is generated. It will be easier to do.

Instead of a radar that only measures two-dimensional information such as the width X [m] and the depth Z [m] of the measurement target, the width X [m], the depth Z [m], and the height Y [of the measurement target A radar capable of measuring three-dimensional information such as [m] may be used.

Further, instead of the monocular camera that only measures two-dimensional information such as the width x [pix] and the height y [pix] of the measurement target, the width x [pix], the height y [pix], and the depth of the measurement target are replaced. A stereo camera capable of measuring three-dimensional information such as z [pix] may be used.

(Detection result of contour point cloud)
FIG. 8 shows the detection result of the contour point cloud of one vehicle of the present disclosure. First, the quadrangular pyramid R is detected for the vehicle C1. Next, contour point clouds are detected on the front surface and the side surface of the quadrangular pyramid R. Then, the contours of the outer edge, the window frame, and the like are reproduced for the vehicle C1.

The detection result of the contour point cloud of one pedestrian of this disclosure is shown in FIG. First, the quadrangular pyramid R is detected for the pedestrian P. Next, a contour point cloud is detected in front of the quadrangular pyramid R. The contours of the limbs, torso, and the like are reproduced for the pedestrian P.

Images of the three vehicles of the present disclosure are shown in FIG. In the following, it was investigated whether the contours of the outer edge, the window frame, etc. were reproduced not only for one vehicle C1 but also for three vehicles C4, C5, and C6.

FIG. 11 shows the detection results of the contour point cloud of the three vehicles of the present disclosure. The upper part of FIG. 11 shows the detection results of the contour point cloud of the vehicles C4, C5, and C6 as seen from the left side. The lower part of FIG. 11 shows the detection results of the contour point cloud of the vehicles C4, C5, and C6 as seen from the right side. For vehicles C4 and C6 at a short distance from the radar device 1 and in the adjacent lane to the own vehicle, contours such as outer edges and window frames are reproduced on the front and side surfaces of the quadrangular pyramid. For the vehicle C5, which is a long distance from the radar device 1 and is in the same lane as the own vehicle, the contours of the outer edge, the window frame, and the like are reproduced in front of the quadrangular pyramid.

FIG. 12 shows the detection results of the prior art and the contour point cloud of the present disclosure. The upper part of FIG. 12 shows the detection results of the contour point cloud of the vehicles C4, C5, and C6 using LiDAR to which the laser is applied. The lower part of FIG. 12 shows the detection results of the contour point cloud of the vehicles C4, C5, and C6 using the invention of the present disclosure that integrates the radar and the image. When LiDAR is used, the accuracy of reproducing the contours of the outer edge, window frame, etc. of the vehicles C4, C5, and C6 is low. When the invention of the present disclosure is used, the accuracy of reproducing the contours of the outer edge, the window frame, etc. of the vehicles C4, C5, and C6 is high.

(How to set the number of grid divisions)
FIG. 13 shows the detection results of the contour point cloud at various grid division numbers of the present disclosure. First, the quadrangular pyramid R is detected for the vehicle C1. Next, contour point clouds are detected on the front surface and the side surface of the quadrangular pyramid R. Then, the contours of the outer edge, the window frame, and the like are reproduced for the vehicle C1. However, the accuracy with which contours such as the outer edge and the window frame are reproduced changes depending on the number of grid divisions on each surface of the quadrangular pyramid R. First, when the number of grid divisions on each surface of the quadrangular pyramid R is 5 in the vertical direction and 5 in the horizontal direction, the accuracy of reproducing the contours of the outer edge and the window frame is low. Next, when the number of grid divisions on each surface of the quadrangular pyramid R is 10 in the vertical direction and 10 in the horizontal direction, the accuracy of reproducing the contours of the outer edge, the window frame, and the like is medium. When the number of grid divisions on each surface of the quadrangular pyramid R is 20 in the vertical direction and 20 in the horizontal direction, the accuracy of reproducing the contours of the outer edge, the window frame, and the like is high.

Therefore, the target contour detection unit 34 sets the number of grids that divide each surface of the quadrangular pyramid R according to the degree of danger of the measurement target as seen from the own vehicle equipped with the radar device 1 and the image pickup device 2. For example, if the risk of the measurement target is high, the contour point cloud of the measurement target can be detected in detail by increasing the number of grids that divide each surface of the quadrangular pyramid R, such as rubbing and slipping. It is possible to recognize the surrounding environment that can be used for automatic driving. On the other hand, if the risk of the measurement target is low, the contour point cloud of the measurement target can be detected at high speed by reducing the number of grids that divide each surface of the quadrangular pyramid R, and the risk is low. It is possible not to wastefully detect the contour point cloud of the measurement target for the measurement target.

FIG. 14 shows a method of setting the first grid division number of the present disclosure. The target contour detection unit 34 sets the number of grids that divide each surface of the quadrangular pyramid R according to the distance between the own vehicle and the measurement target measured by the radar device 1. In the left column of FIG. 14, since the distance between the own vehicle C7 and the other vehicle C8 is short, there is a high possibility that the own vehicle C7 collides with the other vehicle C8. , Increase the number of grids that divide each surface of the quadrangular pyramid R (for example, vertical 5 × horizontal 5). In the right column of FIG. 14, since the distance between the own vehicle C7 and the other vehicle C8 is long, the possibility that the own vehicle C7 collides with the other vehicle C8 is low. , The number of grids that divide each surface of the quadrangular pyramid R is reduced (for example, vertical 3 × horizontal 3). In this way, the degree of danger of the measurement target can be determined based on the distance between the own vehicle and the measurement target, and the detection accuracy of the contour point cloud of the measurement target can be set.

FIG. 15 shows a method of setting the second grid division number of the present disclosure. The target contour detection unit 34 sets the number of grids that divide each surface of the quadrangular pyramid R according to the difference in traveling speed between the own vehicle and the measurement target measured by the radar device 1. In the left column of FIG. 15, since the high-speed vehicle C7, which is the own vehicle, follows the low-speed vehicle C8, which is another vehicle, there is a high possibility that the vehicle C7, which is the own vehicle, collides with the vehicle C8, which is another vehicle. Increase the number of grids that divide each surface of the quadrangular pyramid R (for example, vertical 5 × horizontal 5). In the right column of FIG. 15, since the vehicle C7, which is the own vehicle, follows the vehicle C8 having the same speed as the other vehicle, the possibility that the vehicle C7, which is the own vehicle, collides with the vehicle C8, which is the other vehicle, is low. , The number of grids that divide each surface of the quadrangular pyramid R is reduced (for example, vertical 3 × horizontal 3). In this way, the degree of danger of the measurement target can be determined based on the difference in traveling speed between the own vehicle and the measurement target, and the detection accuracy of the contour point cloud of the measurement target can be set.

FIG. 16 shows a method of setting the third grid division number of the present disclosure. The target contour detection unit 34 sets the number of grids that divide each surface of the quadrangular pyramid R according to the difference in the traveling direction between the own vehicle and the measurement target measured by the radar device 1. In the left column of FIG. 16, since the vehicle C7, which is the own vehicle, heads for an intersection and the vehicle C8, which is another vehicle, also heads for the intersection, the vehicle C7, which is the own vehicle, may collide with the vehicle C8, which is another vehicle. The property is increased, and the number of grids that divide each surface of the quadrangular pyramid R is increased (for example, vertical 5 × horizontal 5). In the right column of FIG. 16, the vehicle C7, which is the own vehicle, is heading toward an intersection, but the vehicle C8, which is another vehicle, is away from the intersection, so that the vehicle C7, which is the own vehicle, collides with the vehicle C8, which is another vehicle. The number of grids that divide each surface of the quadrangular pyramid R is reduced (for example, 3 vertical × 3 horizontal). In this way, the degree of danger of the measurement target can be determined based on the difference in the traveling direction between the own vehicle and the measurement target, and the detection accuracy of the contour point cloud of the measurement target can be set.

FIG. 17 shows a method of setting the fourth grid division number of the present disclosure. The target contour detection unit 34 sets the number of grids that divide each surface of the quadrangular pyramid R according to the approach time of the own vehicle to the measurement target measured by the radar device 1. In the left column of FIG. 17, the distance between the own vehicle C7 and the other vehicle C8 is short, and the own high-speed vehicle C7 follows the other low-speed vehicle C8, and the own vehicle. Since the time for the vehicle C7 to approach the vehicle C8 which is another vehicle is short, the possibility that the vehicle C7 which is the own vehicle collides with the vehicle C8 which is another vehicle is high, and each surface of the quadrangular pyramid R is divided. Increase the number of grids to be used (for example, 5 vertical × 5 horizontal). In the right column of FIG. 17, the distance between the vehicle C7 which is the own vehicle and the vehicle C8 which is another vehicle is long, and the vehicle C7 which is the own vehicle follows the vehicle C8 which is the other vehicle and has the same speed. Since the vehicle C7, which is a vehicle, has a long time to approach the vehicle C8, which is another vehicle, the possibility that the vehicle C7, which is the own vehicle, collides with the vehicle C8, which is another vehicle, is low, and each surface of the quadrangular pyramid R is reduced. Reduce the number of grids to be divided (for example, 3 vertical × 3 horizontal). In this way, the degree of danger of the measurement target can be determined based on the approach time of the own vehicle to the measurement target, and the detection accuracy of the contour point cloud of the measurement target can be set.

The target position measuring device and the target position measuring program of the present disclosure can measure three-dimensional information of the measurement target by using a millimeter wave radar or the like and a monocular camera or the like in combination.

S: Target position measurement system C1, C2, C3, C4, C5, C6, C7, C8: Vehicle P: Pedestrian L: White line A: Antenna P1, P2, P3, P4, P5, P6, P7: Radar reflection point R: Square pyramid stand R1, R2, R3, R4, R5, R6, R7: Sides E4, E5, E6, E7: Edge analysis range V: Disappearance point 1: Radar device 2: Image pickup device 3: Target position measurement device 31 : Measurement target acquisition unit 32: Reflection point plot unit 33: Square pyramid detection unit 34: Target contour detection unit 35: Position conversion unit

Claims (9)

About the measurement target The measurement target acquisition unit that acquires radar reflection point information and images,
A reflection point plotting unit that plots the radar reflection points on the image,
A quadrangular pyramid detection unit that detects a quadrangular frustum that borders the measurement target on the image by performing edge analysis in the vicinity of the radar reflection point on the image.
A target contour detection unit that detects the contour of the measurement target on the image by performing edge analysis in each surface of the quadrangular pyramid on the image.
A position conversion unit that converts the position of the contour of the measurement target on the image to a position in real space,
A target position measuring device characterized by being provided with. The target contour detection unit divides each surface of the quadrangular pyramid into a plurality of grids, executes edge analysis in each grid, and detects a part of the contour of the measurement target in each grid. A part of the contour of the measurement target is converted into a representative point in each grid.
The target position measuring device according to claim 1, wherein the position changing unit converts the position of the representative point into a position in real space in each grid. The target contour detection unit executes edge analysis in each surface of the quadrangular pyramid, detects the contour of the measurement target in each surface of the quadrangular frustum, and has a plurality of surfaces of the quadrangular frustum. It is divided into grids, and a part of the contour of the measurement target is converted into a representative point in each grid.
The target position measuring device according to claim 1, wherein the position changing unit converts the position of the representative point into a position in real space in each grid. The target contour detection unit is characterized in that the number of grids that divide each surface of the quadrangular pyramid is set according to the degree of danger of the measurement target as seen from the own vehicle equipped with the radar device and the image pickup device. The target position measuring device according to claim 2 or 3. The target contour detection unit is characterized in that it sets the number of grids that divide each surface of the quadrangular pyramid according to the distance between the own vehicle and the measurement target measured by the radar device. The target position measuring device according to claim 4. The target contour detection unit sets the number of grids that divide each surface of the quadrangular pyramid according to the difference in traveling speed between the own vehicle and the measurement target measured by the radar device. The target position measuring device according to claim 4 or 5, which is characterized. The target contour detection unit sets the number of grids that divide each surface of the quadrangular pyramid according to the difference in the traveling direction between the own vehicle and the measurement target measured by the radar device. The target position measuring device according to any one of claims 4 to 6, which is characterized. The target contour detection unit is characterized in that the number of grids that divide each surface of the quadrangular pyramid is set according to the approach time of the own vehicle to the measurement target measured by the radar device. , The target position measuring device according to any one of claims 4 to 7. Measurement target acquisition step to acquire radar reflection point information and images, and measurement target acquisition step
A reflection point plotting step for plotting the radar reflection points on the image,
A quadrangular frustum detection step for detecting a quadrangular frustum that borders the measurement target on the image by performing edge analysis in the vicinity of the radar reflection point on the image.
A target contour detection step of detecting the contour of the measurement target on the image by performing edge analysis in each surface of the quadrangular pyramid on the image.
A position conversion step of converting the position of the contour of the measurement target on the image to a position in the real space,
Target position measurement program to make the computer execute in order.

Images