JP7301474B2 - 3D data analysis device and 3D data analysis method - Google Patents

Description

The present invention relates to a three-dimensional data analysis device and a three-dimensional data analysis method, and particularly to a three-dimensional data analysis device and a three-dimensional data analysis method for analyzing three-dimensional data output by a three-dimensional distance measuring sensor. is.

2. Description of the Related Art Conventionally, there has been known a device that analyzes the condition of a space to be measured using a distance measurement sensor. For example, in Patent Document 1, ultrasonic waves are transmitted from the first ultrasonic sensor 11 provided on the electric cart 10 toward the road surface 80, and the reflected waves are received. A control device 30 that determines whether or not a road surface 80 exists at a predetermined position in front of the electric cart 10 by analyzing the there is

Conventionally, there are three-dimensional data analysis apparatuses that analyze three-dimensional data based on detection values of optical three-dimensional distance measuring sensors such as stereo cameras and TOF cameras. The three-dimensional data is, for example, point cloud data in which pixels whose distances are measured are arranged in a three-dimensional orthogonal coordinate system. Ideally, the three-dimensional data represent the surface of each object in space to be ranged as a set of points in a three-dimensional Cartesian coordinate system. By analyzing the three-dimensional data, the three-dimensional data analysis device analyzes the situation of the space to be measured, such as detecting the presence or absence of obstacles.

JP 2017-184358 A

There is a case where the three-dimensional distance measuring sensor described above is provided at a position higher than the ground, and the sensor measures the distance of the real space included in a radially expanding range with the position higher than the ground as a base point. For example, there is a case where a three-dimensional distance measuring sensor is provided in a mobility support vehicle such as an electric wheelchair or a senior car. Conventionally, in this case, there have been the following problems when analyzing three-dimensional data using a three-dimensional data analysis device.

In other words, a blank area, which is a group of areas where the distance has not been measured, may be formed in the image of the ground surface included in the three-dimensional data. A blank area may be formed due to noise, measurement errors, missing measurements, etc., even though the ground exists in the area corresponding to the blank area in the real space. A hole is formed in a region corresponding to , and may be formed due to this hole. However, in the conventional three-dimensional data analysis device, when a blank area is formed in the image of the surface of the ground, it is uniformly determined that a hole is formed in the area corresponding to the blank area in the real space. When a blank area is formed due to noise or the like in a situation where no hole is formed, it may be erroneously determined that a hole is formed in the area corresponding to the blank area.

The present invention has been made to solve such a problem. To suppress erroneous determination that a hole is formed even though a hole is not formed in a hole.

In order to solve the above-described problems, in the present invention, when a blank area, which is a group of areas where the distance is not measured, exists in an image of the surface of the ground included in three-dimensional data, the depth of the blank area is measured. From the edge, it is determined whether or not there is a downward extension image extending downward from the surface corresponding to the surface of the ground, and if there is, a hole is formed in the area corresponding to the blank area in the real space. I'm trying to judge that it is. In the present invention, when it is determined that a downwardly extending image exists, it is determined whether or not the vertical length of the downwardly extending image is equal to or greater than a threshold value. On the other hand, when it is determined that a hole is formed in the area corresponding to the blank area in the real space, of the area facing the base point on the inner surface of the hole, the three-dimensional measurement A theoretical value of the length in the vertical direction of a range-finding target portion that can be measured by the range sensor is calculated, and the threshold value is set based on the calculated theoretical value.

When a hole is formed in the ground in the real space, a part of the area facing the base point of the three-dimensional distance measuring sensor on the inner surface of the hole is subject to distance measurement by the three-dimensional distance measuring sensor. There is a feature that the image of the part will be included in the three-dimensional data. In the three-dimensional data, the image of this portion extends from the far side edge of the blank area downward from the surface corresponding to the surface of the ground.

Based on the above, according to the present invention configured as described above, when a blank area exists, instead of uniformly determining that a hole is formed in the area corresponding to the blank area in the real space, A hole is determined to be formed only if there is a downwardly extending image extending from the far edge of the blank area. Therefore, it is possible to prevent erroneous determination that a hole is formed in the area corresponding to the blank area in the real space, even though the area corresponding to the blank area in the real space is not formed. .

1 is a block diagram showing a functional configuration example of a warning system including a three-dimensional data analysis device according to one embodiment of the present invention; FIG. FIG. 4 is a diagram showing a state in which a three-dimensional ranging sensor is provided in a movement support vehicle; FIG. 4 is a diagram showing an example of an image developed on a three-dimensional orthogonal coordinate system based on three-dimensional data; FIG. 4 is a diagram showing the arrangement relationship in real space of a camera, a ranging range, and the ground; It is a figure which shows a mode that the blank area was seen toward the minus side of z-axis direction. FIG. 13 shows a blank area and a downward extension image; 4 is a view of the image in FIG. 3 viewed from the plus side to the minus side in the y-axis direction; FIG. It is the figure which looked at the image of FIG. 3 toward the plus side from the minus side of the x-axis direction. It is a figure which shows a mode that the camera was arrange|positioned on the ground in which the hole was formed. 4 is a flow chart showing an operation example of the three-dimensional data analysis device according to one embodiment of the present invention;

An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing a functional configuration example of a warning system 2 including a three-dimensional data analysis device 1 according to one embodiment of the present invention. As shown in FIG. 1, the warning system 2 includes a three-dimensional ranging sensor 3, a three-dimensional data generation device 4, a three-dimensional data analysis device 1, and a warning device 5. A three-dimensional data analysis device 1 according to this embodiment is provided in a mobility support vehicle 6 ( FIG. 2 ) and analyzes three-dimensional data generated based on detection values of a three-dimensional ranging sensor 3 . The movement support vehicle 6 is a vehicle such as an electric wheelchair or a senior car that travels on a sidewalk with a person on board.

FIG. 2 is a simplified schematic diagram of the three-dimensional distance measuring sensor 3 installed on the mobility support vehicle 6 to explain the installation position of the three-dimensional distance measuring sensor 3, its measurement axis 7, and its distance measuring range 8. It is a schematic diagram. The three-dimensional ranging sensor 3 is a sensor that three-dimensionally measures a space, and particularly the three-dimensional ranging sensor 3 according to this embodiment is a stereo camera type ranging sensor. In FIG. 2, one camera 9 of two cameras 9, 9 constituting the stereo camera of the three-dimensional distance measuring sensor 3 is illustrated. In the following description, upward or upward means upward in the vertical direction, and downward or downward means downward in the vertical direction. Further, "forward" or "forward" means the direction in which the mobility assistance vehicle 6 moves forward, and "rearward" or "backward" means the direction in which the mobility assistance vehicle 6 moves backward. Also, left or left means facing left when facing forward, and right or right means facing right when facing forward.

As shown in FIG. 2, the camera 9 is provided at the front end of the mobility assistance vehicle 6 at a position higher than the ground 10 on which the mobility assistance vehicle 6 is placed. The height of the camera 9 from the ground 10 is, for example, about 50 cm. As shown in FIG. 2, the camera 9 is mounted on the mobility assistance vehicle 6 so that its measurement axis 7 (optical axis) extends forward while being parallel to the ground 10 . The distance measurement range 8 extends radially from the base point 11 of the camera 9 along the measurement axis 7 toward the front. Target for distance measurement. Hereinafter, the physical space that is included in the distance measurement range 8 and is the object of distance measurement will be referred to as the "object space for distance measurement". In a stereo camera type distance measuring sensor, although the photographing range of one camera 9 and the distance measuring range 8 do not exactly match, they are close to each other. In the embodiment, it is assumed that the imaging range of the camera 9 is the ranging range 8 as it is.

In this embodiment, when the power source of the mobility support vehicle 6 is switched on, power supply to each device of the warning system 2 is started, and the power of these devices is automatically turned on. While the power of the three-dimensional ranging sensor 3 is turned on (=the period during which the mobility support vehicle 6 may travel), the two cameras 9, 9 continuously perform photographing at predetermined intervals, Images (detected values) captured by the two cameras 9 and 9 are output to the three-dimensional data generation device 4 .

The three-dimensional data generating device 4 performs matching based on existing technology on the captured images of the two cameras 9, 9 which are input at predetermined intervals, and generates three-dimensional data. In this embodiment, the three-dimensional data is point cloud data in which the pixels whose distances are measured by the three-dimensional distance measuring sensor 3 are arranged in a three-dimensional orthogonal coordinate system (=each pixel whose distance is measured in the three-dimensional orthogonal coordinate system). point cloud data that holds the coordinates of pixels). Ideally, the three-dimensional data should be such that the surface (the surface photographed by the cameras 9, 9) facing the base point 11 of each object (including the ground 10) in the range-finding space is a point in the three-dimensional orthogonal coordinate system. is represented as a set of

The three-dimensional data generation device 4 continuously generates three-dimensional data at a predetermined cycle and outputs the three-dimensional data to the three-dimensional data analysis device 1 .

As shown in FIG. 1, the three-dimensional data analysis apparatus 1 includes a three-dimensional data acquisition section 13, a blank area detection section 14, and a hole determination section 15 as functional configurations. Each of the functional blocks 13 to 15 can be configured by hardware, DSP (Digital Signal Processor), or software. For example, when configured by software, each of the functional blocks 13 to 15 is actually configured with a CPU, RAM, ROM, etc. of a computer, and a program stored in a recording medium such as RAM, ROM, hard disk, or semiconductor memory. is realized by the operation of

The three-dimensional data acquisition unit 13 receives and acquires three-dimensional data output by the three-dimensional data generation device 4 at predetermined intervals. Each time the 3D data acquisition unit 13 acquires 3D data, it stores the 3D data in a buffer 16 formed in a work area such as a RAM, and notifies the blank area detection unit 14 of the storage.

A blank area detection unit 14 detects a blank area 17 (FIGS. 3, 5 and 6), which is a group of areas whose distance is not measured, from the image of the surface of the ground 10 included in the three-dimensional data. The processing of the blank area detection unit 14 will be described in detail below.

The blank area detection unit 14 performs the following processing on the three-dimensional data stored in the buffer 16 each time it receives a notification from the three-dimensional data acquisition unit 13 that the three-dimensional data has been stored in the buffer 16. . That is, first, the blank area detection unit 14 identifies the image 18 (FIG. 3) of the surface of the ground 10 from the three-dimensional data. The image 18 of the surface of the ground 10 is hereinafter referred to as "ground surface image 18".

FIG. 3 shows an image developed on a three-dimensional orthogonal coordinate system based on three-dimensional data generated based on the detection values of the three-dimensional distance measuring sensor 3 provided on the mobility support vehicle 6 in the manner described in FIG. It is a figure which shows the example of. In FIG. 3 and FIGS. 7 and 8 described later, white pixels are arranged on a black background. In the following description, the x-axis direction of the three-dimensional orthogonal coordinate system corresponds to the front-back direction in the real space, the direction toward the plus side of the x-axis direction corresponds to the front side, and the direction toward the minus side of the x-axis direction corresponds to It shall correspond to the rear. Also, the y-axis direction of the three-dimensional orthogonal coordinate system shall correspond to the horizontal direction in the physical space. The z-axis direction of the three-dimensional orthogonal coordinates corresponds to the vertical direction in the real space, the direction toward the plus side of the z-axis direction corresponds to upward direction, and the direction toward the minus side of the z-axis direction corresponds to downward direction. shall be

In the image illustrated in FIG. 3, the image of the portion surrounded by the frame W1 corresponds to the ground surface image 18. As shown in FIG. In FIG. 3, the image on the positive side in the x-axis direction of the ground surface image 18 surrounded by the frame W1 is the image of the wall erected on the ground 10. As shown in FIG. To explain the ground surface image 18 in detail, as shown in FIG. 3, the ground surface image 18 spreads in the y-axis direction toward the plus side in the x-axis direction. This is because the distance measurement range 8 radially spreads forward along the measurement axis 7 from the base point 11 .

Further, the ground surface image 18 has gradually unstable pixels and rough pixels toward the positive side in the x-axis direction. This is due to the fact that as the position of the object moves away from the three-dimensional distance measuring sensor 3, the accuracy of distance measurement decreases and the density of the pixels with which the three-dimensional distance measuring sensor 3 can capture the object decreases. This is due to the nature of Also, as shown in FIG. 3, the ground surface image 18 essentially extends along a plane parallel to the plane containing the intersecting x- and y-axes. However, the ground surface image 18 is not perfectly flat and has some irregularities.

In the three-dimensional orthogonal coordinate system, the range in which the pixel group forming the ground surface image 18 can be arranged is determined by the specifications of the three-dimensional distance measuring sensor 3, the arrangement position, and the direction of the measurement axis 7. FIG. This is because these define the state of the range-finding range 8 with respect to the ground 10 . In this embodiment, the range in which the pixel group constituting the ground surface image 18 can be arranged in the three-dimensional orthogonal coordinate system (hereinafter referred to as " "ground surface image range") is predetermined. The ground surface image range is defined not as a planar area but as a three-dimensional area having a certain width in the z-axis direction.

Then, the blank area detection unit 14 identifies a group of pixels belonging to the ground surface image range as the ground surface image 18 . Note that when walls, pedestrians, and other obstacles stand on the ground 10 as in the example of FIG. A pixel group belonging to the surface image range cannot be extracted. The blank area detection unit 14 excludes from the ground surface image 18 the range where the pixel group cannot be extracted as the pixel group belonging to the ground surface image range due to the image of the obstacle. It is clearly distinguished from the blank area 17 which will be described in detail.

After specifying the ground surface image 18 , the blank area detection unit 14 specifies an analysis target range in the ground surface image 18 . The analysis target range is an area in which a certain level of pixel accuracy can be ensured. As described above, the pixels of the ground surface image 18 become unstable and coarser toward the plus side in the x-axis direction, and the accuracy of the pixels becomes lower. In this embodiment, the precision of a certain pixel refers to the degree to which the coordinates of that pixel in the three-dimensional orthogonal coordinate system correctly reflect the position of the surface of an object in the real space. The higher the , the stronger the reflection. In this embodiment, the analysis target range is predetermined. In the example of FIG. 3, for example, the range on the minus side of the x-axis direction with respect to the arc K1 is the analysis target range.

After specifying the analysis target range in the ground surface image 18, the blank area detection unit 14 analyzes the analysis target range and determines whether or not the blank area 17 exists within this range. The blank area 17 is an area in the ground surface image 18 that has a size equal to or larger than a predetermined size and has no pixels. In other words, the blank area 17 is a large area within the ground surface image 18 in which the distance is not measured by the three-dimensional ranging sensor 3 . Assuming that a hole 19 (FIGS. 4A and 9) having a size equal to or larger than the predetermined size is formed in the ground 10 in the physical space, the predetermined size serving as a reference for determining whether or not the blank area 17 corresponds to the blank area 17 is determined. In other words, if a hole 19 smaller than the predetermined size is formed in the ground 10 in the real space, is also determined in advance based on the viewpoint that there is no influence on the running of the mobility support vehicle 6 and there is no need to give a warning to the passenger. In FIG. 3, the blank area 17 is a black area surrounded by a frame W2 (=an area without pixels).

There are two reasons why the blank area 17 is formed in the ground surface image 18 as follows. FIG. 4 is a diagram showing a simplified arrangement relationship of the camera 9 of the three-dimensional distance measuring sensor 3, the distance measuring range 8, and the ground 10 in the real space for explaining the cause of the formation of the blank area 17. As shown in FIG. As shown in FIG. 4A , when a hole 19 is formed in the ground 10 in the real space, a blank area 17 is formed in the ground surface image 18 . This is because the three-dimensional distance measuring sensor 3 does not measure the distance in the area of the hole 19 on the plane along the ground 10, and no pixel is formed in the three-dimensional data.

On the other hand, as shown in FIG. 4(B), even if the hole 19 is not formed in the ground 10 in the real space, the three-dimensional data may be affected by noise, measurement errors, missing measurements, etc. A blank area 17 may be formed in the ground surface image 18 of . Missing data means the following. That is, when the three-dimensional distance measuring sensor 3 measures the surface of an object in the distance measurement target space, the three-dimensional distance measuring sensor 3 measures distances at a plurality of discrete measurement points according to the resolution of the sensor. Missing measurement means that the portion between the measurement points is not subject to distance measurement (the distance cannot be measured).

In this embodiment, the measurement axis 7 of the three-dimensional distance measuring sensor 3 extends parallel to the ground 10, as described with reference to FIG. When the three-dimensional ranging sensor 3 is provided in the mobility assistance vehicle 6 in this manner, the ground 10 extending parallel to the measurement axis 7 is compared with an object extending perpendicular to the measurement axis 7. As a result, the density of pixels with which the three-dimensional distance measuring sensor 3 can capture an object (=the density of measurement points theoretically developed on the surface of the object) is small, and is susceptible to noise, measurement errors, and missing measurements. Therefore, the ground surface image 18 tends to have a blank area 17 formed therein. In particular, the greater the distance from the base point 11, the more likely it is to be affected by noise, measurement errors, and missing measurements, and thus the blank area 17 is likely to be formed.

When the blank area detection unit 14 determines that the blank area 17 exists, the blank area detection unit 14 outputs information indicating the range of the blank area 17 in the three-dimensional orthogonal coordinate system (hereinafter referred to as "blank area range information") to the hole determination unit 15. do. The blank area range information is, for example, information representing the outer edge of the blank area 17 by coordinates of a plurality of different points in the three-dimensional orthogonal coordinate system. On the other hand, when the blank area detection unit 14 determines that the blank area 17 does not exist, the blank area detection unit 14 does not output the blank area range information to the hole determination unit 15 and ends the processing in this cycle. In this case, processing by the hole determination unit 15, which will be described later, is not executed in this cycle.

A plurality of blank areas 17 may exist. In this case, the blank area detection unit 14 outputs blank area range information for each of the plurality of blank areas 17 to the hole determination unit 15 . In the following description, for convenience of explanation, when blank area range information is output from the blank area detection unit 14 to the hole determination unit 15, one blank area range information related to one blank area 17 is output. shall be However, when a plurality of pieces of blank area range information relating to a plurality of blank areas 17 are output, the processing described below is executed based on each piece of blank area range information.

The hole determination unit 15 detects a downwardly extending image 20 (FIG. 6 , 7, 8) exist, and if they exist, it is determined that the hole 19 is formed in the area corresponding to the blank area 17 in the physical space. The processing of the hole determination unit 15 will be described in detail below. In this embodiment, the "back side" means the side away from the base point 11 of the three-dimensional distance measuring sensor 3, and means the plus side in the x-axis direction in the three-dimensional orthogonal coordinate system. Further, in this embodiment, the "front side" means the side closer to the base point 11 of the three-dimensional distance measuring sensor 3, and means the minus side in the x-axis direction in the three-dimensional orthogonal coordinate system.

When the blank area range information is input from the blank area detection unit 14, the hole determination unit 15 performs the following processing. That is, the hole determination unit 15 refers to the three-dimensional data stored in the buffer 16, and based on the input blank area range information, determines the blank area 17 in the three-dimensional orthogonal coordinate system in which the image based on the three-dimensional data is developed. identify.

Next, the hole determination unit 15 identifies the far side edge of the blank area 17 . FIGS. 5A and 5B are diagrams showing how the blank area 17 is viewed from the negative side in the z-axis direction. In FIGS. 5A and 5B, the blank area 17 is represented by a line along the entire outer edge thereof. For example, with respect to the blank area 17 illustrated in FIG. 5A, the hole determination unit 15 identifies the portion surrounded by the frame W3 as the far side edge of the blank area 17 . Further, for example, the hole determining unit 15 identifies a portion surrounded by a frame W4 as the far side edge of the blank area 17 illustrated in FIG. 5B.

Next, the hole determination unit 15 determines whether or not there is a downwardly extending image 20 extending from the far side edge of the blank area 17 toward the minus side in the z-axis direction. The downwardly extending image 20 will be described below with reference to FIG. 4(A). Here, when the hole 19 is actually formed in the area corresponding to the blank area 17 in the physical space, one of the areas facing the base point 11 of the three-dimensional distance measuring sensor 3 is located on the inner surface of the hole 19 . The 3D distance measuring sensor 3 is used to measure the range of the part. For example, in FIG. 4A, the area AR1 is the target of distance measurement by the three-dimensional distance measurement sensor 3. FIG. The three-dimensional distance measuring sensor 3 according to this embodiment is a stereo camera type distance measuring sensor. be. Hereinafter, the portion of the inner surface of the hole 19 that is the object of distance measurement by the three-dimensional distance measurement sensor 3 will be referred to as the "object of distance measurement".

Since the distance measurement target portion is the object of distance measurement by the three-dimensional distance measurement sensor 3, the three-dimensional data generated based on the detection value of the three-dimensional distance measurement sensor 3 includes the image of this distance measurement object portion. It is formed. The image of the distance measurement target portion is a group of pixels representing the surface facing the reference point 11 of the distance measurement target portion. The image of this range-finding target portion is the downward extension image 20 . Due to the above-described properties of the part to be distance-measured, the downwardly extending image 20 extends from the far side edge of the blank area 17 toward the minus side in the z-axis direction in the three-dimensional orthogonal coordinate system (=on the surface of the ground 10). downward from the corresponding surface).

FIG. 6 is a diagram schematically showing a blank area 17 having a circular outer edge and a downward extending image 20 formed corresponding to this blank area 17 in a simplified manner. In FIG. 6, portions corresponding to the inner periphery and the bottom of the hole 19 in the real space are indicated by solid lines. The downwardly extending image 20 is the image of the portion shown with oblique lines in FIG. As shown in FIG. 6, when the blank area 17 has a circular outer edge, a downward extending image 20 is formed in a portion facing the base point 11 and exposed so as to be able to be measured by the three-dimensional distance measuring sensor 3 . FIG. 7 is a view of the image in FIG. 3 viewed from the plus side to the minus side in the y-axis direction. In FIG. 7, the image surrounded by the frame W5 is the downward extension image 20. In FIG. FIG. 8 is a view of the image in FIG. 3 viewed from the minus side to the plus side in the x-axis direction. In FIG. 8, the image surrounded by the frame W6 is the downward extension image 20. In FIG.

When determining that the downward extending image 20 does not exist, the hole determination unit 15 determines that the hole 19 is not formed in the area corresponding to the blank area 17 in the physical space. This is because, as described above, when the hole 19 is formed in the region corresponding to the blank region 17, the downwardly extending image 20 should be formed along with it. In this case, the blank area 17 is caused by noise, measurement error, missing measurement, or the like.

On the other hand, when determining that the downwardly extending image 20 exists, the hole determination unit 15 measures the length of the downwardly extending image 20 in the vertical direction (z-axis direction). In this embodiment, the hole determination unit 15 measures the vertical length of the downwardly extended image 20 by the following method. That is, the hole determination unit 15 identifies the pixel located on the most minus side in the z-axis direction among the pixels forming the downwardly extended image 20 . Next, the hole determination unit 15 draws a perpendicular line from the specified pixel to a virtual plane corresponding to the ground 10 in the physical space, and specifies the intersection of the perpendicular line and the virtual plane as the deepest point 21 (FIG. 6). Next, the hole determination unit 15 measures the distance from the specified pixel to the innermost point 21 as the length of the downward extended image 20 in the vertical direction. The deepest point 21 is ideally the farthest point from the base point 11 on the outer edge of the blank area 17 .

In the example of FIG. 6, the hole determination unit 15 measures the length of the line segment LL as the length of the downwardly extended image 20 in the vertical direction. For convenience of explanation, the measured length shall be a value converted into the length of the real space expressed in units such as centimeters and millimeters.

Next, the hole determination unit 15 calculates a theoretical value of the length of the part to be measured (hereinafter referred to as "theoretical length value"). The processing of the hole determination unit 15 when calculating the theoretical length value will be described in detail below. FIG. 9(A) is a diagram showing how the camera 9 is placed on the ground 10 in which a hole 19 is formed, similarly to FIG. FIG. 9(B) is a diagram showing the environment of FIG. 9(A) viewed from above downward. 9A and 9B correspond to each other in the front-rear direction.

In FIG. 9A, "a straight line S1 passing through the base point 11 of the three-dimensional distance measuring sensor 3 and a point P1 (details will be described later) on the front edge of the hole 19" and "corresponding to the deepest point 21 A point P3 is an intersection point with a straight line S2 extending in the vertical direction passing through a point P2 (details will be described later). The theoretical length of the distance measurement target portion corresponds to the theoretically calculated "length of line segment X1 extending from point P2 to point P3". The hole determination unit 15 calculates the theoretical length of the line segment X1 as a theoretical length value. The line segment X1 corresponds to "a second line extending along the distance measurement target portion in the real space" in the claims.

When calculating the theoretical length value, first, the hole determination unit 15 defines a completely flat virtual plane 22 (FIG. 9A) extending along the ground 10 in the real space. The surface 22 is hereinafter referred to as "virtual ground 22". Points P<b>1 and P<b>2 described above, point P<b>4 described later, and line segments L<b>2 , L<b>3 and L<b>4 described later are arranged on the virtual ground 22 .

Next, based on the blank area range information input from the blank area detection unit 14, the hole determination unit 15 calculates the arrangement position of the outer edge 23 (FIG. 9B) of the area corresponding to the blank area 17 in the physical space. do. In this embodiment, the arrangement position of the outer edge 23 is the position of each point relative to the base point 11 when the outer edge 23 is regarded as "a set of points on the virtual ground surface 22". Here, when a perpendicular is drawn from the reference point 11 to the virtual ground 22 in the physical space, the intersection of this perpendicular and the virtual ground 22 is defined as a point P4 (FIGS. 9A and 9B). The hole determination unit 15 acquires the length of the line segment L1 from the base point 11 to the point P4. The length of the line segment L1 is determined by the installation position of the three-dimensional ranging sensor 3. FIG. In this embodiment, installation position information including information indicating the height of the reference point 11 from the virtual ground 22 (=length of the line segment L1) is stored in advance in a memory (not shown). The length of the line segment L1 is obtained based on the installation position information.

Next, the hole determination unit 15 determines the position of a point P2 (FIGS. 9A and 9B) corresponding to the innermost point 21 in the physical space based on the coordinates of the innermost point 21 in the three-dimensional orthogonal coordinate system. calculate. The hole determination unit 15 calculates the position of the point P2 as a relative position from the base point 11 in the physical space. Next, the hole determination unit 15 calculates a line segment L2 (FIGS. 9A and 9B) extending from the point P4, which is the intersection of the line segment L1 and the virtual ground 22, to the point P2. Further, the hole determination unit 15 calculates an intersection point P1 (FIGS. 9A and 9B) between the line segment L2 and the outer edge 23 of the area corresponding to the blank area 17. FIG.

Next, the hole determination unit 15 calculates the length of the line segment L3 extending from the point P4 to the point P1 and the length of the line segment L4 extending from the point P1 to the point P2. A point P1 is a point located on the near side edge of the area corresponding to the blank area 17 in the physical space. A point P2 is a point located at the far side edge of the blank area 17 in the physical space. Based on this, the line segment L3 is defined as the "second line extending from the intersection of the perpendicular line drawn from the base point to the ground 10 and the ground 10 to the front edge of the area corresponding to the blank area 17 in the real space. 1 line”. In addition, the line segment L4 is defined as "the third line extending from the front edge to the back edge of the area corresponding to the blank area 17 in the real space and on the same straight line as the first line." corresponds to the line of

Here, the right-angled triangle whose vertices are the base point 11, the point P4 and the point P1 and the right-angled triangle whose vertices are the points P3, P2 and P1 are in a similar relationship. : length of line segment L3=length of line segment X1:length of line segment L4" holds. The hole determination unit 15 applies the already obtained lengths of the line segments L1, L3, and L4 to the equation S1, calculates the length of the line segment X1, and uses the calculated value as the theoretical length value.

After calculating the theoretical length value as described above, the hole determination unit 15 sets a value obtained by subtracting a predetermined margin from the theoretical length value as the threshold value T1. The significance of the margin will be discussed later. Next, the hole determination unit 15 determines whether or not the vertical length of the downward extended image 20 is equal to or greater than the threshold value T1. When the vertical length of the downwardly extending image 20 is equal to or greater than the threshold value T1, the hole determination unit 15 determines that the hole 19 is formed in the area corresponding to the blank area 17 in the physical space. On the other hand, when the vertical length of the downwardly extended image 20 is not equal to or greater than the threshold value T1, the hole determination unit 15 determines that the hole 19 is not formed in the area corresponding to the blank area 17 in the physical space.

The reason why the hole determination unit 15 makes such a determination is as follows. That is, the threshold value T1 is a value obtained by subtracting a margin from the theoretical value of the length of the distance measurement target portion. Therefore, when the hole 19 is formed in the physical space, the threshold value T1 is set to Predicted value of the vertical length of the downwardly extending image 20 when the hole 19 is formed in the physical space, which is the length of the downwardly extending image 20 formed based on the detected value of can be considered. The fact that the vertical length of the downwardly extending image 20 is equal to or greater than the threshold value T1 means that the length of the downwardly extending image 20 in the vertical direction is theoretically derived assuming that the hole 19 exists. In this case, it can be considered that the hole 19 actually exists in the physical space. Based on the above, the hole determination unit 15 determines that the hole 19 is formed in the area corresponding to the blank area 17 in the physical space when the vertical length of the downwardly extended image 20 is equal to or greater than the threshold value T1. On the other hand, if the vertical length of the downwardly extended image 20 is not equal to or greater than the threshold value T1, it is determined that the hole 19 is not formed in the area corresponding to the blank area 17 in the physical space.

Note that the margin is the length of the downward extension image 20 that is not affected by noise, measurement error, or the like in a situation where the hole 19 is formed in the physical space. Even if it is shorter than the length, it is given in consideration of making the vertical length of the downwardly extended image 20 equal to or greater than the threshold value T1.

According to the configuration of the present embodiment, instead of determining that the hole 19 is formed in the area corresponding to the blank area 17 in the real space for all the blank areas 17, The presence of the extending downward extension image 20 is a condition for determining that the hole 19 is formed. Therefore, when a hole 19 exists in the physical space, the feature that the downwardly extending image 20 is formed along with the blank area 17 corresponding to the hole 19 is preferably used to detect the blank in the physical space. It is possible to prevent an erroneous analysis that the hole 19 is formed in the region corresponding to the region 17 .

Further, in the present embodiment, it is not determined that the hole 19 is formed in the area corresponding to the blank area 17 in the physical space just because the downwardly extending image 20 exists. If the length in the vertical direction is equal to or greater than the threshold, it is determined that the hole 19 is formed in the area corresponding to the blank area 17 in the physical space. Due to this configuration, when the downwardly extending image 20 is accidentally formed due to noise, measurement error, missing measurement, etc. accompanying the blank area 17 that does not originate from the hole 19 in the real space, such a blank It is possible to reduce the possibility of erroneously determining that the hole 19 is formed in the corresponding area of the physical space for the area 17 .

Now, when the hole determination unit 15 determines that the hole 19 is formed in the area corresponding to the blank area 17 in the physical space, it notifies the warning device 5 of that fact. As a result, during the period in which the hole determination unit 15 determines that the hole 19 is formed in the area corresponding to the blank area 17 in the physical space, the hole determination unit 15 continuously notifies the warning device 5 at predetermined intervals. output.

The warning device 5 warns the passenger that there is a hole 19 in front of the mobility support vehicle 6 while the notification is being input from the hole determination unit 15 . The warning is, for example, outputting a specific electronic sound from a speaker (not shown), or outputting a specific wording (for example, "There is a hole 19 in front. Please be careful") from the speaker, Alternatively, for example, a specific LED emits light in a specific manner.

Next, the operation of the three-dimensional data analysis device 1 according to one embodiment of the present invention will be explained using a flowchart. FIG. 10 is a flow chart showing the operation of the three-dimensional data analysis device 1 after the three-dimensional data generation device 4 outputs the three-dimensional data.

As shown in FIG. 10, the three-dimensional data acquisition unit 13 receives and acquires three-dimensional data output by the three-dimensional data generation device 4 at predetermined intervals (step SA1). Next, the three-dimensional data acquisition unit 13 stores the three-dimensional data in the buffer 16 and notifies the blank area detection unit 14 of the storage (step SA2).

The blank area detection unit 14 identifies the ground surface image 18 from the three-dimensional data in response to the notification from the three-dimensional data acquisition unit 13 (step SA3). Next, the blank area detection unit 14 identifies an analysis target range in the ground surface image 18 (step SA4). Next, the blank area detection unit 14 analyzes the analysis target range and determines whether or not the blank area 17 exists within this range (step SA5). If the blank area 17 exists (step SA5: YES), the blank area detection unit 14 outputs blank area range information to the hole determination unit 15 (step SA6). If the blank area 17 does not exist (step SA5: NO), the flowchart of FIG. 10 ends.

When the blank area range information is input from the blank area detection section 14, the hole determination section 15 specifies the blank area 17 in the three-dimensional orthogonal coordinate system in which the image based on the three-dimensional data is developed based on this blank area range information. (step SA7). Next, the hole determination unit 15 identifies the far side edge of the blank area 17 (step SA8). Next, the hole determination unit 15 determines whether or not there is a downwardly extending image 20 extending from the far side edge of the blank area 17 toward the minus side in the z-axis direction (step SA9). ).

If downwardly extending image 20 does not exist (step SA9: NO), hole determination unit 15 determines that hole 19 is not formed in the area corresponding to blank area 17 in the physical space (step SA10). After the processing of step SA10, the flowchart of FIG. 10 ends. If the downwardly extending image 20 exists (step SA9: YES), the hole determination unit 15 measures the length of the downwardly extending image 20 in the vertical direction (z-axis direction) (step SA11). Next, the hole judging section 15 calculates a theoretical length value, which is a theoretical value of the length of the distance measurement target portion (step SA12).

Next, the hole determination unit 15 sets a value obtained by subtracting a predetermined margin from the theoretical length value as the threshold value T1 (step SA13). Next, the hole determination unit 15 determines whether or not the vertical length of the downward extended image 20 is equal to or greater than a threshold value T1 (step SA14). If the vertical length of the downwardly extended image 20 is not equal to or greater than the threshold value T1 (step SA14: NO), the hole determination unit 15 determines that the hole 19 is not formed in the area corresponding to the blank area 17 in the physical space. Determine (step SA15). After the processing of step SA15, the flowchart of FIG. 10 ends.

If the vertical length of the downwardly extended image 20 is equal to or greater than the threshold value T1 (step SA14: YES), the hole determination unit 15 determines that the hole 19 is formed in the area corresponding to the blank area 17 in the physical space. (step SA16). Next, hole determination unit 15 notifies warning device 5 that hole 19 is formed in the area corresponding to blank area 17 in the physical space (step SA17).

Although one embodiment of the present invention has been described above, the above embodiment merely shows an example of specific implementation of the present invention, and the technical scope of the present invention can be construed in a limited manner. It should not be done. Thus, the invention may be embodied in various forms without departing from its spirit or essential characteristics.

For example, in the above-described embodiment, when the downwardly extending image 20 exists and the vertical length of the downwardly extending image 20 is equal to or greater than the threshold value T1, the hole determination unit 15 determines that the blank area 17 in the physical space is It was determined that holes 19 were formed in the corresponding regions. Regarding this point, the hole determination unit 15 may be configured to determine that the hole 19 is formed in the area corresponding to the blank area 17 in the physical space when the downwardly extending image 20 exists. However, in this case, even if the hole 19 is not actually formed and the downwardly extending image 20 is accidentally formed due to noise, measurement error, missing measurement, etc., it is determined that the hole 19 is formed. Therefore, the possibility of erroneously determining the existence of the hole 19 increases compared to the above embodiment.

Regarding the above points, the hole determination unit 15 determines that when the downwardly extending image 20 exists and the vertical length of the downwardly extending image 20 is equal to or greater than a predetermined fixed value threshold value T2, the actual It may be determined that the hole 19 is formed in the area corresponding to the blank area 17 in the space. It should be noted that if the downwardly extending image 20 is accidentally formed due to noise or the like, it is assumed that the length in the vertical direction will be considerably shortened. With this in mind, the threshold T2 is set to a value that allows rejection of accidentally formed short downward extension images 20 . In this case, the threshold value T2 is a value that is not related to the theoretical value of the vertical length of the portion to be measured. Therefore, compared to the above embodiment, the possibility of erroneously determining the existence of the hole 19 increases. do.

Further, in the above embodiment, the three-dimensional ranging sensor 3 is a stereo camera type ranging sensor, but the three-dimensional ranging sensor 3 is a ranging sensor that outputs detection values for generating three-dimensional data. It is not limited to the exemplified types. As an example, it may be a ranging sensor that includes a projector that projects a pattern image and a camera, or a ranging sensor that includes a TOF camera.

Also, in the above embodiments, the measurement axis 7 extends parallel to the ground 10 . However, the measurement axis 7 may be tilted to some extent so that it approaches the ground 10 as it goes forward. In this case, the blank area detection unit 14 and the hole determination unit 15 reflect the inclination of the measurement axis 7 and execute various processes including analysis of the three-dimensional data. Also, the direction in which the measurement axis 7 faces is not limited to the front of the mobility support vehicle 6, and may be the rear or the side.

Further, in the above-described embodiment, the determination result of the hole determination unit 15 is used as a trigger for warning the passenger of the mobility assistance vehicle 6 . On the other hand, the configuration may be such that the determination result of the hole determination unit 15 is used in another mode. As an example, when the movement support vehicle 6 automatically travels on a sidewalk, useful information for determining whether or not there is a hole 19 to be avoided by automatic driving in front of the movement support vehicle 6 A configuration using the determination result of the hole determination unit 15 may be used.

Further, in the above-described embodiment, the three-dimensional ranging sensor 3 is provided on the mobility assistance vehicle 6, but the object on which the three-dimensional ranging sensor 3 is provided is not limited to the mobility assistance vehicle 6. As an example, it may be provided in a vehicle traveling on a roadway.

In addition, a part of the processing explained to be executed by the three-dimensional data analysis device 1 can be performed in cooperation with the three-dimensional data analysis device 1 and an external device, or by the three-dimensional data analysis device 1 alone. good too. In this case, the three-dimensional data analysis device 1 and the external device cooperate to function as a "three-dimensional data analysis device" in claims. As an example, part or all of the processing of the hole determination unit 15 of the three-dimensional data analysis device 1 may be executed by a cloud server that can communicate with the three-dimensional data analysis device 1 via a network.

1 three-dimensional data analysis device 3 three-dimensional distance measuring sensor 6 movement support vehicle 10 ground 11 base point 14 blank area detection unit 15 hole determination unit 17 blank area 18 ground surface image (ground surface image)
19 hole 20 downward extension image

Claims (4)

A three-dimensional data analysis device that analyzes three-dimensional data based on detection values of a three-dimensional ranging sensor whose distance measurement target is a real space included in a radially expanding range with a position higher than the ground as a base point,
a blank area detection unit that detects a blank area, which is a group of areas whose distance is not measured, from the image of the surface of the ground included in the three-dimensional data;
Determining whether or not there is a downwardly extending image extending downward from a surface corresponding to the surface of the ground from the far side edge of the blank area detected by the blank area detection unit, a hole determination unit that determines that a hole is formed in an area corresponding to the blank area in the physical space if it exists,
The hole determination unit
When it is determined that the downwardly extending image exists, it is determined whether or not the vertical length of the downwardly extending image is equal to or greater than a threshold. While determining that a hole is formed,
Supposing that a hole is formed in an area corresponding to the blank area in the physical space, a distance measurement capable of being measured by the three-dimensional distance measurement sensor is performed in an area facing the base point on the inner surface of the hole. A three- dimensional data analysis apparatus, wherein a theoretical value of the vertical length of a distance target portion is calculated, and the threshold value is set based on the calculated theoretical value. The hole determination unit
A perpendicular extending from the base point to the ground in the physical space, and a first line extending from the intersection of the perpendicular and the ground to the front edge of the area corresponding to the blank area in the physical space are at right angles. an intersecting right triangle and
A second line extending along the range-finding target portion in the physical space, and a first line extending from the near side edge to the far side edge of the area corresponding to the blank area in the physical space. Using the fact that the third line, which is a line on a line, is similar to a right-angled triangle that intersects at right angles,
The length of each of the perpendicular, the first line, and the third line is obtained based on the three-dimensional data and the installation position information of the three-dimensional ranging sensor, and based on the obtained length of each line 2. The three-dimensional data analysis apparatus according to claim 1 , wherein the length of the second line is calculated by using the above method and is used as the theoretical value. 3. The three-dimensional data analysis apparatus according to claim 1 , wherein the three-dimensional distance measuring sensor is provided in a mobility support vehicle that travels on a sidewalk. 3D data analysis by a 3D data analysis device that analyzes 3D data based on the detection values of a 3D ranging sensor whose range is measured in the real space that is included in a range that radiates from a position higher than the ground. a method,
a first step in which the blank area detection unit of the three-dimensional data analysis device detects a blank area, which is a group of areas whose distance is not measured, from the image of the surface of the ground included in the three-dimensional data;
The hole determination unit of the three-dimensional data analysis device extends downward from a far side edge of the blank area detected by the blank area detection unit below a surface corresponding to the surface of the ground. a second step of determining whether an existing image exists, and if it exists, determining that a hole is formed in an area corresponding to the blank area in the physical space ;
In the second step, the hole determination unit
When it is determined that the downwardly extending image exists, it is determined whether or not the vertical length of the downwardly extending image is equal to or greater than a threshold. While determining that a hole is formed,
If a hole is formed in an area corresponding to the blank area in the physical space, the distance can be measured by the three-dimensional distance measuring sensor in the area facing the base point on the inner surface of the hole. Calculate the theoretical value of the vertical length of the target portion, and set the threshold value based on the calculated theoretical value
A three-dimensional data analysis method characterized by:

Images