JP7718638B2 - 3D map creation device and 3D map creation method - Google Patents

Description

The present invention relates to a device and method for removing unnecessary objects from a 3D map in a 3D map creation device and 3D map creation method.

As seen in Non-Patent Document 1, there is a view that high-definition maps are not necessary for autonomous driving. However, autonomous vehicles generally use onboard cameras, LiDAR, and other sensors to recognize their surroundings, distinguishing and analyzing other vehicles, pedestrians, roadside trees, lanes, etc., before making decisions on how to control the vehicle. In this case, by integrating high-definition maps containing precise road information and position information for traffic lights, roadside trees, etc. with information detected by sensors, it becomes possible to determine the vehicle's position more accurately. This is also effective when sensor performance is reduced, such as in bad weather.

From this perspective, as shown in Patent Document 1, an invention has been proposed that estimates the surrounding environment with greater accuracy from observation data obtained using sensing equipment and creates an occupancy grid map that describes occupancy probability.

However, because there is no way to determine whether vehicles, people, etc. are unwanted objects, parked vehicles, etc. may be deemed stationary obstacles and registered on the map.

An invention has also been proposed that uses continuously acquired camera images to find moving objects that could become obstacles (Patent Document 2).

However, methods that use images can only find moving objects within the images. The distance sensors used in typical 3D map creation devices measure an area wider than the camera's field of view to create the 3D map, so Patent Document 1 is unable to remove unwanted objects outside the image's field of view, resulting in unwanted objects remaining in the 3D map. Furthermore, it is unable to remove objects that are not moving within a series of images, such as temporarily parked vehicles.

Patent Publication No. 2017-166966 Patent Publication No. 2000-123183

https://jidounten-lab.com/v_3dmap-autonomous-need3D Maps: "We Need It vs. We Don't" The ruthless battle in the autonomous driving industry: Are dynamic maps necessary?

One possible method is to use sensors installed in cars or mobile carts to acquire 3D point clouds of surrounding objects over time and create a 3D map. In order to be able to use the 3D point cloud as a map, the 3D map needs to include only point clouds of primarily fixed objects. When sensor data is acquired, the 3D map also includes point clouds of unwanted objects, so it is necessary to manually remove the point clouds from the map. For example, in a road environment, unwanted objects include parked vehicles and people, and in an indoor environment such as a factory, unwanted objects include temporarily placed luggage.

In light of the above, the objective of the present invention is to automate the process of removing unwanted objects outside the camera's field of view from a 3D map.

The three-dimensional map creation device of the present invention comprises a distance sensor installed on a mobile object that acquires a three-dimensional point cloud representing the mobile object's surrounding environment; a camera that acquires images of the surrounding environment within its angle of view; and a calculation means that creates a three-dimensional map based on the three-dimensional point cloud acquired from the distance sensor and the image acquired from the camera. The calculation means is characterized by comprising: an unwanted object detection unit that detects unwanted objects not required for the three-dimensional map based on the images and outputs the results as unwanted object detection results; a self-location estimation unit that estimates the position of the calculation means; an unwanted object area management map creation unit that creates an unwanted object area management map that indicates unwanted object areas where the unwanted objects may exist based on the three-dimensional point cloud and the unwanted object detection results; a removal unit that uses the unwanted object area management map to remove specific points from the three-dimensional point cloud; and a three-dimensional map creation unit that creates a three-dimensional map using points not removed by the removal unit. This makes it possible to remove unwanted objects that are not required for the three-dimensional map even if they are outside the camera's angle of view.

"3D point cloud" refers to a three-dimensional collection of objects generated as points, created based on the distance between them and the surrounding environment measured by a distance sensor.

"Cameras" include common cameras used to create three-dimensional maps.

"Within the camera's field of view" refers to a specific area within the range, expressed in degrees, that is actually captured when the surrounding environment is photographed with a camera.

A "distance sensor" is a sensor that measures the distance between itself and the surrounding environment.

"Unnecessary objects that are not part of the 3D map" include, for example, people, cars, bicycles, etc. from the surrounding environment.

The "self-position estimation unit" estimates the calculation means's own position on a three-dimensional map.

"Area where unwanted matter may exist" refers to a specific area excluding areas where unwanted matter may not exist.

A "waste area management map" is a map for managing areas that show point clouds that are determined to be waste.

A "specific point cloud" is at least a point cloud that corresponds to an unwanted object area. For example, point clouds managed as road surface areas may also be removed.

This invention makes it possible to automate the removal of unwanted objects from 3D maps, a task that has previously been done manually. It is also possible to remove unwanted objects measured by distance sensors, which measure a wider range than cameras.

A specific aspect of the method for creating an unwanted object area management map is that the space for which the map is to be created is divided into grids, and the unwanted object detection unit manages the grids in the space where unwanted objects are determined to exist as the unwanted object area.

In a specific aspect that takes into account the handling of objects of different sizes, such as vehicles and people, the unwanted object area management map creation unit changes the number of grids managed as the unwanted object area depending on the type of unwanted object detected by the unwanted object detection unit.

A specific aspect of the removal unit that uses unnecessary object areas is that the removal unit removes point clouds that exist on grids managed as unnecessary object areas in the unnecessary object area management map.

A specific aspect of the removal unit that sets the height range to be removed is that the removal unit sets a height threshold for the height of the point cloud to be managed in the grid managed as the unwanted object area, and removes the point cloud that falls within the range indicated by the height threshold.

A specific aspect of the confirmed road surface area is that the 3D map creation device further detects the road surface based on the image or the 3D point cloud, and the unnecessary object area management map creation unit manages the grid in which the area detected as the road surface exists as the confirmed road surface area.

In a specific embodiment using self-location information, the unnecessary object area management map creation is further characterized in that the space through which the 3D map creation device has passed is determined based on the time-series self-location estimated by the self-location estimation unit, and the grid in which the passed space exists is managed as a determined road surface area.

As a specific example of the road surface confirmed area, the unwanted object area management map creation unit detects the road surface by determining an area that is judged to be a road surface from the 3D point cloud acquired by the distance sensor.

A feature of the removal unit that uses unnecessary object areas is that the removal unit removes point clouds that exist on grids managed as road surface determined areas in the unnecessary object area management map and are at a specified height.

A specific aspect of the unwanted object detection unit is that the unwanted object detection unit further detects the unwanted object from the three-dimensional point cloud and outputs the unwanted object detection result.

A specific aspect of the detection unit that utilizes the feature that cameras have higher detection performance than distance sensors is that the unwanted object detection unit detects unwanted objects from the image within the camera's field of view, and detects unwanted objects from the three-dimensional point cloud outside the camera's field of view.

A specific aspect of changing the grid size depending on the type of detected object is that the unwanted object area management map creation unit changes the size of the grid managed as the unwanted object area depending on the type of unwanted object detected by the unwanted object detection unit.

A specific aspect that can be achieved by having a display unit is that the 3D map creation device further comprises a road map and a display unit, and in the created unwanted object area management map, the positions indicated by the grids managed as unwanted object areas are displayed on the road map.

A specific aspect of having unmeasured areas is that the display unit displays on the road map, as unmeasured areas, areas that are not determined as either unwanted object areas or determined road surface areas, and for which no point cloud was obtained by the distance sensor.

In a display mode that utilizes the feature that areas far from roads are unnecessary for autonomous driving, the display unit displays the unmeasured areas or unnecessary object areas that are located at a distance from the road on the road map that is shorter than a threshold value on the road map.

In a specific embodiment in which unwanted object areas or unmeasured areas are not displayed if they are not densely packed, the display unit displays on the road map grids where there are more unwanted object areas or unmeasured areas around the grids that are identified as unwanted object areas or unmeasured areas than a threshold value.

The three-dimensional map creation method of the present invention comprises an unwanted object detection step of detecting unwanted objects from an image acquired from a camera and creating unwanted object detection results; an unwanted object management map creation step of creating an unwanted object management map showing a grid indicating unwanted object areas; an unwanted object point cloud removal step of using the unwanted object management map to remove points indicating unwanted objects from a three-dimensional point cloud acquired from a distance sensor; and a three-dimensional map creation step of creating a three-dimensional map using the point cloud from which the unwanted objects have been removed.

FIG. 1 is a block diagram showing an example of a configuration according to a first embodiment of the present invention. FIG. 1 is an explanatory diagram showing an example of a configuration according to a first embodiment of the present invention. 1 is a block diagram showing an example of a hardware configuration according to a first embodiment of the present invention. 1 is a flowchart of a first embodiment of the present invention. FIG. 10 is an explanatory diagram showing an example of an acquired image and a projected point cloud. FIG. 10 is an explanatory diagram showing an example of an unwanted object detection result. FIG. 10 is an explanatory diagram showing an example of the unwanted object management map. FIG. 10 is an explanatory diagram showing an example of the unwanted object management map. FIG. 10 is a block diagram showing an example of a configuration according to a second embodiment of the present invention. 10A and 10B are explanatory diagrams for explaining a display image displayed on a display unit according to a second embodiment of the present invention. 10A and 10B are explanatory diagrams for explaining a display image displayed on a display unit according to a second embodiment of the present invention.

The 3D map creation device 1 and 3D map creation method according to embodiment 1 of the present invention will be described below with reference to Figures 1 to 8.

As shown in FIG. 2, the three-dimensional map creation device 1 is installed on a moving object 100 such as a person, an automobile, or a mobile robot, and includes a distance sensor 101 that acquires a three-dimensional point cloud 110, a camera 102 that acquires an image 120 shown in FIG. 5(A), an unwanted object detection unit 103 that detects unwanted objects 404 that are not required for the three-dimensional map, a self-position estimation unit 104 that estimates the self-position P of the device 1 (see FIGS. 7 and 8), an unwanted object area management map creation unit 105 that creates an unwanted object area management map 401 that indicates areas where unwanted objects 404 (see FIG. 7) may exist, a removal unit 106 that uses the unwanted object area management map 401 to remove point clouds 405, 505, 515 (see FIGS. 7 and 8) that indicate unwanted objects 404 from the three-dimensional point cloud 110 acquired by the distance sensor 101, and a three-dimensional map creation unit 107 that creates a three-dimensional map using the point clouds not removed by the removal unit 106.

The distance sensor 101 measures the distance to objects around the 3D map creation device 1, acquires a 3D point cloud 110 that indicates the shapes of objects in the surrounding environment, and outputs the information to the unwanted object area management map creation unit 105 and the removal unit 106. An example of a distance sensor 101 is a laser distance measuring device. Specific examples include TOF (Time of Flight) methods, such as LiDAR (Light Detection and Ranging), which calculates the distance to an object from the time it takes for laser light to travel back and forth.

The camera 102 acquires an image 120 (see Figure 5) and outputs the image 120 to the unwanted object detection unit 103. To project the 3D point cloud 110 acquired by the distance sensor 101 onto the image 120, the camera 102 and distance sensor 101 are installed on the mobile object 100 so that the acquisition range of the 3D point cloud 110 overlaps at least a portion of the camera's 102 angle of view 403. For example, the distance sensor 101 acquires the 3D point cloud 110 of the entire periphery of the 3D map creation device 1, and the camera 102 is installed facing the direction in which the mobile object 100 and the 3D map creation device 1 are moving. Figure 2, which shows the 3D point cloud 110, does not represent a point cloud acquired by the distance sensor 101 at a specific time, but rather represents a point cloud created by compiling point clouds from all points at different times to form a complete map. In the following explanation, the 3D point cloud 110 in this embodiment refers to a point cloud at a specific time, corresponding to the limited environment shown in the lower right of Figure 2, and will be described in that sense.

The unwanted object detection unit 103 detects unwanted objects 404 from the image 120 using techniques such as neural networks.

The self-position estimation unit 104 estimates the self-position P. For example, a GPS or a radio wave measurement device (not shown) can be used. It can also be obtained by acquiring sensor data in time series from the distance sensor 101 or camera 102 and comparing the time series data.

The unwanted object area management map creation unit 105 obtains a projected point cloud 130 by projecting the three-dimensional point cloud 110 onto the image 120 acquired by the camera 102, and creates unwanted object area management maps 401 and 501.

Here, an example of an arithmetic circuit 200 that controls 3D map creation, etc. will be described with reference to Figure 3. This arithmetic circuit 200 comprises a CPU 201, RAM 202, ROM 203, counter 204, timer 205, and input/output interface 209, all interconnected via a bus 210. The input/output interface 209 is connected to the distance sensor 101, camera 102, input unit 211, etc. The CPU 201 performs initial settings or specific calculations upon receiving input information.

The CPU 201 generates calculation information to be output to each component and executes 3D map creation by outputting the calculation information under program control. The RAM 202 temporarily reads and writes data for 3D map creation and the like. The ROM 203 stores programs for 3D map creation and the like in read-only mode. The CPU 201 executes calculations related to 3D map creation using data from the distance sensor 101, camera 102, etc. Instead of program control, this can also be implemented by hardware control such as LSI logic.

The counter 204 functions as a count value for creating 3D maps. After power is turned on, the count value is initialized to "0" and the count is calculated by referencing various input information.

Timer 205 performs time calculations related to 3D map creation, etc.

Examples of the input unit 211 include a mouse and keyboard. The input unit 211 may be omitted.

Next, the processing steps for creating a 3D map performed by the 3D map creation device 1 to remove 3D point clouds on unnecessary objects and create a 3D map will be explained with reference to the flowchart in Figure 4.

When processing begins, information from the distance sensor 101, camera 102, etc. is input (step 601).

When the processing of step 601 is completed, the unwanted object detection unit 103 detects unwanted object 404 (step 602).

In the unwanted object detection step 602, unwanted objects 404 are detected based on the image 120.

In this embodiment, the unwanted object detection unit 103 detects unwanted objects 404 from the image 120 because the camera 102 has better detection performance than the distance sensor 101. However, for locations that are always in the blind spot (outside the angle of view 403) of the camera 102, which is fixed forward in the direction of travel, the unwanted object 404 can also be detected from the three-dimensional point cloud 110 acquired by the distance sensor 101.

The unwanted object detection unit 103 detects unwanted objects 404 captured in the input image 120 and outputs the unwanted object detection result 303, which indicates the area surrounded by the rectangle shown in Figure 6, to the unwanted object area management map creation unit 105. A neural network, for example, can be used to detect unwanted objects 404. Unwanted objects 404 can be detected by having the neural network learn the objects that are the target of unwanted objects 404. The unwanted object detection result 303 indicates the unwanted object area on image 120 obtained by inputting image 120. Here, a car 301 and a person 302 have been detected as unwanted objects 404 that are not needed on the three-dimensional map. The unwanted object detection unit 103 outputs not only the location on image 120 but also the type of detected unwanted object 404 as the unwanted object detection result 303 to the unwanted object area management map creation unit 105.

The unwanted object detection unit 103 also trains the neural network to learn road surface areas, and outputs the road surface areas on the image 120 to the unwanted object area management map creation unit 105.

When the processing of step 602 is completed, processing proceeds to step 603, where the unnecessary object area management map creation unit 105 creates the unnecessary object area management map 401 (step 603).

As shown in Figure 7, the unnecessary object area management map 401 divides the space in which the three-dimensional map is to be created into grids, and the unnecessary object detection unit 103 manages the grids 406 in the space in which unnecessary objects 404 are determined to exist as unnecessary object areas U.

As shown in Figure 7, the unnecessary object area management map creation unit 105 divides the space within which the three-dimensional map is to be created into a grid, and creates an unnecessary object area management map 401 that manages whether unnecessary object 404 may exist in the grid-like space at a certain time t. Figure 8 shows an example of an unnecessary object area management map 501 managed by the three-dimensional map creation device 1 at a certain time t+1, after time t.

In Figures 5(A) and 6, a car 301 and a person 302 are captured as unwanted objects 404 within the angle of view 403 of the camera 102. As shown in Figure 7, the unwanted object area management map creation unit 105 first creates a projected point cloud 130 from the three-dimensional point cloud 110 and extracts the three-dimensional point cloud 405 contained in the unwanted object 404. Next, the position of the grid 406 in which the three-dimensional point cloud 405 exists is calculated from the self-position P estimated by the self-position estimation unit 104 and the position of the three-dimensional point cloud 405. The unwanted object area management map creation unit 105 manages the grid at the calculated position as the grid 406 in which the unwanted object 404 may exist.

The unwanted object area management map creation unit 105 can eliminate point clouds obtained from the same unwanted object 404 that are farther away from the center of the point cloud, etc., than a threshold value, based on the representative value of the point cloud 405, and not treat them as grids where unwanted object 404 may exist. The representative value can be selected by selecting the median, average, or shortest distance to each point cloud 405. Furthermore, if the type of unwanted object 404 is obtained from the unwanted object detection unit 103, the representative value can also be selected according to the type of unwanted object 404. For example, if it is detected as a person 302, the value is short, and if it is detected as a car 301, the value is long. For example, for the car on the far right in Figures 5(A) and (B), the background point cloud contained in the rectangular area is located farther away than the car 301, so a representative value such as the average distance of the point cloud contained in this rectangle is calculated, and points farther away from the representative value by more than a threshold value are not removed, and grids containing point clouds that have not been removed are not managed as unwanted object areas.

The unwanted object area management map creation unit 105 can also manage grids that are closer than the management threshold to the point cloud 405 managed as unwanted object 404 as grids where unwanted object 404 may exist. If the type of unwanted object 404 is obtained from the unwanted object detection unit 103, the management threshold can also be set according to the type. For example, if it is detected as a person 302, the threshold is set short, and if it is detected as a car 301, the threshold is set long. Although the projected point cloud 120 does not obtain the point cloud behind car 301, the type "car" is known, so grids in the range of sizes likely to be for a car are managed as unwanted object areas, but if it is a person, the threshold is set short.

The unwanted object area management map creation unit 105 can also detect individual unwanted objects 404 from the three-dimensional point cloud 110 measured by the distance sensor 101 and extract the point cloud of the unwanted objects 404. However, for point clouds within the range that can be projected onto the image 120 acquired by the camera 102 (within the angle of view 403), the unwanted object detection result 303 from the unwanted object detection unit 103 takes priority.

The unwanted object area management map creation unit 105 can also create multiple unwanted object area management maps 401, 501 with different grid sizes depending on the type of unwanted object 404 detected. For example, if people 302 and automobiles 301 are targeted as unwanted objects 404, it is possible to have an unwanted object area management map for people and an unwanted object area management map for automobiles, with the grid size of the unwanted object area management map for people being smaller than the grid size for automobiles.

The unwanted object area management map creation unit 105 can also change the number and/or size of the grids managed as unwanted object areas depending on the type of unwanted object 404 detected by the unwanted object detection unit 103. This is to take into account the different sizes of objects such as cars 301 and people 302.

When a road surface area (see Figure 5(A)) is obtained from the unnecessary object detection unit 103, the unnecessary object area management map creation unit 105 manages the grid containing the point cloud 409 (see Figure 5(B)) projected onto the road surface area as a determined road surface area R.

The unnecessary object area management map creation unit 105 can extract points from the three-dimensional point cloud 110 whose height is lower than a set threshold as a road surface point cloud, and can also manage the grid in which the road surface point cloud 409 (see Figure 5 (B)) is located as a road surface determined area R.

In addition, the unnecessary object area management map creation unit 105 can manage, as a road surface determined area R, grids within a range from the self-position P to the size of the 3D map creation device 1 or the mobile body 100 on which the 3D map creation device 1 is mounted.

The unnecessary object area management map creation unit 105 determines the space that the three-dimensional map creation device 1 has passed through based on the self-position P at time series t and t+1 estimated by the self-position estimation unit 104, and can manage the grid in which the passed space exists as a road surface determined area R.

The three-dimensional map creation device 1 detects road surfaces as described above, and the unnecessary object area management map creation unit 105 manages the grid containing the area detected as a road surface as a road surface confirmed area R.

When the unwanted object area management map creation unit 105 calculates the grid that will become the road surface confirmed area R using the currently acquired point cloud, if the grid was previously managed as a grid where unwanted objects 404 may exist, it updates the grid to be managed as the road surface confirmed area R.

When the processing of step 603 is completed, the processing proceeds to step 604, where the removal unit 106 performs processing to remove the point cloud 405 of the unwanted object 404 (step 604).

In step 604, the unnecessary object area management maps 401, 501 and unnecessary object detection results 303 are used to remove points 405 that exist on the grid managed as unnecessary object presence areas U and road surface confirmation areas R from the acquired three-dimensional point cloud 110 based on predetermined criteria.

Figure 8 shows an example of a junk object area management map 501 and point cloud created at time t+1, the time after Figure 7. Three-dimensional point cloud 505 shows a point cloud newly extracted as junk object 404 at time t+1, when junk object management map 501 in Figure 8 was created. Since junk object management map 501 at time t+1 basically inherits the grid information managed in junk object management map 401 at time t, it can be seen that grids located in positions managed as junk object area U and road surface area R in 401 are still managed as junk object area U and road surface area R. Furthermore, because the information of the newly extracted point cloud can be used, it can also be seen that the number of grids managed as junk object area U is increasing. Three-dimensional point cloud 515 shows a point cloud that will be removed at time t+1, when junk object management map 501 in Figure 8 was created, because they are located on grid 506 managed as junk object area U at time t, or on grid 508 managed as determined road surface area R. By using the unwanted object management map 501, it is also possible to remove point clouds 515 of unwanted objects 404 outside the camera's angle of view 403. By managing unwanted objects 404 by location rather than by object, one advantage is that unwanted objects can be removed without the need to predict their movement.

The three-dimensional map creation unit 107 creates a three-dimensional map by overlaying the point clouds not removed by the removal unit 106 in chronological order.

The removal unit 106 removes the 3D point clouds that exist on the grids 408, 508 managed as the road surface determined area R in the unnecessary object area management map 401, 501 and are at the specified height.

The removal unit 106 also removes point clouds extracted as unwanted objects 404 by the unwanted object detection unit 103 from the three-dimensional point cloud 110 acquired by the distance sensor 101. Furthermore, among point clouds on grids 406 managed as unwanted object areas U on the unwanted object area management maps 401, 501 or grids managed as determined road surface areas R, point clouds located higher than the position of point clouds considered to be road surfaces are removed. A height threshold can be set for the point clouds removed based on the unwanted object area management maps 401, 501, and point clouds located higher than the position of point clouds considered to be road surfaces but lower than the height threshold can be removed. The height threshold can also be set for each grid.

Once processing in step 604 is completed, processing proceeds to step 605, where a three-dimensional map is created (step 605).

In step 605, where a three-dimensional map is created, a three-dimensional map is created using the point cloud from which the unnecessary objects 404 have been removed. When this process is complete, the three-dimensional map creation process ends. This process is performed at specific time intervals.

This first embodiment will be described assuming a road environment as an example, but it is also possible to create a 3D map from which unwanted objects 404 have been removed by predetermining the objects to be detected as unwanted objects 404 in environments where indoor and outdoor environments coexist, such as theme parks and factories, or in indoor environments such as retail stores.

As described above, according to this first embodiment, it is possible to automate the removal of unwanted objects 404 from a three-dimensional map, which has previously been done manually. It is also possible to remove unwanted objects measured by the distance sensor 101, which measures a wider range than the camera 102.

The following describes a 3D map device 7 according to a second embodiment of the present invention, with reference to Figures 9 and 10. In the second embodiment, a road map 708 and a display unit 709 are added to the configuration of the first embodiment. Elements similar to those in the first embodiment are designated by numbers in the 700 range.

The 3D map creation device 7 includes the 3D map creation device 1 of embodiment 1, a road map 708, and a display unit 709, and displays on the road map 708 the positions indicated by the grid points managed as unwanted object areas U in the created unwanted object area management maps 401, 501.

The display unit 709 further displays on the road map 708, as unmeasured areas N, areas that are not classified as either unwanted object areas U or determined road surface areas R, and for which no point cloud was obtained by the distance sensor 701.

The display unit 709 obtains from the road map 708 a road map corresponding to the three-dimensional map created by the three-dimensional map creation unit 707. A road map is represented, for example, as a two-dimensional top view of roads observed from above. Similarly, when a three-dimensional map is observed from an aerial perspective, it can be represented as a two-dimensional top view, allowing a correspondence between the road map and the three-dimensional map. Similarly, a correspondence can also be established between the unnecessary object area management map, which corresponds to the three-dimensional map. If the coordinate system of the three-dimensional map creation unit 707 matches the coordinate system of the road map 708, the corresponding road map 708 is obtained based on the coordinates; if the coordinate systems do not match, the road map 708 is obtained using separately specified coordinates.

Referring to Figure 10, the display unit 709 displays on the road map 708 unmeasured areas N or areas containing unwanted objects that are located at a distance from the road 803 on the road map 708 that is shorter than the threshold value. This utilizes the feature that areas far from the road 803 are not necessary for autonomous driving.

The display unit 709 associates the unwanted object area management map 801 with the road map 708, and as shown in FIG. 10, among the grids on the unwanted object area management map 801 that are not managed as either unwanted object areas or road surface areas R, manages grids 805 that do not contain point clouds on the three-dimensional map as unmeasured areas N. The display unit 709 creates an image in which the positions of grids 805 managed as unmeasured areas N and grids 806 managed as unwanted object areas (see FIG. 11) on the road map 708 are indicated in a pre-specified color, and displays the created image as the display image 802. The pre-specified color may be changed for positions managed as unmeasured areas N and positions managed as unwanted object areas.

Display image 802 is an image created using an unwanted object area management map 801, which includes areas managed as unmeasured areas N, and road map 708. Display image 802 shown in the example of Figure 10 shows roads 803 and stationary objects 804 included in road map 708, as well as the positions of grid cells managed as unwanted object areas in unwanted object area management map 801 and the positions of grid cells managed as unmeasured areas N, indicated in different colors.

The display unit 709 can also indicate in a specified color only the grids 805 that are managed as unwanted object areas and unmeasured areas N within a specified range from the road.

As shown in FIG. 11, the display unit 709 can also indicate in a specified color when there are more lattices managed as unwanted object areas U and unmeasured object areas N around a lattice 806 managed as unwanted object areas U and a lattice 805 managed as unmeasured object areas N than a threshold. In other words, it can also indicate in a specified color areas where lattices managed as unwanted object areas and lattices managed as unmeasured object areas N are concentrated. This makes it possible to visualize key areas that require remeasurement on the map, making the creation of 3D maps more efficient.

As shown in FIG. 11, the display unit 709 can also display on the road map 708 only grids 901 where there are more grids 806 designated as unwanted object areas U or grids 805 designated as unmeasured areas N around the grid 806 designated as unwanted object areas U or grids 805 designated as unmeasured areas N than a threshold value. Conversely, grids 902 where there are fewer grids than the threshold value can be hidden.

Embodiment 2 not only achieves the same effects as embodiment 1, but also makes it possible to visualize locations that require remeasurement, thereby utilizing 3D maps and road maps to further improve the efficiency of 3D map creation and autonomous driving.

By being able to remove unwanted objects outside the field of view 403 of the camera 102, the system can improve the efficiency of creating 3D maps, contributing to the development of autonomous driving and other technologies, and is of great industrial value.

1. 3D map creation device
7. 3D map creation device
100 Mobile
101 Distance Sensor
102 Camera
103 Unwanted object detection unit
104 Self-position estimation part
105 Unwanted Area Management Map Creation Department
106 Removal section
107 3D Map Creation Department
110 3D point cloud
120 images
130 Projection point cloud
200 Arithmetic circuit
301 Automobiles
302 people
303 Unwanted object detection results
401 Unwanted Area Management Map
403 angle of view
404 Unwanted items
405 3D point cloud
406 Lattice
407 Road surface
408 Lattice
409 point cloud
505 point cloud
515 point cloud
501 Unwanted Area Management Map
502 point cloud
506 Lattice
707 3D Map Creation Department
709 Display section
802 display images
803 Road
804 Stationary object
805 Lattice
P self-position
R Road surface determination area
U Unwanted area t, t+1 Time

Claims (17)

a distance sensor installed on a moving body for acquiring a three-dimensional point cloud representing the surrounding environment of the moving body; a camera for acquiring an image representing the surrounding environment within a field of view; and a calculation means for creating a three-dimensional map based on the three-dimensional point cloud acquired from the distance sensor and the image acquired from the camera,
The calculation means
an unwanted object detection unit that detects unwanted objects that are not required for the three-dimensional map based on the image and outputs the detected unwanted objects as unwanted object detection results;
a self-position estimation unit that estimates the position of the moving object;
an unwanted object area management map creation unit that creates an unwanted object area management map that indicates an unwanted object area that is an area where the unwanted object may exist, based on the three-dimensional point cloud acquired in time series and the unwanted object detection result;
a removal unit that removes specific points from the three-dimensional point cloud using the unnecessary object area management map;
a three-dimensional map creation unit that creates a three-dimensional map using the point cloud that was not removed by the removal unit;
A three-dimensional map creation device comprising: The 3D map creation device according to claim 1, characterized in that the unnecessary object area management map divides the space for which the map is to be created into a grid, and manages the grid in the space in which the unnecessary object detection unit determines that unnecessary objects exist as the unnecessary object area. The 3D map creation device described in claim 2, characterized in that the unnecessary object area management map creation unit changes the number of grids managed as unnecessary object areas depending on the type of unnecessary object detected by the unnecessary object detection unit. The 3D map creation device according to claim 2, characterized in that the removal unit removes point clouds that exist on grids managed as the unnecessary object areas in the unnecessary object area management map. The 3D map creation device according to claim 4, characterized in that the removal unit sets a height threshold for the height of the managed point cloud in the grid managed as the unnecessary object area, and removes the point cloud that falls within the range indicated by the height threshold. The 3D map creation device of claim 2, characterized in that the 3D map creation device further detects road surfaces based on the image or the 3D point cloud, and the unnecessary object area management map creation unit manages the grid containing the area detected as a road surface as a confirmed road surface area. The 3D map creation device of claim 2, characterized in that the unnecessary object area management map creation unit further determines the space through which the 3D map creation device has passed based on the time-series self-position estimated by the self-position estimation unit, and manages the grid in which the passed space exists as a determined road surface area. The 3D map creation device described in claim 6, characterized in that the unnecessary object area management map creation unit detects road surfaces by determining areas to be judged as road surfaces from the 3D point cloud acquired by the distance sensor. The 3D map creation device according to claim 6, characterized in that the removal unit removes point clouds that exist on grids managed as road surface determined areas in the unnecessary object area management map and are at a specified height. The 3D map creation device according to claim 1, characterized in that the unnecessary object detection unit further detects unnecessary objects from the 3D point cloud and outputs the unnecessary object detection result. The 3D map creation device according to claim 10, characterized in that the unwanted object detection unit detects unwanted objects from the image within the camera's field of view, and detects unwanted objects from the 3D point cloud outside the camera's field of view. The 3D map creation device described in claim 2, characterized in that the unnecessary object area management map creation unit manages the area of the 3D map using a grid, and changes the size of the grid managed as the unnecessary object area depending on the type of unnecessary object detected by the unnecessary object detection unit. The 3D map creation device according to any one of claims 1 to 12, characterized in that it uses a grid to manage the areas of the 3D map, and further comprises a road map and a display unit, and in the created unwanted object area management map, the positions indicated by the grid managed as unwanted object areas are displayed on the road map. The three-dimensional map creation device according to claim 6 or 7, characterized in that the three-dimensional map creation device manages areas of the three-dimensional map using a grid, and further comprises a road map and a display unit, and in the created unnecessary object area management map, positions indicated by the grid managed as unnecessary object areas are displayed on the road map, and the display unit displays areas that are not designated as unnecessary object areas or determined road surface areas and for which no point cloud was obtained by the distance sensor as unmeasured areas on the road map. The 3D map creation device described in claim 14, characterized in that the display unit displays, on the road map, the unmeasured areas or the unwanted object areas that are located at a distance from the road on the road map that is shorter than a threshold value. The 3D map creation device of claim 14, characterized in that the display unit displays, on the road map, grids in which the number of grids designated as unwanted object areas or unmeasured areas surrounding the grids designated as unwanted object areas or unmeasured areas is greater than a threshold value. A method for creating a three-dimensional map, in which a computer executes the following steps:
an unwanted object detection step of detecting unwanted objects from an image acquired from the camera and creating an unwanted object detection result;
a waste management map creation step of creating a waste management map showing a grid indicating a waste area based on the three-dimensional point cloud acquired from the distance sensor and the waste detection result ;
an unwanted object point cloud removal step of removing a point cloud indicating unwanted objects from the three-dimensional point cloud acquired from the range sensor by using the unwanted object management map;
a three-dimensional map creation step of creating a three-dimensional map using the point cloud from which the point cloud indicating the unnecessary objects has been removed in the unnecessary object point cloud removal step ;
A three-dimensional map creation method comprising: