JP7651320B2 - Information processing device, information processing method, and program - Google Patents

Description

The present invention relates to a technology for creating three-dimensional map information of an environment in which a mobile object moves.

There are methods for measuring the position and orientation of a moving body such as an autonomous mobile robot, and for creating electronic map data to be used for position and orientation measurement. A known technology is Simultaneous Localization and Mapping (SLAM), which uses cameras and laser range scanners (laser range finders, Laser Imaging Detection and Ranging (LIDAR)) as sensors. If the path of the moving body changes, or if the location of surrounding objects changes, new map elements are added to the map data.

Non-Patent Document 1 discloses a technique for reducing the memory capacity required to hold map data (three-dimensional map information) by deleting images contained in a map element if the images have similar characteristics to those contained in other map elements.

Raul Mur-Artal, J.M.M. Montiel,Member, IEEE,and Juan D. Tardos,Member, IEEE, “ORB-SLAM: a Versatile and AccurateMonocular SLAM System” 2015

In the method of Non-Patent Document 1, if a map element cannot be complemented by another map element, it cannot be deleted, which may result in insufficient storage capacity.

The present invention was made in consideration of the above problems, and aims to reduce the memory capacity required to store map elements of a three-dimensional map of the environment in which a mobile object moves.

In order to solve the above problems, the information processing device according to the present invention has the following configuration: an acquisition means for acquiring three-dimensional map information from a storage means for storing three-dimensional map information composed of a plurality of map elements, the map elements being the position and orientation of an imaging device estimated by capturing an image of an environment with the imaging device mounted on a moving body, three-dimensional position information in the environment of feature points included in an image obtained by capturing the image of the environment, and the image, and a determination means for determining one of the map elements as a deletion target to be deleted from the storage means when, for two map elements having similar positions and orientations of the imaging devices included in the map elements, a common area captured in a common field of view in the images has a size that is not similar in features that is equal to or larger than a threshold value .

The present invention makes it possible to reduce the memory capacity required to store map elements of a three-dimensional map of the environment in which a mobile object moves.

1 is a diagram illustrating a hardware configuration of an information processing device according to a first embodiment. 1 is a functional block diagram of an information processing device according to a first embodiment. FIG. 1 is a functional block diagram of a mobile body system and a map creation system according to a first embodiment. 4 is an operation flowchart of the information processing device according to the first embodiment. 4 is an operation flowchart of the information processing device according to the first embodiment. FIG. 13 is a diagram showing an example of a comparison of map elements. 10 is an operation flowchart of the information processing device according to the second embodiment. FIG. 13 is a functional block diagram of an information processing device according to a third embodiment. 13 is an operation flowchart of the information processing device according to the third embodiment. FIG. 13 is a diagram showing an example of the selection of a map element.

Below, preferred embodiments of the present invention are described in detail with reference to the drawings. Note that these embodiments are merely examples and are not intended to limit the scope of the present invention.

First Embodiment
As a first embodiment of an information processing device according to the present invention, an information processing device that determines map elements to be deleted based on the degree of change in the object placement around a sensor will be described below. Specifically, when the object placement around a sensor is changed, different map elements before and after the placement change are stored, which requires storage capacity. Therefore, in the case of a temporary placement change, the map elements after the placement change are deleted because they are unlikely to be used again.

In this embodiment, the sensor is a camera that outputs a two-dimensional array of brightness value data. The camera may be a monochrome camera, a color camera, or a stereo camera. The internal parameters of the camera are assumed to be known through prior calibration.

FIG. 1 is a diagram showing the hardware configuration of an information processing device according to the first embodiment. The information processing device 100 includes a CPU 101, a memory 102, a storage device 103, an input device 104, and an output device 105, which are interconnected by a bus 106.

The CPU 101 executes various functions described below by executing a determination program 107 for map elements to be deleted that is stored in the storage device 103. The memory 102 temporarily stores programs and data that the CPU 101 reads from the storage device 103. The memory 102 is also used as an area for the CPU 101 to execute various programs. The storage device 103 stores the operating system (OS), map elements for traveling along a route, and map elements obtained during traveling.

The input device 104 is a functional unit that accepts input from the user. Specifically, a keyboard and a mouse are used. The output device 105 outputs information input by the input device 104 and the results of execution of a program executed by the CPU 101. The communication interface 107 communicates information via a network, and the communication interface may be Ethernet, or any type of interface such as USB, serial communication, or wireless communication.

Figure 2 is a functional block diagram of the information processing device 100 according to the first embodiment. The information processing device 100 includes an acquisition unit 201, a determination unit 202, and a movement/deletion unit 203.

The acquisition unit 201 acquires and outputs, as environmental condition information, images captured by a camera from the position and orientation measurement unit 301 of the mobile system 300 described below, three-dimensional position information of feature points detected from the images, and map elements linked to the capture time and the camera position and orientation information.

The determination unit 202 determines the map elements to be deleted or moved from the map elements input from the acquisition unit 201, and outputs them.

The move/delete unit 203 moves or deletes the target map element input by the decision unit 202 to a specified location.

Figure 3 is a functional block diagram of a mobile body system 300 and a map creation system 310 according to the first embodiment. Information processing of the mobile body system 300 and the map creation system 310 is performed by the same hardware configuration as that of the information processing device 100. The mobile body system 300 includes a position and orientation measurement unit 301, a mobile body control unit 302, and a mobile body system communication unit 303. The map creation system 310 includes a map generation unit 311, a map storage unit 312, an environment measurement unit 313, and a map creation system communication unit 314. The mobile body system 300, the map creation system 310, and the information processing device 100 can all be mounted on a mobile body. The information processing device 100 is located on a server outside the mobile body, and the mobile body can be controlled by communication between the information processing device 100 and the mobile body via a network.

Examples of the mobile object include an AMR (autonomous mobile robot), an AGV (automatic guided vehicle), an autonomous vehicle, a cleaning robot, a delivery robot, and a drone. The mobile object is equipped with a camera as an imaging device.

The position and orientation measurement unit 301 measures the position and orientation of the mobile system 300 from the surrounding environment, and outputs the results as map elements. Although the measurement method is not shown, it uses an image captured by a camera mounted on the mobile system 300, and calculates using the distance from the camera to the object and the position information of the image feature points. As long as the features of the image feature points are obtained from the gradient value of the brightness, various types of features such as SIFT, SURF, and FAST can be used.

The mobile control unit 302 controls the travel of the mobile system 300 based on the results of the position and orientation measurement unit 301 and information and instructions received from the mobile system communication unit 303 described below, and outputs the results of the position and orientation measurement unit 301 to the mobile system communication unit 303.

The mobile system communication unit 303 outputs the results of the position and orientation measurement unit 301 to the map creation system 310, which is an external module, and inputs information from the map creation system 310.

The map generation unit 311 generates three-dimensional map information (hereinafter, "map" will be explained as "three-dimensional map") from the position and orientation measurement results measured by the mobile system 300 received from the map creation system communication unit 314 described below. The three-dimensional map information is composed of multiple map elements.

The map storage unit 312 stores the map generated by the map generation unit 311 and information received from the environmental measurement unit described below, and outputs the stored information to the map creation system communication unit 314.

The environmental measurement unit 313 uses a sensor to output that information. Specifically, it is the image and time when the map element was created.

The map creation system communication unit 314 is connected to the mobile system communication unit 303 of the mobile system 300 and the acquisition unit 201 of the information processing device 100, and inputs and outputs information related to maps and map elements.

Figures 4 and 5 are operational flowcharts of the information processing device 100 according to the first embodiment. Hereinafter, it is assumed that the flowcharts are realized by the CPU executing a control program. The operation of the information processing device is started when the memory capacity of the map storage unit 312 reaches a predetermined capacity, or when a predetermined time has elapsed since the moving object started to move.

In step S401, a group of map elements is obtained from the map creation system 310.

In step S402, the map elements to be deleted are determined from the map elements obtained in step S401.

In step S403, the map element determined in step S402 is deleted.

Here, we will explain in detail the determination step of step S402.

In step S501, an unselected map element is selected from the group of map elements obtained in step S401.

In step S502, from among the map elements acquired in step S401, map elements other than the map element selected in S501 are targeted, and map elements with camera positions and orientations similar to the camera position and orientation associated with the map element selected in S501 are extracted. Map elements with similar camera positions and orientations (differences within a specified range) are extracted. Then, the map element with the most recent shooting time is extracted.

In step S503, a common area photographed in a common field of view between the map element selected in step S501 and the map element extracted in step S502 is determined using the characteristics of the image feature points associated with each map element.

In step S504, the two image areas, the common area found in S503 of the image linked to the map element selected in S501, and the common area found in S503 of the image linked to the map element extracted in S502, are divided into grid areas with the same number of divisions.

In step S505, the average brightness value is calculated within each grid area of the two image areas, and the number of grid areas or the percentage of the image where the absolute value of the difference between the average brightness values for each grid area is equal to or greater than a predetermined threshold value P is determined.

In step S506, it is determined whether the number of regions found in step S505 is equal to or greater than a predetermined threshold value R. If the number of regions found in step S505 is equal to or greater than the predetermined threshold value R, the process proceeds to step S507. If the number of regions found in step S505 is equal to or less than the predetermined threshold value R, the image is important for improving the position and orientation accuracy in the future, and is likely to be used in the future, so it is not subject to deletion.

In step S507, the map element selected in step S501 is determined to be deleted.

In step S508, it is determined whether to continue the process of determining which map elements to delete. If there are any unselected map elements remaining from the group of map elements acquired in step S401, the process returns to step S501 and continues. If not, the process proceeds to step S403 and deletes the map elements.

Figure 6 is a schematic diagram of map elements to which environmental condition information and position and orientation information are linked. 601 and 602 are example images linked to map elements selected/extracted in steps S501 and S502, respectively. Object 603 has been added to the latest captured image 602 since image 601 was captured. Area 604 is the common area determined in step S503. On the other hand, area 605 is not a common area and cannot be compared. Locations where there are changes in object placement within this common area 604 are calculated. The image from step S504 is divided as shown in 606, and locations where there are changes in object placement from step S505 are marked as partial area 607. Partial area 607 where there is a change in object placement is an area where the features are not similar between images 601 and 602. If the condition for deletion is that a grid area of 20% or more of the image area has changed, then the partial area 607 has changed by 6/28, or approximately 21%, and the captured image 602 is determined to be a map element to be deleted.

As described above, the information processing device 100 according to the first embodiment can reduce memory capacity by deleting map elements based on the degree of change in the placement of objects around the sensor.

<Modification 1-1>
In the first embodiment, map elements whose object location has changed are to be deleted, but map elements to be deleted may be determined based on the usage history of the map elements. When a moving body moves autonomously, it estimates its own position and orientation by referring to map elements that have been created in advance or map elements that have been created during previous travel. For this reason, the use time and number of times of use of the map elements used in the estimation may be stored, and map elements that have been used a long time ago or have been used infrequently may be determined to be to be deleted.

<Modification 1-2>
Map elements may be assigned a degree of importance, and map elements with a high degree of importance may be excluded from the determination of map elements to be deleted as shown in the first embodiment and the modified examples. This degree of importance is set based on a user's instruction or the history of map usage. If the environment in which a moving object moves is measured before executing the process described in this embodiment and a three-dimensional map of the environment is created in advance, the importance of the map elements of the three-dimensional map can be set high and excluded from the determination of map elements to be deleted.

<Modification 1-4>
In the first embodiment and the modified example, the map element to be deleted is determined from the difference in the numerical value obtained from the image that is the environmental condition information, but the map element to be deleted may be determined from the contents of the image. Specifically, an object present in the image is detected by machine learning, image recognition, or the like. It is determined whether or not the detected object is likely to move. If a moving object is present in the image and its proportion in the image is small, the object area is treated as an area 408 that cannot be compared, and it is determined whether or not there is a change in the object arrangement in other parts. Also, if the moving object occupies a large proportion of the image, it is determined as a map element to be deleted. In addition to the size of the object, the range, speed, and frequency of the movement may be taken into consideration to determine the map element to be deleted. For example, in the case of a scene where the object passes by quickly, it may not be present the next time it is photographed, so it can be determined as a deletion target.

<Modification 1-5>
Instead of recognizing objects that may move as in Modification 1-4, an object may be recognized as having actually moved and determined as a deletion target. Also, a threshold may be used to determine the amount of movement or the number of objects that have moved to select deletion targets.

Second Embodiment
In the first embodiment, an example of determining a map element to be deleted based on the degree of change in the object arrangement around the sensor has been described. In addition, when the brightness around the sensor is changed, the sensor obtains different information even with the same arrangement, so that different map elements before and after the brightness change are stored, which may increase the storage capacity. Therefore, in the second embodiment, an example of determining a map element to be deleted based on the degree of change in brightness around the sensor will be described. Specifically, the brightness changes gradually over time, and map elements with a large difference in brightness that are unlikely to be used are deleted. In addition, when a temporary brightness change occurs, the map element is determined to be deleted. In this embodiment, the sensor is a camera that outputs a two-dimensional array of brightness value data.

The hardware configuration of the information processing device according to the second embodiment is the same as that shown in FIG. 1, so a description thereof will be omitted.

The functional block diagram of the information processing device 100 according to the second embodiment is the same as that shown in FIG. 2, and the acquisition unit 201 also acquires images as environmental condition information and map elements to which position and orientation information are linked. The mobile system 300 and map creation system 310 according to the second embodiment are also the same as those shown in FIG. 3, but input and output brightness information at the time the map elements were created as information measured by the environmental measurement unit 313.

Figure 7 is an operational flowchart of step S402 of the information processing device 100 according to the second embodiment. The process in Figure 4 is the same as that in the first embodiment.

Steps S701 to S703 and steps S706 to S707 are the same as steps S501 to S503 and steps S507 to S508 in FIG. 5, so the explanation will be omitted.

In step S704, the average pixel value of each image is calculated for the two image regions determined in step S703.

In step S705, the average brightness value is calculated for each image of the two image regions, and it is determined whether the absolute value of the brightness difference is greater than or equal to a predetermined threshold value P.

As described above, the information processing device 100 according to the second embodiment can reduce memory capacity by deleting map elements based on the degree of brightness fluctuation around the sensor.

<Modification 2-1>
In the first embodiment, an example was described where the brightness fluctuates slowly. However, in the case of temporary brightness fluctuations, the fluctuation in the average brightness value of the image in the time direction may be found, and map elements at points in time with high kurtosis may be determined to be targets for deletion.

<Modification 2-2>
In the first embodiment, an example was described in which brightness information obtained from an image was used, but the comparison may also be performed using brightness information obtained from a sensor other than a camera. Specifically, a luminance meter is provided that is fixed in the sensor while maintaining a predetermined relative position and orientation relationship, and illuminance measured in synchronization with the sensor measurement is stored as environmental condition information. The illuminance is compared with the illuminance at the time of the map element to be compared, and if the absolute value of the illuminance difference is equal to or greater than a predetermined threshold, the map element may be determined to be deleted.

<Modification 2-3>
In the first embodiment and the modified example, the determination is made using direct brightness information obtained from an image or a sensor, but the determination may be made from other brightness information. Specifically, weather forecasts and weather result information provided on the Internet or the like are received and used. This weather forecast and result information are linked to the map element creation position and stored. If the map element to be compared is a current one, the change in brightness may be determined based on the change in weather using weather forecast information, or if the map element to be compared is a past map element, the weather, cloud cover, sunrise, and sunset information from the weather result information.

Third Embodiment
In the first and second embodiments, the information processing device 100 determines the map elements to be deleted, but in the third embodiment, an example will be described in which the map elements to be deleted are selected in the first and second embodiments, and the information is presented to the user to delete the map elements. Here, the method of the first embodiment will be used for the explanation, but there is no problem if the method of the second embodiment is used.

The hardware configuration of the information processing device according to the third embodiment is the same as that shown in FIG. 1, so a description thereof will be omitted.

Figure 8 is a functional block diagram of an information processing device 100 according to the third embodiment. In addition to Figure 2, a display unit 204 is added. The display unit 204 displays the information selected by the decision unit 202 and receives instructions from the user.

Figure 9 is a flowchart of the information processing device 100 according to the third embodiment. Steps S901 to S907 are the same as steps S501 to S507 in Figure 5, so their explanation will be omitted.

In step S908, information about the map elements to be deleted determined in step S907 is output to the output device 105.

In step S909, the system waits for input from the user. If a deletion instruction is received from the user, the system proceeds to the next step S910, where the target map element is deleted.

In step S911, it is determined whether to continue the selection process. If the selection process is to be continued, the process returns to step S901 and is repeated. If the selection process is not to be continued, the process ends.

Figure 10 shows the output in step S908 of the flowchart in Figure 9. Display area 1001 displays information 1002 of the map element to be deleted, along with information 1003 of map elements that are frequently used while driving and that are photographed in a similar location and with a similar angle of view compared to the map element to be deleted. The user selects the map elements to be deleted from this information, and deletes the selected map elements in step S910.

As described above, the information processing device 100 according to the third embodiment allows the user to select map elements to be deleted, and by deleting the selected map elements, storage capacity can be reduced.

Other Embodiments
In the above embodiment, an example of deleting a selected map element has been described, but the map element may be moved to a predetermined location instead of being deleted. Specifically, the storage capacity of the map storage unit 312 can be reduced by moving the map element from the memory 102 in the information processing device 100 to the storage device 103, or by moving the map element from the storage device 103 in the information processing device 100 to a storage device of an external system.

Other Embodiments
The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the programs. The programs may also be provided by recording them on a computer-readable recording medium.

Claims (9)

an acquisition means for acquiring three-dimensional map information from a storage means for storing three-dimensional map information composed of a plurality of map elements, the three-dimensional map information being defined as map elements including a position and orientation of an imaging device mounted on a moving body that is estimated by capturing an image of an environment with the imaging device, three-dimensional position information in the environment of feature points included in the image captured by capturing the image of the environment, and the image;
and a determination means for determining that one of the map elements is to be deleted from the storage means when, in two map elements in which the position and orientation of the imaging device contained in the map element are similar, the size of an area in the images in which the features are not similar in a common area captured in a common field of view is equal to or larger than a threshold value. The information processing device according to claim 1, characterized in that the determining means determines the second map element to be deleted when a size of a dissimilar area between images included in the first map element and the second map element is equal to or larger than a threshold value, between a first map element in first three-dimensional map information created in advance for the environment, and a second map element in second three-dimensional map information created in association with the movement of the moving body and having a position and orientation similar to that included in the first map element. The information processing device according to claim 1 or 2, characterized in that the region in which the features between the images are not similar is a partial region of the images. 3. The information processing device according to claim 1, wherein the determining means determines one of the map elements to be deleted from the storage means when a size of an area in which the luminance characteristics between the images are not similar is equal to or larger than a threshold value. The information processing device according to any one of claims 1 to 4, characterized in that the determining means determines the map element to be deleted from two map elements in which the difference in the position and orientation of the imaging device contained in the map element is within a predetermined range. 6. The information processing apparatus according to claim 1 , wherein a difference between average values of luminance values of pixels in an area having dissimilar features between the images is equal to or greater than a threshold value. The information processing device according to claim 1 or 2, characterized in that the determination means determines one of the map elements to be deleted from the storage means when the size of an area in which the features obtained from the brightness gradient values between the images are not similar is equal to or greater than a threshold value. an acquisition step of acquiring three-dimensional map information composed of a plurality of map elements, the map elements being the position and orientation of an imaging device mounted on a moving body estimated by capturing an image of an environment with the imaging device, three-dimensional position information in the environment of feature points included in the image captured by capturing the image of the environment, and the image;
and a determining step of determining, for two map elements having similar positions and orientations of the imaging devices contained in the map elements, that one of the map elements is to be deleted when the size of an area having dissimilar features in a common area captured in a common field of view in the images is equal to or larger than a threshold . A program that causes a computer to function as each of the means of an information processing device according to any one of claims 1 to 7.

Images