JP7657580B2 - Information processing device, information processing method, and program - Google Patents

Description

The present invention relates to a technology for estimating the position of a moving object from sensor input information.

There is a technology for estimating the position of a moving object equipped with a sensor that measures the distance between the sensor and objects in the environment, based on the measurement information obtained from the sensor. In this case, the positional relationship between the moving object and the sensor is measured (calibrated) in advance, so that the position of the moving object can be estimated with higher accuracy. In Patent Document 1, the sensor measures the reference target when the robot is moved a predetermined amount so that the robot and the reference target have a predetermined relative positional relationship, and this is repeated until the measurement result falls within the error range.

JP 2009-279677 A

However, the method of Patent Document 1 does not take into consideration the possibility that the relative positional relationship between the measurement device (sensor) and the robot (mobile body) may change to an inappropriate positional relationship due to some unpredictable cause, making it difficult to detect the occurrence of such an event. Furthermore, if such an event occurs, it will ultimately be impossible to accurately estimate the position of the mobile body. The present invention has been made in consideration of the above problems, and aims to make it possible to detect when the position and orientation of the sensor with respect to the mobile body has shifted from the calibrated position and orientation.

The information processing device according to the present invention for solving the above-mentioned problems has the following configuration: That is, the information processing device has an estimation means for estimating first position information indicating a position of a moving body equipped with a sensor based on input information input from the sensor when the moving body makes a predetermined movement, an acquisition means for acquiring second position information indicating a position of the moving body that is expected to be obtained based on the input information input from the sensor when the moving body makes the predetermined movement while maintaining the position and orientation of the sensor relative to the moving body held in the holding means, and an output means for outputting information regarding changes in the position and orientation of the sensor relative to the moving body when the moving body makes the predetermined movement and the position and orientation of the sensor relative to the moving body held in the holding means, based on the first position information and the second position information.

The present invention makes it possible to detect when the position and orientation of a sensor relative to a moving object deviates from the maintained position and orientation.

FIG. 1 is a diagram showing an example of a system configuration of an information processing system. FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device; FIG. 1 is a diagram showing an example of a hardware configuration of an information processing device. A flowchart for explaining a process executed by an information processing device. A flowchart for explaining a process executed by a change amount acquisition unit. A diagram explaining a method for acquiring the amount of change in the attitude parameter of a moving object. A flowchart for explaining a process executed by a change amount acquisition unit. A diagram explaining a method for acquiring the amount of change in the position parameter of a moving object. A flowchart for explaining a process executed by an information processing device. FIG. 1 shows an example of a GUI. FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device; A flowchart for explaining a process executed by an information processing device.

<Embodiment 1>
In this embodiment, the present invention is applied to an information processing system including a mobile body equipped with a sensor for measuring the distance between an object in the environment and the sensor. In this embodiment, the mobile body is specifically an automatic guided vehicle. When the mobile body plans a route using a position and orientation estimated based on the measurement value of the sensor and travels along the plan, it is necessary to convert the measurement results from the sensor into a format used in the movement control of the mobile body. This conversion requires a relative positional relationship between the mobile body and the sensor. Therefore, the positional relationship between the mobile body and the sensor is measured (calibrated) in advance before the mobile body starts traveling. The relative positional relationship between the calibrated sensor and the mobile body is defined as an appropriate positional relationship. However, while the mobile body travels, the sensor may move from the position or orientation of the calibrated state. In other words, the relative positional relationship between the sensor and the mobile body may become inappropriate due to the movement of the mobile body. If there is a change in the actual physical relative positional relationship, the mobile body travels based on the position estimation result of the mobile body that includes the change in the position and orientation of the sensor as noise, so that the mobile body may deviate from the specified route. Therefore, in this embodiment, a change in the positional relationship between the sensor and the mobile body when the mobile body is moved straight ahead a specified distance is detected. This allows the system to detect a change in the relative positional relationship between the sensor and the moving object, and if there is a change, notify the user that the relative positional relationship between the sensor and the moving object is inappropriate. In addition, a method for allowing the user to correct the positional relationship between the sensor and the moving object will be described. In this embodiment, the sensor refers to a stereo camera (image sensor).

Figure 1 shows an example of the configuration of an information processing system 1 in this embodiment. The information processing system 1 is composed of a mobile object 10 that travels along a predetermined route, a sensor 12 that measures the shape of objects present in the environment, an information processing device 11 that estimates the position and orientation of the sensor, a control device 13 that controls the mobile object, and an output device that outputs information regarding deviations in positional relationship. Note that in this embodiment, the mobile object moves on a horizontal plane, and is, for example, an unmanned guided vehicle that transports luggage.

In this embodiment, movement control refers to controlling the motor, which is an actuator serving as the control device 13 equipped on the moving body, and the steering, which changes the direction of the wheels. Specifically, the control device 13 causes the moving body to move straight forward, backward, left and right, and to turn at a predetermined angle. By inputting control information into the control device 13, the moving body is moved along a predetermined path. Furthermore, movement control information is information necessary to control the control device 13, and refers to the rotational torque and traveling direction of the wheels. By setting the number of rotations of the wheels and the traveling direction in advance as movement control information for a predetermined operation, the moving body can be made to move along a fixed trajectory regardless of the position and orientation of the moving body or the sensor.

In this embodiment, the position of the sensor 12 is defined by three parameters (X, Y, Z) that represent the position of the sensor 12 in an arbitrary world coordinate system defined in real space. The orientation of the sensor 12 is defined by three parameters (Roll, Pitch, Yaw) that represent the orientation of the sensor 12. These six parameters combined indicate the position and orientation of the sensor 12. The three-dimensional coordinate system defined on the sensor, with the optical axis of the sensor 12 as the Z axis, the horizontal direction of the image as the X axis, and the vertical direction as the Y axis, is called the sensor coordinate system.

Meanwhile, the position of a moving body in this embodiment is defined by two position parameters of the moving body on an arbitrary world coordinate system defined in real space (positions X and Y on a plane horizontal to the floor of the real space). The orientation of the moving body is defined by and one orientation parameter (direction of rotation on a plane horizontal to the floor of the real space). A total of three parameters, combining these, define the position and orientation of the moving body. A two-dimensional coordinate system defined on the moving body, with the Y axis representing the moving direction of the moving body and the X axis representing the direction perpendicular to the moving direction, is called the moving body coordinate system. It is assumed that the moving body in this embodiment moves on a horizontal plane.

Furthermore, in this embodiment, the relative positional relationship between the sensor 12 and the moving body refers to the relative position and orientation between the origin of the sensor coordinate system and the origin of the moving body coordinate system, and is expressed by a total of six parameters: three parameters that represent the position and three parameters that represent the orientation. This relative position and orientation is used to convert the position and orientation between the sensor coordinate system and the moving body coordinate system. It is assumed that this positional relationship between the sensor and the moving body has been calibrated in advance using known technology. In the following process, if the orientation (position) of the sensor changes from its initial state due to vibration of the moving body or some other external factor, it is detected that the positional relationship has changed from the one previously known by calibration.

Fig. 2 is a diagram showing an example of the functional configuration of the information processing device 11 in this embodiment. The information processing device 11 is composed of a movement control unit 111, a position and orientation estimation unit 112, a position and orientation conversion unit 113, a change amount acquisition unit 114, a calibration necessity determination unit 115, a relative position and orientation calibration unit 116, and an output unit 117. The position and orientation estimation unit 112 is connected to the sensor 12. The movement control unit 111 is connected to the control device 13. The information processing device 11 is also connected to the control device 13 that controls the movement of the moving body, and the output device 14 that outputs information about the relative positional relationship between the sensor and the moving body.

The movement control unit 111 generates movement control information for controlling the movement of the moving body and outputs it to the control device 13 and the change amount acquisition unit 114.

The position and orientation estimation unit 112 estimates the position and orientation of the sensor based on the measurement value obtained from the sensor 12, and outputs the position and orientation of the sensor to the position and orientation conversion unit 113. The sensor 12 is, for example, a stereo camera, and acquires image information as input information. Hereinafter, the input information from the sensor 12 will be described as measurement information or measurement value for convenience. The position and orientation conversion unit 113 converts the position and orientation of the sensor input by the position and orientation estimation unit 112 from the sensor coordinate system to the moving body coordinate system, and outputs it to the change amount acquisition unit 114. The change amount acquisition unit 114 outputs the change amount of the relative positional relationship between the sensor 12 and the moving body based on the conversion result input by the position and orientation conversion unit 113 and the displacement of the position and orientation of the moving body when a predetermined operation that is known in advance is performed. The acquired change amount is output to the calibration necessity determination unit 115 or the output unit 117. The calibration necessity determination unit 115 determines whether calibration is necessary using the change amount of the relative position and orientation as an input. The determination result is output to the relative position and orientation calibration unit 116. The relative position and orientation calibration unit 116 calibrates the relative position and orientation between the sensor 12 and the moving body based on the result of the determination of whether or not calibration is required. The output unit 117 outputs output information indicating whether the positional relationship between the sensor 12 and the moving body after a predetermined operation is maintained based on the amount of change in the relative positional relationship between the sensor 12 and the moving body. That is, the output unit 117 outputs output information according to the difference between the position of the moving body estimated by the position and orientation conversion unit 113 (first position information) and the predicted position of the moving body after a predetermined operation associated with the predetermined operation (second position information) to the output device 14. The second position information is the expected position of the moving body that is obtained when the moving body performs a predetermined operation while maintaining the relative positional relationship between the sensor and the moving body. This makes it possible to detect that the relative positional relationship between the sensor and the moving body measured in advance has shifted before or during driving, and to notify the user of this.

Figure 3 is a diagram showing the hardware configuration of the information processing device 11. H11 is a CPU, which controls various devices connected to the system bus H20. H12 is a ROM, which stores the BIOS program and the boot program. H13 is a RAM, which is used as the main storage device of the CPU H11. H14 is an external memory, which stores the programs processed by the information processing device 11. The input unit H15 is a keyboard, mouse, or robot controller, which performs processing related to the input of information, etc. The display unit H16 outputs the calculation results of the information processing device 11 to a display device according to instructions from H11. The display device may be a liquid crystal display device, a projector, an LED indicator, or any other type. H17 is a communication interface, which communicates information via a network. The communication interface may be Ethernet (registered trademark), or may be USB, serial communication, wireless communication, or any other type. H17 is an I/O, which inputs an image from a sensor H18. The sensor H18 is the sensor 12 described above. H19 is the control device 13 described above.

Figure 4 is a flowchart showing the processing procedure in this embodiment. The processing consists of initialization S101, movement control S102, position and orientation estimation S103, position and orientation conversion S104, change amount acquisition S105, calibration necessity determination S106 and S107, relative position and orientation calibration S108, and system termination determination S109. The processing shown in the flowchart in Figure 4 is executed by the computer CPU H11 in Figure 3, which is a computer, in accordance with a computer program stored in external memory H14. In the following description, each process (step) is represented by adding an S to the beginning, and the notation of the process (step) is omitted.

In S101, the system is initialized. That is, a program is read from the external memory H14, and the information processing device 11 is put into an operable state. In addition, parameters of each device connected to the information processing device 11 and the relative position and orientation of the sensor 12 and the mobile object are read into the RAM H13. In addition, each control device of the mobile object is started up, and put into an operable and controllable state.

In S102, the movement control unit 111 causes the moving object to perform a predetermined action based on control information for the moving object to perform the predetermined action. Here, the predetermined action is to move straight a predetermined distance. Control information such as how many meters to move forward may be acquired or generated. Note that in this embodiment, control is performed to fix the moving direction to forward (or backward) so that the moving object moves straight. This makes it possible to detect changes in the attitude (orientation) of the sensor without human intervention.

In S103, the position and orientation estimation unit 112 estimates the position and orientation of the sensor based on the measurement information measured by the sensor 12 at a second time point after the predetermined operation. If the sensor is an image sensor, the measurement information is image information. Here, the trajectory of the position and orientation of the sensor is estimated based on the measurement information measured by the sensor 12 between the first time point when the predetermined operation is started and the second time point after the predetermined operation. The estimated position and orientation of the sensor is called the first position information. In this embodiment, SLAM (Simultaneous Localization and Mapping) technology is used. A specific example of SLAM processing will be described. SLAM is a method of using feature points detected on an image to estimate the position and orientation of a sensor, etc. The information processing device holds, for example, three-dimensional point cloud information indicating the structural features of walls and stationary objects in a scene in which a moving body carrying a sensor runs as map information of the scene. Specifically, the information on the three-dimensional point cloud in the scene is the three-dimensional coordinates of the feature points in the world coordinate system. In this embodiment, map information of the scene is generated in advance based on images of the shapes of objects existing in the environment captured while a mobile body is traveling, and is stored in a storage device. Based on the input image and the map information, the information processing device minimizes the difference (reprojection error) between the positions of the feature points (edge features indicating the surrounding shape) on the input image and the positions when the feature points on the map are projected onto the input image using the position and orientation of the sensor when the input image was captured. This estimates the position and orientation of the sensor. Any other method may be used as long as it is a method of estimating the position and orientation of the sensor from measurement information.

Note that here we use the method of Engel et al. (J. Engel, T. Schps, and D. Cremers. LSD-SLAM: Large-Scale Direct Monocular SLAM. ECCV, 2014). This method simultaneously performs self-location estimation and map creation in a wide-area environment such as outdoors. When a moving object is stationary before starting to move, the position and orientation estimation unit 112 outputs the origin of the sensor coordinate system as the position and orientation of the sensor. In addition, while the moving object is performing a specified operation, the position and orientation of the sensor continues to be measured at regular time or distance intervals, and all measurement information is output to the position and orientation conversion unit 113. Note that the measurement information is array data of six position and orientation parameters.

In S104, the position and orientation conversion unit 113 converts the trajectory of the position and orientation of the sensor 12 input by the position and orientation estimation unit 112 from the sensor coordinate system to the moving body coordinate system. That is, the estimated change (trajectory) of the position and orientation of the sensor is converted to the position and orientation of the moving body based on the known positional relationship. Specifically, the position and orientation of the sensor 12 is rotated or translated based on the relative position and orientation of the sensor 12 and the moving body stored in a storage unit such as a RAM, so that each axis (X, Y, Z) of the coordinate system for the three position parameters and the three orientation parameters is aligned with the moving body coordinate system. Next, since the moving body coordinate system has three degrees of freedom, the six position and orientation parameters of the sensor 12 are reduced to three parameters. Specifically, data that cannot exist in the moving body coordinate system is deleted from the array data of the position information. For the position parameters, data on the Z axis, which is the vertical direction, is deleted. On the other hand, for the orientation parameters, data on Roll (X axis) and Pitch (Y axis) other than Yaw (Z axis), which is the rotation direction on the horizontal plane, is deleted. The multiple pieces of position information of the sensor 12 converted into the moving body coordinate system are output to the change amount acquisition unit 114. When moving a moving body in a straight line, the position of the moving body at a first time point when a predetermined motion is started and the position of the moving body at a second time point after the predetermined motion may be acquired based on the positional relationship between the sensor and the moving body. This has the effect of reducing the processing load because it can be estimated with simpler calculations than estimating the trajectory of the moving body. When the trajectory of the moving body is used, it has the effect of being able to estimate the amount of change in the attitude of the sensor with higher accuracy.

In S105, the change amount acquisition unit 114 acquires the amount of change between the relative position and orientation stored in the RAM and the actual physical relative position relationship between the moving body and the sensor, using multiple pieces of position information of the sensor 12 input by the position and orientation conversion unit 113. That is, the position and orientation conversion unit 113 acquires the result of converting the position (first position information) of the moving body or sensor after a predetermined operation from the measurement information measured by the sensor when the moving body performs a predetermined operation. Then, the predicted position (second position information) of the moving body or sensor when performing a predetermined operation in a preset positional relationship is compared with the position of the moving body or sensor after the predetermined operation. If the difference in the position of the moving body or sensor from the first position information and the second position information is within a predetermined threshold, it can be said that the relative positional relationship between the sensor and the moving body has not changed from the preset value. If the moving body moves straight as in this embodiment, the change amount of the orientation parameter in the relative position and orientation is obtained. The details of the process are explained below.

Figure 5 is a flowchart explaining the process executed by the change amount acquisition unit 114. In S201, the change amount acquisition unit 114 judges whether the trajectory represented by the position information input by the position and orientation conversion unit 113 is sufficient in quantity to estimate the change amount of the sensor's orientation. For example, the Euclidean distance between the position parameters (X, Y) of the latest position and orientation and the origin (0, 0) is calculated from the position information, and if the distance is equal to or greater than a predetermined threshold, it is determined that the number of position and orientation data is sufficient, and the process proceeds to S202. If not, the change amount acquisition process is terminated without executing the change amount acquisition. The more position information there is, the more accurately the change amount of the sensor's orientation can be estimated. Note that this step may be skipped if the change amount of the sensor's orientation is calculated only from the positions of the moving body at the first time point when the predetermined operation is started and the second time point when the predetermined operation is ended.

The following process will be explained with reference to FIG. 6. FIG. 6(a) is a diagram in which the position information of the sensor 12 in the moving body coordinate system is mapped onto a two-dimensional plane. The moving body 10 equipped with the sensor 12 moves linearly for a predetermined distance in the direction shown in the figure. D101 is the origin of the moving body coordinate system and is also the initial position of the sensor 12. The black circle in D102 is a plot of each position parameter included in the sensor position information. The trajectory is shown by plotting in the moving body coordinate system the positions that the moving body has passed through between a first point in time when the moving body starts a predetermined operation and a second point in time after it has moved by executing the predetermined operation.

In S202, the change amount acquisition unit 114 estimates first position information indicating the position of the moving body after a predetermined movement from the measurement information measured when the moving body performs a predetermined movement based on the positional relationship between the sensor and the moving body. The first position information may be the trajectory of the sensor, and an approximate straight line is acquired by inputting all position parameters included in the position information of the sensor 12. In FIG. 6, D103 is the approximate straight line. When the trajectory of the moving body is used, the change in the attitude of the sensor shown in D104 can be detected more accurately. In addition, in order to maintain the detection accuracy of the change in the attitude of the sensor, the points whose positions could not be accurately estimated or measured may not be included in the approximate straight line of D1003. For example, when the reliability of the acquired position information of the sensor or the moving body is lower than a predetermined value, it is suppressed so as not to be used in estimating the approximate straight line. In addition, when the change in attitude is detected from the position of the sensor at the first time when the predetermined movement is started and the position of the sensor after the predetermined movement, this step may be skipped. In this case, the amount of information to be handled is reduced, which has the effect of reducing the processing load.

In S203, the change amount acquisition unit 114 outputs the difference in the position of the sensor from the first position information and the second position information indicating the predicted position of the moving body (sensor) associated with the specified operation. That is, the change amount of the attitude of the sensor is acquired based on the position (trajectory) of the moving body (sensor) before and after the specified operation. Specifically, the angle D104 formed by the approximation line D103 and the Y axis in FIG. 6 is acquired. In the moving body coordinate system, since the Y axis direction is the moving direction of the moving body, when the moving body moves straight, the X coordinate of the position parameter should always be 0 (predicted position). Therefore, if the approximation line of the position information deviates from the Y axis, it can be seen that the coordinate system conversion of the position information by the position and attitude conversion unit 113 (sensor coordinate system → moving body coordinate system) is incorrectly converted. This incorrect conversion is caused by a change from the relative positional relationship between the sensor and the moving body stored in the RAM, that is, a change in the attitude parameter of the sensor. The change amount of the attitude parameter is the angle formed by the approximation line of the position information and the Y axis. The angle between these two lines is calculated using trigonometric functions. This completes the process of S105.

The amount of change in the sensor's attitude obtained in the above manner is output to the calibration necessity determination unit 115. In S106, the calibration necessity determination unit 115 obtains the amount of change from the change amount acquisition unit 114. If the obtained amount of change is equal to or greater than a predetermined threshold, it is determined that calibration of the relative position and attitude is necessary, and if not, calibration is not required. In S107, if the calibration necessity determination unit 115 determines that calibration is necessary, the process proceeds to S108, and if not, the process skips to S109. The determination result here is input to the output unit 117, and output information is presented to the user via a predetermined output device 14. For example, if it is determined that calibration is necessary, output information is output to inform the user that the mounting position and attitude of the sensor has shifted. The output device 14 and output information will be described later.

In S108, the relative position and orientation calibration unit 116 corrects the relative position and orientation based on the amount of change in the relative position and orientation acquired by the change amount acquisition unit 114 so that it matches the actual physical relative positional relationship between the moving body and the sensor. That is, the amount of change in the orientation of the sensor is estimated from the first position information, and the relative positional relationship between the sensor and the moving body is updated according to the amount of change. Specifically, the angle of the amount of change is subtracted from the parameter that is Yaw (Z axis) in the moving body coordinates among the orientation parameters of the relative position and orientation. The relative position and orientation after calibration is overwritten and stored in H13, which is a RAM.

In S109, it is determined whether or not to terminate the system. Specifically, if the amount of change acquired by the amount of change acquisition unit 114 is equal to or less than the threshold value, or if an instruction to terminate from the user is received via the input unit H15, the calibration of the relative position and orientation is deemed complete and the system is terminated. If not, the system returns to S101 and continues processing.

In the first embodiment, when a moving body moves straight, the position and orientation are measured based on an image acquired from a sensor, and the amount of change in the orientation parameters of the relative position and orientation between the sensor and the moving body is acquired based on position information consisting of the position and orientation at multiple points in time. Furthermore, the orientation parameters are corrected from the acquired amount of change. This makes it possible to correct the relative position and orientation between the sensor and the moving body without using a highly accurate amount of movement of the moving body, and makes it possible to easily correct changes in the relative positional relationship between the moving body and the position and orientation estimation device.

(Variation 1-1)
In the first embodiment, the position and orientation estimation unit 112 measures the position and orientation by the SLAM technique using a camera. However, the sensor combined with the SLAM technique is not limited to a camera, and any sensor may be used. The position and orientation may be measured by combining the SLAM technique with an active distance sensor such as LiDAR, Time of Flight, or pattern light projection.

The position and orientation estimation unit 112 is not limited to SLAM technology, and may use any technology that can measure position and orientation. For example, it may use a method based on measuring radio wave intensity in wireless communication such as Wi-Fi. It may also use a method that uses a magnetic tracking sensor or a time difference in detection by multiple optical sensors. It may also use a method that uses images acquired by a camera installed in a specific location, such as a surveillance camera, and performs model fitting based on CAD data.

(Variation 1-2)
In the first embodiment, the position and orientation conversion unit 113 converts the position and orientation of the sensor 12 from the sensor coordinate system to the moving body coordinate system and outputs the converted position and orientation to the change amount acquisition unit 114. However, as long as the coordinate system can be unified to a single coordinate system when the change amount acquisition unit 114 acquires the change amount of the relative position and orientation, the coordinate system may be unified to any coordinate system. For example, when unifying to the sensor coordinate system, the following processing may be performed. The position and orientation conversion unit 113 converts the position and orientation of the moving body estimated based on the movement control information acquired from the movement control unit 111 from the moving body coordinate system to the sensor coordinate system, and outputs the converted position and orientation to the change amount acquisition unit 114. On the other hand, the position and orientation of the sensor obtained from the position and orientation estimation unit 112 may be output to the change amount acquisition unit 114 as it is in the sensor coordinate system, and the change amount acquisition unit 114 may acquire the change amount based on both the position and orientation of the sensor and the moving body in the sensor coordinate system.

(Variation 1-3)
In the first embodiment, the information processing system 10 is not limited to an unmanned guided vehicle (mobile body). For example, the information processing system 10 may be an automatic driving vehicle or an autonomous mobile robot, and the movement control described in the present embodiment may be applied to them. It may also be applied to an unmanned aerial vehicle 10' as shown in FIG. 8(b). Specifically, it is a drone. In the case of an aerial vehicle, for example, the inclination of the sensor 12 can be detected by moving up and down a predetermined distance in the direction of gravity (Z axis). Alternatively, it may be applied to a robot arm equipped with a sensor on the arm.

(Modification 1-4)
The output unit 117 outputs output information indicating whether the positional relationship after the predetermined operation is the maintained positional relationship based on the trajectory (first position information) of the moving body estimated after the predetermined operation and the predicted trajectory (second position information) of the moving body after the predetermined operation. For example, when the change amount of the sensor or the position and orientation of the moving body acquired by the change amount acquisition unit 114 is greater than a predetermined threshold, output information indicating whether the positional relationship between the sensor and the moving body is the maintained positional relationship is output to the output device. For example, if three-color lights of red, yellow, and green are used for the presentation unit, red is presented when the change amount of the relative position and orientation is equal to or greater than a predetermined threshold, and green is presented when it is less than the threshold. This makes it possible to visualize the change amount acquisition result to the user. The threshold may be set in stages, and the notification method may be changed depending on which threshold range the change amount falls within. For example, a first threshold at a stage where calibration is definitely required and a second threshold for notifying that the position estimation has not yet been affected but that the initial position and orientation has changed are set in advance. For example, when the acquired change in the position and orientation of the sensor is equal to or greater than a first threshold, the light turns red, when the change is less than the first threshold and equal to or greater than a second threshold, the light turns yellow, and when the change is equal to or less than the second threshold, the light turns green. That is, when the difference in the position of the moving body indicated by the first measurement information and the second measurement information is greater than the first threshold, the output information is output to the first output device. When the difference in the position of the moving body indicated by the first measurement information and the second measurement information is greater than a second threshold that is smaller than the first threshold, the output information is output to the second output device. Also, as long as the information on the change amount acquisition result can be presented to the user, an output device such as an LED lamp or a liquid crystal display may be used. The output device may be an audio output device such as a speaker, and may be configured to play specific audio information (such as an alarm sound or a melody) according to the map update status. Also, the output device may be a projection device. For example, when it is detected that the relative positional relationship between the sensor and the moving body is inappropriate, the correct mounting position and orientation of the sensor may be projected by a projection device or a display device. Specifically, by projecting the appropriate position and orientation of the sensor using a projector, the attachment position and orientation of the sensor can be presented to the user in an easy-to-understand manner.

(Modification 1-5)
It may be set so that the moving body automatically performs a predetermined operation at a predetermined timing. For example, when the information processing system 1 is started or ended, the moving body may be made to move straight ahead a predetermined distance and immediately return as a predetermined operation. Since a change in the positional relationship between the moving body and the sensor can be detected before the moving body starts autonomous driving using the sensor, the moving body can be controlled more safely. If the attitude of the sensor has changed, the sensor will pass through the trajectory of D103' twice as shown in FIG. 6B. If the attitude of the sensor has not changed, the trajectory will move forward a predetermined distance on the Y axis and return. Since the same trajectory is moved twice, the amount of measurement information increases, and the amount of change in the attitude of the sensor can be obtained more accurately. In addition, when the moving body is started or ended, the moving body may be set to go to a predetermined location (for example, a charging spot or a waiting space). In such a case, the amount of change in the attitude of the sensor may be obtained at the timing when the moving body approaches the predetermined location (the timing when the target point and the position of the moving body are within a predetermined range). By checking the relative positional relationship between the sensor and the moving body at a predetermined timing in this way, the position of the moving body can be measured more accurately and more stably. In addition, by setting a trigger at a timing that allows the user to immediately check the detection results, the above processing can be executed at any timing, making it easy to detect whether the relative positional relationship between the sensor and the moving object is inappropriate.

<Embodiment 2>
In the first embodiment, when the movement trajectory of the moving body becomes a straight line, the amount of change in the relative attitude parameter between the sensor and the moving body is acquired based on the position information of the sensor, and the relative position and attitude between the moving body and the sensor is corrected based on the acquired result. According to this embodiment, the relative position parameter between the sensor and the moving body can also be corrected. Therefore, in the second embodiment, the amount of change in the relative position parameter between the sensor and the moving body is acquired based on the position information of the sensor when the moving body turns, and the relative position and attitude is corrected using the acquired result.

The configuration diagram in this embodiment is the same as FIG. 2 which explains the configuration of the information processing device 11 described in embodiment 1, and therefore the explanation will be omitted. The hardware configuration may also be as shown in FIG. 3, as in embodiment 1. The diagram of the processing procedure in this embodiment is the same as FIG. 4 which explains the processing procedure of the information processing device 11 described in embodiment 1, and therefore the explanation will be omitted. Only the processing procedures which differ from embodiment 1 will be explained, and explanations of the other procedures will be omitted as they are the same as the processing in FIG. 4.

In S102, the movement control unit 111 controls the movement of the moving body by outputting movement control information to the control device 13 for rotating the moving body by a predetermined amount of rotation. Note that in this embodiment, the movement of the moving body is controlled so that the position parameters do not move from the origin, which is the initial position, in the moving body coordinate system, and only the attitude parameters change in a rotational motion.

In S105, the change amount acquisition unit 114 acquires the amount of change between the relative position and orientation stored in the RAM and the actual physical relative position relationship between the moving body and the sensor, using the position information of the sensor 12 input by the position and orientation conversion unit 113. In the case where the moving body is rotating in such a way that only the orientation changes, as in this embodiment, the amount of change in the position parameter of the relative position and orientation is obtained.

Figure 7 is a flowchart showing the detailed flow of the change amount acquisition process. In S301, the change amount acquisition unit 114 determines whether the number of positions and orientations in the position information input by the position and orientation conversion unit 113 is sufficient for acquiring the amount of change. Since the position and orientation estimation unit 112 measures the position and orientation at regular time intervals, the number of pieces of position and orientation data are proportional to the movement time. If the movement time is equal to or longer than the regular time, it is determined that the number of pieces of data is sufficient, and the process proceeds to S302. If not, the change amount acquisition process ends without executing the change amount acquisition.

The following process will be explained with reference to FIG. 8(a). FIG. 8(a) is a diagram in which the position information of the sensor 12 in the moving body coordinate system is mapped onto a two-dimensional plane. D201 is the origin of the moving body coordinate system, and is also the initial position of the sensor 12. The black circles in D202 are plots of each position parameter included in the position information. The moving body 10 performs an operation of turning in the traveling direction shown in the figure.

In S302, the change amount acquisition unit 114 inputs all position parameters included in the position information of the sensor 12 and finds an approximate circle. In this embodiment, since a rotational movement is performed in which only the orientation parameters change, the actual position parameters do not change from the origin. However, if there is a change in the position parameters of the relative position and orientation, the position information of the sensor 12 will have a trajectory close to a circle, and so this approximate circle is found. In Figure 6, D203 is the approximate circle.

In S303, the change amount acquisition unit 114 acquires the difference between the center coordinate of the approximation circle and the initial position coordinate. Since the initial position coordinate is the origin, the difference is the center coordinate of the approximation circle of the position information, which becomes the change amount. In FIG. 8, the center coordinate D204 of the approximation circle of the position information has changed from the origin in the X-axis direction, and therefore it can be seen that there is a change in the X-coordinate of the position parameter. The change amount acquired in this manner is output to the calibration necessity determination unit 115.

In S108, the relative position and orientation calibration unit 16 corrects the relative position and orientation based on the amount of change in the relative position and orientation acquired by the change amount acquisition unit 114, as in the first embodiment, so that it matches the actual physical relative positional relationship between the moving body and the sensor. Specifically, among the position parameters of the relative position and orientation, the amount of change is subtracted from the parameters in the X-axis direction and the Y-axis direction in the moving body coordinates. In the example of FIG. 8, since there is a change in the X-axis direction, the amount of change is subtracted from the X coordinate. The relative position and orientation after calibration is overwritten and stored in H13, which is a RAM.

In the second embodiment, when a moving body undergoes a rotational motion that changes only its orientation, the amount of change in the relative position parameters between the moving body and the sensor is obtained from the position information of the sensor. Furthermore, the relative position and orientation are corrected from the obtained amount of change. This makes it possible to correct the relative position and orientation between the sensor and the moving body without using a highly accurate amount of movement of the moving body, and makes it possible to easily correct changes in the relative positional relationship between the moving body and the position and orientation estimation device.

(Variation 2-1)
In the first and second embodiments, the movement control unit 111 controls the movement of the moving body so that the moving body moves straight (first embodiment) or turns (second embodiment), but this is not limited thereto. The moving body may be controlled in any manner as long as the moving trajectory is composed of straight movement, turning, or both. For example, the moving trajectory may be a U-turn, an L-shaped movement, or a rectangular movement trajectory. In the case of moving control in which straight movement and turning are mixed like this, straight line parts and non-straight line parts, for example, corners or turning points in the path, are separated from the trajectory of the position information of the sensor 12. The change amount acquisition process described in the first and second embodiments is applied to each separated trajectory to acquire the change amount of each parameter of the position and orientation of the relative position and orientation.

(Variation 2-2)
In the first and second embodiments, the movement control unit 111 moves the moving body itself by outputting movement control information to the control device 13. However, the moving body itself does not need to move as long as it can move along a predetermined movement trajectory. For example, the moving body may be placed on a device such as a moving stage or a rotating plate and moved or rotated. In this case, since the measurement information can be obtained by the sensor without controlling the moving body itself, the change in the positional relationship between the sensor and the moving body can be detected with higher accuracy.

<Embodiment 3>
In the first and second embodiments, when the moving body is controlled to move in a prescribed manner such as going straight or turning, the amount of change in the relative position and orientation between the sensor and the moving body is acquired based on the position and orientation of the sensor, and the relative position and orientation are corrected based on the amount of change. In the third embodiment, a method for automatically calibrating the relative position and orientation between the sensor and the moving body while the moving body automatically travels along a predetermined route will be described. In this embodiment, in particular, a case will be described in which a section that forms a prescribed movement is set in advance together with route information. Note that, in this embodiment, the prescribed movement section is a section that forms a prescribed movement such as going straight or turning.

An example of the system configuration in this embodiment is similar to that in embodiment 1, and is composed of a mobile body 10 having an information processing device and a sensor 12 as shown in FIG. 1. The hardware configuration in this embodiment is similar to that in embodiment 1, and is shown in FIG. 3. The functions of each piece of hardware are similar to those in embodiment 1, so their explanation will be omitted. An example of the functional configuration in this embodiment is shown in FIG. 11. It differs from FIG. 2 in embodiment 1 in that it has a movement path storage unit 216 and a prescribed movement determination unit 217. Here, only the parts that differ from embodiment 1 will be explained, and explanations of the similar parts will be omitted.

The movement path storage unit 216 stores route information necessary for a moving body to travel a predetermined route, and specified movement section information. This information includes the final point and intermediate points of the moving body's travel path, and the start and end points of the specified movement section, and is array data of three parameters, the position and orientation of the moving body.

The movement control unit 211 generates movement control information using the route information stored in the movement route storage unit as input, and outputs the information to the control device 13. Alternatively, the movement control unit 211 acquires movement control information generated by an external device or the like, or movement control information that has been set in advance, and controls the moving object based on the acquired movement control information.

The prescribed movement determination unit 217 receives the position and orientation of the sensor 12 input by the position and orientation conversion unit 113 and the prescribed movement section information stored in the movement path storage unit, and determines whether the current position of the moving body is inside or outside the prescribed movement section. The determination result is output to the change amount acquisition unit 114.

Figure 9 is a diagram showing a flowchart of the processing procedure in this embodiment. Explanations of the same processes as those in Figure 4, which explains the processing procedure of the information processing device 11 described in embodiment 1, will be omitted, and only steps S302, S305, S306, and S309 that are different from embodiment 1 will be explained.

In S302, the movement control unit 111 acquires movement control information based on the route information stored in the movement route storage unit 216 so as to travel along a specified route, and outputs the information to the control device 13, thereby controlling the movement of the moving body.

In S305, the prescribed movement determination unit 217 determines whether the moving object is within the prescribed movement section. Based on the position and orientation of the sensor 12 input by the position and orientation conversion unit 113 and the prescribed movement section information stored by the movement path storage unit, a determination is made as to whether the current position of the moving object is within or outside the prescribed movement section. When the moving object is within the prescribed movement section, the type information of the prescribed movement is also output as the determination result. The type information of the prescribed movement is, for example, a movement pattern including a straight line as described in the first embodiment and a turn as described in the second embodiment. Furthermore, even if the position of the moving object is within the prescribed movement section, if the travel time within the prescribed movement section is short, the number of position and orientation data in the position information of the sensor 12 may not be sufficient to obtain the amount of change. Therefore, if the moving object has not traveled within the prescribed movement section for a predetermined time or more, the determination result is output as false even if the moving object is within the prescribed movement section.

In S306, if the result of the determination by the prescribed movement determination unit 217 is true, the process proceeds to S307. If not, the process skips to S310. In S307, the change amount acquisition unit 114 uses the position information of the sensor 12 input by the position and orientation conversion unit 113 to acquire the amount of change between the relative position and orientation stored in the RAM and the physical relative positional relationship between the actual moving body and the sensor. At this time, the change amount acquisition method is determined based on the type information of the prescribed movement input by the prescribed movement determination unit. For example, if the type of prescribed movement is straight ahead, the method described in embodiment 1 is used, and if it is turning, the method described in embodiment 2 is used.

In S309, the relative position and orientation calibration unit 216 corrects the relative position and orientation based on the amount of change in the relative position and orientation acquired by the change amount acquisition unit 114 so that it matches the physical relative positional relationship between the actual moving body and the sensor. In this embodiment, the correction is made in accordance with the parameters acquired by the change amount acquisition unit 114. Specifically, if the type of prescribed movement is straight ahead, the orientation parameters are corrected, and if it is turning, the position parameters are corrected.

In the third embodiment, when a section of a specified movement is set in advance together with route information in the autonomous driving of a moving body, it is determined whether the moving body is inside the specified movement section. Based on the result of the determination, acquisition of the amount of change in the relative position and orientation of the sensor and the moving body and implementation of correction are dynamically switched. This makes it possible to correct the relative position and orientation while the moving body is autonomously driving, and makes it easy to correct changes in the relative position relationship between the moving body and the position and orientation estimation device.

(Variation 3-1)
In the third embodiment, the moving path storage unit stores route information and prescribed moving section information required for the moving body to travel along a predetermined path. Furthermore, the prescribed moving judgment unit performs a determination as to whether the position of the moving body is inside or outside the prescribed moving section based on the prescribed moving section information. However, this is not limited to these, and any method may be used as long as the prescribed moving judgment unit can determine whether the position of the moving body is inside or outside the prescribed moving section based on the information stored in the moving path storage unit. For example, the moving path storage unit may store a history of the moving trajectory of the moving body, and the prescribed moving judgment unit may extract a section that is a prescribed movement such as going straight or turning by using least squares fitting to the history of the moving trajectory.

Furthermore, the prescribed movement determination unit may make a determination using information other than the information stored in the movement path storage unit. For example, a sensor capable of observing a moving object from a third-party perspective, such as a surveillance camera, may be used to obtain the movement trajectory of the moving object, and based on the movement trajectory, the inside or outside of the prescribed movement section may be determined, and the type of prescribed movement may be determined.

The output unit 117 may output output information related to the calibration of the relative position and orientation, such as a specified movement section, the amount of change in the relative position and orientation, and calibration parameters. FIG. 10 shows an example of a GUI 100 that uses a display as the output device 14 and presents route information and specified movement section information of a moving body, the results of the amount of change in the relative position and orientation, and calibration parameters.

G110 is a GUI that presents information about the moving object's route and the specified travel section. G111 is a moving object that is traveling automatically, G112 is the start point of the travel route, and G113 is the end point. The arrow line connecting the start point G112 and the end point G113 is the travel route, and in particular the solid arrow indicates the specified travel section.

G120 is a GUI for selecting the type of movement control for a moving object. Specifically, in the movement control for turning a moving object, the type is selected as either one in which only the attitude parameters change as shown in the second embodiment, or one in which both the position parameters and the attitude parameters change as in an automobile.

G130 is a GUI that presents the results of the change in the relative position and orientation. G131 displays the change in the acquired relative position and orientation numerically, dividing it into position and orientation. G132 displays each axis (solid arrow) of the coordinate system to which the acquired change has been added to the current moving body coordinate system (dotted arrow). This makes it possible to visually understand the change in the relative position and orientation. The user can decide whether or not to correct the sensor mounting position and orientation. Note that Figure 10 shows an example in which only the orientation (angle) has changed.

G140 is a GUI that allows the user to adjust the calibration of the relative position and orientation. In G141, automatic calibration while driving can be enabled by clicking a checkbox. In G142, the threshold adjustment bar can be moved left and right to determine the threshold to be used when determining whether calibration is necessary.

As described above, a GUI that outputs the change amount acquisition results and displays the input of parameters during calibration can encourage the user to understand intuitively, further improving convenience when correcting the relative position and orientation.

<Embodiment 4>
In the first and second embodiments, when the moving body is controlled to move in a specified direction such as going straight or turning, the amount of change in the relative position and attitude of the sensor and the moving body is acquired based on the position and attitude of the sensor, and the relative position and attitude is corrected based on the amount of change. In both the first and second embodiments, the moving body is described as an automatic guided vehicle, and is assumed to move on the ground, i.e., on a two-dimensional plane, so the position and attitude of the moving body has three degrees of freedom (two position parameters and one attitude parameter).

In this embodiment, we will explain the case where the method is applied to a moving body such as a drone that can move and rotate in both the vertical direction (perpendicular to the floor in real space). Therefore, in embodiment 4, we will explain a method for correcting the relative position and orientation when the movement trajectory of a drone is a straight line. In the case of such a moving body, the position and orientation of the moving body have six degrees of freedom (three position parameters and three orientation parameters).

The system configuration example in this embodiment is the same as that in the first embodiment, and is composed of a mobile body 10 having an information processing device and a sensor 12 as shown in FIG. 1. However, the difference is that the moving direction of the mobile body 10 can move not only horizontally but also vertically. In addition, the posture parameter of the mobile body is one parameter (Yaw) in the first embodiment, whereas in this embodiment, there are three parameters (Roll, Pitch, Yaw), and rotation around the three axes of XYZ is possible. Here, the Z axis is a coordinate axis perpendicular to the floor of the real space, and Roll and Pitch are defined as posture parameters that represent rotation around the X axis and Y axis, respectively. The hardware configuration in this embodiment is the same as that in the first embodiment, and is shown in FIG. 3. The functions of each hardware are the same as those in the first embodiment, so the description will be omitted. A diagram of the processing procedure in this embodiment is shown in FIG. 12. Only the processing procedures different from the first embodiment will be described, and the other procedures will be omitted as they are the same as those in FIG. 4.

Since the drone can move three-dimensionally, in this embodiment, it moves on the coordinate axes in the order of X-axis, Y-axis, and Z-axis. The order of the coordinate axes in which it moves can be any order. The flowchart in Figure 12 shows the processing procedure for each axis.

In S401, the movement control unit 111 determines the route and stopping points for the drone to move and generates control information. When a drone moves forward, backward, left, or right, it is prone to wobbling and has difficulty moving in a straight line. In this embodiment, it is necessary to make it possible to consider the movement trajectory of the moving body as a straight line. A stopping point is provided along the movement path where the drone can hover or land to stabilize, and the movement trajectory is pseudo-straightened by using the position and orientation of the sensor at that point. Linearization refers to making it easier to find an approximate straight line for the movement trajectory by using a method such as eliminating position measurement data that deviates from a straight line or averaging a group of position measurement data that includes variation.

In S102, the movement control unit 111 inputs the control information generated in S401 and moves the drone. At that time, the drone is temporarily stopped at the stopping point.

In S103, the position and orientation estimation unit 112 estimates the position and orientation of the sensor near the stopping point, and outputs the average value as the estimation result to the position and orientation conversion unit 113.

In S104, the position and orientation conversion unit 113 converts the trajectory of the position and orientation of the sensor 12 input by the position and orientation estimation unit 112 from the sensor coordinate system to the moving body coordinate system. In the first embodiment, the moving body coordinate system had three degrees of freedom, so the six position and orientation parameters were reduced to three parameters, but in this embodiment, the moving body has six degrees of freedom, so there is no need to reduce the parameters.

In S105, the change amount acquisition unit 114 acquires the change amount between the relative position and orientation stored in the RAM and the physical relative position relationship between the actual moving body and the sensor, using the position information at the start point and the stop point of the sensor 12 input by the position and orientation conversion unit 113. Since the moving body is moved in a straight line, the change amount of the orientation parameter in the relative position and orientation can be obtained, but the rotation component of the change amount obtained differs depending on the moving direction. Specifically, the rotation component of Pitch can be obtained from the position information when moving on the X axis, the Yaw can be obtained from the position information when moving on the Y axis, and the Roll can be obtained from the position information when moving on the Z axis. Note that the same method as in the first embodiment can be used to calculate the change amount regardless of the moving direction. In the case of Z-axis movement, the moving direction in the first embodiment is replaced by the Z axis, the coordinate axis perpendicular to it is replaced by the Y axis, and the comparison axis when calculating the change amount is replaced by the Z axis, so the change amount of the orientation parameter Roll is the angle between the approximate straight line of the sensor's position information and the Z axis. On the other hand, in the case of movement along the X-axis, the direction of travel in Example 1 is replaced with the X-axis, the coordinate axis perpendicular to it is replaced with the Z-axis, and the axis to be compared when calculating the amount of change is replaced with the X-axis, so the amount of change in the attitude parameter Pitch is the angle between the approximation line of the sensor's position information and the X-axis.

In S108, the relative position and orientation calibration unit 116 corrects the relative position and orientation so that it matches the actual physical relative positional relationship between the moving body and the sensor, based on the amount of change in the relative position and orientation acquired by the change amount acquisition unit 114. The relative position and orientation are corrected by subtracting the amount of change in the three orientation parameters calculated in S105 from each component of the relative position and orientation.

In the fourth embodiment, for a moving body that can move in any direction in three-dimensional space, it is possible to correct the relative position and orientation between the sensor and the moving body without using a highly accurate amount of movement of the moving body, and it is possible to easily correct changes in the relative positional relationship between the moving body and the position and orientation estimation device.

(Modification 4-1)
In the fourth embodiment, the movement control unit 111 moves the drone along the coordinate axes of the X-axis, Y-axis, and Z-axis. However, the movement direction is not limited to these, and may be any direction that is not along the coordinate axes, as long as the change amount acquisition unit 114 can compare the predicted position of the sensor with the position of the moving body. The change amount acquisition unit determines a two-dimensional plane to be used as a reference, and then compares the predicted position of the sensor with the position of the moving body. The two-dimensional plane to be used as a reference is determined by the rotation component of the change amount to be acquired. Specifically, the rotation component to be acquired is the XY plane in the case of Yaw, the YZ plane in the case of Roll, and the ZX plane in the case of Pitch. For example, when it is desired to acquire the change amount of the rotation component of Roll, the movement control unit 111 moves the moving body straight on the YZ plane and compares the predicted position of the sensor with the position of the moving body. Furthermore, even if the moving body is moved in any direction in the XYZ space that is not on the YZ plane, the predicted position of the sensor and the position of the moving body may be projected onto the YZ plane, respectively, and the change amount of Pitch may be acquired by comparing the position information after the plane projection.

(Modification 4-2)
In the fourth embodiment, the position and orientation estimation unit 112 estimates the position and orientation of the sensor near the stop point and outputs the average value as the estimation result. However, this is not limited to this, and any position and orientation group of the estimation result may be linearized. For example, since a drone is likely to wobble immediately after starting to move, the position and orientation may be estimated at a certain time interval after a certain time has elapsed since the start of movement. Alternatively, after moving a certain distance from the start of movement, the position and orientation may be estimated at a certain distance interval and output to the relative position and orientation change amount acquisition unit 114. Furthermore, regardless of the distance or time from the start of movement, a position and orientation that deviates from a straight line may be excluded.

(Modification 4-3)
In the fourth embodiment, the movement control unit 111 corrects the three attitude parameters of the relative position and attitude by moving in a straight line in each direction of the X-axis, Y-axis, and Z-axis. However, the three position parameters of the relative position and attitude may be corrected by rotating the drone in turn around each coordinate axis instead of moving in a straight line. Here, the rotation may be a method of rotating the drone in the air by controlling its movement, or a method of fixing the drone to a jig so that the drone rotates around a designed rotation axis and rotating the drone. As in the case of moving in a straight line, the position parameters that can be corrected differ depending on the rotation axis. Specifically, the Y and Z coordinates can be corrected by X-axis rotation (Roll), the Z and X coordinates can be corrected by Y-axis rotation (Pitch), and the X and Y coordinates can be corrected by Z-axis rotation (Yaw). Note that the same method as in the second embodiment can be used for the method of acquiring the change amount during rotation and the calibration method. For example, in the case of X-axis rotation, the two-dimensional plane that maps the position information in the second embodiment is replaced with a YZ plane, and the change amount of the position parameter Y and Z coordinates is the difference between the center coordinates of the approximation circle of the sensor's position information and the initial position coordinates (origin).

(Modification 4-4)
In the fourth embodiment, since it is difficult for a drone to move stably in a straight line, the position and orientation of the sensor near the stopping point are used to acquire the amount of change, but this is not limited to drones. For example, the present invention can be applied to moving objects that move on a two-dimensional plane, such as automated guided vehicles, self-driving cars, and autonomous mobile robots.

(Modification 4-5)
In the modified example 4-2, the position and orientation estimation unit 112 removes positions and orientations that deviate from a straight line in order to linearize the group of estimated sensor positions and orientations, but this is not limited to drones. For example, it can also be applied to moving bodies that move on a two-dimensional plane, such as automated guided vehicles, self-driving cars, and autonomous mobile robots.

Of the above-mentioned processing units, the position and orientation measurement unit 112 and the movement control unit 111 may instead use a trained model that has been machine-learned. In that case, for example, multiple combinations of input data and output data to the processing unit are prepared as training data, knowledge is acquired from them by machine learning, and a trained model is generated that outputs output data for the input data based on the acquired knowledge. The trained model can be configured, for example, as a neural network model. Then, the trained model performs the processing of the processing unit by operating in cooperation with a CPU or GPU as a program for performing processing equivalent to that of the processing unit. The trained model may be updated after a certain amount of processing as necessary.

The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a data communication network or various storage media. The computer (or CPU, MPU, etc.) of the system or device then reads and executes the program. The program may also be provided by recording it on a computer-readable recording medium.

10 Information processing system 11 Information processing device 12 Sensor 13 Control device

Claims (18)

an estimation means for estimating first position information indicating a position of a moving object mounted with a sensor based on input information input from the moving object when the moving object moves in a predetermined manner;
an acquisition means for acquiring second position information indicating a position of the moving object that is expected to be obtained based on input information input from the sensor when the moving object held by the holding means makes the predetermined movement while maintaining the position and orientation of the sensor with respect to the moving object;
an output means for outputting information regarding a change in a position and orientation of the sensor relative to the moving body when the moving body makes the predetermined movement and a change in a position and orientation of the sensor relative to the moving body held by the holding means, based on the first position information and the second position information;
13. An information processing device comprising: 2 . The information processing apparatus according to claim 1 , wherein the output means outputs output information indicating a difference in position of the moving object based on the first position information and the second position information. The information processing device according to claim 2, characterized in that the output means outputs the output information indicating that the position and orientation of the sensor with respect to the moving body is the held position and orientation when a difference in position of the moving body based on the first position information and the second position information is less than a predetermined threshold value. The information processing device according to claim 2, characterized in that the output means outputs the output information indicating that the position and orientation of the sensor relative to the moving body is not the retained position and orientation when a difference in position of the moving body based on the first position information and the second position information is equal to or greater than a predetermined threshold value. The information processing device according to claim 2, characterized in that the output means outputs to a display device the output information indicating that the position and orientation of the sensor relative to the moving body is not the retained position and orientation when a difference in position of the moving body based on the first position information and the second position information is equal to or greater than a predetermined threshold value. The information processing device according to claim 2, characterized in that the output means outputs the output information for displaying the position of the moving body indicated by the first or second position information on a display device when a difference in the position of the moving body based on the first position information and the second position information is equal to or greater than a predetermined threshold value. The information processing device according to claim 2, characterized in that the output means outputs the output information indicating that the position and orientation of the sensor relative to the moving body is not the retained position and orientation to an audio output device when a difference in position of the moving body based on the first position information and the second position information is equal to or greater than a predetermined threshold. 3. The information processing device according to claim 2, wherein the output means outputs the output information indicating the position and orientation of the sensor that satisfies the held position and orientation to a projection device or a display device when a difference in position of the moving body based on the first position information and the second position information is greater than a predetermined value. 3. The information processing device according to claim 2, wherein the output means outputs different output information depending on the magnitude of the difference between the positions of the moving body based on the first position information and the second position information based on a plurality of threshold values. The information processing device according to claim 8, characterized in that the output means outputs the output information to a first output device when a difference in the position of the moving body based on the first position information and the second position information is greater than a first threshold, and outputs the output information to a second output device when a difference in the position of the moving body based on the first position information and the second position information is greater than a second threshold that is smaller than the first threshold. The predetermined movement is a movement of the moving body moving in a straight line for a predetermined distance,
9. The information processing device according to claim 8, characterized in that the output means outputs the output information indicating a change in attitude of the sensor based on a difference in position of the moving body based on the first position information and the second position information. The predetermined movement is a movement in which the moving body turns by a predetermined angle,
9. The information processing device according to claim 8, characterized in that the output means outputs the output information indicating a change in the position of the sensor based on a difference between the position of the moving body based on the first position information and the second position information. 9. The information processing device according to claim 8, further comprising an update means for updating the position and orientation of the sensor relative to the moving body based on the output information when a difference in the position of the moving body based on the first position information and the second position information is greater than a predetermined threshold value. a measuring means for measuring a position of the sensor from the input information based on map information obtained by measuring a moving path of the moving object in advance;
14. The information processing device according to claim 1, wherein the estimation means estimates the position of the moving body by converting the position or orientation of the sensor measured by the measurement means based on the held position and orientation of the sensor with respect to the moving body. The information processing device according to any one of claims 1 to 14, characterized in that the sensor is an imaging device that captures an image of an object present in the environment. The information processing device according to any one of claims 1 to 14, characterized in that the sensor is a distance sensor that measures the distance between an object present in the environment and the sensor. A program for causing a computer to function as each of the means possessed by an information processing device according to any one of claims 1 to 15. an estimation step of estimating first position information indicating a position of a moving object equipped with a sensor based on input information input from the moving object when the moving object moves in a predetermined manner;
an acquisition step of acquiring second position information indicating a position of the moving object that is expected to be obtained based on input information input from the sensor when the moving object moves in the predetermined manner while maintaining the position and orientation of the sensor relative to the moving object held by a holding means;
and an output step of outputting information regarding changes in the position and orientation of the sensor relative to the moving body when the moving body performs the predetermined movement, and the position and orientation of the sensor relative to the moving body held by the holding means, based on the first position information and the second position information.

Images