JP7830385B2 - Information processing program, information processing device, and information processing method - Google Patents

Description

Embodiments of the present invention relate to an information processing program, an information processing device, and an information processing method.

A technique has been proposed to estimate the depth information of each pixel from image data from a single viewpoint by learning the relationship between multiple training datasets, each consisting of image data captured from a single viewpoint and depth information for each pixel. For training, it is necessary to use training data that has been calibrated so that the temporal and spatial correspondence between image data captured by a camera sensor and point cloud data with depth information defined for each pixel measured by a distance measuring sensor is accurate. Therefore, a technique for calibrating the training data using calibration parameters has been disclosed.

However, conventional techniques used calibration parameters to correct and use training data that contained temporal and spatial errors that could not be fully corrected even with calibration parameters, which sometimes resulted in a decrease in training accuracy. In other words, conventional techniques struggled to support the provision of training data suitable for high-accuracy training.

Special Publication No. 2022-543017

Park, Chanoh, et al. "Spatiotemporal camera-LiDAR calibration: A targetless and structureless approach." IEEE Robotics and Automation Letters 5.2 (2020): 1556-1563.

The problem that this invention aims to solve is to provide an information processing program, an information processing device, and an information processing method capable of providing training data suitable for high-precision learning.

The information processing program of this embodiment is an information processing program that causes a computer to execute the following steps: an image acquisition step of acquiring image data; a point cloud acquisition step of acquiring point cloud data in which depth information is defined for each of a plurality of points, where the calibration parameters used to calibrate at least one of the temporal and spatial errors of the image data are unknown and the temporal and spatial errors are uncalibrated; a calibration parameter calculation step of calculating the calibration parameters for each training data including the image data and the point cloud data; and a determination step of determining the effectiveness of the training data used to calculate the calibration parameters based on the calibration parameters.

A schematic diagram of an information processing system. Schematic diagram of the waveform before applying the time offset. Schematic diagram of the waveform after applying a time offset. An illustrative diagram of reprojection error. An illustrative diagram of the position trajectory before applying calibration parameters. An illustrative diagram of the positional trajectory after applying calibration parameters. A flowchart illustrating the process for determining the effectiveness of training data. A flowchart illustrating the learning process. Hardware configuration diagram.

The information processing program, information processing device, and information processing method will be described in detail below with reference to the attached drawings.

Figure 1 is a schematic diagram of an example of the information processing system 1 of this embodiment.

The information processing system 1 comprises an information processing device 10, a UI (user interface) unit 12, a shooting sensor 14, and a distance measuring sensor 16. The information processing device 10, the UI unit 12, the shooting sensor 14, and the distance measuring sensor 16 are communicated together via a bus 18 or the like.

The UI unit 12 has a display function and an input function. The display function displays various types of information. Examples of the display function include a display, projection device, etc. The input function accepts user input. Examples of the input function include pointing devices such as a mouse and touchpad, a keyboard, etc. A touch panel integrating the display and input functions may also be used.

The imaging sensor 14 is a fixed-focus camera that obtains image data by capturing images from a single viewpoint. The imaging sensor 14 sequentially outputs the captured image data to the information processing device 10 in a time-series manner.

The distance measuring sensor 16 obtains point cloud data in which depth information is defined for each of the multiple points within the distance measuring range in real space. The distance measuring sensor 16 is, for example, a laser sensor, radar sensor, millimeter-wave sensor, etc. The distance measuring sensor 16 sequentially outputs the point cloud data obtained by distance measurement over time to the information processing device 10.

The shooting range of the image sensor 14 and the distance measurement range of the distance measurement sensor 16 are pre-adjusted so that at least a portion of them overlap. Preferably, the shooting range of the image sensor 14 and the distance measurement range of the distance measurement sensor 16 match.

The information processing device 10 performs processes such as determining the validity of training data, which includes image data and point cloud data, and training the learning model.

The information processing device 10 comprises a storage unit 20, a communication unit 22, and a processing unit 24. The storage unit 20, communication unit 22, UI unit 12, imaging sensor 14, and distance measuring sensor 16 are communicated with the processing unit 24.

The memory unit 20 stores various types of data. The communication unit 22 is a communication interface for communicating with external information processing devices, etc., via a network.

The memory unit 20, UI unit 12, image sensor 14, and distance measuring sensor 16 may be configured to communicate with the processing unit 24 via wired or wireless connections. For example, at least one of the memory unit 20, UI unit 12, image sensor 14, and distance measuring sensor 16 may be connected to the processing unit 24 via a network or the like.

Furthermore, the storage unit 20 may be located outside the information processing device 10. Alternatively, the storage unit 20 and at least one of the one or more functional units included in the processing unit 24 (described later) may be mounted on an external information processing device that is communicated to the processing unit 24 via a network or the like.

The processing unit 24 performs information processing in the information processing device 10. The processing unit 24 includes an image acquisition unit 24A, a point cloud acquisition unit 24B, a calibration parameter calculation unit 24C, a determination unit 24D, an output control unit 24E, a calibration unit 24F, and a learning unit 24G.

The image acquisition unit 24A, point cloud acquisition unit 24B, calibration parameter calculation unit 24C, determination unit 24D, output control unit 24E, calibration unit 24F, and learning unit 24G are implemented by, for example, one or more processors. For example, each of the above units may be implemented by having a processor such as a CPU (Central Processing Unit) execute a program, i.e., by software. Each of the above units may also be implemented by a dedicated IC or other processor, i.e., by hardware. Each of the above units may also be implemented using a combination of software and hardware. When multiple processors are used, each processor may implement one of the above units, or two or more of the above units.

The image acquisition unit 24A acquires image data. The image acquisition unit 24A sequentially acquires image data output sequentially from the imaging sensor 14.

The point cloud acquisition unit 24B acquires point cloud data. The point cloud acquisition unit 24B sequentially acquires the point cloud data output sequentially from the distance measuring sensor 16.

As described above, point cloud data is data in which depth information is defined for each of multiple points. Depth information represents the distance from the distance measuring sensor 16 to the point, which is an element in real space. Each of the multiple points constituting the point cloud data corresponds to one of the pixels that make up the image data.

The point cloud data acquired by the point cloud acquisition unit 24B is point cloud data with unknown calibration parameters and uncalibrated temporal and spatial errors. In other words, the point cloud acquisition unit 24B acquires point cloud data with unknown calibration parameters and uncalibrated temporal and spatial errors.

Calibration parameters are parameters used to calibrate at least one of the temporal and spatial errors in image data. In this embodiment, calibration parameters are described as representing temporal and spatial errors.

The temporal and spatial error of the temporal element refers to the time offset, which represents the difference in acquisition timing between image data and point cloud data. The image data acquisition timing refers to the timing when the image acquisition unit 24A acquires the image data from the imaging sensor 14. The point cloud data acquisition timing refers to the timing when the point cloud acquisition unit 24B acquires the point cloud data from the distance measuring sensor 16. Note that the image data acquisition timing may also be the timing when the imaging sensor 14 captures the image data. In this case, the point cloud data acquisition timing may also be the timing when the distance measuring sensor 16 measures the distance of the point cloud data.

The time offset is represented by the difference in the acquisition start times and acquisition update speeds of the image acquisition unit 24A and the point cloud acquisition unit 24B, respectively. The acquisition start time refers to the time when the image acquisition unit 24A and the point cloud acquisition unit 24B each began acquiring image data and point cloud data, respectively. The update speed refers to the speed at which the image acquisition unit 24A and the point cloud acquisition unit 24B each acquire image data and point cloud data, respectively.

In other words, the fact that the calibration parameters used to correct the temporal and spatial errors of the temporal elements are unknown means that the discrepancies in the acquisition start times and acquisition update rates of the image acquisition unit 24A and the point cloud acquisition unit 24B are unknown. Furthermore, the fact that the temporal and spatial errors of the temporal elements are uncalibrated means that the discrepancies in the acquisition start times and acquisition update rates of the image acquisition unit 24A and the point cloud acquisition unit 24B for at least one of the image data and point cloud data are uncalibrated.

The temporal-spatial error of spatial elements refers to at least one of the following: the spatial correspondence between pixels constituting image data and points constituting point cloud data is unknown, or there is a discrepancy in the correspondence. The spatial correspondence is represented by internal parameters that describe the physical structure of the imaging sensor 14, and external parameters that describe the spatial and relative positional relationship between the imaging sensor 14 and the distance measuring sensor 16.

In other words, the fact that the calibration parameters used to correct the temporal and spatial errors of spatial elements are unknown means that at least one of the intrinsic and extrinsic parameters is unknown. Furthermore, the fact that the temporal and spatial errors of spatial elements are uncalibrated means that there is no accurate correspondence between each of the multiple points constituting the point cloud data and each of the multiple pixels constituting the image data. When an accurate correspondence is established, each of the multiple pixels constituting the image data and each of the multiple points constituting the point cloud data represents an element at the same location and time of the same object in real space, for each pair of pixels and points.

Here, if the image sensor 14 and the distance measuring sensor 16 are hardware-synchronized, the image acquisition unit 24A and the point cloud acquisition unit 24B can acquire synchronized image data and point cloud data from the image sensor 14 and the distance measuring sensor 16, respectively. Furthermore, if hardware synchronization of the image sensor 14 and the distance measuring sensor 16 is difficult due to design reasons or other factors, the image acquisition unit 24A and the point cloud acquisition unit 24B can acquire synchronized image data and point cloud data based on the timestamps obtained from the image sensor 14 and the distance measuring sensor 16, respectively. However, errors may occur in the timestamps of the image sensor 14 and the distance measuring sensor 16 due to factors such as temperature and GPS (Global Positioning System) reception conditions. Therefore, the point cloud data may contain temporal and spatial errors represented by such temporal elements, or the temporal and spatial errors may be unknown.

Furthermore, in order to associate each of the multiple pixels constituting the image data with each of the multiple points constituting the point cloud data so that they represent the same elements in real space, it is necessary to perform the association using internal parameters that represent the physical structure of the imaging sensor 14 and external parameters that represent the spatial and relative positional relationship between the imaging sensor 14 and the distance measuring sensor 16. However, there are cases where at least one of these internal and external parameters is unknown. That is, point cloud data may contain temporal and spatial errors of the spatial elements represented by these internal and external parameters, or the temporal and intervalal errors may be unknown.

The point cloud acquisition unit 24B acquires point cloud data with uncalibrated temporal and spatial errors because the calibration parameters used to correct these errors are unknown.

The calibration parameter calculation unit 24C calculates calibration parameters for each training data set, which includes image data and point cloud data.

In this embodiment, the calibration parameter calculation unit 24C acquires a group consisting of multiple image data and multiple point cloud data points in a time-series sequence, acquired by the image acquisition unit 24A and the point cloud acquisition unit 24B, respectively, as training data.

A sequence is a predetermined time period. This predetermined time period may be, for example, 10 seconds, 20 seconds, etc., but is not limited to this time. That is, the calibration parameter calculation unit 24C acquires multiple image data and multiple point cloud data that are consecutive in time series, acquired by the image acquisition unit 24A and the point cloud acquisition unit 24B, respectively, during one sequence, as training data. The calibration parameter calculation unit 24C may also use a pair of one image data and one point cloud data as training data; it is not limited to acquiring image data and point cloud data for one sequence as training data.

In this embodiment, the calibration parameter calculation unit 24C calculates a calibration parameter and an error evaluation value representing the magnitude of the temporal and spatial errors expressed by the calibration parameter for each training data set.

For example, the calibration parameter calculation unit 24C calculates the calibration parameters and error evaluation values using one of the following three calculation methods: the first to the third calculation method.

First, let's explain the first calculation method.

In the first calculation method, the calibration parameter calculation unit 24C calculates a time offset as a calibration parameter, which indicates the timing difference between the acquisition of image data and point cloud data included in the training data, based on at least one of the following: position and orientation, relative angle, relative distance, angular velocity, and velocity of the respective imaging sensor 14 and distance measuring sensor 16. The calibration parameter calculation unit 24C also calculates an error evaluation value for the calculated calibration parameter.

In detail, the calibration parameter calculation unit 24C calculates the time offset by estimating the acquisition start time and acquisition update rate of the image acquisition unit 24A and the point cloud acquisition unit 24B, respectively.

Specifically, the calibration parameter calculation unit 24C determines the correspondence between multiple time-series image data and multiple point cloud data that constitute the training data by aligning the trajectories acquired, for example, from VO (Visual Odometry) and LO (LiDAR Odometry), and calculates a rough time offset. Then, the calibration parameter calculation unit 24C determines the initial values for the data acquisition start time and acquisition update speed of the image acquisition unit 24A and the point cloud acquisition unit 24B from the calculated rough time offset. The calibration parameter calculation unit 24C generates the initial values for the data acquisition start time and acquisition update speed from the rough time offset, and uses these generated initial values for calculating the calibration parameters described later.

Next, the calibration parameter calculation unit 24C determines at least one of the following from VO and LO: the position and orientation of the imaging sensor 14 and the distance measuring sensor 16, relative angle, relative distance, angular velocity, and speed.

The calibration parameter calculation unit 24C determines at least one of the following parameters for the distance measuring sensor 16 that acquired the point cloud data: position, orientation, relative angle, relative distance, angular velocity, and velocity. The relative angle and relative distance of the distance measuring sensor 16 refer to the relative angle and relative distance of the distance measuring sensor 16 at the time of distance measurement for each of the multiple point cloud data points in the time series constituting the learning data, relative to the distance measuring sensor 16 at the time of distance measurement for the point cloud data acquired at the preceding timing.

The calibration parameter calculation unit 24C needs to determine at least one of the following parameters from the position, orientation, relative angle, relative distance, angular velocity, and speed of the distance measuring sensor 16 that acquired point cloud data, based on the time-series position and orientation information obtained from the LO.

Similarly, the calibration parameter calculation unit 24C determines at least one of the following parameters for the imaging sensor 14 that captured the image data: position and orientation, relative angle, relative distance, angular velocity, and velocity. The relative angle and relative distance of the imaging sensor 14 refer to the relative angle and relative distance of the imaging sensor 14 at the time of each image capture of the multiple time-series image data constituting the training data, relative to the imaging sensor 14 at the time of capture of the image data acquired at the preceding timing.

The calibration parameter calculation unit 24C only needs to determine at least one of the following parameters of the image sensor 14 that acquired the image data: position, orientation, relative angle, relative distance, angular velocity, and speed, based on the time-series position and orientation information obtained from VO.

The calibration parameter calculation unit 24C then interpolates at least one of the time-series position and orientation, relative angle, relative distance, angular velocity, and velocity of the imaging sensor 14 or distance measuring sensor 16 so that the update speeds of the image acquisition unit 24A and the point cloud acquisition unit 24B are the same. Interpolation means adjusting at least one of the time-series position and orientation, relative angle, relative distance, angular velocity, and velocity of the imaging sensor 14 or distance measuring sensor 16 so that the update speeds of the image acquisition unit 24A and the point cloud acquisition unit 24B are the same. Then, the calibration parameter calculation unit 24C calculates the time offset, which is the difference in acquisition timing, by determining the acquisition start time or update speed that yields a high correlation from the cross-correlation function for waveforms a and b obtained by normalizing at least one of the interpolated position and orientation, relative angle, relative distance, angular velocity, and velocity.

Waveform a represents the transition of at least one of the following time series—position, orientation, relative angle, relative distance, angular velocity, and velocity—at the time of acquisition (i.e., shooting) for each of the multiple image data points included in the training data along their respective time series. Waveform b represents the transition of at least one of the following time series—position, orientation, relative angle, relative distance, angular velocity, and velocity—at the time of acquisition (i.e., distance measurement) for each of the multiple point cloud data points included in the training data along their respective time series.

The calibration parameter calculation unit 24C then uses i as the value of the number of image data points at each acquisition timing for at least one of the complementary position/orientation, relative angle, relative distance, angular velocity, and velocity obtained from the image data and point cloud data included in the training data. The calibration parameter calculation unit 24C also uses N as the value of the larger of the image data and point cloud data points, k as the sum of the number values of the image data and point cloud data points, and conj() as the function that converts each element of the array to its complex conjugate. Then, at a specific update speed, the array c showing the correlation between k combinations of image data and point cloud data acquisition start times is expressed by the following equation (1) from the cross-correlation function.

The calibration parameter calculation unit 24C obtains the above sequence c for multiple update speeds and searches for at least one of the data acquisition start time and update speed with a high correlation using a grid search. The calibration parameter calculation unit 24C calculates the time offset, represented by the difference between at least one of the data acquisition start time and update speed obtained through this search, as the calibration parameter.

Figure 2A is a schematic diagram of an example of waveforms a and b before applying the calculated time offset. Figure 2B is a schematic diagram of an example of waveforms a and b after applying the calculated time offset.

Return to Figure 1 and continue the explanation.

Furthermore, the calibration parameter calculation unit 24C calculates an error evaluation value that represents the magnitude of the temporal and spatial errors expressed by the calculated calibration parameters. Specifically, the calibration parameter calculation unit 24C calculates a higher error evaluation value the lower the correlation between the image data and point cloud data included in the training data. Correlation represents the degree of agreement and relationship between the temporal and spatial elements expressed by the temporal and spatial errors. In the first calculation method, the calibration parameter calculation unit 24C calculates a higher error evaluation value the lower the correlation represented by the above-mentioned array c. Methods for calculating the error evaluation value include using the overall correlation of multiple image data and multiple point cloud data included in the explored training data as the error evaluation value, or dividing multiple image data and multiple point cloud data along the time series included in the training data into multiple time intervals and using the correlation of each time interval as the error evaluation value.

Next, we will explain the second calculation method.

In the second calculation method, the calibration parameter calculation unit 24C reprojects the pixels constituting the image data onto the image data included in the training data or other image data, using the orientation information of the distance measuring sensor 16 at the time of acquisition of the point cloud data, which is presumed to have been acquired at the same timing as the image data. The calibration parameter calculation unit 24C then calculates the calibration parameter based on the reprojection error, which represents at least one of the following: the distance between the reprojected pixels and the pixels before reprojection, the difference in pixel values, and the difference in pixel feature quantities. In the second calculation method, the calibration parameter calculation unit 24C calculates a calibration parameter that includes at least one of the following: a time offset indicating the timing difference between the acquisition of the image data and the point cloud data included in the training data; an internal parameter indicating the structure of the imaging sensor 14 that captured the image data; and an external parameter indicating the spatial relationship between the imaging sensor 14 and the distance measuring sensor 16.

In detail, the calibration parameter calculation unit 24C determines the correspondence between multiple time-series image data and multiple point cloud data included in the training data by aligning the trajectories acquired, for example, from VO and LO, and calculates and applies a rough time offset. The calibration parameter calculation unit 24C generates initial values for the data acquisition start time and acquisition update speed from the rough time offset, and uses these generated initial values for calculating the calibration parameters described later.

Next, the calibration parameter calculation unit 24C defines the world coordinates of the time-series point cloud data included in the training data based on the orientation of the distance measuring sensor 16 at the initial time _t0 , and then obtains the orientation T _iL of the distance measuring sensor 16 at the time of acquisition (distance measurement) for each of the multiple point cloud data in the time series. The calibration parameter calculation unit 24C also determines the corresponding pixel P ^ijI between image _Ii of image data acquired at the same time as the point cloud data and image _Ij of other image data other than the said image data, using feature points or optical flow. Then, the calibration parameter calculation unit 24C determines the position p ^iw in world coordinates of pixel P _ijI by triangulation, based on the pixel correspondence and the relative orientation (T _iL ) ^{- 1 T jL} _of the imaging _sensor ¹⁴ obtained from the distance measuring _sensor ¹⁶ . Furthermore, the calibration parameter calculation unit 24C projects the position p _i ^w onto the image data I _k using the orientation T _k ^L at another time, and calculates ^the reprojection error p _ki using at _{least one of the following: the distance on the image coordinates to pixel P ki I} ^, the difference in pixel values between pixel P i j ^I and pixel P _ki ^I , and the difference in feature quantities between pixel P _{i j} ^I and pixel P _{ki I.}

The calibration parameter calculation unit 24C then converts the orientation of the distance measuring sensor 16 at the time of distance measurement for each of the multiple point cloud data in the time series included in the learning data into a continuous value by linear interpolation. Using the obtained reprojection error p _ki , the calibration parameter calculation unit 24C solves at least one of the following: the external parameter T _L ^C , the internal parameter π, and the time offset Δt _k, which encompasses the difference between the update rate and the initial time, using a nonlinear least squares method such as the LM (Levenberg Marquardt algorithm). Through this process, the calibration parameter calculation unit 24C calculates at least one of the time offset Δt _k , the internal parameter π, and the external parameter T _L ^C as calibration parameters.

Figure 3 is an illustrative diagram of the reprojection error p _ki .

The reprojection error p _ki is expressed by the following equation (2).

In equation (2), t _k represents the initial value of the time when image I _k was acquired.

The calibration parameter calculation unit 24C calculates the reprojection error p _ki for each feature point of all pixels pe _i* on the image data of neighboring image data with similar acquisition times for the image data included in the training data. Then, the calibration parameter calculation unit 24C obtains the sum of the individual reprojection errors p _ki of all the multiple image data included in the training data as the reprojection error of the image data.

Furthermore, the calibration parameter calculation unit 24C calculates an error evaluation value that represents the magnitude of the temporal and spatial errors expressed by the calculated calibration parameters. Specifically, the calibration parameter calculation unit 24C uses at least one of the following: the sum of the reprojection errors of all image data within the sequence included in the training data, or the sum of the reprojection errors of each pixel included in each image data. The larger the sum of the reprojection errors, the higher the error evaluation value.

Next, we will explain the third calculation method.

In the third calculation method, the calibration parameter calculation unit 24C calculates calibration parameters, including at least one of a time offset, an internal parameter, and an external parameter, based on the difference between the position information of the object observed by the point cloud data and the position information of the object observed by the image data.

In detail, the calibration parameter calculation unit 24C determines the correspondence between multiple time-series image data and multiple point cloud data that constitute the training data by aligning the trajectories acquired, for example, from VO and LO, and calculates and applies a rough time offset. The calibration parameter calculation unit 24C generates initial values for the data acquisition start time and acquisition update speed from the rough time offset, and uses these generated initial values for calculating the calibration parameters described later.

Next, the calibration parameter calculation unit 24C detects the same object observed in each of the multiple point cloud data included in the training data. The calibration parameter calculation unit 24C can detect the object, for example, by a machine learning-trained 3D object detector or by semantic segmentation of the point cloud data. The calibration parameter calculation unit 24C also calculates the 3D position x _i ' of the detected object relative to the distance measuring sensor 16. Furthermore, the calibration parameter calculation unit 24C calculates the size of the detected object using the point cloud data.

The calibration parameter calculation unit 24C then detects the object that appears in multiple image data included in the training data. The calibration parameter calculation unit 24C detects the object using a machine learning-based object detector or semantic segmentation. The calibration parameter calculation unit 24C then calculates the three-dimensional position x i of the object relative to the imaging sensor 14 using the size of the object calculated from the point cloud data, the size of the object on the image data I _i , and the internal parameter _π . The internal parameter π can be determined in advance or by using a predetermined initial value.

The calibration parameter calculation unit 24C calculates the difference te i between the three-dimensional position x _i ' of the object calculated from the point cloud data and the three-dimensional position x _i of the object calculated from the image data. The calibration parameter calculation unit 24C solves the difference te _i by converting the three-dimensional position of the object calculated from the image data and point cloud data into a continuous value by linear interpolation using a nonlinear least squares method such as the LM method, for at least one of the time offset Δt _i , the intrinsic parameter π, and the external parameter _TL ^C , which encompass the difference between the update speed and the initial time. Through this calculation process, the calibration parameter calculation unit 24C calculates at least one of the time offset Δt _i , the _intrinsic parameter π, and the external parameter _TL ^C as calibration parameters.

The positional difference te _i is expressed by the following equation (3), where ti _i is the synchronization time before calibration.

In equation (3), _ti represents the initial value of the time when image _Ii was acquired.

For the position difference te _i , you can use the sum of all synchronized data.

Figure 4A is an illustrative diagram of an example of the trajectories of positions xi _' and _xi before applying calibration parameters. Figure 4B is an illustrative diagram of an example of the trajectories of positions _xi ' and _xi after applying calibration parameters. In Figures 4A and 4B, the vertical axis represents the vertical horizontal distance from the imaging sensor 14 and the distance measuring sensor 16 to the object, and the horizontal axis represents the horizontal horizontal distance.

Furthermore, the calibration parameter calculation unit 24C calculates an error evaluation value that represents the magnitude of the temporal and spatial errors expressed by the calculated calibration parameters. Specifically, the calibration parameter calculation unit 24C calculates an error evaluation value that is the sum of the differences in the positions of all image data and all point cloud data within the sequence included in the training data, or a higher error evaluation value the larger the position difference in each data point.

Return to Figure 1 and continue the explanation.

Furthermore, the calibration parameter calculation unit 24C may further calculate a confidence level for the training data used to calculate the error evaluation value, based on the calculated error evaluation value. The confidence level is lower the larger the temporal and spatial error represented by the error evaluation value. The confidence level is an evaluation value representing the effectiveness of the training data. A higher confidence level indicates higher effectiveness as training data.

Furthermore, the calibration parameter calculation unit 24C may normalize at least one of the variances of the error evaluation values within multiple image data and multiple point cloud data included in the training data, or within multiple training data and multiple point cloud data, and then calculate the confidence level for the training data.

The calibration parameter calculation unit 24C stores the learning data, the calibration parameters calculated from the learning data, the error evaluation value, and the confidence level in the storage unit 20, associating them with each other.

The determination unit 24D determines the validity of the training data used to calculate the calibration parameters, based on the calibration parameters. Determining the validity of the training data means determining whether the calibrated training data, obtained by calibrating the training data using the calibration parameters calculated from the training data, is valid training data for training the learning model.

In detail, the determination unit 24D determines that the higher the error evaluation value of the temporal and spatial errors represented by the calibration parameters, the lower the effectiveness of the training data used to calculate the calibration parameters. Conversely, the lower the error evaluation value, the higher the effectiveness of the training data used to calculate the calibration parameters representing the temporal and spatial errors represented by the error evaluation value.

Specifically, for example, the determination unit 24D pre-sets a threshold value for use in determining effectiveness based on the error evaluation value. If the error evaluation value is less than the threshold value, the determination unit 24D determines that the training data used to calculate the calibration parameters representing the temporal and spatial errors expressed by the error evaluation value is highly effective training data. Conversely, if the error evaluation value is equal to or greater than the threshold value, the determination unit 24D determines that the training data used to calculate the calibration parameters representing the temporal and spatial errors expressed by the error evaluation value is less effective training data. The determination unit 24D may also determine that the training data deemed less effective is invalid training data.

Furthermore, the determination unit 24D may recalculate the error evaluation value based on the variance within the training data, or within each of the multiple image data and multiple point cloud data included in the training data. If the recalculated error evaluation value is less than a predetermined threshold, the determination unit 24D determines that the training data used to calculate the calibration parameters representing the temporal and spatial errors expressed by the error evaluation value is highly effective training data. If the error evaluation value is greater than or equal to the threshold, the determination unit 24D determines that the training data used to calculate the calibration parameters representing the temporal and spatial errors expressed by the error evaluation value is less effective training data. The determination unit 24D may also determine that the training data determined to be less effective is invalid training data.

Furthermore, the determination unit 24D may determine that a lower value represents the effectiveness judgment result, where the error evaluation value is higher.

The determination unit 24D then stores the effectiveness determination result in the storage unit 20, associating it with the error evaluation value and learning data used in the determination.

Therefore, the memory unit 20 stores the effectiveness determination result, calibration parameters, error evaluation value, and confidence level for each training data set, which includes multiple image data and multiple point cloud data for one sequence.

The output control unit 24E outputs the judgment result of the judgment unit 24D. For example, the output control unit 24E outputs the training data and the judgment result of the effectiveness of the training data in association with each other to the UI unit 12. By outputting the judgment result to the UI unit 12, the user can be provided with a visual representation of the training data effectiveness judgment result. The user can then instruct the system to remove training data judged to have low effectiveness and use training data judged to have high effectiveness for training the learning model, thereby enabling the use of training data suitable for high-precision learning.

Furthermore, the output control unit 24E outputs the training data and the results of the effectiveness assessment of the training data to an external information processing device via the communication unit 22. The external information processing device, based on the effectiveness assessment results, removes training data deemed to have low effectiveness and uses training data deemed to have high effectiveness for training the learning model, thereby enabling the use of training data suitable for high-precision learning.

Furthermore, the output control unit 24E may output the training data and the result of determining the effectiveness of the training data to the calibration unit 24F.

In this case, the output control unit 24E outputs the training data whose effectiveness is determined to be above a predetermined value, and the calibration parameters calculated from the training data, to the calibration unit 24F. Alternatively, the output control unit 24E outputs the training data acquired by the image acquisition unit 24A and the point cloud acquisition unit 24B, the calibration parameters calculated from the training data, and at least one of the effectiveness determination result of the training data and the confidence level for the training data, to the calibration unit 24F.

Effectiveness exceeding a predetermined value means that the error evaluation value, which represents the magnitude of the temporal and spatial errors expressed by the calibration parameters calculated from the training data, is equal to or greater than the threshold mentioned above.

The calibration unit 24F calibrates at least one of the image data and point cloud data included in training data whose effectiveness is above a predetermined value, using calibration parameters calculated from the training data. That is, for training data determined by the determination unit 24D to be valid because its error evaluation value is below a threshold, the calibration unit 24F calibrates at least one of the image data and point cloud data included in the training data, using calibration parameters calculated from the training data. The calibration unit 24F should calibrate at least one of the image data and point cloud data included in the training data using the calibration parameters so that the temporal and spatial errors between the image data and point cloud data are canceled out. A known method may be used for the calibration process of at least one of the image data and point cloud data using the calibration parameters.

Furthermore, regardless of whether the effectiveness is above or below a predetermined value, the calibration unit 24F may calibrate at least one of the image data and point cloud data contained in each of the learning data stored in the storage unit 20 using calibration parameters calculated from the learning data.

The learning unit 24G learns a learning model that outputs estimated depth information for each pixel constituting the image data, based on the training data calibrated by the calibration unit 24F. The learning model is a machine learning model.

For example, consider a case where the calibration unit 24F calibrates at least one of the image data and point cloud data included in training data whose effectiveness is above a predetermined value, using calibration parameters calculated from the training data. In other words, consider a case where the calibration unit 24F calibrates training data that the determination unit 24D has determined to be valid because its error evaluation value is below a threshold, using the calibration parameters.

In this case, the learning unit 24G inputs image data included in the learning data, which has been determined to be valid learning data and calibrated according to the calibration parameters, into the learning model. It then adjusts the parameters of the learning model so that the error between the estimated depth information for each pixel output from the learning model and the depth information for each point defined by the point cloud data included in the learning data is minimized. Through this parameter adjustment process, the learning unit 24G learns the learning model.

The learning unit 24G can learn a highly accurate learning model that outputs highly accurate estimated depth information from image data by training the learning model using learning data that has been determined to be valid and calibrated according to the calibration parameters.

Furthermore, consider a scenario where, for example, the calibration unit 24F, regardless of whether its effectiveness is above or below a predetermined value, calibrates at least one of the image data and point cloud data contained in all the learning data received from the output control unit 24E using calibration parameters calculated from the learning data.

In this case, the learning unit 24G inputs the image data included in the calibrated training data into the learning model and calculates the error between the estimated depth information for each pixel output from the learning model and the depth information for each point defined by the point cloud data included in the training data. The learning unit 24G then adjusts the parameters of the learning model so that the smaller the temporal and spatial error represented by the calibration parameters calculated by the calibration parameter calculation unit 24C, the greater the confidence level, and the smaller the error, the greater the weighted error. Specifically, the learning unit 24G calculates a weighted error by weighting the error between the estimated depth information and the depth information so that the value increases with higher confidence levels. For example, the learning unit 24G calculates a weighted error by weighting the error between the estimated depth information and the depth information by the confidence level. The learning unit 24G then adjusts the parameters of the learning model so that the weighted error is minimized. Through this parameter adjustment process, the learning unit 24G trains the learning model.

The learning unit 24G can train a highly accurate learning model that outputs highly accurate estimated depth information from image data by adjusting the parameters of the learning model so that the weighted error, which is weighted according to the confidence level, is minimized.

Alternatively, the learning unit 24G may generate a permutation of the training data according to the confidence level, and input the image data contained in the training data into the learning model according to the permutation. More specifically, the learning unit 24G may retrieve all the training data stored in the storage unit 20, sort the training data in descending order of confidence level, and then input the image data contained in the training data into the learning model in descending order of confidence level for training.

Furthermore, the learning unit 24G may sort the training data, which has been calibrated using calibration parameters, in descending order of confidence level, and then input the image data contained in the training data into the learning model in descending order of confidence level for training.

The learning unit 24G generates permutations of training data according to the confidence level, and inputs the image data contained in the training data into the learning model according to the permutations. This allows for faster convergence of learning, improved learning efficiency, reduced learning time, and high-precision learning.

Next, an example of the information processing flow executed by the information processing device 10 of this embodiment will be described.

Figure 5 is a flowchart showing an example of the flow of the learning data effectiveness determination process performed by the information processing device 10 of this embodiment.

The calibration parameter calculation unit 24C acquires training data including image data acquired from the imaging sensor 14 by the image acquisition unit 24A and point cloud data acquired from the distance measuring sensor 16 by the point cloud acquisition unit 24B (step S100). For example, the calibration parameter calculation unit 24C acquires a group consisting of multiple image data and multiple point cloud data for one sequence along a time series as training data.

The calibration parameter calculation unit 24C calculates calibration parameters for the training data acquired in step 100 (step S102). The calibration parameter calculation unit 24C also calculates an error evaluation value representing the magnitude of the temporal and spatial errors expressed by the calibration parameters calculated in step S102 (step S104). Furthermore, based on the error evaluation value calculated in step S104, the calibration parameter calculation unit 24C calculates a confidence level for the training data used to calculate the error evaluation value, with a lower confidence level associated with larger temporal and spatial errors (step S106).

The determination unit 24D determines whether the error evaluation value calculated in step S104 is below the threshold (step S108). If it determines that the error evaluation value is below the threshold (step S108: Yes), the determination unit 24D determines that the effectiveness of the training data acquired in step S100 is high (step S110). Then, the process proceeds to step S114.

On the other hand, if the error evaluation value is determined to be above the threshold (step S108: No), the determination unit 24D determines that the effectiveness of the training data acquired in step S100 is low (step S112). Then, the process proceeds to step S114.

In step S114, the determination unit 24D stores the learning data acquired in step 100, the calibration parameters calculated in step S102, the effectiveness determination result from step S110 or step 112, and the error evaluation value and confidence level calculated in steps S104 and S106, respectively, in the storage unit 20, associating them with each other.

The output control unit 24E outputs the determination result of the determination unit 24D in step S110 or step S112 to the UI unit 12 and an external information processing device, etc. (step S116). Then, this routine terminates.

The processing unit 24 executes steps S100 to S114 for each different sequence of training data, thereby storing the calibration parameters, effectiveness determination results, error evaluation values, and reliability scores in the storage unit 20 for each of the multiple training data sets.

Next, the learning process executed by the processing unit 24 will be described.

Figure 6 is a flowchart showing an example of the learning process flow executed by the processing unit 24.

The calibration unit 24F calibrates at least one of the image data and point cloud data contained in each of the training data stored in the storage unit 20, using calibration parameters calculated from the training data (step S200). As described above, the calibration unit 24F may also calibrate at least one of the image data and point cloud data contained in training data for which the error evaluation value is less than a threshold and is determined to have high effectiveness, using calibration parameters calculated from the training data.

Then, the learning unit 24G learns a learning model that outputs estimated depth information for each pixel constituting the image data from the image data, based on the training data calibrated in step S200 (step S202). Finally, this routine terminates.

As described above, the information processing program executed by the information processing device of this embodiment comprises: an image acquisition step for acquiring image data; a point cloud acquisition step for acquiring point cloud data in which depth information is defined for each of multiple points, where the calibration parameters used to calibrate at least one of the temporal and spatial errors of the image data are unknown and the temporal and spatial errors are uncalibrated; a calibration parameter calculation step for calculating calibration parameters for each training data including image data and point cloud data; a determination step for determining the validity of the training data used to calculate the calibration parameters based on the calibration parameters; and an information processing program to be executed by a computer.

Conventional techniques disclose methods for calculating calibration parameters for uncalibrated training data where the calibration parameters are unknown, and then using these parameters to calibrate the training data. However, training data calibrated using these calibration parameters may contain training data that failed to be calibrated. That is, training data with temporal and spatial errors that cannot be fully calibrated even with the use of calibration parameters cannot have those errors corrected by applying the calibration parameters. Using such calibrated training data in training a model results in using training data that failed to be calibrated, which can prevent high-accuracy learning. In other words, conventional techniques did not provide training data suitable for high-accuracy learning.

On the other hand, the information processing program of this embodiment determines the effectiveness of the training data used to calculate the calibration parameters, based on the calibration parameters calculated for each uncalibrated training data set, which includes image data and point cloud data. In other words, the information processing program of this embodiment determines the effectiveness of the training data after calibration using the calibration parameters.

Therefore, the information processing program of this embodiment can determine the effectiveness of the training data, which has been calibrated using calibration parameters, indicating whether or not it is suitable for training a learning model. Furthermore, the information processing program of this embodiment can provide the effectiveness of the training data as data to be used for training a learning model. Therefore, the information processing program of this embodiment can support the provision of training data suitable for high-precision learning.

Therefore, the information processing program of this embodiment can support the provision of training data suitable for high-precision learning.

Furthermore, since the information processing program of this embodiment learns the learning model according to the effectiveness of the training data, it can perform highly accurate learning.

Next, an example of the hardware configuration of the information processing device 10 of the above embodiment will be described.

Figure 7 is a hardware configuration diagram of an example of the information processing device 10 in the above embodiment.

The information processing device 10 of the above embodiment comprises a control device such as a CPU (Central Processing Unit) 90D, storage devices such as a ROM (Read Only Memory) 90E, RAM (Random Access Memory) 90F, and HDD (Hard Disk Drive) 90G, an I/F unit 90B which serves as an interface to various devices, an output unit 90A which outputs various information such as output information, an input unit 90C which accepts user operations, and a bus 90H which connects each unit. It has a hardware configuration that utilizes a standard computer.

In the information processing device 10 of the above embodiment, the CPU 90D reads the information processing program from the ROM 90E onto the RAM 90F and executes it, thereby realizing each of the above components on the computer.

Furthermore, the information processing program for executing each of the above processes performed by the information processing device 10 in the above embodiment may be stored in the HDD 90G. Alternatively, the information processing program for executing each of the above processes performed by the information processing device 10 in the above embodiment may be pre-installed and provided in the ROM 90E.

Furthermore, the information processing program for executing the above-described process performed by the information processing device 10 of the above embodiment may be provided as a computer program product by being stored in an installable or executable file format on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD (Digital Versatile Disc), or flexible disk (FD). Alternatively, the program for executing the above-described process performed by the information processing device 10 of the above embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. Furthermore, the program for executing the above-described process performed by the information processing device 10 of the above embodiment may be provided or distributed via a network such as the Internet.

While embodiments of the present invention have been described above, these embodiments are presented as examples only and are not intended to limit the scope of the invention. This novel embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. This embodiment and its variations are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as described in the claims.

10 Information Processing Device 14 Imaging Sensor 16 Distance Measuring Sensor 24A Image Acquisition Unit 24B Point Cloud Acquisition Unit 24C Calibration Parameter Calculation Unit 24D Judgment Unit 24E Output Control Unit 24F Calibration Unit 24G Learning Unit

Claims (13)

Image acquisition step to obtain image data,
A point cloud acquisition step in which depth information is defined for each of a plurality of points, for which the calibration parameters used to calibrate the temporal and spatial errors of at least one of the temporal and spatial elements of the aforementioned image data are unknown and the temporal and spatial errors are uncalibrated;
A calibration parameter calculation step, which calculates the calibration parameter for each training data including the image data and the point cloud data,
A determination step in which the effectiveness of the training data used to calculate the calibration parameters is determined based on the calibration parameters,
An information processing program that causes a computer to execute something. The calibration parameter calculation step is as follows:
The calibration parameters and an error evaluation value representing the magnitude of the temporal and spatial errors expressed by the calibration parameters are calculated.
The aforementioned determination step is,
The higher the error evaluation value, the lower the effectiveness is judged.
The information processing program according to claim 1. The calibration parameter calculation step is as follows:
Based on at least one of the position, orientation, relative angle, relative distance, angular velocity, and speed of the distance measuring sensor that acquired the point cloud data and the imaging sensor that captured the image data, a time offset indicating the timing difference between the acquisition of the image data and the point cloud data included in the training data is calculated as the calibration parameter.
The information processing program according to claim 1. The calibration parameter calculation step is as follows:
Based on a reprojection error that represents at least one of the distance, the difference in pixel values, and the difference in pixel features between the reprojected pixels, which are reprojected onto the image data or other image data included in the training data, and the pixels constituting the image data, using the attitude information of the distance sensor at the time of acquisition of the point cloud data, which is presumed to have been acquired at the same time as the image data, and the pixels constituting the image data,
The calibration parameter is calculated, which includes at least one of the following: a time offset indicating the timing difference between the acquisition of the image data and the point cloud data included in the learning data; an internal parameter indicating the structure of the imaging sensor that captured the image data; and an external parameter indicating the spatial relationship between the imaging sensor and the distance measuring sensor.
The information processing program according to claim 1. The calibration parameter calculation step is as follows:
Based on the difference between the position information of an object observed by the point cloud data and the position information of the object observed by the image data, the calibration parameter is calculated, which includes at least one of the following: a time offset indicating the timing difference between the acquisition of the image data and the point cloud data included in the training data; an internal parameter indicating the structure of the imaging sensor that captured the image data; and an external parameter indicating the spatial relationship between the imaging sensor and the distance measuring sensor that acquired the point cloud data.
The information processing program according to claim 1. The calibration parameter calculation step is as follows:
Based on the error evaluation value, a confidence level for the training data is further calculated, with the confidence level decreasing as the temporal and spatial errors represented by the error evaluation value increase.
The information processing program according to claim 2. An output control step that outputs the determination result of the determination step,
The information processing program according to claim 1, which includes the following: A calibration step in which at least one of the image data and point cloud data included in the training data whose effectiveness is greater than or equal to a predetermined value is calibrated by the calibration parameter calculated from the training data,
A learning step of training a learning model that outputs estimated depth information for each pixel constituting the image data from the image data based on the calibrated training data,
The information processing program according to claim 1, which includes the following: The aforementioned learning steps are:
The image data included in the calibrated training data, whose effectiveness is equal to or greater than a predetermined value, is input to the training model.
The parameters of the learning model are adjusted so that the error between the estimated depth information for each pixel output from the learning model and the depth information for each point defined by the point cloud data included in the calibrated learning data is minimized.
The information processing program according to claim 8. The aforementioned learning steps are:
The image data included in the calibrated training data is input to the training model, and the error between the estimated depth information for each pixel output from the training model and the depth information for each point defined by the point cloud data included in the calibrated training data is calculated.
The parameters of the learning model are adjusted so that the weighted error, which is obtained by weighting the error according to the confidence level, is minimized as the temporal and spatial errors represented by the calibration parameters calculated from the learning data become smaller.
The information processing program according to claim 9. The aforementioned learning steps are:
The system generates permutations of the training data according to the magnitude of the confidence level, where the temporal and spatial errors represented by the calibration parameters are smaller, and inputs the image data included in the training data into the training model according to the permutations.
The information processing program according to claim 8. Image acquisition unit that acquires image data,
A point cloud acquisition unit acquires point cloud data in which depth information is defined for each of a plurality of points, for which the calibration parameters used to calibrate at least one of the temporal and spatial errors of the temporal and spatial elements of the aforementioned image data are unknown and the temporal and spatial errors are uncalibrated.
A calibration parameter calculation unit calculates the calibration parameter for each training data including the image data and the point cloud data,
A determination unit that determines the effectiveness of the learning data used to calculate the calibration parameters based on the calibration parameters,
An information processing device equipped with the following features. Image acquisition step to obtain image data,
A point cloud acquisition step in which depth information is defined for each of a plurality of points, for which the calibration parameters used to calibrate the temporal and spatial errors of at least one of the temporal and spatial elements of the aforementioned image data are unknown and the temporal and spatial errors are uncalibrated;
A calibration parameter calculation step, which calculates the calibration parameter for each training data including the image data and the point cloud data,
A determination step in which the effectiveness of the training data used to calculate the calibration parameters is determined based on the calibration parameters,
Information processing methods including