JP7653102B2 - Parameter estimation device, parameter estimation method, and parameter estimation program - Google Patents

Description

The present invention relates to a parameter estimation device, a parameter estimation method, and a parameter estimation program.

Bundle Adjustment is a technique that detects feature points from images of the same subject captured by multiple cameras, associates the detected feature points between the cameras (between images), and then estimates the camera parameters and the 3D coordinates of each feature point. Bundle Adjustment estimates optimal camera parameters and the 3D coordinates of each feature point by performing iterative calculations using a cost function that minimizes the reprojection error of each feature point associated between the cameras.

FIG. 14 is a schematic diagram for explaining the outline of the bundle adjustment. FIG. 14 shows, as an example, a case where the same subject is captured by two cameras (camera j and camera j+1) from different positions. As shown in FIG. 14, in the bundle adjustment, for example, feature points are detected from an image captured by camera j and an image captured by camera j+1. The feature points detected from both images are associated with each other for each feature point at the same location of the subject. Here, the two-dimensional coordinates of the feature point detected from the image captured by camera j are u _i,j , the two-dimensional coordinates of the feature point detected from the image captured by camera j+1 are u _i,j+1 , and the three-dimensional coordinates of a certain feature point are p _i .

In the bundle adjustment, iterative calculation is performed to minimize the reprojection error between the three-dimensional coordinate p _i of the feature point reprojected onto the screen of camera j and the two-dimensional coordinate u _i,j of the feature point reprojected onto the screen of camera j, and the reprojection error between the three-dimensional coordinate p i+1 of the feature point reprojected onto the screen of camera _j+1 and the two-dimensional coordinate u _{i,j+1 of the feature point reprojected onto the screen of camera j+1,} by a cost function expressed by the following formula (1), for example. As a result, the three-dimensional coordinate p _i of the feature point is estimated. For example, the Gauss-Newton method or the like is used for the iterative calculation.

In the above formula (1), the function F( _pi | _fj , _rj , _dj , _Rj , _tj ) is a function that projects the three-dimensional coordinates _pi of the feature point onto the screen of the camera j. Here, _fj , _rj , _dj , _Rj , and _tj are the camera parameters of the camera j. Specifically, _fj is the camera focal length, _rj is the camera resolution, _dj is the camera distortion, _Rj is the camera rotation angle, and _tj is the camera position. In the case of the camera parameters of the camera j+1, similarly, fj ₊₁ is the camera focal length, rj ₊₁ is the camera resolution, dj ₊₁ is the camera distortion, Rj ₊₁ is the camera rotation angle, and tj ₊₁ is the camera position.

In general, when the object for which the 3D coordinates of feature points are to be acquired is a geometric structure such as a building, a method is widely used to detect feature points and match between cameras using SIFT (Scale-Invariant Feature Transform) features. On the other hand, when the object for which the 3D coordinates of feature points are to be acquired is a person, 2D pose estimation techniques such as OpenPose (see Non-Patent Document 1) and PersonLab (see Non-Patent Document 2) are used. These techniques detect parts of a person, such as joints, eyes, nose, and ears. Then, for example, under the assumption that the length of a person's skeleton is invariant, estimation is performed using Bundle Adjustment, regarding the detected parts of the person as feature points (see Non-Patent Document 3).

Z. Cao, G. Hidalgo, T. Simon, S.E. Wei, and Y. Sheikh, “OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields”, arXiv: 1812.08008v2 [cs.CV], pp.1-14, May 2019. G. Papandreou, T. Zhu, L. C. Chen, S. Gidaris, J. Tompson, and K. Murphy, “PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Model”, ECCV, 2018. K. Takahashi, D. Mikami, M Isogawa, and H. Kimata, “Human Pose as Calibration Pattern; 3D Human Pose Estimation with Multiple Unsynchronized and Uncalibrated Cameras”, CVPR, pp.1888-1895, 2018. T. Ohashi1, Y. Ikegami, and Y. Nakamura, “Synergetic Reconstruction from 2D Pose and 3D Motion for Wide-Space Multi-Person Video Motion Capture in the Wild”, arXiv: 2001.05613v2 [cs.CV], pp.1-12, Oct 2020.

In Bundle Adjustment, a method is widely used in which parameters are updated to minimize the mean squared error (MSE) of the reprojection error in the process of iterative calculations that optimize each parameter. However, even in the process of feature point detection by the aforementioned OpenPose and PersonLab, estimation errors are superimposed on the two-dimensional coordinates of the feature points, so that each feature point of the subject in an image (e.g., each frame of a video) has at most as many sampling points as there are cameras. Therefore, if estimation errors in the two-dimensional coordinates of feature points are superimposed due to large noise, the estimation accuracy of the camera parameters and the three-dimensional coordinates of each feature point can be significantly impaired.

In particular, as shown in FIG. 15, for example, even if the estimation errors for both feature points are similar (for example, an estimation error of 2 pixels for both) when the subject appears small on the screen and when it appears large on the screen, the estimation errors for both are different in three-dimensional space. In this case, the smaller the subject appears on the screen, the larger the estimation error will be, and the larger the subject appears on the screen, the smaller the estimation error will be. Therefore, in conventional methods that treat a subject that appears small on the screen in the same way as a subject that appears large on the screen and perform estimation, there is a possibility that the estimation accuracy of the camera parameters and the three-dimensional coordinates of each feature point will decrease.

In particular, as shown in FIG. 16, for example, there may be cases where the image area of the feature point to be detected overlaps with, for example, an image area of another part of the human subject, or an image area of another person appearing in the same image, resulting in the presence of an occluded image area. In such cases, the estimation accuracy of the camera parameters and the three-dimensional coordinates of each feature point may decrease in the conventional method that performs estimation by treating a subject with an occluded image area in the same way as a subject with no occluded image area.

In this way, conventional bundle adjustment detects feature points of a subject from images captured by multiple cameras, and estimates the three-dimensional coordinates and camera parameters of each feature point (hereinafter collectively referred to as the "parameter group") by matching the two-dimensional coordinates of each feature point between images. However, there is a problem in that if the detection of feature points contains estimation errors, the accuracy of estimating the parameter group can be significantly impaired.

In view of the above circumstances, the present invention aims to provide a parameter estimation device, a parameter estimation method, and a parameter estimation program that can improve the estimation accuracy in a technology that estimates a group of parameters based on the two-dimensional coordinates of feature points detected from images captured by multiple cameras.

One aspect of the present invention is a parameter estimation device that includes an acquisition unit that acquires subject posture group data indicating data in which two-dimensional coordinates of the same feature point of the subject are associated with each other in each image of the same subject captured by a plurality of cameras from different positions; a weighting factor calculation unit that calculates a visibility rate indicating the proportion of the area of the feature point that is not occluded by areas other than the feature point based on the subject posture group data acquired by the acquisition unit and calculates a weighting factor based on the calculated visibility rate; and a parameter estimation unit that estimates a parameter value including the three-dimensional coordinate value of the feature point based on a calculation using the weighting factor calculated by the weighting factor calculation unit.

Also, one aspect of the present invention is a parameter estimation method having an acquisition step of acquiring subject posture group data indicating data in which two-dimensional coordinates of the same feature point of the subject are associated with each other in each image of the same subject captured by a plurality of cameras from different positions; a weighting factor calculation step of calculating a visibility rate indicating the proportion of the area of the feature point that is not occluded by areas other than the feature point based on the subject posture group data acquired in the acquisition step, and calculating a weighting factor based on the calculated visibility rate; and a parameter estimation step of estimating a value of a parameter including a coordinate value of the three-dimensional coordinate of the feature point based on a calculation using the weighting factor calculated in the weighting factor calculation step.

Another aspect of the present invention is a parameter estimation program for causing a computer to function as the above-mentioned parameter estimation device.

The present invention makes it possible to improve the estimation accuracy in a technique for estimating a group of parameters based on the two-dimensional coordinates of feature points detected from images captured by multiple cameras.

1 is a block diagram showing the overall configuration of an object 3D reconstruction apparatus 100 according to a first embodiment of the present invention. 4 is a flowchart showing the operation of the object 3D reconstruction apparatus 100 according to the first embodiment of the present invention. 2 is a block diagram showing the functional configuration of an optimization unit 102 of the object 3D reconstruction apparatus 100 according to the first embodiment of the present invention. FIG. FIG. 2 is a diagram showing an example of a connection relationship between feature points in a human skeleton or the like. 11 is a diagram for explaining a method of deriving a weighting coefficient based on an occupancy rate within a screen. FIG. 4 is a flowchart showing an operation of an optimization unit 102 of the object 3D reconstruction apparatus 100 according to the first embodiment of the present invention. FIG. 11 is a block diagram showing the overall configuration of an object 3D reconstruction device 100a according to a second embodiment of the present invention. 10 is a flowchart showing the operation of an object 3D reconstruction device 100a according to the second embodiment of the present invention. FIG. 11 is a block diagram showing the functional configuration of an optimization unit 102a of an object 3D reconstruction device 100a according to a second embodiment of the present invention. 11 is a diagram for explaining a method of deriving a weighting factor based on a visibility rate for self-occlusion. FIG. 11 is a diagram for explaining a method of deriving a weighting factor based on a visibility rate with respect to occlusion by another person; FIG. 10 is a flowchart showing the operation of an optimization unit 102a of an object 3D reconstruction apparatus 100 according to the second embodiment of the present invention. 1 is a block diagram showing an example of a hardware configuration of an object 3D reconstruction device according to the first to third embodiments of the present invention. FIG. 13 is a schematic diagram for explaining an overview of Bundle Adjustment. 11A and 11B are diagrams for explaining differences in the size of a subject within a screen and two-dimensional estimation errors and three-dimensional estimation errors. 11A and 11B are diagrams for explaining a decrease in the estimation accuracy of two-dimensional coordinates due to obstruction by another person;

Below, embodiments of the parameter estimation device, parameter estimation method, and parameter estimation program of the present invention will be described with reference to the drawings.

First Embodiment
The following describes an object 3D reconstruction device 100 according to a first embodiment of the present invention. Note that the object 3D reconstruction device 100 described below is one example of an embodiment of a parameter estimation device of the present invention, and the present invention is not necessarily limited to the configuration of the object 3D reconstruction device 100 described below.

[Configuration of the object 3D reconstruction device]
The overall configuration of the object 3D reconstruction device 100 will be described below. FIG. 1 is a block diagram showing the overall configuration of the object 3D reconstruction device 100 in the first embodiment of the present invention. As shown in FIG. 1, the object 3D reconstruction device 100 includes an inter-camera correspondence unit 101 and an optimization unit 102. Note that in the object 3D reconstruction device 100 shown in FIG. 1, the optimization unit 102 has a different configuration from that of a conventional general object 3D reconstruction device, and the other configurations are the same as those of the conventional object 3D reconstruction device.

The camera correspondence unit 101 accepts input of subject posture group data output from an external device (not shown). The subject posture group data here refers to data generated from images of a person as a subject captured simultaneously from different positions by multiple cameras. The subject posture group data includes data indicating the positions of the joints, eyes, nose, ears, etc. of the person as a subject in each image, detected by a two-dimensional posture estimation technique represented by OpenPose, PersonLab, etc.
The camera correspondence unit 101 recognizes the positions of the joints, eyes, nose, ears, etc. of the person who is the subject in each image as feature points based on the input subject posture group data. The camera correspondence unit 101 associates feature points of the same part of the same person in images captured simultaneously by different cameras (for example, feature points indicating the positions of right elbows, or feature points indicating the positions of left knees, etc.). Hereinafter, data in which these feature points are associated is referred to as "camera-correlated subject posture group data." The camera correspondence unit 101 generates camera-correlated subject posture group data and outputs it to the optimization unit 102.

The optimization unit 102 receives input of the camera-matched subject posture group data output from the camera-matching unit 101. The optimization unit 102 executes a bundle adjustment using feature points based on the camera-matched subject posture group data to optimize a parameter group. The parameter group here refers to the camera parameter values of each camera that captured the original image, and the three-dimensional coordinates of each feature point. The camera parameters include, for example, the camera position, imaging direction, rotation angle, field of view, focal length, resolution, and distortion.

The optimization unit 102 outputs the optimized parameter group to an external device (not shown). The optimized parameter group here refers to a parameter group optimized for the purpose of minimizing the mean square error of the difference (reprojection error) between the actual 3D coordinates of the feature points and the estimated 3D coordinates.

The inter-camera correspondence unit 101 may be provided in an external device. In other words, the inter-camera corresponded subject posture group data may be generated in an external device (not shown), and the subject 3D reconstruction device 100 may be configured to acquire the inter-camera corresponded subject posture group data from the external device.

[Operation of the Object 3D Reconstruction Apparatus]
An example of the operation of object 3D reconstruction apparatus 100 will be described below. Fig. 2 is a flowchart showing the operation of object 3D reconstruction apparatus 100 in the first embodiment of the present invention. This flowchart starts when object posture group data output from an external device is input to object 3D reconstruction apparatus 100.

The camera-to-camera correspondence unit 101 of the object 3D reconstruction device 100 accepts input of object posture group data output from an external device. The camera-to-camera correspondence unit 101 recognizes the positions of the joints, eyes, nose, ears, etc. of the person who is the object in each image as feature points based on the input object posture group data. The camera-to-camera correspondence unit 101 matches feature points of the same part of the same person in images captured simultaneously by different cameras (step S101). The camera-to-camera correspondence unit 101 generates camera-to-camera corresponded object posture group data and outputs it to the optimization unit 102.

The optimization unit 102 receives the inter-camera-correlated subject posture group data output from the inter-camera correspondence unit 101. The optimization unit 102 executes a bundle adjustment using feature points based on the inter-camera-correlated subject posture group data to optimize the parameter group (step S102). The optimization unit 102 outputs the optimized parameter group to an external device. This completes the operation of the object 3D reconstruction device 100 shown in the flowchart of FIG. 2.

[Configuration of the optimization unit]
The configuration of the optimization unit 102 will be described in more detail below. Fig. 3 is a block diagram showing the functional configuration of the optimization unit 102 of the object 3D reconstruction device 100 in the first embodiment of the present invention. As shown in Fig. 3, the optimization unit 102 includes a parameter initialization unit 1021, a weighting factor initialization unit 1022, a parameter optimization unit 1023, an object orientation reprojection unit 1024, and a weighting factor calculation unit 1025.

Note that in the optimization unit 102 shown in FIG. 3, the configuration differs from the conventional technology in that the weight coefficient calculation unit 1025 calculates weight coefficients based on the on-screen occupancy rate of the feature points derived for each camera and each feature point, as described below, and the parameter optimization unit 1023 applies weighting calculations to the cost function of the bundle adjustment using the calculated weight coefficients. The other configurations are the same as those of the optimization unit of the conventional object 3D reconstruction device.

The parameter initialization unit 1021 acquires the camera-correlated subject posture group data output from the camera-correlated unit 101. The parameter initialization unit 1021 initializes the camera parameters for each camera and the three-dimensional coordinates for each feature point based on the acquired camera-correlated subject posture group data. That is, the parameter initialization unit 1021 initializes the parameters of "number of cameras x number of camera parameters + number of feature points x three-dimensional coordinates". The parameter initialization unit 1021 outputs the initialized parameter group to the parameter optimization unit 1023.

The weighting factor initialization unit 1022 acquires the camera-matched subject posture group data output from the camera-matching unit 101. The weighting factor initialization unit 1022 initializes the weighting factor for each camera and each feature point based on the acquired camera-matched subject posture group data. In other words, the weighting factor initialization unit 1022 initializes the weighting factor of "number of cameras x number of feature points". The weighting factor initialization unit 1022 outputs the initialized weighting factor group to the parameter optimization unit 1023.

The parameter optimization unit 1023 acquires the camera-correlated subject posture group data output from the camera-corresponding unit 101, the parameter group output from the parameter initialization unit 1021, and the weighting coefficient group output from the weighting coefficient initialization unit 1022 or the weighting coefficient calculation unit 1025. The parameter optimization unit 1023 updates the parameter group based on the acquired camera-correlated subject posture group data and weighting coefficient group. The parameter optimization unit 1023 outputs the updated parameter group to the subject posture reprojection unit 1024 or an external device (not shown).

The subject pose reprojection unit 1024 acquires the updated parameter group output from the parameter optimization unit 1023. The subject pose reprojection unit 1024 projects the three-dimensional coordinates of each feature point onto the screen of each camera. The subject pose reprojection unit 1024 connects the mutually connected feature points, for example, with a line having a predetermined thickness, and generates an image of a silhouette for each camera and each subject. The subject pose reprojection unit 1024 outputs the generated camera and subject silhouette group to the weighting coefficient calculation unit 1025.

The weighting coefficient calculation unit 1025 acquires the group of silhouettes for each camera and each subject output from the subject orientation reprojection unit 1024. The weighting coefficient calculation unit 1025 calculates the screen occupancy rate of each feature point from the silhouette based on the acquired group of silhouettes for each camera and each subject, and calculates a weighting coefficient. The weighting coefficient calculation unit 1025 outputs the group of weighting coefficients calculated for each camera and each feature point (i.e., consisting of a weighting coefficient of the number of cameras x the number of feature points) to the parameter optimization unit 1023.

When the parameter optimization unit 1023 acquires the weighting coefficient group output from the weighting coefficient calculation unit 1025, it updates the parameter group based on the weighting coefficient group and the inter-camera associated subject posture group data output from the inter-camera association unit 101. The optimization unit 102 performs an iterative calculation to update the above-mentioned parameter group a predetermined number of times. Alternatively, the optimization unit 102 performs an iterative calculation to update the above-mentioned parameter group until the values of the parameter group converge. When the iterative calculation is completed, the parameter optimization unit 1023 sets the updated parameter group as an optimized parameter group, and outputs the parameter group to an external device (not shown).

More specifically, for example, during iterative calculations in the bundle adjustment, the object posture reprojection unit 1024 reprojects onto the screen of each camera using each camera parameter in the optimization process in each step of the iterative calculation and the three-dimensional coordinates of each feature point. The weighting factor calculation unit 1025 calculates the screen occupancy rate of the feature point relative to the camera resolution (for example, the ratio of the number of pixels of the silhouette of the feature point to the number of pixels of the camera). The weighting factor calculation unit 1025 derives a weighting factor based on the screen occupancy rate. The parameter optimization unit 1023 incorporates the weighting factor into a cost function to perform iterative calculations. As a result, the object 3D reconstruction device 100 in the first embodiment can suppress random noise that is likely to be superimposed on an object that appears small on the screen.

When reprojecting onto the screen of each camera, the subject orientation reprojection unit 1024 does not just project the feature points as points, but also takes into account the connection between the feature points (such as the human skeleton) as shown in FIG. 4, for example, and draws the lines between the connected feature points as line segments with width, circles, etc. The subject orientation reprojection unit 1024 draws a silhouette that fits the shape of the subject's feature points using line segments with width, circles, etc.

The weighting coefficient calculation unit 1025 generates a partial silhouette as a figure in which a line segment connecting a target feature point to another feature point connected to the target feature point is truncated at a certain internal division ratio, as shown in FIG. 5, for example. For example, FIG. 5 shows a partial silhouette in which a line segment connecting a target feature point to another feature point connected to the target feature point is truncated at the center of the line segment. The weighting coefficient calculation unit 1025 derives the screen occupancy rate of the feature point (silhouette) by calculating the number of pixels of the partial silhouette relative to the number of pixels of the camera (or the area ratio of the partial silhouette to the entire screen).

The smaller the feature point of the subject appears on the screen, the smaller the screen occupancy value becomes, and the larger the feature point of the subject appears on the screen, the larger the screen occupancy value becomes. The weighting coefficient calculation unit 1025 uses, for example, the value of the screen occupancy rate as the weighting coefficient. For example, the screen occupancy rate v _i,j can be expressed as the following formula (2).

Here, s _i,j represents the number of pixels (or area) of the partial silhouette, and S represents the number of pixels of the camera (or the area of the screen). The weighting coefficient w _i,j can be expressed as a function of the following equation (3).

The function w(v) can be expressed as the following equation (4).

In addition, the weighting coefficient calculation unit 1025 may limit the weighting coefficient when the screen occupancy rate is greater than a predetermined value. In this case, the function w(v) can be expressed, for example, as in the following equation (5).

The cost function for bundle adjustment in the first embodiment, incorporating the above weighting coefficients, is expressed as the following equation (6).

Here, W _i,j is a weighting coefficient. As in formula (1), in formula (2), the function F(p _i |f _j , r _j , d _j , R _j , t _j ) is a function for projecting the three-dimensional coordinate p _i of the feature point onto the screen of camera j. Here, f _j , r _j , d _j , R _j , and t _j are camera parameters of camera j. Specifically, f _j is the camera focal length, r _j is the camera resolution, d _j is the camera distortion, R _j is the camera rotation angle, and t _j is the camera position.

[Optimization section operation]
An example of the operation of the optimization unit 102 will be described in more detail below. Fig. 6 is a flowchart showing the operation of the optimization unit 102 of the object 3D reconstruction device 100 according to the first embodiment of the present invention. The operation shown in this flowchart is started when the inter-camera associated object posture group data output from the inter-camera association unit 101 is input to the optimization unit 102.

The parameter initialization unit 1021 acquires the camera-correlated subject posture group data output from the camera-correlated unit 101. The parameter initialization unit 1021 initializes the parameters of "number of cameras x number of camera parameters + number of feature points x three-dimensional coordinates" based on the acquired camera-correlated subject posture group data (step S201). The parameter initialization unit 1021 outputs the initialized parameter group to the parameter optimization unit 1023.

The weighting factor initialization unit 1022 acquires the camera-matched subject posture group data output from the camera-matching unit 101. Based on the acquired camera-matched subject posture group data, the weighting factor initialization unit 1022 initializes the "number of cameras x number of feature points weighting factor" by setting it to a value of, for example, 1.0 (step S202). Note that this initialization is intended to normalize the weighting factor group, and the initialized value is not limited to 1.0. The weighting factor initialization unit 1022 outputs the initialized weighting factor group to the parameter optimization unit 1023.

The parameter optimization unit 1023 acquires the camera-correlated subject posture group data output from the camera-corresponding unit 101, the parameter group output from the parameter initialization unit 1021, and the weighting coefficient group output from the weighting coefficient initialization unit 1022 or the weighting coefficient calculation unit 1025. The parameter optimization unit 1023 updates the parameter group based on the acquired camera-correlated subject posture group data and weighting coefficient group.

The parameter optimization unit 1023 then calculates the cost using a predetermined cost function (for example, the cost function expressed by the above-mentioned equation (6)) and updates the parameter set so as to minimize the cost. At this time, for example, the parameter optimization unit 1023 counts the number of times the parameter set is updated. Alternatively, for example, the parameter optimization unit 1023 calculates the maximum absolute error before and after updating the parameter set (step S203).

If the number of times the parameter group is updated is less than the predetermined number, or if the maximum value of the absolute error before and after the parameter group update is greater than the predetermined value (step S204, NO), the parameter optimization unit 1023 outputs the updated parameter group to the subject pose reprojection unit 1024. The subject pose reprojection unit 1024 acquires the optimized parameter group output from the parameter optimization unit 1023. The subject pose reprojection unit 1024 projects the three-dimensional coordinates of each feature point onto the (virtual) screen of each camera. The subject pose reprojection unit 1024 connects the connected feature points with, for example, a line having a predetermined thickness, and generates a silhouette for each camera and each subject (step S205). The subject pose reprojection unit 1024 outputs the generated camera and subject silhouette group to the weight coefficient calculation unit 1025.

The weighting coefficient calculation unit 1025 acquires the group of silhouettes by camera and by subject output from the subject orientation reprojection unit 1024. The weighting coefficient calculation unit 1025 calculates the screen occupancy rate of each feature point from a silhouette based on the acquired group of silhouettes by camera and by subject, and calculates a weighting coefficient by camera and by feature point based on the screen occupancy rate (step S206). The weighting coefficient calculation unit 1025 outputs the calculated group of weighting coefficients consisting of the number of cameras x the number of feature points to the parameter optimization unit 1023. Then, the process returns to step S203, and the update of the parameter group is repeated.

On the other hand, if the number of updates of the parameter group reaches a predetermined number, or if the maximum absolute error before and after the update of the parameter group is equal to or less than a predetermined value (step S204, YES), the parameter optimization unit 1023 outputs the optimized parameter group to an external device (not shown). This completes the operation of the optimization unit 102 shown in the flowchart of FIG. 6.

When calculating the weighting coefficient group in the process of step S206, the weighting coefficient calculation unit 1025 may refer to the absolute error before and after the update of the parameter group described above and adjust the weighting coefficient according to the value of the absolute error. For example, if the absolute error before and after the update of the parameter group is relatively large, it is possible that the optimization of each parameter is not sufficient. If the weighting coefficient is considered at such a stage where there is a high degree of uncertainty, there is a possibility that the coordinates will converge to unintended coordinates. Therefore, it is desirable for the weighting coefficient calculation unit 1025 to calculate the weighting coefficient group after the absolute error before and after the update of the parameter group has become relatively small (for example, after it has become equal to or greater than a predetermined value).

Note that while the bundle adjustment can simultaneously optimize both the camera parameters and the three-dimensional coordinates of the feature points, it may also be configured to optimize only the three-dimensional coordinates of the feature points. In other words, the values of the camera parameters of each camera may be known, and only the three-dimensional coordinates of the feature points may be optimized. In this case, the object 3D reconstruction device 100 acquires a parameter set for each camera from an external device (not shown). Then, the parameter initialization unit 1021 and the parameter optimization unit 1023 perform processing using the acquired known camera parameters.

Non-Patent Document 4 proposes a method of determining the maximum value of Part Confidence MAP (PCM) superimposed from multiple viewpoints as the coordinates of a person's joints. Even with this method, it is possible to configure a configuration that performs optimization taking into account the screen occupancy rate by changing the following equations (7) and (8) described in Non-Patent Document 4 to the following equations (9) and (10), respectively. Note that, unlike the previous equations, the following equation (9) uses the letter x as a variable representing a weighting coefficient.

As described above, the object 3D reconstruction device 100 in the first embodiment of the present invention reprojects onto the screen of each camera using the camera parameters and the 3D coordinates of each feature point at each step of the iterative calculation during the iterative calculation of the optimization process in the bundle adjustment. When reprojecting onto the screen of each camera, the object 3D reconstruction device 100 not only projects the feature points as points for each object, but also draws a silhouette that fits the shape of the object using line segments or circles with width, taking into account the connection relationship of the feature points, as shown in FIG. 4, for example. The object 3D reconstruction device 100 generates a partial silhouette that targets the center of a line segment from a target feature point to another feature point connected to the target feature point, as shown in FIG. 5, for example. The object 3D reconstruction device 100 calculates the screen occupancy rate based on the ratio of the number of pixels of the partial silhouette to the number of pixels of the camera (or the area ratio of the partial silhouette to the entire screen). The object 3D reconstruction device 100 derives a weighting coefficient based on the calculated screen occupancy rate, incorporates it into a predetermined cost function, and performs iterative calculation.

By being equipped with such a configuration, the object 3D reconstruction device 100 in the first embodiment of the present invention can suppress random noise that tends to be superimposed on an object that appears relatively small on the screen. Therefore, the object 3D reconstruction device 100 can further improve the estimation accuracy in the technology that estimates a group of parameters based on the two-dimensional coordinates of feature points detected from images captured by multiple cameras.

Second Embodiment
The object 3D reconstruction device 100a according to the second embodiment of the present invention will be described below. Note that the object 3D reconstruction device 100a described below is one example of an embodiment of a parameter estimation device of the present invention, and the present invention is not necessarily limited to the configuration of the object 3D reconstruction device 100a described below.

[Configuration of the object 3D reconstruction device]
The overall configuration of the object 3D reconstruction device 100a will be described below. FIG. 7 is a block diagram showing the overall configuration of the object 3D reconstruction device 100a in the second embodiment of the present invention. As shown in FIG. 7, the object 3D reconstruction device 100a includes an inter-camera correspondence unit 101 and an optimization unit 102a. In the object 3D reconstruction device 100a shown in FIG. 7, the optimization unit 102a has a different configuration from that of a conventional general object 3D reconstruction device, and the other configurations are the same as those of the conventional object 3D reconstruction device.

The camera correspondence unit 101 accepts input of subject posture group data output from an external device (not shown). The subject posture group data here refers to data generated from images of a human subject captured simultaneously from different positions by multiple cameras. The subject posture group data includes data indicating the positions of the joints, eyes, nose, ears, etc. of the human subject in each image, detected by two-dimensional posture estimation technology such as OpenPose and PersonLab.

The camera correspondence unit 101 recognizes the positions of the joints, eyes, nose, ears, etc. of the human subject in each image as feature points based on the input subject posture group data. The camera correspondence unit 101 matches feature points of the same part of the same person in images captured simultaneously by different cameras (for example, feature points indicating the positions of right elbows, or feature points indicating the positions of left knees, etc.). The camera correspondence unit 101 generates camera-matched subject posture group data in which these feature points are matched, and outputs the data to the optimization unit 102a.

The optimization unit 102a receives input of the camera-correlated subject posture group data output from the camera-correlated unit 101. The optimization unit 102a executes bundle adjustment using feature points based on the camera-correlated subject posture group data to optimize the parameter group. The parameter group here refers to the camera parameter values of each camera that captured the original image and the three-dimensional coordinates of each feature point. The camera parameters are, for example, the camera position, image capture direction, rotation angle, viewing angle, focal length, resolution, and distortion. The optimization unit 102a outputs the optimized parameter group to an external device (not shown). The optimized parameter group here refers to a parameter group optimized for the purpose of minimizing the mean square error of the difference (reprojection error) between the actual three-dimensional coordinates of the feature points and the estimated three-dimensional coordinates of the feature points.

The inter-camera correspondence unit 101 may be provided in an external device. In other words, the inter-camera corresponded subject posture group data may be generated in an external device (not shown), and the subject 3D reconstruction device 100a may be configured to acquire the inter-camera corresponded subject posture group data from the external device.

[Operation of the Object 3D Reconstruction Apparatus]
An example of the operation of the object 3D reconstruction device 100a will be described below. Fig. 8 is a flowchart showing the operation of the object 3D reconstruction device 100a in the second embodiment of the present invention. This flowchart starts when object posture group data output from an external device is input to the object 3D reconstruction device 100a.

The camera-to-camera correspondence unit 101 of the object 3D reconstruction device 100a accepts input of object posture group data output from an external device. The camera-to-camera correspondence unit 101 recognizes the positions of the joints, eyes, nose, ears, etc. of the person who is the object in each image as feature points based on the input object posture group data. The camera-to-camera correspondence unit 101 matches feature points of the same part of the same person in images captured simultaneously by different cameras (step S301). The camera-to-camera correspondence unit 101 generates camera-to-camera-correlated object posture group data and outputs it to the optimization unit 102a.

The optimization unit 102a receives input of the inter-camera-correlated subject posture group data output from the inter-camera correspondence unit 101. The optimization unit 102a executes a bundle adjustment using feature points based on the inter-camera-correlated subject posture group data to optimize the parameter group (step S302). The optimization unit 102a outputs the optimized parameter group to an external device. This completes the operation of the object 3D reconstruction device 100a shown in the flowchart of FIG. 8.

[Configuration of the optimization unit]
The configuration of the optimization unit 102a will be described in more detail below. Fig. 9 is a block diagram showing the functional configuration of the optimization unit 102a of the object 3D reconstruction device 100a in the second embodiment of the present invention. As shown in Fig. 9, the optimization unit 102a includes a parameter initialization unit 1021, a weighting factor initialization unit 1022, a parameter optimization unit 1023a, a weighting factor calculation unit 1025a, an invisible area including partial silhouette derivation unit 1026, and a visible area partial silhouette derivation unit 1027.

Note that the optimization unit 102a shown in FIG. 9 differs from the conventional technology in that the weight coefficient calculation unit 1025a calculates weight coefficients based on the visibility rates of the feature points derived for each camera and each feature point, as described below, and the parameter optimization unit 1023a applies weighting calculations to the cost function of the bundle adjustment using the calculated weight coefficients. The other configurations are the same as those of the optimization unit of the conventional object 3D reconstruction device.

The parameter initialization unit 1021 acquires the camera-correlated subject posture group data output from the camera-correlated unit 101. The parameter initialization unit 1021 initializes each camera parameter for each number of cameras and the three-dimensional coordinates for each feature point based on the acquired camera-correlated subject posture group data. In other words, the parameter initialization unit 1021 initializes the parameter "number of cameras x number of camera parameters + number of feature points x three-dimensional coordinates". The parameter initialization unit 1021 outputs the initialized parameter group to the parameter optimization unit 1023a.

The weighting factor initialization unit 1022 acquires the camera-matched subject posture group data output from the camera-matching unit 101. The weighting factor initialization unit 1022 initializes the weighting factor for each camera and each feature point based on the acquired camera-matched subject posture group data. In other words, the weighting factor initialization unit 1022 initializes the weighting factor of "number of cameras x number of feature points". The weighting factor initialization unit 1022 outputs the initialized weighting factor group to the parameter optimization unit 1023.

The parameter optimization unit 1023a acquires the camera-correlated subject posture group data output from the camera-corresponding unit 101, the parameter group output from the parameter initialization unit 1021, and the weighting coefficient group output from the weighting coefficient initialization unit 1022 or the weighting coefficient calculation unit 1025a. The parameter optimization unit 1023a updates the parameter group based on the acquired camera-correlated subject posture group data and weighting coefficient group. The parameter optimization unit 1023a outputs the updated parameter group to the invisible area including partial silhouette derivation unit 1026 and the visible area partial silhouette derivation unit 1027, or to an external device (not shown).

The invisible area including part silhouette derivation unit 1026 acquires the updated parameter group output from the parameter optimization unit 1023a. The invisible area including part silhouette derivation unit 1026 projects the three-dimensional coordinates of each feature point on the screen of each camera. The invisible area including part silhouette derivation unit 1026 connects the mutually connected feature points with, for example, a line having a predetermined thickness, and generates an image of a silhouette for each camera, subject, and feature point.

Then, the invisible-area-including partial silhouette derivation unit 1026 generates a partial silhouette, which is a figure obtained by cutting off the line segment connecting the target feature point and another feature point connected to the target feature point at a certain internal division ratio (for example, at the center of the line segment). The partial silhouette generated by the invisible-area-including partial silhouette derivation unit 1026 also includes parts of the subject that cannot be seen due to obstruction by, for example, a person appearing in the foreground. The invisible-area-including partial silhouette derivation unit 1026 outputs the generated partial silhouettes to the weighting coefficient calculation unit 1025a as "a group of invisible-area-including partial silhouettes by camera, subject, and feature point."

The visible area partial silhouette derivation unit 1027 acquires the optimized parameter group output from the parameter optimization unit 1023a. The visible area partial silhouette derivation unit 1027 reprojects the three-dimensional coordinates of each feature point onto the screen of each camera. At this time, the visible area partial silhouette derivation unit 1027 reprojects the feature points onto the screen of each camera in order from the most distant feature point, taking into account the depth. The visible area partial silhouette derivation unit 1027 then connects the feature points that are connected to each other with, for example, a line having a predetermined thickness, and generates silhouette images for each camera, subject, and feature point.

The visible area partial silhouette derivation unit 1027 then generates a partial silhouette, which is a figure obtained by truncating the line segment connecting the target feature point with another feature point connected to the target feature point at a certain internal division ratio (for example, at the center of the line segment). The partial silhouette generated by the visible area partial silhouette derivation unit 1027 does not include parts of the subject that cannot be seen due to obstruction by, for example, a person appearing in the foreground. The visible area partial silhouette derivation unit 1027 outputs the partial silhouette of only the visible area to the weighting coefficient calculation unit 1025a as a "group of visible area partial silhouettes by camera, subject, and feature point".

The weighting coefficient calculation unit 1025a acquires the group of invisible area inclusive partial silhouettes by camera, subject, and feature point output from the invisible area inclusive partial silhouette derivation unit 1026. The weighting coefficient calculation unit 1025a also acquires the group of visible area partial silhouettes by camera, subject, and feature point output from the visible area partial silhouette derivation unit 1027. The weighting coefficient calculation unit 1025a calculates the visibility of each feature point from the partial silhouette based on the acquired group of invisible area inclusive partial silhouettes by camera, subject, and feature point, and the partial silhouette based on the group of visible area partial silhouettes by camera, subject, and feature point, and calculates a weighting coefficient. The weighting coefficient calculation unit 1025a outputs the calculated weighting coefficient group for each camera and feature point (i.e., consisting of a weighting coefficient of the number of cameras x the number of feature points) to the parameter optimization unit 1023a.

When the parameter optimization unit 1023a acquires the weighting coefficient group output from the weighting coefficient calculation unit 1025a, it updates the parameter group based on the weighting coefficient group and the inter-camera associated subject posture group data output from the inter-camera association unit 101. The optimization unit 102a performs an iterative calculation to update the above-mentioned parameter group a predetermined number of times. Alternatively, the optimization unit 102a performs an iterative calculation to update the above-mentioned parameter group until the values of the parameter group converge. When the iterative calculation is completed, the parameter optimization unit 1023a sets the updated parameter group as an optimized parameter group and outputs the parameter group to an external device (not shown).

More specifically, for example, during iterative calculations in the bundle adjustment, the invisible area including partial silhouette derivation unit 1026 and the visible area partial silhouette derivation unit 1027 reproject onto the screen of each camera using each camera parameter of the optimization process in each step of the iterative calculation and the three-dimensional coordinates of each feature point. The weighting coefficient calculation unit 1025a calculates the visibility rate of the feature point (i.e., the ratio of the visible area partial silhouette to the invisible area including partial silhouette). The weighting coefficient calculation unit 1025a derives a weighting coefficient based on the visibility rate. The parameter optimization unit 1023a incorporates the weighting coefficient into a cost function and performs iterative calculations. As a result, the object 3D reconstruction device 100a in the second embodiment can suppress noise due to self-occlusion and occlusion by others.

When reprojecting onto the screen of each camera, the invisible area including partial silhouette derivation unit 1026 and the visible area partial silhouette derivation unit 1027 do not just project feature points as points, but also take into account the connection between feature points (such as the skeleton of a person) as shown in FIG. 4, for example, and draw the lines between connected feature points as line segments with width, circles, etc. The invisible area including partial silhouette derivation unit 1026 and the visible area partial silhouette derivation unit 1027 draw a silhouette that fits the shape of the subject using line segments with width, circles, etc.

The weighting coefficient calculation unit 1025a generates a partial silhouette in the form of a figure in which a line segment connecting a target feature point to another feature point connected to the target feature point is truncated at a certain internal division ratio, as shown in Fig. 10 and Fig. 11, for example. For example, Fig. 10 and Fig. 11 show a partial silhouette in the case where a line segment connecting a target feature point to another feature point connected to the target feature point is truncated at the center of the line segment.

Figure 10 is a diagram showing a case where self-occlusion occurs. Figure 10 shows an example of an image showing two people. In Figure 10, part of the partial silhouette of feature point j, which corresponds to the position of the right hip joint of the person appearing on the left side of the image, is occluded by the partial silhouette of feature point j, which corresponds to the position of the left thigh of that person.

Figure 11 is a diagram showing a case where occlusion by another person occurs. Figure 10 shows an example of an image showing two people. The person on the left side of the image is located closer to the viewer (from the position of the camera that captured the image) than the person on the right side of the image. Therefore, as shown in Figure 10, part of the partial silhouette of feature point j, which corresponds to the position of the right elbow of the person on the right, is occluded by the partial silhouette of feature point j, which corresponds to the position of the fist of the person on the left.

The weighting coefficient calculation unit 1025a derives the visibility rate of each feature point by calculating the ratio of the number of pixels in the partial silhouette of the visible area to the number of pixels in the partial silhouette including the invisible area (or the area ratio of the partial silhouette of the visible area to the partial silhouette including the invisible area).

The visibility rate becomes smaller as the ratio of the self-occluded area or the area occluded by another person becomes larger, and becomes larger as the ratio of the self-occluded area or the area occluded by another person becomes smaller. The weighting coefficient calculation unit 1025a uses, for example, the value of the visibility rate as the weighting coefficient. For example, the visibility rate v _i,j can be expressed as the following formula (11).

Here, s _i,j represents the number of pixels (or area) of the visible partial silhouette, and S represents the number of pixels (or area) of the entire partial silhouette including the invisible part. The weighting coefficient w _i,j can be expressed as a function of the following formula (12).

The function w(v) can be expressed as follows:

In addition, the weighting coefficient calculation unit 1025 may limit the weighting coefficient when the visibility rate is greater than a predetermined value. In this case, the function w(v) can be expressed, for example, as in the following equation (14).

The cost function for bundle adjustment in the second embodiment, incorporating the above weighting coefficients, is expressed, for example, as in the following equation (15).

Here, W _i,j is a weighting coefficient. As in equation (1), in equation (15), the function F(p _i |f _j , r _j , d _j , R _j , t _j ) is a function for projecting the three-dimensional coordinate p _i of the feature point onto the screen of camera j. Here, f _j , r _j , d _j , R _j , and t _j are camera parameters of camera j. Specifically, f _j is the camera focal length, r _j is the camera resolution, d _j is the camera distortion, R _j is the camera rotation angle, and t _j is the camera position.

[Optimization section operation]
An example of the operation of the optimization unit 102 will be described in more detail below. Fig. 12 is a flowchart showing the operation of the optimization unit 102a of the object 3D reconstruction device 100a in the second embodiment of the present invention. The operation shown in this flowchart is started when the inter-camera associated object posture group data output from the inter-camera association unit 101 is input to the optimization unit 102a.

The parameter initialization unit 1021 acquires the camera-correlated subject posture group data output from the camera-correlated unit 101. The parameter initialization unit 1021 initializes the parameters of "number of cameras x number of camera parameters + number of feature points x three-dimensional coordinates" based on the acquired camera-correlated subject posture group data (step S401). The parameter initialization unit 1021 outputs the initialized parameter group to the parameter optimization unit 1023a.

The weighting factor initialization unit 1022 acquires the camera-matched subject posture group data output from the camera-matching unit 101. Based on the acquired camera-matched subject posture group data, the weighting factor initialization unit 1022 initializes the "number of cameras x number of feature points weighting factor" by setting it to a value of, for example, 1.0 (step S402). Note that this initialization is intended to normalize the weighting factor group, and the initialized value is not limited to 1.0. The weighting factor initialization unit 1022 outputs the initialized weighting factor group to the parameter optimization unit 1023a.

The parameter optimization unit 1023a acquires the camera-correlated subject posture group data output from the camera-corresponding unit 101, the parameter group output from the parameter initialization unit 1021, and the weighting coefficient group output from the weighting coefficient initialization unit 1022 or the weighting coefficient calculation unit 1025a. The parameter optimization unit 1023a updates the parameter group based on the acquired camera-correlated subject posture group data and weighting coefficient group.

Here, the parameter optimization unit 1023a calculates the cost using a predetermined cost function (for example, the cost function expressed by the above-mentioned equation (15)) and updates the parameter group so as to minimize the cost. At this time, for example, the parameter optimization unit 1023a counts the number of times the parameter group is updated. Alternatively, for example, the parameter optimization unit 1023a calculates the maximum value of the absolute error before and after updating the parameter group (step S403).

If the number of times the parameter group has been updated is less than the predetermined number, or if the maximum absolute error before and after the parameter group update is greater than a predetermined value (step S404: NO), the parameter optimization unit 1023a outputs the updated parameter group to the invisible area including partial silhouette derivation unit 1026 and the visible area partial silhouette derivation unit 1027.

The invisible area including part silhouette derivation unit 1026 acquires the updated parameter set output from the parameter optimization unit 1023a. The invisible area including part silhouette derivation unit 1026 reprojects each feature point for each camera. The invisible area including part silhouette derivation unit 1026 projects the three-dimensional coordinates of each feature point on the screen of each camera based on the updated parameter set. The invisible area including part silhouette derivation unit 1026 connects the mutually connected feature points with, for example, a line having a predetermined thickness, and generates an image of a silhouette for each camera, each subject, and each feature point. Then, the invisible area including part silhouette derivation unit 1026 generates a figure as a partial silhouette by cutting off the line segment connecting the target feature point and another feature point connected to the target feature point at a certain internal division ratio (for example, at the center position of the line segment).

The visible area partial silhouette derivation unit 1027 acquires the updated parameter set output from the parameter optimization unit 1023a. The visible area partial silhouette derivation unit 1027 reprojects each feature point for each camera, starting from the most distant feature point, taking into account the depth. The visible area partial silhouette derivation unit 1027 projects the three-dimensional coordinates of each feature point onto the screen of each camera based on the updated parameter set. The visible area partial silhouette derivation unit 1027 then connects the mutually connected feature points with, for example, a line having a predetermined thickness, and generates an image of a silhouette for each camera, subject, and feature point. The visible area partial silhouette derivation unit 1027 then generates a figure as a partial silhouette by truncating the line segment connecting the target feature point and another feature point connected to the target feature point at a certain internal division ratio (for example, at the center position of the line segment) (step S405).

The invisible area including partial silhouette derivation unit 1026 outputs the generated partial silhouette including the invisible area to the weighting coefficient calculation unit 1025a as a "group of invisible area including partial silhouettes by camera, by subject, and by feature point." The visible area partial silhouette derivation unit 1027 outputs the generated partial silhouette of only the visible area to the weighting coefficient calculation unit 1025a as a "group of visible area including partial silhouettes by camera, by subject, and by feature point."

The weighting coefficient calculation unit 1025a acquires the group of invisible area inclusive partial silhouettes by camera, subject, and feature point output from the invisible area inclusive partial silhouette derivation unit 1026. The weighting coefficient calculation unit 1025a also acquires the group of visible area partial silhouettes by camera, subject, and feature point output from the visible area partial silhouette derivation unit 1027. The weighting coefficient calculation unit 1025a calculates the visibility rate of each feature point from the partial silhouette based on the group of invisible area inclusive partial silhouettes by camera, subject, and feature point acquired and the partial silhouette based on the group of visible area partial silhouettes by camera, subject, and feature point, and calculates the weighting coefficient by camera and feature point based on the visibility rate (step S406). The weighting coefficient calculation unit 1025a outputs the calculated weighting coefficient group consisting of the number of cameras x the number of feature points to the parameter optimization unit 1023a. Then, the process returns to step S403, and the update of the parameter group is repeated.

On the other hand, if the number of updates of the parameter group reaches a predetermined number, or if the maximum absolute error before and after the update of the parameter group is equal to or less than a predetermined value (step S404, YES), the parameter optimization unit 1023a outputs the optimized parameter group to an external device (not shown). This completes the operation of the optimization unit 102a shown in the flowchart of FIG. 12.

When calculating the weighting coefficient group in the process of step S406, the weighting coefficient calculation unit 1025a may refer to the absolute error before and after the update of the parameter group described above and adjust the weighting coefficient according to the value of the absolute error. For example, if the absolute error before and after the update of the parameter group is relatively large, it is possible that the optimization of each parameter is not sufficient. If the weighting coefficient is considered at such a stage where there is a high degree of uncertainty, there is a possibility that the coordinates will converge to unintended coordinates. Therefore, it is desirable for the weighting coefficient calculation unit 1025a to calculate the weighting coefficient group after the absolute error before and after the update of the parameter group has become relatively small (for example, after it has become equal to or greater than a predetermined value).

Note that while the bundle adjustment can simultaneously optimize both the camera parameters and the three-dimensional coordinates of the feature points, it may also be configured to optimize only the three-dimensional coordinates of the feature points. In other words, the values of the camera parameters of each camera may be known, and only the three-dimensional coordinates of the feature points may be optimized. In this case, the object three-dimensional reconstruction device 100a acquires a parameter set for each camera from an external device (not shown). Then, the parameter initialization unit 1021 and the parameter optimization unit 1023a perform processing using the acquired known camera parameters.

As described above, the object 3D reconstruction device 100a in the second embodiment of the present invention performs reprojection on the screen of each camera, sorted in consideration of depth, using each camera parameter and the 3D coordinates of each feature point in each step of the iterative calculation during the iterative calculation of the optimization process in the bundle adjustment. When performing reprojection on the screen of each camera, the object 3D reconstruction device 100a not only projects the feature points as points for each object, but also draws a silhouette that fits the shape of the object using a line segment or a circle with width, taking into consideration the connection relationship of the feature points, as shown in FIG. 4, for example. The object 3D reconstruction device 100a generates a partial silhouette that targets the center of a line segment from a target feature point to another feature point connected to the target feature point, as shown in FIG. 10 and FIG. 11, for example. The object 3D reconstruction device 100a calculates the visibility rate by the ratio of the number of pixels of the partial silhouette in the visible area to the number of pixels of the entire partial silhouette including the invisible area (or the area ratio of the partial silhouette in the visible area to the entire partial silhouette including the invisible area). The object 3D reconstruction device 100a derives a weighting factor based on the calculated visibility rate, incorporates it into a specified cost function, and performs an iterative calculation.

By being provided with such a configuration, the object 3D reconstruction device 100a in the second embodiment of the present invention can suppress noise due to self-occlusion and occlusion by others. Therefore, the object 3D reconstruction device 100a can further improve the estimation accuracy in a technique for estimating a group of parameters based on the two-dimensional coordinates of feature points detected from images captured by multiple cameras.

Third Embodiment
An object 3D reconstruction apparatus according to the third embodiment of the present invention will now be described.

The object 3D reconstruction device 100 in the first embodiment described above was configured to apply weighting calculations to the cost function of the bundle adjustment using weighting coefficients calculated based on the on-screen occupancy rate of the feature points derived for each camera and each feature point. The object 3D reconstruction device 100a in the second embodiment described above was configured to apply weighting calculations to the cost function of the bundle adjustment using weighting coefficients calculated based on the visibility rate of the feature points derived for each camera and each feature point. The object 3D reconstruction device may be configured to perform both weighting calculations based on the on-screen occupancy rate and weighting calculations based on the visibility rate.

For example, the subject 3D reconstruction device may be configured to perform a weighting calculation based on the screen occupancy rate, and then further perform a weighting calculation based on the visibility rate. Or, conversely, the subject 3D reconstruction device may be configured to perform a weighting calculation based on the screen occupancy rate, and then further perform a weighting calculation based on the visibility rate.

Alternatively, for example, the object 3D reconstruction device may be configured to perform a weighting calculation based on the screen occupancy rate and a weighting calculation based on the visibility rate in parallel, and perform an iterative calculation based on the results of both weighting calculations. For example, the object 3D reconstruction device may be configured to use the average value of the parameter values obtained by both weighting calculations in the next iterative calculation.

As described above, the object 3D reconstruction device in the third embodiment of the present invention performs reprojection on the screen of each camera, sorted in consideration of depth, using each camera parameter and the 3D coordinates of each feature point in each step of the iterative calculation during the iterative calculation of the optimization process in the bundle adjustment. When reprojecting on the screen of each camera, the object 3D reconstruction device not only projects the feature points as points for each object, but also draws a silhouette that fits the shape of the object using a line segment or a circle with width, taking into consideration the connection relationship of the feature points, as shown in FIG. 4, for example. The object 3D reconstruction device generates a partial silhouette that targets the center of the line segment from the target feature point to another feature point connected to the target feature point, as shown in FIG. 5, FIG. 10, and FIG. 11, for example. The object 3D reconstruction device calculates the screen occupancy rate by the ratio of the number of pixels of the partial silhouette to the number of pixels of the camera (or the area ratio of the partial silhouette to the entire screen). The object 3D reconstruction device also calculates the visibility rate based on the ratio of the number of pixels in the partial silhouette of the visible region to the number of pixels in the entire partial silhouette including the invisible region (or the area ratio of the partial silhouette of the visible region to the entire partial silhouette including the invisible region). The object 3D reconstruction device derives a weighting factor based on the calculated screen occupancy rate and visibility rate, incorporates it into a predetermined cost function, and performs an iterative calculation.

By being provided with such a configuration, the object 3D reconstruction device in the third embodiment of the present invention can suppress random noise that tends to be superimposed on objects that appear relatively small on the screen. Furthermore, by being provided with such a configuration, the object 3D reconstruction device in the third embodiment of the present invention can suppress noise due to self-occlusion and occlusion by others. Therefore, the object 3D reconstruction device can further improve the estimation accuracy in the technology that estimates a group of parameters based on the two-dimensional coordinates of feature points detected from images captured by multiple cameras.

Below is shown an example of a hardware configuration of the object 3D reconstruction device in the first to third embodiments configured by a computer and a software program. FIG. 13 is a block diagram showing an example of a hardware configuration of the object 3D reconstruction device in the first to third embodiments of the present invention. As shown in FIG. 13, the object 3D reconstruction device is configured with a CPU (Central Processing Unit) 10, memory 11, data storage unit 12, program storage device 13, and optimal parameter group output unit 15 connected by a bus.

The CPU 10 executes a program. The memory 11 is a storage medium such as a RAM (Random Access Memory) in which programs and data accessed by the CPU 10 are stored. The data storage unit 12 stores various input data. The data storage unit 12 may be a storage unit such as a disk device. The program storage unit 13 stores a three-dimensional reconstruction program 14 of an object, which is a software program that causes the CPU 10 to execute the three-dimensional reconstruction process of the object in each of the above-mentioned embodiments. The optimal parameter group output unit 15 outputs an optimized parameter group generated by the CPU 10 executing the three-dimensional reconstruction program 14 of the object loaded in the memory 11, for example, via a network. The optimal parameter group output unit 15 may be a storage unit that stores the optimized parameter group, such as a disk device.

According to the above-described embodiment, the parameter estimation device includes an acquisition unit, a weighting factor calculation unit, and a parameter estimation unit. For example, the parameter estimation device is the object 3D reconstruction device 100 in the embodiment, the acquisition unit is the parameter initialization unit 1021, the weighting factor initialization unit 1022, and the parameter optimization unit 1023 in the embodiment, the weighting factor calculation unit is the weighting factor calculation unit 1025 in the embodiment, and the parameter estimation unit is the parameter optimization unit 1023 in the embodiment.

The acquisition unit acquires subject posture group data indicating data in which two-dimensional coordinates of the same feature points of a subject are associated with each other in each image captured by multiple cameras of the same subject from different positions. For example, the multiple cameras are camera j and camera j+1 in the embodiment, and the subject posture group data is subject posture group data with inter-camera correspondence in the embodiment.

The weighting coefficient calculation unit calculates an on-screen occupancy rate indicating the ratio of the feature point area to the entire image area based on the subject posture group data acquired by the acquisition unit, and calculates a weighting coefficient based on the calculated on-screen occupancy rate. For example, the ratio of the feature point area to the entire image area is the number of pixels of the partial silhouette relative to the number of pixels of the camera in the embodiment (or the area ratio of the partial silhouette to the entire screen).

The parameter estimation unit estimates the values of parameters including the coordinate values of the three-dimensional coordinates of the feature points based on a calculation using the weighting coefficients calculated by the weighting coefficient calculation unit. For example, the calculation using the weighting coefficients is an iterative calculation using a cost function expressed by equation (6) in the embodiment, and the values of parameters including the coordinate values of the three-dimensional coordinates of the feature points are a parameter group consisting of the camera parameter values of each camera and the three-dimensional coordinates of each feature point in the embodiment.

In the parameter estimation device described above, the screen occupancy rate may be calculated from the ratio of the number of pixels of a silhouette figure drawn based on the positional relationship between a first feature point of a subject and a second feature point of the subject to the number of pixels of the image. For example, the first feature point is a target feature point in the embodiment, and the second feature point is another feature point connected to the target feature point in the embodiment.

In the above parameter estimation device, the silhouette figure may be a figure generated by adding width to a line segment in a range from the position of the first feature point to a position that is a predetermined internal division ratio of the line segment connecting the first feature point and the second feature point.

In the parameter estimation device described above, the weighting coefficient calculation unit may calculate the weighting coefficient by further using a visibility rate indicating the proportion of the area of the feature point that is not obscured by areas other than the feature point. For example, the feature point area is the area of the entire partial silhouette in the embodiment, and the area that is not obscured by areas other than the feature point is the area of the partial silhouette in the embodiment excluding the invisible area (i.e., the visible area).

In the parameter estimation device, the parameters may further include at least one of the external parameters of the camera and the internal parameters of the camera.

Furthermore, according to the above-described embodiment, the parameter estimation device includes an acquisition unit, a weighting factor calculation unit, and a parameter estimation unit. For example, the parameter estimation device is the object 3D reconstruction device 100a in the embodiment, the acquisition unit is the parameter initialization unit 1021, the weighting factor initialization unit 1022, and the parameter optimization unit 1023a in the embodiment, the weighting factor calculation unit is the weighting factor calculation unit 1025a in the embodiment, and the parameter estimation unit is the parameter optimization unit 1023a in the embodiment.

The acquisition unit acquires subject posture group data indicating data in which two-dimensional coordinates of the same feature points of a subject are associated with each other in each image captured by multiple cameras of the same subject from different positions. For example, the multiple cameras are camera j and camera j+1 in the embodiment, and the subject posture group data is subject posture group data with inter-camera correspondence in the embodiment.

The weighting coefficient calculation unit calculates a visibility rate indicating the proportion of the area of the feature points that is not obscured by areas other than the feature points based on the subject posture group data acquired by the acquisition unit, and calculates a weighting coefficient based on the calculated visibility rate. For example, the area of the feature points is the area of the entire partial silhouette in the embodiment, and the area that is not obscured by areas other than the feature points is the area of the partial silhouette in the embodiment excluding invisible areas (i.e., the visible area).

The parameter estimation unit estimates the values of parameters including the coordinate values of the three-dimensional coordinates of the feature points based on a calculation using the weighting coefficients calculated by the weighting coefficient calculation unit. For example, the calculation using the weighting coefficients is an iterative calculation using a cost function expressed by equation (15) in the embodiment, and the values of the parameters including the coordinate values of the three-dimensional coordinates of the feature points are a parameter group consisting of the camera parameter values of each camera and the three-dimensional coordinates of each feature point in the embodiment.

In the above parameter estimation device, the visibility may be calculated from the ratio of the number of pixels in an area of the silhouette figure that is not obscured by areas other than the feature points to the number of pixels in the silhouette figure that is drawn based on the positional relationship between a first feature point of the subject and a second feature point of the subject. For example, the first feature point is a target feature point in the embodiment, and the second feature point is another feature point that is connected to the target feature point in the embodiment.

In the above parameter estimation device, the silhouette figure may be a figure generated by adding width to a line segment in a range from the position of the first feature point to a position that is a predetermined internal division ratio of the line segment connecting the first feature point and the second feature point.

In addition, in the above parameter estimation device, the unoccluded area may be identified by detecting overlapping areas by reprojecting feature points on the camera screen in order from the most distant feature points, taking into account the depth.

In the above parameter estimation device, the weighting coefficient calculation unit may calculate the weighting coefficient by further using a screen occupancy rate indicating the ratio of the area of the feature point to the entire area of the image. For example, the ratio of the area of the feature point to the entire area of the image is the number of pixels of the partial silhouette relative to the number of pixels of the camera in the embodiment (or the area ratio of the partial silhouette to the entire screen).

In the parameter estimation device, the parameters may further include at least one of the external parameters of the camera and the internal parameters of the camera.

The object 3D reconstruction device in each of the above-mentioned embodiments may be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to realize the function. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs (Read Only Memory), and CD-ROMs, and storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may include a medium that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in such a case. Furthermore, the above-mentioned program may be a program for realizing part of the above-mentioned function, or may be a program that can realize the above-mentioned function in combination with a program already recorded in the computer system, or may be a program that is realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).

Although an embodiment of the present invention has been described above with reference to the drawings, it is clear that the above embodiment is merely an example of the present invention, and the specific configuration of the present invention is not limited to the above embodiment. Therefore, components may be added, omitted, replaced, or other design changes may be made within the technical scope that does not deviate from the spirit and gist of the present invention.

10...CPU, 11...memory, 12...data storage unit, 13...program storage device, 14...3D reconstruction program, 15...optimum parameter group output unit, 100 (100a)...object 3D reconstruction device, 101...camera correspondence unit, 102 (102a)...optimization unit, 1021...parameter initialization unit, 1022...weighting coefficient initialization unit, 1023 (1023a)...parameter optimization unit, 1024...object orientation reprojection unit, 1025 (1025a)...weighting coefficient calculation unit, 1026...invisible area including partial silhouette derivation unit, 1027...visible area partial silhouette derivation unit

Claims (8)

an acquisition unit that acquires subject posture group data indicating data in which two-dimensional coordinates of the same feature point of the same subject are associated with each other in each of images captured by a plurality of cameras from different positions of the same subject;
a weighting coefficient calculation unit that calculates a visibility ratio indicating a ratio of an area of the feature points that is not occluded by an area other than the feature points, based on the subject posture group data acquired by the acquisition unit, and calculates a weighting coefficient based on the calculated visibility ratio;
a parameter estimation unit that estimates values of parameters including coordinate values of three-dimensional coordinates of the feature points based on a calculation using the weighting coefficients calculated by the weighting coefficient calculation unit;
Equipped with
The visibility is calculated from a ratio of the number of pixels of a region in the silhouette figure that is not obscured by a region other than the feature points to the number of pixels of the silhouette figure that is drawn based on a positional relationship between a first feature point of the subject and a second feature point of the subject.
Parameter estimator. 2. The parameter estimation device according to claim 1, wherein the silhouette figure is a figure generated by adding width to a line segment in a range from a position of the first feature point to a position at a predetermined internal division ratio of a line segment connecting the first feature point and the second feature point. an acquisition unit that acquires subject posture group data indicating data in which two-dimensional coordinates of the same feature point of the same subject are associated with each other in each of images captured by a plurality of cameras from different positions of the same subject;
a weighting coefficient calculation unit that calculates a visibility ratio indicating a ratio of an area of the feature points that is not occluded by an area other than the feature points, based on the subject posture group data acquired by the acquisition unit, and calculates a weighting coefficient based on the calculated visibility ratio;
a parameter estimation unit that estimates values of parameters including coordinate values of three-dimensional coordinates of the feature points based on a calculation using the weighting coefficients calculated by the weighting coefficient calculation unit;
Equipped with
The unoccluded regions are identified by reprojecting the feature points on the screen of the camera in order of depth, starting from the furthest feature point, to detect overlapping regions.
Parameter estimator. The parameter estimation device according to claim 1 , wherein the weighting coefficient calculation unit calculates the weighting coefficient by further using a screen occupancy rate indicating a ratio of an area of the feature point to an entire area of the image. The parameter estimation device according to claim 1 , wherein the parameters further include at least one of an external parameter of the camera and an internal parameter of the camera. an acquisition step of acquiring subject posture group data indicating data in which two-dimensional coordinates of identical feature points of an identical subject are associated with each other in respective images captured by a plurality of cameras from different positions of the same subject;
a weighting coefficient calculation step of calculating a visibility ratio indicating a ratio of an area of the feature points that is not occluded by an area other than the feature points, based on the subject posture group data acquired in the acquisition step, and calculating a weighting coefficient based on the calculated visibility ratio;
a parameter estimation step of estimating values of parameters including coordinate values of three-dimensional coordinates of the feature points based on a calculation using the weighting coefficients calculated in the weighting coefficient calculation step;
having
The visibility is calculated from a ratio of the number of pixels of a region in the silhouette figure that is not obscured by a region other than the feature points to the number of pixels of the silhouette figure that is drawn based on a positional relationship between a first feature point of the subject and a second feature point of the subject.
Parameter estimation methods. an acquisition step of acquiring subject posture group data indicating data in which two-dimensional coordinates of identical feature points of an identical subject are associated with each other in respective images captured by a plurality of cameras from different positions of the same subject;
a weighting coefficient calculation step of calculating a visibility ratio indicating a ratio of an area of the feature points that is not occluded by an area other than the feature points, based on the subject posture group data acquired in the acquisition step, and calculating a weighting coefficient based on the calculated visibility ratio;
a parameter estimation step of estimating values of parameters including coordinate values of three-dimensional coordinates of the feature points based on a calculation using the weighting coefficients calculated in the weighting coefficient calculation step;
having
The unoccluded regions are identified by reprojecting the feature points on the screen of the camera in order of depth, starting from the furthest feature point, to detect overlapping regions.
Parameter estimation methods. A parameter estimation program for causing a computer to function as the parameter estimation device according to any one of claims 1 to 5 .

Images