JP7816752B2 - Image processing device, image processing method, and program - Google Patents

Description

This disclosure relates to an image processing device and image processing method that enable the construction of three-dimensional shapes from multiple images, as well as a program for implementing these.

In recent years, technology for constructing the three-dimensional shape of objects captured in images has been attracting attention. A well-known example of such technology is Structure from Motion (SfM). In SfM, feature values (e.g., SIFT features, SURF features) are calculated for each of multiple images of the object taken from different viewpoints, and feature points are extracted. The extracted feature points are then matched between images to extract matching feature point pairs. The geometric relationship between the feature point pairs is then calculated using, for example, robust estimation, and incorrect feature point pairs are eliminated.

Then, for each pair of feature points, a fundamental matrix is calculated based on these geometric relationships, and the geometric relationship for each pair of feature points is adjusted to minimize the difference between the calculated fundamental matrices. A three-dimensional shape (point cloud) is then constructed based on the adjusted geometric relationships.

Furthermore, the robust estimation described above does not completely remove incorrect feature point pairs, and the accuracy of the reconstructed three-dimensional shape is not high. Therefore, Patent Document 1 discloses technology to suppress the extraction of incorrect feature point pairs.

Specifically, the technology disclosed in Patent Document 1 first uses the conventional method described above to extract corresponding feature point pairs from multiple images, and then constructs a temporary three-dimensional shape using the extracted feature point pairs. Next, corresponding points or line segments are specified in each image of the object. The total number of points or line segments specified at this time is two or more.

Next, a numerical matrix indicating the geometric relationship between the specified points or lines is calculated, and the calculated numerical matrix is used to eliminate incorrect feature point pairs from the previously extracted feature point pairs. The three-dimensional shape is then reconstructed using only the feature point pairs that were not eliminated. Using the technology disclosed in Patent Document 1, the accuracy of the reconstructed three-dimensional shape can be improved.

International Publication No. 2019/065784

As mentioned above, with the technology disclosed in Patent Document 1, corresponding points or line segments are specified for each image, but the points or line segments are specified by the worker finding the corresponding points or line segments with the naked eye. As a result, specifying points or line segments is a significant burden for the worker.

One example of the objective of the present disclosure is to provide an image processing device, image processing method, and program that resolves the above-mentioned problems and can reduce the burden on workers when constructing a three-dimensional shape of an object using multiple images, while improving the accuracy of the three-dimensional shape.

In order to achieve the above object, an image processing device according to one aspect of the present disclosure includes:
a combination extraction unit that extracts combinations of corresponding feature points from each of a plurality of images of the object;
a temporary three-dimensional shape constructing unit that constructs a temporary three-dimensional shape of the object, the temporary three-dimensional shape being configured as a point cloud of the feature points, using a combination of the extracted feature points;
a feature point identification unit that, when a position on the object is designated in one of the plurality of images, identifies the corresponding feature point in accordance with the designated position, and further identifies the feature point corresponding to the identified feature point in another of the images in which a position is not designated;
a feature point extraction unit that selects, from the identified feature points in the image in which a position on the object is specified, any feature point that satisfies a set condition, and extracts, in the other image, the feature point that corresponds to the selected feature point;
a matrix calculation unit that calculates a numerical matrix that represents a geometric relationship between the selected feature points and the extracted feature points;
Using the numerical matrix, a combination of feature points having a contradictory geometric relationship is identified from among the extracted combinations of feature points, and
a three-dimensional shape constructing unit that constructs a three-dimensional shape of the object using a combination of the feature points other than the identified combination of the feature points;
The present invention is characterized by the following features.

In order to achieve the above object, an image processing method according to one aspect of the present disclosure includes:
a combination extraction step of extracting combinations of corresponding feature points from each of a plurality of images of the object;
a temporary three-dimensional shape construction step of constructing a temporary three-dimensional shape of the object, the temporary three-dimensional shape being configured as a point cloud of the feature points, using a combination of the extracted feature points;
a feature point identifying step of, when a position on the object is specified in one of the plurality of images, identifying the corresponding feature point in accordance with the specified position, and further identifying the feature point corresponding to the identified feature point in another of the images in which a position is not specified;
a feature point extraction step of selecting, from the feature points identified in the image in which a position on the object is specified, any feature point that satisfies a set condition, and extracting, in the other image, the feature point corresponding to the selected feature point;
a matrix calculation step of calculating a numerical matrix that represents a geometric relationship between the selected feature points and the extracted feature points;
Using the numerical matrix, a combination of feature points having a contradictory geometric relationship is identified from among the extracted combinations of feature points, and
a three-dimensional shape construction step of constructing a three-dimensional shape of the object using a combination of the feature points other than the identified combination of the feature points;
The present invention is characterized by having the following:

Furthermore, in order to achieve the above object, a program according to one aspect of the present disclosure includes:
On the computer,
a combination extraction step of extracting combinations of corresponding feature points from each of a plurality of images of the object;
a temporary three-dimensional shape construction step of constructing a temporary three-dimensional shape of the object, the temporary three-dimensional shape being configured as a point cloud of the feature points, using a combination of the extracted feature points;
a feature point identifying step of, when a position on the object is specified in one of the plurality of images, identifying the corresponding feature point in accordance with the specified position, and further identifying the feature point corresponding to the identified feature point in another of the images in which a position is not specified;
a feature point extraction step of selecting, from the feature points identified in the image in which a position on the object is specified, any feature point that satisfies a set condition, and extracting, in the other image, the feature point corresponding to the selected feature point;
a matrix calculation step of calculating a numerical matrix that represents a geometric relationship between the selected feature points and the extracted feature points;
Using the numerical matrix, a combination of feature points having a contradictory geometric relationship is identified from among the extracted combinations of feature points, and
a three-dimensional shape construction step of constructing a three-dimensional shape of the object using a combination of the feature points other than the identified combination of the feature points;
The method is characterized in that:

As described above, according to the present disclosure, when constructing a three-dimensional shape of an object using multiple images, it is possible to improve the accuracy of the three-dimensional shape while reducing the burden on the worker.

FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus according to the first embodiment. FIG. 2 is a block diagram showing a specific configuration of the image processing device according to the first embodiment. FIG. 3 is a diagram showing an example of a plurality of images to be processed. FIG. 4 is a diagram showing an example of a plurality of images of an object displayed on a screen. FIG. 5 is a diagram illustrating a process for identifying a combination of feature points having a contradictory geometric relationship. FIG. 6 is a diagram showing an example of a pair of images from which combinations of feature points have been extracted. FIG. 7 is a diagram showing an example of the three-dimensional coordinates and rotation matrix of the camera of the initial pair of images, which are obtained from the camera matrix. FIG. 8 is a diagram showing an example of a combination of an image newly selected after the selection of the initial pair of images and feature points extracted therefrom. FIG. 9 is a diagram illustrating the calculation process of the reprojection error of a feature point in an object. FIG. 10 is a flowchart showing the operation of the image processing device according to the first embodiment. FIG. 11 is an explanatory diagram for explaining the functions of the image processing device according to the second embodiment. FIG. 12 is a flowchart showing the operation of the image processing device according to the second embodiment. FIG. 13 is a block diagram showing an example of a computer that realizes the image processing device according to the first and second embodiments.

(Embodiment 1)
The image processing device according to the first embodiment will be described below with reference to FIGS.

[Device configuration]
First, the schematic configuration of the image processing device according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the schematic configuration of the image processing device according to the first embodiment.

The image processing device 10 in embodiment 1 shown in Figure 1 is a device that constructs a three-dimensional shape of an object from multiple images of the object. As shown in Figure 1, the image processing device 10 includes a combination extraction unit 11, a temporary three-dimensional shape construction unit 12, a feature point identification unit 13, a feature point extraction unit 14, a matrix calculation unit 15, and a three-dimensional shape construction unit 16.

The combination extraction unit 11 extracts corresponding combinations of feature points from each of multiple images of the object. The temporary 3D shape construction unit 12 uses the extracted combinations of feature points to construct a temporary 3D shape of the object composed of a point cloud of feature points.

When a position on an object is specified in one of the multiple images, the feature point identification unit 13 identifies a corresponding feature point in accordance with the specified position. The feature point identification unit 13 also identifies a feature point corresponding to the identified feature point in another image in which a position is not specified.

The feature point extraction unit 14 selects any feature point that satisfies a set condition from among the identified feature points in an image in which the position on the object is specified. The feature point extraction unit 14 also extracts feature points in another image that correspond to the selected feature point. The matrix calculation unit 15 calculates a numerical matrix that represents the geometric relationship between the selected feature point and the extracted feature point.

The three-dimensional shape construction unit 16 uses a numerical matrix to identify, from among the extracted combinations of feature points, combinations of feature points that have contradictory geometric relationships between the feature points, and constructs the three-dimensional shape of the object using combinations of feature points other than the identified combinations of feature points.

In this way, in embodiment 1, after the three-dimensional shape of the object is constructed, a position is specified on one image and feature points around the specified position are identified in order to improve the accuracy of the three-dimensional shape. Furthermore, when one of these is selected, corresponding feature points are extracted on the remaining images. The three-dimensional shape of the object is then reconstructed using the selected or extracted feature points.

Accordingly, according to embodiment 1, when constructing a three-dimensional shape of an object using multiple images, the accuracy of the three-dimensional shape can be improved. Furthermore, in embodiment 1, when constructing a three-dimensional shape, the user does not need to precisely specify points or line segments on the image, but only needs to specify a general position on the image, thereby reducing the burden on the user compared to conventional methods.

Next, the configuration and functions of the image processing device in embodiment 1 will be specifically explained using Figures 2 to 9. Figure 2 is a block diagram showing the specific configuration of the image processing device in embodiment 1.

As shown in FIG. 2, in embodiment 1, the image processing device 10 includes a combination extraction unit 11, a temporary 3D shape construction unit 12, a feature point identification unit 13, a feature point extraction unit 14, a matrix calculation unit 15, and a 3D shape construction unit 16, as well as an image acquisition unit 17, a filtering unit 18, an input acceptance unit 19, and a display unit 20. In FIG. 2, 21 is a display device.

The image acquisition unit 17 acquires multiple image data of an object from an external device, such as an imaging device, a terminal device, or a storage device that stores image data. Figure 3 shows an example of multiple images to be processed. In the example of Figure 3, four images are shown, but the number of images to be processed in embodiment 1 is not particularly limited. In Figure 3, 30 indicates the object.

In the first embodiment, the combination extraction unit 11 first identifies feature points for each image data by calculating, for example, SIFT features or SURF features from the image. Then, the combination extraction unit 11 extracts corresponding feature points between the images as combinations of corresponding feature points. In the example of Figure 3, the circled portion indicates one of the feature points of the object 30. Hereinafter, multiple images from which combinations of corresponding feature points have been extracted will be referred to as a "group image."

The filtering unit 18 calculates the geometric relationship between feature points for each combination of feature points, identifies incorrect combinations of feature points based on the calculation results, and eliminates the identified combinations of feature points. The filtering process by the filtering unit 18 is performed using robust estimation, as in conventional SfM. In embodiment 1, in addition to the filtering process by the 3D shape construction unit 16 (hereinafter referred to as the "second filtering process") described below, the filtering unit 18 also performs a filtering process (hereinafter referred to as the "first filtering process"), which more reliably eliminates incorrect combinations of feature points.

In embodiment 1, the temporary 3D shape construction unit 12 calculates temporary 3D coordinates using a combination of extracted feature points, and then uses the calculated temporary 3D coordinates to construct a temporary 3D shape of the object composed of a point group of feature points.

The three-dimensional coordinate calculation process and three-dimensional shape construction process by the temporary three-dimensional shape construction unit 12 are performed in the same manner as the three-dimensional coordinate calculation process and three-dimensional shape construction process by the three-dimensional shape construction unit 16, which will be described later. The only difference between the two is whether or not a second filtering process is performed on the combination of feature points to be processed. Therefore, details of the three-dimensional coordinate calculation process and three-dimensional shape construction process by the temporary three-dimensional shape construction unit 12 will be substituted for the detailed description of the three-dimensional coordinate calculation process and three-dimensional shape construction process by the three-dimensional shape construction unit 16, which will be described later.

The input receiving unit 19 receives a position specification on the image to be processed. The position specification is performed, for example, by the user on the image displayed on the screen of the display device 21 using an input device such as a touch panel or mouse. The position specification may also be performed by the user via an external terminal device.

Once the three-dimensional shape of the object has been constructed by the three-dimensional shape construction unit 16, the display unit 20 displays the constructed three-dimensional shape as a two-dimensional image on the screen of the display device 21. Specifically, the display unit 20 generates image data for displaying the constructed three-dimensional shape on a two-dimensional screen, and outputs the generated image data to the display device 21.

Furthermore, when image data of multiple images is acquired by the image acquisition unit 17, the display unit 20 can also display the images of the acquired image data on the screen of the display device 21. In this case, as shown in FIG. 4, the user can specify a position on the object in one of the images displayed on the screen via the input acceptance unit 19. FIG. 4 is a diagram showing an example of multiple images of an object displayed on the screen. In the example of FIG. 4, the user specifies a position on the object 30 with a cursor (arrow) 31 displayed on the screen.

In embodiment 1, as shown in FIG. 4, when a user specifies a position on an object 30 using a cursor 31, the feature point identification unit 13 identifies a feature point 32 corresponding to the specified position. Specifically, the feature point identification unit 13 sets, for example, a circular area with a radius r centered on the specified position in the image where the position is specified, and identifies feature points that exist in the set area. Note that the radius r of the circular area is set as appropriate.

The feature point identification unit 13 also identifies feature points corresponding to the identified feature points in images other than the image in which the position is specified. Note that the feature points to be identified are feature points that have already been extracted from the image by the combination extraction unit 11 and further filtered by the filtering unit 18.

Furthermore, the feature point identification unit 13 presents the identified feature points on the object 30 on the screen along with each of the multiple images. Specifically, the feature point identification unit 13 sends data indicating the coordinates of the identified feature points for each image to the display unit 20. Upon receiving the data, the display unit 20 generates image data in which marks indicating the feature points (● in Figure 4) are superimposed on each image, and outputs the generated image data to the display device 21. As a result, the identified feature points 32 and each image (images 1 to 4) are displayed on the screen of the display device 21, as shown in Figure 4.

As shown in Figure 4, when feature points around the specified position are displayed on the screen, the user can use the input device to specify one of the feature points on the image where the position was specified. In this case, the input receiving unit 19 notifies the feature point identification unit 13 of the specified feature point.

In the first embodiment, when a feature point is specified on an image in which a position on the object 30 is specified on the screen, the feature point extraction unit 14 selects the specified feature point. In the first embodiment, the feature point specified by the user corresponds to the "feature point that satisfies the set conditions" described above. Then, as described above, the feature point extraction unit 14 extracts feature points corresponding to the selected feature point in images other than the image in which a position on the object 30 is specified.

In embodiment 1, the matrix calculation unit 15 calculates the fundamental matrix (see reference) shown in equation 1 below as a numerical matrix that represents the geometric relationship between feature points selected on an image whose position is specified and corresponding feature points extracted from another image.

References: Richard Hartley and Andrew Zisserman, “Multiple View Geometry in Computer Vision Second Edition”, Cambridge University Press, March 2004.

In embodiment 1, the three-dimensional shape construction unit 16 first uses the numerical matrix (fundamental matrix) calculated by the matrix calculation unit 15 to identify combinations of feature points that have contradictory geometric relationships from the combinations of feature points extracted by the combination extraction unit 11 (excluding those excluded by the filtering unit 18).

The process of identifying a combination of feature points having a contradictory geometric relationship performed by the three-dimensional shape constructing unit 16 will be specifically described with reference to Fig. 5. Fig. 5 is a diagram illustrating the process of identifying a combination of feature points having a contradictory geometric relationship. In the example of Fig. 5, the combination of feature point p _i and feature point p' _i is the subject of determination as to whether it is incorrect.

In Fig. 5, E indicates the epipolar plane, _Oi indicates the center position of the camera for one image, and _O'i indicates the center position of the camera for the other image. _ei indicates the intersection of one image with the epipolar plane, and _e'i indicates the intersection of the other image with the epipolar plane. Furthermore, in Fig. 5, the parallelograms shown on the left and right each indicate the frame of an image.

5, point P in one image is a point indicating a specified position (specified point), and point P' in the other image is a point corresponding to the specified point (corresponding point). The epipolar line intersecting with specified point P is defined as epipolar line _ln , and the epipolar line intersecting with corresponding point P' is defined as epipolar line l' _n .

The three-dimensional shape constructing unit 16 calculates the epipolar line l' ₁ corresponding to the epipolar line _l1 on which the feature point p _i exists, using the relationship shown in the following equation 2 for the epipolar line. In equation 2, F is the fundamental matrix shown in equation 1. x is the coordinate of the epipolar line l.

The three-dimensional shape construction unit 16 then determines whether the distance d between the calculated epipolar line l' ₁ and the feature point _p'i is equal to or greater than a threshold, and if the distance d is equal to or greater than the threshold, determines that the combination of feature point p _i and feature point _p'i is incorrect. In this case, the three-dimensional shape construction unit 16 identifies the combination of feature point p _i and feature point _p'i as a combination of feature points with a contradictory geometric relationship.

Furthermore, when the three-dimensional shape construction unit 16 identifies a combination of feature points with a contradictory geometric relationship, it calculates three-dimensional coordinates for constructing a three-dimensional shape using the combination of feature points that was not identified.

The calculation process of three-dimensional coordinates by the three-dimensional shape construction unit 16 will be described in detail using Figures 6 to 8. Figure 6 is a diagram showing an example of a pair of images from which combinations of feature points have been extracted. Figure 7 is a diagram showing an example of the three-dimensional coordinates and rotation matrix of the camera of the initial pair of images, calculated from the camera matrix. Figure 8 is a diagram showing an example of an image newly selected after the selection of the initial pair of images and a combination of feature points extracted from it.

As shown in FIG. 6 , the 3D shape constructing unit 16 first selects images 51 and 52 as a set of paired images (initial paired images). In this case, feature points (m ₁ to m ₅ ) extracted from image 51 correspond to feature points (m' ₁ to m' ₅ ) extracted from image 52. m ₁ and m' 1 , m ₂ and m' ₂ , m ₃ and m' ₃ , m ₄ and m' ₄ , and m ₅ and _{m' 5 are} combinations of feature points (hereinafter also referred to as "feature point pairs"). In the example of FIG. 6 , image 51 is captured by camera 41, and image 52 is captured by camera 42. In FIG. 6 , M (M ₁ to M ₅ ) are the 3D coordinates on the object 30 corresponding to each feature point.

Next, the three-dimensional shape constructing unit 16 calculates the camera matrix Q of the camera 41 that captured the image 51 and the camera matrix Q' of the camera 42 that captured the image 52, using the feature point pairs (m ₁ to m ₅ , m' ₁ to m' ₅ ) extracted from each of the initial pair images. Furthermore, when the position of the camera 41 is taken as the origin, the camera matrix Q and the camera matrix Q' can be expressed by the following equations 3 and 4, respectively.

In the above equation 3, I is the rotation matrix of the camera 41. As shown in Fig. 7, the position of the camera 41 is the origin, so I = (1, 1, 1). Also, in the above equation 4, R is the rotation matrix of the camera 42 (R = ( _Rx , _Ry , _Rz )). t is the translation matrix, which corresponds to the three-dimensional coordinates of the position of the camera 42 (t = ( _tx , _ty , _tz )).

Therefore, in this case, the rotation matrix R and the translation matrix t can be calculated by performing inverse calculations from the camera matrix Q and the camera matrix Q'. Specifically, the three-dimensional shape constructor 16 calculates the rotation matrix R and the translation matrix t by solving the equations shown in the following equations 5 to 7 using the coordinates of each feature point. In equations 5 to 7, m is the coordinate on image 51 obtained by normalizing m (m ₁ to m ₅ ). Similarly, m' is the coordinate on image 52 obtained by normalizing m'(m' ₁ to m' ₅ ). E is the essential matrix, and K is the camera calibration matrix.

The calibration matrix K can be calculated from the following equations 8 and 9. Note that c _x and c _y are the center coordinates of the camera.

Next, the three-dimensional shape construction unit 16 calculates the three-dimensional coordinates M (M ₁ to M ₅ ) of the extracted feature points using the three-dimensional coordinates of the positions of each camera and the rotation matrix. Specifically, the three-dimensional shape construction unit 16 calculates the three-dimensional coordinates M by solving the following equation 10. Furthermore, the matrix A in equation 10 is expressed by equation 11. In equation 11, q ^iT is a row of the camera matrix Q, and q' ^iT is a row of the camera matrix Q'.

Next, as shown in FIG. 8, the three-dimensional shape construction unit 16 newly selects one image 53 from among the images from which feature points have been extracted, other than the initial pair of images, and creates a new pair of images consisting of the newly selected image 53 and one of the initial pair of images. Image 53 was captured by camera 43.

Then, the three-dimensional shape construction unit 16 identifies feature points (m'' ₁ to m'' ₃ ) of image 53 that correspond to the feature points of image 52, and sets the feature points of image 52 and the feature points of image 53 as feature point pairs. Then, the three-dimensional shape construction unit 16 calculates a camera matrix Qn of the camera 43 that captured image 53. The camera matrix Qn can be expressed by the following equation 12.

Specifically, the 3D shape construction unit 16 calculates Rn and tn of the camera matrix Qn of the camera 43 by solving the equation shown in Equation 13 below using the identified feature points of the image 53.

In Expression 13, M _i is the three-dimensional coordinate of a feature point in newly selected image 53 that is common to image 52. M _i is the normalized coordinate of the feature point in newly selected image 53. d i indicates the distance between camera 43 that captured image 53 and M _i as shown in Expression 14 below.

Next, the three-dimensional shape construction unit 16 calculates the three-dimensional coordinates M i of the identified feature points (m'' ₁ to m'' ₃ ) of the image 53 using Qn and tn of the calculated camera matrix _Qn of the camera 43. In this case as well, the three-dimensional shape construction unit 16 calculates the three-dimensional coordinates M (M ₁ to M ₃ ) of the feature points by solving the above equation 10. Through the above processing, the three-dimensional shape construction unit 16 can calculate the three-dimensional coordinates of the object.

Furthermore, in embodiment 1, the three-dimensional shape construction unit 16 can use the reprojection error to determine whether a feature point whose three-dimensional coordinates have been calculated is appropriate. Specifically, for one combination of feature points, the three-dimensional shape construction unit 16 reprojects the three-dimensional coordinates obtained from the combination onto the original image using the camera matrix corresponding to the image from which the feature points were extracted, and compares the two-dimensional coordinates of the projected position with the two-dimensional coordinates of the position at the time of extraction. In embodiment 1, the three-dimensional shape construction unit 16 then calculates the difference (reprojection error) between the former two-dimensional coordinates and the latter two-dimensional coordinates as the comparison result, and determines that the target feature point is appropriate if the calculated reprojection error is equal to or less than a threshold. On the other hand, the three-dimensional shape construction unit 16 determines that the target feature point is inappropriate if the calculated reprojection error exceeds the threshold.

In addition, in embodiment 1, for each combination of feature points for which three-dimensional coordinates have been calculated, the three-dimensional shape construction unit 16 can select all or some of the feature points that make up this combination and determine whether each of the selected feature points is appropriate. For example, if a combination of feature points is made up of three feature points, the three-dimensional shape construction unit 16 can determine whether each of the three feature points is appropriate, or can select only two of the feature points and determine whether each of the two selected feature points is appropriate.

The process of determining the suitability of feature points by the three-dimensional shape construction unit 16 will be specifically described using Fig. 9. Fig. 9 is a diagram illustrating the process of calculating the reprojection error of feature points in an object. The example of Fig. 9 shows an example of reprojecting one of the feature points on the image 52 shown in Fig. 6. Furthermore, in Fig. 9, the three-dimensional coordinates corresponding to the feature point, i.e., the three-dimensional coordinates in the world coordinate system, are defined as ( _Xw , _Yw , _Zw ), and the two-dimensional coordinates of the position at the time of extraction of the feature point are defined as ( _xf , _yf ). Furthermore, the coordinates of the feature point in the camera coordinate system are defined as ( _Xc , _Yc , _Zc ).

First, the three-dimensional shape construction unit 16 calculates the coordinates ( _XC , _YC , _ZC ) of the feature points in the camera coordinate system from the camera matrix Q' (=[R|t]) and the three-dimensional coordinates ( _XW , _YW , _ZW ) of the feature points using the following equation 15.

Next, the three-dimensional shape constructing unit 16 normalizes the coordinates ( _Xc , _Yc , _Zc ) of the feature points in the camera coordinate system using the following equation (16).

Next, the three-dimensional shape construction unit 16 applies the coordinates normalized by the above equation 16 and the internal parameters of the camera (in the example of FIG. 9, the focal length f of the camera 42 and the coordinates (c _x , c _y ) of the image center position) to the following equation 17, and calculates the two-dimensional coordinates (x _p , y _p ) when the feature points are reprojected onto the image 52.

Next, the three-dimensional shape construction unit 16 uses the following equation 18 to calculate the difference df between the two-dimensional coordinates ( _xp , _yp ) after reprojection calculated from the above equation 17 and the two-dimensional coordinates ( _xf , _yf ) of the position at the time of extraction of the feature point.

Then, if the calculated difference df is equal to or less than the threshold value G, the three-dimensional shape construction unit 16 determines that the target feature point is appropriate; if the calculated difference df exceeds the threshold value G, it determines that the target feature point is inappropriate.

After the judgment, the three-dimensional shape construction unit 16 excludes the three-dimensional coordinates corresponding to the feature points judged to be inappropriate, and all feature points corresponding to these three-dimensional coordinates. In embodiment 1, the three-dimensional shape construction unit 16 can also exclude only the feature points judged to be inappropriate, and not exclude the three-dimensional coordinates corresponding to the feature points judged to be inappropriate, or any feature points among the combinations of feature points corresponding to these three-dimensional coordinates other than those judged to be inappropriate. However, in this case, when the three-dimensional shape construction unit 16 has completed the judgment of suitability for all feature points, it excludes all three-dimensional coordinates that have fewer than two feature points corresponding to each of the three-dimensional coordinates.

Then, the 3D shape construction unit 16 recalculates the 3D coordinates using the remaining feature points, and constructs point cloud data of the object using the 3D coordinates of each feature point finally obtained, which is used as the 3D shape of the object.

In addition, in the above example, the three-dimensional shape construction unit 16 constructs a three-dimensional shape from scratch again using a combination of feature points, but the embodiment is not limited to this example. For example, the three-dimensional shape construction unit 16 can also construct a final three-dimensional shape by adjusting the coordinates of each feature point that constitutes the provisional three-dimensional shape by performing bundle adjustment using the numerical matrix calculated by the matrix calculation unit 15. Note that even in this mode, combinations of feature points that have contradictory geometric relationships between them are essentially identified and excluded.

[Device operation]
Next, the operation of the image processing device 10 in embodiment 1 will be described with reference to FIG. 10. FIG. 10 is a flow diagram showing the operation of the image processing device in embodiment 1. In the following description, reference will be made to FIGS. 1 to 9 as appropriate. In embodiment 1, an image processing method is implemented by operating the image processing device 10. Therefore, the description of the image processing method in embodiment 1 will be replaced by the following description of the operation of the image processing device 10.

As shown in FIG. 10, first, the image acquisition unit 17 acquires multiple image data of an object from an external device (step A1). Furthermore, when step A1 is executed, the display unit 20 displays images of each acquired image data on the display screen of the display device 21.

Next, the combination extraction unit 11 identifies feature points from the image for each image data acquired in step A1, and further extracts corresponding feature points between the images as combinations of corresponding feature points (step A2).

Next, the filtering unit 18 calculates the geometric relationship between the feature points for each combination of feature points extracted in step A2, identifies incorrect combinations of feature points based on the calculation results, and eliminates the identified combinations of feature points (step A3).

Next, the temporary 3D shape construction unit 12 calculates temporary 3D coordinates using the combination of feature points after processing in step A3, and constructs a temporary 3D shape of the object composed of a point group of feature points using the calculated temporary 3D coordinates (step A4).

Next, when the user specifies the position of the object on the image displayed on the screen of the display device 21, the input acceptance unit 19 accepts the position specification (step A5).

Next, the feature point identification unit 13 identifies feature points corresponding to the positions accepted in step A5, and further identifies feature points corresponding to the previously identified feature points in images other than the image in which the positions are specified (step A6). Furthermore, the feature point identification unit 13 displays the identified feature points on the display screen of the display device 21 via the display unit 20 (step A7).

Next, when a feature point is specified on the image whose position is specified on the screen of the display device 21, the feature point extraction unit 14 selects the specified feature point and further extracts feature points corresponding to the selected feature point on other images (step A8).

Next, the matrix calculation unit 15 calculates a numerical matrix that represents the geometric relationship between the feature points selected on the image whose position is specified and the feature points extracted from other images (step A9).

Next, the 3D shape construction unit 16 uses the numerical matrix calculated in step A9 to identify combinations of feature points that have contradictory geometric relationships from among the combinations of feature points after processing in step A3 (step A10).

Next, the three-dimensional shape construction unit 16 calculates three-dimensional coordinates for constructing a three-dimensional shape using combinations of feature points not identified in step A10 (step A11).

Next, the 3D shape construction unit 16 determines the suitability of the feature points whose 3D coordinates were calculated in step A11, and then eliminates inappropriate feature points and recalculates the 3D coordinates using appropriate feature points. The 3D shape construction unit 16 then constructs point cloud data of the object using the 3D coordinates of each feature point finally obtained, and uses this as the 3D shape of the object (step A12).

Then, the display unit 20 displays the three-dimensional shape constructed in step A12 as a two-dimensional image on the screen of the display device 21 (step A13).

[Effects of the First Embodiment]
As described above, according to the first embodiment, a user can check feature points in an image and specify any feature point simply by specifying a position on the image. Once a feature point is specified, the three-dimensional shape of the object is reconstructed based on the geometric relationship between the specified feature point and its corresponding feature point. As a result, the accuracy of the three-dimensional shape is improved. Furthermore, the user does not need to precisely specify a point or line segment on the image, but only needs to specify a general position on the image, which reduces the burden on the user compared to conventional methods.

[program]
The program in the first embodiment may be a program that causes a computer to execute steps A1 to A13 shown in Fig. 10. By installing and executing this program on a computer, the image processing device and image processing method in the first embodiment can be realized. In this case, the processor of the computer functions as a combination extraction unit 11, a temporary 3D shape construction unit 12, a feature point identification unit 13, a feature point extraction unit 14, a matrix calculation unit 15, a 3D shape construction unit 16, an image acquisition unit 17, a filtering unit 18, an input acceptance unit 19, and a display unit 20 to perform processing. Examples of the computer include a general-purpose PC, a smartphone, and a tablet terminal device.

The program in embodiment 1 may also be executed by a computer system constructed by multiple computers. In this case, for example, each computer may function as one of the combination extraction unit 11, temporary 3D shape construction unit 12, feature point identification unit 13, feature point extraction unit 14, matrix calculation unit 15, 3D shape construction unit 16, image acquisition unit 17, filtering unit 18, input acceptance unit 19, and display unit 20.

(Embodiment 2)
The image processing device according to the second embodiment will be described below with reference to FIGS.

[Device configuration]
First, a description will be given of the configuration of the image processing device in embodiment 2. The image processing device in embodiment 2 has the same configuration as the image processing device 10 in embodiment 1 shown in Fig. 2. Therefore, in the following description, Figs. 2 to 9 will be referred to as appropriate.

However, the image processing device in embodiment 2 differs from the image processing device in embodiment 1 in terms of the functionality of the feature point identification unit 13. The differences from embodiment 1 are explained below using Figure 11. Figure 11 is an explanatory diagram for explaining the functionality of the image processing device in embodiment 2.

In the second embodiment, as in the first embodiment, the feature point identification unit 13 identifies a feature point corresponding to a specified position on one image, and then identifies a feature point corresponding to this feature point on another image where a position is not specified.

However, in embodiment 2, when the feature point identification unit 13 identifies feature points on each image, it calculates three-dimensional coordinates for each identified feature point using a virtual three-dimensional shape. Furthermore, for each identified feature point, the feature point identification unit 13 calculates an index for the calculated three-dimensional coordinates using the calculated three-dimensional coordinates, the two-dimensional coordinates on the image in which each feature point was identified, and the camera parameters of the image in which each feature point was identified. The index indicates the error between the calculated three-dimensional coordinates and the actual three-dimensional coordinates at the corresponding position on the object 30.

Here, we will explain some specific examples of indices. Specific examples of indices include reprojection error and epipolar error.

First, we will explain the case where reprojection error is used as the index. In this case, the feature point identification unit 13 reprojects the feature points onto the original image using the three-dimensional coordinates and the camera parameters (camera matrix) of the original image, and calculates the difference between the two-dimensional coordinates after reprojection and the two-dimensional coordinates at the time of identification as the reprojection error. Specifically, as shown in Figure 10, the feature point identification unit 13 calculates the difference df for each identified feature point using the above-mentioned equations 15 to 18.

Next, we will explain the case where epipolar error is used as an index. In this case, as shown in Figure 5, for each identified feature point, the feature point identification unit 13 uses Equation 1 above to find an epipolar line on the image where the corresponding feature point is located that corresponds to the epipolar line on which the corresponding feature point is located. Then, for each identified feature point, the feature point identification unit 13 calculates the distance d between the feature point and the epipolar line, and sets the calculated distance d as the epipolar error.

The feature point identification unit 13 then presents the calculated index in addition to the identified feature points. In this case, as shown in FIG. 11, the feature point identification unit 13 can also present the index using an icon that changes depending on the value of the calculated index. In the example of FIG. 11, an icon 33 is displayed at the location of the feature point. The feature point identification unit 13 sets the size of the icon 33 depending on the magnitude of the error indicated by the index, and displays the icon 33 of the set size on the display screen of the display device 21 via the display unit 20.

As in embodiment 1, when feature points around the specified position are displayed as icons 33 as shown in Figure 11, the user uses the input device to specify one of the feature points on the image where the position was specified.

However, in embodiment 2, the index at the feature point is indicated by the size of the icon 33, so the user can specify the feature point so that the icon 33 becomes smaller in all images. In this case, the feature points selected and extracted by the feature point extraction unit 14 are feature points with small reprojection error or epipolar error. Therefore, the numerical matrix calculated by the matrix calculation unit 15 is accurate, resulting in improved accuracy of the three-dimensional shape of the object.

Furthermore, in embodiment 2, the change in the icon is not limited to a change in size. The change in the icon may also be a change in color (hue, shade, etc.), a change in shape, etc. Specifically, the indicator at the feature point may be indicated by a change in the color of the icon 33 (large error: red, medium error: yellow, small error: blue, etc.) or by a change in the shape of the icon 33 (large error: x, medium error: △, small error: ○, etc.). The ranges for large error, medium error, and small error are set appropriately in advance using thresholds.

[Device operation]
Next, the operation of the image processing device in embodiment 2 will be described with reference to FIG. 12. FIG. 12 is a flow diagram showing the operation of the image processing device in embodiment 2. In the following description, reference will be made to FIGS. 1 to 9 and 11 as appropriate. Also in embodiment 2, the image processing method is implemented by operating the image processing device. Therefore, the description of the image processing method in embodiment 2 will be replaced by the description of the operation of the image processing device below.

As shown in FIG. 12, first, the image acquisition unit 17 acquires multiple image data of an object from an external device (step B1). Step B1 is the same as step A1 shown in FIG. 10. When step B1 is executed, the display unit 20 displays images of each acquired image data on the display screen of the display device 21.

Next, the combination extraction unit 11 identifies feature points from the image for each image data acquired in step B1, and further extracts corresponding feature points between the images as combinations of corresponding feature points (step B2). Step B2 is the same as step A2 shown in Figure 10.

Next, the filtering unit 18 calculates the geometric relationship between the feature points for each combination of feature points extracted in step B2, identifies incorrect combinations of feature points based on the calculation results, and eliminates the identified combinations of feature points (step B3). Step B3 is the same as step A3 shown in Figure 10.

Next, the temporary 3D shape construction unit 12 calculates temporary 3D coordinates using the combination of feature points after processing in step B3, and constructs a temporary 3D shape of the object composed of a point group of feature points using the calculated temporary 3D coordinates (step B4). Step B4 is the same step as step A4 shown in Figure 10.

Next, when the user specifies the position of the object on the image displayed on the screen of the display device 21, the input accepting unit 19 accepts the position specification (step B5). Step B5 is the same as step A5 shown in FIG. 10.

Next, the feature point identification unit 13 identifies feature points corresponding to the positions accepted in step B5, and further identifies feature points corresponding to the previously identified feature points in images other than the image in which the positions are specified (step B6). Step B6 is the same as step A6 shown in Figure 10.

Next, the feature point identification unit 13 calculates an index for each feature point and sets the size of the icon 33 according to the calculated value. The feature point identification unit 13 then displays the icon with the set size on the display screen of the display device 21 via the display unit 20 as the identified feature point (step B7).

Next, when a feature point is specified on the image whose position is specified on the screen of the display device 21, the feature point extraction unit 14 selects the specified feature point and further extracts feature points corresponding to the selected feature point on other images (step B8).

Next, the matrix calculation unit 15 calculates a numerical matrix that represents the geometric relationship between the feature points selected on the image whose position is specified and the feature points extracted from other images (step B9). Step B9 is the same as step A9 shown in Figure 10.

Next, the three-dimensional shape construction unit 16 uses the numerical matrix calculated in step B9 to identify combinations of feature points that have contradictory geometric relationships from among the combinations of feature points obtained after the processing of step B3 (step B10). Step B10 is the same as step A10 shown in Figure 10.

Next, the three-dimensional shape construction unit 16 calculates three-dimensional coordinates for constructing a three-dimensional shape using combinations of feature points not identified in step B10 (step B11). Step B11 is the same as step A11 shown in Figure 10.

Next, the three-dimensional shape construction unit 16 determines the suitability of the feature points whose three-dimensional coordinates were calculated in step B11, and then eliminates inappropriate feature points and recalculates the three-dimensional coordinates using appropriate feature points. The three-dimensional shape construction unit 16 then constructs point cloud data of the object using the three-dimensional coordinates of each feature point finally obtained, and uses this as the three-dimensional shape of the object (step B12). Step B12 is the same step as step A12 shown in Figure 10.

Then, the display unit 20 displays the three-dimensional shape constructed in step A12 as a two-dimensional image on the screen of the display device 21 (step B13). Step B13 is the same as step A13 shown in Figure 10.

[Effects of the Second Embodiment]
As described above, in the second embodiment, the user can confirm the feature points to be designated by the icons 33 displayed on the image. That is, in the second embodiment, the user can designate feature points so as to reduce the reprojection error, epipolar error, or the like, thereby further improving the accuracy of the three-dimensional shape.

[Modification]
In the above example, the image processing device always displays the calculated indices on the screen and prompts the user to specify feature points, but embodiment 2 may have a different configuration. Specifically, embodiment 2 may have a configuration in which, after the feature point identification unit 13 calculates an index for each feature point, the feature point extraction unit 14 selects and extracts feature points for each image based on the calculated indices without receiving a specification from the user. In this configuration, appropriate feature points are automatically extracted.

[program]
The program in the first embodiment may be a program that causes a computer to execute steps B1 to B13 shown in Fig. 12. By installing and executing this program in a computer, the image processing device and image processing method in the second embodiment can be realized. In this case, the processor of the computer functions as a combination extraction unit 11, a temporary 3D shape construction unit 12, a feature point identification unit 13, a feature point extraction unit 14, a matrix calculation unit 15, a 3D shape construction unit 16, an image acquisition unit 17, a filtering unit 18, an input acceptance unit 19, and a display unit 20 to perform processing. Examples of the computer include a general-purpose PC, a smartphone, and a tablet terminal device.

The program in embodiment 2 may also be executed by a computer system constructed by multiple computers. In this case, for example, each computer may function as one of the combination extraction unit 11, temporary 3D shape construction unit 12, feature point identification unit 13, feature point extraction unit 14, matrix calculation unit 15, 3D shape construction unit 16, image acquisition unit 17, filtering unit 18, input acceptance unit 19, and display unit 20.

[Physical configuration]
A computer that realizes the image processing device by executing the programs in the first and second embodiments will now be described with reference to Fig. 13. Fig. 13 is a block diagram showing an example of a computer that realizes the image processing device in the first and second embodiments.

As shown in FIG. 13, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader/writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so that they can communicate data with each other.

Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111. In this embodiment, the GPU or FPGA can execute the programs in the embodiments.

The CPU 111 deploys the program in this embodiment, which is composed of a group of codes stored in the storage device 113, into the main memory 112 and executes each code in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as DRAM (Dynamic Random Access Memory).

Furthermore, the program in this embodiment is provided in a state stored on a computer-readable recording medium 120. Note that the program in this embodiment may also be distributed over the Internet connected via the communication interface 117.

Specific examples of the storage device 113 include a hard disk drive and semiconductor storage devices such as flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and mouse. The display controller 115 is connected to the display device 119 and controls the display on the display device 119.

The data reader/writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes the processing results of the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, and optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

Note that the image processing device in this embodiment can also be realized by using hardware corresponding to each part, rather than a computer on which a program is installed. Furthermore, the image processing device may be realized in part by a program and the remaining part by hardware.

Some or all of the above-described embodiments can be expressed by (Appendix 1) to (Appendix 15) described below, but are not limited to the following descriptions.

(Appendix 1)
a combination extraction unit that extracts combinations of corresponding feature points from each of a plurality of images of the object;
a temporary three-dimensional shape constructing unit that constructs a temporary three-dimensional shape of the object, the temporary three-dimensional shape being configured as a point cloud of the feature points, using a combination of the extracted feature points;
a feature point identification unit that, when a position on the object is designated in one of the plurality of images, identifies the corresponding feature point in accordance with the designated position, and further identifies the feature point corresponding to the identified feature point in another of the images in which a position is not designated;
a feature point extraction unit that selects, from the identified feature points in the image in which a position on the object is specified, any feature point that satisfies a set condition, and extracts, in the other image, the feature point that corresponds to the selected feature point;
a matrix calculation unit that calculates a numerical matrix that represents a geometric relationship between the selected feature points and the extracted feature points;
Using the numerical matrix, a combination of feature points having a contradictory geometric relationship is identified from among the extracted combinations of feature points, and
a three-dimensional shape constructing unit that constructs a three-dimensional shape of the object using a combination of the feature points other than the identified combination of the feature points;
An image processing device comprising:

(Appendix 2)
2. The image processing device according to claim 1,
the feature point identification unit further presents the identified feature points together with each of the plurality of images on a screen;
When any of the feature points in the image whose position on the object is specified is designated on the screen, the feature point extraction unit selects the designated feature point, using the designation as the setting condition.
1. An image processing device comprising:

(Appendix 3)
3. The image processing device according to claim 1,
The feature point identification unit
calculating three-dimensional coordinates for each of the identified feature points using the virtual three-dimensional shape, and calculating an index indicating an error between the calculated three-dimensional coordinates and actual three-dimensional coordinates at the corresponding position of the object using the calculated three-dimensional coordinates, the two-dimensional coordinates on the image where each feature point is identified, and the camera parameters of the image where each feature point is identified;
presenting the calculated index in addition to the identified feature points;
1. An image processing device comprising:

(Appendix 4)
4. The image processing device according to claim 3,
the feature point identification unit calculates a re-projection error of the feature point as the indicator, and presents an icon whose size changes according to the calculated value of the re-projection error, thereby presenting the indicator.
1. An image processing device comprising:

(Appendix 5)
4. The image processing device according to claim 3,
the feature point identification unit calculates an epipolar error of the feature point as the indicator, and presents the indicator by presenting an icon that changes depending on the calculated value of the epipolar error.
1. An image processing device comprising:

(Appendix 6)
a combination extraction step of extracting combinations of corresponding feature points from each of a plurality of images of the object;
a temporary three-dimensional shape construction step of constructing a temporary three-dimensional shape of the object, the temporary three-dimensional shape being configured as a point cloud of the feature points, using a combination of the extracted feature points;
a feature point identifying step of, when a position on the object is specified in one of the plurality of images, identifying the corresponding feature point in accordance with the specified position, and further identifying the feature point corresponding to the identified feature point in another of the images in which a position is not specified;
a feature point extraction step of selecting, from the feature points identified in the image in which a position on the object is specified, any feature point that satisfies a set condition, and extracting, in the other image, the feature point corresponding to the selected feature point;
a matrix calculation step of calculating a numerical matrix that represents a geometric relationship between the selected feature points and the extracted feature points;
Using the numerical matrix, a combination of feature points having a contradictory geometric relationship is identified from among the extracted combinations of feature points, and
a three-dimensional shape construction step of constructing a three-dimensional shape of the object using a combination of the feature points other than the identified combination of the feature points;
An image processing method comprising:

(Appendix 7)
7. The image processing method according to claim 6, further comprising:
In the feature point specifying step, the specified feature points are further displayed on a screen together with each of the plurality of images;
In the feature point extraction step, when any of the feature points in the image whose position on the object is specified is specified on the screen, the specified feature point is selected with the specification being set as the setting condition.
An image processing method comprising:

(Appendix 8)
8. The image processing method according to claim 6, further comprising:
In the feature point specifying step,
calculating three-dimensional coordinates for each of the identified feature points using the virtual three-dimensional shape, and calculating an index indicating an error between the calculated three-dimensional coordinates and actual three-dimensional coordinates at the corresponding position of the object using the calculated three-dimensional coordinates, the two-dimensional coordinates on the image where each feature point is identified, and the camera parameters of the image where each feature point is identified;
presenting the calculated index in addition to the identified feature points;
An image processing method comprising:

(Appendix 9)
9. The image processing method according to claim 8, further comprising:
In the feature point identification step, a reprojection error of the feature point is calculated as the index;
presenting the indicator by presenting an icon whose size changes depending on the calculated value of the reprojection error;
An image processing method comprising:

(Appendix 10)
9. The image processing method according to claim 8, further comprising:
In the feature point identifying step, an epipolar error of the feature point is calculated as the index, and an icon that changes depending on the calculated value of the epipolar error is presented to present the index.
An image processing method comprising:

(Appendix 11)
On the computer,
a combination extraction step of extracting combinations of corresponding feature points from each of a plurality of images of the object;
a temporary three-dimensional shape construction step of constructing a temporary three-dimensional shape of the object, the temporary three-dimensional shape being configured as a point cloud of the feature points, using a combination of the extracted feature points;
a feature point identifying step of, when a position on the object is specified in one of the plurality of images, identifying the corresponding feature point in accordance with the specified position, and further identifying the feature point corresponding to the identified feature point in another of the images in which a position is not specified;
a feature point extraction step of selecting, from the identified feature points in the image in which a position on the object is specified, any feature point that satisfies a set condition, and extracting, in the other image, the feature point that corresponds to the selected feature point;
a matrix calculation step of calculating a numerical matrix that represents a geometric relationship between the selected feature points and the extracted feature points;
Using the numerical matrix, a combination of feature points having a contradictory geometric relationship is identified from among the extracted combinations of feature points, and
a three-dimensional shape construction step of constructing a three-dimensional shape of the object using a combination of the feature points other than the identified combination of the feature points;
A program that executes.

(Appendix 12)
12. The program of claim 11,
In the feature point specifying step, the specified feature points are further displayed on a screen together with each of the plurality of images;
In the feature point extraction step, when any of the feature points in the image whose position on the object is specified is specified on the screen, the specified feature point is selected with the specification being set as the setting condition.
A program characterized by:

(Appendix 13)
13. The program according to claim 11 or 12,
In the feature point specifying step,
calculating three-dimensional coordinates for each of the identified feature points using the virtual three-dimensional shape, and calculating an index indicating an error between the calculated three-dimensional coordinates and actual three-dimensional coordinates at the corresponding position of the object using the calculated three-dimensional coordinates, the two-dimensional coordinates on the image where each feature point is identified, and the camera parameters of the image where each feature point is identified;
presenting the calculated index in addition to the identified feature points;
A program characterized by:

(Appendix 14)
14. The program of claim 13,
In the feature point identification step, a reprojection error of the feature point is calculated as the index;
presenting the indicator by presenting an icon whose size changes depending on the calculated value of the reprojection error;
A program characterized by:

(Appendix 15)
14. The program of claim 13,
In the feature point identifying step, an epipolar error of the feature point is calculated as the index, and an icon that changes depending on the calculated value of the epipolar error is presented to present the index.
A program characterized by:

As described above, according to the present disclosure, when constructing a three-dimensional shape of an object using multiple images, it is possible to improve the accuracy of the three-dimensional shape while reducing the burden on the worker. The present invention is useful in fields where it is necessary to construct the three-dimensional shape of an object from images.

REFERENCE SIGNS LIST 10 Image processing device 11 Combination extraction unit 12 Temporary three-dimensional shape construction unit 13 Feature point identification unit 14 Feature point extraction unit 15 Matrix calculation unit 16 Three-dimensional shape construction unit 17 Image acquisition unit 18 Filtering unit 19 Input reception unit 20 Display unit 30 Object 31 Cursor 32 Feature point 33 Icon 41, 42 Camera 51, 52 Image 110 Computer 111 CPU
112 Main memory 113 Storage device 114 Input interface 115 Display controller 116 Data reader/writer 117 Communication interface 118 Input device 119 Display device 120 Recording medium 121 Bus

Claims (6)

a combination extraction unit that extracts combinations of corresponding feature points from each of a plurality of images of the object;
a temporary three-dimensional shape constructing unit that constructs a temporary three-dimensional shape of the object, the temporary three-dimensional shape being configured as a point cloud of the feature points, using a combination of the extracted feature points;
a feature point identification unit that, when a position on the object is designated in one of the plurality of images, identifies the corresponding feature point in accordance with the designated position, and further identifies the feature point corresponding to the identified feature point in another of the images in which a position is not designated;
a feature point extraction unit that selects, from the feature points identified in the image in which a position on the object is specified, any feature point that satisfies a set condition, and extracts, in the other image, the feature point that corresponds to the selected feature point;
a matrix calculation unit that calculates a numerical matrix that represents a geometric relationship between the selected feature points and the extracted feature points;
Using the numerical matrix, a combination of feature points having a contradictory geometric relationship is identified from among the extracted combinations of feature points, and
a three-dimensional shape constructing unit that constructs a three-dimensional shape of the object using a combination of the feature points other than the identified combination of the feature points;
Equipped with
The feature point identification unit
calculating three-dimensional coordinates for each of the identified feature points using the virtual three-dimensional shape, and calculating an index indicating an error between the calculated three-dimensional coordinates and actual three-dimensional coordinates at the corresponding position of the object using the calculated three-dimensional coordinates, the two-dimensional coordinates on the image where each feature point is identified, and the camera parameters of the image where each feature point is identified;
presenting the calculated index in addition to the identified feature points;
1. An image processing device comprising: 2. The image processing device according to claim 1,
the feature point identification unit further presents the identified feature points together with each of the plurality of images on a screen;
When any of the feature points in the image whose position on the object is specified is designated on the screen, the feature point extraction unit selects the designated feature point, using the designation as the setting condition.
1. An image processing device comprising: 2. The image processing device according to claim 1 ,
the feature point identification unit calculates a re-projection error of the feature point as the indicator, and presents an icon whose size changes according to the calculated value of the re-projection error, thereby presenting the indicator.
1. An image processing device comprising: 2. The image processing device according to claim 1 ,
the feature point identification unit calculates an epipolar error of the feature point as the indicator, and presents the indicator by presenting an icon that changes depending on the calculated value of the epipolar error.
1. An image processing device comprising: a combination extraction step of extracting combinations of corresponding feature points from each of a plurality of images of the object;
a temporary three-dimensional shape construction step of constructing a temporary three-dimensional shape of the object, the temporary three-dimensional shape being configured as a point cloud of the feature points, using a combination of the extracted feature points;
a feature point identifying step of, when a position on the object is specified in one of the plurality of images, identifying the corresponding feature point in accordance with the specified position, and further identifying the feature point corresponding to the identified feature point in another of the images in which a position is not specified;
a feature point extraction step of selecting, from the identified feature points in the image in which a position on the object is specified, any feature point that satisfies a set condition, and extracting, in the other image, the feature point that corresponds to the selected feature point;
a matrix calculation step of calculating a numerical matrix that represents a geometric relationship between the selected feature points and the extracted feature points;
Using the numerical matrix, a combination of feature points having a contradictory geometric relationship is identified from among the extracted combinations of feature points, and
a three-dimensional shape construction step of constructing a three-dimensional shape of the object using a combination of the feature points other than the identified combination of the feature points;
and
In the feature point specifying step,
calculating three-dimensional coordinates for each of the identified feature points using the virtual three-dimensional shape, and calculating an index indicating an error between the calculated three-dimensional coordinates and actual three-dimensional coordinates at the corresponding position of the object using the calculated three-dimensional coordinates, the two-dimensional coordinates on the image where each feature point is identified, and the camera parameters of the image where each feature point is identified;
presenting the calculated index in addition to the identified feature points;
An image processing method comprising: On the computer,
a combination extraction step of extracting combinations of corresponding feature points from each of a plurality of images of the object;
a temporary three-dimensional shape construction step of constructing a temporary three-dimensional shape of the object, the temporary three-dimensional shape being configured as a point cloud of the feature points, using a combination of the extracted feature points;
a feature point identifying step of, when a position on the object is specified in one of the plurality of images, identifying the corresponding feature point in accordance with the specified position, and further identifying the feature point corresponding to the identified feature point in another of the images in which a position is not specified;
a feature point extraction step of selecting, from the identified feature points in the image in which a position on the object is specified, any feature point that satisfies a set condition, and extracting, in the other image, the feature point that corresponds to the selected feature point;
a matrix calculation step of calculating a numerical matrix that represents a geometric relationship between the selected feature points and the extracted feature points;
Using the numerical matrix, a combination of feature points having a contradictory geometric relationship is identified from among the extracted combinations of feature points, and
a three-dimensional shape construction step of constructing a three-dimensional shape of the object using a combination of the feature points other than the identified combination of the feature points;
Execute
In the feature point specifying step,
calculating three-dimensional coordinates for each of the identified feature points using the virtual three-dimensional shape, and calculating an index indicating an error between the calculated three-dimensional coordinates and actual three-dimensional coordinates at the corresponding position of the object using the calculated three-dimensional coordinates, the two-dimensional coordinates on the image where each feature point is identified, and the camera parameters of the image where each feature point is identified;
presenting the calculated index in addition to the identified feature points;
program.