JP6604502B2 - Depth map generation apparatus, depth map generation method, and program - Google Patents

Description

  The present invention relates to a depth map generation apparatus, a depth map generation method, and a program for generating a depth map of an image.

  3D restoration technology based on multi-viewpoint images is attracting attention not only in the computer vision research community, but also in a wide range of fields such as digital archives of cultural assets and the entertainment industry (see Patent Document 1).

JP 2013-019801 A

The above-described three-dimensional restoration technique based on the multi-viewpoint image uses a triangulation technique, and finally generates a three-dimensional point group by integrating the depth maps at the respective viewpoints.
However, simply converting the three-dimensional point group calculated from the depth map at each viewpoint into the world coordinate system (hereinafter also referred to as world coordinates) leaves many miscorresponding points in the finally obtained three-dimensional restoration result. End up. This is because there is a miscorresponding point in the depth map of the viewpoint used for integration, which affects the three-dimensional restoration result.

  The present invention has been made in view of such a situation. When generating a three-dimensional point group, the number of erroneous correspondence points in the depth map used for integration is reduced, and the occurrence of erroneous correspondence points in the three-dimensional restoration result is suppressed. A depth map generation apparatus, a depth map generation method, and a program are provided.

In order to solve the above-described problem, the depth map generation apparatus of the present invention, based on a multi-view stereo algorithm, a depth map generation unit that generates a depth map from images of different viewpoints using a phase-only correlation method; For each of the edges to be connected from the node corresponding to the pixel of the target viewpoint that is the target for detecting the erroneous corresponding point, for each of the correct corresponding point node of the label of the correct corresponding point and the erroneous corresponding point node of the label of the incorrect corresponding point On the other hand, the first weighting number is given by comparing the pixel with the corresponding pixel of the reference viewpoint which is another viewpoint in the vicinity of the target viewpoint, and the pixel node at the target viewpoint is adjacent to the pixel. For the edge connecting the node of another adjacent pixel, the difference in depth between the pixel at the target viewpoint and the adjacent pixel The graph is generated by assigning the corresponding second weighting number so that the integrated value of each weighting number of the edge through which the cutting line of the graph cut that cuts the graph passes is minimized, and the node in the graph Each corresponding to the correct corresponding point or the label of the erroneous corresponding point, and an erroneous corresponding point removing unit for removing the erroneous corresponding point , wherein the first weighting number includes the node of the positive corresponding point and the pixel A first A weighting number for edges connecting between nodes corresponding to, and a first B weighting number for edges connecting nodes corresponding to the pixels and nodes of the miscorresponding points, and the first A weighting number is The numerical value corresponding to the peak value of the phase-only correlation function with the other viewpoint is set higher as the correlation value is higher. The number of digits is a value corresponding to the difference in depth of the same pixel from a reference viewpoint that is another viewpoint in the vicinity, and the larger the difference is, the higher the value is set, and the second weighting number is adjacent in the same viewpoint. numerical as the difference in depth is small with the node of pixels, characterized that you are set high.

  The depth map generation device of the present invention further includes an artifact removal unit that removes an artifact between different viewpoints after the erroneous correspondence point removal unit performs the erroneous correspondence point removal, and the artifact removal unit includes the target The difference between the depth of the pixel at the viewpoint and the depth obtained by converting the pixel at the comparative viewpoint, which is a viewpoint separated from the target viewpoint by a predetermined distance, into the coordinate system of the target viewpoint is determined, and the difference is preset. It is determined whether or not the pixel of the target viewpoint is an artifact compared with the threshold value.

  In the depth map generation device according to the present invention, when the threshold includes the first threshold and the second threshold, and the artifact removal unit determines that each of the target viewpoint and the comparative viewpoint is when the difference is equal to or less than the first threshold. If the same pixel is restored at a similar depth in step (a), the pixel of the target viewpoint is determined not to be an artifact, and the difference exceeds the first threshold and is less than the second threshold. When the same pixel is restored at different depths in each of the target viewpoint and the comparative viewpoint, it is determined that the pixel of the target viewpoint is an artifact, and the difference is greater than or equal to the second threshold value , Assuming that the pixel of the target viewpoint is the pixel in a different area from the pixel of the target viewpoint. And judging not an artifact of cell.

  In the depth map generation device of the present invention, the artifact removal unit determines the comparison viewpoint as the target viewpoint when a parallax angle with respect to the restored pixel of the target viewpoint and the comparison viewpoint is less than a preset third threshold. It is determined that it is not used for comparison with.

  In the depth map generation apparatus of the present invention, when generating a depth map from the images of a plurality of different viewpoints, the pixel matching process between the images is performed by a hierarchical search that gradually increases the resolution of the pixels. The depth map for each pixel in the depth map is corrected using a weighted median filter for the depth map after completion of the matching process for each layer in the hierarchical search.

  In the depth map generation device of the present invention, the weight value of each of the pixels in the weighted median filter window includes a distance from a target pixel, a luminance value of the pixel in the image of the hierarchy, and a phase-only correlation function. It is characterized by being set corresponding to the peak value.

The depth map generation method of the present invention includes a depth map generation process in which a depth map generation unit generates a depth map from images of different viewpoints using a phase-only correlation method based on a multi-view stereo algorithm, The removal unit connects from the node corresponding to the pixel of the target viewpoint, which is the target for detecting the incorrect corresponding point, to each of the positive corresponding point node of the correct corresponding point label and the incorrect corresponding point node of the incorrect corresponding point label. For each edge to be given, a first weighting number is assigned by comparing the pixel with a corresponding pixel of a reference viewpoint that is another viewpoint in the vicinity of the target viewpoint, and a node of the pixel in the target viewpoint The edge connecting the node of another adjacent pixel adjacent to the pixel is the depth between the pixel at the target viewpoint and the adjacent pixel. To graph generated by applying a second number weighted corresponding to Kino difference, do the integrated value of the weighted number of the edges cut line of the graph cut for cutting the graph passes is minimized, the distributing each of the nodes in the graph on the label of the positive corresponding points or the erroneous corresponding points, erroneous and the corresponding point removal process erroneously removing corresponding points, only contains the first number weighted, the positive corresponding point A first A weighting number for an edge connecting the node corresponding to the pixel and a node corresponding to the pixel, and a first B weighting number for an edge connecting the node corresponding to the pixel and the node corresponding to the miscorresponding point, The 1A weighting number is a numerical value corresponding to the peak value of the phase-only correlation function with the other viewpoint, and the higher the correlation value, the higher the numerical value is set. The 1B weighting number is a numerical value corresponding to the difference in depth of the same pixel from a reference viewpoint that is another viewpoint in the vicinity, and the larger the difference is, the higher the numerical value is set, and the second weighting number is the same. numerical smaller difference in the depth of a node of the adjacent pixels at the viewpoint characterized that you are set high.

The program according to the present invention is based on a multi-view stereo algorithm and uses a phase-only correlation method to generate a depth map from a plurality of different viewpoint images. And for each of the miscorresponding point nodes of the label of the miscorresponding point, for each edge connected from the node corresponding to the pixel of the target viewpoint that is the target of detecting the miscorresponding point, the pixel and the vicinity of the target viewpoint A first weighting number is assigned by comparison with a corresponding pixel in the reference viewpoint, which is another viewpoint, and a node of a pixel in the target viewpoint is connected to a node of another adjacent pixel adjacent to the pixel. A second weighting number corresponding to a difference in depth between a pixel at the target viewpoint and the adjacent pixel with respect to an edge; To given graph generated by performed so integrated value of the weighted number of the edges cut line of the graph cut for cutting the graph passes is minimized, said each of the nodes of the graph positive corresponding point Or it is a program for allocating to the label of the miscorresponding point and operating as miscorresponding point removing means for removing the miscorresponding point , wherein the first weighting number corresponds to the node of the correct corresponding point and the pixel A first A weighting number for an edge connecting nodes to be connected, and a first B weighting number for an edge connecting a node corresponding to the pixel and a node of the miscorresponding point, and the first A weighting number is the other The numerical value corresponding to the peak value of the phase-only correlation function with respect to the other viewpoint is set higher as the correlation value is higher. The B weighting number is a numerical value corresponding to the difference in depth of the same pixel from a reference viewpoint that is another viewpoint in the vicinity. The larger the difference is, the higher the numerical value is set, and the second weighting number is the same in the viewpoint. This is a program in which the numerical value is set higher as the difference in depth from the adjacent pixel node is smaller .

  As described above, according to the present invention, when generating a three-dimensional point group, depth map generation that reduces the number of erroneous correspondence points in the depth map used for integration and suppresses the occurrence of erroneous correspondence points in the three-dimensional restoration result. An apparatus, a depth map generation method, and a program can be provided.

It is a figure showing the structure of the arithmetic processing apparatus in one Embodiment of this invention. It is a figure showing the outline | summary of the one-dimensional phase only correlation method in one Embodiment of this invention. It is a figure showing the relationship between a three-dimensional point and parallax in one Embodiment of this invention. It is a figure showing the correlation function by one-dimensional phase only correlation method for every stereo pair in one Embodiment of this invention. It is a flowchart showing the operation | movement procedure of the production | generation of the mesh model of the arithmetic processing unit in one Embodiment of this invention. It is a figure showing the operation | movement procedure of the function calculation apparatus in each hierarchy of an image pyramid in one Embodiment of this invention. It is a flowchart showing the search procedure of the depth in one Embodiment of this invention. It is a figure showing the matching window defined in each of the stereo pair in one Embodiment of this invention. It is a flowchart showing the operation | movement procedure of the function calculation apparatus in each hierarchy of an image pyramid in one Embodiment of this invention. It is a figure explaining the relationship of the depth map of the upper layer, middle layer, and lower layer in an image pyramid. It is a figure which shows the relationship between the coordinate system in a depth map, and the coordinate system in a window. It is a figure explaining the process of the graph cut which removes the miscorresponding point in a depth map. It is a figure which shows the removal of the miscorresponding point by the process of the graph cut in a depth map. It is a figure explaining the consistency between the depth maps of a some viewpoint when an artifact generate | occur | produces. It is a figure explaining the determination processing whether the pixel of the target viewpoint is a positive corresponding point or an artifact.

  An embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 shows the configuration of the arithmetic processing unit. The arithmetic processing device 100 is, for example, a CPU (Central Processing Unit), and is connected to the storage device 200 and the output device 300 via the bus 400.

  The arithmetic processing device 100 reads the multi-viewpoint images from the storage device 200 and normalizes the parallax of the stereo pairs constituting the multi-viewpoint images. Here, normalization refers to, for example, a stereo pair parallax in which an object is imaged from a plurality of cameras that are not installed on the same plane, from a reference viewpoint and a camera that is virtually installed near the reference viewpoint. The parallax is converted using the camera parameters (for example, coordinate system, focal length) so that the parallax with the other viewpoint (near viewpoint) is obtained. Details of normalization will be described later with reference to FIG.

  Further, the arithmetic processing device 100 determines and cuts out a matching window from the stereo pair whose parallax is normalized, and calculates a correlation function of the stereo pair for each cut out matching window. The arithmetic processing device 100 integrates only correlation functions having a high correlation value map corr and confidence value map conf among the correlation functions calculated for each matching window, and generates a depth map dep based on the integrated correlation function. . Here, the depth map is map information representing the depth of the surface shape of the object captured in the multi-viewpoint image. In addition, the arithmetic processing device 100 generates a mesh model in which the surface shape of the object is restored based on the generated depth map dep, and outputs data representing the generated mesh model to the output device 300.

  Hereinafter, a correlation function (hereinafter referred to as “POC function”) based on a position-only correlation method (hereinafter referred to as “POC function”), which is an image matching method focusing on the phase component of the image signal, is used as an example of the correlation function. The explanation will be continued as being possible.

  First, a case will be described in which image deformation between stereo pairs is expressed using the parallel movement amount of the matching window and the parallax magnification of the matching window.

An image matching method based on the one-dimensional phase-only correlation method (one-dimensional POC) will be described.
In the parallel stereo pair, the width direction of the matching window cut out from one image of the stereo pair (hereinafter referred to as “reference viewpoint image”) and the other image of the stereo pair (hereinafter referred to as “neighbor viewpoint image”). )) Matches the width direction of the matching window. In this case, since the extracted matching windows are arranged in the one-dimensional direction, the arithmetic processing device 100 calculates the parallel movement amount of the corresponding point in the matching window in which the attention point on the object surface is imaged based on the one-dimensional POC function. Can be calculated with high accuracy.

  FIG. 2 shows an outline of the one-dimensional phase-only correlation method. When a one-dimensional image signal f (n) and a one-dimensional image signal g (n) are given for N (N is a positive integer) points in the one-dimensional image, the one-dimensional image signal f (n) The parallel movement amount between the one-dimensional image signal g (n) is calculated based on the one-dimensional POC. Here, it is assumed that the discrete time index n of the one-dimensional image signal is n = −M,..., M (M is a positive integer). Further, N = 2M + 1.

  The one-dimensional discrete Fourier transform (DFT) of the one-dimensional image signal f (n) is expressed by the equation (1). Further, the one-dimensional discrete Fourier transform result of the one-dimensional image signal g (n) is expressed by Expression (2).

Here, k = −M,..., M is a discrete frequency index. WN = e−j2π / N is a twiddle factor. AF (k) and AG (k) are amplitude spectra. ΘF (k) and θG (k) are phase spectra. Further, the normalized mutual power spectrum R (k) is expressed by the equation (3).

Here, G (k) with a superscript line is a complex conjugate of G (k). ΘF (k) −θG (k) is a phase difference spectrum between the one-dimensional image signal f (n) and the one-dimensional image signal g (n).

  Further, the one-dimensional POC function r (n) is expressed by Expression (4) as a one-dimensional inverse discrete Fourier transform (Inverse Discrete Fourier Transform: IDFT) of the normalized mutual power spectrum R (k).

  When the one-dimensional image signal f (n) and the one-dimensional image signal g (n), which are stereo pairs, are translated from each other by a minute parallel movement amount δ, the one-dimensional image signal f (n) and one-dimensional The POC function r (n) of the image signal g (n) is expressed by equation (5).

  Equation (5) represents the general form of the POC function when the one-dimensional image signal is translated slightly by the translation amount δ. Here, α is a parameter introduced to express the height (peak value) of the correlation peak. When uncorrelated noise is added to the image, the value of the correlation peak height α decreases, so α ≦ 1 is actually satisfied. In this case, the height α of the correlation peak of the POC function corresponds to an index of similarity between the one-dimensional image signal f (n) and the one-dimensional image signal g (n). Further, the position coordinates of the correlation peak correspond to the parallel movement amount δ of the one-dimensional image signal f (n) and the one-dimensional image signal g (n).

  Therefore, the similarity between the one-dimensional image signals is calculated based on the correlation peak height α. The parallel movement amount δ between the one-dimensional image signals is calculated with subpixel accuracy based on the position coordinates of the correlation peak.

  In an actual multi-viewpoint image, an estimation error may occur in the parallel movement amount δ due to the distortion caused by the change in the baseline length of the viewpoint and the influence of noise generated in the digital image. Accordingly, various high accuracy techniques that are important when calculating the parallel movement amount δ using a one-dimensional POC function will be described below.

  In the one-dimensional discrete Fourier transform (DFT), since it is assumed that the signal waveform circulates periodically, discontinuity at the end points of the signal waveform becomes a problem. In order to reduce the influence of this discontinuity, the window function w (n) is applied to the one-dimensional image signals f (n) and g (n). In the following, it is assumed that the one-dimensional Hanning window shown in Expression (6) is used as the window function.

  Further, it is known that in natural images, energy is generally concentrated on low frequency components, and the energy of high frequency components is smaller than that of low frequency components. For this reason, when disturbances such as aliasing (frequency component aliasing), noise (noise), and distortion are added to the image, the signal-to-noise ratio (S / N ratio) of the high frequency component is significantly reduced.

  Therefore, the arithmetic processing unit 100 (see FIG. 1) uses a low-pass spectrum weighting when calculating the normalized mutual power spectrum R (k) in order to suppress the influence of high-frequency components with low reliability. Apply function H (k). In the following, it is assumed that a Gaussian function represented by Expression (7) is used as the spectrum weighting function.

  Here, σ is a constant representing the width of the Gaussian function. Hereinafter, as an example, σ = √ (0.5). In this case, the correlation peak model is a Gaussian type expressed by Equation (8).

  Since the translation amount δ is generally a real value, the position coordinate of the correlation peak of the POC function exists between sampling grid points, that is, between pixels of an image. Therefore, the arithmetic processing apparatus 100 fits the equation (8) representing the correlation peak model to the numerical data of the actually calculated POC function to obtain the position coordinates of the correlation peak existing between the pixels of the image. presume. In this case, the height α of the correlation peak and the parallel movement amount δ are the fitting parameters. In the following, as an example of the fitting method, the “Levenberg-Marquardt method”, which is one of the solutions of the nonlinear least square problem, is used.

  When the image processing method based on the POC function is applied to the correspondence between stereo pairs, the arithmetic processing device 100 defines a matching window around the coordinates of the corresponding point in the image obtained by capturing the attention point on the object surface, A POC function between the cut out windows is calculated. In this case, it is difficult to calculate the position coordinates of the correlation peak of the POC function with high accuracy based on only one set of windows because of the influence of noise.

Therefore, the arithmetic processing device 100 determines and cuts out matching windows from a plurality of stereo pairs in which the point of interest is imaged, and extracts a one-dimensional image signal from the plurality of cut out matching windows. The one-dimensional POC function based on the one-dimensional image signal is integrated by averaging based on a plurality of stereo pairs. Hereinafter, the averaged POC function is referred to as “average POC function”. By using the average POC function to calculate the correlation peak α and the parallel movement amount δ, it is possible to suppress the influence of noise.
This completes the description of the image matching method based on the one-dimensional phase-only correlation method (one-dimensional POC).

  Returning to FIG. 1, the description of the configuration of the arithmetic processing unit 100 will be continued. The arithmetic processing device 100 includes a function calculation device 110, a depth map generation device 120, and a mesh model generation device 130. The function calculation device 110 calculates the coordinates of the correlation peak of the POC function, that is, the coordinates of the three-dimensional point (attention point) imaged on the image, based on the translation window δ of the matching window. The function calculation device 110 includes a normalization unit 111, a window setting unit 112, a function calculation unit 113, and a function integration unit 114.

  The normalization unit 111 parallelizes the reference viewpoint image and the near viewpoint image that form the stereo pair. Further, the normalization unit 111 normalizes the parallax of a plurality of stereo pairs based on the target pixel and the camera parameter. By normalizing the parallax of a plurality of stereo pairs, the function calculation apparatus 110 can integrate the correlation function even when the cameras are not installed on the same plane.

Among the multi-view images V = {V ₀ ,..., V _F−1 } whose camera parameters (for example, the coordinate system and the focal length) are known, the normalization unit 111 selects the reference viewpoint image V _R εV, A near viewpoint image C = {C ₀ ,..., C _K−1 } ⊂V− {V _R } picked up from a near viewpoint is read from the storage device 200. Here, F is the number of multi-viewpoint images. K is the number of the near viewpoint images C.

Normalizing unit 111, the K sets of stereo pair consisting of a set of the reference viewpoint image V _R and one of neighboring viewpoint image C, and collimated by the camera parameters. Further, the function calculation apparatus 110 determines and cuts out a matching window from the reference viewpoint image and the near viewpoint image constituting the parallelized stereo pair, and normalizes the parallax of the extracted matching window.

The normalization unit 111 normalizes the parallax of the matching window cut out from the stereo pair as follows. Here, to match the predetermined world coordinate, the camera coordinates of the reference viewpoint image V _R is determined in advance.

When the normalization unit 111 calculates the parallax of the matching window in the paralleled stereo pair V _R -C _i (C _i εC) for the three-dimensional point M = (X, Y, Z), the reference viewpoint image V _R and the near viewpoint image C _i are collimated by camera parameters. Here, the rotation matrix Ri with respect to the camera coordinates of the reference viewpoint image V _R is represented by the formula (9).

  In this case, the relationship between the three-dimensional point Mi = (Xi, Yi, Zi) in the parallelized stereo pair and the three-dimensional point M is expressed by Expression (10).

Further, the relationship between the three-dimensional point Mi = (Xi, Yi, Zi) in the parallel stereo pair V _R -C _i and the parallax d _i is expressed by Expression (11).

Here, (ui, vi) is at the reference viewpoint image _{V R} which collimates represent pixel coordinates of the corresponding point of the three-dimensional point Mi. Further, (u0i, v0i) represents the angle around the reference viewpoint image _{V R} which is collimated. Βi represents the focal length. B _i represents the baseline length between stereo pairs.

3D point relationship between M and the disparity _{d i,} based on the equation (10) and (11) is represented by the formula (12).

Between the reference viewpoint image _{V R} and the neighboring view image _{C j} ∈C- _{{C i},} holds a relationship represented by the formula (13).

  In addition, the coordinates of the three-dimensional point M in the predetermined world coordinates are expressed by equations (14) to (16) based on the equations (12) and (13).

A parallax _{d i} stereopair _{V R} - Ci was collimated, the parallax _{d j} stereopair _{V R} -C _j that collimation holds true relationship expressed by the equation (17).

Therefore, a parallax _{d i} in the stereo pair _{V R} -C _i was collimated, the disparity _{d j} in the stereo pair _{V R} -C _j were collimated, and pixel coordinates of the corresponding point in the reference viewpoint image _{V R,} camera parameters Are related by the parallax magnification s. Here, the camera parameters are the focal length βi and the baseline length B _i between the stereo pairs.

That is, the normalized parallax d is determined by considering the parallax magnification s of each stereo pair. Stereopair _{V R -C i (i = 0} , ..., K-1) was collimated if given, the parallax d _i of each stereo pair collimated, in a normalized parallax d of the formula The relationship expressed by (18) holds. The normalization part 111 normalizes the parallax of the paralleled stereo pair by Formula (18).

Here, si is a magnification of parallax of each paralleled stereo pair. Further, Table for pixel coordinates of the corresponding point (u, v), the magnification si parallax of each stereo pair collimated using Formula (19) based on equation (17) in the reference viewpoint image V _R which is not collimated Is done.

  Here, | s | is expressed by Expression (20).

In this case, the three-dimensional point Mi in the parallel stereo pair V _R -C _i is represented by the equation (21).

  Further, the window setting unit 112 determines a matching window for a stereo pair in which parallax is normalized, with a corresponding point in an image obtained by capturing a point of interest on the object surface as a center. The function calculation unit 113 calculates a POC function between matching windows determined by the window setting unit 112. The function integration unit 114 also integrates a plurality of POC functions calculated by the function calculation unit 113.

FIG. 3 shows the relationship between the three-dimensional point and the parallax. Parallax at collimated stereo pair _{V R -C i (C i ∈C} ) is in line-of-sight direction through the three-dimensional point M and the reference viewpoint image _{V R,} a small amount .DELTA.M = ([Delta] X from the 3D point M, [Delta] Y, It is determined based on a point M ′ (= M + ΔM) located at a position shifted by ΔZ).

In this case, three-dimensional point M 'is a disparity _{d i} in the stereo pair _{V R} -C _i was collimated, based on the equation (12) is represented by the formula (22).

Here, in the stereo pair V _R -C _i was collimated with respect to the parallax d _i of the three-dimensional point M located at the true position, if the error [delta] _i occurs in the disparity d _i of the three-dimensional point M, The relationship represented by Expression (23) is established between the parallax d _i of the three-dimensional point M and the parallax d ′ _i of the three-dimensional point M ′.

If local image deformation was assumed that only the image deformation due to translation, matching windows f _i cut defining around the corresponding point in the reference viewpoint image V _R, the corresponding point in the vicinity viewpoint image C _i center translation amount [delta] _i between the matching window g _i cut defining a is calculated based on the position coordinates of the correlation peak of the 1-dimensional POC function between the matching window f _i and matching window g _i.

Here, the movement amount [delta] _i parallel in the stereo pair _{V R} -C _i was collimated, parallel movement amount [delta] _j in the collimated stereo pair _{V R -C j (C j ∈C-} {C i}) is If they do not match, the position coordinates of the correlation peak based on the stereo pair V _R -C _i was collimated, the position coordinates of the correlation peak based on collimated stereo pair V _R -C _j, do not coincide.

For example, in FIG. 3, the base length B1 of the parallelized stereo pair V _R -C ₁ is longer than the base length B _{0 of} the parallel stereo pair V _R -C ₀ . Therefore, for the same amount of change ΔZ in the depth direction, the translation amount δ ₁ of the corresponding point in the parallelized stereo pair V _R -C ₁ is equal to the corresponding point in the parallelized stereo pair V _R -C ₀ . Is greater than the parallel movement amount δ0.

  Returning to FIG. 1, the description of the configuration of the arithmetic processing unit 100 will be continued. The window setting unit 112 enlarges or reduces the matching window determined and cut out from each paralleled stereo pair based on the normalized parallax. The function calculation unit 113 calculates a one-dimensional POC function between matching windows determined and cut out by the window setting unit 112.

  The function integration unit 114 integrates the one-dimensional POC function by averaging the one-dimensional POC function calculated by the function calculation unit 113 between the stereo pairs. Thereby, the position coordinates of the correlation peak of the one-dimensional POC function calculated for each matching window are unified among the stereo pairs. Here, based on the parallax d normalized with respect to the three-dimensional point M and the parallax d ′ normalized with respect to the three-dimensional point M ′ (= d + δ), the three-dimensional point M ′ is expressed by the equation (24). expressed.

Therefore, the translation amount of the matching window f _i cut out around the corresponding point in the parallel stereo pair V _R -C _i and the matching window g _i cut out around the corresponding point in the neighboring viewpoint image C _i is siδ. Therefore, the window size of the matching window _{f i} and matching window _{g i} are respectively defined as siw. The window size w is a predetermined reference size.

Further, the window size siw is enlarged or reduced by (1 / si) times, so that the window size of the matching window of the one-dimensional image signal is made common to the window size w. Hereinafter, the matching window _{f i} which is enlarged or reduced window size siw within (1 / si) times, referred to as the matching window fi '. Similarly, the matching window _{g i} which is enlarged or reduced window size siw within (1 / si) times, referred to as the matching window gi '.

Matching window f _{i 'and} the matching window g _i' coordinates of the correlation peak of the 1-dimensional POC function calculated on the basis of the corresponds to the matching window f _{i 'and} the matching window g _i' translation amount between [delta]. Here, the size of the matching window cut out from the parallel stereo pair V _R -C _j (C _j εC- {C _i }) is the window size sjw. The correlation peak of the one-dimensional POC function of the parallel stereo pair V _R -C _j is obtained by expanding or reducing the window size sjw different from the window size siw by (1 / sj) times, thereby making the parallel stereo pair It coincides with the correlation peak of the one-dimensional POC function of V _R -C _i .

  In addition to the above-described enlargement or reduction of the matching window, the window setting unit 112 may further enlarge or reduce the matching window so that the window size corresponds to the surface shape of the target object.

  Further, the window setting unit 112 executes image matching for each of a plurality of matching windows whose window sizes (magnifications) are changed when determining the magnification or reduction ratio of the matching window described above, thereby performing one-dimensional POC. You may employ | adopt the window size from which the correlation peak of a function becomes the highest.

  In consideration of three-dimensional restoration based on multi-viewpoint images, the window sizes of the matching windows cut out from the left and right images in a plurality of stereo pairs are mutually related between the stereo pairs. (For example, stereo pairs located in close positions have the same window size). Therefore, the window setting unit 112 may add restrictions to the size of the matching window by comparing the window sizes of other stereo pairs with each other based on multi-view images of three or more viewpoints.

In FIG. 4, the correlation function (one-dimensional POC function) by the one-dimensional phase-only correlation method is shown for each stereo pair. FIG. 4A shows a one-dimensional POC function when a matching window is cut out from all parallel stereo pairs with the same window size w without considering the normalized parallax magnification s. ing. In this case, the parallel movement amount δ _i between the matching windows is different for each stereo pair. That is, the calculated position coordinates of the correlation peak of the one-dimensional POC function do not match between the stereo pairs.

On the other hand, FIG. 4B shows a one-dimensional POC function in the case where a matching window is cut out from each paralleled stereo pair with the same window size siw in consideration of the normalized parallax magnification s. ing. In this case, since the normalized parallax magnification s is taken into account, the parallel movement amount δ _i between the matching windows becomes the same for each stereo pair. That is, the calculated position coordinates of the correlation peak of the one-dimensional POC function match between stereo pairs. Therefore, the one-dimensional POC function calculated based on the parallelized stereo pairs can be integrated because the position coordinates are identical between the stereo pairs.

  The function integration unit 114 integrates the one-dimensional POC function calculated for each stereo pair by averaging the stereo pairs. The coordinates of the three-dimensional point M (attention point) in the predetermined world coordinates are expressed by Expression (25) using the parallel movement amount δ based on the position coordinates of the correlation peak of the integrated one-dimensional POC function.

Here, when the three-dimensional point M is not reflected in the near viewpoint image C _i ∈C due to occlusion (a state in which the object in the front conceals the object in the back), or a plurality of parallaxes at the object boundary. When a matching window is cut out from a region having the same, it is expected that a very large error occurs in the position coordinates of the correlation peak of the one-dimensional POC function calculated from the matching window. Further, it is experimentally known that when the image deformation between matching windows cannot be approximated only by translation, the correlation peak height α of the POC function calculated from the matching window is lowered.

  Therefore, the function integration unit 114 calculates only the average POC function of the average POC function (integrated one-dimensional POC function) whose correlation peak height α is equal to or higher than the threshold thcorr (image similarity is high). The data is output to the generation device 120.

  The depth map generation device 120 generates a depth map based on the average POC function output from the function integration unit 114. Thereby, the depth map generation apparatus can generate the depth map while suppressing the influence of the occlusion and the object boundary. The depth map generation device 120 includes a depth map generation unit 121, a filter unit 122, an erroneous corresponding point removal unit 123, and an artifact removal unit 124.

  The mesh model generation device 130 generates a mesh model based on the depth map calculated by the depth map generation device 120 and outputs data representing the generated mesh model to the storage device 200 and the output device 300.

Storage device 200 stores the reference viewpoint image V _R, and the neighboring viewpoint image C, a data representing a depth map, and the data representing the mesh model. The storage device 200 stores various parameters and programs used for processing.

  Data representing the mesh model is input to the output device 300. The output device 300 displays the surface shape of the restored object based on the data representing the mesh model. The output device 300 is, for example, a liquid crystal display device.

Next, the operation procedure of the function calculation device will be described.
FIG. 5 is a flowchart showing an operation procedure for generating a mesh model of the arithmetic processing unit. Normalization unit 111 of the function calculation unit 110 (see FIG. 1) is a reference view image _{V R} and near view image C, read from the storage device 200 (step S1). Normalizing unit 111 collimates the reference viewpoint image V _R and neighboring viewpoint images C forming a stereo pair, it is normalized parallax collimated stereo pair (step S2).

  The window setting unit 112 (see FIG. 1) enlarges or reduces the window size of the matching window cut out from the stereo pair based on the parallax magnification (step S3). The function calculation unit 113 (see FIG. 1) calculates a one-dimensional POC function between the enlarged or reduced matching windows (step S4).

  The function integration unit 114 (see FIG. 1) integrates one-dimensional POC functions whose correlation peak heights are equal to or greater than a threshold among the calculated one-dimensional POC functions (step S5). The depth map generator 121 (see FIG. 1) generates a depth map based on the integrated one-dimensional POC function (step S6). The mesh model generation device 130 generates a mesh model based on the depth map (step S7).

<How to search depth using an image pyramid>
When searching for the depth of the three-dimensional point M (calculating coordinates), the function calculation device 110 may combine a coarse / fine search using an image pyramid. Thereby, the depth map generating apparatus 120 can calculate the depth map representing the world coordinates of the three-dimensional point M group with a smaller number of matchings compared to the case where the image pyramid is not used.

Input information to the depth search process, the reference viewpoint image _{V R,} near view image C = {C0, ..., CK -1} is. Further, output information from the depth search processing is a depth map dep (I _Z (u, v) described later), a correlation value map corr (I _α (u, v) described later), and a confidence value map conf. . Each parameter used in the depth map calculation process includes a correlation peak height threshold thcorr, an image pyramid hierarchy number H, a reference size window size w, and a line used for calculating an average POC function. The number L.

FIG. 6 shows an operation procedure of the function calculation device in each layer of the image pyramid. Normalizing unit 111 (see FIG. 1) includes a reference viewpoint image V _R, and a near view image C captured from the viewpoint of its neighbors, read from the storage device 200. Then, the normalizing unit 111 parallelizes the stereo pair V _R -C _i (i = 0,..., K−1). Furthermore, the function calculation unit 113 generates an image pyramid (hierarchical image) of the parallel stereo pair V _R -C _i .

Hereinafter, the reference viewpoint images constituting the parallel stereo pair V _R -C _i are denoted as Vrect, i, 0, and the near viewpoint images are denoted as Crect, i, 0. In addition, for the image pyramid hierarchy h = 1,..., H−1, an image obtained by multiplying the reference viewpoint image Vrect, i, 0 by (2−h) is referred to as a reference viewpoint image Vrect, i, h. Further, for the hierarchy h = 1,..., H−1 of the image pyramid, an image obtained by multiplying the near viewpoint image Clect, i, 0 by (2−h) is referred to as a near viewpoint image Clect, i, h.

The function calculation unit 113 (see FIG. 1) determines a start coordinate Zinit for searching for depth by executing matching of the entire image with respect to the uppermost layer H-1 of the image pyramid.
This start coordinate Zinit is based on the position coordinates of the correlation peak of the one-dimensional POC function between the reference viewpoint image Vrect, i, H-1 and the neighboring viewpoint image Clect, i, H-1, and each stereo pair (i = 0,..., K-1) are determined by calculating the parallax.

Function calculating section 113, any point in the reference viewpoint image _{V R} m = the (u, v), the depth (depth), and calculates the correlation value and the confidence value, the depth map dep (m), the correlation value map corr (m) and the confidence value map conf (m) are stored in the storage device 200. Here, the function calculation unit 113, the coordinates of an arbitrary point in the reference viewpoint image V _R m, by repeating the process while changing over the reference viewpoint image V _R, the depth map dep (m), the correlation value map corr (m) and the confidence value map conf (m) are calculated.

  Hereinafter, corresponding points obtained by projecting the three-dimensional points Mi on the reference viewpoint images Vrect, i, h (i = 0,..., K−1) constituting one of the stereo pairs are denoted by mi, h = (ui, h, vi, h). In addition, corresponding points obtained by projecting the three-dimensional point Mi to the near viewpoint images Clect, i, h (i = 0,..., K−1) constituting the other of the stereo pairs are represented by m′i, h = (u ′). i, h, v′i, h).

FIG. 7 is a flowchart showing a depth search procedure. Normalizing unit 111, the reference viewpoint image _V ratio of parallax at any point m on the _{R si (i = 0, ...} , K-1) , and is calculated based on the equation (19). In this case, the corresponding point mi = (ui, vi) in the parallelized reference viewpoint image Vrect, i, 0 is calculated based on the projection matrix used for the deformation of the images constituting the parallelized stereo pair (step Sa1).

The function calculation unit 113 initially sets an image pyramid hierarchy h = H−1 and a three-dimensional point M = Minit, and searches for coordinates of the three-dimensional point M in world coordinates from the top layer H-1 of the image pyramid. Start. Here, Minit, in the reference viewpoint image _{V R,} is a three-dimensional point depth on the line of sight which passes through the coordinates of an arbitrary point m is in the position of Zinit (step Sa2).

The window setting unit 112 defines the matching windows f _i and h from the reference viewpoint images Vrect, i and h with the window size siw × the number of lines L as the center and the corresponding points mi and h as the center. Further, the window setting unit 112 determines and cuts the matching windows g _i , h from the neighboring viewpoint images Vrect, i, h around the corresponding points mi, h with a size of window size siw × number of lines L. The function calculation unit 113 calculates the one-dimensional POC function ri, h based on the extracted matching windows f _i , h and the matching windows gi, h (step Sa3).

  The function integration unit 114 calculates the average POC function “rave, h” by averaging the one-dimensional POC function having the correlation peak height αi ≧ threshold thcorr among the one-dimensional POC functions “ri, h” (step Sa4). ). Hereinafter, the number of one-dimensional POC functions for which the correlation peak height αi ≧ threshold thcorr is expressed as K ′.

  The function integration unit 114 performs function fitting on the average POC function “rave”, “h”, thereby calculating the correlation peak height α of the average POC function and the position coordinates of the correlation peak of the average POC function (translation amount δ). Equivalent). The function integration unit 114 updates the coordinates of the three-dimensional point M in the world coordinates (camera coordinates of the reference viewpoint image) based on the parallel movement amount δ and Expression (25) (step Sa5).

  The function integration unit 114 determines whether the hierarchy of the image pyramid is the lowest layer (step Sa6). When the hierarchy of the image pyramid is the lowest layer (step Sa6: Yes), the depth map generation unit 121 generates a depth map of the three-dimensional point M group in which the correlation value map corr and the confidence value map conf are equal to or greater than a threshold ( Step Sa7).

  Here, the depth map dep (m) is obtained by using the one-dimensional POC function “rave, 0” and the three-dimensional point “M = (X, Y, Z)” when the hierarchy is h = 0 (lowermost layer). It is defined in advance as shown in (26). Similarly, the correlation value map corr (m) is defined in advance as shown in Expression (27). Similarly, the confidence value map conf (m) is defined in advance as in Expression (28).

  On the other hand, in step Sa6, when the hierarchy of the image pyramid is not the lowest layer (step Sa6: No), the function integration unit 114 subtracts the value 1 from the hierarchy h, and returns the process to step Sa3 (step Sa8).

  As described above, the depth map generation unit 121 generates a depth map dep based on the correlation value map corr and the confidence value map conf that are equal to or greater than the threshold value, thereby calculating highly reliable coordinates for the three-dimensional point M group. Can do.

  This completes the description of the case where the image deformation between stereo pairs is expressed using the parallel movement amount of the matching window and the parallax magnification of the matching window.

  Next, not only the translation amount of the matching window and the parallax magnification of the matching window, but also the image deformation between stereo pairs is expressed using the matching window scaling ratio and the matching window skew. The case where it does is demonstrated.

  The images between the stereo pairs are deformed non-linearly according to the surface shape of the captured object to be three-dimensionally restored and the positional relationship of the camera. The window setting unit 112 (see FIG. 1) of the function calculation device 110 determines a matching window so as to reduce image deformation between stereo pairs. More specifically, the window setting unit 112 determines a matching window based on at least one of an enlargement / reduction ratio based on a line segment and an inclination based on a normal vector.

  Thereby, even when the image calculation between the stereo pairs is large, such as when the base line length between the stereo pairs is long or when the object to be three-dimensionally restored is greatly inclined with respect to the camera, the function calculating device 110 The correlation function of the stereo pair can be calculated with high accuracy. The function calculation device 110 can calculate the coordinates of the three-dimensional point (attention point) captured in the image with high accuracy. In addition, the depth map generation apparatus 120 can generate a depth map by executing high-precision image matching.

  Since the vertical coordinate axis or horizontal coordinate axis and the epipolar line are parallel between the parallel stereo pairs, the image deformation between the stereo pairs is limited to a one-dimensional direction. Therefore, when it is assumed that the object to be restored in three dimensions is locally a plane, the image between the matching viewpoint window of the parallel viewpoint reference viewpoint image Vrect, i and the neighboring viewpoint image Crect, i. The deformation can be expressed using the scaling ratio of the matching window and the skew (for example, skew slope) of the matching window.

In the following, in the paralleled stereo pair V _R -C _i , that is, in the parallelized stereo pair Vrect, i-Clect, i, the method of the three-dimensional point Mi = [Xi, Yi, Zi] and the three-dimensional point Mi A method of calculating the enlargement / reduction ratio ξi and the slope κi in the case where the line vector ni = [nX, i, nY, i, nZ, i] is given will be described.

  Note that the coordinate system of the three-dimensional point Mi is represented by the camera coordinate system of the parallelized reference viewpoint image Vrect, i. The coordinate system of the normal vector ni is also expressed by the camera coordinate system of the parallelized reference viewpoint image Vrect, i. The camera coordinates of these reference viewpoint images Vrect, i are rotated based on the camera parameters of the neighboring viewpoint images Clect, i when the reference viewpoint images Vrect, i and the neighboring viewpoint images Cect, i of the stereo pair are parallelized. Has been.

  FIG. 8 shows a matching window defined for each stereo pair. In FIG. 8, the stereo pair Vrect, i-Clect, i is parallelized so that the epipolar line is parallel to the horizontal coordinate axis. Further, it is assumed that the three-dimensional plane (object surface) is greatly inclined with respect to the camera.

  First, the case of reducing the image deformation based on the enlargement / reduction ratio ξi will be described with reference to FIG. Assume that a unit line segment centered on the three-dimensional point Mi is assumed on the intersection line between the three-dimensional point Mi and the normal vector ni and the three-dimensional plane (object surface) and the epipolar plane.

  The lengths w1, i of the line segments projected on the reference viewpoint image Vrect, i are expressed by Expression (29). Further, the length w2, i of the line segment projected on the near viewpoint image Clect, i is expressed by Expression (30).

  Here, ψ1, i represents an angle formed by the optical axis and the line of sight in the reference viewpoint image Vrect, i. Ψ2, i represents an angle formed by the optical axis and the line of sight in the near viewpoint image Clect, i. Moreover, φ1, i represents an angle formed by the projection vector obtained by projecting the normal vector ni onto the epipolar plane and the line of sight. Φ2, i represents an angle formed by a projection vector obtained by projecting the normal vector ni onto the epipolar plane and the line of sight.

  The length w1, i of the line segment in which the unit line segment is projected on the reference viewpoint image Vrect, i and the length w2, i of the line segment in which the unit line segment is projected on the near viewpoint image Crect, i The relationship is expressed by Expression (31) or Expression (32) based on Expression (29) and Expression (30).

  Further, the matching window enlargement / reduction ratio ξi is expressed by Expression (33). Even when the stereo pair Vrect, i-Clect, i is parallelized so that the epipolar line is parallel to the vertical coordinate axis, the scaling ratio ξi of the matching window is expressed by the equation (33).

  The window setting unit 112 (refer to FIG. 1) sets the window size in the horizontal direction (width direction) of the near viewpoint image Clect, i to siw pixels based on the parallax magnification si and the window size w. The vertical window size of Crect, i is set to L pixels (see the matching window shown in the lower right of FIG. 8).

  On the other hand, the window setting unit 112 refers to the window size in the horizontal direction (width direction) of the reference viewpoint image Vrect, i based on the parallax magnification si, the window size w, and the enlargement / reduction ratio ξi, with reference to ξisiw pixels. The vertical window size of the viewpoint image Vrect, i is set to L pixels (see the matching window shown in the lower left of FIG. 8). In this way, the window setting unit 112 can reduce local image deformation between stereo pairs based on the enlargement / reduction ratio ξi.

  In addition, instead of multiplying the horizontal window size of the reference viewpoint image Vrect, i by ξi, the one-dimensional POC function ri, h is obtained when the horizontal window size of the neighboring viewpoint image Cect, i is multiplied by 1 / ξi. The position coordinates of the correlation peak are multiplied by ξi. For this reason, the position coordinates of the correlation peak of the one-dimensional POC function rj, h calculated based on another stereo pair Vrect, j-Clect, j (j∈ {1,..., K-1}-{i}) Does not coincide with the position coordinates of the correlation peak of the one-dimensional POC function ri, h.

Next, a case where image deformation is reduced based on the inclination κi will be described with reference to FIG.
A straight line is assumed on a three-dimensional plane (object surface) determined by the three-dimensional point Mi and the normal vector ni. When this straight line is projected onto the reference viewpoint image Vrect, i, it passes through the corresponding point mi = [ui, vi] of the three-dimensional point Mi in the reference viewpoint image Vrect, i and is parallel to the vertical coordinate axis. It is assumed on a three-dimensional plane (object surface).

The straight line assumed on the three-dimensional plane in this way passes through the corresponding point mC _i = [uC _i , vi] of the three-dimensional point Mi in the near viewpoint image Clect, i, and is a near viewpoint image as a straight line having an inclination κi. It will be projected onto Crect, i. The straight line assumed on the three-dimensional plane is represented by [X, Y, Z] satisfying the equation (34).

Here, the coordinates (u0i, v0i) represent the center coordinates of the reference viewpoint image Vrect, i.
Β i represents the focal length of the reference viewpoint image Vrect, i. || tcam, i || represents the baseline length between stereo pairs. Further, v′i represents an arbitrary vertical coordinate on the reference viewpoint image Vrect, i.

  As shown in FIG. 8, when the near viewpoint image Crect, i is in the positive direction of the X axis with respect to the reference viewpoint image Vrect, i, the equation (35) is obtained by solving the equation (34) for the inclination κi. Is obtained.

  On the other hand, when the near viewpoint image Crect, i is in the negative direction of the X axis with respect to the reference viewpoint image Vrect, i, the sign of the slope κi is reversed. Therefore, equation (35) is more generally represented by equation (36). Even when the stereo pair Vrect, i-Clect, i is parallelized so that the epipolar line is parallel to the vertical coordinate axis, the inclination κi is expressed by Expression (33).

  The window setting unit 112 determines the matching window in the near viewpoint image Crect, i so that, for example, the skew inclination becomes κi. Further, the window setting unit 112 determines the center coordinates of each line constituting the matching window based on the positional deviation between the coordinate axis orthogonal to the epipolar line and the center coordinates of the matching window. Thus, the window setting unit 112 determines a matching window based on the inclination κi so as to reduce local image deformation between stereo pairs.

  The function integration unit 114 (see FIG. 1) integrates the one-dimensional POC function calculated for each stereo pair based on the enlargement / reduction ratio ξi and the inclination κi by averaging the stereo pairs. The function integration unit 114 calculates only the average POC function whose correlation peak height α is equal to or higher than the threshold thcorr (high image similarity) from the average POC function (integrated one-dimensional POC function). 120 (see FIG. 1).

The depth map generation device 120 performs depth map (to be described later) from a multi-viewpoint image by depth search using plane sweeping and hierarchical search using image pyramids based on the average POC function output from the function integration unit 114. I _Z (u, v)) is generated.

<How to search depth using an image pyramid>
FIG. 9 is a flowchart showing an operation procedure of the function calculation device in each layer of the image pyramid.
(Step Sb1) The normalization unit 111 (see FIG. 1) reads from the storage device 200 the reference viewpoint image Vrect, i whose camera parameters are known and the near viewpoint image Clect, i whose camera parameters are known. . The normalization unit 111 generates K parallel stereo pairs Vrect, i-Clect, i (i = 0,..., K−1). Here, the normalization unit 111 is a hierarchy obtained by reducing the image by 1/2 for each hierarchy for K parallel stereo pairs Vrect, i-Clect, i (i = 0,..., K−1). A number H of image pyramids is generated.

  (Step Sb2) The normalization unit 111 calculates the parallax magnification si in each stereo pair for the coordinates mi = [ui, vi] on the reference viewpoint image Vrect, i.

  (Step Sb3) Since the normal vector ni of the three-dimensional plane (object surface) to be three-dimensionally restored has not been calculated, the function integration unit 114 (see FIG. 1) is the uppermost layer H-1 of the image pyramid. The average POC function “rave, h” is calculated while changing the depth of the three-dimensional point Mi and the normal vector ni (image matching).

  Here, the step size for changing the depth of the three-dimensional point Mi may be a length corresponding to a quarter of the window size w. The average POC function for nine normal vectors n rotated in the range of ± (π / 8) around the X axis and the Y axis with reference to the normal vector ni facing the reference viewpoint image Vrect, i. Rave, h may be calculated.

  (Step Sb4) The function integration unit 114 selects the depth of the three-dimensional point Mi having the highest correlation peak height αi of the average POC function and the normal vector ni. Further, the function integration unit 114 updates the depth of the three-dimensional point Mi based on the position coordinates of the correlation peak of the average POC function “rave”, h. Note that the normal vector ni may be determined with higher accuracy on the basis of the selected normal vector ni.

  (Step Sb5) The function integration unit 114 calculates the average POC function “rave, h” between stereo pairs based on the normal vector ni selected in the top layer H−1, and the correlation peak height αi of the average POC function. The depth of the three-dimensional point Mi where becomes the highest is updated. Further, the function integration unit 114 updates the depth of the three-dimensional point Mi based on the position coordinates of the correlation peak of the average POC function “rave”, h.

  (Step Sb6) The function integration unit 114 determines whether or not the hierarchy of the image pyramid is the lowest layer h = 0. When the hierarchy of the image pyramid is the lowest layer (step Sb6: Yes), the function integration unit 114 proceeds with the process to step Sb8. On the other hand, when the hierarchy of the image pyramid is not the lowest layer (step Sb6: No), the function calculation unit 113 advances the process to step Sb7.

(Step Sb7) The function integration unit 114 subtracts the value 1 from the hierarchy h (moves the process to the next lower hierarchy), and returns the process to step Sb5.
(Step Sb8) The function integration unit 114 determines the updated depth of the three-dimensional point Mi as the depth of the three-dimensional point Mi corresponding to the coordinate mi.

  (Step Sb9) The function integration unit 114 determines whether or not the depth of the three-dimensional point Mi has been determined for all coordinates mi on the reference viewpoint image Vrect, i. When the depth of the three-dimensional point Mi is determined for all the coordinates mi on the reference viewpoint image Vrect, i (step Sb9: Yes), the function integration unit 114 includes the average POC function (integrated one-dimensional POC function). Only the average POC function whose correlation peak height α is equal to or greater than the threshold thcorr (the image similarity is high) is output to the depth map generation device 120, and the process is terminated. On the other hand, when the depth of the three-dimensional point Mi is not defined for any coordinate mi on the reference viewpoint image Vrect, i (step Sb9: No), the function integration unit 114 returns the process to step Sb2.

  The function calculation device 110 changes the reference viewpoint image Vrect, i for searching the depth, and similarly executes the process for the reference viewpoint image Vrect, i. The depth map generation device 120 (depth map generation unit 121) generates a depth map based on the average POC function output from the function integration unit 114.

<Repair missing pixels using weighted median filter>
In the hierarchical search using the image pyramid described above, the matching result of one pixel in the upper layer affects a plurality of pixels in the lower layer. For this reason, the erroneous correspondence of the pixels in the upper layer causes the erroneous correspondence of many pixels in the final depth map. In addition, since the window size relative to the image size increases relative to the upper layer, the range affected by the object boundary and the occlusion boundary increases. In order to solve this problem, in the present embodiment, the accuracy of the depth map is increased by using a weighted median filter. As a result, in the present embodiment, the weighted median filter is used to correct isolated miscorresponding points and artifacts at the object boundary and the occlusion boundary.

That is, in the present embodiment, in response to the lack of pixels in the upper layer matching, the above-described weighted median filter is used to perform the repair process, thereby causing an unexpected correspondence in the upper layer, an object boundary, and an occlusion boundary. Suppresses propagation of artifacts that occur in the vicinity to the lower layer.
In the present embodiment, the depth estimation result itself by window matching based on POC (Phase Only Correlation) for MVS (Multi View Stereo) is used for the final depth map. Therefore, the map value is not updated using the weighted median filter.

For each hierarchy in the image pyramid, depth map I _Z ^h (u, v) (depth map dep), correlation value map I _α ^h (u, v) (correlation value map corr), normal vector map I _Ψ ^h ( u, v) and the normal vector map I _φ ^h (u, v) are generated, the depth map I _Z ^h (u, v), the correlation value map I _α is calculated using a weighted median filter. ^{In each of h} (u, v) and normal vector map I _Ψ ^h (u, v), the pixel missing in the matching process is repaired. In each of the depth map I _Z ^h (u, v), the correlation value map I _α ^h (u, v), and the normal vector map I _Ψ ^h (u, v), the symbol h indicates the corresponding hierarchy in the image pyramid. ing. Here, each of u and v represents coordinate values on the X axis and the Y axis that form a two-dimensional plane in a three-dimensional space composed of the X axis, the Y axis, and the Z axis.

FIG. 10 is a diagram illustrating the relationship between the depth maps of the upper layer, the middle layer, and the lower layer in the image pyramid. In FIG. 10, each of the upper layer depth map I _Z ^H-1 (u, v), the middle layer depth map I _Z ^h (u, v), and the lower layer depth map I _Z ⁰ (u, v) is an image pyramid. Is configured. One pixel matched in the upper layer depth map I _Z ^H-1 (u, v) becomes four pixels in the middle layer depth map I _Z ^h (u, v). In addition, one pixel matched in the middle layer depth map I _Z ^h (u, v) becomes four pixels in the lower layer depth map I _Z ⁰ (u, v). For this reason, one pixel matched in the upper depth map I _Z ^H-1 (u, v) becomes 16 pixels in the lower depth map I _Z ⁰ (u, v). In the example in FIG. 10, the lower layer, which is the original image data, is reduced by 2 pixels vertically and horizontally as it becomes higher, and matching processing for each pixel is performed. In the case of reduction by n pixels in the vertical and horizontal directions, the number of pixels increases in units of n2 as it goes down, so that the lack of one pixel in the upper layer becomes the lack of a large number of pixels as it propagates to the lower layer. The other correlation value maps I _α ^h (u, v) and normal vector maps I _Ψ ^h (u, v) are similar in the image pyramid.

In the present embodiment, the weight function in the weighted median filter is the product of the bilateral weight in the original image (weight based on the luminance difference in the Gaussian filter kernel) and the correlation value of the corresponding pixel in the correlation value map. Is used.
In the following, the weighted median filter in this embodiment will be described by taking the depth map I _Z ^h (u, v) in the hierarchy h as an example.
The weighted median filter in the present embodiment uses the numerical value of the target pixel (depth numerical value of the depth map) that is a pixel to be filtered, and the numerical value of other pixels (depth map) in the peripheral region centered on the target pixel. This is a filter that is determined by the weighted median of the numerical value of the depth.

In the calculation of the weighted median, the numbers (a _k ) _{k = 0, 1,..., N−1} sorted in order from the smallest or largest _, and the weight (ω _k ) _{k = of} each element (corresponding to the pixel position) _{When each of 0, 1,..., N−1} is given, the weighted median value a _med of the sequence (ak) is obtained by the following equation (37). In the formula (37), each of n and l indicates the number of a term in the sequence. Here, l indicates the number of a term in the target pixel number sequence.

In the weighted median filter, the map value I of other pixels around the target pixel (a region including other pixels centered on the target pixel set in advance) for the target pixel whose data is to be restored by lacking. Each of _med (i, j) and its weight w _med (i, j) is sorted in ascending order or descending order. Then, a weighted median a _med is _obtained as the sorted sequence and the weight (w _k ) of each element. Here, (i, j) are coordinates in the window of the median filter, i = −M _ω ,..., M _ω , j = −M _ω ,..., M _ω , and the window size is N _ω. × N _ω (N _ω = 2M _ω +1) pixels.

FIG. 11 is a diagram showing the relationship between the coordinate system in the depth map and the coordinate system in the window. In FIG. 11, a depth map I _Z ^h (u, v) in the middle layer indicates the depth of each pixel of vertical u pixels × horizontal v pixels. Here, the window of the weighted median filter corresponds to the map value I _med (i, j). In map value _I med (i, j), as described _above, i = -M _ω, ..., a _{M ω, j = -M ω,} ..., a M _omega. The window size (number of pixels) of the window is N _ω × N _ω (N _ω = 2M _ω +1).
Here, the pixel coordinate (u ′, v ′) of the target pixel in the coordinate system of the depth map I _Z ^h (u, v) of the hierarchy h (middle layer) and the map value I _med (i, j) in the coordinate system in the window. ) Is expressed by the following equation (38).

The weight ω _med (i, j) is expressed by the following equation (39).

In the equation (39), the pixel coordinate (u ′, v ′) of the target pixel indicates the pixel at the window center of the weighted median filter in the coordinate system of the depth map I _Z ^h (u, v). I _V ^h (u, v) represents the luminance value of each pixel of the image data captured by the camera of the target viewpoint in the hierarchy h of the image pyramid. Each of σ _ω1 and σ _ω2 is an arbitrary parameter.
The above equation (32) is formed based on the following three assumptions.
Other pixel values in the vicinity of the target pixel (coordinate values within a predetermined range) in the same image data are values that are close in depth in the depth map.
A depth value in the depth map is close to another pixel having a luminance value close to the luminance value of the target pixel in the same image data.
-The lower the peak value of the POC function, the lower the reliability of the depth value in the depth map.

In each of the above equations (38) and (39), the depth map outside the image and the luminance value of the depth map I _Z ^h (u, v) and the luminance value I _V ^h (u, v) are set to 0. .
Using the map value I _med (i, j) of each pixel in the window described above, the map value (for example, the depth value) is sorted to generate a sequence (a _k ). Weights ω _med (i, j) of each pixel in the window using, by sorting the weight ω _med (i, j), and generates a weight sequence (omega _k). Then, the numerical sequence (a _k ) and the weighted numerical sequence (ω _k ) are substituted into the equation (37), the weighted median value a _med is calculated, and the depth map I _Z ^h (u, v) obtained by filtering the weighted median filter. The depth of the target pixel at the pixel coordinates (u ′, v ′) at. Substitution processing according to the following expression (40) is performed.

The update process of the depth map I _Z ^h (u, v) using the above-described weighted median filter, that is, the pixel coordinates (u ′, v ′) as the target pixel are sequentially changed, and the depth map I _Z ^h (u , V) calculates the weighted median value a _med of all the pixels, and updates the map value of each pixel to the calculated weighted median value.
In the above description, the depth map I _Z ^h (u, v) is taken as an example, but the correlation value map I _α ^h (u, v) and the normal vector map I _Ψ ^h (u, v) For each of them, a weighted median filter is used to perform a map value update process similar to that for the depth map I _Z ^h (u, v), that is, a process for updating each map. At this time, in the weighted median filter, the map value I _med (i, j) in the window is a numerical value of each map, for example, a correlation value map I _α ^h (u, v), Substitute into the equation corresponding to equation (38). Similarly, in the case of the normal vector map I _Ψ ^h (u, v), the normal vector is substituted as a map value into an expression corresponding to the expression (38).

As described above, according to the present embodiment, the depth map I _Z ^h (u, v) and the correlation value map I _α ^h (u, v) are applied to all layers other than the lowest layer in the image pyramid by the weighted median filter. v) and normal vector maps I _Ψ ^h (u, v) to update the map value of each map, so that the lack of pixels caused by window matching can be repaired, and the lack of pixels in the upper layer map It is possible to suppress the influence from propagating to the lower layer map.

<Elimination of erroneous correspondence points in the depth map>
Here, the miscorresponding point removing unit 123 in the depth map generating device 120 performs the following in the depth map (depth map dep generated by the depth map generating unit 121) of each viewpoint by the miscorresponding point removing process and the artifact removing process described below. Remove false correspondence points and remove artifacts in the depth map of each viewpoint (proposed method). The depth map dep generated by the above-described depth map generation unit 121 will be described below as a depth map I _Z (u, v).

-Miscorresponding point removal processing In this processing method, the miscorresponding points are removed using a graph cut method. Even if it is a miscorresponding point in the depth map of a certain viewpoint, in the depth map of another viewpoint (another viewpoint in the vicinity) that restores the coordinate point of the same three-dimensional coordinates (world coordinates), It may not be possible. For this reason, in the corresponding point removal process according to the present embodiment, the depth map of the target viewpoint to be evaluated for the corresponding point is consistent with the depth map of the reference viewpoint that is another viewpoint near the camera position. As a result, a process for removing an erroneous correspondence point of the depth map at the target viewpoint is performed. In particular, the depth map generation using the hierarchical search in the image pyramid hierarchy as described above spreads from the upper layer to the lower layer of the image pyramid and easily occurs in a cluster shape on the lower depth map.
Therefore, in the present embodiment, for the miscorresponding points generated in a cluster shape in the lower layer of the image pyramid, the consistency between the viewpoint (target viewpoint) where the miscorresponding point occurred and the other viewpoints (reference viewpoints) in the vicinity are adjusted. In order to take this, the erroneous corresponding point removal using the graph cut in the graph theory is performed.

The graph cut is a weighted directed graph including an S node (Source node: lnlier) and a T node (Sink node: Outlier), and an S-T cut is obtained and binary labeling of each of the S node and the T node is performed. It is a technique to do. Here, the S-T cut refers to a cut line that minimizes the weight of the edge from the S node to the node corresponding to the pixel or the edge from the node corresponding to each pixel to the T node. Binary labeling is performed.
At this time, the ST cut is performed by allocating each term of the energy function to the edge weight of the directed graph and solving the energy minimization problem. In addition, when the ST cut is applied to image processing, it is necessary to consider the connection relationship with other pixels in the vicinity of the pixel that is the target node.

FIG. 12 is a diagram illustrating a graph cut process for removing an erroneous corresponding point in the depth map I _Z (u, v). In FIG. 12, a graph to be cut is shown. That is, in the target viewpoint V _R ∈V, the node (Node: each pixel) of the pixel m of each pixel coordinate (u, v) on the depth map I _Z (u, v) is a positive corresponding point (correct 3 A binary labeling problem is solved by a graph cut whether it is a (dimensional restoration point) or an erroneous correspondence point (an incorrect three-dimensional restoration point). In FIG. 12, each node indicates a pixel m, and an S node (Source node) is a label for correct correspondence (Inlier), and a T node (Sink node) is a label for incorrect correspondence (Outlier). For each pixel that is a pixel, an edge (edge1) extending from the S node and an edge (edge2) extending from each pixel to the T node are shown.

Also, because of the connection relationship with other adjacent pixels, the corresponding nodes between the target pixel and each of four neighboring pixels adjacent to the target pixel) Add a bi-directional edge (edge3) to The four pixels described above are, for example, in three-dimensional coordinates, the pixel coordinates (u−1, v) and pixel coordinates (u) of pixels adjacent in the vertical and horizontal directions with respect to the pixel coordinates (u, v) of the target pixel. , V−1), pixel coordinates (u + 1, v), and pixel coordinates (u, v + 1). A set representing nodes of all pixels is N, and a set representing edges between all pixels is E. A weight is added to each edge, and the weight of the edge from the S node to each of the pixel nodes (edge1) is E _data1 (m), and the edge (edge2) from each of the pixel nodes to the T node The weight is E _data2 (m), and the weight of an edge (edge3) representing the relationship between nodes of adjacent pixels (for example, m ₁ -m ₂ ) is E _smooth (m ₁ -m ₂ ). At this time, the graph cut becomes a problem of minimizing the total energy E _all (N, E) of the following equation (41).

The weight E _data1 (m) is the weight of each edge (edge1) from the label S to the node of the pixel, and is a numerical value representing the degree of likelihood of a positive corresponding point of the pixel corresponding to the node. That is, as the weight E _data1 (m) indicating that the pixel m is a positive corresponding point is larger, the edge from the S node toward the node of the pixel m is less likely to be included in the ST cut. In the present embodiment, it is assumed that a pixel having a high peak value of the POC function is likely to be a positive corresponding point. Therefore, the weight E _data1 (m) is expressed by the following equation (42) in the present embodiment. In the equation (42), λ _E1 represents the priority of cost, and σ _E1 represents a parameter. I _α (m) is a correlation value map indicating the correlation value of each pixel.

On the other hand, the weight E _data2 (m) is a weight of each edge (edge2) from the pixel node to the label T, and is a numerical value representing the degree of likelihood of an erroneous corresponding point of the pixel corresponding to the node. That is, as the weight E _data2 (m) indicating that the pixel m is an erroneous correspondence point is larger, the edge from the node of the pixel m toward the T node is less likely to be included in the ST cut. In the present embodiment, the greater the difference between the depth of the target pixel and the depth of the pixel that has been reprojected to a nearby viewpoint, the higher the possibility that the target pixel is a miscorresponding point. Assumes. Therefore, the weight E _data2 (m) is expressed by the following equation (43) in the present embodiment. In equation (43), λ _E2 represents the priority of cost, and σ _E2 represents a parameter. C is a perspective group near When Creating a V _R. Z 'represents the depth of the pixels when a three-dimensional point of the reference viewpoint V _R pixel was converted into 3-dimensional point other viewpoints C _k of neighboring. I _{Z, Ck} (m ′) represents the depth of the pixel m ′ at the viewpoint C _k in the vicinity of the target viewpoint corresponding to the pixel m of the target viewpoint.

In the above equation (43), each of the depth Z ′ and the pixel m ′ satisfies the relationship represented by the following equation (44). In this equation (44), R _Ck from the camera coordinates at the viewpoint V _R of the target, indicates the rotation of the camera coordinate other viewpoints C _k in the vicinity of the viewpoint V _R of this subject. Also, t _Ck from the camera coordinates at the viewpoint V _R of the subject shows a parallel movement amount of the camera coordinates of the other viewpoints C _k in the vicinity of the viewpoint V _R of this subject. Further, each of A and A _Ck show perspective C _k internal parameters of each camera in the vicinity of the viewpoint V _R and the reference viewpoint V _R of the reference. s indicates an enlargement rate or reduction rate. Further, I _Z (m) indicates the depth of the target pixel m in the depth map I _Z (u, v) of the target viewpoint.

On the other hand, the weight E _smooth (m ₁ -m ₂ ) indicates the ease with which the labels of the target pixel and the adjacent pixels match. That is, as the weight E _smooth (m ₁ −m ₂ ) increases, the edge (edge 3) between each of m ₁ and m ₂ is less likely to be included in the ST cut. In this embodiment, the labels of the correct corresponding point and the incorrect corresponding point change at the edge (edge3) where the depth of the target pixel becomes larger than the depth of other pixels adjacent to the target pixel. It is assumed that it is easy to do. Therefore, the weight E _smooth (m) is expressed by the following equation (45) in the present embodiment. In the equation (45), λ _E3 represents the priority of cost, and σ _E2 represents a parameter. Each of I _Z (m ₁ ) and I _Z (m ₂ ) is determined by each of the target pixel m ₁ in the depth map I _Z (u, v) of the target viewpoint and the pixel m ₂ adjacent to the pixel m _1. Depth of.

Then, the miscorresponding point removal unit 123 uses the graph shown in FIG. 12 defined as described above and each of the equations (41) to (45) to change the depth of the target viewpoint by the graph cut method. An ST cut is obtained in the map, and an erroneous corresponding point removal process is performed. The miscorresponding point removal unit 123 determines a pixel corresponding to a node having the same label as the S node as a correct corresponding point by the obtained ST cut, and a pixel corresponding to a node having the same label as the T node. Are determined to be miscorresponding points. The miscorresponding point removal unit 123 removes the pixels determined to be miscorresponding points from the depth map, so that the miscorresponding points in each of the depth map I _Z (u, v) and the correlation value map I _α (u, v). The value of the pixel determined as 0 is set to 0. The miscorresponding point removal unit 123 obtains the above-described ST cut for all viewpoints included in the multi-viewpoint image V, and performs processing for removing the miscorresponding points in the depth map of each viewpoint.

  According to the present embodiment, as described above, the correlation value of the POC function is used for weighting, the ST cut is obtained, and the labeling as to whether each pixel is a positive corresponding point or a false corresponding point by graph cutting. By removing the pixels determined to be miscorresponding points, it is possible to effectively remove miscorresponding points left when the coordinate points of the three-dimensional coordinates are transformed into world coordinates, It is possible to improve the accuracy of restoring the shape of the three-dimensional image restored from the viewpoint depth map.

FIG. 13 is a diagram illustrating the removal of erroneous corresponding points by the graph cut processing in the depth map. FIG. 13 (a) shows an image data captured by the camera located in the target viewpoint V _R. FIG. 13B shows a depth map generated from the image data of FIG. Each of FIG. 13C and FIG. 13D shows a depth map generated from the image data of the camera arranged at the reference viewpoint C _k which is another viewpoint in the vicinity. Create graphs as shown in Fig. 13 (b) to Fig. 12, calculate weights for each edge from Fig. 13 (c) and Fig. 13 (d), and erroneously correspond to pixel m that leaves the edge with T node. Let it be a point. FIG. 13E shows a portion of the pixel m of the erroneous corresponding point as white. FIG. 13 (f) shows a depth map in which the white pixels in FIG. 13 (e) are deleted from FIG. 13 (b).

-Artifact removal processing In this processing method, artifacts are removed using a threshold processing method. This artifact is a structure generated on the surface of an object that does not actually exist in a captured three-dimensional object, which is restored by a multi-viewpoint stereo algorithm. Artifacts tend to be restored in regions that are object boundaries and occlusion boundaries at the target viewpoint. In the multi-viewpoint stereo algorithm, the depth of the coordinate point of the three-dimensional coordinate is estimated for each small area of the matching window, so that the foreground included in the matching window is also included in the pixel that is the background at the object boundary. In order to estimate the depth, it is restored as an artifact.

  Also, unlike the removal of erroneous corresponding points already described, the artifacts that are the targets of the removal processing that will be described are not different in depth difference in depth maps of other viewpoints in the vicinity of the target viewpoint camera position. It is hard to appear. For this reason, the artifact cannot be effectively removed in the already described erroneous corresponding point removal process. For this reason, it is necessary to ensure consistency between the depth map of the camera position of the target viewpoint and the depth map of the camera position of another viewpoint at a position greatly different from the target point.

However, between the viewpoints with greatly different camera positions, the range of the restoration area in the target object that is the target for restoring the three-dimensional shape is easily changed. For this reason, if there is a difference between the depth maps of the target viewpoint and the viewpoint to be compared, the difference is caused by an artifact, or a different area between different viewpoints is restored to a three-dimensional shape. It is difficult to judge whether they are different.
For this reason, in the present embodiment, a plurality of depth maps including artifact pixels are used, and a state in which the consistency between the depth maps is not obtained is classified into cases, thereby the threshold value described below for the artifact pixels. Remove by processing.

  FIG. 14 is a diagram for explaining consistency between depth maps of a plurality of viewpoints when an artifact occurs. In FIG. 14A, an artifact is generated in the three-dimensional shape reproduced by the depth maps of the viewpoint View1, the viewpoint View2, and the viewpoint View3 that are different positions in the camera position. Even in an area observed from the viewpoint View1 as an object boundary or an occlusion boundary (for example, the boundary 502 by the protrusion of the shape Object2), it is not observed as an object boundary or an occlusion boundary from each of the other viewpoint View2 and the viewpoint View3.

  Similarly, even in an area observed from the viewpoint View3 as an object boundary or an occlusion boundary (for example, a boundary 501 due to the protrusion of the shape Object2), it is not observed as an object boundary or an occlusion boundary from each of the other viewpoint View1 and the viewpoint View2. . Further, the artifact has a characteristic that is likely to occur outside the shape as shown in FIG. For this reason, the artifact reproduced from the depth map of the viewpoint View1 is observed closer to the shape reproduced from the depth maps of the other viewpoint View2 and the viewpoint View3. Here, the near side indicates a distance closer to the viewpoint with respect to the depth direction at each viewpoint.

FIG. 14B shows the consistency of the depth maps of the viewpoint View1, the viewpoint View2, and the viewpoint View3, and the relationship between the pixels in the reproduced three-dimensional coordinates. In the following, the consistency and relationship of the coordinate points of coordinate point A, coordinate point A ′, coordinate point B, coordinate point B ′, coordinate point C, coordinate point C ′, and coordinate point D are shown below.
In each of the coordinate point A and the coordinate point A ′, consistency between the coordinate points cannot be achieved by the depth maps of the viewpoint View1 and the viewpoint View2. However, both the coordinate point A of the shape Object1 restored from the depth map of the viewpoint View and the coordinate point A ′ of the shape Object2 restored from the viewpoint View2 are the pixels of the positive corresponding points. In each of the coordinate point A and the coordinate point A ′, an area restored from each of the depth map of the viewpoint View1 and the depth map of the viewpoint View2 is an example. The difference in depth in each of the depth maps used for reproduction of the coordinate point A and the coordinate point A ′ becomes large.

  In addition, in each of the coordinate point B and the coordinate point B ′ and each of the coordinate point C and the coordinate point C ′, consistency between the viewpoint View1 and the viewpoint View2 is not achieved. That is, the coordinate point B restored from the depth map of the viewpoint View1 is a pixel of a positive corresponding point, and the coordinate point B 'restored from the depth map of the viewpoint View3 is an artifact pixel. Similarly, the coordinate point C ′ restored from the depth map of the viewpoint View1 is an artifact pixel, and the coordinate point C restored from the depth map of the viewpoint View3 is a pixel of a positive corresponding point. Each of the coordinate point C ′ and the coordinate point B ′ is generated at an occlusion boundary in each of the viewpoint View1 and the viewpoint View3.

  Here, in each of the coordinate point B and the coordinate point B ′ and each of the coordinate point C and the coordinate point C ′, the regions restored from the depth map of the viewpoint View 1 and the depth map of the viewpoint View 3 are the same. This is an example. The difference in depth in each of the depth maps used for reproducing the coordinate points B and B 'and the coordinate points C and C' becomes small. Further, the depth of the coordinate point B ′ that is an artifact is restored to a position closer to the viewpoint View 1 with respect to the coordinate point B when the coordinate point B that is the positive corresponding point is observed from the restored view View1. The depth of the coordinate point C ′, which is an artifact, is restored to a position closer to the viewpoint View3 with respect to the coordinate point C when observed from the viewpoint View3 where the coordinate point C, which is a positive corresponding point, is restored.

  At the coordinate point D, the difference in depth between the depth map of the viewpoint View 2 and the depth map of the viewpoint View 3 is small, and consistency is obtained as being within a predetermined range. From this result, at the position of the coordinate point D, the same coordinate point D in the same region of the object is restored from the depth map of the viewpoint View2 and the depth map of the viewpoint View3.

Based on the consistency of the depth map in FIG. 14 described above and the correspondence between the coordinate points, a threshold value is set for determining whether to be an artifact or a positive corresponding point. Then, whether artifact removing section 124, the threshold value processing using the threshold value set, the depth map I _{Z (u,} v) of the target viewpoint V _R in is either positive corresponding points or artifacts every pixel Judgment is made. The threshold value to be set is expressed by the following equation (46). The artifact removing unit 124 determines that the pixel m satisfying the following expression (46) is an artifact.

In the above equation (46), each of th ₁ and th ₂ is a parameter that is arbitrarily set. Z 'is a coordinate point in the three-dimensional coordinate of the target of the viewpoint V _R, shows the depth in the three-dimensional coordinates coordinate transformation on the coordinate system of the other viewpoint C _k of neighboring. I _{Z, Ck} (m ′) indicates the depth of the pixel m ′ corresponding to the pixel m in the depth map of another viewpoint C _{k in} the vicinity.
Further, each of the depth Z ′ and the pixel m ′ satisfies the relationship of the formula (44) already described. The viewpoint C _k in the above equation (46) is a viewpoint that satisfies the following equation (47).

In the above equation (47), th ₃ is a parameter that is arbitrarily set. deg _(V R, _{C k)} shows a parallax angle between the other viewpoint _{C k} in the vicinity of the viewpoint _{V R} and viewpoint _{V R.}
Artifact removing section 123, if the other viewpoint C _k adjacent against the viewpoint V _R of interest there are a plurality, and a candidate all other viewpoints adjacent, for each candidate is evaluated by the expression (46) . Then, when one or more candidates satisfy the expression (46), the artifact removal unit 123 determines a pixel in the target depth map as an artifact.
At this time, in (46), the artifact removal unit 123 normalizes the difference between the depth I _{Z, Ck} (m ′) and the depth Z ′ by dividing the difference by the depth I _{Z, Ck} (m ′). However, the pixel m that is equal to or less than the threshold value th ₁ as a parameter is a coordinate point restored as the same pixel as the pixel m ′, and is determined as a positive corresponding point and is not removed.

On the other hand, the artifact removal unit 123 restores a different coordinate point in a region in which the pixel m having a normalization difference equal to or greater than the threshold value th _{2 as a} parameter is different, and determines that it is a positive corresponding point and does not remove it. Then, the artifact removal unit 123 removes the pixel m that exceeds the threshold th ₁ and is less than the threshold th ₂ as a pixel different from the corresponding pixel m ′ in the same region, and removes it as an artifact.
Further, the artifact removal unit 123 locates another viewpoint with a viewing angle equal to or smaller than the threshold th ₃ that is a parameter in the equation (47) with respect to the target viewpoint, and determines artifacts. Because it is not suitable, it is not used as another viewpoint compared with the target viewpoint.

The artifact removing unit 123 evaluates all the pixels on the depth map I _Z (u, v) of all the viewpoints V in the multi-viewpoint image by each of the expressions (46) and (47). Then, the artifact removing unit 123 sets the pixel m that satisfies the conditions of Expressions (46) and (47) to 0 in the depth map I _Z (u, v) and the correlation value map I _α (u, v). To remove artifacts.

FIG. 15 is a diagram illustrating a process of determining whether a target viewpoint pixel is a positive corresponding point or an artifact.
FIG. 15A shows a coordinate point A (pixel m) restored from the depth map of the target viewpoint View2 (V _R ) and a coordinate point A ′ restored from the depth map of the other viewpoint View1 (C _k ). Comparison with is shown. In FIG. 15A, the artifact removing unit 123 rotates the coordinate point A of the viewpoint View2 by the parallax angle deg (V _R , C _k ) and performs coordinate conversion to the coordinate system of the viewpoint View1. The artifact removing unit 123 then converts the depth Z ′ of the coordinate point A from the viewpoint View1 and the depth I _{Z, Ck} (u, v) of the corresponding coordinate point A ′ (pixel m ′) from the viewpoint View1. ) (I _{Z, Ck} (u, v) −Z ′).

Then, the artifact removing unit 123 obtains a normalized difference by dividing the obtained difference (I _{Z, Ck} (u, v) −Z ′) by I _{Z, Ck} (u, v). The artifact removal unit 123 determines whether or not the obtained normalized difference satisfies the above equation (46). Since the normalized difference exceeds the threshold th ₂ , the pixel m at the coordinate point A is a positive corresponding point. Is determined. At this time, the artifact removing unit 123 determines whether or not the viewpoint View1 (C _k ) satisfies the expression (47), and the parallax angle deg (V _R , C _k ) between the viewpoint View2 and the viewpoint View1 is a threshold value. Since it exceeds th3, it is determined that the viewpoint View1 is suitable for determining an artifact used for comparison with the viewpoint View2.

On the other hand, FIG. 15B shows a coordinate point B (pixel m) restored from the depth map of the target viewpoint View3 (V _R ) and a coordinate point restored from the depth map of the other viewpoint View1 (C _k ). Comparison with B ′ is shown. In FIG. 15B, the artifact removing unit 123 rotates the coordinate point B of the viewpoint View3 by the parallax angle deg (V _R , C _k ) and performs coordinate conversion to the coordinate system of the viewpoint View1. The artifact removing unit 123 then converts the depth Z ′ of the coordinate point B from the viewpoint View1 and the depth I _{Z, Ck} (u, v) of the corresponding coordinate point B ′ (pixel m ′) from the viewpoint View1. ) (I _{Z, Ck} (u, v) −Z ′).

Then, the artifact removing unit 123 obtains a normalized difference by dividing the obtained difference (I _{Z, Ck} (u, v) −Z ′) by I _{Z, Ck} (u, v). The artifact removing unit 123 determines whether or not the obtained normalized difference satisfies the above equation (46). Since the normalized difference is equal to or greater than the threshold th ₁ and equal to or less than the threshold th ₂ , the coordinate point B Pixel m is determined to be an artifact. At this time, the artifact removing unit 123 determines whether or not the viewpoint View1 (C _k ) satisfies the expression (47), and the parallax angle deg (V _R , C _k ) between the viewpoint View3 and the viewpoint View1 is a threshold value. th ₃ is exceeded, and it is determined that the viewpoint View1 is suitable for determining the artifact used for comparison with the viewpoint View3.

According to the present embodiment, as described above, by performing the artifact removal processing using the threshold processing on each pixel of the target viewpoint using the reference viewpoints that are separated by a distance obtained by Expression (47). The artifacts that cannot be removed by the already described erroneous corresponding point removal process can be effectively removed, and the restoration accuracy of the shape of the three-dimensional image restored from the depth map of each viewpoint can be improved. .
Further, in the present embodiment, pixels that are determined to be artifacts are removed from at least one of the plurality of reference viewpoints. However, considering the possibility that the corresponding pixels of the reference viewpoint are artifacts, The artifact removing unit 124 may be configured to perform artifact determination by majority vote.

  The mesh model generation device 130 generates a mesh model based on the depth map calculated by the depth map generation device 120 and outputs data representing the generated mesh model to the storage device 200 and the output device 300.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  In addition, a program for realizing each device described above is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed, thereby executing an execution process. May be. Here, the “computer system” may include an OS and hardware such as peripheral devices.

  Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

Further, the “computer-readable recording medium” refers to a volatile memory (for example, DRAM (Dynamic) in a computer system serving as a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Random Access Memory)) that holds a program for a certain period of time is also included.
The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
The program may be for realizing a part of the functions described above.
Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

  DESCRIPTION OF SYMBOLS 100 ... Arithmetic processing apparatus, 110 ... Function calculation apparatus, 111 ... Normalization part (reception part, parallelization part), 112 ... Window setting part, 113 ... Function calculation part, 114 ... Function integration part, 120 ... Depth map generation apparatus , 121 ... Depth map generation unit, 122 ... Filter unit, 123 ... Mismatch point removal unit, 124 ... Artifact removal unit, 130 ... Mesh model generation device, 200 ... Storage device, 300 ... Output device, 400 ... Bus, 110 ... Memory

Claims (8)

A depth map generating unit that generates a depth map from images of different viewpoints using a phase-only correlation method based on a multi-view stereo algorithm;
For each of the edges to be connected from the node corresponding to the pixel of the target viewpoint that is the target for detecting the erroneous corresponding point, for each of the correct corresponding point node of the label of the correct corresponding point and the erroneous corresponding point node of the label of the incorrect corresponding point On the other hand, the first weighting number is given by comparing the pixel with the corresponding pixel of the reference viewpoint which is another viewpoint in the vicinity of the target viewpoint, and the pixel node at the target viewpoint is adjacent to the pixel. The graph is cut with respect to the graph generated by assigning the second weighting number corresponding to the difference in depth between the pixel at the target viewpoint and the adjacent pixel to the edge connecting the node of another adjacent pixel. Each of the nodes in the graph is performed such that the integrated value of each weighting number of the edge through which the cutting line of the graph cut passes is minimized. The allocation to the label of the positive corresponding points or the erroneous corresponding points, and erroneous corresponding point removing unit which removes erroneous corresponding points,
Equipped with a,
The first weighting number is
A first A weighting number for an edge connecting between the node corresponding to the positive corresponding point and the node corresponding to the pixel, and a first B weighting number corresponding to an edge connecting the node corresponding to the pixel and the node corresponding to the erroneous corresponding point And
The first A weighting number is a numerical value corresponding to the peak value of the phase-only correlation function with the other viewpoint, and the higher the correlation value, the higher the numerical value is set,
The 1B weighting number is a numerical value corresponding to the difference in depth of the same pixel from a reference viewpoint that is another viewpoint in the vicinity, and the numerical value is set higher as the difference increases,
The smaller the depth difference from the adjacent pixel node in the viewpoint with the same second weighting number, the higher the numerical value is set.
Depth map generation device according to claim and this. After the erroneous corresponding point removal unit performs the erroneous corresponding point removal, further comprising an artifact removing unit for removing artifacts between the different viewpoints,
The artifact removal unit calculates a difference between the depth of the pixel at the target viewpoint and the depth obtained by converting the pixel at the comparison viewpoint, which is a viewpoint separated from the target viewpoint by a predetermined distance, into the coordinate system of the target viewpoint. The depth map generation apparatus according to claim 1 , wherein the depth map generation apparatus determines and determines whether or not the pixel of the target viewpoint is an artifact by comparing the difference with a preset threshold value. The threshold comprises each of a first threshold and a second threshold;
The artifact removing unit is
If the difference is less than or equal to the first threshold, since the same pixel is restored at the same depth in each of the target viewpoint and the comparative viewpoint, it is determined that the pixel of the target viewpoint is not an artifact,
When the difference exceeds the first threshold and is less than the second threshold, the same pixel is restored at a different depth in each of the target viewpoint and the comparative viewpoint, and thus the target viewpoint Determine that the pixel is an artifact,
When the difference is equal to or greater than the second threshold, the pixel of the target viewpoint is determined not to be an artifact, assuming that the pixel of the target viewpoint is the pixel in a region different from the pixel of the target viewpoint. The depth map generating apparatus according to claim 2 . The artifact removing unit is
The comparative viewpoint is determined not to be used for comparison with the target viewpoint when a parallax angle with respect to the restored pixel of the target viewpoint and the comparative viewpoint is less than a preset third threshold value. The depth map generation device according to claim 2 or claim 3 . When generating a depth map from the images of a plurality of different viewpoints, a pixel matching process between the images is performed by a hierarchical search that gradually increases the resolution of the pixels,
The depth value for each pixel in the depth map is corrected using a weighted median filter for the depth map after completion of the matching process in each hierarchy in the hierarchical search. The depth map generation device according to any one of claims 1 to 4 . The weight value of each of the pixels in the weighted median filter window is set corresponding to the distance from the target pixel, the luminance value of the pixel in the image of the hierarchy, and the peak value of the phase-only correlation function. The depth map generating apparatus according to claim 5 , wherein A depth map generating unit that generates a depth map from images of a plurality of different viewpoints using a phase-only correlation method based on a multi-view stereo algorithm;
The miscorresponding point removal unit corresponds to each pixel of the target viewpoint that is a target for detecting the miscorresponding point for each of the correct corresponding point node of the correct corresponding point label and the miscorresponding point node of the miscorresponding point label. For each edge to be connected from the node, a first weighting number is assigned by comparing the pixel with a corresponding pixel of a reference viewpoint that is another viewpoint in the vicinity of the target viewpoint. A graph generated by assigning a second weighting number corresponding to a difference in depth between a pixel at the target viewpoint and the adjacent pixel to an edge connecting a node and a node of another adjacent pixel adjacent to the pixel. hand, do as the integrated value of each weighting number of the edges cut line of the graph cut for cutting the graph passes is minimized, put on the graph Sorting each of said nodes to said label of said positive corresponding points or the erroneous corresponding points, the corresponding point removal process erroneously removing erroneous corresponding points,
Only including,
The first weighting number is
A first A weighting number for an edge connecting between the node corresponding to the positive corresponding point and the node corresponding to the pixel, and a first B weighting number corresponding to an edge connecting the node corresponding to the pixel and the node corresponding to the erroneous corresponding point And
The first A weighting number is a numerical value corresponding to the peak value of the phase-only correlation function with the other viewpoint, and the higher the correlation value, the higher the numerical value is set,
The 1B weighting number is a numerical value corresponding to the difference in depth of the same pixel from a reference viewpoint that is another viewpoint in the vicinity, and the numerical value is set higher as the difference increases,
The smaller the depth difference from the adjacent pixel node in the viewpoint with the same second weighting number, the higher the numerical value is set.
Depth map generation method which is characterized a call. Computer
A depth map generating means for generating a depth map from images of different viewpoints using a phase-only correlation method based on a multi-view stereo algorithm;
For each of the edges to be connected from the node corresponding to the pixel of the target viewpoint that is the target for detecting the erroneous corresponding point, for each of the correct corresponding point node of the label of the correct corresponding point and the erroneous corresponding point node of the label of the incorrect corresponding point On the other hand, the first weighting number is given by comparing the pixel with the corresponding pixel of the reference viewpoint which is another viewpoint in the vicinity of the target viewpoint, and the pixel node at the target viewpoint is adjacent to the pixel. The graph is cut with respect to the graph generated by assigning the second weighting number corresponding to the difference in depth between the pixel at the target viewpoint and the adjacent pixel to the edge connecting the node of another adjacent pixel. Proceed as integrated value of the weighted number of the edges cut line of the graph cut through is minimized, each of the nodes of the graph The positive corresponding point or distributed to the label of the erroneous corresponding point, the corresponding point removal means false for removing erroneous corresponding points,
Is a program for operating as
The first weighting number is
A first A weighting number for an edge connecting between the node corresponding to the positive corresponding point and the node corresponding to the pixel, and a first B weighting number corresponding to an edge connecting the node corresponding to the pixel and the node corresponding to the erroneous corresponding point And
The first A weighting number is a numerical value corresponding to the peak value of the phase-only correlation function with the other viewpoint, and the higher the correlation value, the higher the numerical value is set,
The 1B weighting number is a numerical value corresponding to the difference in depth of the same pixel from a reference viewpoint that is another viewpoint in the vicinity, and the numerical value is set higher as the difference increases,
A program in which a numerical value is set higher as a difference in depth from an adjacent pixel node in the viewpoint having the same second weighting number is smaller .