JP6494402B2 - Image processing apparatus, imaging apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an alignment technique for generating an image in which a focal length and a viewpoint are changed from a plurality of images having different viewpoints.

There is a method in which the focal length and the viewpoint position are changed using a plurality of images shot at different viewpoint positions, and an image shot at the changed viewpoint (arbitrary viewpoint) is generated by post-processing. Hereinafter, an image obtained from an arbitrary viewpoint is referred to as an “arbitrary viewpoint image”. In the arbitrary viewpoint image generation process, a hidden area (also referred to as occlusion) that is an unphotographed area is generated by being hidden by the subject on the near side. For this reason, an interpolation process for filling the shielding area using a plurality of images taken from different viewpoints is performed.
In the apparatus disclosed in Patent Document 1, an image photographed at a position close to a photographing position corresponding to an arbitrary viewpoint is used in order to interpolate a shielding area generated in the arbitrary viewpoint image.

JP 2013-77882 A

When performing interpolation processing of a shielding area using a plurality of images, if the amount of camera movement to a different viewpoint position is small, the size of the shielding area captured by images with different viewpoints is small. For this reason, the size of the occluded area that can be interpolated is reduced, but the amount of misalignment is small because the number of corresponding points between images is large. On the other hand, if the amount of movement of the camera to a different viewpoint position is large, the size of the occluded area captured by images with different viewpoints is large, but the number of corresponding points is small. For this reason, depending on the shooting situation, there is a possibility that the amount of misalignment becomes large or the alignment cannot be performed.
An object of the present invention is to reduce the amount of misalignment between images when processing a plurality of images obtained at different viewpoint positions, and to increase the shielding area that can be interpolated in the shielding area interpolation process. is there.

  An apparatus according to a first aspect of the present invention includes a first image, input means for inputting a plurality of second images acquired at a different viewpoint from the first image, the first image, and the first image Alignment means for aligning the first and second images using corresponding points with the second image, and detecting a shielding area shielded by a subject area in the first and second images A detecting unit; a plurality of the second images obtained from the input unit; a determining unit that determines the second image to be used for the interpolation processing of the shielding region in the first image; and the first unit The image processing apparatus includes an interpolation unit that acquires an image, the second image determined by the determination unit, and information on the shielding area detected by the detection unit and performs interpolation processing on the shielding area. The determining means is not shielded by the subject area in the second image with respect to the number of corresponding points between images used by the positioning means and the size of the shielding area detected by the detecting means. An image to be used for the interpolation process is determined from the plurality of second images using the information on the size of the region.

  An apparatus according to a second aspect of the present invention includes: a first image; input means for inputting a plurality of second images acquired from a different viewpoint from the first image; the first image; Alignment means for aligning the first and second images using corresponding points with the second image, and calculating a misalignment amount between the first image and the second image A plurality of second images obtained from the calculating means, a detecting means for detecting a shielding area shielded by a subject area in the first and second images, and the input means; Determination means for determining the second image used for interpolation processing of the shielding area, the first image, the second image determined by the determination means, and the shielding detected by the detection means Interpolation processing of the shielding area by acquiring area information It comprises interpolation means for performing. The determination unit is a region that is not shielded by the subject region in the second image with respect to the misalignment amount calculated by the calculation unit and the size of the shielding region detected by the detection unit. An image to be used for the interpolation process is determined from the plurality of second images using the size information.

  According to the present invention, when processing a plurality of images obtained at different viewpoint positions, it is possible to reduce the amount of misalignment between images and increase the shielding area that can be interpolated in the shielding area interpolation process. it can.

It is a figure which shows the structural example of the apparatus which concerns on 1st and 2nd embodiment of this invention. It is a flowchart explaining the process in 1st Embodiment. It is a figure explaining the corresponding point search in 1st Embodiment. It is a figure explaining the hierarchy process which concerns on the corresponding point search in 1st Embodiment. It is a figure explaining the detection of the shielding area in a 1st embodiment. It is explanatory drawing of the imaging position in 1st Embodiment. It is a figure which illustrates the image image | photographed at the reference | standard position of FIG. 6, and each imaging position. It is a figure which illustrates the relationship between the number of corresponding points according to camera movement in 1st Embodiment, and the shielding area currently image | photographed. It is a figure explaining the alignment in 1st Embodiment. It is a figure explaining the interpolation of the shielding area in a 1st embodiment. It is a figure explaining the relationship between the camera position and focal distance in 1st Embodiment. It is a figure explaining the production | generation of the arbitrary viewpoint images in 1st Embodiment. It is a flowchart explaining the process in 2nd Embodiment of this invention. It is a figure explaining the movement amount of the camera according to the distance information to a to-be-photographed object and the magnitude | size of a shielding area | region in 2nd Embodiment. It is a figure explaining the movement amount of the camera in case a background is a multilayer in 2nd Embodiment. It is a figure which shows the structural example of the apparatus which concerns on 3rd Embodiment of this invention. It is a flowchart explaining the process in 3rd Embodiment. It is a figure explaining calculation of the amount of shift of position alignment in a 3rd embodiment.

Hereinafter, embodiments of the present invention will be described by exemplifying an imaging apparatus including an image processing apparatus with reference to the accompanying drawings.
[First Embodiment]
A first embodiment of the present invention will be described with reference to FIGS. In the present embodiment, an image of a different viewpoint (an image of another viewpoint) among the number of points (corresponding points) having a high correlation in a set of image feature points used for alignment between images and an area (shielded area) hidden by the subject. ) To calculate the size of the area being photographed. Based on the calculation result, another viewpoint image used in the interpolation process for filling the occluded region is determined. In the following, the positional relationship of the subject will be described by defining the side closer to the imaging device as the near side and the side away from the imaging device as the back side.

  FIG. 1 is a system diagram illustrating a configuration example of an image processing apparatus according to the present embodiment. In addition to image information at the time of shooting, camera parameters such as distance information from the imaging device to each subject (distance information in the depth direction, hereinafter referred to as subject distance information) and focal length at the time of shooting are acquired. To do. One or more of the functional blocks shown in FIG. 1 may be realized by hardware such as an ASIC or a programmable logic array (PLA), or may be realized by a programmable processor such as a CPU or MPU executing software. May be. Further, it may be realized by a combination of software and hardware. Therefore, in the following description, even when different functional blocks are described as the operation subject, the same hardware can be realized as the subject.

  An image input unit 101 illustrated in FIG. 1 performs input processing of each image data of a reference image and another viewpoint image as a captured image. Each image data is output to the corresponding point search unit 102 and the image determination unit 105. The corresponding point search unit 102 searches (detects) corresponding points in the reference image and the different viewpoint images. Detection information of the corresponding point search unit 102 is output to the occlusion area detection unit 103, the image determination unit 105 (1205 will be described in the second embodiment), and the alignment unit 106, respectively. The occlusion area detection unit 103 detects an occlusion area using information on corresponding points detected by the corresponding point search unit 102. The occlusion area detection unit 103 outputs occlusion area detection information to the image determination unit 105, the alignment unit 106, and the image interpolation unit 107, respectively. The distance information input unit 104 performs subject distance information input processing and outputs subject distance information to the arbitrary viewpoint image generation unit 108.

  The image determination unit 105 acquires image data from the image input unit 101 and determines an image to be used in the occlusion region interpolation processing based on each information acquired from the corresponding point search unit 102 and the occlusion region detection unit 103. The image data determined by the image determination unit 105 is output to the alignment unit 106. The alignment unit 106 deforms the different viewpoint image determined by the image determination unit 105 and performs an alignment process between the reference image and the different viewpoint image. The image data after the alignment process is output to the image interpolation unit 107. The image interpolation unit 107 performs an interpolation process for filling the occlusion area in the reference image using the different viewpoint image data transformed by the alignment unit 106. The image interpolation unit 107 outputs the reference image data subjected to the interpolation processing to the arbitrary viewpoint image generation unit 108. The arbitrary viewpoint image generation unit 108 geometrically deforms the reference image after the interpolation processing using the deformation parameter calculated based on the subject distance information, and generates arbitrary viewpoint image data. The image output unit 109 outputs arbitrary viewpoint image data generated by the arbitrary viewpoint image generation unit 108.

FIG. 2 is a flowchart illustrating an example of processing performed when an imaging apparatus including the image processing apparatus according to the present embodiment performs imaging. The processing shown in each step from S201 to S207 is sequentially executed. Each step is executed mainly by a control unit (CPU) (not shown in FIG. 1) or each block shown in FIG.
S201 [Acquisition of multiple images]
The image input unit 101 acquires a plurality of images captured by the imaging device. The plurality of images are images taken from different viewpoints. For example, the first image taken in time series is set as a reference image. One or more images taken from a different viewpoint from the reference image are set as different viewpoint images. Here, each data of the reference image and the different viewpoint image is input from the image input unit 101 to the corresponding point search unit 102 and the image determination unit 105, respectively.

S202 [corresponding point search]
The corresponding point search unit 102 detects corresponding points in the input reference image and another viewpoint image. The search for corresponding points will be described with reference to FIGS.
FIG. 3 is an explanatory diagram of the corresponding point search, in which the reference image 301 is shown on the left side and another viewpoint image 311 is shown on the right side. Feature points 302 to 306 in the reference image 301 and feature points 312 to 316 in the different viewpoint image 311 are shown. When a feature point in the reference image 301 is denoted by P, a feature point in the different viewpoint image 311 is denoted by Q, and a set of corresponding feature points is represented by (P, Q), in FIG. 3, (302, 312), (303, 313), (304, 314).
In the example of FIG. 3, the feature points 305 and 306 of the reference image 301 and the feature points 315 and 316 of the different viewpoint image 311 indicate that there is no corresponding point (that is, there is no corresponding pixel).

FIG. 4 is an explanatory diagram of hierarchical processing in corresponding point search. FIG. 4 is a diagram showing the reference image 431 and its low resolution image 401 on the left side, and another viewpoint image 441, its low resolution image 411 and a motion vector on the right side. A feature point in each image, search ranges 444 and 413 in another viewpoint image 441 and its low resolution image 411 are shown, respectively.
First, the corresponding point search unit 102 detects feature points in the image. Using the bandpass filter and the edge detector for the reference image 301 and the different viewpoint image 311 in FIG. 3, a region where the signal change is large or a region where the edge strength is large is extracted, and the feature point is detected.

式（１）中の各記号の意味は、以下のとおりである。
ｆ（ｉ，ｊ）：基準画像の画素値
ｇ（ｉ，ｊ）：別視点画像の画素値
Ｒ_ＳＡＤ：相関値
変数ｉ，ｊは、画像に設定される２次元座標系にて座標軸ごとの座標値を表す整数の変数である。基準画像４３１と別視点画像４４１の画素位置ごとに、画素値の差分の絶対値が計算され、その総和を算出することにより相関値Ｒ_ＳＡＤが得られる。ＳＡＤは“Ｓｕｍ ｏｆ Ａｂｕｓｏｌｕｔｅ Ｄｉｆｆｅｒｅｎｃｅ”の略号である。 Next, the corresponding point search unit 102 detects the presence or absence of a correspondence relationship between images by calculating the degree of correlation between feature points in each image. The following equation (1) is used to calculate the correlation value.

The meaning of each symbol in Formula (1) is as follows.
f (i, j): pixel value of the reference image g (i, j): pixel value of the different viewpoint image R _SAD : correlation value The variables i, j are for each coordinate axis in the two-dimensional coordinate system set in the image. It is an integer variable that represents the coordinate value. The absolute value of the difference between the pixel values is calculated for each pixel position of the reference image 431 and the different viewpoint image 441, and the correlation value R _SAD is obtained by calculating the sum. SAD is an abbreviation for “Sum of Absolute Difference”.

  The corresponding point search unit 102 determines a set of feature points whose calculated correlation value is equal to or less than a predetermined threshold as corresponding points. In the present embodiment, the calculation of the correlation value using the absolute difference value of the pixel value will be described as an example. However, the present invention is not limited to this, and other calculation methods may be used.

  With reference to FIG. 4, hierarchical processing using images with different resolutions will be specifically described for feature point detection and corresponding point search using the degree of correlation. First, the corresponding point search process is performed using the low resolution image 401 of the reference image 431 and the low resolution image 411 of the different viewpoint image 441. In the low-resolution image 411 of another viewpoint image, the search range 413 is shown in a solid rectangular frame. Corresponding points determined to have a high correlation within the search range 413 in the low resolution image 411 with respect to the feature points 402 of the low resolution image 401 of the reference image 431 are feature points 412. A motion vector 421 is calculated from the search result of the corresponding points.

  The corresponding point search unit 102 arranges the detected motion vector 421 at a corresponding position in the same size image (431, 441). In the reference image 431 and the different viewpoint image 441, the corresponding points are the feature point 432 and the feature point 442, respectively. That is, the motion vector 421 is a vector having a feature point 432 as a start point and a feature point 442 as an end point. A search range 444 centered on the feature point 442 arranged in the different viewpoint image 441 is shown in a solid-line rectangular frame. Corresponding point search processing is performed in the search range 444. As a result, the feature point 443 within the search range 444 detected as the corresponding point becomes the corresponding point of the feature point 432 in the reference image 431. In this way, the corresponding point search unit 102 performs hierarchical processing using images with different resolutions. Since the search range is large in the low-resolution image, the corresponding points can be detected even when the camera movement amount is large.

As a result of the above processing, as shown in FIG. 3, the correspondence relationship between the feature points is obtained between the reference image 301 and the different viewpoint image 311. That is, the respective correspondence relations are taken with respect to the set of feature points (302, 312), (303, 313), and (304, 314). On the other hand, the feature points 305 and 315 and the feature points 306 and 316 in FIG. Corresponding point searching unit 102, the corresponding relationship information between the detected image, _(referred to as _{F in)} corresponding points, the number of feature points _{(F all} hereinafter) shielding the information area detection unit 103, the image determining unit 105, positioning Output to the unit 106. Next, the process proceeds to S203 in FIG.

S203 [Detection of shielding area]
The occlusion area detection unit 103 detects an occlusion area based on the correspondence between images obtained by the corresponding point search unit 102. With reference to FIG. 5, the detection of the shielding area will be described. FIG. 5 is an explanatory diagram of detection of the occluded region, and shows a reference image 501 and another viewpoint image 511. Feature points 502 to 506 in the reference image 501 and feature points 512 to 516 in the different viewpoint image 511 are shown, respectively. An image of a subject located closest to the front side in the reference image 501 is referred to as a subject image 507, and an image of a subject located closest to the front side in the different viewpoint image 511 is referred to as a subject image 517. A position corresponding to the subject image 507 of the reference image 501 in the different viewpoint image 511 is indicated by a dotted line 518. FIG. 5 illustrates pairs of feature points (502, 512), (503, 513), and (504, 514) corresponding to the reference image 501 and the different viewpoint image 511. Assume that the feature points 505 and 506 and the feature points 515 and 516 are not in correspondence. An image 521 shows an example of an image that is used to calculate the size of the occluded area captured in the different viewpoint image 511.

  The shielding area is an area hidden by the subject. In other words, in the image taken from a certain viewpoint position, the area hidden by the subject area on the near side is the shielding area, but in the image photographed at a different viewpoint position, a part of the shielding area is the subject area. Can be an area that has been captured without being hidden. For this reason, based on the information on the correspondence acquired from the corresponding point search unit 102, the region where the feature points that do not have the correspondence are present is a candidate for the shielding region captured in the different viewpoint image. Since a feature point with low correlation is determined to have no corresponding point, a feature point having a correlation value equal to or greater than a threshold can be obtained by using the equation (1) for calculating the correlation value as a feature point having no correspondence relationship. It can be determined that there is. The shielding area detection unit 103 determines whether or not the shielding area candidate is a shielding area by using the position of the subject area (subject image area) of the reference image in another viewpoint image. Specifically, in the case of FIG. 5, since the feature point 515 of the different viewpoint image 511 is included in the region indicated by the dotted line 518, it is determined that the reference image 501 is within the shielding region. On the other hand, since the feature point 516 of the different viewpoint image 511 is not included in the area indicated by the dotted line 518, it is determined to be outside the shielding area.

In the image 521, a dotted line 522 indicates a subject area of the reference image, and a captured shielding area 523 is an area indicated by hatching. The size of the occlusion area 523 is calculated as the sum of feature points that have no corresponding relationship on the dotted line 522 that indicates the position of the subject area of the reference image. The occlusion area detection unit 103 calculates the size of the occlusion area (denoted as S _in ) captured in the different viewpoint image, and also indicates the size of the entire occluded area (denoted as S _all ), that is, the dotted line 522. Calculate the sum of the feature points within the range. The shielding area detection unit 103 outputs detection information (S _in , S _all ) relating to the shielding area to the image determination unit 105, the alignment unit 106, and the image interpolation unit 107, respectively. Next, the process proceeds to S204 of FIG.

S204 [Determination of image used for interpolation]
The image determination unit 105 determines another viewpoint image to be used in the interpolation process for filling the occluded region based on the information obtained from the corresponding point search unit 102 and the occluded region detection unit 103. The image determination process will be described with reference to FIGS. FIG. 6 to FIG. 8 are diagrams for explaining a process for determining another viewpoint image used for interpolation of the occluded region.

  FIG. 6 illustrates the positional relationship between the subjects 605 to 607 and a plurality of shooting positions 601 to 604. Assume that the subjects 605, 606, and 607 are positioned in the order closer to the imaging device in the depth direction. FIG. 7 illustrates the reference image 611 photographed at the photographing position 601 and the different viewpoint images 612, 613, and 614 photographed at the photographing positions 602, 603, and 604, respectively. FIG. 8 is a diagram for explaining the relationship between the number of corresponding points at each photographing position and the shielding area being photographed. The relationship between the number of corresponding points regarding the images photographed at each of the photographing positions 601 to 604 and the shielding area being photographed is illustrated. The horizontal axis represents the number of corresponding points, and the vertical axis represents the size of the shielding area being photographed. The shooting position 602 corresponds to the point 702, the shooting position 603 corresponds to the point 703, and the shooting position 604 corresponds to the point 704.

  The occlusion area generated in the reference image 611 in FIG. 7 is an area hidden by the subject 605 in FIG. In practice, it is necessary to consider the shielding area 609 generated by the subject 606, but in order to make the explanation easier to understand in the following processing, description will be made using one shielding area.

  By sequentially moving the camera from the shooting position 601 of the reference image 611 to the shooting positions 602 to 604 and to the right side in FIG. 6, it is possible to take an area corresponding to the shielding area in the reference image 611. In this case, the different viewpoint image 612 photographed at the photographing position 602 has a small movement amount of the camera, and thus the size of the shielded area 615 being photographed is small. That is, only a part of the area corresponding to the shielding area in the reference image 611 is captured. In the different viewpoint image 613 photographed at the photographing position 603, the size of the shielded area 615 being photographed becomes large. In the different viewpoint image 614 captured at the imaging position 604, the moving amount of the camera is further large, and therefore the size of the captured shielding area 617 is increased. Thus, it can be seen that the larger the movement amount of the camera with respect to the shooting position 601, the larger the size of the shielding area shot in the different viewpoint image.

  On the other hand, in the case of the different viewpoint image 612 photographed at the photographing position 602, the moving amount of the camera is small, and the range of images that match the reference image 611 is large. For this reason, the number of corresponding points between the reference image 611 and the different viewpoint image 612 is large. On the other hand, in the case of the different viewpoint image 614 photographed at the photographing position 604, the movement amount of the camera is large, and the range of the image that matches the reference image 611 is small. For this reason, the number of corresponding points between the reference image 611 and the different viewpoint image 614 is small. Thus, it can be seen that the smaller the amount of movement of the camera with respect to the shooting position 601, the greater the number of corresponding points between the shot images, and thus the smaller the shift in alignment.

  When the above results are shown in FIG. 8, the number of corresponding points is relatively large at the photographing position 602, as indicated by a point 702, and the size of the shielded area being photographed is small. At the photographing position 603, as indicated by a point 703, the number of corresponding points is small compared to the point 702, and the size of the shielded area being photographed is large. At the photographing position 604, as indicated by a point 704, the number of corresponding points is smaller than that at the point 703, and the size of the shielding area being photographed is large. As an image used for interpolating the shielding area, an image in which the size of the shielding area being photographed is large and the amount of misalignment is small is preferable. However, since the number of corresponding points and the size of the occluded area being photographed are in a trade-off relationship, an image that satisfies both conditions cannot be selected. Therefore, a process of selecting an image in which the number of corresponding points and the size of the captured shielding area are balanced, such as a point 703 shown in FIG. 8, is executed. As a result, the interpolation process can be performed using an image in consideration of the size of the shielded region to be interpolated and the amount of misalignment. The image determination unit 105 performs determination using the following equation (2) when selecting an image.

式（２）中の各記号の意味は以下のとおりである。
Ｆ_ｉｎ：検出された対応点数
Ｆ_ａｌｌ：画像の全特徴点数
Ｓ_ｉｎ：撮影されている遮蔽領域の大きさ
Ｓ_ａｌｌ：遮蔽領域の全体の大きさ
Ｒ_ｏｕｔ：評価値
式（２）に示すαとβは、画像間での対応点数の比率と、撮影されている遮蔽領域の大きさの比率に対して乗算される、重み付け係数をそれぞれ表している。つまり画像決定部１０５は、重み付け演算によって式（２）から評価値Ｒ_ｏｕｔを算出し、当該評価値が最大となる画像（別視点画像）を決定する。本例では説明の便宜上、画像の選択と同時に決定処理が行われるものとするが、選択された画像を所定の基準にしたがってチェックした上で画像の決定処理を行ってもよい。次に図２のＳ２０５に処理を進める。

The meaning of each symbol in the formula (2) is as follows.
F _in : the number of detected corresponding points F _all : the number of all feature points of the image S _in : the size of the shielded area being photographed S _all : the overall size of the shielded area R _out : the evaluation value α shown in Expression (2) And β respectively represent weighting coefficients that are multiplied by the ratio of the number of corresponding points between the images and the ratio of the size of the shielded area being shot. That is, the image determination unit 105 calculates the evaluation value R _out from Expression (2) by weighting calculation, and determines an image (another viewpoint image) that maximizes the evaluation value. In this example, for convenience of explanation, it is assumed that the determination process is performed simultaneously with the selection of the image. However, the image determination process may be performed after checking the selected image according to a predetermined standard. Next, the process proceeds to S205 in FIG.

・ S205 [Alignment]
The alignment unit 106 deforms the different viewpoint image determined by the image determination unit 105 using the information from the corresponding point search unit 102, and performs alignment processing. FIG. 9 is an explanatory diagram of the alignment process. FIG. 9A shows the correspondence between the reference image 801 and the different viewpoint image 811, and FIG. 9B shows the reference image 821 and the different viewpoint image 831. The correspondence between is shown.

式（３）中の各記号の意味は以下のとおりである。
（ｘ，ｙ）：別視点画像の座標
（ｘ^*，ｙ^*）：別視点画像の射影変換後の座標
ｈ_ｉｊ：射影変換係数（ｉ,ｊ＝１〜３） The alignment unit 106 calculates a transformation matrix using a set of corresponding points in order to deform the different viewpoint image 811. The feature points 802 to 808 in the reference image 801 in FIG. 9A correspond to the feature points 812 to 818 in the different viewpoint image 811, respectively. By using this correspondence, the process of calculating the projective transformation coefficient shown in the following equation (3) is executed. The alignment process is executed by transforming the different viewpoint image using Expression (3).

The meaning of each symbol in Formula (3) is as follows.
(X, y): coordinates of another viewpoint image (x ^* , y ^* ): coordinates after projective transformation of another viewpoint image h _ij : projective transformation coefficient (i, j = 1 to 3)

射影変換係数の未知数は９個であるが、射影変換係数が定数倍された場合でも変換後の結果は影響を受けないため、実際の自由度は８となる。そのため、ｈ_３３の値を１として分母の式を両辺にかけると、式（４）は下式（５）で表される。 When formula (3) is expanded to calculate the projective transformation coefficient shown in formula (3), the following formula (4) is obtained.

The number of unknown projective transformation coefficients is 9, but even if the projective transformation coefficient is multiplied by a constant, the result after transformation is not affected, so the actual degree of freedom is 8. Therefore, when h ₃₃ is set to 1 and the denominator formula is applied to both sides, formula (4) is expressed by the following formula (5).

式（５）は連立一次方程式であるため、４組以上の対応点を与えて、最小二乗法を用いて解くことにより、射影変換係数の値をそれぞれ算出することができる。具体的には、図９（Ｂ）に示すように、実線８２６，８３６の矩形枠内の範囲（遮蔽領域の周辺域を含む）内にある４組の対応点を使用する。対応点の組は（８０２，８１２）、（８０３，８１３）、（８０４，８１４）、（８０５，８１５）であり、これらの対応点が優先的に使用される。つまり遮蔽領域の補間で使用する別視点画像の領域は、撮影されている遮蔽領域のみである。このため、遮蔽領域周辺の位置合わせのズレ量が重要となる。したがって遮蔽領域周辺の対応点を優先して用いることで、補間時の位置合わせのズレ量を小さくすることができる。次に図２の２０６に処理を進める。

Since Equation (5) is a simultaneous linear equation, it is possible to calculate the values of projective transformation coefficients by giving four or more pairs of corresponding points and solving using the least square method. Specifically, as shown in FIG. 9B, four sets of corresponding points in the range (including the peripheral area of the shielding area) within the rectangular frame of solid lines 826 and 836 are used. Corresponding point pairs are (802, 812), (803, 813), (804, 814), (805, 815), and these corresponding points are preferentially used. That is, the area of the different viewpoint image used for the interpolation of the occluded area is only the occluded area being photographed. For this reason, the amount of misalignment around the shielding area is important. Therefore, by using the corresponding points around the shielding area with priority, the amount of misalignment during interpolation can be reduced. Next, the process proceeds to 206 in FIG.

S206 [Image interpolation]
The image interpolation unit 107 performs an occlusion region interpolation process using the different viewpoint image after deformation input from the alignment unit 106 and the detection result of the occlusion region detection unit 103. FIG. 10 is a diagram for explaining the interpolation of the shielding area. 10A illustrates a reference image 901, FIG. 10B illustrates another image 911 after deformation, and FIG. 10C illustrates an image 921 after interpolation of the occluded region. A black area 902 indicates a shielding area. FIG. 10D shows the amount of change in the composition ratio used for interpolation with a graph line 931. The horizontal axis represents the amount of misalignment, and the vertical axis represents the amount of change in the composition ratio. As an example, the graph line 931 is shown as a straight line rising to the right.

  The image interpolation unit 107 performs an interpolation process by synthesizing the occlusion area captured in the modified different viewpoint image with the occlusion area detected in the reference image. A region 912 illustrated in FIG. 10C indicates a shielding region that is captured in the different viewpoint image 911. In the image 921 after interpolation, a region 922 indicates a region where another viewpoint image is synthesized. A black area 923 shown in the image 921 is an area that has not been captured in the different viewpoint image 911 and remains as a shielding area.

  In the interpolation process, the misalignment amount may increase depending on the different viewpoint image determined by the image determination unit 105. For example, in a different viewpoint image that has been shot, there may be a case where shooting is performed in a scene where there are few features of the subject in the background, and there are cases where the number of detected feature points is small. Another example is a case where the subject is far away and parallax is not easily generated, and the shielding area is not photographed unless the camera is largely moved. In such a case, the number of corresponding points decreases, and the amount of misalignment may increase. Therefore, in this embodiment, image interpolation processing is performed in consideration of the amount of misalignment.

式（６）中の各記号の意味は以下のとおりである。
γ ：重み付け係数（合成比率は「γ：１−γ」である。０≦γ≦１）
Ｉｎ１：基準画像の画素値
Ｉｎ２：変形後の別視点画像の画素値
Ｏｕｔ：補間後の画素値
γ値については、位置合わせのズレ量に基づいて決定され、画像補間部１０７はγ値を変化させることで、基準画像と別視点画像との合成比率を変更する。対応点数が少ない場合には、位置合わせのズレを目立ちにくくするための設定が行われる。すなわち画像補間部１０７は、図１０（Ｄ）に示すように、グラフ線９３１に基づいてγ値の変化量を小さく設定することで、基準画像と別視点画像とを合成する範囲を大きくする。位置合わせのズレ量が大きいほど、γ値の変化量が相対的に大きく設定される。次に図２のＳ２０７に処理を進める。 The image interpolation unit 107 performs weighted addition (weighted addition) using the following equation (6).

The meaning of each symbol in Formula (6) is as follows.
γ: weighting coefficient (composition ratio is “γ: 1−γ”, 0 ≦ γ ≦ 1)
In1: Pixel value of the reference image In2: Pixel value of another viewpoint image after transformation Out: Pixel value after interpolation The γ value is determined based on the amount of misalignment, and the image interpolation unit 107 changes the γ value. By doing so, the composition ratio of the reference image and the different viewpoint image is changed. When the number of corresponding points is small, a setting is made to make the misalignment inconspicuous. That is, as shown in FIG. 10D, the image interpolation unit 107 sets a small amount of change in the γ value based on the graph line 931 to increase the range in which the reference image and the different viewpoint image are combined. The larger the misalignment amount, the larger the change amount of the γ value is set. Next, the process proceeds to S207 in FIG.

S207 [Generation of arbitrary viewpoint image]
The arbitrary viewpoint image generation unit 108 generates an arbitrary viewpoint image of the captured image using the interpolated image input from the image interpolation unit 107. In this embodiment, as an example of an arbitrary viewpoint image, the size of the main subject (focused subject) is fixed or fixed at a predetermined variable magnification, and then at an arbitrary shooting position and an arbitrary focal length of the shooting optical system. A captured image is virtually generated. More specifically, the background region other than the main subject is subjected to deformation processing based on a virtual shooting position (viewpoint position) and a focal length determined by a user instruction or image analysis. An arbitrary viewpoint image is generated by synthesizing the transformed background image and main subject region image. At this time, the background area after the deformation may include a shielding area that is shielded by the main subject area before the deformation. In that case, the interpolated image generated in step S206 is deformed and synthesized with the image of the main subject area, thereby generating an image with no missing images (or fewer missing images). In S207, deformation parameters necessary for generating an arbitrary viewpoint image are calculated based on the subject distance information input from the distance information input unit 104. The object distance information calculation process may use an active method using a laser or the like, or a passive method using a stereo method or the like, and is not limited to a specific means. FIG. 11 is a schematic diagram illustrating the relationship between the camera position and the focal length at an arbitrary viewpoint. In FIG. 11, in order to make the explanation of the deformation parameter calculation method easy to understand, it is assumed that a focused subject (main subject) and a background are photographed as two subjects. A focused subject is a subject that is focused during shooting. In the optical axis of the imaging system, the right side in FIG. 11 is the subject side, and the left side is the imaging surface side. Also, an area that is the same distance as the in-focus distance with respect to the in-focus subject is defined as the in-focus subject area. Each symbol in FIG. 11 indicates the following contents.

y ₁ : Size of focused subject y ₂ : Background size f _w : Focal length at the time of shooting y _w1 : Size on the image plane at the time of shooting corresponding to y ₁ y _w2 : Shooting corresponding to y ₂ Size on the image plane at the time S _w1 : Distance to the focused subject at the time of shooting S _w2 : Distance to the background at the time of shooting f _T : Focal length at an arbitrary viewpoint y _T1 : When shooting at an arbitrary viewpoint Size on the image plane corresponding to y ₁ y _T2 : Size on the image plane corresponding to y ₂ when shooting from an arbitrary viewpoint S _T1 : Subject distance at the arbitrary viewpoint (distance to the focused subject)
S _T2 : Subject distance at an arbitrary viewpoint (distance to the background)
t ₃ : Amount of camera movement relative to the time of shooting

式（７）おいて、背景の変倍率をＮ＝ｙ_Ｔ２／ｙ_ｗ２により定義する。任意視点画像生成部１０８は、撮影時の焦点距離ｆ_ｗと任意視点での焦点距離ｆ_Ｔ、被写体距離情報Ｓ_ｗ１、Ｓ_ｗ２により、背景の変倍率Ｎを算出する。任意視点画像生成部１０８は、算出した変倍率Ｎ（変形パラメータ）を用いて背景領域の幾何変形を行い、合焦被写体像と幾何変形処理後の背景画像とを合成して任意視点画像を生成する。その際には、撮影されなかった遮蔽領域が存在する場合、変倍率の値によっては、補間されていない領域が画像に現れることになる。図１２は任意視点画像の生成を説明する図である。基準画像１１０１、補間後の背景領域の画像１１１１、生成された任意視点画像１１２１を例示する。 When generating an image in which the background image is enlarged without changing the size of the focused subject image, the size y _w1 on the image plane of the focused subject at the time of shooting is adjusted when shooting from an arbitrary viewpoint. The size y _{T1 of the} focused subject on the image plane is the same. For this reason, the following formula (7) is established as the scaling factor of the background from the similarity relationship of the triangle and the lens formula.

In equation (7), the scaling factor of the background is defined by N = y _T2 / y _w2 . The arbitrary viewpoint image generation unit 108 calculates the background scaling factor N based on the focal length f _w at the time of shooting, the focal length f _T at the arbitrary viewpoint, and the subject distance information S _w1 and S _w2 . The arbitrary viewpoint image generation unit 108 performs geometric deformation of the background region using the calculated scaling factor N (deformation parameter), and generates an arbitrary viewpoint image by synthesizing the focused subject image and the background image after the geometric deformation processing. To do. In this case, if there is a shielded area that has not been photographed, an uninterpolated area appears in the image depending on the value of the scaling factor. FIG. 12 is a diagram for explaining generation of an arbitrary viewpoint image. The reference image 1101, the background region image 1111 after interpolation, and the generated arbitrary viewpoint image 1121 are illustrated.

  In the image 1111 of the background region after interpolation shown in FIG. 12, dotted lines 1112 and 1113 are regions interpolated from different viewpoint images, and the black region 1114 is a region that is not interpolated. Therefore, the arbitrary viewpoint image generation unit 108 changes the black area 1114 in the image, that is, the magnification ratio at which the occlusion area does not appear, and generates the arbitrary viewpoint image 1121 by performing geometric deformation processing using the changed magnification ratio. To do.

In the present embodiment, an image in which both are balanced is selected and used based on the size of the occluded area captured in the different viewpoint image and the number of corresponding points used for alignment. In the determination process of the image used for interpolation, the first ratio occupied by the number of detected corresponding points with respect to the total number of feature points (total number) of the image, and the shielding area captured with respect to the entire size of the shielding area An evaluation value is calculated based on the second ratio occupied by the size of. According to the present embodiment, by using an image having the maximum evaluation value, the interpolation processing can be performed so that the amount of misalignment is small and the shielding area that can be interpolated is large.
Further, in the present embodiment, after generating an interpolated image obtained by performing interpolation from another viewpoint image with respect to the occluded region of the reference image, the interpolated image is generated based on virtual shooting conditions for generating an arbitrary viewpoint image. . Not limited to such an example, an arbitrary viewpoint image may be generated using a reference image, and interpolation may be performed using a different viewpoint image for the occluded region generated at this time.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In this embodiment, in order to determine an image to be used in the interpolation process of the occluded area, in addition to the process of the first embodiment, priority is given to the number of corresponding points between the images and the size of the occluded area captured by another viewpoint image. The process of changing the degree and determining the image is performed. In the following, by using the same reference numerals as those used in the first embodiment, the detailed description thereof will be omitted, and differences will be mainly described. The way of omitting such description is the same in the embodiments described later.

  In FIG. 1, the difference from the first embodiment is that subject distance information output from the distance information input unit 204 is also input to the image determination unit 1205, and the image determination unit 1205 determines an image used for interpolation processing. It is to be used as information.

Image processing in this embodiment will be described with reference to the flowchart of FIG. The difference from FIG. 2 is the priority changing process S1304 for determining an image. Therefore, description of S201 to S207 is omitted, and only S1304 will be described.
In step S <b> 1304 subsequent to step S <b> 203, the image determination unit 1205 changes the priority for determining an image based on each piece of information acquired from the corresponding point search unit 202, the occlusion area detection unit 203, and the distance information input unit 204. The priority is the priority of the number of corresponding points between images and the size of the occluded area captured by the different viewpoint image, which are used to determine another viewpoint image used in the occlusion area interpolation process. The amount of camera movement required to interpolate the shielding area varies depending on the positional relationship and size of the subject photographed to generate the arbitrary viewpoint image. For this reason, the priority of the number of corresponding points between images and the size of the shielding area being photographed is changed in consideration of subject information. The amount of camera movement necessary to interpolate the shielding area will be described with reference to FIGS. 14 and 15 are diagrams showing the amount of camera movement depending on the positional relationship and size of the subject. 14 and 15, regions 1401, 1411, 1501, 1511, 1601, and 1602 indicated by dotted lines indicate shielding regions by the subject.

  FIGS. 14A and 14B are diagrams for explaining the movement amount of the camera according to the distance information to the subject. FIG. 14A shows a case where the distance from the camera to the subject 1403 is short, and FIG. 14B shows a case where the distance from the camera to the subject 1404 is long. In FIG. 14A, the shielding region 1401 can be photographed even when the amount of movement of the camera is small as indicated by the photographing position 1402. On the other hand, in the case of FIG. 14B, since it is difficult for parallax to occur, the size of the shielding area that can be photographed becomes small at the photographing position 1412 close to the reference position. Therefore, all the shielding areas cannot be imaged unless the movement amount of the camera is increased and the camera is moved from the reference position to the imaging position 1413.

  FIGS. 14C and 14D are diagrams illustrating the amount of camera movement according to the size of the shielding area. FIG. 14C shows a case where the size of the shielding area by the subject 1503 is small, and FIG. 14D shows a case where the size of the shielding area by the subject 1504 is large. In FIG. 14C, as shown in the shooting position 1502, the shielding region can be shot even if the camera movement amount is small. On the other hand, in FIG. 14D, the shielding area that can be photographed at the photographing position 1512 close to the reference position is small. Therefore, all the shielding areas cannot be imaged unless the movement amount of the camera is increased and the camera is moved from the reference position to the imaging position 1513.

FIG. 15 is a diagram for explaining the movement amount of the camera when the background is multilayer. As shown in FIG. 15, when there are a plurality of subjects in the background, when shooting is performed at the shooting position 1603 close to the reference position, the shielding region 1601 by the subject 1605 can be photographed, but the shielding region 1602 by the subject 1606 can be photographed. Can not. Therefore, in order to capture all of the shielding areas 1601 and 1602, it is necessary to move the camera to an imaging position 1604 that is distant from the reference position. Since the size of the shielding area increases as the number of subjects in the background increases, all the shielding areas cannot be captured unless the camera movement amount is increased. In view of such a tendency, the image determination unit 1205 determines which of the corresponding number of points between images and the size of the captured shielding area is based on the subject distance information, the size of the shielding area, and the number of subjects. Change the priority of selecting images with priority. As a method of changing the priority, the values of α and β are changed in the above equation (2). For example, when there is a multi-layered background as shown in FIG. 15, the size of the shielding area to be interpolated becomes large. Therefore, the image determination unit 1205 increases the priority with respect to the size of the shielded area by setting the β value, which is a weighting coefficient for the ratio of the size of the shielded area being photographed, to be larger than the α value. On the contrary, by making the α value larger than the β value, the priority with respect to the ratio of the number of corresponding points between images can be increased. The image determination unit 1205 calculates the evaluation value R _out from the equation (2) by weighting calculation according to the priority, and determines the image having the maximum evaluation value as another viewpoint image to be used in the interpolation process.

  In the present embodiment, by changing the number of corresponding points between images and the priority of the size of the occluded area captured by another viewpoint image, the image used in the interpolation process of the occluded area can be represented by distance information in the depth direction. It can be determined according to certain subject distance information.

[Third Embodiment]
Next, a third embodiment of the present invention will be described. In the present embodiment, an image to be used for the interpolation processing of the shielding area is determined based on the amount of misalignment and the size of the shielding area being photographed. The difference from the first embodiment is that the number of corresponding points between images is changed in the equation (2).

  FIG. 16 is a system diagram illustrating a configuration example of the image processing apparatus according to the present embodiment. The difference from the configuration example shown in FIG. 1 is that a positional deviation amount calculation unit 1705 is provided. The positional deviation amount calculation unit 1705 acquires information on corresponding points from the corresponding point search unit 1702, and calculates a temporary conversion coefficient using the information. The misregistration amount calculation unit 1705 calculates the misregistration amount based on the provisional conversion coefficient, and outputs the misregistration amount as a calculation result to the image determination unit 1706. Note that the units indicated by 1701 to 1704 correspond to the units indicated by 101 to 104 in FIG. 1, and the units indicated by 1707 to 1710 correspond to the units indicated by 106 to 109 in FIG. Therefore, detailed description thereof will be omitted, and differences will be mainly described.

FIG. 17 is a flowchart illustrating an example of image processing in the present embodiment. The difference from the image processing example shown in FIG. 2 is S1804 for performing a positional deviation amount calculation process and S1805 for performing a process for determining an image to be used for interpolation. Therefore, only S1804 and S1805 will be described below.
In step S1804 following step S203, the positional deviation amount calculation unit 1705 calculates a provisional conversion coefficient using the corresponding points input from the corresponding point search unit 1702, and the registration deviation amount based on the provisional conversion coefficients. Is calculated. Processing for calculating the amount of misalignment will be described with reference to FIG.

18A illustrates the reference image 1901, FIG. 18B illustrates the different viewpoint image 1911, and FIG. 18C illustrates the deformed different viewpoint image 1921. In the different viewpoint image 1921 after deformation, the subject positions 1922 and 1923 and the detected feature points 1924 are shown, and the subject positions 1925 and 1926 in the reference image 1901 are shown by dotted lines, and the detected feature points 1927 are shown.
Using the formula described in S205 in FIG. 2, four or more sets are selected from the input corresponding points, and a temporary conversion coefficient calculation process is performed. Then, another viewpoint image is deformed using the calculated conversion coefficient. As a result, the positions of the subjects in the different viewpoint image 1921 after the deformation are moved to positions 1922 and 1923. The positional shift at that time is a difference between the subject positions 1925 and 1926 (see the dotted line) in the reference image 1901 and the subject positions 1922 and 1923 in the different viewpoint image 1921 after the deformation, and this difference becomes a misalignment. . Therefore, the inter-pixel distance between the feature point 1924 in the different viewpoint image 1921 and the feature point 1927 in the reference image 1901 is defined as a deviation amount. The following formula (8) is used for the calculation process of the inter-pixel distance.

式（８）中の各記号の意味は以下のとおりである。
（ｘ_ｉ，ｙ_ｉ）：基準画像の特徴点座標
（ｘ_ｉ ^*，ｙ_ｉ ^*）：変形された別視点画像の特徴点座標
Ｅ_ｉ：ｉ番目の特徴点における画素間距離
本実施形態では、画素間距離としてユークリッド距離を用いるが、この算出方法に限定される訳ではなく、他の方法を用いてもよい。

The meaning of each symbol in Formula (8) is as follows.
(X _i , y _i ): feature point coordinates (x _i ^* , y _i ^* ) of the reference image: feature point coordinates E _{i of the} modified different viewpoint image: inter-pixel distance at the i-th feature point Although the Euclidean distance is used as the inter-pixel distance, the present invention is not limited to this calculation method, and other methods may be used.

式（９）中の各記号の意味は以下のとおりである。
Ｅ_ｉ：ｉ番目の特徴点における画素間距離
Ｅ_ａｖｇ：画素間距離の平均
Ｎ：特徴点の総数
σ：画素間距離の標準偏差 Next, the positional shift amount calculation unit 1705 calculates the maximum value of the inter-pixel distance E _i as a representative value of the shift amount for each image. In the calculation process, in order to calculate the maximum value of the inter-pixel distance excluding outliers and noise components, the degree of value variation is taken into account using the standard deviation. The standard deviation (denoted as σ) of the inter-pixel distance calculated at each feature point is calculated using the following equation (9).

The meaning of each symbol in Formula (9) is as follows.
E _i : Inter-pixel distance at the i-th feature point E _avg : Average inter-pixel distance N: Total number of feature points σ: Standard deviation of inter-pixel distance

The positional deviation amount calculation unit 1705 determines the maximum value of the inter-pixel distance within the range of 3σ. FIG. 18D illustrates a graph 1931 that represents the standard deviation of the inter-pixel distance. The horizontal axis represents the inter-pixel distance, and the vertical axis represents the number (frequency). A maximum distance between pixels in the range of 3σ in the graph 1931 is indicated by a point 1932. The value at point 1932 is determined as the amount of deviation. That position shift amount calculation unit 1705, the maximum value of the inter-pixel distance in the range of 3 [sigma], is calculated as the representative value of the positional deviation amount in each image (referred to as E _in), and outputs the image determining unit 1706.

式（１０）中の各記号の意味は以下のとおりである。
Ｅ_ｉｎ：位置合わせのズレ量の代表値
Ｓ_ｉｎ：撮影されている遮蔽領域の大きさ
Ｓ_ａｌｌ：遮蔽領域の全体の大きさ
Ｒ_ｏｕｔ：評価値
α、βは、位置合わせのズレ量の逆数と撮影されている遮蔽領域の大きさの比率に対する重み付け係数をそれぞれ表している。画像決定部１７０６は、式（１０）で算出した評価値Ｒ_ｏｕｔが最大となる画像を決定し、当該画像の情報を位置合わせ部１７０７に出力する。 In step S1805 of FIG. 17, the image determination unit 1706 determines another viewpoint image to be used in the occlusion region interpolation processing based on the information acquired from the occlusion region detection unit 1703 and the positional deviation amount calculation unit 1705. At that time, the evaluation value is calculated using the following equation (10).

The meaning of each symbol in the formula (10) is as follows.
E _in : representative value of misregistration amount S _in : size of shielded area being photographed S _all : total size of shielded area R _out : evaluation value α and β are reciprocals of misalignment amount of alignment And weighting coefficients for the ratio of the size of the shielded area being photographed. The image determination unit 1706 determines an image having the maximum evaluation value R _out calculated by Expression (10), and outputs information on the image to the alignment unit 1707.

  In the present embodiment, an image to be used in the shielding area interpolation processing is determined based on the amount of misalignment and the size of the shielding area being photographed. That is, the evaluation value is calculated based on the reciprocal of the amount of misalignment (representative value) and the ratio of the size of the shielded area being captured to the overall size of the shielded area. By using the image having the maximum evaluation value, the interpolation processing is performed so that the amount of misalignment is small and the shielding area that can be interpolated is large.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 101 ... Image input part, 102 ... Corresponding point search part, 103 ... Occlusion area detection part, 104 ... Distance information input part, 105 ... Image determination part, 106 ... Positioning part 107: Image interpolation unit, 108: Arbitrary viewpoint image generation unit, 109: Image output unit

Claims (18)

An input means for inputting a first image and a plurality of second images acquired at a different viewpoint from the first image;
Alignment means for aligning the first and second images using corresponding points between the first image and the second image;
Detecting means for detecting a shielding area shielded by a subject area in the first and second images;
Determining means for obtaining a plurality of the second images from the input means and determining the second image to be used for interpolation processing of the shielding area in the first image;
Interpolating means for acquiring the first image, the second image determined by the determining means, and information on the shielding area detected by the detecting means and performing interpolation processing of the shielding area,
The determining means is not shielded by the subject area in the second image with respect to the number of corresponding points between images used by the positioning means and the size of the shielding area detected by the detecting means. An image processing apparatus for determining an image to be used for interpolation processing from a plurality of the second images using information on the size of a region.   The determination unit acquires distance information in the depth direction relating to the first and second images, and sets a priority for the number of corresponding points between the images used by the alignment unit and information on the size of the shielding area. The image processing apparatus according to claim 1, wherein an image to be used for interpolation processing is determined from a plurality of the second images.   The alignment means performs alignment with the first and second images by deforming the second image according to corresponding points between the first image and the second image. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   The image processing apparatus according to claim 3, wherein the alignment unit performs alignment by preferentially using the corresponding points around the shielding area in the first image.   The detection means detects a region that is shielded by a subject region in the first image and that is not shielded by the subject region in the second image. The image processing apparatus according to any one of 1 to 4.   The image processing apparatus according to claim 5, wherein the detection unit detects, as the shielding area, an area in the second image that does not have a corresponding point with the first image.   The determination means is configured to calculate the second ratio with respect to a first ratio occupied by the detected number of corresponding points with respect to a total number of feature points in the first image and a size of the shielding area in the first image. An evaluation value is calculated from the second ratio occupied by the size of the area not covered by the subject area in the image of the image, and an image to be used for interpolation processing from the plurality of the second images using the evaluation value. The image processing apparatus according to claim 1, wherein the image processing apparatus is determined.   8. The image processing apparatus according to claim 7, wherein the determination unit calculates the evaluation value by multiplying the first ratio and the second ratio by a weighting coefficient.   The interpolation means calculates a misalignment amount between the first image and the second image, and changes a ratio of combining the first and second images using the misalignment amount. The image processing apparatus according to claim 1, wherein the image processing apparatus is characterized.   10. The image processing according to claim 1, further comprising a generation unit configured to generate an arbitrary viewpoint image using an image generated by interpolation by the interpolation unit and an image of the subject area. apparatus. Having an instruction means for instructing virtual shooting conditions in an arbitrary viewpoint image;
The generation unit generates the arbitrary viewpoint image by deforming at least a part of the image of the subject region and the image generated by interpolation by the interpolation unit according to an instruction of a virtual shooting condition by the instruction unit. The image processing apparatus according to claim 10. An input means for inputting a first image and a plurality of second images acquired from a different viewpoint from the first image;
Alignment means for aligning the first and second images using corresponding points between the first image and the second image;
Calculating means for calculating a misalignment amount between the first image and the second image;
Detecting means for detecting a shielding area shielded by a subject area in the first and second images;
Determining means for obtaining a plurality of the second images from the input means and determining the second image to be used for interpolation processing of the shielding area in the first image;
Interpolating means for acquiring the first image, the second image determined by the determining means, and information on the shielding area detected by the detecting means and performing interpolation processing of the shielding area,
The determination unit is a region that is not shielded by the subject region in the second image with respect to the misalignment amount calculated by the calculation unit and the size of the shielding region detected by the detection unit. An image processing apparatus for determining an image to be used for an interpolation process from a plurality of the second images using information on the size of the image.   The determination means occupies the reciprocal of the misalignment amount and the size of the region not shielded by the subject region in the second image relative to the size of the shield region in the first image. The image processing apparatus according to claim 12, wherein an evaluation value is calculated from the ratio, and an image to be used for interpolation processing is determined from the plurality of second images using the evaluation value.   The image processing apparatus according to claim 13, wherein the determination unit calculates the evaluation value by multiplying a reciprocal of the misalignment amount and the ratio by a weighting coefficient.   An imaging apparatus comprising the image processing apparatus according to claim 1. An image processing method executed by an image processing apparatus that processes a plurality of images with different viewpoints,
An input step of inputting a first image and a plurality of second images acquired at a different viewpoint from the first image;
A detection step of detecting a shielding area shielded by a subject area in the first and second images;
A determination step of acquiring a plurality of the second images and determining the second image to be used for the interpolation processing of the shielding area in the first image;
An alignment step of aligning the first and second images using corresponding points between the first image and the second image;
An interpolation step for acquiring the first image, the second image determined in the determination step, and information on the shielding area detected in the detection step and performing interpolation processing on the shielding area;
In the determining step, the corresponding points between the first image and the second image and information on the shielding area detected in the detecting step are acquired, the number of corresponding points between the images, and the shielding area An image to be used for interpolation processing is determined from a plurality of the second images by using information on the size of a region not covered by the subject region in the second image with respect to the size. An image processing method. An image processing method executed by an image processing apparatus that processes a plurality of images with different viewpoints,
An input step of inputting a first image and a plurality of second images acquired at a different viewpoint from the first image;
A calculation step of calculating a misalignment amount between the first image and the second image;
A detection step of detecting a shielding area shielded by a subject area in the first and second images;
A determination step of acquiring a plurality of the second images and determining the second image to be used for the interpolation processing of the shielding area in the first image;
An alignment step of aligning the first and second images using corresponding points between the first image and the second image;
An interpolation step for acquiring the first image, the second image determined in the determination step, and information on the shielding area detected in the detection step and performing interpolation processing on the shielding area;
In the determination step, an area that is not shielded by the subject area in the second image with respect to the misalignment amount calculated in the calculation step and the size of the shielding area detected in the detection step An image processing method characterized in that an image to be used for interpolation processing is determined from a plurality of the second images using information on the size of the image. A program for causing a computer of an image processing apparatus to execute each step of the image processing method according to claim 16 or 17.

Images