JP6026979B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program for estimating the depth of a scene from a multi-viewpoint video.

  In the field of video media, in recent years, the flow of high reality and personalization has been rapidly progressing. 3D video that enables a highly realistic video experience by viewing the video three-dimensionally has been widely used in movies, televisions, game machines, and the like. In addition, one of the factors that accelerates the flow of high reality is the increase in screen size, such as 4K video with a resolution four times that of full HD and Super Hi-Vision video that NHK is developing. In addition, the spread of video-on-demand services via a network is remarkable, and it can be seen that personalization of video for viewing a favorite content on a favorite device by a user such as a television or a smartphone is progressing.

  Against this background, free viewpoint video is one of the new video media that further accelerates the high presence and personalization. In order to synthesize a free viewpoint video, there is a ray space method as a typical technique for interpolating images between viewpoints from multi-viewpoint images (see, for example, Non-Patent Document 1). In this method, a multi-viewpoint image is converted into a space called Epipolar Plane Image (hereinafter referred to as an EPI image), and the depth of the scene is estimated for each pixel by estimating the inclination of the ray, and the image between the viewpoints is interpolated. is there.

  However, this ray space method assumes pixel-by-pixel processing, and does not take into account region-based depth estimation that considers connectivity between pixels. There is a problem that is likely to occur. Further, there is a Segment-Based Depth Estimation method that divides a viewpoint image into regions based on color information and estimates the gradient of light rays for each region using an EPI image (see, for example, Non-Patent Document 2). .

  However, since this method has a region division shape depending on one viewpoint image, there is a possibility that a contradiction with the region division shape on another viewpoint image may occur. In addition, since the depth is estimated after the region shape is estimated, the optimum solution of both cannot be derived.

“Expression of three-dimensional spatial information by light group and its application” Journal of Television Society 50 (9), 1312-1318, 1996-09-20 "SEGMENT-BASED MULTI-VIEW DEPTH MAP ESTIMATION USING BELIEF PROPAGATION FROM DENSE MULTI-VIEW VIDEO" 3DTV Conference: The True Vision-Capture, Transmission and Display of 3D Video, 2008, pp.193-196

  Since the region division result based on the independent color information in each viewpoint image of the multi-viewpoint image is different for each viewpoint image, the region division images cannot maintain consistency with each other. For this reason, when estimating the depth of a scene for each region, the depth estimation result depends only on the region division result of one viewpoint image because only the region division result of one image is used. There is a problem that the error becomes large. In addition, since the region division itself results in a region division resulting from the number of regions set in advance, there is a problem in that it cannot cope with a flexible shape change or a change in the number of regions.

  The present invention has been made in view of such circumstances. An image processing apparatus and an image processing apparatus capable of performing area division that can respond to a flexible shape change and a change in the number of areas and performing depth estimation with a small error. It is an object to provide a method and an image processing program.

  The present invention inputs image information point conversion means for converting the input EPI image into point information combining color information and coordinate information, and outputting the point information and cluster information of a straight line cluster. Updating the cluster information to output the updated cluster information, to input the updated cluster information, to determine whether it is necessary to further perform the cluster division, the cluster determination means, When it is determined by the divided cluster determination means that it is necessary to perform the cluster division, the straight line cluster for performing straight line detection on the EPI image is further divided, and cluster information of the divided straight line cluster is obtained. The cluster division is performed by the cluster dividing unit that outputs to the cluster updating unit and the divided cluster determining unit. If it is determined that there is no necessity, characterized in that it comprises the the pixels belonging to the linear cluster of linear region information, and an area calculation unit that calculates the depth information and the color information output.

  The present invention inputs an image information point conversion step for converting the input EPI image into point information obtained by combining color information and coordinate information, and outputting the point information and cluster information of a straight line cluster. Updating the cluster and outputting the updated cluster information; inputting the updated cluster information; and determining whether or not further cluster splitting is required; and When it is determined by the divided cluster determination step that the cluster division is necessary, the straight line cluster for performing straight line detection on the EPI image is further divided, and the cluster information of the divided straight line cluster is obtained. A cluster dividing step for outputting to the cluster updating step and a divided cluster determining step; An area calculation step of calculating and outputting straight line area information, depth information and color information from pixels belonging to the straight line cluster when it is determined that the cluster division is not necessary. And

  The present invention is an image processing program for causing a computer to function as the image processing apparatus.

  According to the present invention, it is possible to perform an area division that can cope with a flexible shape change and a change in the number of areas, and to obtain an effect that depth estimation with a small error can be performed.

It is a block diagram which shows the structure of one Embodiment of this invention. It is explanatory drawing which shows the operation | movement which acquires an EPI image. It is a figure which shows an example of the EPI image acquired by the acquisition operation | movement shown in FIG. It is explanatory drawing which shows the example of the cluster centering on a straight line. It is explanatory drawing which shows the structure of a point information table. It is a block diagram which shows the structure of the division | segmentation cluster determination part 3 shown in FIG. It is a figure which shows the structure of the table which the area | region calculation part 5 outputs as a final result. It is explanatory drawing which shows the example of the EPI image reproduced from the result shown in FIG.

  Hereinafter, a depth estimation apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the depth estimation apparatus in the embodiment. The depth estimation apparatus shown in this figure is configured by a computer device and includes an image information point conversion unit 1, a cluster update unit 2, a divided cluster determination unit 3, a cluster division unit 4, and a region calculation unit 5. In the present embodiment, an EPI image composed of light rays on a straight line acquired by a camera array arranged in parallel on a straight line is a processing target.

  FIG. 2 is an explanatory diagram illustrating an operation of acquiring an EPI image. Cameras (linear camera arrays) that acquire images are arranged in a line in the x-axis direction. Each of the cameras constituting the linear camera array acquires an image having the vertical direction (height) as the v-axis and the horizontal direction (width) as the u-axis. As shown in FIG. 2, an image obtained by arranging a plurality of images acquired by the linear camera array in the x-axis direction and cutting out on the ux plane at v = a is an EPI image. FIG. 3 is a diagram showing an example of a simple EPI image acquired by the acquisition operation shown in FIG.

  On the EPI image, one straight line can be regarded as one light ray, and the inclination of the light ray represents the distance from the light source position to the camera. By performing linear fitting on the EPI image based on the luminance value of the pixel in the observation viewpoint image, the light ray is expressed as a linear parameter composed of the intercept and the inclination, and the unobserved viewpoint image between the cameras is interpolated. (See FIG. 4). FIG. 4 is an explanatory diagram showing an example of a cluster centered on a straight line.

  Next, the processing operation of the depth estimation apparatus shown in FIG. 1 will be described. First, the image information point conversion unit 1 converts the image information into point information for the acquired EPI image. Next, the cluster update unit 2 receives the point information and the initial cluster information as input, and updates the number of member pixels of the cluster and the centroid information. Then, the cluster update unit 2 updates the cluster information for all the pixels on the EPI image, and the cluster division unit 4 performs straight line detection on the EPI image by repeating the division of the cluster.

  Next, when the divided cluster determination unit 3 determines that the number of straight lines is such that the EPI image can be sufficiently expressed, the update and division of the cluster are finished, and finally, the region calculation unit 5 performs the member pixel of each cluster. Output the ray thickness and slope intercept.

Next, detailed processing operations of the components shown in FIG. 1 will be described. First, the processing operation of the image information point conversion unit 1 will be described. The image information point conversion unit 1 receives RGB (x, y) of the EPI image as input, and receives point information (point set E = {i: 1 ≦ i ≦ N}, the number N of points, and information {X _i , Y _i , Z _i } are output, RGB (x, y) is the color (R (x, y), G (x, y), B) in the pixel at coordinates (x, y) as shown in FIG. (X, y)) and R (x, y), G (x, y), and B (x, y) are the intensities of red, green, and blue colors (integers from 0 to 255).

The point set is generated by the following processes (1) to (3).
(1) E = φ (φ represents a hatched zero and is an empty set: the same applies hereinafter), i = 1.
(2) The following (i) to (iii) are executed for all combinations of x = 0,..., X−1, y = 0,..., Y−1 (where X and Y are the width of the EPI image) height).
(I) Let E ← EU {i}.
(Ii) X _i = x, Y _i = y, Z _i = RGB (x, y)
(Iii) Let i ← i + 1.
(3) N = i.
The point information table shown in FIG. 5 is generated based on the point information generated in this way. The image information point conversion unit 1 outputs this point information table to the cluster update unit 2 as point information.

Next, the processing operation of the cluster update unit 2 will be described. The cluster updating unit 2 inputs the point information {X _i , Y _i , Z _i } and the cluster information {N _k , E _k , A _k , B _k , C _k } (k = 1,..., J) The updated cluster information {N _k , E _k , A _k , B _k , C _k } is output.

The cluster update unit 2 performs clustering with a straight line y = A _k x + B _{k in} the x and y axis directions representing pixels and a point C _{k in the} z (RGB) axis direction representing color intensity. All points belong to the nearest cluster E _k of the straight line (A _k x + B _k , C _k ) that becomes the centroid. When the cluster update unit 2 is executed for the first time, the cluster number j = 1, and the initial cluster centroids A _k , B _k , C _k (k = 1) are executed as random values and the cluster E ₁ = E. . From the second time on, the cluster information {N _k , E _k , A _k , B _k , C _k } (k = 1,..., J) output from the cluster dividing unit 4 is executed as an input. The cluster update process is performed according to the following (1) to (5).

(1) _Let E _{k (old)} = E _k (k = 1, 2,..., J).
(2) The distance D _{i, k} with each cluster k (k = 1,..., J) is calculated for all points i∈E.
F _d (p, q) is a function for calculating the distance between the vectors p and q.
(3) Update cluster E _k for all points iεE. Specifically, the nearest cluster
And the slope A _k , intercept B _k, and average value C _k of the straight line clusters are updated for all clusters E _k (k = 1, 2,..., J). Specifically, the slope inclination A _k , intercept B _k, and average value C _k that minimize L _k and G _k below are obtained.
(4) For all k (k = 1, 2,..., J), the process ends when E _{k (old)} = E _k .
(5) Return to (1) and repeat the process.

For example, the distance function F _d can be given as follows:
α is a coefficient for adjusting the weight of the distance between pixels and the distance between colors. F _d may use the Mahalanobis distance or the like instead of the weighted Euclidean distance function.

Next, the processing operation of the cluster dividing unit 4 will be described. The cluster dividing unit 4 includes cluster information {N _k , E _k , A _k , B _k , C _k } (k = 1, 2,..., J) and a divided cluster index L = {l ₁ , l ₂ ,. l _n } and cluster information {N _k , E _k , A _k , B _k , C _k } (k = 1, 2,..., j, where j is the value before division To n increase (that is, j + n) is output.

Specifically, the calculation is performed for all lεL as in the following (1) to (4).
(1) A ′ = A _i , B ′ = B _i , and C ′ = C _i .
(2) B _i = B ′ + ε _y , Ci = C ′ + ε _z (ε is a random value or a unit vector in a direction in which Y and Z have large dispersion).
(3) Let j ← j + 1.
(4) Let B _i = B′−ε _y and C _i = C′−ε _z .

Next, the processing operation of the divided cluster determination unit 3 will be described. The divided cluster determination unit 3 receives the cluster information {N _k , E _k , A _k , B _k , C _k } and point information {X _i , Y _i , Z _i }, and the divided cluster index L = {l ₁ , L ₂ ,..., L _n } and cluster information {N _k , E _k , A _k , B _k , C _k }. When the cluster to be divided is not found (that is, when L = φ), the divided cluster determination unit 3 outputs the cluster information to the region calculation unit 5 as it is. When a cluster to be divided is found (that is, when L ≠ φ), a divided cluster index L and cluster information {N _k , E _k , A _k , B _k , C _k } are output.

  FIG. 6 is a block diagram showing a configuration of the divided cluster determination unit 3 shown in FIG. The divided cluster determination unit 3 includes a cluster division unit 31, a cluster update unit 32, an information amount criterion calculation unit 33, an information amount criterion calculation unit 34, and an index generation unit 35.

The cluster dividing unit 31 and the cluster updating unit 32 shown in FIG. 6 have the same functions as the cluster dividing unit 4 and the cluster updating unit 2 shown in FIG. For this reason, the divided cluster determination unit 3 can divide and update the cluster, and by calculating the information amount (information amount reference calculation unit 34) at that time, the information amount (information) when the cluster is not divided Compared with the quantity reference calculation unit 33), it is possible to determine whether or not the cluster should be divided. Compare the information criterion I _k of the cluster before dividing in each cluster of k = 1,..., j with the information criterion I ′ _k of the cluster after dividing, and if I _k > I ′ _k , the cluster is divided As a better cluster, k is added to the set of index information L.

Specifically, the calculation is performed by the following processing operations (1) to (9).
(1) k = 1 and index information L = φ.
(2) End when k = j + 1.
(3) The information amount criterion calculation unit 33 calculates the information amount criterion I _k for the cluster information {N _k , E _k , A _k , B _k , C _k }.
(4) Assume that {N ′ ₁ , E ′ _k , A ′ ₁ , B ′ ₁ , C ′ ₁ } = {N _k , E _k , A _k , B _k , C _k }.
(5) The cluster division unit 31 divides the cluster as cluster information {N ′ ₁ , E ′ ₁ , A ′ ₁ , B ′ ₁ , C ′ ₁ } and division index information {1}, and the cluster information after the division {N ′ ₁ , E ′ _k , A ′ ₁ , B ′ ₁ , C ′ ₁ }, {N ′ ₂ , E ′ ₂ , A ′ ₂ , B ′ ₂ , C ′ ₂ } are obtained.
(6) The cluster updating unit 32 obtains the cluster information {N ′ ₁ , E ′ _k , A ′ ₁ , B ′ ₁ , C ′ ₁ }, {N ′ ₂ , E ′ ₂ , A ′ ₂ , B after the division. ′ ₂ , C ′ ₂ } and updated cluster information {N ′ ₁ , E ′ _k , A ′ ₁ , B ′ ₁ , C ′ ₁ }, {N ′ ₂ , E ′ ₂ , A ′ ₂ , B ′ ₂ , C ′ ₂ }.
(7) The information criterion calculation unit 34 obtains cluster information {N ′ ₁ , E ′ _k , A ′ ₁ , B ′ ₁ , C ′ ₁ }, {N ′ ₂ , E ′ ₂ , A ′ ₂ , B ′. ₂ , C ′ ₂ }, the information criterion l ′ _k is calculated.
(8) The index generation unit 35 sets the index information L ← L 部 {k} when I _k ≧ I ′ _k .
(9) Set k ← k + 1 and return to (2).

The information amount criteria I _k and I ′ _k can be obtained by using AIC / BIC as follows, for example.
AIC I _k = -2 lnL (E _k ) + q
I ′ _k = −2 lnL (E ′ ₁ ) L (E ′ ₂ ) + 2q
BIC I _k = −2 lnL (E _k ) + qlnN _{k
I 'k = -2lnL (E'} 1) L (E '2) + 2qlnN k
L (E _k ) is the likelihood of the cluster E _k , q is the number of parameters of the cluster, and N _k is the number of samples of the cluster
Σ _ky and Σ _kz are variance covariance matrices of (Y _i − (A _k X _i + B _k )) and (Z _i −C _k ) in the cluster iεE _k .
When the variance-covariance matrix is a diagonal matrix, q = 9 (total number of parameters of A _k , number of parameters of B _k , number of parameters of C _k , parameter number of 1 for Σ _ky , parameter number of 3 for Σ _kz )

Next, the processing operation of the area calculation unit 5 will be described. The area calculation unit 5 inputs the cluster information {N _k , E _k , A _k , B _k , C _k } (k = 1,..., J), the depth information A _k , the color information C _k , and the area information W _k is output. FIG. 7 is a diagram showing a configuration of a table output by the area calculation unit 5 as a final result. The depth information A _k is the cluster information A _k , and the smaller the inclination, the deeper the object. The color information C _{k is} also cluster information C _k , and has the intensity R _k , G _k , and B _k of each color. The area information w _k is the size of the cluster (thickness of the straight line), and the processing operations (1) to (3) below are performed for each cluster {N _k , E _k , A _k , B _k , C _k }. Can be obtained.

(1) For all points iεE _k belonging to the cluster, the difference H _{i, k} from the centroid A _k , B _k is calculated by the following equation.
H _{i, k} = | Y _i − (A _k X _k + B _k ) |
(2) Find the maximum value max _i {H _{i, k} } of H _{i, k} among all points i∈E _k .
(3) Obtain the region w _k from the maximum difference value max _i {H _{i, k} } by the following equation.
FIG. 8 shows an EPI image reproduced from the result of FIG. It can be seen that the original EPI image can be reproduced with the obtained straight line.

  As described above, in order to perform simultaneous estimation of the shape of the divided region and the depth of the divided region, first, an EPI image cross-sectional image is extracted from the multi-viewpoint image. Then, the same processing is performed on all EPI image cross-sectional images. In order to realize this, the depth estimation apparatus according to the present embodiment includes an image information point conversion unit 1 that converts an EPI image slice image into point information, and a cluster update unit 2 that sequentially updates clustering information. A divided cluster determination unit 3 that determines whether or not to divide the cluster, a cluster division unit 4 that divides the cluster, a region calculation unit 5 that calculates a linear parameter, a size of the cluster with the straight line as the center of gravity, and the like, Equipped with. Thereby, in the area division estimation and the depth estimation, it is possible to simultaneously estimate the inclination, intercept and width of the light ray with respect to the EPI image cross-sectional image by a new framework called linear clustering. In addition, in the simultaneous estimation of the ray depth and the region, the end determination condition is used as the information criterion.

  With this configuration, it is possible to perform region division with consistency between viewpoint images. In addition, it is possible to perform region division with a high degree of freedom in which the number of region divisions does not depend on the initial value. Furthermore, depth estimation independent of only a specific image is possible (region shape and depth can be estimated simultaneously).

  You may make it implement | achieve the depth estimation apparatus in embodiment mentioned above with a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. is there. Therefore, additions, omissions, substitutions, and other modifications of the components may be made without departing from the technical idea and scope of the present invention.

  It can be applied to applications where it is essential to perform depth estimation with a small error as well as to be able to divide the region to accommodate flexible shape changes and changes in the number of regions.

  DESCRIPTION OF SYMBOLS 1 ... Image information point conversion part, 2 ... Cluster update part, 3 ... Divided cluster determination part, 4 ... Cluster division part, 5 ... Area calculation part

Claims (3)

Image information point conversion means for converting the input EPI image into point information combining color information and coordinate information and outputting the point information;
Cluster update means for inputting the point information and the cluster information of the straight line cluster, updating the cluster information and outputting the updated cluster information;
The updated cluster information is input, the information criterion I _k of the cluster before the division is compared with the information criterion I ′ _k of the cluster after the division, and when I _k > I ′ _k , the cluster A split cluster determination means for determining whether or not further cluster splitting needs to be performed,
When the divided cluster determination unit determines that the cluster division is necessary, the straight line cluster for performing straight line detection on the EPI image is further divided, and the cluster information of the straight line cluster after the division is obtained. Cluster dividing means for outputting to the cluster updating means,
An area calculation means for calculating and outputting straight line area information, depth information or color information from pixels belonging to the straight line cluster when the divided cluster determination means determines that the cluster division is not necessary. Bei to give a,
The _{I k,}
I _k = -2 lnL (E _k ) + q,
The I _'k,
I ′ _k = −2 lnL (E ′ ₁ ) L (E ′ ₂ ) + 2q
Asking for,
Or
The _{I k,}
I _k = −2lnL (E _k ) + qlnN _k,
The I _'k,
I ′ _k = −2 lnL (E ′ ₁ ) L (E ′ ₂ ) + 2qlnN _k
(Where L (E _k ) is the likelihood of cluster E _k , q is the number of parameters in the cluster, and N _k is the number of samples in the cluster)
An image processing apparatus. An image information point conversion step for converting the input EPI image into point information combining color information and coordinate information and outputting the point information;
A cluster update step of inputting the point information and cluster information of a straight line cluster, updating the cluster information and outputting the updated cluster information;
The updated cluster information is input, the information criterion I _k of the cluster before the division is compared with the information criterion I ′ _k of the cluster after the division, and when I _k > I ′ _k , the cluster A split cluster determination step for determining whether or not further cluster splitting needs to be performed by determining to further split
When it is determined by the divided cluster determination step that the cluster division is necessary, the straight line cluster for performing straight line detection on the EPI image is further divided, and cluster information of the divided straight line cluster is obtained. A cluster dividing step for outputting to the cluster update step;
An area calculation step for calculating and outputting straight line area information, depth information or color information from pixels belonging to the straight line cluster when it is determined by the divided cluster determination step that the cluster division is not necessary; I have a,
The _{I k,}
I _k = -2 lnL (E _k ) + q,
The I _'k,
I ′ _k = −2 lnL (E ′ ₁ ) L (E ′ ₂ ) + 2q
Asking for,
Or
The _{I k,}
I _k = −2lnL (E _k ) + qlnN _k,
The I _'k,
I ′ _k = −2 lnL (E ′ ₁ ) L (E ′ ₂ ) + 2qlnN _k
(Where L (E _k ) is the likelihood of cluster E _k , q is the number of parameters in the cluster, and N _k is the number of samples in the cluster)
An image processing method.   An image processing program for causing a computer to function as the image processing apparatus according to claim 1.