JP6401674B2 - Image processing method, image processing apparatus, and image processing program - Google Patents

Description

  The present invention relates to an image processing method, an image processing apparatus, and an image processing program for generating a light field image.

  Utilization of light information, such as focus adjustment after image capture and restoration of three-dimensional information, due to the practical use of light field cameras that record light information (light field images) including the direction of light entering the image sensor Research and development of new image processing technologies and applications has been actively conducted. A method of recording light ray information has been present for a long time, and a method of photographing a large number of cameras closely arranged in synchronization is well known (for example, see Non-Patent Document 1). This method requires a system in which a large number of cameras of the same type are moved in synchronization with each other, and since strict camera calibration is required, it has been considered difficult to implement in terms of cost and labor.

  However, in a light field camera (for example, see Non-Patent Document 2) that has recently been sold to the general public, a large number of cameras are arranged by arranging microlens arrays before and after the main lens inside the camera. It is possible to record the light beam information in the same way as in the case of the above (it is possible to adopt a different configuration by changing the installation position of the microlens array).

B. Wilburn, N. Joshi, V. Vaish, E.-V. Talvala, E. Antunez, A. Barth, A. Adams, M. Horowitz, and M. Levoy, “High performance imaging using large camera arrays,” ACM Transactions on Graphics, vol. 24. p. 765, 2005. R. Ng, M. Levoy, G. Duval, M. Horowitz, and P. Hanrahan, “Light Field Photography with a Hand-held Plenoptic Camera,” Stanford Tech Rep. CTSR, pp. 1-11, 2005.

  Unlike a conventional camera, a light field camera with a microlens array can record light beam information. However, in this camera recording method, the spatial resolution and the angular resolution are a trade-off, and it is difficult to achieve the resolution required for photography with a conventional camera with a practically usable imaging device. In addition, the amount of light is insufficient due to attenuation of light by the microlens array, so that it is easily affected by noise, and in order to compensate for it, it is also affected by blurring. As described above, there is a problem that it is difficult to obtain a quality that can be easily achieved by a conventional camera in imaging using a light field camera.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing method, an image processing apparatus, and an image processing program capable of easily generating a light field image.

  One aspect of the present invention is an image processing method for generating a light field image from a light field dictionary composed of a finite basis vector that can represent an input image including a defocused image and an arbitrary light field image by linear combination of the basis vectors. And estimating a coefficient vector for the light field dictionary that optimizes an error between the focusing process result for the light field image and the input image, and a focal blur amount of each sub-aperture image of the light field image. An image processing method including a coefficient vector estimation step and a light field image generation step of generating a light field image from the coefficient vector.

  One aspect of the present invention is the image processing method, further including a focal blur amount estimation step for estimating a focal blur amount of the sub-aperture image, and the coefficient vector estimation step includes the focus blur amount based on the focal blur amount. Estimate the coefficient vector.

  One aspect of the present invention is the image processing method, wherein in the focal blur amount estimation step, a norm of a gradient of the sub-aperture image is set as the focal blur amount.

  One aspect of the present invention is the image processing method, wherein in the focal blur amount estimation step, a dispersion of a gradient of the sub-aperture image is set as the focal blur amount.

  One aspect of the present invention is the image processing method, wherein in the focal blur amount estimation step, a gradient ratio obtained by applying two different types of point spread functions to the sub-aperture image is calculated as the focal blur. Amount.

  One aspect of the present invention is an image processing device that generates a light field image from a light field dictionary including a finite basis vector that can represent an input image including a defocused image and an arbitrary light field image by linear combination of the basis vectors. And estimating a coefficient vector for the light field dictionary that optimizes an error between the focusing process result for the light field image and the input image, and a focal blur amount of each sub-aperture image of the light field image. An image processing apparatus includes a coefficient vector estimation unit and a light field image generation unit that generates a light field image from the coefficient vector.

  One aspect of the present invention is an image processing program for causing a computer to execute the image processing method.

  According to the present invention, the light field information is restored by referring to the light field dictionary based on a general image that does not include the light beam information, so that the light field image can be easily generated.

It is a block diagram which shows the structure of the light field image generation apparatus by one Embodiment of this invention. 3 is a flowchart showing a processing operation of the light field image generation device 100 shown in FIG. 1.

  Hereinafter, a light field image generating device according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a light field image generation device according to the embodiment. As illustrated in FIG. 1, the light field image generation device 100 includes an image input unit 101, a dictionary input unit 102, a patch generation unit 103, a coefficient vector estimation unit 104, and a light field image generation unit 105.

  The image input unit 101 inputs an image that does not include light ray information that is a source of the light field image from the outside. Hereinafter, this image is referred to as an input image. The dictionary input unit 102 inputs a light field dictionary from the outside. Hereinafter, this light field dictionary is referred to as an LF dictionary. The patch generation unit 103 divides the input image into patches (small regions) having a predetermined size, and generates an image patch group. The coefficient vector estimation unit 104 estimates a coefficient vector group corresponding to each image patch from the image patch group and the LF dictionary. The light field image generation unit 105 generates a light field patch group from the coefficient vector group and the LF dictionary, and generates and outputs a light field image from the light field patch group. Hereinafter, the light field patch is referred to as an LF patch, and the light field image is referred to as an LF image.

  The LF image is an image in which information of light rays incident on a certain imaging system is recorded. The LF image may be expressed in any way. In general, a normal image is represented by a two-dimensional array in which pixels are arranged vertically and horizontally, whereas an LF image is a four-dimensional array in which two dimensions are added to express angles in two directions. expressed. When the spatial resolution is H (height) xW (width) and the angular resolution is NxM, the LF image can be represented by a four-dimensional array of HxWxNxM (Reference 1: “M. Levoy and P. Hanrahan,“ Light Field rendering, ”Proc. 23rd Annu. Conf. Comput. Graph. Interact. Tech.-SIGGRAPH '96, pp. 31-42, 1996.”).

  When generating an image focused on an arbitrary distance from the LF image, the resolution of the generated image is HxW unless special processing for increasing the resolution such as resampling or super-resolution is performed. In addition, it can be expressed as N × M multi-viewpoint images having a resolution of HxW. In this case, each viewpoint image is a collection of ray information as an image for each direction, and is the same as an image captured by each camera when the same ray information is captured using a number of cameras. Hereinafter, an image at each viewpoint in this expression is referred to as a sub-aperture image. Alternatively, an image recorded on an image sensor via a main lens and a microlens array may be directly input as an image captured by a general LF camera. In addition, any type of LF image may be output.

  Next, the processing operation of the light field image generating apparatus 100 shown in FIG. 1 will be described with reference to FIG. FIG. 2 is a flowchart showing the processing operation of the light field image generating apparatus 100 shown in FIG.

  First, the image input unit 101 inputs an image that does not include the light ray information that is the basis of the LF image (step S101). This image is hereinafter referred to as an input image. Subsequently, the dictionary input unit 102 inputs an LF dictionary (step S102). Any LF dictionary may be used, but in the present embodiment, it is assumed that an arbitrary LF image can be represented by a linear combination of the basis vectors. The basis vectors may be generated by any method such as principal component analysis or independent principal component analysis, but in the following, sparse coding (Reference 2: “M. Elad and M. Aharon,“ Image denoising via sparse and redundant representation. over learned dictionaries, "IEEE Transations Image Process., vol. 15, no. 12, pp. 3736-3745, 2006."). Thus, it is assumed that an arbitrary LF image can be expressed by a very small number of coefficient vectors.

  Next, the patch generation unit 103 divides the input image into patches having a predetermined size, and generates an image patch group (step S103). The size of the patch may be any size, but is determined by the input LF dictionary. A plurality of patches may be overlapped. In the following description, it is assumed that a patch having a spatial resolution of wxh is used.

  Next, the coefficient vector estimation unit 104 estimates a coefficient vector for the LF dictionary of each image patch from the image patch group and the LF dictionary (step S104). At this time, on the condition that the original image patch can be generated by performing the focusing process on the restored LF patch, and that the sub-aperture image obtained from the restored LF patch contains as little focus blur as possible. To do. Any method may be used for estimating the coefficient vector that satisfies such conditions, but a method using sparse coding will be described below.

となる。 In sparse coding, an arbitrary LF image vector is expressed by a coefficient vector corresponding to an LF dictionary composed of LF bases. If the LF image to be restored is L, the dictionary is D, and the coefficient vector is α,

It becomes.

とすることで係数ベクトルαを推定できる。第三項はスパース項であり、λはそのパラメータである。 In order that the sub-aperture image obtained from the generated LF image does not include defocusing, the vector of the input image is I, the focal length f, the focusing operator at the depth of field d is R (f, d), If the operator that extracts the (i, j) th sub-aperture image is C (i, j) and the focal blur evaluation function is E (x),

Thus, the coefficient vector α can be estimated. The third term is a sparse term, and λ is its parameter.

  Any method may be used to create the focused image. Well-known methods include shift addition and Fourier slice methods (Reference 3: “R. Ng,“ Fourier slice photography, ”ACM SIGGRAPH 2005 Pap.-SIGGRAPH '05, p. 735, 2005.”) and so on. Although the above method requires the focal length and depth of field of the input image, these may be input from the outside and used, or may be estimated and used.

  The coefficient vector may be estimated by a method that does not use these parameters. For example, when restoration is performed on the assumption that the focal length and depth of field of the input image are the same as the focal length and depth of field of the main lens in the LF image imaging system, In addition averaging and Fourier slicing, by setting the slice angle to 0, a focusing operator that does not depend on parameters can be defined.

とすることで係数ベクトルαを推定できる。また第二項は勾配ベクトルを正規化することでＬ２ノルムに置き換えることも可能であり、一般的なＬ１正則化で高速に解を得ることができる。 Any defocus evaluation function may be used. For example, the gradient vector of the image may be obtained and the number of non-zero pixels may be used as the evaluation function. When using such an evaluation function,

Thus, the coefficient vector α can be estimated. The second term can be replaced with the L2 norm by normalizing the gradient vector, and a high-speed solution can be obtained by general L1 regularization.

としてあらかじめ求め、

とすることで係数ベクトルを推定できる。 Any normalization method can be used. For example, the gradient of the input image

As in advance,

Thus, the coefficient vector can be estimated.

Further, by using the normalized gradient vector of the input image instead of the input image in the first term, it is possible to omit the gradient operator and the normalization operator in the second term as follows.

  Since the coefficient vector obtained in this case corresponds to the gradient of the LF image, the light field image generation unit 105 described later may perform processing for estimating the LF image from the gradient.

  In the above description, an evaluation function having a larger value as the focal blur is stronger is used as the focal blur evaluation function in order to optimize the minimization problem. However, the optimization problem is minimized or the dual problem is minimized. In the case where the first term and the second term are separated into different problems and a solution is obtained by repetitive calculation or the like, an evaluation function that takes a smaller value as the defocus is stronger may be used. For example, the variance of the image gradient may be evaluated. Alternatively, conversion to a frequency domain such as Fourier transform or wavelet transform may be performed to evaluate the strength of the high frequency component.

と推定される。 In the above description, a function capable of calculating an evaluation value by a linear operation is used. However, an evaluation function obtained by a non-linear operation may be used. For example, as shown in Reference 4: “S. Zhuo and T. Sim,“ Defocus map estimation from a single image, ”Pattern Recognit., Vol. 44, no. 9, pp. 1852-1858, Sep. 2011.” As described above, there is a method of estimating the blur amount by comparing the gradient amounts of the images subjected to the Gaussian filter with two different parameters. In this method, a point spread function representing defocus is assumed to be a Gaussian function, and its parameters are estimated. If the original image is I, the parameters of an arbitrary Gaussian filter are σ ₁ , σ ₂ : σ ₁ <σ ₂ , the operator of the Gaussian filter is G (x, y, σ), and the gradient operator is ∇, the position (x , Y), σ (x, y) is

It is estimated to be.

  It is also possible to apply the same method to different point spread functions such as an average filter. Besides the method of estimating parameters assuming a model of point spread function, Reference 5: “H. Zhang, J. Yang, Y. Zhang, and TS Huang,“ Sparse Representation Based Blind Image Deblurring, ”IEEE Int. Conf. Multimed. Expo, pp. 1-6, 2011. ”A method of directly estimating a point spread function may be used.

と推定される。 Alternatively, instead of focusing blur evaluation on the sub-aperture image, an evaluation function that evaluates the amount of blur due to the point spread function is used by assuming that the input image is a convolution of the sub-aperture image corresponding to the center of the main lens and a point spread function. May be. Any method may be used for estimating the point spread function. For example, a vector having the same size as the kernel of the point spread function around the position (x, y) of the sub-aperture image is represented by A (x, y), (x , Y) where the pixel value of the input image is b (x, y) and the kernel of the point spread function is k (x, y).

It is estimated to be.

としてパッチ全体で点広がり関数を推定するなどしてもよい。 If the patch size is larger than the kernel size

For example, a point spread function may be estimated for the entire patch.

  The blur amount may be evaluated by the point spread function estimated in this way. For example, the blur amount may be larger as the variance of the kernel of the point spread function is higher, and the blur amount may be smaller as the kernel variance is lower.

  In the above description, the corresponding coefficient vector is estimated with the input image as a vector as it is, but normalization, redundancy reduction, and the like may be added to the input image as necessary. For example, a process may be added in which an average value is subtracted from each patch to obtain an image vector, and after the LF patch is restored, the average value is added back. Further, arbitrary transformation such as DCT transformation and Fourier transformation, and processing such as quantization may be added as necessary.

  Next, the light field image generation unit 105 generates an LF patch group from the coefficient vector group and the LF dictionary, and generates an LF image from the LF patch group. Any method for generating the LF image may be used. In general, a method is used in which all patches are matched to corresponding image positions, and an average of the portions where a plurality of patches overlap is obtained. Finally, the generated LF image is output and the process is terminated (step S105).

  In the above embodiment, the imaging position of the input image is the same as the imaging position of the sub-aperture image corresponding to the center of the main lens of the LF, but may be the same as the imaging position of another sub-aperture image of the LF, or may be a different position. Good. In such a case, it is assumed that a focused image at the same position as the input image can be generated using the LF image, and the operator is R (f, d, X, Y, Z). A coefficient vector can be generated as shown in Equation (3) and Equation (3).

  As described above, when a light field image is generated from an input image having no ray information, a coefficient vector for expressing an image patch obtained by dividing the input image as a linear combination of basis vectors that are elements of the LF dictionary is represented. The light field image can be easily generated from the input image using the coefficient vector calculated to minimize the predetermined error.

  In the above-described embodiment, the order of some processes may be changed. In addition, all or part of the light field image generation device in the above-described embodiment may be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be a program for realizing a part of the above-described functions, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. is there. Therefore, additions, omissions, substitutions, and other modifications of the components may be made without departing from the technical idea and scope of the present invention.

  It can be applied to applications indispensable for generating a light field image from a general image that does not include light ray information and performing focus adjustment and estimation of three-dimensional information.

  DESCRIPTION OF SYMBOLS 100 ... Light field image generation apparatus, 101 ... Image input part, 102 ... Dictionary input part, 103 ... Patch generation part, 104 ... Coefficient vector estimation part, 105 ... Light field image Generator

Claims (7)

An image processing method for generating a light field image from a light field dictionary composed of a finite basis vector that can represent an input image including a defocus and an arbitrary light field image by linear combination of basis vectors,
A coefficient vector estimation step for estimating a coefficient vector for the light field dictionary for optimizing an error between the focus processing result for the light field image and the input image, and a defocus amount of each sub-aperture image of the light field image. When,
A light field image generation step of generating a light field image from the coefficient vector. A defocus amount estimation step for estimating a defocus amount of the sub-aperture image;
The image processing method according to claim 1, wherein in the coefficient vector estimation step, the coefficient vector is estimated based on the focal blur amount.   The image processing method according to claim 2, wherein, in the focal blur amount estimation step, a norm of a gradient of the sub-aperture image is used as the focal blur amount.   The image processing method according to claim 2, wherein in the focal blur amount estimation step, a variance of a gradient of the sub-aperture image is used as the focal blur amount.   The image processing method according to claim 2, wherein in the focal blur amount estimating step, a ratio of gradients obtained by applying two different types of point spread functions to the sub-aperture image is used as the focal blur amount. An image processing device that generates a light field image from a light field dictionary composed of a finite basis vector that can represent an input image including a defocus and an arbitrary light field image by linear combination of basis vectors,
Coefficient vector estimation means for estimating a coefficient vector for the light field dictionary for optimizing an error between the focus processing result for the light field image and the input image, and a defocus amount of each sub-aperture image of the light field image When,
An image processing apparatus comprising: a light field image generation unit configured to generate a light field image from the coefficient vector.   An image processing program for causing a computer to execute the image processing method according to claim 1.

Images