JP5515864B2 - Image processing method, image processing apparatus, and image processing program - Google Patents

Description

  The present invention relates to an image processing method, an image processing apparatus, and a program that enable a stereoscopic effect to be manipulated (emphasized or suppressed) when an image is stereoscopically viewed.

In human visual stereoscopic processing, a stereoscopic image is recognized based on an image (video) viewed with the left and right eyes. Therefore, in order to make an image (video) appear three-dimensionally, a set of images (video) for the left eye image (video) and the right eye image (video) is required.
A so-called “stereoscopic” technique is known in which a two-dimensional image is processed to make a planar image appear stereoscopically. Among them, there is a technique for generating two images for the left eye and right eye from one two-dimensional image. This calculates the binocular parallax with respect to the object of interest (the object to be imaged) in the image, and shifts each pixel position of the image in the horizontal direction by an amount based on the parallax. Is a process for generating

As a method for calculating binocular parallax, there is a method of estimating the actual depth (relative distance from the camera) of an object based on color information such as brightness and saturation.
For example, in Patent Document 1, an integrated value of high frequency components, luminance contrast, integrated luminance value, and integrated saturation value for each region are defined as image feature amounts representing the perspective of an image, and any one or combination thereof is determined. A method is disclosed in which the depth correction amount is calculated so that a difference is produced between the regions, and the degree of perspective is determined from the magnitude.
In this method, the depth amount is corrected so as to conveniently give a front-to-back relationship between regions. For this purpose, an appropriate image feature amount is selected, and there is flexibility to cope with various cases. In addition, it lacks decisive power such as automatic determination. Furthermore, it can be said that it is inappropriate for an object in which it is difficult to give a context within a region such as the same plane.

Many methods for adjusting the stereoscopic effect are provided on the premise of binocular stereoscopic vision.
For example, in Patent Document 1, an average value of motion vectors corresponding to the amount of movement of a region of interest in a video is obtained, and whether the region of interest is a subject or a background is determined from the magnitude relationship compared with the average value. A method for adjusting the horizontal phase amount in accordance with the difference between the subject and the background is presented. However, this method cannot give a stereoscopic effect to the background itself or the inside of the subject surface.
Further, in Patent Document 2, edge information of a video signal is obtained, and a front-rear relationship is obtained by regarding a dense area of edge information as a foreground and a sparse area as a background, and a large parallax is given to the foreground and a small background. A method is presented. However, since this method targets the entire region with dense edges, the background cannot be specified because the entire surface is the target of the foreground, and the background cannot be specified for materials with dense structures such as fur and fabric. I can't find it.
Further, in Patent Document 3, since there are generally many images in which the subject is in focus, it is considered that the closer the object is, the higher the high frequency component, contrast, luminance, and saturation, and the higher frequency integrated value, luminance contrast, luminance A method is proposed in which the depth correction amount is determined by determining that an area having a larger integrated value and saturated integrated value is an object existing ahead, and the parallax amount is linearly associated with the depth amount. However, this method is intended to give parallax between areas, and is inappropriate for the purpose of giving parallax inside a small area, for example.

  On the other hand, there is a method for reproducing as CG (computer graphics) instead of manipulating actual images and videos for the same purpose of realistic reproduction of texture. For this purpose, it is necessary to accurately measure and accurately reflect the reflection characteristics of the actual material. A typical method is to paste a material into a certain shape (three-dimensional shape such as a plane or curved surface), and place a light source with a known light color in a specified direction so that it has a certain strength. In addition to irradiating with light, the camera can be photographed from almost all directions to photograph every aspect of the object, and the illumination direction and observation direction can be given as parameters from the large number of images obtained in this way. This is a method of configuring and using a function that outputs a reflected color. This method is called BRDF (bidirectional reflection distribution function). A method in which this method is applied to a woven fabric is presented in Non-Patent Document 1, for example. In this method, since the appearance of the surface of the moving subject can be calculated as a change in reflected light from the viewpoint of an observer who performs stationary observation, the reality can be improved. However, this method cannot provide a stereoscopic effect that is superior to stereoscopic viewing using parallax information.

  As described above, in the conventional method, it is a difficult problem to reproduce or enhance the three-dimensional effect (such as the thickness of the material surface) in each plane of the object.

Japanese Patent No. 3235776 Japanese Patent Laid-Open No. 10-336702 Japanese Patent Laid-Open No. 10-191397

Yuki Takeda, Hiromi Tanaka, “Reconstruction of microscopic surface geometry of woven fabrics based on reflected light analysis,” IPSJ “Image Recognition and Understanding Symposium (MIRU'06)”, pp. 1084-1089, July 2006 “Wavelet Beginners Guide”, Susumu Sugawara, Tokyo Denki University Press, 1995 “Wavelets and Orthogonal Function Systems”, G. G. By Walter Susumu Sugawara, Takeshi Sasayo, Translated by Ryuichi Kanno, Tokyo Denki University Press, 2001

  As described above, the conventional technique enhances the stereoscopic effect by giving a sense of front and back and a sense of depth by giving parallax between regions (for example, individual objects) in an image (video) or between individual surfaces. It does not take into account manipulating the stereoscopic effect inside individual surfaces or areas. Therefore, if the conventional technique is applied to the inside of the surface or the area as it is, there is a problem that a stereoscopic image becomes unnatural and has a strong sense of incongruity, such as a double image or a double layer.

  In view of the above problems, the present invention makes a natural three-dimensional feel when binocularly viewed with the object surface, the area inside, the undulation and thickness of the surface derived from the detailed structure and material as the direct processing target. It is another object of the present invention to provide an image processing method that makes it possible to generate a left-eye image and a right-eye image from only one two-dimensional image.

As means for solving the above-mentioned problems, the first invention should be applied to a frequency decomposition means for calculating a frequency component by spatially resolving a spatial frequency component of one two-dimensional image and an image to be three-dimensionalized. A parallax amount designating unit for designating a parallax amount, a parallax amount allocating unit for allocating the parallax amount to a frequency component obtained by decomposition, and a frequency synthesis for synthesizing each frequency component obtained through the parallax amount allocating unit And an image output means for outputting a left-eye image and a right-eye image that enable binocular stereoscopic vision from the result obtained by the frequency synthesizing means. It is.
The second invention is the image processing method according to claim 1, wherein the frequency component to which the parallax amount is allocated by the parallax amount allocating unit is a frequency component of brightness of the two-dimensional image. .
According to a third aspect of the present invention, the frequency component to which the parallax amount is allocated by the parallax amount allocating means is the frequency component that is divided into two after the frequency decomposition is repeated a desired number of times, and the lower frequency component is the lower frequency component. component, at the time of the frequency component of the higher frequency side and high frequency components, Ri high frequency component der brightness of the two-dimensional image, the parallax amount is a frequency component other than the high frequency components parallax amount is allocated 2. The image processing method according to claim 1, wherein the image processing method is not assigned .
According to a fourth aspect of the present invention, the frequency component to which the parallax amount is allocated by the parallax amount allocating means is the frequency component that is divided into two after repeating the desired number of frequency decompositions, and the lower frequency component is the lower frequency component. component, at the time of the frequency component of the higher frequency side and high frequency components, wherein Ri low frequency components der brightness of the two-dimensional image, the parallax amount is a frequency component other than the low-frequency frequency components in which the parallax amount is allocated The image processing method according to claim 1, wherein no image is assigned .
The fifth invention is the image processing method according to any one of claims 1 to 4, wherein the amount of parallax is horizontal parallax.
The sixth aspect of the present invention is the image processing method according to claim 5, wherein the horizontal parallax is large for a component having a high lightness and small for a component having a low lightness. .
The image processing method according to claim 5, wherein the seventh invention provides the horizontal parallax to be large for a component having a high saturation and to be small for a component having a low saturation. It is.
Further, according to an eighth aspect of the invention, there is provided a frequency resolving means for calculating a frequency component by spatially resolving a spatial frequency component of one two-dimensional image, and a parallax amount specifying means for specifying a parallax amount to be given to an image to be three-dimensionalized. Obtained by the parallax amount allocating means for allocating the parallax amount to the frequency component obtained by the decomposition, the frequency synthesizing means for synthesizing each frequency component obtained through the parallax amount allocating means, and the frequency synthesizing means. From the obtained result, the image processing apparatus includes at least an image output unit that outputs a left-eye image and a right-eye image that enables binocular stereoscopic vision.
The ninth aspect of the present invention is the image processing device according to claim 8, wherein the frequency component to which the parallax amount is allocated by the parallax amount allocating unit is a frequency component of brightness of the two-dimensional image. .
According to a tenth aspect of the invention, the frequency component to which the parallax amount is allocated by the parallax amount allocating means is the frequency component divided into two after repeating the desired number of frequency decompositions, the frequency component on the lower frequency side being the low frequency component, at the time of the frequency component of the higher frequency side and high frequency components, wherein Ri high frequency component der brightness of the two-dimensional image, the parallax amount is a frequency component other than the low-frequency frequency components in which the parallax amount is allocated The image processing apparatus according to claim 8, wherein no image is assigned .
According to an eleventh aspect of the invention, the frequency component to which the parallax amount is allocated by the parallax amount allocating means is the frequency component with the lower frequency among the frequency components divided into two after repeating the desired number of frequency decompositions. component, at the time of the frequency component of the higher frequency side and high frequency components, wherein Ri low frequency components der brightness of the two-dimensional image, the parallax amount is a frequency component other than the low-frequency frequency components in which the parallax amount is allocated The image processing apparatus according to claim 8, wherein no image is assigned .
The twelfth invention is the image processing apparatus according to any one of claims 8 to 11, wherein the parallax amount is horizontal parallax.
The thirteenth aspect of the present invention is the image processing apparatus according to claim 12, wherein the horizontal parallax is large for a component having a high lightness and small for a component having a low lightness. is there.
The fourteenth aspect of the present invention is the image processing according to claim 12, wherein the horizontal parallax is large for a component with high saturation of an image and small for a component with low saturation. Device.
A fifteenth aspect of the invention is an image processing program that causes an arithmetic device to execute the image processing method according to any one of claims 1 to 7.

According to the first invention, a single two-dimensional image of the surface of the object is decomposed into individual frequency components by applying spatial frequency decomposition, and a predetermined horizontal parallax amount is obtained for a desired spatial frequency component. By emphasizing and generating a left-eye image and a right-eye image by frequency synthesis, the stereoscopic effect on the object surface can be emphasized or suppressed without discomfort under viewing conditions that allow binocular stereoscopic viewing. Is possible.
In addition, the present invention is particularly effective in enhancing the thickness of the object surface, and is particularly effective for materials that humans feel thickness, such as carpets, furs, woolen fabrics, thick fabrics, and clothing. is there. In addition, since it is possible to generate a pseudo-parallax even if it is not an image or video originally taken for stereoscopic viewing, it is utilized when generating a pseudo-stereo image or stereo video from an existing image or video. be able to.
By observing these in a binocular stereoscopic environment, the stereoscopic effect on the surface is enhanced or suppressed without a sense of incongruity.
For example, a binocular stereoscopic image, video, or animation can be easily realized from an image, video, or animation captured by a single imaging device, which is realistic in a binocular stereoscopic viewing environment. Pseudo stereoscopic vision can be realized.
According to the second invention, it is possible to enhance or suppress the stereoscopic effect on the object surface without a sense of incongruity by decomposing the spatial frequency component of brightness.
By separating the lightness component and the chromaticity component and processing only the lightness component, it is possible to effectively enhance or suppress the stereoscopic effect while suppressing the color change to the maximum.
Further, according to the third invention, it is possible to enhance or suppress the stereoscopic effect on the object surface without a sense of incongruity by decomposing and processing the high-frequency component of brightness.
In particular, since human vision is highly sensitive to lightness components mainly for medium and high frequency components, applying the present invention to high frequency components of lightness has an effective effect on stereoscopic effect. Can be obtained.
The present invention is particularly suitable for emphasizing the surface three-dimensional effect of details of materials with dense hair such as thick fur. Also, it is effective for scenes where the camera gradually approaches and zooms in on the video, and scenes where the surface of the target object is more closely projected.
Further, according to the fourth invention, it is possible to enhance or suppress the stereoscopic effect on the object surface without a sense of incongruity by decomposing and processing the high frequency component of brightness.
Since the low frequency component of the brightness mainly reflects the surface shape of the object (uneven shape with a large area over the entire surface), the three-dimensional effect of the surface shape can be effectively enhanced or suppressed.
For example, the present invention is a comparison of objects such as the back, abdomen, buttocks, or the rise of clouds floating in the air, sand dunes in the desert, wave swells on the sea surface, etc. It is suitable for emphasizing or suppressing the three-dimensional effect of a large area. In addition, it is effective in projecting a wide area of the object in a zoom-out scene where the camera gradually takes a long distance in the video.
In addition, according to the fifth aspect, by adopting horizontal parallax as the parallax amount, it is possible to enhance or suppress the stereoscopic effect on the object surface without a sense of incongruity under viewing conditions that allow binocular stereoscopic viewing. Become.
Further, according to the sixth aspect of the invention, the horizontal parallax is adopted as the amount of parallax, and the stereoscopic effect on the surface of the object is uncomfortable by giving a large value to a component having a high brightness and a small value to a component having a low brightness. It is possible to emphasize and suppress without any loss.
In particular, in the case of a material having a multi-layered network structure on the surface structure, such as multi-layer gauze, cotton, woolen fabric, or nylon scrubbing, the reflection is strong and bright as it is closer to the surface, and the reflection is weak and dark as it goes deeper. By giving a large amount of parallax to a component with high brightness and giving a small amount of parallax to a component with low brightness, the depth difference between the front side and the back side can be gradually emphasized, resulting in a natural sense of depth. Can be emphasized.
In addition, according to the seventh aspect, by adopting horizontal parallax for the amount of parallax, giving a large value for a component with high saturation and a small value for a component with low saturation, the stereoscopic effect on the object surface is uncomfortable. It is possible to emphasize and suppress without any loss.
This is based on the rule of thumb that the saturation of the surface of the target object in the vicinity is strong and the saturation of the surface of the target object in the distance is weak. For example, costumes and curtains with flashy colors, Persian carpets, etc. This is effective when it is desired to obtain a three-dimensional surface effect of a colorful material, and is effective for reproducing a change in the three-dimensional effect on the object surface during zoom-in / zoom-out in an image.
According to the eighth invention, the spatial frequency decomposition is applied to one two-dimensional image of the surface of the object to be decomposed into individual frequency components, and a predetermined horizontal parallax amount is obtained for the desired spatial frequency component. By emphasizing and generating a left-eye image and a right-eye image by frequency synthesis, the stereoscopic effect on the object surface can be emphasized or suppressed without discomfort under viewing conditions that allow binocular stereoscopic viewing. Is possible.
In addition, the present invention is particularly effective in enhancing the thickness of the object surface, and is particularly effective for materials that humans feel thickness, such as carpets, furs, woolen fabrics, thick fabrics, and clothing. is there. In addition, since it is possible to generate a pseudo-parallax even if it is not an image or video originally taken for stereoscopic viewing, it is utilized when generating a pseudo-stereo image or stereo video from an existing image or video. be able to.
By observing these in a binocular stereoscopic environment, the stereoscopic effect on the surface is enhanced or suppressed without a sense of incongruity.
For example, a binocular stereoscopic image, video, or animation can be easily realized from an image, video, or animation captured by a single imaging device, which is realistic in a binocular stereoscopic viewing environment. Pseudo stereoscopic vision can be realized.
According to the ninth aspect, by decomposing the spatial frequency component of the brightness, it is possible to enhance or suppress the stereoscopic effect on the object surface without a sense of incongruity.
By separating the lightness component and the chromaticity component and processing only the lightness component, it is possible to effectively enhance or suppress the stereoscopic effect while suppressing the color change to the maximum.
In addition, according to the tenth invention, it is possible to enhance or suppress the stereoscopic effect on the object surface without a sense of incongruity by decomposing and processing the high-frequency component of brightness.
In particular, since human vision is highly sensitive to lightness components mainly for medium and high frequency components, applying the present invention to high frequency components of lightness has an effective effect on stereoscopic effect. Can be obtained.
The present invention is particularly suitable for emphasizing the surface three-dimensional effect of details of materials with dense hair such as thick fur. Also, it is effective for scenes where the camera gradually approaches and zooms in on the video, and scenes where the surface of the target object is more closely projected.
In addition, according to the eleventh aspect, it is possible to enhance or suppress the stereoscopic effect on the object surface without a sense of incongruity by decomposing and processing the high-frequency component of brightness.
Low-frequency component of brightness The low-frequency component mainly reflects the surface shape of the object (uneven shape having a large area over the entire surface), so that the stereoscopic effect of the surface shape can be effectively enhanced or suppressed.
For example, the present invention is a comparison of objects such as the back, abdomen, buttocks, or the rise of clouds floating in the air, sand dunes in the desert, wave swells on the sea surface, etc. It is suitable for emphasizing or suppressing the three-dimensional effect of a large area. In addition, it is effective in projecting a wide area of the object in a zoom-out scene where the camera gradually takes a long distance in the video.
Further, according to the twelfth aspect, by adopting horizontal parallax as the parallax amount, it is possible to enhance or suppress the stereoscopic effect on the object surface without a sense of incongruity under viewing conditions that allow binocular stereoscopic viewing. Become.
Further, according to the thirteenth aspect, by adopting horizontal parallax as the amount of parallax and giving a large value to a component having a high lightness and a small value to a component having a low lightness, the stereoscopic effect on the object surface is uncomfortable. It is possible to emphasize and suppress without any loss.
In particular, in the case of a material having a multi-layered network structure on the surface structure, such as multi-layer gauze, cotton, woolen fabric, or nylon scrubbing, the reflection is strong and bright as it is closer to the surface, and the reflection is weak and dark as it goes deeper. By giving a large amount of parallax to a component with high brightness and giving a small amount of parallax to a component with low brightness, the depth difference between the front side and the back side can be gradually emphasized, resulting in a natural sense of depth. Can be emphasized.
According to the fourteenth aspect of the present invention, the horizontal parallax is adopted as the parallax amount, and the stereoscopic effect on the object surface is uncomfortable by giving it large to a component with high saturation and small to a component with low saturation. It is possible to emphasize and suppress without any loss.
This is based on the rule of thumb that the saturation of the surface of the target object in the vicinity is strong and the saturation of the surface of the target object in the distance is weak. For example, costumes and curtains with flashy colors, Persian carpets, etc. This is effective when it is desired to obtain a three-dimensional surface effect of a colorful material, and is effective for reproducing a change in the three-dimensional effect on the object surface during zoom-in / zoom-out in an image.
In addition, according to the fifteenth aspect, by causing the present invention to be executed by a computing device such as a personal computer, the stereoscopic effect on the object surface can be emphasized or suppressed without discomfort under observation conditions that allow binocular stereoscopic viewing. Is possible.
According to the first to fifteenth inventions, in a video, by appropriately selecting and applying the high-frequency component stereoscopic effect enhancement method and the low-frequency component stereoscopic effect enhancement method of the present invention on the time axis, It is also possible to emphasize or suppress the movement of viscous objects such as swells on the water surface and the movement of fabrics such as flags fluttering in the wind.

1 is a schematic diagram illustrating an example of an image processing method of the present invention. The processing flowchart which showed an example of the image processing method of this invention. The processing flowchart which showed an example of the frequency decomposition process of this invention. The processing flowchart which showed an example of the parallax amount allocation process of this invention. The processing flowchart which showed an example of the frequency synthesis process of this invention. Explanatory drawing which shows the concept of wavelet transformation and reverse transformation. The figure which shows the shape of the wavelet function and scaling function of a Daubechies wavelet (10th order). The conceptual diagram which shows an example of the image presentation environment which can display the binocular parallax image of the observation target object which displays the image produced | generated by this invention. The conceptual diagram which shows the mode of the single virtual image which an observer can perceive in the environment of FIG.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 8 is an example of an image presentation environment in which a binocular parallax image of an observation target object can be presented for displaying an image generated using the present invention. FIG. 9 is an example of a conceptual diagram showing a state of a single virtual image that can be perceived by the observer in the environment of FIG.

The images displayed in the left-eye image 1 and the right-eye image 2 are reflected by the fusion mirror 3 and are observed with both eyes of the observer. When the same image is displayed on both images, the viewer can perceive as if a single virtual image 4 exists in front as shown in FIG. 9 by appropriate arrangement of the image and the mirror. Bring.
In addition, changing the arrangement of the image and the mirror changes the parallax angle, which changes the sense of distance to the image. In addition, the parallax angle changes and the sense of distance also changes by adding different amounts of translation to the images displayed in both images without changing the arrangement of the images and mirrors.

  FIG. 1 is a schematic diagram showing an example of the image processing method of the present invention.

In the image input means S1, an image to be three-dimensionalized is input to the apparatus of the present invention.
In the image input means S1, two-dimensional image data I (x, y) is input from a storage device such as a hard disk device, a communication line through a network, a video signal line connected to a video device or the like.

Next, in the image conversion means S3, the image input from the image input means is converted, and the lightness image L (x, y), the chromaticity images C ₁ (x, y), C ₂ (x, y), and the chromaticity are displayed. Degree image S (x, y) is output.

  Next, the parallax amount designating unit S2 inputs the parallax amount to be given to the image to be three-dimensionalized to the apparatus of the present invention. The parallax amount input means S2a inputs the parallax amount H. This can be realized, for example, by inputting an arbitrary numerical value from an input screen presented on a computer terminal or by selecting from a plurality of candidate values. Or you may input via the input device etc. which arranged the button, the dial, etc.

  The parallax amount of the present invention assumes binocular parallax, and is intended for parallax in a direction parallel to the direction of both eyes of the observer and horizontal parallax in the horizontal direction of the parallax image (the horizontal direction of the image) Adjust them accordingly.

  For example, the parallax amount input means S2a gives a predetermined numerical value h as the horizontal parallax amount H (H = h). Next, the correction coefficient setting means S2b receives the lightness component image L (x, y) and the saturation component image S (x, y) output from the image conversion means S3, and monotonically increases the numerical value k and the lightness v. The parallax correction coefficient K is set in one of the function f (v) and the monotonically increasing function g (s) of the saturation s (K = k, K = f (v), K = g (s) ).

  Here, the parallax correction coefficient is given as a monotonically increasing function of brightness, because the surface part of the object surface appears brighter and the deeper part appears darker due to the nature of light reflection. Yes.

  Also, the parallax correction factor is given as a monotonically increasing function of saturation because the closer to the human eye looks brighter and the farther away the object appears to be weaker. Further, the surface layer portion is vivid, that is, the saturation is high, and conversely, the deep portion corresponds to the light color, that is, the saturation is low.

  Next, in the parallax correction unit S2c, a predetermined parallax amount H is received from the parallax input unit S2a, a correction coefficient K is received from the correction coefficient specifying unit S2b, and H ′ = K × H as a parallax correction amount H ′. Is calculated.

Subsequently, the frequency selection means S4 calculates the maximum decomposition level M of the image received from the image conversion means S3, and selects a desired decomposition level from decomposition levels 1 to M.
Here, the decomposition level corresponds to the number of decompositions when an image is decomposed into frequency components. For example, when the frequency is defined in units of pixels, which are the minimum structural unit of an image, the finest, that is, the highest frequency is when two pixels are one wavelength. This corresponds to the frequency 1/2 at the decomposition level 1. Next, at decomposition level 2, the frequency becomes ½ of the frequency, and thereafter, the same is repeated, and at decomposition level n, frequency becomes 1/2 ⁿ .
Note that the frequency value at each decomposition level depends on the unit size when the frequency is defined.

  Next, the frequency decomposition means S5 performs spatial frequency decomposition on the image converted by the image conversion means S3.

Here, when calculating individual frequency components by spatial frequency decomposition of a two-dimensional image, it can be performed on various image signals. For example, there is a method of calculating individual frequency components by spatially resolving lightness components and chromaticity components.
Since the present invention relates to human stereoscopic vision, it is considered appropriate to apply it to signals that are meaningful in terms of visual perception. Regarding the lightness component and chromaticity component, which are image signal components, the sensitivity characteristic for the lightness component shows a bandpass type, while the sensitivity characteristic for the chromaticity component is low-passed, with respect to the spatial frequency perception of human visual perception. Since the lightness component is mainly used for the perception of the medium and high frequency components, the spatial frequency processing in the present invention is preferably performed on the lightness component. Hereinafter, the case where the brightness component is applied will be described.

In the parallax amount instruction means S2, a horizontal parallax amount is designated. In the parallax amount specifying means S2a, the predetermined parallax amount h is received and H = h is set, and then in the correction coefficient specifying means S2b,
As the parallax correction coefficient K, any one of a numerical value Kv, a brightness function Kl, and a saturation function Ks is received, and K = Kv, K = Kl, and K = Ks corresponding to each of them.

  Then, in the parallax amount correcting means S2c, the predetermined parallax amount H is received from the parallax amount specifying means S2a, the parallax correction coefficient K is received from the correction coefficient specifying means S2b, and the parallax correction amount H ′ is H ′ = K × H Is calculated.

  Next, in the parallax amount allocating unit S6, the parallax correction amount H ′ specified by the parallax amount specifying unit S2 is allocated to the frequency component obtained by the frequency decomposing unit S5.

  At this time, the parallax correction amount H ′ is a method in which the high frequency component is shifted to the left and right by a predetermined horizontal parallax amount and the other frequency components are not shifted depending on the type of image. A new horizontal parallax amount that is shifted left and right by the amount of parallax and other frequency components are not shifted, or a value obtained by multiplying a predetermined horizontal parallax amount by a constant or a correction coefficient given by a monotonically increasing function of the brightness component Adopting a method of shifting the desired frequency component, adopting a value obtained by multiplying a predetermined horizontal parallax amount by a constant or a correction coefficient given by a monotonically increasing function of the saturation component as a new horizontal parallax amount, A method for shifting the frequency component is appropriately selected, or a combination thereof is used to assign the parallax amount.

Specifically, among the frequency components F _i (x, y) received from the frequency resolution unit S5, the parallax correction amount H ′ as the horizontal parallax amount for the frequency component at the resolution level i to which 1 is assigned. Are assigned symmetrically in the horizontal direction, and the frequency component image F _i ^L (x, y) for the left eye and the right eye are not assigned to the frequency component of the decomposition level i to which 0 is assigned, without assigning a horizontal parallax amount. A frequency component image F _i ^R (x, y) is generated as follows.

Next, in the frequency synthesizer S7, the decomposed frequency components are synthesized to form an image.
In the frequency synthesizing unit S7, the process proceeds in the opposite direction to the frequency decomposition method. An inverse wavelet transform is applied to the decomposition level n low frequency image and the decomposition level n wavelet coefficients to obtain a decomposition level n-1 low frequency image. Next, by applying wavelet inverse transformation to the low-frequency image at decomposition level n-1 and the wavelet coefficients, a low-frequency image at decomposition level n-2 is obtained. By sequentially repeating this operation, an image at the decomposition level 0, that is, the original image can be reconstructed.

  By performing frequency decomposition by wavelet transform, wavelet coefficients for each decomposition level are obtained. The decomposition level corresponds to a spatial resolution of one power of 2 due to the two-scale relationship. For example, the decomposition level i is the spatial resolution of the original image.

Twice the spatial resolution, which is

This corresponds to a double spatial frequency component.

  Therefore, by applying the synthesis method without applying horizontal parallax to a desired frequency component, that is, a wavelet coefficient having a desired decomposition level, and applying no horizontal parallax to other wavelet coefficients, only the desired frequency component is obtained. Left eye image with horizontal parallax

And right-eye images are obtained.

Next, the image output means S8 outputs the result obtained by the frequency synthesizing means S7 as an image of a desired format.
Specifically, the image output means S8 outputs the frequency composite image for the left eye from the frequency composition means.

And right eye frequency composite image

The chromaticity images C ₁ (x, y) and C ₂ (x, y) are received from the image converting means S2, and color conversion is applied from these to the left eye image I which is a desired two-dimensional image. ^L (x, y) and right eye image I ^R (x, y) are output.

  The medium for outputting the result as an image is not particularly limited as long as the image can be displayed, and examples thereof include a liquid crystal display, plasma, CRT, and organic EL.

FIG. 2 is a processing flowchart showing an example of the image processing method of the present invention. Hereinafter, an embodiment of the present invention will be described using the processing flow of FIG.
First, in the image reading process P1, the two-dimensional image data I (x, y) to be processed is read.

Next, in the image conversion process P2, the two-dimensional image data I (x, y) read in the image reading process P1 is separated into a brightness component and a chromaticity component, and the two-dimensional image L (x, y) of the brightness component is separated. y) and a chromaticity component two-dimensional image C ₁ (x, y), C ₂ (x, y) and a saturation component two-dimensional image S (x, y), respectively.

  As a method for separating the color information into lightness information and chromaticity information, any color system such as Y (R−Y) (BY), CIELAB, or HSV may be used. Here, when the RGB color system is used as a reference for the most commonly used two-dimensional image color system, Y (R−Y) (BY) is used for broadcasting, and brightness Y Is calculated by Equation 1, and the saturation S is calculated by Equation 2.

  Y = 0.299R + 0.587G + 0.114B (Formula 1)

  On the other hand, in the parallax amount input process P3, first, the parallax amount H is input via the parallax amount input means S2a. This can be realized, for example, by inputting an arbitrary value from an input screen presented on a computer terminal or by selecting from a plurality of candidate values. Or you may input from the input device etc. which arranged the button, the dial, etc.

  In the correction coefficient calculation process P4, the lightness image L (x, y) and the saturation image S (x, y) are received from the image conversion P2, and the processing value k, the function f (v) of the lightness v, the saturation s The correction coefficient K is calculated in one of the functions g (s) (K = k, K = f (v), K = g (s)).

  Here, as the brightness function f (v), for example, if the maximum brightness value of the brightness image L (x, y) is Vmax and the minimum value is Vmin, then f = v (L, x, y) and f (v ) = V / (Vmax−Vmin). Similarly, as for the saturation function g (s), assuming that the maximum value of the saturation of the saturation image S (x, y) is Smax and the minimum value is Smin, s = S (x, y) and g ( s) = s / (Smax−Smin). Of course, the present invention is not limited to these, and various functions can be selected within a monotonically increasing range.

  Next, in the parallax amount correction processing P5, the lightness component image L (x, y) and the saturation component image S (x, y) are received from the image conversion processing P2, and a predetermined amount of parallax is received from the parallax amount input processing P3. H is received, the correction coefficient K is received from the correction coefficient calculation process P4, and H ′ = K × H is calculated as the parallax correction amount H ′.

  Next, in the maximum decomposition level calculation process P6, the lightness component image L (x, y) output in the image conversion process P2 is received, the width width and height height are read, and the smaller value of width and height is obtained. The maximum power of 2 that does not exceed is calculated, and that power is substituted into M as the maximum decomposition level.

  Next, in the frequency selection process P7, the maximum decomposition level M is received from the maximum decomposition level calculation process P6, a desired decomposition level is designated between decomposition levels 1 to M, and the parallax is determined according to each decomposition level. It is stored in list (i) which is a list for designating whether to give or not. Here, a smaller frequency represents a higher frequency, and a larger level represents a lower frequency.

  As for the selection method, in order to instruct the processing according to the nature of the subject of the image, when processing only the high frequency component, select the decomposition level of the high frequency component, and when processing only the low frequency component Selects the decomposition level of the low frequency component, selects only the decomposition level of the desired frequency component when processing only the desired frequency, and selects the entire decomposition level when processing regardless of the frequency.

  As a method for realizing list (i), which is a list for specifying parallax, for example, list (i) = 1 is given when the decomposition level i is selected, and list (i) = 0 is given when not selected. Output. If the frequency component division number is M, list (i) can be realized, for example, as data of length M bits, or can be realized as an array of size M.

In the frequency decomposition process P8, the brightness image L (x, y) calculated in the image conversion process P2 is sequentially subjected to spatial frequency conversion from the decomposition level 1 to the maximum decomposition level M obtained in the maximum decomposition level calculation process. The frequency component image F _i (x, y) at each decomposition level is calculated.

Here, a frequency component decomposition method will be described. FIG. 3 is a process flow diagram showing an example of the frequency decomposition process of the present invention. First, c ⁰ = L (x, y), the decomposition level is i, 1 is substituted for i (P8b), and the first spatial frequency transformation is applied (P8c). Thereby, decomposition coefficients c ¹ and d ¹ of decomposition level 1 are obtained (P8d).
Next, i is updated with a value obtained by adding 1 to the value of i (P8e), and compared with the maximum number of decomposition levels M (P8f). If the value of i is M or less, decomposition coefficients of decomposition level 2 by applying processing again from P8c against ^{c 1} until ^P8f, the determined c 2, ^{d 2,} less, the processing from P8c to P8f The decomposition coefficients c ⁱ and d ⁱ are sequentially calculated by repeating. When the value of i exceeds M, the frequency decomposition process is terminated. As a result, frequency component images F _i (x, y) = c ^{i at} the respective decomposition levels are obtained.

Next, with regard to the frequency conversion method, many methods have been devised, particularly in the field of acoustic technology. For example, the Fourier transform method and the wavelet transform method are representative. In the present invention, a case where a wavelet transform method with high spatial localization accuracy is applied is introduced here as a more preferable example (although various frequency conversion methods can be used as appropriate).
The wavelet transform is a method for decomposing and generating a signal using two types of functions (scaling function and wavelet function) that satisfy a two-scale relationship and an orthonormal system condition. The basic content of the wavelet transform is described in, for example, “Non-Patent Document 2”. A function system for configuring a wavelet is described in, for example, “Non-Patent Document 3”.
Examples of wavelet functions include Daubechies wavelets and cardinal spline wavelets. A wavelet function that is straightforward and smooth in nature is suitable for the present invention.

From the image to be processed, a region having the maximum area and the width and height of the image are equal to each other and the value is equal to a power of 2 is extracted as a processed image. For this image, a scaling factor and a wavelet coefficient of a desired wavelet function are prepared, and wavelet transformation is performed. As shown in FIG. 6, every time wavelet transform is performed, the frequency is halved and decomposed into a high frequency component and a low frequency component. Here, the state of the original image before conversion is called decomposition level 0, the case where the conversion is applied once is called decomposition level 1, and the state where the conversion is repeated n times is called decomposition level n. Therefore, when the repeated conversion to n times, as the highest frequency spatial frequency of the original image, its 1/2, 1/4-fold, ..., are disassembled into 1/2 ⁿ times the frequency component .

  The following relationship holds between the decomposition component at decomposition level k and the decomposition component at decomposition level k-1 (0 <k <n).

These formulas represent the original signal ^ck and the left eye signal obtained by shifting the original signal to the right by a predetermined horizontal parallax amount H.

And a signal for the right eye obtained by shifting the original signal to the left by a predetermined horizontal parallax amount H

Is a filter:

Depending on the resolution one step lower:

This means that it can be disassembled. Where the function

Is called a wavelet function, also a function

Is called a scaling function.

  FIG. 7 shows waveforms of the Daubechies wavelet (10th order) wavelet function and scaling function.

In the parallax amount allocation processing P9, the parallax correction amount H ′ is received from the parallax amount correction processing P5, the instruction list list (i) is received from the frequency selection processing P7, and each frequency component image calculated from the frequency resolution processing P8 is received. F i receives _(x, y), the frequency component image F _{i (x,} y) of the designated resolution level i the parallax correction amount H 'given as horizontal parallax with respect to the left-eye frequency component image F _i ^L (x, y) and the right-eye frequency component image F _i ^R (x, y) are output.

Next, the parallax amount allocation process P9 will be described in detail. FIG. 4 is a process flow diagram showing an example of the parallax amount allocation process P9 of the present invention.
First, the selection list list (i) is received from the frequency selection instruction process P5, the parallax correction amount H ′ is received from the parallax amount correction process P5, and each frequency component image from the frequency resolution process P8 to the decomposition levels 1 to M is received. F _i (x, y) is received.
Now, 1 is assigned to the decomposition level i (P9a), the value of list (i) is compared with 1 (P9b), and if they match, the horizontal parallax is symmetrical to the frequency component image F _i (x, y) at the decomposition level i. And the frequency component image for left eye F _i ^L (x, y) and the frequency component image for right eye F _i ^R (x, y) at the decomposition level i are calculated. Specifically, the frequency component image for the left eye at the decomposition level i by giving the parallax H ′ in the horizontal right direction to the frequency component image F _i (x, y) at the decomposition level i.

Similarly, a frequency component image for right eye at decomposition level i is given by giving a parallax H ′ in the horizontal left direction to the frequency component image F _i (x, y) at decomposition level i.

Is calculated. This process (from P9e to P9i) is repeated while increasing the value of x by 1 from the value of x to 0 to width and further increasing the value of y by 1 from the value of y to 0 to height.
On the other hand, if the value of list (i) does not match 1, the frequency component image F _i (x, y) at the decomposition level i is converted to the frequency component image F _i ^L (x, y) for the left eye at the decomposition level i and Substitute into the frequency component image F _i ^R (x, y) for the right eye (F _i ^L (x, y) = F _i (x, y), F _i ^R (x, y) = F _i (x, y) )). This process (from P9j to P9n) is repeated while increasing the value of x by 1 from the value of x to 0 to width, and is further repeated while increasing the value of y by 1 from the value of y to 0 to height.
Next, 1 is added to the value of i and compared with M. If it is less than or equal to M, the same processing (from P9b to P9o) is repeated again, and the processing ends when M is exceeded.

In the frequency synthesis process P8, the left-eye frequency component image F _i ^L (x, y) and the right-eye frequency component image in which a predetermined horizontal parallax is given to the desired frequency component in the parallax amount allocation process P7. From F _i ^R (x, y), starting from the decomposition level M, the inverse image of the frequency decomposition process is applied to calculate a frequency image of the decomposition level M−1 that is one step lower. This is repeated in sequence until a resolution level 0 frequency image F ₀ ^L (x, y) and a right eye frequency image F ₀ ^R (x, y) are finally obtained.

Next, the frequency synthesis process P10 will be described in detail.
In FIG. 6, let us consider that the flow of an arrow going from left to right is advanced from right to left in the reverse direction. Using the low-frequency component and the high-frequency component at the maximum decomposition level M, the frequency is doubled each time the wavelet inverse transformation is performed, and the decomposition level is reduced to one less frequency component. Therefore, when the conversion is repeated ^M times, frequency components of the maximum decomposition level are doubled, quadrupled,..., ^2M times. The frequency component of ^2M times represents a level where decomposition is not performed, and the frequency component of the original image is synthesized after all.
Hereinafter, the processing will be specifically described.
FIG. 5 is a process flow diagram showing an example of the frequency synthesis process of the present invention.
The left-eye frequency component images F _i ^L (x, y) and the right-eye frequency component images F _i ^R (x, y) from the resolution levels 1 to M are received from the parallax amount allocation process P9.
Now, assuming that the decomposition level is i, first, the maximum decomposition level number M is substituted for i (P10a). Left eye frequency component image

And right eye frequency component images

The inverse frequency transform image (Equation 3) and (Equation 4) of the frequency transform P8c is applied to (P10b), and the frequency component image for the left eye having a level of decomposition one less

And right eye frequency component images

Is calculated.

  Next, the value of i is updated with a value reduced by 1 and compared with 1. If it is 1 or more, the inverse transformation process (from P10b to P10e) is applied again, and if it is 0, the process is terminated. As a result, the frequency composite image for the left eye is finally obtained.

And right eye frequency composite image

Is obtained.

  Next, in the image conversion process P11, the frequency image for the left eye which is the output of the frequency synthesis process P10

And right eye frequency image

And the chromaticity images C ₁ (x, y) and C ₂ (x, y) separated in the image conversion process P2 are received and converted into the same image format as the original image I (x, y). The left-eye two-dimensional image I ^L (x, y) and the right-eye two-dimensional image I ^R (x, y) are output.

  This time, a still image is used as an example to explain the present invention. The present invention is intended for general two-dimensional images, and the application target is not limited to still images. For example, the present invention can be similarly applied to each frame image of 2D video, CG (computer graphics) image, and each frame image of 3DCG video.

  In addition, there is a method of combining MTD (Modified Time Difference time difference modification) particularly when a two-dimensional image is targeted. The MTD is a method for obtaining a stereoscopic effect in which, for example, a subject appears to jump out of the background by giving a time delay to the two-dimensional image projected on the other eye with respect to the two-dimensional image projected on one eye. It is. Therefore, it is considered effective to apply the image processing method of the present invention to the two-dimensional video (image) of each frame, and to apply the MTD only to the high frequency component of the luminance component of the video.

  The present invention is a method for generating a left-eye image and a right-eye image for binocular stereoscopic viewing by image processing a single two-dimensional image, and is a stereoscopic image represented by a 3D movie that has recently become widespread. It can be applied to images and stereoscopic images in general.

DESCRIPTION OF SYMBOLS 1 ... Left eye image 2 ... Right eye image 3 ... Fusion mirror 4 ... Virtual image 5 ... Parallax angle D1 ... Daubechies wavelet (10th order) scaling Function D2 ... Daubechies wavelet (tenth) wavelet function P1 ... Image reading process P2 ... Image conversion process P3 ... Parallax amount input process P4 ... Correction coefficient calculation process P5 ... Parallax Quantity correction processing P6... Maximum decomposition level calculation processing P7... Frequency selection instruction processing P8... Frequency decomposition processing P8a... Set decomposition level to variable i P8b. Frequency conversion process P8d: Stores decomposition coefficient P8e: Adds 1 value to P8f ... Compares i value with M P9 ... Parallax amount allocation process P9a ... Sets variable i to 1 Setting P9 ... Compare list (i) with 1 P9c ... Set 0 to variables x, y P9d ... Set 0 to variables x, y P9e ... Left eye and right eye frequency components Is set with a horizontal parallax H 'symmetrically set to the right and left in the x direction. P9f: Add 1 to the value of the variable x P9g ... Compare the value of the variable x to the width P9h P9i for converting the value of variable y to 1 P9i: Compare the value of variable y with height P9j: Set the frequency component for the left eye and right eye frequency components P9k: Add 1 to the value of variable x P9l: Compare the value of variable x with width P9m ... Convert the value of variable y to 1 P9n ... Compare the value of variable y with height P9o ... Add 1 to the value of variable i P9p ..Comparing the value of the variable i with M P10... Frequency composition processing P10a... Setting M to the variable i P10b. Frequency inverse transform P10c from the decomposition coefficient and decomposition image of bell i... Decomposition image of decomposition level i-1 is calculated P10d ... The value of variable i is decremented by 1 P10e ... The value of variable i is compared with 1 P11 ... image conversion processing S1 ... image input means S2 ... parallax amount designation means S2a ... parallax amount input means S2b ... correction coefficient designation means S2c ... parallax amount correction means S3 ... image Conversion means S4... Frequency selection means S5... Frequency decomposition means S6... Parallax amount assignment means S7.

Claims (15)

Frequency resolving means for calculating a frequency component by spatially resolving a spatial frequency component of one two-dimensional image;
A parallax amount designation means for designating a parallax amount to be given to an image to be three-dimensionalized;
Parallax amount allocating means for allocating the parallax amount to the frequency component obtained by the decomposition;
Frequency synthesis means for synthesizing each frequency component obtained through the parallax amount allocation means;
From the result obtained by the frequency synthesizing means, an image output means for outputting a left-eye image and a right-eye image that enables binocular stereoscopic vision;
An image processing method comprising:   The image processing method according to claim 1, wherein the frequency component to which the parallax amount is allocated by the parallax amount allocation unit is a frequency component of brightness of the two-dimensional image. The frequency component to which the amount of parallax is allocated by the parallax amount allocating means is a frequency component that is divided into two frequency components after repeating frequency decomposition a desired number of times, the frequency component on the lower frequency side is the low frequency component, and the frequency on the higher frequency side at the time of the components and high frequency components, according to the are two-dimensional high frequency components der of the brightness of the image, the frequency components other than the high frequency components parallax amount is assigned characterized in that it does not assign the amount of parallax Item 8. The image processing method according to Item 1. The frequency component to which the amount of parallax is allocated by the parallax amount allocating means is a frequency component that is divided into two frequency components after repeating frequency decomposition a desired number of times, the frequency component on the lower frequency side is the low frequency component, and the frequency on the higher frequency side at the time of the components and high frequency components, wherein Ri low frequency components der brightness of the two-dimensional image, characterized in that it does not assign the amount of parallax in the frequency components other than the low-frequency frequency components in which the parallax amount is allocated The image processing method according to claim 1.   The image processing method according to claim 1, wherein the amount of parallax is horizontal parallax.   The image processing method according to claim 5, wherein the horizontal parallax is given to a component having a high lightness of the image and small to a component having a low lightness.   The image processing method according to claim 5, wherein the horizontal parallax is given to a component having a high saturation of an image and is given a small value to a component having a low saturation. Frequency resolving means for calculating a frequency component by spatially resolving a spatial frequency component of one two-dimensional image;
A parallax amount designation means for designating a parallax amount to be given to an image to be three-dimensionalized;
Parallax amount allocating means for allocating the parallax amount to the frequency component obtained by the decomposition;
Frequency synthesis means for synthesizing each frequency component obtained through the parallax amount allocation means;
From the result obtained by the frequency synthesizing means, an image output means for outputting a left-eye image and a right-eye image that enables binocular stereoscopic vision;
An image processing apparatus comprising at least   The image processing apparatus according to claim 8, wherein the frequency component to which the parallax amount is allocated by the parallax amount allocation unit is a frequency component of brightness of the two-dimensional image. The frequency component to which the amount of parallax is allocated by the parallax amount allocating means is a frequency component that is divided into two frequency components after repeating frequency decomposition a desired number of times, the frequency component on the lower frequency side is the low frequency component, and the frequency on the higher frequency side at the time of the components and high frequency components, wherein Ri high frequency component der brightness of the two-dimensional image, characterized in that it does not assign the amount of parallax in the frequency components other than the low-frequency frequency components in which the parallax amount is allocated The image processing apparatus according to claim 8. The frequency component to which the amount of parallax is allocated by the parallax amount allocating means is a frequency component that is divided into two frequency components after repeating frequency decomposition a desired number of times, the frequency component on the lower frequency side is the low frequency component, and the frequency on the higher frequency side at the time of the components and high frequency components, wherein Ri low frequency components der brightness of the two-dimensional image, characterized in that it does not assign the amount of parallax in the frequency components other than the low-frequency frequency components in which the parallax amount is allocated The image processing apparatus according to claim 8.   The image processing apparatus according to claim 8, wherein the amount of parallax is horizontal parallax.   The image processing apparatus according to claim 12, wherein the horizontal parallax is given to a component having a high lightness of the image and small to a component having a low lightness.   The image processing apparatus according to claim 12, wherein the horizontal parallax is large for a component with high saturation of an image and small for a component with low saturation.   An image processing program for causing an arithmetic device to execute the image processing method according to claim 1.