JP6217486B2 - Stereo image generating apparatus, stereo image generating method, and stereo image generating program - Google Patents

Description

  The present invention relates to a stereoscopic image generation technique, and more particularly to a stereoscopic image generation technique for correcting a pair of images forming a stereoscopic video.

  In recent years, 3D image display devices capable of displaying stereoscopic images (3D images) have begun to spread. Broadcasting using 3D video signals is also performed in television broadcasting, and 3D signal recording apparatuses capable of recording and reproducing 3D video signals are also becoming popular. The content of a 3D video signal for broadcasting or package media (hereinafter referred to as “3D content”) is usually a two-viewpoint video signal captured by a twin-lens camera. According to the two-viewpoint video signal, a realistic 3D image can be expressed.

  In such a stereoscopic image display device, there is a mismatch of focusing on the display surface while adjusting the convergence angle of the eye to the point where the object being watched by the observer has popped out. Induces eye fatigue. Moreover, the depth feeling suitable for the observer varies depending on the distance between the observer and the display surface of the display and the individual differences of the observer.

  With respect to these problems, Patent Document 1 discloses that the parallax between a pair of input images is converted into a parallax with a suitable sense of depth regardless of the distance between the viewer and the display surface or the individual differences of the viewers. A technique for displaying a stereoscopic image by changing it is disclosed.

JP 2011-35712 A

  However, in the technique disclosed in Patent Document 1, various methods have been proposed as methods for adjusting parallax. However, since the selection criterion of these methods is fixedly determined regardless of the parallax of the image itself, video In some cases, the optimum adjustment method is not always selected.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a stereoscopic image generation technique capable of appropriately adjusting parallax and providing a stereoscopic image suitable for an observer.

  In order to solve the above-described problem, a stereoscopic image generation apparatus according to an aspect of the present invention includes a parallax information acquisition unit that acquires parallax information of a pair of images forming a stereoscopic video, and a projection of a stereoscopic video based on the parallax information. A protrusion / retraction degree acquisition unit that acquires the degree of retraction, a parallax adjustment unit that adjusts the parallax information to generate parallax adjustment information, and a plurality of depth models for deepening at least some of the regions in the image. A combination unit that selects at least one and combines the selected at least one depth model and the parallax adjustment information by the parallax adjustment unit at a combination ratio according to the degree of popping out or retracting, and parallax adjustment after combining An image generation unit that generates a new pair of images in which the parallax of the pair of images is changed based on the information.

  Another aspect of the present invention is a stereoscopic image generation method. In this method, a step of acquiring parallax information of a pair of images forming a stereoscopic video, a step of acquiring a degree of popping out or retracting a stereoscopic video based on the parallax information, and adjusting the parallax information to obtain parallax adjustment information. Selecting at least one of a plurality of depth models for increasing the depth of at least some of the regions in the image, and popping out the selected at least one depth model and the parallax adjustment information Alternatively, the method includes a step of combining at a combining ratio corresponding to the degree of retraction, and a step of generating a new pair of images in which the parallax of the pair of images is changed based on the combined parallax adjustment information.

  It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, etc. are also effective as an aspect of the present invention.

  According to the present invention, it is possible to provide a stereoscopic image suitable for an observer.

1 is a block diagram of a stereoscopic image generation device according to Embodiment 1 of the present invention. It is a block diagram of the image signal conversion part of FIG. It is a block diagram of the histogram generation part of FIG. It is a block diagram of the parallax adjustment part of FIG. It is a figure explaining an example of pop-out frequency F_sum which the pop-out frequency calculation part of FIG. 4 calculates. 6A is a diagram illustrating an example of partial parallax adjustment performed by the partial parallax adjustment unit illustrated in FIG. 4, and FIG. 6B illustrates an example of total parallax adjustment performed by the total parallax adjustment unit illustrated in FIG. 4. FIG. FIGS. 7A and 7B are diagrams illustrating the synthesis of the partial adjustment parallax value and the overall parallax adjustment value by the synthesis unit in FIG. 4. It is a figure which shows an example of the stereo image pair input into the stereo image production | generation apparatus of FIG. FIG. 9A to FIG. 9D are diagrams illustrating shift image generation processing by the stereoscopic image generation apparatus according to the first embodiment. 6 is a configuration diagram of a parallax adjustment unit according to Embodiment 2. FIG. Fig.11 (a)-FIG.11 (c) are figures explaining an example of three types of basic depth models. FIG. 12A and FIG. 12B are diagrams illustrating the synthesis of the partial adjustment parallax value and the depth value of at least one type of basic depth model by the synthesis unit according to the second embodiment. FIG. 13A to FIG. 13D are diagrams illustrating shift image generation processing by the stereoscopic image generation apparatus according to the second embodiment.

[Embodiment 1]
FIG. 1 is a block diagram of a stereoscopic image generating apparatus 1 according to Embodiment 1 of the present invention. A 3D signal recording device 2 and a stereo display device 3 are connected to the stereoscopic image generating device 1. The 3D signal recording device 2 records and holds 3D video. The stereoscopic image generation device 1 reads 3D video from the 3D signal recording device 2. The stereo display device 3 receives the left eye image and the right eye image from the stereoscopic image generation device 1 and displays them as a stereoscopic image.

  The stereoscopic image generation apparatus 1 includes an input signal acquisition unit 11, a 3D signal decoding unit 12, a 3D signal parallax detection unit 13, and an image signal conversion unit 14.

  The input signal acquisition unit 11 acquires a 3D video signal and a 3D format identification signal output from the 3D signal recording device 2 that is a 3D video signal transmission source. The input signal acquisition unit 11 includes, for example, an input terminal and an input interface. The 3D format identification signal is a signal indicating whether the 3D content format is a side-by-side format, a top-and-bottom format, a line-by-line format, a field sequential format, or the like.

  As an example, transmission / reception of 3D signals is standardized by HDMI (registered trademark) 1.4a, and in this embodiment, the input signal acquisition unit 11 performs 3D video based on the standard of HDMI (registered trademark) 1.4a. The signal and the 3D format identification signal are acquired from the 3D signal recording device 2. The 3D video signal and 3D format identification signal acquired by the input signal acquisition unit 11 are supplied to the 3D signal decoding unit 12.

  The 3D signal decoding unit 12 separates the 3D video signal into a left eye image signal and a right eye image signal based on the 3D format identification signal. The left eye image signal and the right eye image signal are supplied to the 3D signal parallax detection unit 13 and the image signal conversion unit 14. The left eye image signal and the right eye image signal form a stereo image pair.

  The 3D signal parallax detection unit 13 uses, for example, a stereo matching technique, which is a representative example of the parallax calculation technique, and uses the left eye image signal (or right eye image signal) of the stereo image pair as a reference between the left eye image signal and the right eye image signal. The 3D signal parallax value DPT is detected for each pixel, and parallax information of the stereo image pair is acquired. The stereo matching method is a pixel block in one captured image (reference image), that is, a pixel group for each small region constituting a part of the reference image, and the correlation destination is the other captured image (comparison image). In this method, the parallax value is obtained for each pixel block in the captured image (reference image) (see, for example, Japanese Patent Application Laid-Open No. 2003-16427). When the value of the 3D signal parallax value DPT is positive, the “protruding direction” is set, and when the value is negative, the “retracting direction” is set.

  Here, the parallax is detected from the left-eye image signal and the right-eye image signal. However, when the stereoscopic video signal includes parallax information, the parallax information may be acquired from the signal. Various methods can be used as a method of acquiring the parallax information, and the 3D signal parallax detection unit 13 is an example of the parallax information acquisition unit.

  Here, the 3D signal parallax value DPT is based on the left parallax value DPT_L indicating the amount of shift of the corresponding right-eye image pixel with respect to the pixel of the left-eye image, and the corresponding left-eye image pixel with respect to the pixel of the right-eye image. It includes a right parallax value DPT_R indicating the amount of pixel displacement.

  The 3D signal parallax value DPT (at least one of DPT_L and DPT_R) is supplied to the image signal converter 14.

  The image signal conversion unit 14 adjusts the parallax by shifting the pixel of at least one of the left-eye image signal and the right-eye image signal that are the stereo image pair output from the 3D signal decoding unit 12 and outputs the result. The image signal conversion unit 14 shifts pixels using, for example, a pseudo-stereoscopic image generation device described in Japanese Patent Application Laid-Open No. 2005-151534. The pseudo-stereoscopic image generation apparatus described in the publication performs so-called “2D3D conversion” and generates a pseudo-stereoscopic image from a 2D image.

  FIG. 2 is a configuration diagram of the image signal conversion unit 14. The image signal conversion unit 14 includes a histogram generation unit 141L, a histogram generation unit 141R, a left eye parallax adjustment unit 143L, a right eye parallax adjustment unit 143R, a left eye shift image generation unit 145L, and a right eye shift image generation unit 145R.

  The histogram generation unit 141L, the left eye parallax adjustment unit 143L, and the left eye shift image generation unit 145L convert the left eye image signal by shifting the pixels of the left eye image signal based on the left eye image signal and the left parallax value DPT_L. The histogram generation unit 141L receives the left parallax value DPT_L, generates a histogram indicating the distribution of the parallax values, and supplies information regarding the pop-up frequency to the left-eye parallax adjustment unit 143L. The left-eye parallax adjustment unit 143L receives the left parallax value DPT_L for each pixel, adjusts the left parallax value DPT_L for each pixel based on the information regarding the pop-out frequency given from the histogram generation unit 141L, and sets the left parallax adjustment value DPT_L_adj. This is supplied to the left eye shift image generation unit 145L. The left eye shift image generation unit 145L receives the left eye image signal, shifts the left eye image based on the left parallax adjustment value DPT_L_adj given from the left eye parallax adjustment unit 143L, and outputs the shifted left eye image signal.

  The histogram generation unit 141R, the right eye parallax adjustment unit 143R, and the right eye shift image generation unit 145R convert the right eye image signal by shifting the pixels of the right eye image signal based on the right eye image signal and the right parallax value DPT_R. The histogram generation unit 141R receives the right parallax value DPT_R, generates a histogram indicating the distribution of the parallax values, and supplies information regarding the pop-up frequency to the right-eye parallax adjustment unit 143R. The right eye parallax adjustment unit 143R receives the right parallax value DPT_R for each pixel, adjusts the right parallax value DPT_R for each pixel based on the information regarding the pop-out frequency given from the histogram generation unit 141R, and sets the right parallax adjustment value DPT_R_adj. This is supplied to the right eye shift image generation unit 145R. The right eye shift image generation unit 145R receives the right eye image signal, shifts the right eye image based on the right parallax adjustment value DPT_R_adj given from the right eye parallax adjustment unit 143R, and outputs the shifted right eye image signal.

  The left eye image signal conversion process and the right eye image signal conversion process are the same except for the difference between the image signal and the parallax value for the left eye or the right eye. The processing will be described without distinguishing the right eye, and reference will be made to the reference numerals without distinguishing between L and R.

  FIG. 3 is a configuration diagram of the histogram generation unit 141. The histogram generation unit 141 includes a selector group 1411, a cumulative counter group 1412, and a holding register group 1413.

  The input 3D signal parallax value DPT of each pixel is supplied to the selector group 1411. This selector group 1411 is composed of a plurality of selectors, and divides the possible range of the 3D signal parallax value DPT into a plurality of level regions, and the 3D signal parallax value DPT enters any level region for each pixel. Determine whether.

  In the present embodiment, as an example, assuming that the range of the 3D signal parallax value DPT is −128 to 127, 128 is added to this to obtain 256 levels of 0 to 255, and the level of the 3D signal parallax value DPT is obtained. The obtained range is divided into 16 level regions. Corresponding to the 16 level regions, the selector group 1411 is provided with 16 selectors. The first selector corresponds to the level region in the range of 0 ≦ DPT <16, the second selector corresponds to the level region in the range of 16 ≦ DPT <32, and the third selector has 32 Corresponding to the level region in the range of ≦ DPT <48,..., The sixteenth selector corresponds to the level region in the range of 240 ≦ DPT <256. In the selector group 1411, the selector corresponding to the level region to which the input 3D signal parallax value DPT belongs outputs “1”, and the other selectors output “0”.

  The cumulative counter group 1412 includes 16 counters corresponding to the 16 selectors. Each counter counts the output of the corresponding selector. The i-th counter counts the number of times “1” output from the i-th selector. The holding register group 1413 includes 16 registers corresponding to the 16 counters. Each register holds the output of the corresponding counter. The value of each register is output to the parallax adjustment unit 143.

  For example, if the level of the 3D signal parallax value DPT is 40 pixels, “1” is output from the third selector, and “0” is output from the other selectors. The cumulative counter group 1412 includes 16 counters, counts up the signals output from the 16 selectors, and increments the count value by one. This count value indicates the total number of pixels included in the level region divided by the selector group 1411. The holding register group 1413 includes 16 registers, and holds the count value output from each of the 16 counters. As described above, the histogram generation unit 141 counts the number of pixels included in the divided level region, generates histogram data, and supplies the histogram data to the parallax adjustment unit 143.

  FIG. 4 is a configuration diagram of the parallax adjustment unit 143. The parallax adjustment unit 143 includes a pop-up frequency calculation unit 1431, a partial parallax adjustment unit 1432, an overall parallax adjustment unit 1433, and a synthesis unit 1434.

  The pop-out frequency calculation unit 1431 receives the histogram data from the histogram generation unit 141, calculates the pop-out frequency F_sum, and supplies it to the synthesis unit 1434.

  FIG. 5 is a diagram illustrating an example of the pop-out frequency F_sum calculated by the pop-out frequency calculation unit 1431.

  The graph of FIG. 5 shows a histogram of the parallax value DPT input to the pop-out frequency calculation unit 1431. The horizontal axis represents a level region L obtained by dividing the parallax value DPT into 16 steps from the retracting side to the protruding side. The vertical axis represents the number (frequency) Hist [L] of pixels in each level region. Hist [0] to Hist [7] indicate the number of pixels in the retracting direction from the display surface, and Hist [8] to Hist [15] indicate the number of pixels in the protruding direction from the display surface. Hist [0] represents the number of pixels at the back, and Hist [15] represents the number of pixels at the front.

As an example, the pop-out frequency calculation unit 1431 calculates the pop-out frequency F_sum using Expression (1).
F_sum = Hist [12] + Hist [13] + Hist [14] + Hist [15]
... Formula (1)

  Note that the method of calculating the pop-out frequency F_sum is not limited to the expression (1), and any index can be used as long as it represents the degree of pop-out of the pixels in the image. As in Expression (1), the total number of pixels in the image whose degree of pop-up is greater than or equal to the threshold value may be used as an index, or the number of pixels belonging to a level region that shows a peak in the histogram may be used as an index. In this embodiment, a histogram of parallax values of an image is obtained and the pop-out frequency is calculated from the histogram. However, the pop-out degree may be obtained without using the histogram. For example, the number of pixels having a parallax value DPT equal to or greater than a certain threshold value may be counted, and the value may be used as an index representing the degree of popping out.

  The pop-up frequency F_sum calculated in this way is supplied to the synthesis unit 1434.

  Refer to FIG. 4 again. The partial parallax adjustment unit 1432 receives the 3D signal parallax value DPT, obtains the partial parallax adjustment value DPT_lim, and supplies the partial parallax adjustment value DPT_lim to the synthesis unit 1434.

  FIG. 6A is a diagram illustrating an example of partial parallax adjustment by the partial parallax adjustment unit 1432.

  The graph of FIG. 6A shows the relationship between the parallax value DPT and the adjusted parallax value DPT_lim. The broken line shown in FIG. 6A is the parallax value when the parallax adjustment is not performed, and the solid line is the parallax value after the adjustment when the parallax adjustment is performed.

The partial parallax adjustment unit 1432 calculates a partial parallax adjustment value based on the partial parallax adjustment characteristics illustrated in FIG. This calculation is expressed by equation (2).
DPT_lim (x, y) = LIM (DPT (x, y) ≧ LIM)
DPT_lim (x, y) = DPT (x, y) (DPT (x, y) <LIM)
... Formula (2)
Here, DPT (x, y) represents a parallax value of a pixel having horizontal coordinates x and vertical coordinates y, and DPT_lim (x, y) represents a partial parallax adjustment value of a pixel having horizontal coordinates x and vertical coordinates y.

  In the case of the partial parallax adjustment characteristic shown in FIG. 6A, the parallax value is limited by the threshold value LIM for pixels whose parallax value is larger than the threshold value LIM among the pixels felt in front of the display surface.

  Refer to FIG. 4 again. The overall parallax adjustment unit 1433 receives the 3D signal parallax value DPT, obtains the overall parallax adjustment value DPT_ofst, and supplies it to the synthesis unit 1434.

  FIG. 6B is a diagram illustrating an example of the overall parallax adjustment by the overall parallax adjustment unit 1433.

  The graph in FIG. 6B shows the relationship between the parallax value DPT and the adjusted parallax value DPT_ofst. The broken line shown in FIG. 6B is the parallax value when the parallax adjustment is not performed, and the solid line is the parallax value after the adjustment when the parallax adjustment is performed.

The overall parallax adjustment unit 1433 calculates an overall parallax adjustment value based on the overall parallax adjustment characteristics illustrated in FIG. This calculation is expressed by equation (3).
DPT_ofst (x, y) = DPT (x, y) −OFST
... Formula (3)
Here, DPT (x, y) represents the parallax value of the pixel with the horizontal coordinate x and the vertical coordinate y, and DPT_ofst (x, y) represents the overall parallax adjustment value of the pixel with the horizontal coordinate x and the vertical coordinate y.

  In the case of the overall parallax adjustment characteristic in FIG. 6B, the parallax value is reduced by the offset amount OFST as a whole. When the parallax is reduced, both the pixels felt in front of the display surface and the pixels felt in the back of the display surface can be felt in the back. In other words, it feels like the entire screen is retracted.

  The partial adjustment parallax value DPT_lim and the total parallax adjustment value DPT_ofst calculated in this way are supplied to the synthesis unit 1434.

  Refer to FIG. 4 again. The combining unit 1434 determines a ratio for combining the partial parallax adjustment value DPT_lim and the total parallax adjustment value DPT_ofst based on the pop-up frequency F_sum, and combines the partial parallax adjustment value DPT_lim and the total parallax adjustment value DPT_ofst with the determined combining ratio. Then, the parallax adjustment value DPT_adj is calculated. The combined parallax adjustment value DPT_adj is supplied to the shift image generation unit 145.

  FIGS. 7A and 7B are diagrams illustrating the synthesis of the partial adjustment parallax value DPT_lim and the total parallax adjustment value DPT_ofst by the synthesis unit 1434. FIG.

  The horizontal axis in FIG. 7A represents the pop-out frequency F_sum, and the vertical axis represents the partial parallax adjustment gain Glim that is multiplied by the partial parallax adjustment value DPT_lim. The partial parallax adjustment gain is reduced as the pop-out frequency increases.

  In FIG. 7B, the horizontal axis represents the pop-up frequency F_sum, and the vertical axis represents the overall parallax adjustment gain Gofst that is multiplied by the overall parallax adjustment value DPT_ofst. The overall parallax adjustment gain is increased as the pop-out frequency increases.

Here, the relationship of Expression (4) is established between the partial parallax adjustment gain Glim and the overall parallax adjustment gain Gofst.
Glim + Gofst = 1.0 Formula (4)

  The gain characteristics shown in FIGS. 7A and 7B are merely examples, and the gain characteristics are not limited to this. In general, the greater the degree of popping, the greater the degree of suppression of parallax by overall parallax adjustment, while the ratio of total parallax adjustment and partial parallax adjustment is determined so that the degree of suppression of parallax by partial parallax adjustment decreases. Is preferred.

When the pop-up frequency F_sum is given, the combining unit 1434 obtains the partial parallax adjustment gain Glim and the total parallax adjustment gain Gofst based on the gain characteristics shown in FIGS. ) And Expression (6), the partial parallax adjustment gain Glim and the total parallax adjustment gain Gofst are respectively multiplied by the partial parallax adjustment threshold LIM and the total parallax adjustment offset value OFST for correction.
LIM ′ = 255− (255−LIM) × Glim (5)
OFST ′ = OFST × Gofst (6)

Further, the synthesis unit 1434 obtains the partial parallax adjustment value DPT_lim ′ (x, y) using the corrected partial parallax adjustment threshold LIM ′ as in Expression (7), and after correction as in Expression (8). The total parallax adjustment value DPT_ofst ′ (x, y) is obtained using the total parallax adjustment offset value OFST ′.
DPT_lim ′ (x, y) = LIM ′ (DPT (x, y) ≧ LIM ′)
DPT_lim ′ (x, y) = DPT (x, y) (DPT (x, y) <LIM ′)
... Formula (7)
DPT_ofst ′ (x, y) = DPT (x, y) −OFST ′
... Formula (8)

Finally, as shown in Expression (9), the synthesis unit 1434 performs the partial parallax adjustment value DPT_lim ′ (x, y) obtained by Expression (7) and the overall parallax adjustment value obtained by Expression (8). DPT_ofst ′ (x, y) is combined to obtain a final combined parallax adjustment value DPT_adj.
DPT_adj (x, y)
= DPT_Lim '(x, y) + DPT_ofst'-DPT (x, y)
= DPT_Lim ′ (x, y) −OFST ′
... Formula (9)

  In this way, the combination ratio of the partial parallax adjustment and the total parallax adjustment is determined based on the pop-out frequency F_sum.

  The final parallax adjustment data DPT_adj is supplied to the shift image generation unit 145.

  The shift image generation unit 145 shifts the pixels of any one of the input stereo image pairs (for example, the left-eye image signal) based on the parallax adjustment value DPT_adj output from the parallax adjustment unit 143, thereby A shifted image signal is generated. If the parallax adjustment value DPT_adj is the same as the original parallax value DPT, the pixel is not shifted. However, for a pixel whose parallax adjustment value DPT_adj is different from the original parallax value DPT, the pixel is shifted by the difference. A method for generating a shift image is described in, for example, Japanese Patent Application Laid-Open No. 2005-151534.

  The effect of the shift image generation process of this embodiment will be described with reference to FIGS. 8 and 9A to 9D.

  FIG. 8 is a diagram illustrating an example of a stereo image pair input to the stereoscopic image generation apparatus 1. FIG. 8 shows a left eye image IL and a right eye image IR. The subject OB1 has a parallax value in the pop-out direction (DPT value 30), the subject OB2 is on the display surface and has no parallax (DPT value 0), and the subject OB3 has a parallax value in the retracting direction (DPT value -5). doing. This image is an example in which the subject OB1 has excessive parallax in the protruding direction and is a burden on the eyes of the observer.

  FIG. 9A to FIG. 9D are diagrams illustrating shift image generation processing by the stereoscopic image generation apparatus 1.

  FIG. 9A is the input right-eye image IR shown in FIG. FIG. 9B is an image obtained by performing partial parallax adjustment on the input right-eye image IR using the partial parallax adjustment characteristics illustrated in FIG. The gray portion in FIG. 9B is an occlusion area generated by shifting the pixel of the protruding subject OB1, and there is no corresponding pixel in the input right-eye image signal IR. In the occlusion area, pixel data is generated by performing pixel interpolation based on the peripheral shift pixel data that can be normally generated, and thus it may be difficult to reproduce the input image signal. In this example, since the occlusion area is large, there is a high risk of causing image quality degradation.

  FIG. 9C is an image in which the overall parallax adjustment is performed on the input right-eye image IR using the overall parallax adjustment characteristic illustrated in FIG. In the overall parallax adjustment, an occlusion area does not occur, but since the image is shifted overall, the depth of all subjects changes in the retracting direction, and not only the subject OB1 that is in front of the display surface but also the display surface or display The objects OB2 and OB3 that are in the back of the surface are also retracted in the back. Also, by shifting the whole to the right, a black area is created at the left end.

  FIG. 9D is a shifted right-eye image generated by combining the partial parallax adjustment characteristics and the overall parallax adjustment characteristics according to the pop-out frequency by the shift image generation processing of the present embodiment. The occlusion area is reduced by suppressing the shift due to the partial parallax adjustment for the subject OB1 having the parallax in the excessive protruding direction, and the whole is not excessively retracted by suppressing the shift due to the overall parallax adjustment for all the subjects OB1 to OB3. Become.

  In the shift image generation processing of the present embodiment, the partial parallax adjustment characteristics and the overall parallax adjustment characteristics are synthesized based on the characteristics of the input image, particularly the histogram analysis. The movement of the subject in the retracting direction due to the parallax adjustment can be suppressed to be small. Accordingly, it is possible to perform parallax adjustment that suppresses excessive pop-up without significant image quality deterioration.

  In the examples of FIGS. 9A to 9D, the case where the right eye image is pixel-shifted based on the right parallax adjustment value has been described. However, in the case of the left-eye image, the pixel is similarly set based on the left parallax adjustment value. Can be shifted. In the present embodiment, the pixel shift is described for each of the left-eye image and the right-eye image so as not to shift the center of the image. However, the pixel shift may be performed for only the left-eye image or only the right-eye image.

  The stereo display device 3 generates and displays a stereoscopic image using the stereo image pair whose parallax is adjusted in this way. A method for generating a stereoscopic image using a stereo image pair is described in, for example, Japanese Patent Application Laid-Open No. 2005-151534.

[Modification of Embodiment 1]
As a modification of the first embodiment, a shift image may be generated based on the retraction frequency instead of the pop-out frequency. In this case, in the histogram of FIG. 5, the retraction frequency F_sum is taken as an example.
F_sum = Hist [0] + Hist [1] + Hist [2] + Hist [3]
The partial parallax adjustment and the total parallax adjustment may be combined based on the retraction frequency F_sum.

However, in this case, the partial parallax adjustment characteristic restricts the parallax value with the threshold value -LIM for pixels whose parallax value is smaller than the negative threshold value -LIM among the pixels that are felt behind the display surface as follows. Become a thing.
DPT_lim (x, y) = − LIM (DPT (x, y) ≦ −LIM)
DPT_lim (x, y) = DPT (x, y) (DPT (x, y)> − LIM)

Also, the overall parallax adjustment characteristic adjusts the parallax value so that the entire screen pops forward by increasing the parallax value by the offset amount OFST as a whole as follows.
DPT_ofst (x, y) = DPT (x, y) + OFST

  As described above, according to the present embodiment, the degree of popping or retracting is detected from the parallax information acquired from the stereo image pair, and the combination of partial parallax adjustment and overall parallax adjustment is performed according to the degree of popping or retracting. Determine the ratio. This makes it possible to limit the amount of projection or withdrawal that is larger than a certain value while suppressing image distortion caused by pixel shift, and to adjust the parallax of stereoscopic images that have excessive projection or withdrawal, ensuring high quality and safety. 3D images can be provided.

  In this embodiment, partial parallax adjustment and overall parallax adjustment are described as examples of a plurality of parallax adjustment methods, but two or more parallax adjustment methods are synthesized using a method other than this as the parallax adjustment method. May be. For example, a parallax adjustment method based on selection of a basic depth model described in the second embodiment described later may be combined.

[Embodiment 2]
The stereoscopic image generation apparatus 1 according to Embodiment 2 of the present invention differs from Embodiment 1 in the configuration and operation of the parallax adjustment unit 143 of the image signal conversion unit 14, and the other configuration and operation are the same. Here, description of the configuration and operation common to Embodiment 1 will be omitted as appropriate, and configuration and operation different from Embodiment 1 will be described.

  FIG. 10 is a configuration diagram of the parallax adjustment unit 143 according to the second embodiment. The parallax adjustment unit 143 includes a pop-up frequency calculation unit 1431, a partial parallax adjustment unit 1432, a plurality of basic depth models 1435a, 1435b, 1435c, and a synthesis unit 1434.

  The configurations and operations of the pop-out frequency calculation unit 1431 and the partial parallax adjustment unit 1432 are the same as those in Embodiment 1, and thus description thereof is omitted.

  The basic depth model indicates depth values for the basic scene structure. The basic depth model approximates an actual three-dimensional scene, which is originally a complicated structure, using a curved surface or a plane. Here, it is assumed that there are three types of basic depth models 1435a, 1435b, and 1435c, but the number of basic depth models is arbitrary.

  Fig.11 (a)-FIG.11 (c) are figures explaining an example of three types of basic depth models 1435a, 1435b, and 1435c.

  The basic depth model 1435a of type 1 shown in FIG. 11A defines the depth value of the entire screen by a three-dimensional structure having a spherical concave surface. The periphery of the screen has no depth with respect to the display surface, and gradually retracts from the periphery of the screen toward the center of the screen, and becomes the farthest distance at the center of the screen.

  A basic depth model 1435b of type 2 shown in FIG. 11B increases the depth value at the top of the screen.

  In the basic depth model 1435c of type 3 shown in FIG. 11C, the upper portion of the screen is approximated as a distant view, and the depth value is reduced toward the lower portion of the lower portion of the screen.

  These three types of basic depth models 1435a, 1435b, and 1435c are described in detail in Japanese Patent Application Laid-Open No. 2005-151534.

  The partial adjustment parallax value DPT_lim calculated by the partial parallax adjustment unit 1432 and the three types of basic depth models 1435a, 1435b, and 1435c are supplied to the synthesis unit 1434.

  Refer to FIG. 10 again. The combining unit 1434 determines a ratio of combining the partial parallax adjustment value DPT_lim and the depth value of at least one type of basic depth model based on the pop-up frequency F_sum, and the partial parallax adjustment value DPT_lim with the determined combining ratio. The parallax adjustment value DPT_adj is calculated by combining the depth values of at least one type of basic depth model. The combined parallax adjustment value DPT_adj is supplied to the shift image generation unit 145.

  FIGS. 12A and 12B are diagrams for explaining the synthesis of the partial adjustment parallax value DPT_lim and the depth value of at least one type of basic depth model by the synthesis unit 1434.

  The horizontal axis in FIG. 12A represents the pop-up frequency F_sum, and the vertical axis represents the partial parallax adjustment gain Glim that is multiplied by the partial parallax adjustment value DPT_lim. The partial parallax adjustment gain is reduced as the pop-out frequency increases.

  FIG. 12B shows a method of selecting and mixing three types of basic depth models according to the pop-up frequency F_sum. As the pop-up frequency increases, the basic depth model of type 1 that increases the depth at the center of the screen is selected. As the pop-out frequency decreases, the basic depth model of type 3 that does not adjust the depth at the bottom of the screen is selected. When the pop-up frequency is intermediate, the type 2 basic depth model is selected, but when the pop-up frequency increases, type 3 is mixed with type 2, and when the pop-out frequency decreases, type 1 is mixed with type 2. A method for synthesizing different types of basic depth models is described in JP-A-2005-151534.

When the pop-up frequency F_sum is given, the synthesizer 1434 obtains the partial parallax adjustment gain Glim based on the gain characteristics shown in FIG. 12A, and obtains the partial parallax adjustment gain Glim as shown in Expression (10). The parallax adjustment threshold LIM is multiplied and corrected.
LIM ′ = 255− (255−LIM) × Glim Expression (10)

Further, the combining unit 1434 obtains the partial parallax adjustment value DPT_lim ′ (x, y) using the corrected partial parallax adjustment threshold LIM ′ as in Expression (11).
DPT_lim ′ (x, y) = LIM ′ (DPT (x, y) ≧ LIM ′)
DPT_lim ′ (x, y) = DPT (x, y) (DPT (x, y) <LIM ′)
... Formula (11)

Finally, as shown in Expression (12), the synthesis unit 1434 has at least one selected according to the partial parallax adjustment value DPT_lim ′ (x, y) obtained by Expression (11) and the pop-up frequency F_sum. The depth value Basic of the basic depth model is synthesized to obtain a final synthesized parallax adjustment value DPT_adj.
DPT_adj (x, y)
= DPT_Lim ′ (x, y) + Basic (12)

  In this way, the combination ratio of the partial parallax adjustment and the basic depth model is determined based on the pop-up frequency F_sum.

  The final parallax adjustment data DPT_adj is supplied to the shift image generation unit 145. The configuration and operation of the shift image generation unit 145 are the same as those in the first embodiment.

  FIG. 13A to FIG. 13D are diagrams illustrating shift image generation processing by the stereoscopic image generation apparatus according to the second embodiment.

  FIG. 13A is the input right-eye image IR described in Embodiment 1 with reference to FIG. FIG. 13B is an image obtained by performing partial parallax adjustment on the input right-eye image IR using the partial parallax adjustment characteristics similar to those in the first embodiment. A gray portion in FIG. 13B is an occlusion area generated by shifting the pixel of the protruding subject OB1, and there is no corresponding pixel in the input right-eye image signal IR. In the occlusion area, pixel data is generated by performing pixel interpolation based on the peripheral shift pixel data that can be normally generated, and thus it may be difficult to reproduce the input image signal. In this example, since the occlusion area is large, there is a high risk of causing image quality degradation.

  FIG. 13C shows a basic depth model selected and mixed according to the pop-up frequency. Here, by mixing type 1 and type 2 basic depth models, the pixels are gently moved to the back throughout the screen.

  FIG. 13D is a shifted right-eye image generated by combining the partial parallax adjustment characteristics and the basic depth model according to the pop-out frequency by the shift image generation processing of the present embodiment. The occlusion area is reduced by suppressing the shift due to the partial parallax adjustment of the subject OB1 having the parallax in the excessive protruding direction, and all the subjects OB1 to OB3 are gently moved to the back by using the mixed basic depth model. So the whole will not retract too much.

  In the shift image generation processing of the present embodiment, the partial parallax adjustment characteristics and the basic depth model are synthesized based on the characteristics of the input image, particularly the histogram analysis, so that the occlusion area generated by the partial parallax adjustment is suppressed and mixed. The basic depth model can make the movement of the subject gentle for the observer. Accordingly, it is possible to perform parallax adjustment that suppresses excessive pop-up without significant image quality deterioration.

[Modification of Embodiment 2]
In the second embodiment, similarly to the modification of the first embodiment, a shift image may be generated based on the retraction frequency instead of the pop-out frequency. In this case, as the basic depth model, a curved surface that defines the depth value so that at least a part of the region in the image pops out is used. For example, a plurality of basic depth models are used such as a model in which the center of the screen pops out by a three-dimensional structure having a spherical convex surface, and a model in which the top or bottom of the screen pops out. At least one basic depth model is selected according to the retraction frequency, and the parallax adjustment value is obtained by combining the partial parallax adjustment and the depth value of at least one basic depth model based on the retraction frequency, and the combined parallax adjustment value A shift image is generated based on the above. Thereby, it is possible to generate a stereoscopic image in which excessive retraction is suppressed.

  The processing described in the above embodiments can be realized as a transmission, storage, and reception device using hardware, as well as stored in a ROM (Read Only Memory), a flash memory, or the like. It can also be realized by existing firmware or software such as a computer. The firmware program and software program can be recorded on a computer-readable recording medium, provided from a server through a wired or wireless network, or provided as a data broadcast of terrestrial or satellite digital broadcasting Is also possible.

  The present invention has been described based on the embodiments. The embodiments are exemplifications, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are within the scope of the present invention. .

  DESCRIPTION OF SYMBOLS 1 Stereo image production | generation apparatus, 2 3D signal recording device, 3 Stereo display apparatus, 11 Input signal acquisition part, 12 3D signal decoding part, 13 3D signal parallax detection part, 14 Image signal conversion part, 141 Histogram generation part, 143 Parallax adjustment Unit, 145 shift image generation unit, 1411 selector group, 1412 cumulative counter group, 1413 holding register group, 1431 pop-up frequency calculation unit, 1432 partial parallax adjustment unit, 1433 overall parallax adjustment unit, 1434 synthesis unit, 1435 basic depth model.

Claims (3)

A parallax information acquisition unit that acquires parallax information of a pair of images forming a stereoscopic video;
A popping / retracting degree acquisition unit that acquires a popping or retracting degree of a stereoscopic image based on the parallax information;
A parallax adjustment unit that adjusts the parallax information to generate parallax adjustment information;
Selecting at least one of a plurality of depth models for increasing the depth of at least some of the regions in the image, and calculating the depth value of the selected at least one depth model and the parallax adjustment information by the parallax adjustment unit; A synthesizing unit that synthesizes at a synthesis ratio according to the degree of protrusion or retraction;
An image generation unit that generates a new pair of images in which the parallax of the pair of images is changed based on the parallax adjustment information after combining ,
The parallax adjustment unit is a partial parallax adjustment unit that suppresses parallax of pixels having parallax that deviates from a threshold;
As the degree of popping out or retracting is larger, the synthesizing unit selects a depth model that makes the depth deeper from among the plurality of depth models, while reducing the degree of suppressing parallax by the partial parallax adjusting unit, wherein the plurality of stereoscopic image generation apparatus characterized that you determine the synthesis ratio for synthesizing the parallax adjustment information according to the depth value and the partial parallax adjustment of the at least one depth model of depth model. Obtaining parallax information of a pair of images forming a stereoscopic video;
Obtaining a degree of projection or withdrawal of a stereoscopic image based on the parallax information;
Adjusting the parallax information to generate parallax adjustment information;
Selecting at least one of a plurality of depth models for increasing the depth of at least a part of the region in the image, and projecting or retracting the depth value of the selected at least one depth model and the parallax adjustment information Synthesizing at a synthesis ratio according to the degree;
Generating a new pair of images in which the parallax of the pair of images is changed based on the parallax adjustment information after combining ,
The step of generating the parallax adjustment information is a partial parallax adjustment step for suppressing parallax of pixels having parallax that deviates from a threshold value,
In the synthesizing step, as the degree of popping out or retracting is larger, a depth model that makes the depth deeper is selected from among the plurality of depth models, while the degree of suppressing parallax by the partial parallax adjustment step is reduced. , the stereoscopic image generation method comprising that you determine the synthesis ratio for synthesizing the parallax adjustment information by the partial parallax adjustment step and the depth value of the at least one depth model of the plurality of depth model. Obtaining parallax information of a pair of images forming a stereoscopic video;
Obtaining a degree of projection or withdrawal of a stereoscopic image based on the parallax information;
Adjusting the parallax information to generate parallax adjustment information;
Selecting at least one of a plurality of depth models for increasing the depth of at least a part of the region in the image, and projecting or retracting the depth value of the selected at least one depth model and the parallax adjustment information Synthesizing at a synthesis ratio according to the degree;
Generating a new pair of images in which the parallax of the pair of images is changed based on the parallax adjustment information after synthesis ,
The step of generating the parallax adjustment information is a partial parallax adjustment step for suppressing parallax of pixels having parallax that deviates from a threshold value,
In the synthesizing step, as the degree of popping out or retracting is larger, a depth model that makes the depth deeper is selected from among the plurality of depth models, while the degree of suppressing parallax by the partial parallax adjustment step is reduced. It said plurality of at least one three-dimensional image generating program which is characterized that you determine the synthesis ratio for synthesizing the parallax adjustment information by the partial parallax adjustment step and the depth value of the depth model of depth model.