JP3089306B2 - Stereoscopic imaging and display device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a three-dimensional image pickup and display device for taking and displaying a three-dimensional image.

[0002]

2. Description of the Related Art As a conventional three-dimensional image pickup apparatus, for example, there is one as shown in FIG. This is originally two cameras fixed on two heads, respectively.
The two cameras correspond to the left and right eyes of a human. Each camera is fixed to the rotation controllers 4a and 4b, and is rotatable in the horizontal direction. The camera interval control unit 5 can change the interval between the two cameras, and the entire camera is fixed to a tripod 6. Here, a plan view of this conventional stereoscopic image pickup apparatus is shown in FIG.
(A) and (b).

FIG. 22A shows a case of parallel photographing in which the optical axes of the camera are parallel, and FIG. 22B shows a case of congestion photographing in which the optical axes of the camera are inclined so that the two optical axes intersect. It is. By the movement of the rotation control units 4a and 4b, the two cameras are rotated to correspond to parallel shooting and convergence shooting. Also,
The camera interval can be changed by changing Wc. Now, in the imaging of a stereoscopic image, the distance dx to the convergence point and the camera interval Wc greatly affect the image quality of the stereoscopic image. The camera interval Wc affects the size of the displayed subject and the size of the reproduced horizontal parallax. The distance dx to the convergence point is greatly related to the magnitude of the horizontal parallax (affecting the binocular fusion range of the observer) and the distortion of the captured image. In conventional three-dimensional image capturing, the distance dx to the convergence point and the camera interval Wc are set based on experience of the photographer so that the photographed image can be seen clearly by the observer. Here, “to make the image look beautiful” mainly means that the horizontal parallax of the stereoscopic image falls within the fusion range of the observer when the observer views the image.

Next, the fusion range will be described in more detail. FIG. 23 is a schematic diagram of a conventional stereoscopic image display device for reproducing a stereoscopic image signal recorded by a stereoscopic image pickup device. In this method, two types of left and right images picked up by a stereoscopic image pickup device are reproduced by projectors 9 and 10, polarized by polarizing plates 11 having polarization directions perpendicular to each other, and formed on a screen 12. The observer wears polarized glasses 13,
The image for the right eye is projected only to the right eye of the observer, and the image for the left eye is projected to the left eye of the observer.

FIG. 24 is a plan view of a screen portion of the stereoscopic image display device. Assuming that the point images obtained by the stereoscopic image pickup device are IR and IL for the right eye and the left eye, respectively, the observer at the binocular distance We perceives that the subject virtually exists at the intersection C1 of the line of sight. In this way, the observer can perceive the depth of the displayed image using binocular stereovision. Also, as shown in FIG. 25, CRTs 21 and 2 provided with two orthogonal polarizing plates 24 without using a screen.
There is also a conventional stereoscopic image display method in which the two images are synthesized by the half mirror 23 and the corresponding images are observed with the glasses 25 provided with a polarizing plate, but the principle by which the observer perceives the depth is All are the same.

Next, the depth perception will be described in more detail with reference to FIG. This is a diagram for explaining the formation of the depth perception of the observer by the binocular parallax.
P is the subject (point), CL is the rotation center of the observer's left eye 26, CR is the rotation center of the observer's right eye 27, FL, F
R is the fovea of the observer's left eye and right eye (the highest resolution part on the retina, and when a human gazes at the object, an image of the object is formed in the fovea). Assume that the observer gazes at point P. At this time, the left and right eyeballs are oriented so as to form an image of the point P in the fovea FL and FR. At this time, the image of the point A closer to the gazing point P is formed on AL and AR on the retina of the left and right eyes.
Similarly, the image of the point B farther than the gazing point P is formed on BL and BR on the retinas of the left and right eyes. At this time, the image of point A of each eye and the point B
Are different from each other in distance ALBL and ARBR. This difference in distance corresponds to the distance in the depth direction between point A and point B. In other words, in other words, the binocular disparity PBA between the point A and the point B is represented by δBA = γB−γA.
Corresponding to the distance in the depth direction between the point B and the point B, the human calculates this difference in the brain and perceives the depth position of the subject. That is, LAB
To perceive. Also, if humans are far away from the subject,
The perceptual sensitivity of the absolute distances LA and LB of the point A or the point B is low, and the perceptual sensitivity of the relative distance LAB is high.

[0007]

However, in the above configuration, an actual image is displayed on the display screen 12 (FIG. 2).
4) The observer perceives that the subject is located at the position C1 despite the image being formed on the top. That is, the focus position of the eyes is different between an image that looks real and an image that looks like this. In other words, the convergence of the observer (closed eyes, that is, turning both eyes to the close position C1) and accommodation (focusing of the eyes)
Is in an inconsistent state. If the amount of contradiction becomes large, humans cannot perceive the left and right images as one, and the perception of a double image will result. Alternatively, even if the image does not become a double image, a three-dimensional image having a very uncomfortable feeling is obtained, and the viewer may feel uncomfortable.

More specifically, in FIG.
When the observer gazes at the point P, the range of the size of the binocular disparity δ at which a human can calculate the depth is approximately ± 0.5 °, and when the depth distance is further increased (the binocular disparity δ
When the absolute value of the distance increases, the sense of distance in the depth direction decreases. When the absolute value of the distance further increases, the object is observed twice, and the depth becomes difficult to understand and becomes very difficult to observe. In FIG. 20, when points A, B, and P are displayed on the three-dimensional image display device in FIG. 25, and the position of the reproduced stereoscopic image is away from the position 28 on the CRT surface displaying the image, The deviation between the focus adjustment position of the eye lens (CRT surface) and the depth position of the displayed image increases. There is a permissible range for the deviation between the position of the CRT surface and the stereoscopic image presentation position. If the deviation exceeds this range, the intended subject cannot be imaged in the fovea of both eyes. Perceive multiple images.

In view of the above, it can be understood that the depth perception range of the human being in the binocular stereoscopic view has only a certain limited range centered on the gazing point P. In order to perceive a wider three-dimensional world, the human also moves the gazing point P in the depth direction, and at each gazing point position, 3
Perceive a dimensional structure and synthesize it. However, in the conventional three-dimensional image display device, the entire space of the subject cannot be observed because there is an allowable range of a shift between a focus adjustment position (CRT surface) of an eye lens and a depth position of a display image. Further, since there is a shift between the focus adjustment position of the eye lens (CRT surface) and the depth position of the display image,
It is difficult to quickly move the gazing point, and these causes overlap, and the conventional three-dimensional image display apparatus is tired and hard to see.

Further, in order to improve such a drawback, in a conventional three-dimensional image pickup apparatus, a photographer measures a photographing distance at a photographing site so that the photographer does not become in such a state, and displays the measured distance. Assuming the size of the image display device and the viewing distance of the audience, Wc and dx in FIG. 22 were determined.
The calculation relies on rough shooting distances and the experience of the photographer, and is not accurate or focuses on one subject, and camera parameters are set so that this can be easily and stereoscopically viewed with two eyes. Further, when the depth to be reproduced changes with time, such as when the subject moves in the depth direction, Wc and dx cannot be optimally determined.

An object of the present invention is to provide a three-dimensional image pickup and display device capable of obtaining a three-dimensional image which is easy to see and is less tiring when an observer views the object, in consideration of the conventional problems as described above. Things.

[0012]

SUMMARY OF THE INVENTION The present invention provides an image pickup section for picking up an object from a plurality of viewpoints, a convergence angle moving mechanism for changing the convergence angle of the image pickup section, and extracting image data from each image pickup section. Signal processing unit, a parallax calculating unit that calculates the parallax of an image using the output of the signal processing unit, and a minimum value (parallax of the closest subject) among the outputs of the parallax calculating unit.
And a depth display position calculation for calculating a depth position of the closest object reproduced when an image captured using the output of the parallax processing unit is displayed on the stereoscopic image display device. Unit, using the output of the depth display position calculation unit, a fusion range confirmation unit to determine whether or not this is within the fusion allowable range of the observer viewing the playback image, using the output of the fusion range confirmation unit, This is a stereoscopic image capturing apparatus including a convergence angle control unit that controls a convergence angle moving mechanism unit so that a subject at the closest point is within a fusion allowable range of an observer.

Further, the present invention provides a parallax calculating unit for calculating binocular parallax or a three-dimensional position of a subject using images obtained by capturing the subject from at least two directions, and a depth of the subject based on an output of the parallax calculating unit. a gazing point calculator for calculating a position, a depth position indicated by the output of the gazing point calculator 3
A stereoscopic image display apparatus comprising: a parallax control unit that translates the image in parallel so as to reproduce the image at a specified depth distance from the surface of the three-dimensional stereoscopic image display unit.

Further, the present invention is a three-dimensional image pickup and display device comprising the three-dimensional image pickup device according to any one of claims 1 to 4 and the three-dimensional image display device according to any one of claims 5 to 8.

[0015]

According to the present invention, the binocular parallax of a picked-up image is detected, and the depth position of the nearest object to be observed by the observer is calculated from the parallax, which is within the allowable range of fusion of the observer who views the reproduced image. The convergence angle moving mechanism is controlled as described above.

Further, according to the present invention, an optimum fixation point at which the observer can perceive the depth world of the subject in the widest range is calculated from the binocular parallax of the image, and this is specified from the surface of the stereoscopic image display unit or from the surface. Control to reproduce the distance.

Further, according to the present invention, a three-dimensional image is displayed on a three-dimensional image display device according to any one of the fifth to eighth aspects by using a captured image obtained by the three-dimensional image capturing device according to any one of the first to fourth aspects. .

[0018]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the drawings showing embodiments thereof.

FIG. 1 is a block diagram of a stereoscopic image pickup apparatus according to a first embodiment of the present invention. In FIG.
Denotes a lens, 3a and 3b denote a camera body, 4a and 4b denote a rotation control unit which is a convergence angle moving mechanism unit, and 5 denotes a camera interval control unit, which are the same as those in the prior art. 14a, 1
4b is a signal processing unit for converting a signal from the camera body into a luminance signal, 15 is a parallax calculating unit for calculating horizontal parallax from left and right images, 16 is a parallax processing unit for detecting minimum parallax, 17
Is a depth display position calculation unit that calculates the spatial position of the subject closest to the observer, 18 is a fusion range confirmation unit that checks whether the spatial position of the subject closest to the observer is within the fusion range of the observer, 19 Is the rotation control unit 4 such that the spatial position of the subject closest to the observer is within the fusion range of the observer.
A convergence angle control unit that controls a and 4b. Here, the lenses 1 and 2 and the camera bodies 3a and 3b constitute an imaging unit.

Before describing the operation of the thus configured stereoscopic image pickup apparatus of the present embodiment, model theory of an image pickup system and a display system and characteristics relating to a binocular fusion range of an observer will be described. I do.

First, a model of an imaging system and a display system will be described.
This will be described with reference to FIG. FIG. 2A shows a model of the imaging system. The photographing camera is symmetrical, and the convergence point F (0, dx,
The optical axis is oriented at 0). At this time, the coordinate origin is set at the center of the two cameras, and the direction in which the two cameras are arranged is x-axis, the depth direction is y-axis, and the height direction is z-axis. The left and right lenses 1 and 2 are (−Wc, 0, 0), (Wc,
0, 0), and the focal length of the lens with respect to the imaging planes (positions of the imaging element or the film) 7 and 8 is f. At this time, if dx = ∽, the optical axes of the two cameras become parallel (FIG. 2A, broken line).

FIG. 2B shows a model of a display system (model of a three-dimensional image display device). The coordinates of the display system have the origin at the center of both eyes of the observer, a line connecting the centers of both eyes as the X axis, the depth direction as the Y axis, and the height direction as the Z axis. At this time, the distance from the observer to the stereoscopic image display screen 12 is ds, the left and right eyes are located at (-We, 0, 0) and (We, 0, 0), respectively. Assuming that the object is Pr and the object for the left eye is Pl, the observer considers Pr and (We, 0,
It is perceived that there is a point image at the intersection P of the line segment connecting P1 and (−We, 0, 0). Generally, FIG.
When a stereoscopic image display apparatus using a flat screen as shown in FIG. 2B is used, the position where dx = ∽ and the image at the infinity point (the center position of the captured image) is changed to the left and right images in FIG. When displayed at the positions of NL (-We, ds, 0) and NR (We, ds, 0), the x direction of the imaging space is reproduced without distortion. When the y direction of the imaging space is (Equation 1), the image is reproduced without distortion.

[0023]

(Equation 1)

Here, M is the ratio of the size of the image on the imaging surfaces 7 and 8 to the size of the image on the display screen 12 (provided that the center of the observer is on the Y axis). In this case, when observing the reproduced image at the point at infinity, the lines of sight of both eyes are parallel as shown by the broken lines in FIG. Since the subject is present at a distance of 0 from the point at infinity, the two lines of sight of the observer are parallel or face inward, and do not face outward. This condition is a condition for maximizing the reproduction range of the stereoscopic image, since two lines of sight cannot face outward from parallel as human characteristics.

Among the distortion-free reproduction conditions, dx shown in FIG.
If the condition of ∽ = し な い is not satisfied, that is, if dx has a finite value, the reproduced stereoscopic image is necessarily spatially distorted. Only when it is not within the fusion range of the observer, dx is changed so that the observer can fuse. At this time, dx is set to have as large a value as possible in order to minimize the distortion of the reproduced stereoscopic image. The actual control of dx will be described later. At this time, on the stereoscopic image reproducing side, an image at an infinite point is displayed in FIG.
(B) Play back to NL and NR. For this purpose, FIG.
Using the fact that the horizontal coordinate of the point at infinity on the imaging planes 7 and 8 in (a) is f × Wc / dx, ΔS is set so that the relationship of (Equation 2) is established. The image is translated to the right by ΔS, and the left image is translated to the left by ΔS to display the left and right images.

[0026]

(Equation 2)

Next, the binocular fusion characteristics of a human will be described. The present invention controls the parameter dx of the image pickup device so that when the stereoscopic image is displayed on the stereoscopic image display device, the observer can display the stereoscopic image within a range where the left and right images can be merged. In), it is necessary for the observer to know the conditions for binocular fusion. For example, in FIG.
It is assumed that the observer is gazing at the position of the point P. Here, the actual image is formed on the display screen 12. However, the line of sight of both eyes points to point P. Adduction of both eyes to see a point close to this is called vergence eye movement, P
The point is called a convergence point, and the convergence eye movement is controlled using the focus information of the eye and the retinal projection position information of the point P.

In this case, since the focus information and the retina information are inconsistent, a burden is imposed on the observer. If the contradiction is small, the observer can tolerate it. However, if the contradiction is large, the observer cannot fuse the left and right images into one image, and eventually perceives a double image. Even if the double image is not perceived, the visual system is strained and causes fatigue. There is a result of measuring the allowable range of the focus position of the eye (referred to as accommodation) and the position of the convergence point (referred to as convergence) based on the retinal projection position (Nikkei Electronics vol.444, p205-214 (1
988))). This is shown in FIG. The convergence on the horizontal axis is the distance between the intersection of the line of sight of the two eyes and the observer, and the vertical axis is the distance between the position where the lens of the eye is in focus and the observer (unit: D, diopter, reciprocal of distance) ). 4 rising right
The straight line at 5 degrees is the condition under which the convergence and the accommodation match, the shaded portion is the allowable range within the depth of focus of the eye, and α is the fusion limit line when the image is displayed for 0.5 seconds. In the area between the upper and lower fusion limit lines, the observer can fuse the stereoscopic image, and does not become a double image. Accordingly, dx in FIG. 2 is imaged under the condition that all three-dimensional images are reproduced in the area where the fusion can be performed, and is controlled so as to have as large a value as possible.

First, it is assumed that dx = ∽ (a sufficiently large value) as an initial value. In FIG. 1, a parallax calculation unit 15 calculates a parallax map (three-dimensional map) from left and right eye images output from the signal processing units 14a and 14b. Many calculation methods have been proposed, such as a correlation matching method for calculating the correlation between the luminance patterns of the left and right images and a method for matching the edge information of the left and right images. Here, for example, the correlation between the luminance patterns is calculated. A method using the correlation matching method will be described. The detailed operation of the parallax calculation unit 15 will be described with reference to FIG. In FIG. 4, consider left and right images of size N × M.
Consider a block window of n × n pixels (3 × 3 pixels in the figure) in the left image. The same image as the block window is searched for in the right image using a window of the same size. At this time, the horizontal component Δx of the vector (Δx, Δy) indicating the shift of the left and right block positions is
The binocular parallax of the left and right images at the center coordinates of the block window. If the position of the block window of the reference left image is translated across the entire screen and the position of the corresponding block of the right image (binocular disparity) is obtained in all cases, the disparity map of the entire screen (each image on the screen) Which indicates the depth distance at the place). Here, the shift between the left and right images at the coordinates (x, y) of the image, that is, the binocular disparity (Δx, Δy) is represented by (Equation 3).
Indicated by

[0030]

(Equation 3)

Here,

[0032]

(Equation 4)

Is as follows. However, Σ in (Equation 3) indicates that the coordinates xk and yk are changed within the n × n block window to obtain the sum within the absolute value. N is the number of pixels in the block window. Of the binocular disparities Δx and Δy, Δx directly indicates the depth position. When the left image is the reference, when the value of the binocular disparity is positive, the right image is located on the right side, the left image is located on the left side with respect to the reference image, and indicates the depth side from the depth position of the binocular disparity 0, When the value of the binocular parallax is negative, it indicates that the subject exists on the near side from the depth position of the binocular parallax 0.

Based on the parallax map obtained as described above (calculation of the parallax Δx at each coordinate for the entire screen), the parallax processing unit 16 determines the minimum value of the parallax of the entire image (most to the observer). A parallax of a subject reproduced at a close position is extracted, and this is defined as Δxmin. Next, the depth reproduction position Ypmin of the stereoscopic image in the stereoscopic image display shown in FIG. 5 is calculated by the depth display position calculation unit 17 based on Δxmin, and the calculation formula is represented by (Equation 5).

[0035]

(Equation 5)

Ypmin represents the Y coordinate of the distance from the origin of the display coordinate system to the position of the subject of interest. Here, FIG. 5 shows the display model of FIG. 2B overlaid with the binocular fusion range of the observer of FIG. The shaded area is an area where the observer can integrate both eyes.

Here, F (ds) min is given by the lower curve in FIG. For example, when the viewing distance to the screen is ds = 71 cm, F (ds) min = 29 cm, and the observer can perform binocular fusion within a range from infinity to 29 cm. From FIG. 3, when the viewing distance is 71 cm or more, the observer can perform binocular fusion from infinity to F (ds) min. When the viewing distance is 71 cm or less, the upper curve in FIG. 3 also limits the binocular fusion range, and F (d
s) The distance from max to F (ds) min is the binocular fusion range. Hereinafter, a case where the viewing distance is 71 cm or more will be described. Here, the fusion range confirmation unit 17 determines whether or not Ypmin is at the boundary of the binocular fusion range. The criterion for the evaluation is whether or not (Equation 6) holds.

[0038]

(Equation 6)

Here, ΔF is an arbitrarily small value.
When (Equation 6) holds, the reproduction display position of the stereoscopic image is within the binocular fusion range, and the fusion range confirmation unit 18 sends the current value of dx to the convergence angle control unit 19 without any change. 19 is output. When Ypmin satisfies (Equation 7), the reproduction display position of Ypmin is completely within the binocular fusion range, and the fusion range confirmation unit 18 determines that the current dx is Δ
The value added by dx is set as a new dx and the convergence angle control unit 19
Output to

[0040]

(Equation 7)

However, if the later-described θ (see FIG. 6) approaches 0, the later-described camera optical axis angle θ hardly changes, so that dx need not be updated. Further, as shown in FIG. 5, when Ypmin satisfies (Equation 8), the image reproduced in Ypmin is an image that cannot be binocularly fused for the observer.

[0042]

(Equation 8)

In this case, the current dx is subtracted by Δdx to obtain a new dx. If the value obtained by subtracting ΔF from dx is smaller than dx0 (> 0), dx0
Is a new dx. Here, the calculation of dx0 will be described. The fusion range checking unit 18 calculates the y coordinate of Ypmin as shown in FIG.
Is calculated as dx0 under the condition of F (ds) min. The equation for calculating dx is obtained by (Equation 5) and (Equation 2), eliminating ΔS, setting Yp = F (ds) min, and solving with dx, which is given by (Equation 9).

[0044]

(Equation 9)

Using the dx obtained in this manner, the convergence angle control unit 19 controls the rotation control units 4a and 4b, and the camera bodies 3a and 3b move the optical axis of the camera to the convergence point shown in FIG. Is controlled to face. FIG. 6A shows the definition of the camera control angle θ at this time. When θ = 0, the optical axes of the two cameras are parallel. Here, the update of dx is performed by Δdx, but in the case of the condition of (Expression 8), dx_0 expressed by (Expression 9) is obtained.
May be directly substituted for dx. Also, instead of controlling dx, only when (Equation 8) holds, the camera interval W
Change c to Wc- △ Wc (△ Wc is an arbitrary small value),
This may be repeated until (Equation 6) holds.

As described above, the point Ypmin of the subject closest to the observer is defined as F (d
s) Controlled to be min. At this time, if Ypmin is in the binocular fusion range under the condition that the optical axes of the camera bodies 3a and 3b are parallel, the optical axes of the two cameras are kept parallel.

As described above, according to the present embodiment, the parallax of the imaging screen is calculated, and the optical axis of the stereoscopic imaging camera is controlled so that the minimum value of the parallax is reproduced within the fusion range of the observer. In addition, it is possible to capture an image that allows the observer to fuse the reproduced and displayed stereoscopic image in the widest range.

Also, in FIG. 6B of the first embodiment, θ =
Since the change range of dx is large near 0, dx is dx_∽
Here, θ may be set to 0. At this time, the initial value of dx is dx_∽. Further, the curve of FIG. 6B may be approximated by a straight line. Also, in the condition determination formulas (Equation 6) to (Equation 8), the offset Yp_off is added to Ypmin, and the determination condition is shifted in the Y-axis direction so that the position of Ypmin falls within the fusion range (shaded area). May be adjusted.

FIG. 7 is a block diagram of a stereoscopic image pickup apparatus according to a second embodiment of the present invention. In FIG.
Is a lens, 3a and 3b are camera bodies, 4a and 4b are rotation controllers, 5 is a camera interval controller, 17 is a depth display position calculator that calculates the spatial position of the subject closest to the observer, 19
Is a convergence angle control unit that controls the rotation control units 4a and 4b so that the spatial position of the subject closest to the observer is within the fusion range of the observer, and these are the same as those in the related art.
The difference from the first embodiment of the present invention is that the input of the depth display position calculation unit 17 is directly using the coordinates x0, y0 of the closest subject and the spatial position of the subject closest to the observer. Is that a fusion range determination unit 20 that calculates the value of dx to be within the fusion range of the user is used.

The operation of the thus configured stereoscopic image pickup apparatus of the second embodiment will be described below.

Under the imaging conditions shown in FIG. 2A, the coordinates of the point of the subject which is closest to the camera is represented by N (x
0, y0, z0). The photographer measures this distance before photographing, and inputs x0 and y0 to the depth display position calculation unit 17. The input method is something like a computer keyboard,
Anything that can input numerical data is acceptable. Based on this, the depth display position calculation unit 17 calculates the display position Y of the point N in the three-dimensional image reproduction display coordinate system in FIG.
Calculate coordinates. The calculation formula is shown by (Equation 10).

[0052]

(Equation 10)

Here, ΔS in (Equation 10) is the first value of the present invention.
Is determined by (Equation 2) in the same manner as in the embodiment. Here, first, Ypmin is calculated under the condition of dx = ∽ (the optical axes of the two cameras are parallel). Next, using this Ypmin, the fusion range determination unit 20 determines the actual convergence point dx of the camera. The determination method is that if (Equation 11) is satisfied, d
x = ∽ (sufficiently large value), otherwise, dx_0 shown in Expression 9 is calculated, and this is set as a new dx.

[0054]

[Equation 11]

Based on the dx thus obtained,
The convergence angle control unit 19 calculates the angle θ between the optical axes of the two cameras. θ is the same as that shown in FIG. The rotation controllers 4a and 4b control the optical axis of the camera so that the value of θ determined in this manner is obtained.

As described above, according to the present embodiment, the position of the subject closest to the camera is input, and the convergence point dx that reproduces the position within the fusion range of the observer is determined. By controlling the optical axis of the stereoscopic imaging camera, it is possible to capture an image that allows the viewer to fuse the reproduced and displayed stereoscopic image in the widest range.

Here, in the second embodiment of the present invention, the photographer directly inputs x0 and y0 to the depth display position calculator 17, but this is automatically performed by using a distance measuring device such as an ultrasonic sensor. May be entered. Alternatively, only x0 = a fixed value may be fixed and only the value of y0 may be input.

FIG. 8 is a block diagram of a three-dimensional image pickup apparatus according to a third embodiment of the present invention. The block configuration is the same as in FIG. 8, reference numerals 1 and 2 denote lenses, 3a and 3b.
Is a camera body, 4a and 4b are rotation control units, 5 is a camera interval control unit, 14a and 14b are signal processing units that convert signals from the camera body into luminance signals, and 15 is a parallax that calculates horizontal parallax from left and right images. A calculation unit, 16 is a parallax processing unit that detects the maximum parallax, 17 is a depth display position calculation unit that calculates the spatial position of the subject farthest from the observer, and 18 is a spatial position of the subject farthest from the observer. This is a fusion range confirmation unit that checks whether or not it is within the fusion range. Reference numeral 19 denotes a convergence angle control unit that controls the rotation control units 4a and 4b so that the spatial position of the subject farthest from the observer falls within the fusion range of the observer.

The difference from the first embodiment is that the parallax processing unit 1
6 calculates the maximum value of the parallax, the depth display position calculation unit 17 calculates the spatial position of the subject farthest from the observer, and the fusion range check unit 18 calculates the spatial position of the subject farthest from the observer. The point is to check whether it is within the fusion range of the observer. In the first embodiment, the control is performed for the subject having the minimum parallax value and the closest value.

The operation of the stereoscopic image pickup apparatus according to the third embodiment having the above-described configuration will be described below.

In the first and second embodiments of the present invention, the viewing distance ds of the observer is greater than 71 cm, and the binocular fusion range is ∽.
ＦF (ds) min, and ds was controlled so that F (ds) min was within the range of binocular fusion of the observer. Here, ds <71
cm, as described above, F (ds) max to F (ds) m
In order to control the distance to “in” to be the binocular fusion range and to set F (ds) min to be within the fusion range, the first aspect of the present invention will be described.
In the third embodiment of the present invention, control is performed such that F (ds) max falls within the binocular fusion range.

First, it is assumed that dx = ∽ (a sufficiently large value) as an initial value. Next, similarly to the first embodiment of the present invention, the parallax calculating unit 15 calculates a parallax map using the image signals obtained by the camera bodies 3a and 3b and the signal processing units 14a and 14b. The calculation method is exactly the same as in the first embodiment of the present invention. Based on the parallax map, the parallax processing unit 16 calculates the maximum value of the parallax Δxmax (the parallax of the subject farthest from the camera). Next, the depth display position calculator 1
7 calculates the position at which the farthest object is reproduced and displayed using (Equation 5). The coordinate system is the same as the three-dimensional image display coordinate system shown in FIG. Here, FIG.
Is written by superimposing the binocular fusion range of the observer of FIG. 3 on the display model of FIG. 2B. The shaded area is an area where the observer can integrate both eyes. Here, F (ds) max is given by the upper curve in FIG.

At this time, the fusion range confirming unit 18 determines that the relationship of (Equation 12) is satisfied from the relationship between the value of Ypmax and the value of the binocular fusion limit point F (ds) max at the distant point of the observer. Since the reproduction display position of the stereoscopic image is within the binocular fusion range, the fusion range confirmation unit 18 outputs the current value of dx to the convergence angle control unit 19 as it is.

[0064]

(Equation 12)

When Ypmax satisfies (Equation 13),

[0066]

(Equation 13)

Since the reproduction display position of Ypmax is completely within the binocular fusion range, the current dx is subtracted by Δdx to obtain a new dx. If the value obtained by subtracting ΔF from dx is smaller than dx1 (> 0),
Let dx1 be the new dx.

Here, the calculation of dx1 will be described. The fusion range checking unit 18 determines the y coordinate of Ypmax as F (d
s) Calculate dx under the condition of max as dx1. The formula for calculating dx is the same as in the first embodiment of the present invention (Equation 5).
And (Equation 2), ΔS is eliminated, and Yp = F (ds) min
And further solved by dx to obtain (Equation 14).

[0069]

[Equation 14]

As shown in FIG. 9, Ypmax is given by (Equation 1)
If 5) is satisfied, the image reproduced at Ypmax is an image that cannot be binocularly fused by the observer.

[0071]

(Equation 15)

In this case, the fusion range checking unit 18 determines the current d
The value obtained by adding Δdx to x is output to the convergence angle control unit 19 as a new dx. However, if θ (FIG. 6) described later approaches 0, the optical axis angle θ of the camera described later hardly changes, so that dx need not be updated.

Using the dx obtained in this way, the convergence angle control unit 19 controls the rotation control units 4a and 4b, and the camera bodies 3a and 3b move the camera to the convergence point shown in FIG. Control is performed so that the optical axis is oriented. The definition of the camera control angle θ at this time is the same as that in FIG. 6A of the first embodiment of the present invention. When θ = 0, the optical axes of the two cameras are parallel.
Here, the update of dx is performed by Δdx, but in the case of the condition of (Expression 15), the value of dx_1 expressed by (Expression 14) may be directly substituted for dx.

As described above, the point Ypmax of the object farthest from the observer is the limit far point F (d
s) Controlled to be max. At this time, if Ypmax is not within the binocular fusion range even under the condition that the optical axes of the camera bodies 3a and 3b are parallel, the optical axes of the two cameras are kept parallel. Also, instead of controlling dx, (Equation 15)
Is satisfied, the camera interval Wc is calculated as Wc− △ Wc (△
Wc may be changed to an arbitrary small value), and this may be repeated until (Equation 12) holds.

As described above, according to the present embodiment, the parallax of the imaging screen is calculated, and the optical axis of the stereoscopic imaging camera is controlled so that the maximum value of the parallax is reproduced within the fusion range of the observer. In addition, it is possible to capture an image that allows the observer to fuse the reproduced and displayed stereoscopic image in a wide range.

In the first to third embodiments of the present invention, since the output of the parallax calculating section 15 contains noise depending on the state of the subject, a spatio-temporal low-pass filter is added to the output of the parallax calculating section 15. May be applied to reduce the noise, and since the control of the two cameras may be slow, the output Δxmin of the parallax processing unit 16 or the output dx of the fusion range confirmation unit 18 and the determination of the fusion range may be performed to prevent abrupt operation. In the output dx of the unit 20,
A low-pass filter in the time direction may be applied.

FIG. 10 shows a configuration diagram of a stereoscopic image display apparatus according to a fourth embodiment of the present invention. 10, 29 is a parallax calculator, 30 is a gazing point calculator, 31
Denotes a parallax control unit, and 32 denotes an image display unit.

The operation of the thus configured stereoscopic image display section of the present embodiment will be described below. First, the parallax calculation unit 29 calculates a parallax map (three-dimensional map) from the left and right eye images. The calculation method is the same as that of the first embodiment of the present invention, and uses a correlation matching method or the like for calculating the correlation between the luminance patterns of the left and right images. That is, in FIG. 4, a block window of n × n pixels (3 × 3 pixels in the figure) is considered in the left image, and the same image as the block window is searched for in the right image using the same size window. The horizontal shift component Δx of the shift (Δx, Δy) of the block position is the binocular parallax of the left and right images at the center coordinates of the block window. If the position of the block window of the reference left image is translated across the entire screen and the position of the corresponding block of the right image (binocular disparity) is obtained in all cases, the disparity map of the entire screen (each image on the screen) Which indicates the depth distance at the place). The calculation formula is shown in the first embodiment,

[0079]

(Equation 16)

Here,

[0081]

[Equation 17]

Is as follows.

On the basis of the disparity map obtained as described above (calculation of the disparity Δx at each coordinate for the entire screen), the gazing point calculating unit 30 calculates the average value of the disparity of the entire image or the central part of the screen. Weighted average value Δxav
Calculate e. FIG. 11 shows an example of a screen position and a weight coefficient. In FIG. 11, the weighting coefficient of an area having a half size of the screen size X × Y is simply K2, and the other coefficients are K1. The shape of the region may be another shape, for example, a circle, or the weighting coefficient may be continuously changed from the periphery to the center. At this time, the average value Δxave is given by (Equation 1)
8).

[0084]

(Equation 18)

Where K = K2 (area A), K = K1 (area B),
When K1 = K2, a simple average of the entire screen is taken.

Using the average value of the binocular parallax Δxave, the parallax control unit 31 controls the horizontal read timing of the left and right images and translates the images in the horizontal direction. FIG.
Indicates a horizontal scanning period of the video signal. Point A
It is assumed that L and AR indicate the same position of the same subject in the left image and the right image, respectively. The binocular parallax Δx at the position A described above is shown in the figure. As shown in this figure, the right image is shifted in the direction of canceling it by the average of the binocular parallax Δxave (the horizontal read timing of the image is shifted by Δxave). In this way, an image portion having a binocular parallax of Δxave is reproduced on the display surface of the image display unit 32 (reproduced at the same position in the left and right images). Here, the entire image is translated in parallel.
As described above at 0, this corresponds to moving LA and LB, and if the controlled binocular parallax amount is small, the observer does not notice much of this change. Also, when the value of Δxave changes too fast in time and the display screen moves frequently, low-pass filtering is performed on this signal to control the display image using only slow movement. good.

The gazing point calculation unit 30 calculates the maximum value of the parallax instead of the average value of the parallaxes of the entire image. When the maximum value is a positive value, the parallax control unit 31 causes the display surface of the image display unit 32 to operate. If the maximum deviation of the left and right images above is set so as not to exceed the distance between the eyes of the observer (about 65 mm), the line of sight of the observer will not spread more than parallel, and the displayed image will be displayed with both eyes. It can be controlled to be within the fusion range. Further, the gazing point calculation unit 30 replaces the average value of the parallax of the entire image with
The minimum value of the parallax is calculated, and when the value is a negative value, the parallax control unit 31 determines the amount of deviation between the left and right images on the display surface of the image display unit 32 so that the minimum value does not become equal to or less than a predetermined value β. With such a setting, the observer's viewpoint is located very close, and it is possible to eliminate a large mismatch between the focus information of the eyes from the three-dimensional image display surface and the convergence angle of the line of sight. Left and right images can be controlled to facilitate eye fusion.

If the shift amount of the display image is changed from Δxave to Δxave−α (α is an arbitrary value), Δxave
Can be set to a certain binocular parallax value α. The case of α = 0 indicates the image display surface position, and this position is on the near side or the rear side with respect to the image display surface depending on the value of α. At this time, the control was performed using one binocular parallax average value for one screen. However, in the case of a moving image, the binocular parallax average value for each screen (one frame of an NTSC image) is obtained, and this is used as time-series data. The image display unit may be controlled using a signal that has been low-pass filtered in the time axis direction.

In the fourth embodiment, only the right image is shifted. However, the right image may be shifted by a half shift to be controlled, and the left image may be shifted by the same amount in the opposite direction. A median (median processing filter) may be applied instead of the average binocular parallax.

Further, in this embodiment, the display image is controlled only in the horizontal direction using the displacement Δxave due to the horizontal parallax. However, Δyave is calculated using the binocular parallax Δy calculated by the parallax calculator 29, and the vertical If the image reading position in the direction is also controlled, a three-dimensional image with reduced vertical parallax can be displayed, and the three-dimensional image can be more easily viewed. Very severe interference with the eye fusion function).

As described above, according to this embodiment, the average value of the binocular parallax (probably the depth position near the center of the parallax map)
Can be controlled on the display display surface or at a predetermined position, so that the observer can always perceive the depth world of the subject in the widest range.

FIG. 13 is a block diagram of a stereoscopic image display apparatus according to a fifth embodiment of the present invention. In FIG. 13, 2
Reference numeral 9 denotes a parallax calculating unit, 31 denotes a parallax control unit, and 32 denotes an image display unit, which are the same as those in FIG. 10 of the present invention. 10 differs from the configuration of FIG. 10 in that the gaze sensor unit 33, the gaze detection unit 34, and the gazing point determination unit 35 measure the gaze indicating the location of the image viewed by the observer of the display image. The difference is that a function of controlling the binocular parallax amount by the control amount calculation unit 36 using the output and the viewer's line of sight information is added.

The operation of the stereoscopic image display apparatus according to the fifth embodiment configured as described above will be described below.

First, similarly to the fourth embodiment of the present invention, the parallax calculator 29 calculates a parallax map (three-dimensional map) from the images for the left and right eyes. The calculation method may be the same as that of the fourth embodiment. For example, a correlation matching method for calculating the correlation between the luminance patterns of the left and right images, such as the algorithm shown in FIG. 4, can be used. The parallax map obtained as described above (parallax Δx at each coordinate for the entire screen)
The control amount calculation unit 36 receives the position (point of gaze) of the observer on the display surface of the image display unit 32 as an input, and the control amount calculation unit 36 reads the left and right images as input signals of the parallax control unit 31 horizontally. Calculate the timing deviation.

Here, a method of detecting the gazing point of the observer (the gazing point) will be described. First, the direction of the optical axis of the human eyeball is detected by the visual sensor unit 33 and the line-of-sight detection unit 34. There are various detection methods, such as a scleral reflection method, a corneal reflection method, and a method of photographing an eye part with a TV camera, and any of them may be adopted. Here, the scleral reflection method will be described. FIG. 14 is a principle diagram of the line-of-sight sensor unit 33 in the scleral reflection method. (A) is a front view of the eyeball,
FIG. 2B is a diagram viewed from the side. As shown in FIG. 3A, the eye movement in the horizontal direction is as follows. The infrared LED irradiates the eye with weak infrared light by an infrared LED, and the reflected light of the infrared light from the eye is measured by a photodiode. At this time, set the directivity of the photodiode as shown by the dotted line on both sides of the iris,
If the difference between the outputs of the photodiodes is calculated by an operational amplifier, horizontal eye movement can be detected. As for the vertical eye movement, as shown in FIG. 3B, infrared light is applied to the lower side of the eyeball, and the reflected light is detected using a photodiode having directivity directed to the lower part of the iris. The horizontal and vertical eye movement signals obtained as described above are
It is converted into a line-of-sight angle (line-of-sight signal). That is, it is measured that there is a line of sight at an angle of a horizontal ax degree and a horizontal ay degree with respect to a predetermined reference point (the optical axis of the eyeball of the observer faces the direction of the angle).

Next, the gaze point determination unit 35 calculates which part of the image display unit 32 the observer is gazing at based on the line-of-sight signal. Specifically, as shown in FIG. 15, the distance between the observer and the display surface of the image display unit 32 is L, the reference point is at the center of the display surface, and the two-dimensional coordinate system x, If y is defined, the viewpoint F (x, y) represented by the two-dimensional coordinate system on the display surface is expressed by the following equation (Equation 19) using the line-of-sight signal (ax, ay).

[0097]

[Equation 19]

However, it is assumed that the observer is located in front of the center of the display surface, and the display surface is not inclined with respect to the observer.

Using the gaze point of the observer (the position of the image that the observer is gazing at) on the screen obtained in this way, the gaze point determination unit 35 further looks at the same place where the observer is long. Calculate the point that is. FIG. 16A shows an example of a temporal change of the x component of the viewpoint F. Assuming that the observer looks at the positions x1 to x5 in this way, the movement of the human eyeball is very fast when the viewpoint changes from x1 to x2, x2 to x3, etc. (Saccade). Also, when the observer blinks, the viewpoint measurement becomes impossible, and the waveform changes drastically due to the high-speed movement of the eyelids. The gaze point determination unit 35 removes such a transient state, and detects a state in which the observer is looking at the positions x1 to x5 (gaze state). In the detection method, as shown in FIG. 16B, the moving speed of the viewpoint is calculated, and when the moving speed is equal to or less than a certain threshold value β, the observer may be in a gaze state (FIG. 16B). * Part). Here, only the x component of the viewpoint is used, but the gaze state may be detected using a two-dimensional velocity using both the x and y components. FIG. 16C shows the temporal change of the gaze point of the observer extracted in this manner.
Shown in

Next, as shown in FIG. 16 (d), the data at the point of interest is held in the 0th order, and the time without the point of interest data is interpolated. Thereafter, as shown in FIG. 16 (e), the temporal change of the detected gazing point is smoothed using a low-pass filter. As described above, the gazing point determination unit 35
Detects the position of the display image that the observer is gazing at and outputs it to the control amount calculation unit 36 as a gazing point signal. Here, the gaze point signal is processed with respect to the direction of the line of sight of the single eye, but the gaze direction of both the left and right eyes may be detected, and the gaze point signal may be calculated using the average value.

Based on the parallax map and the gazing point signal obtained as described above, the control amount calculation unit 36 determines the control amount of the horizontal read timing of the left and right images of the parallax control unit 31. This will be described with reference to FIG. F indicates the gaze position of the observer indicated by the gaze point signal at a certain time. With this position as the center, the binocular disparity Δx indicated by the disparity map in the range of M × M pixels is averaged. This is defined as the average value Δxave of the binocular parallax described in the fourth embodiment of the present invention. Using this, the parallax control unit 31 controls the horizontal read timing of the left and right images, and translates the images in the horizontal direction. As the control method, the same method as that described with reference to FIG. 12 in the fourth embodiment of the present invention is used. That is, the average Δxave of binocular parallax
The right image is shifted by the same amount (the horizontal read timing of the image is shifted by Δxave).

Further, in the fifth embodiment of the present invention, the control amount calculation unit 36 averages the binocular disparities in a square centered on the gaze position. Thus, a weighted average calculation with a high center weight may be performed.

Although the line-of-sight signal is detected only from the direction of the optical axis of the viewer's eyeball, this case is effective for a case where the human does not move his head much. Considering the free viewing state, the human gaze is given as a composite value of the direction of the eyeball and the direction of the head. Such a head movement
Regarding gaze detection that integrates eye movements, a method of imaging the head of an observer with a TV camera, and a method disclosed in Japanese Patent Application No. Hei.
Although there is a method using a magnetic field generator and a magnetic field detector mounted on the head as shown in FIG. 6, any method may be adopted.

As described above, according to this embodiment, the binocular disparity at the position of the image which the observer is currently looking at is 0.
(The image is reproduced at a position on the display surface) or a predetermined value, and the stereoscopic image is always formed around the position of the image intended by the observer.
The depth world of the subject can be viewed over a wide range.

FIG. 18 is a configuration diagram of a stereoscopic image display apparatus according to a sixth embodiment of the present invention. FIG. 1A shows a stereoscopic image capturing and recording unit, and FIG. 1B shows a stereoscopic image display unit. FIG.
In 8, the parallax calculator 29, 31 the parallax controller, 3
Reference numeral 2 denotes an image display unit, and reference numeral 36 denotes a control amount calculation unit, which are the same as in the fifth embodiment (FIG. 13) of the present invention. Fifth
The third embodiment differs from the third embodiment in that a stereoscopic imaging camera 37, a recording device 38 as a signal recording unit, a gaze intention designating unit 39, and a reproducing device 40 as a reproducing unit are added.

The operation of the display section of the sixth embodiment having the above-described configuration will be described below.

First, in FIG. 18A, the photographer takes a picture using the stereoscopic camera 37. The stereoscopic imaging camera 37 may be a conventional stereoscopic camera in which two conventional video cameras are arranged side by side on the same camera platform, and may be a stereoscopic image capturing apparatus as described in the first embodiment of the present invention. There may be. The captured image is recorded by the recording device 38 (an object obtained by synchronizing two conventional VTRs). At this time, the photographer inputs the position of the image intended to be watched by the viewer (gaze intention signal) by the gaze intention designation unit 39. The gaze intention signal is a signal that indicates a certain point in an image being captured by inputting an existing pointing device such as a mouse or a tablet as an input device, and is a signal that changes with time. Is indicated by Fix (xt, yt). Here, xt and yt are outputs of the pointing device at time t, that is, two-dimensional coordinate values on the captured image. The gaze intention signal is recorded together with the photographed image using the recording device 38 (VTR). In particular,
The gaze intention signal is sampled at a field frequency of 1/60 second and inserted into a vertical blanking period (vertical blanking period) of the image signal. This gaze intention designating section 39
May be attached to an editor or a VTR to record a gaze intention signal when editing a captured image.

The image signal and the gaze intention signal recorded as described above are processed by the three-dimensional image display unit shown in FIG. 18B, and a three-dimensional image is displayed. Playback device 40 (V
TR) reproduces an image signal and a gaze intention signal. Subsequent operations are the same as those in the fifth embodiment (FIG. 13) of the present invention in which the gazing point signal is replaced with the gazing intention signal obtained from the reproducing apparatus 40. That is, based on the parallax map calculated by the parallax calculation unit 29 and the gaze intention signal reproduced by the reproduction device 40, the control amount calculation unit 36 determines the control amount of the horizontal read timing of the left and right images of the parallax control unit 31. decide. In this case as well, as shown in FIG. 17, with the image position desired to be watched by the observer indicated by the watch intention signal as the center,
The binocular parallax Δx indicated by the parallax map in the range of M × M pixels is averaged, and the average is defined as the average binocular parallax Δxave described in the fourth embodiment of the present invention. Using this, the parallax control unit 31 controls the horizontal read timing of the left and right images,
Translate the image horizontally. The control method is the same as that described in the fourth embodiment with reference to FIG. That is,
The right image is shifted by the average of the binocular parallax Δxave (the horizontal read timing of the image is shifted by Δxave).

In FIG. 18, instead of reproducing the gaze intention signal by the reproducing device 40, the output of the gaze intention designation unit 39 is directly input to the control amount calculation unit 36, and the viewer can watch the directly reproduced image. The gaze intention signal may be input while inputting.

Although the fourth to sixth embodiments have been described with respect to only the binocular three-dimensional image, the multiview three-dimensional image display device may also be configured to display two images projected on the left and right eyes of the viewer. Similar processing can be performed on images. However, in this case, since there are a plurality of pairs of images for the left and right eyes that are viewed at a time, it is necessary to perform the same processing on all of them.

As described above, according to the present embodiment, by controlling the reading position of the stereoscopic image using the gaze intention signal, the image program creator, the photographer, and the viewer can view the image intended for the part. Can be controlled to a position on the display surface or to a predetermined position, and a stereoscopic image with a wide fusion range is always constructed centering on the position of the image intended by the observer. You can enjoy the world of depth.

FIG. 19 is a block diagram of a stereoscopic image pickup and display device according to a seventh embodiment of the present invention. In FIG. 19, 1 and 2 are lenses, 3a and 3b are camera bodies, 4a and
4b is a rotation control unit, 5 is a camera interval control unit, 14a, 14
b is a signal processing unit that converts a signal from the camera body into a luminance signal, 15 is a parallax calculation unit that calculates horizontal parallax from left and right images, 16 is a parallax processing unit that detects the minimum parallax, and 17 is the most A depth display position calculation unit that calculates the spatial position of a close subject, 18 is a fusion range confirmation unit that checks whether or not the spatial position of the subject closest to the observer is within the fusion range of the observer, and 19 is a fusion range check unit that most closely observes the observer. Rotation control unit 4 so that the spatial position of a close subject falls within the fusion range of the observer.
These are convergence angle control units for controlling a and 4b, which are the same as the configuration (FIG. 1) of the stereoscopic image pickup apparatus according to the present invention. Reference numeral 30 denotes a gazing point calculation unit, 31 denotes a parallax control unit, and 32 denotes an image display unit, which are the same as those of the stereoscopic image display device (FIG. 10) according to the fourth embodiment of the present invention.

That is, in this embodiment, both the image pickup unit and the display unit are controlled so that the displayed stereoscopic image is within the range of the observer's binocular fusion.

The operation of the three-dimensional image pickup and display apparatus according to the seventh embodiment having the above-described configuration will be described below. First, dx shown in FIG. 2 (a) is imaged under the condition that all three-dimensional images are reproduced in this fusible area and under control to have as large a value as possible. This operation is exactly the same as that of the first embodiment, but its outline will be described. For this purpose, it is assumed that dx = ∽ (a sufficiently large value) as an initial value. In FIG. 19, the parallax calculation unit 15 calculates a parallax map (a three-dimensional map in which the parallax Δx at each coordinate is calculated for the entire screen) from the left and right eye images output from the signal processing units 14a and 14b. Based on this disparity map, the disparity processing unit 16 extracts the minimum value of the disparity of the entire image (the disparity of the subject reproduced at the position closest to the observer) and sets this as Δxmin.
Next, based on Δxmin, the depth reproduction position Ypmin of the stereoscopic image in the stereoscopic image display shown in FIG. 5 is calculated by the depth display position calculator 17 as shown in (Equation 20).

[0115]

(Equation 20)

If Ypmin is in the boundary region of the binocular fusion range, dx is set to the current value. Otherwise, in the case of (Equation 21), a small value Δdx is added to dx. And

[0117]

(Equation 21)

In the case of (Equation 22), a small value Δ is obtained from dx.
Subtract dx.

[0119]

(Equation 22)

These calculation methods are exactly the same as in the first embodiment. Using the dx obtained in this manner, the convergence angle control unit 19 controls the rotation control units 4a and 4b. As described above, the point of the subject closest to the observer Ypmin
Is controlled to be F (ds) min which is the near point of the fusion range.

Further, using the output of the parallax calculating section 15, the gazing point calculating section 30 calculates the average value or weighted average value Δxave of the parallax of the entire image, and the parallax control section 31 obtains the Δxave.
The left and right images are scrolled so as to eliminate the shift of the left and right images by the minute. This processing is the same as in the fourth embodiment of the present invention.

By performing the above, the image display unit 3
The convergence point of the imaging unit and the display image position are controlled so that the stereoscopic image displayed in 2 is within the binocular fusion range of the observer. This allows the observer to always view a stereoscopic image having a wide range of binocular fusion, and reduces eye fatigue during viewing.

In the seventh embodiment, the operation of the convergence angle control unit 19 and the operation of the parallax control unit 31 are performed simultaneously. However, in order to stabilize the operation of the entire system,
The two control units may alternately operate, or one may operate after the other has stabilized. In the seventh embodiment, the stereoscopic image pickup device of the first embodiment is used for the image pickup section. Instead, the stereoscopic image pickup device of the second or third embodiment is used. Is also good.

[0124]

As is apparent from the above description, the present invention detects the binocular disparity of an image and determines the optimum convergence point of the camera from which the observer can perceive the subject's depth world in the widest range. By calculating and controlling the camera's optical axis to point at the convergence point, and by controlling the display image so that the binocular fusion range of the display image is wide, a stereoscopic image that is easy to see and has less eye fatigue is displayed. It has the advantage of being able to.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a stereoscopic image capturing apparatus according to a first embodiment of the present invention.

FIG. 2 is an explanatory diagram of an imaging coordinate system and a display coordinate system of a stereoscopic image.

FIG. 3 is an explanatory diagram of allowable ranges of a human adjustment mechanism and a congestion mechanism.

FIG. 4 is an explanatory diagram of a parallax calculation method.

FIG. 5 is an explanatory diagram of a binocular fusion range in a stereoscopic image display device (viewing distance> 71 cm).

FIG. 6A is a diagram for explaining the definition of the distance dx to the convergence point and the rotation angle θ of the optical axis, and FIG. 6B is a diagram illustrating the distance dx to the convergence point; FIG. 6 is a diagram illustrating a relationship between rotation angles θ.

FIG. 7 is a configuration diagram of a stereoscopic image capturing apparatus according to a second embodiment of the present invention.

FIG. 8 is a configuration diagram of a stereoscopic image capturing apparatus according to a third embodiment of the present invention.

FIG. 9 is an explanatory diagram of a binocular fusion range in the stereoscopic image display device (viewing distance <71 cm).

FIG. 10 is a configuration diagram of a stereoscopic image display device according to a fourth embodiment of the present invention.

FIG. 11 is an explanatory diagram of a technique for calculating a weighted parallax average value.

FIG. 12 is a diagram illustrating an operation of a parallax control unit 12 according to the fourth embodiment.

FIG. 13 is a configuration diagram of a stereoscopic image display device according to a fifth embodiment of the present invention.

FIGS. 14A and 14B are diagrams for explaining the operation of the line-of-sight sensor unit 14 in the fifth embodiment.

FIG. 15 is an explanatory diagram of a method of calculating coordinates of a gazing point on a display image.

FIGS. 16A to 16E are explanatory diagrams of a method of calculating a gazing point position.

FIG. 17 is an explanatory diagram of a calculation method of the control amount of the horizontal read timing of the left and right images.

FIGS. 18A and 18B are configuration diagrams of a stereoscopic image capturing and displaying apparatus according to a sixth embodiment of the present invention.

FIG. 19 is a configuration diagram of a stereoscopic image capturing and displaying apparatus according to a seventh embodiment of the present invention.

FIG. 20 is an explanatory diagram of formation of depth perception of an observer by binocular parallax.

FIG. 21 is a schematic diagram of a conventional three-dimensional image pickup device.

FIG. 22A is an explanatory diagram of a conventional stereoscopic image photographing method (parallel photographing), and FIG. 22B is an explanatory diagram of a conventional stereoscopic image photographing method (convergence photographing).

FIG. 23 is a schematic view of a conventional stereoscopic image display device.

FIG. 24 is an explanatory diagram of a stereoscopic image display position.

FIG. 25 is an external view of a conventional three-dimensional image display device.

[Explanation of symbols]

 1, 2 Lens 3a, 3b Camera body 4a, 4b Rotation control unit 5 Camera interval control unit 12 Screen 14a, 14b Signal processing unit 15, 29 Parallax calculation unit 16 Parallax processing unit 17 Depth display position calculation unit 18 Fusion range confirmation unit 19 Convergence angle control unit 20 Fusion range determination unit 30 Gaze point calculation unit 31 Parallax control unit 33 Gaze sensor unit 34 Gaze detection unit 35 Gaze point determination unit 36 Control amount calculation unit 38 Recording device 39 Gaze intention designation unit 40 Playback device

Claims (14)

(57) [Claims] An imaging unit for imaging a subject from a plurality of viewpoints, a convergence angle moving mechanism for changing a convergence angle of the imaging unit, or a camera interval control mechanism for changing a camera interval, and an image from each imaging unit A signal processing unit for extracting data, a disparity calculation unit for calculating a disparity of an image using an output of the signal processing unit, and a disparity for detecting a minimum value (disparity of the closest subject) among outputs of the disparity calculation unit A processing unit, and a depth display position calculation unit that calculates the depth position of the closest object reproduced when an image captured using the output of the parallax processing unit is displayed on the stereoscopic image display device. Using the output of the depth display position calculation unit, a fusion range confirmation unit that determines whether or not this is within the fusion allowable range of the observer watching the reproduced image, and using the output of the fusion range confirmation unit to determine the closest point Suffered A convergence angle control unit that controls the convergence angle moving mechanism unit so that the object falls within the fusion allowable range of the observer.
Or stereoscopic imaging apparatus characterized by comprising a camera interval control unit for controlling the camera interval control mechanism. 2. The parallax processing unit detects a maximum value (parallax of a farthest object) among outputs of the parallax calculation unit, and the depth display position calculation unit uses the output of the parallax processing unit to perform imaging. Calculates the depth position of the farthest object reproduced when the displayed image is displayed on the stereoscopic image display device, and the convergence angle control unit or the camera interval control unit uses the output of the fusion range confirmation unit. 2. The three-dimensional image pickup apparatus according to claim 1, wherein the convergence angle moving mechanism or the camera interval control mechanism is controlled so that the object at the farthest point is within the fusion allowable range of the observer. 3. An imaging unit for imaging a subject from a plurality of viewpoints, a convergence angle moving mechanism unit for changing a convergence angle of the imaging unit or a camera interval control mechanism unit for changing a camera interval, and A depth display position calculation unit that calculates the depth position of the closest object reproduced when the captured image is displayed on the stereoscopic image display device using the position data of the closest object, and a depth display position calculation unit A fusion range determining unit that calculates a condition under which the reproduced display image of the near-point subject falls within the binocular fusion range of the observer using the output of the observer, and a convergence angle moving mechanism using the output of the fusion range confirmation unit A three-dimensional image pickup device, comprising: a convergence angle control unit that controls a unit or a camera interval control unit that controls the camera interval control mechanism unit. 4. A depth display position calculation unit, using position data of a farthest object among the objects to be imaged, of a farthest object reproduced when a captured image is displayed on a stereoscopic image display device. Calculate the depth position, the fusion range determination unit, using the output of the depth display position calculation unit,
A condition under which the reproduced display image of the distant subject is included in the binocular fusion range of the observer is calculated, and the convergence angle control unit or the camera interval control unit uses the output of the fusion range confirmation unit to move the convergence angle. 4. The three-dimensional image pickup device according to claim 3, wherein the control unit controls a mechanism unit or a camera interval control mechanism unit. 5. Using the image when capturing a subject from at least two directions, and the parallax calculation section for calculating binocular parallax or three-dimensional position of the object, the depth position location of the object from the output of the parallax calculation section A gaze point calculation unit for calculating, and a parallax control for translating the image so as to reproduce a depth position indicated by an output of the gaze point calculation unit on a surface of the three-dimensional stereoscopic image display unit or a specified depth distance from the surface. And a stereoscopic image display device. 6. The parallax control unit controls the parallax control unit by averaging the entire screen or a small area in the image or outputting a weighted average obtained by adding a large weight to the center of the screen among the outputs of the parallax calculation unit. The stereoscopic image display device according to claim 5, wherein a depth position to be performed is calculated. 7. A depth-of-view position to be controlled by the parallax control unit by calculating a maximum value or a minimum value in an entire screen or a small area in an image from among outputs of the parallax calculation unit. The stereoscopic image display device according to claim 5, wherein is calculated. 8. A gazing point calculating section for detecting a gazing point of an observer; a gazing point determining section for detecting a gazing position in a display image of the observer from an output of the gazing point detecting section; The three-dimensional object according to claim 5, further comprising: a control amount calculating unit that calculates an average value of binocular parallax of a display image position required from an output of the parallax calculating unit and an output of the fixation point determining unit. Image display device. 9. The gazing point determining unit extracts only a period during which the line of sight movement speed of the observer is low, and performs a hold process and a low-pass filtering process to detect a gazing position of the observer. The three-dimensional image display device according to claim 8. 10. A control amount calculating unit for averaging an output of the parallax calculating unit within a predetermined range around a gazing point on a display image of the observer indicated by an output of the gazing point determining unit. 9. The stereoscopic image display device according to claim 8, wherein a depth position to be controlled by the parallax control unit is calculated. 11. A gaze intention designating unit for designating a photographed image portion intended to be watched by an observer, an output of the gaze intention designating unit, and a photographing object when photographing a subject from at least two directions. A signal recording unit for recording an image, a reproducing unit for reproducing a captured image and a gaze intention signal, a parallax calculating unit for calculating binocular parallax or a three-dimensional position of a subject, an output of the parallax calculating unit, and the gaze A stereoscopic image, comprising: a parallax control unit configured to control an image portion indicated by the gaze intention signal using the intention signal so as to reproduce the image portion at a specified depth distance from the surface of the three-dimensional stereoscopic image display unit. Display device. 12. The stereoscopic image display device according to claim 11, wherein the gaze intention signal uses an output of the gaze intention designating unit instead of the gaze intention signal reproduced by the reproducing unit. 13. A three-dimensional image pickup and display device comprising: the three-dimensional image pickup device according to any one of claims 1 to 4; and the three-dimensional image display device according to any one of claims 5 to 8. . 14. A three-dimensional image pickup and display device, comprising: the three-dimensional image pickup device according to claim 1; and the three-dimensional image display device according to claim 11 or 12.