JP3416706B2 - Pupil position detecting device and method, viewpoint position detecting device and method, and stereoscopic image display system - Google Patents

Description

Detailed Description of the Invention

[0001]

TECHNICAL FIELD The present invention relates to a person to be measured (observer).
The present invention relates to a pupil position detecting device and a pupil position detecting method for detecting the pupil position, and more particularly to a pupil position detecting device and a pupil position detecting method used in combination with a display device. The present invention further relates to a viewpoint position detecting device and method using the pupil position detecting device and method, and a stereoscopic image display system.

[0002]

2. Description of the Related Art Conventionally, there is a so-called direct-view type stereoscopic image display device capable of stereoscopic vision with the naked eye. There are various types of three-dimensional image display devices, for example, Japanese Patent Laid-Open No. 09-31129.
Japanese Patent Publication No. 4 discloses a rear cross lenticular type. FIG. 7 is a perspective view of an essential part showing an example of a rear cross lenticular stereoscopic image display device. 6 in the figure
Is a display device for displaying an image, and is composed of, for example, a liquid crystal element (LCD). In the figure, a polarizing plate, a color filter, an electrode, a black matrix, an antireflection film, etc. are omitted.

Reference numeral 10 is a backlight (surface light source) which serves as an illumination light source. Display device 6 and backlight 1
Between 0, a mask substrate (mask) 7 having a mask pattern having checkered openings 8 is arranged. The mask pattern is manufactured by patterning a metal vapor deposition film such as chromium or a light absorbing material on the mask substrate 7 made of glass or resin. The backlight 10, the mask substrate 7 and the like constitute one element of the light source.

A first lenticular lens 3 and a second lenticular lens 4 made of transparent resin or glass are arranged between the mask substrate 7 and the display device 6. The first lenticular lens 3 is a vertical cylindrical lens array configured by vertically arranging vertical cylindrical lenses in the left-right direction, and the second lenticular lens 4 is a horizontal lenticular lens formed by vertically arranging horizontal cylindrical lenses in the vertical direction. It is a cylindrical lens array.

As shown in the figure, the image displayed on the display device 6 is obtained by dividing left and right parallax images R and L into a large number of horizontal stripe pixels R and L in the vertical direction, respectively, and for example, LRLRLLR from the screen. ...
Is a horizontal stripe image that is alternately arranged to form one image.

Light from the backlight 10 is emitted from the mask substrate 7
The display device 6 is illuminated through each of the openings 8 and the left and right stripe pixels R and L are separately observed for both eyes of the observer.

That is, the mask substrate 7 is the backlight 1.
It is illuminated by 0, and light is emitted from the opening 8. The first lenticular lens 3 is arranged on the observer side of the mask substrate 7, and the lens curvature is designed so that the mask substrate 7 is located at substantially the focal position of each of the cylindrical lenses. In this cross section, the second lenticular lens 4 has no optical effect, so that the light beam emitted from one point on the aperture 8 is converted into a substantially parallel light beam in this cross section.

The pair of openings and the light-shielding portion of the mask pattern are set so as to substantially correspond to one pitch of the first lenticular lens 3.

The first lenticular lens 3 is based on the relationship between the optical distance from the predetermined position of the observer to the first lenticular lens 3 and the optical distance from the first lenticular lens 3 to the mask pattern. It is possible to make the light from the openings 8 uniformly gather to the left eye or the right eye over the entire width of the screen by determining the pitch of the pair of openings and the pitch of the pair of openings of the mask pattern and the light shielding section.
In this way, the left and right stripe pixels on the display device 6 are horizontally observed separately in the left and right eye regions.

The second lenticular lens 4 condenses all the luminous flux emitted from each point on the opening 8 of the mask 7 onto the stripe pixel for the right eye or the left eye of the display device 6, illuminates and transmits it. It diverges only in the vertical direction according to the NA at the time of focusing, and provides an observation area in which the left and right stripe pixels are uniformly separated from the predetermined eye height of the observer over the entire width in the vertical direction of the screen. .

However, the viewing angle of such a stereoscopic image display device is narrow, and if the observer's viewpoint deviates from the viewing angle, the stereoscopic display cannot be recognized. Therefore, by detecting the viewpoint position of the observer and controlling the image display in response to the movement of the viewpoint position, by dynamically moving the viewing angle range in which stereoscopic viewing is possible, a substantial viewing angle is obtained. Have been proposed. For example, Japanese Patent Laid-Open No. 10-2323
Japanese Unexamined Patent Publication No. 67 discloses a technique in which a mask pattern or a lenticular lens is moved in parallel with a display surface to enlarge a stereoscopically visible region (viewing angle).

FIG. 8 is a diagram of a stereoscopic image display device disclosed in Japanese Patent Laid-Open No. 10-232367. 8, the same components as those of FIG. 7 are designated by the same reference numerals, and the description thereof will be omitted. The stereoscopic image display device in FIG. 8 does not have the second lenticular lens 4 in FIG. 7 because it has a single lenticular lens structure.

In the stereoscopic image display device having such a structure, control according to the movement of the observer 54 is performed as follows. First, the position sensor 51 detects a horizontal shift of the observer 54 from a preset reference position, sends the information to the control unit 52, and the control unit 52 causes the display drive circuit to respond to the shift information. When the image control signal is output to 50, the display drive circuit 50 displays the first or second horizontal stripe image on the display 6. At the same time, the control unit 52 generates an actuator drive signal based on the displacement information, and drives the actuator 53 that moves the mask pattern 7 in the horizontal direction, so that the observer 54 can separate the left and right stripe images at the best position. Move to. As a result, even if the viewpoint position of the observer 54 changes, the stereoscopic viewable range is expanded.

In addition to the method of physically moving the mask pattern, the mask pattern is formed by a transmissive liquid crystal display element as described in, for example, Japanese Patent Laid-Open No. 10-78563, and the pattern (opening) is formed. The viewing angle can also be expanded by controlling the size of the.

FIG. 9 is a diagram of a stereoscopic image display device disclosed in Japanese Patent Laid-Open No. 10-78563. 9, the same components as those of FIG. 7 or 8 are designated by the same reference numerals, and the description thereof will be omitted.

The stereoscopic image display device shown in FIG. 9 is a cross-lenticular type direct-viewing stereoscopic image display device in which the mask pattern 7 can dynamically change the size of a pattern (opening) of a transmissive liquid crystal element or the like. The display portion has the same configuration as that of the display device shown in FIG. 7, except that the display portion is configured by various elements. On the other hand, as a display control unit, a position sensor 51 that detects the position of the observer 54 as in the display device shown in FIG. 8, and the opening area and shape of the mask pattern are changed according to the position information detected by this position sensor. It has a control unit 55.

Specifically, the control unit 55 changes the positions and widths of the opening (the transparent part of the pattern) and the light-shielding part (the same black part) according to the position of the observer, so that the left and right of the observer can be changed. By following the movement of, it realizes a good stereoscopic image display without crosstalk or reverse stereoscopic vision. Since the principle and specific method of expanding the viewing angle by moving the mask pattern and changing the pattern are not directly related to the present invention, detailed description thereof will be omitted here and will not be repeated here.

As described above, when the display device is controlled according to the position of the observer's viewpoint, an image display suitable for the observer's viewpoint can be obtained even if the detection accuracy is low or the processing time for detection is long. I can't do it. Therefore, how accurately the observer's viewpoint position is detected in a short time is very important in the performance of this type of display device.

As a method of detecting the viewpoint position of the observer (measured person), a method of irradiating the observer with infrared light and detecting the reflex of the retina (for example, Banno "Design of pupil photographing optical system for viewpoint detection Law ", IEICE Transactions D-II, Vol. J74
-D-II, No.6, pp.736-747, described in June 1991), the pupil position is detected and this pupil position is used as the viewpoint position, or the detected pupil position is used from the visible image. A method of detecting the viewpoint position is known.

These methods utilize the fact that the human pupil has the property of retroreflecting near infrared light (returning light in the same direction as the incident direction). Since the pupillary reflected light is obtained as a sharp reflection peak and usually has a higher reflectance than that of the face and the like, by photographing the observer using an infrared imaging device with the light source and the optical axis coaxial, only the pupil part is A bright image can be taken. If this photographed image is binarized with an appropriate threshold value, an accurate viewpoint position can be detected from the extracted pupil position.

[0021]

However, the size of the human pupil changes depending on the ambient brightness. Therefore, for example, when a viewpoint detection device using infrared light is arranged near the above-mentioned stereoscopic image display device to detect the viewpoint position of the observer, not only the brightness and contrast of the display device itself but also the displayed image are displayed. The size of the observer's pupil changes depending on the color tone, the environmental brightness of the display device, and the like. In the following description, the above-described various factors are combined, and the brightness that the observer senses is expressed as the screen brightness of the display device.

FIG. 5 is a conceptual diagram of an image obtained by the infrared image pickup device installed near the stereoscopic image display device. Figure 5
As shown in FIG. 5, when the display of the display device is dark, the pupil opens and a large reflection image is obtained (FIG. 5A), but when the display is bright, the pupil contracts and the reflection image also becomes small. (FIG.5 (b)). When the obtained reflection image becomes small, it becomes difficult to distinguish between the reflection image other than the observer's pupil and the reflection image of the pupil, which adversely affects the pupil position detection accuracy and the position detection processing time. become.

As a method for solving such a problem, for example, in the invention described in Japanese Patent Laid-Open No. 11-72697, an infrared image is subjected to differential calculation processing to emphasize a reflection image, and then the absolute value of the calculation result is calculated. It is shown that the position of the pupillary reflection image is detected by obtaining and comparing with the threshold value.

However, in this method, noise is also emphasized in the same manner as in the pupil reflection image, so that it is considered difficult to detect the noise.

Therefore, it is an object of the present invention to provide a pupil detecting apparatus and method capable of detecting the position of the pupil with high accuracy in a short time. Another object of the present invention is to provide a viewpoint position detection device and method and a stereoscopic image display system using pupil position information obtained by using the pupil position detection device or method of the present invention.

[0026]

That is, the gist of the present invention is a pupil position detecting device for detecting the pupil position of a subject and outputting pupil position information, and presenting a predetermined image to the subject. Brightness control means for controlling the screen brightness of the display means, light irradiation means for irradiating the measurement subject with light, image acquisition means for acquiring the image of the measurement subject, and image acquisition by the image acquisition means And a detection means for detecting the pupil position of the measurer and outputting the pupil position information. After the screen brightness of the display means is set to a predetermined value or less by using the brightness control means, irradiation by the light irradiation means and image acquisition are performed. The pupil position detecting device is characterized in that it is performed.

Another aspect of the present invention is a pupil position detecting method for detecting the pupil position of the subject and outputting the pupil position information, which is a display means for presenting a predetermined image to the subject. From the brightness control step of controlling the screen brightness, the light irradiation step of irradiating the measured person with light, the image acquisition step of acquiring the image of the measured person, and the image acquired in the image acquisition step, the pupil of the measured person A detection step of detecting the position and outputting pupil position information, and the brightness control step is performed to reduce the screen brightness of the display means to a predetermined value or less, and then the light irradiation step and the image acquisition step are executed. The method of detecting the pupil position is as follows.

Further, another gist of the present invention is that the pupil position detecting device of the present invention, the visible image acquiring means for acquiring a visible image of the measurement subject, and the visible image acquired by the visible image acquiring means are used to determine the pupil position. A template creating means for creating a template for pattern matching using information, and pattern matching with the visible image acquired by the visible image acquiring means using the template created by the template creating means to detect the viewpoint position of the measurement subject. However, it has a matching means for outputting the detection result as viewpoint position information, and a control means for controlling to create a template again by using the pupil position detecting device and the template creating means when a predetermined condition is satisfied. The viewpoint position detection device is defined as

Another subject matter of the present invention is a stereoscopic image display system having a viewpoint position detection device of the present invention and a stereoscopic image display means controllable by a brightness control means. A stereoscopic image display device having a mask pattern for controlling a light path, wherein the stereoscopic image display system is characterized in that the position of the mask pattern or the pattern is changed using the viewpoint position information received from the viewpoint position detection device. .

Another subject of the present invention is a pupil position detecting method for detecting the pupil position of the subject and outputting the pupil position information, which is a predetermined image for the subject externally connected. A brightness control step of controlling the screen brightness of the display means for presenting, an infrared light irradiation step of irradiating the person to be measured with infrared light, an infrared image acquisition step of acquiring an infrared image of the person to be measured, From the infrared image acquired in the infrared image acquisition step, having a detection step of detecting the pupil position of the measurement subject and outputting pupil position information, by the brightness control step,
A pupil position detecting method is characterized in that an infrared light irradiation step and an infrared image acquisition step are executed after the screen brightness of the display means is set to a predetermined value or less.

Another aspect of the present invention is that, in addition to the pupil position detecting method of the present invention, a visible image acquisition step of acquiring a visible image of the measurement subject and a pupil image from the visible image acquired in the visible image acquisition step. Using the template creation step of creating a template for pattern matching using position information and the template created in the template creation step, pattern matching is performed with the visible image acquired by the visible image acquisition step to determine the viewpoint position of the measurement subject. And a matching step of detecting the result and outputting the result as viewpoint position information.

Another aspect of the present invention is that, in addition to the pupil position detecting method of the present invention, a pupil is detected from a visible image acquiring step for acquiring a visible image of a subject and a visible image acquired in the visible image acquiring step. The template creation step that creates a template for pattern matching using position information, and the template created in the template creation step are used to perform pattern matching with the visible image acquired in the visible image acquisition step to determine the viewpoint position of the measurement subject. It has a matching step of detecting and outputting the detection result as viewpoint position information, and a control step of controlling to create a template again using the pupil position detecting method and the template creating step when a predetermined condition is satisfied. It lies in the characteristic viewpoint position detection method.

Another aspect of the present invention resides in a computer-readable storage medium in which the pupil position detecting method of the present invention is stored as a computer-executable program.

Another aspect of the present invention resides in a computer-readable storage medium that stores the viewpoint position detection method of the present invention as a computer-executable program.

[0035]

BEST MODE FOR CARRYING OUT THE INVENTION The present invention will be described below in detail based on its preferred embodiments with reference to the drawings. In the following description, a stereoscopic image capable of expanding the stereoscopically visible range by controlling the stereoscopic image display device based on the viewpoint position of the observer detected by the viewpoint position detecting device having the pupil position detecting device of the present invention. The display system will be described, but it goes without saying that the pupil position detecting device of the present invention can be used in any field and application.

In the following description, the viewpoint position means the coordinates of a point indicating the position of the eyes of the observer.
The viewpoint position information output by the viewpoint position detection device does not necessarily have to be the coordinate value of one point, and may be information indicating a certain area. In some cases, it may be sufficient to know the position of the entire eye, and it may be appropriately selected depending on the application.

(Overall Configuration of Stereoscopic Image Display System) FIG.
FIG. 1 is a diagram showing an appearance of a stereoscopic image display system according to this embodiment. In FIG. 1, the stereoscopic image display system is
It is composed of a photographing unit 1, a viewpoint position detection unit 2, and a stereoscopic image display unit 3. The image capturing unit 1 is built in the base of the stereoscopic image display unit 3, and includes a visible image capturing unit 11 that captures a visible image for viewpoint position detection and tracking, and an infrared image capturing unit that captures an infrared image for pupil position detection. 12 and. Visible image capturing unit 11
The infrared image capturing unit 12 and the infrared image capturing unit 12 are both stereoscopic image display units 3.
The angle of view and the shooting direction are determined according to the position where this system is installed and the predicted position of the observer so that the observer's head front face (face) of the stereoscopic image displayed in FIG. For example, when the display unit is installed on a desk and an observer sits on a chair and observes the display unit, the camera unit is arranged in a state having a slight elevation angle as shown in FIG. 1B.

Visible image photographing section 11 and infrared image photographing section 1
Both 2 can be configured by a video camera, and the infrared image capturing unit 12 is configured to cause only infrared light to enter the internal light receiving element by a filter or the like. Also,
It is preferable that the infrared light source for infrared image capturing by the LED or the like illuminates the observer with the same optical axis as the lens center of the infrared image capturing unit 12, and thus is incorporated in the infrared image capturing unit 12, for example. The infrared light is emitted coaxially with the center of the lens of the infrared image capturing section 12 by using an optical path switching means such as a half mirror. Further, the optical axes of the visible image capturing unit 11 and the infrared image capturing unit 12 may be coaxially arranged by an optical path switching means such as a half mirror.

The viewpoint position detection unit 2 includes an infrared image photographing unit 12
The position of the observer's pupil is detected from the infrared image photographed in 1., and the viewpoint position is detected while following the movement of the observer using the obtained pupil position and the visible image photographed by the visible image photographing unit 11. To do. The viewpoint position information and the like obtained by the viewpoint position detection unit 2 is supplied to a display control unit (not shown). As will be described later, the display control unit controls the display content and / or the brightness of the stereoscopic image display unit 3 during the pupil position detection process, thereby realizing the speedup and the detection accuracy of the pupil position detection process. The display control unit also generates stereoscopic image data to be displayed on the stereoscopic image display unit 3.

The viewpoint position detection unit 2 can be composed of, for example, a general-purpose computer device capable of storing the image signal output by the photographing unit 1. Although it is possible to configure a computer apparatus similarly display control unit general purpose, it is preferred to have a circuit capable of performing graphics accelerator such as image processing speed. Viewpoint position detection unit 2,
It is also possible to configure the display control unit by the same computer device.

The stereoscopic image display section 3 displays the image data for display supplied from the display control section in accordance with the control signal also supplied from the display control section. In addition, in the present embodiment, the stereoscopic image display unit 3 is a direct-view stereoscopic display device having a configuration capable of expanding the viewing angle by following the viewpoint position of the observer. It may be a mask pattern moving type disclosed in Japanese Patent Laid-Open No. 10-232367, a mask pattern changing type disclosed in Japanese Patent Laid-Open No. 10-78563, or a display device of another system. The movement or change of these mask patterns is performed by the control unit included in the stereoscopic image display unit 3.

(Internal Structure) FIG. 2 is a block diagram showing the structure of the stereoscopic image display system shown in FIG. In FIG. 2, the same components as those in FIG. 1 are designated by the same reference numerals, and duplicated description will be omitted. As illustrated, the viewpoint position detection unit 2 includes a visible image storage unit 21, a pattern matching determination unit 22, a template creation unit 23, a pupil position detection processing unit 24, an infrared image storage unit 25, and infrared light. The light emission control unit 26 is included.

Visible image storage unit 21 and infrared image storage unit 2
5 is used as a means for storing the image data photographed by the corresponding photographing units 11 and 12. Even if the semiconductor memory such as RAM is used, a magnetic / optical storage device such as a magnetic disk or an optical disk is used. You may comprise.

The pattern matching determination unit 22 uses the template supplied from the template creation unit 23 to display the positional information of the region having the highest correlation with the template among the images stored in the visible image storage unit 21. 27 and the stereoscopic image display unit 3. In addition, when pattern matching fails, the infrared light emission control unit 26 is output to cause the infrared light emission unit 13 to emit light.

The template creation unit 23 uses the position information supplied from the pupil position detection processing unit 24 to extract the template for pattern matching used in the pattern matching determination unit 22 from the image data stored in the visible image storage unit 21. To create.

The pupil position detection processing unit 24 receives a signal notifying that the infrared light emission control unit 26 has emitted the infrared light emission unit 13, and stores an infrared image based on the signal information. The pupil position is detected from the infrared image stored in the unit 25, and the position information is supplied to the template creating unit 23. Further, the position information may be supplied to the stereoscopic image display unit 3 and the display control unit 27.

The infrared light emission control unit 26 controls the lighting of the infrared light emission unit 13 under the control of the control unit (not shown), the pupil position detection processing unit 24, and the pattern matching determination unit 22.

As described above, the display control section 27 generates display image data based on the observer's viewpoint position information and the like. Further, by controlling the brightness of the stereoscopic image display unit 3, it is possible to speed up the pupil position detection process and improve the detection accuracy. The respective components of the viewpoint position detection unit 2 and the display control unit 27 operate under the control of a control unit (not shown) that controls the entire system.

In the stereoscopic image display section 3, a control unit (not shown) moves or changes the mask pattern based on the observer's viewpoint position information supplied from the pattern matching determination section 22 and / or the pupil position detection section 24. By doing so, the viewing angle is moved in accordance with the movement of the observer, and the viewing angle at which stereoscopic viewing is possible is expanded.

In the above-mentioned structure, the pupil position detecting device according to the present invention comprises the infrared image photographing section 12 and the infrared light emitting section 1.
3, a pupil position detection unit 24, an infrared image storage unit 25, and an infrared light emission control unit 26, and the viewpoint position detection device has a configuration in which the imaging unit 1 and the viewpoint position detection unit 2 are combined. However, both the pupil position detecting device and the viewpoint position detecting device do not necessarily need the constituent elements included in the image capturing unit 1, and only the parts for detecting the pupil position and the viewpoint position from the acquired image are used as the pupil position detecting device and the viewpoint position, respectively. It may be a detection device.

(Operation During Infrared Image Acquisition) Next, the operation during infrared image acquisition in the stereoscopic image display system according to this embodiment will be described with reference to the flowchart shown in FIG. The acquisition of the infrared image is performed every time the observer changes, if the pupil position cannot be detected from the acquired infrared image, or if the viewpoint position detection process described later cannot follow the movement of the observer, This is performed when it becomes necessary to perform the viewpoint position detection again, such as when it is set to periodically acquire.

First, the screen brightness is controlled (step S
120). As described above, the human pupil expands and contracts according to the brightness, particularly the brightness around the gazing point. Therefore, the infrared light emission control unit 26 notifies the display control unit 27 of the fact that light will be emitted before the infrared light source built in the infrared image capturing unit 12 is turned on, and the display control unit 27 causes the stereoscopic image display unit 3 to operate. Controls to reduce the screen brightness of. As a result, the observer's pupil is enlarged, so that the size of the pupil (retina) reflection image in the infrared image acquired by the infrared image capturing unit 12 is also enlarged, and the pupil position detection accuracy is improved.

The screen brightness control of the stereoscopic image display section 3 by the display control section 27 will be described in more detail. The method of lowering the screen brightness is not particularly limited, but for example, 1) lowers the brightness of the stereoscopic image display unit 3) power source of the stereoscopic image display unit 3 (backlight when the stereoscopic image display unit 3 has a backlight) Turn off the power of the light) 3) Display black image data 4) If the stereoscopic image display unit 3 has a barrier mask and the pattern of the barrier mask is controllable, a method such as setting the barrier mask to black is used. Can be used. However, when the stereoscopic image display unit 3 uses a display element that is not a self-luminous type such as a liquid crystal display, a method other than 3) is preferable. It is also possible to use any combination of 1) to 4) and other methods.

The degree of lowering the luminance in method 1) will be described. The most simple example is to control the brightness to the lower limit value that is possible in the stereoscopic image display unit 3. However, in actuality, a sufficient effect can be obtained by decreasing the brightness to a certain degree according to the environment in which the stereoscopic image display system is placed. It is possible to obtain

Therefore, the brightness may be experimentally adjusted in advance, and the control content by the display control unit 27 may be set so as to reduce the screen brightness to a value at which a sufficient effect is obtained. The essence of the present invention is to acquire an infrared image in a state where the screen brightness obtained in this manner is lower than the screen brightness.

Returning to FIG. 3, when the screen brightness control processing in step S120 is completed, the display control unit 27 notifies the infrared light emission control unit 26 and a control unit (not shown) of that fact. In response to this, the infrared light emission control unit 26
Causes the infrared light emitting unit 13 to emit light (step S130)
At the same time, the pupil position detection processing unit 24 is notified that infrared light has been emitted.

Then, the infrared image photographed by the infrared image photographing section 12 is loaded into the infrared image storage section 25 (step S140), and the infrared light emitting section 13 is turned off. At this time, in order to perform template creation and pattern matching with high accuracy, it is preferable to capture the visible image captured by the visible image capturing unit 11 at the same frame timing in the visible image storage unit 21.

Since it takes a certain amount of time to expand the human pupil, the infrared light may be emitted after a few seconds from the end of the brightness control. In this case, even if the display control unit 27 adjusts the time for which the brightness control end notification is output, the time from when the infrared light emission control unit 26 receives the brightness control end notification until the infrared light emission unit 13 emits light. The time may be adjusted or both may be combined.

Next, the specific operation of the entire system shown in FIG. 2 will be described using the flowchart shown in FIG.

First, each part of the photographing unit 1, the viewpoint position detection unit 2, the stereoscopic image display unit 3, and the display control unit 27 is turned on and initialized (step S10). Next, it is checked whether a new observer has been detected.

For detection of a new observer, for example, a photographed image of the visible image acquisition unit 11 of the photographing unit 1 is processed and observed by detecting that a switch or the like provided on a chair on which the observer sits is switched from OFF to ON. The system automatically detects that a person has arrived at the observation position, or provides a switch that allows the observer to operate at the observation position so that the observer can turn on the switch. It is also possible to detect by the active action of.

Next, the infrared image acquisition process (steps S120 to S140) described with reference to FIG. 3 is performed (step S12). Next, the pupil position detection processing unit 24
Detects the pupil position from the infrared image captured in the infrared image storage unit 25 (step S13). The detection of the pupil position is basically performed by binarizing the image. However, the reflection image that appears on the left in the infrared image is the right eye of the observer, and the reflection image that appears on the right is the left eye of the observer. As shown in FIG. 5, in the infrared image, the pupil portion becomes much brighter than the other portions due to the retinal reflection of the near infrared light.

Further, since the reflection image due to retinal reflection is a partial area within the image, the position of the reflection image detected in step S13 is not the pinpoint coordinate value indicating one point within the image. The process of obtaining the pupil position, which is the pinpoint image coordinate value from the reflection image, may be appropriately performed before the template creation stage of pattern matching.

When the retina reflection image can be detected in step S14, it cannot be detected normally in step S15 because the observer is blinking or it is difficult to distinguish it from other reflection images such as the frames of glasses. In that case, the process returns to step S12 and the image is acquired again.

When the pupil position detection is normally completed in step S14, the screen brightness lowered in the infrared image acquisition processing performed in step S12 is returned to the preset normal brightness (step S15), and the template is displayed. The creation process of is started (step S16). Although any pattern matching template can be used in the system of the present embodiment, a case will be described here as an example where two child templates and one parent template are used. Each type of template will be further described with reference to FIGS. 6 (a) and 6 (b).

FIGS. 6A and 6B are views for explaining the child template and the parent template used in this embodiment, respectively. As shown in the figure, the two child templates are templates in which the viewpoint positions of the right eye and the left eye are set as base points (indicated by x in the figure), and the parent template includes the viewpoint positions of the right eye and the left eye. It is one template with a point as a base point. Here, the viewpoint position in the template is a coordinate indicating one point in the image.

In the present embodiment, the template creation process starts with the creation of the child template. The template creation unit 23 uses the pupil positions (coordinate values on the image) of the right eye and the left eye detected by the pupil position detection processing unit 24 from the infrared image, from the visible image stored in the visible image storage unit 21. , A child template 1 with the right eye pupil position as a base point, and a child template 2 with the left eye pupil position as a base point are created. The size of the child template is determined by the following equation according to the distance between the pupil positions of the right and left eyes. Distance between the average human right eye and left eye pupil positions: Measured distance between the right eye and left eye pupil positions = Average human eye and eyebrow size: Child template size here As the average value of the distance between the pupil positions and the size in which the eyes and the eyebrows enter, for example, a value obtained statistically can be used.

When the creation of the child template is completed, the template creating unit 23 creates the parent template. As described above, the parent template is a template that includes two pupil positions with the midpoint of the two pupil positions of the right eye and the left eye as the base point. The size of the parent template is determined from the following equation according to the distance between the positions of the right and left eye pupils. Average distance between the right and left eye pupil positions of a human: Measured distance between the right and left eye pupil positions = Size of an average human face: size of parent template When creating a child template alike,
A value obtained statistically can be used as the average value. The template created by the template creating unit 23 is supplied to the pattern matching determination unit 22.

When the templates have been created, pattern matching using these templates is performed, and the viewpoint position is detected by following the change in the viewpoint position of the observer (step S17). Specifically, pattern matching is performed using the visible image captured in the visible image storage unit 21 and the template created by the template creating unit 23.

The pattern matching determination unit 22 first performs pattern matching between the parent template and the visible image.
Pattern matching can be performed using, for example, a normalized correlation function. For pattern matching using the normalized correlation function, see, for example, "Matrox Imaging Library Version 5.1 User Guide (Matr
ox Imaging Library Version 5.1 User Guide) ”pages 154-155. The value obtained by the normalized correlation function is expressed as 0 to 100 (%), and 100% means perfect match. Further, the pattern matching is sometimes called template matching.

In the present embodiment, if a degree of correlation exceeding 85% is obtained, it is considered that the pattern matching has succeeded. In the pattern matching immediately after the template is created, since the image that is the source of the template and the image data to be subjected to the pattern matching are the same, basically, a correlation of approximately 100% should be obtained.

When the pattern matching result between the parent template and the visible image satisfies a predetermined degree of correlation, it is determined that the pattern matching using the parent template has succeeded, and the pattern matching determination unit 22 sets the search area for the viewpoint position. I do. That is, the left half in the parent template is the search area for the right eye viewpoint position, and the right half in the parent template is the search area for the left eye viewpoint position. Then, pattern matching between the child template and the visible image is performed based on the set search area. In this way, pattern matching is performed hierarchically, the search range is narrowed in stages, and the right and left eye viewpoint positions are constrained to prevent detection of an incorrect viewpoint position (not a failure), and to follow accurately. You can Further, while the pattern matching is successful, the pattern matching by the child template may be performed, and when the pattern matching is unsuccessful, the parent template and the child template may be used to perform the pattern matching hierarchically.

As a result of the pattern matching, when the maximum correlation value satisfies the predetermined correlation value, it is determined that the pattern matching is successful in step S18, and step S19 is performed.
Move to. On the other hand, if the maximum correlation value is less than the predetermined correlation value, it is determined that the pattern matching has failed, the process returns to step S12, and the infrared image acquisition operation is performed again.

When it is determined in step S18 that the pattern matching is successful, the pattern matching determination unit 22 finally obtains the viewpoint position information (viewpoint position coordinates on the image) to the display control unit 27 and the stereoscopic image display unit 3. Is output for each of the right eye and the left eye.

In step S19, the display controller 27
Generates display image data according to the viewpoint position of the observer based on the viewpoint position information and outputs the display image data to the stereoscopic image display unit 3. Further, in the stereoscopic image display unit 3, a control unit (not shown) moves the mask pattern in the horizontal direction or changes the mask pattern based on the viewpoint position information, so that the viewer can view the stereoscopic viewing angle. Follow the viewpoint position of.

That is, in the case of the format of FIG. 8 in which the stereoscopic image display unit 3 moves the mask pattern, the control unit 52 controls the actuator 53 for horizontally moving the mask pattern based on the viewpoint position information, and the stereoscopic image display unit. If 3 is the form of FIG. 9 for changing the mask pattern, the control unit 55
Changes the mask pattern based on the viewpoint position information.

In step S20, it is determined whether or not the system is to be terminated. If the system operation is not instructed, the visible image is captured from the visible image photographing section 11,
After storing in the visible image storage unit 21, the process returns to step S17.

After that, the pattern matching with respect to the visible image is continued, and if the pattern matching fails, a new infrared image is automatically acquired and the template is recreated. When the end process is instructed in step S20, a predetermined end process is performed (step S21), and the series of processes is ended.

[0079]

[Other Embodiments] In the above-described embodiments, when the pattern matching result is unsuccessful, the infrared image acquisition is performed again to perform the pattern matching with high accuracy and good followability. In an environment where the person to be measured does not move, the same effect can be expected by periodically recreating the template. In particular, when it is not desirable to repeatedly obtain the infrared image using the brightness control for the same observer, it is preferable to perform such processing.

In the brightness control process, when the screen brightness suddenly decreases, some viewers may be surprised or misunderstand that the stereoscopic image display unit 3 is out of order. Therefore, it is preferable to display a message to the observer that the screen brightness will decrease before the screen brightness decreases. At this time, if a message indicating that the user should not move so as to detect the viewpoint position is also displayed, it is possible to detect the pupil position more reliably in subsequent infrared image acquisition.

After displaying these messages, a stereoscopic image is displayed, and the observer operates a switch such as a remote controller at a position where the stereoscopic image can be correctly observed (stereoscopically viewed). Then, the brightness control process is started. By using the signal as a signal, the range in which the observer's pupil position exists in the acquired infrared image can be narrowed down, and detection can be performed with higher accuracy.

Similarly, when the user has difficulty in stereoscopic viewing, by operating this switch, the pupil position detecting operation starting from the infrared image acquiring operation can be redone. With such a configuration, the template is updated at an appropriate timing, it becomes possible to detect the viewpoint position with higher accuracy, and as a result, it is possible to realize a stereoscopic image display system having a wide stereoscopically visible range. .

When infrared light is emitted, the amount of light emission is changed according to the brightness of the image display unit 3 and the distance to the observer, or the pupil position detection from the infrared image fails and the process is redone. In this case, the light emission amount can be changed (strengthen or weakened) from the previous one. Such control of the light emission amount increases the probability of successful pupil position detection from the infrared image, and as a result leads to obtaining a highly accurate viewpoint position detection result.

Further, in the above-described embodiment, the case where the viewpoint position is detected by using the detection result of the pupil position detecting device according to the present invention has been described as an example. Can be used.

Further, the specific method described in the above embodiments, such as the pattern matching method and the template creation method, is not limited to those described in the embodiments, and is equally applicable. Needless to say, may be used.

In the above embodiment, the viewpoint position, which is the coordinate of the pinpoint, is output. However, as in the above embodiment, the finally obtained viewpoint position is displayed in the stereoscopic image display unit. If used for control,
Since the minimum control is possible if the center positions of the viewpoint positions of the right eye and the left eye are known, the center position of the template may be output to the stereoscopic image display unit 3.

Further, in the above-mentioned embodiment, only the case where the liquid crystal display device using the backlight is used as the stereoscopic image display section 3 has been described, but the backlight and the mask pattern are displayed by the self-luminous display device such as CRT. It is also possible to realize a function equivalent to the combination of.

Furthermore, as a means for lowering the screen brightness,
Although only the control of the stereoscopic image display unit 3 is disclosed, the same effect can be obtained even if a movable physical shading device is installed between the stereoscopic image display unit 3 and the observer. In this case, for example, a retractable black screen or the like may be arranged on the entire surface of the display device, and the screen of the display device may be covered by the screen when the infrared image is acquired.

Even when the present invention is applied to a system including a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), an apparatus including one device (for example, a copying machine, a facsimile). Device).

Further, an object of the present invention is to supply a storage medium (or recording medium) recording a program code of software for realizing the functions of the above-described embodiments to a system or apparatus, and to supply a computer of the system or apparatus ( Needless to say, this can also be achieved by the CPU or MPU) reading and executing the program code stored in the storage medium.

In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instructions of the program code. It goes without saying that a case where some or all of the actual processing is performed and the functions of the above-described embodiments are realized by the processing is also included.

Further, after the program code read from the storage medium is written in the memory provided in the function expansion card inserted into the computer or the function expansion unit connected to the computer, based on the instruction of the program code, Needless to say, this also includes a case where a CPU or the like included in the function expansion card or the function expansion unit performs a part or all of the actual processing and the processing realizes the functions of the above-described embodiments.

When the present invention is applied to the above storage medium, the storage medium stores the program code corresponding to the above-described flowchart (shown in FIG. 3 and / or FIG. 4).

[0094]

As described above, according to the present invention,
In a pupil position detection device that detects the pupil position of an observer using a retina reflection image that uses infrared light, the pupil position can be detected with high accuracy in a short time by lowering the brightness of the display device when acquiring an infrared image. Can be realized.

[Brief description of drawings]

FIG. 1 is a diagram showing an appearance of a stereoscopic image display system according to an embodiment of the present invention.

FIG. 2 is a block diagram showing a configuration of a stereoscopic image display system according to an embodiment of the present invention.

FIG. 3 is a flowchart illustrating an infrared image acquisition operation according to the embodiment of the present invention.

FIG. 4 is a flowchart illustrating an operation of the stereoscopic image display system according to the embodiment of the present invention.

FIG. 5 is a diagram showing an example of an image obtained by infrared light.

FIG. 6 is a diagram illustrating a configuration of a template used in the embodiment of the present invention.

FIG. 7 is a perspective view showing a configuration of a rear cross lenticular stereoscopic image display device.

FIG. 8 is a perspective view showing an example of a stereoscopic image display device that performs display control according to a viewpoint position.

FIG. 9 is a perspective view showing an example of a stereoscopic image display device that performs display control according to a viewpoint position.

─────────────────────────────────────────────────── ─── Continuation of front page (58) Fields surveyed (Int.Cl. ⁷ , DB name) H04N 13/00 A61B 3/113 G02B 27/22

Claims (32)

(57) [Claims] 1. A pupil position detecting device for detecting a pupil position of a measurement subject and outputting pupil position information, wherein a brightness control means for controlling screen brightness of a display means for presenting a predetermined image to the measurement subject. A light irradiating means for irradiating the person to be measured with light, an image acquiring means for acquiring an image of the person to be measured, and an image acquired by the image acquiring means to detect a pupil position of the person to be measured. A detecting unit that outputs the pupil position information, and after the screen brightness of the display unit is set to a predetermined value or less by using the brightness control unit, irradiation by the light irradiation unit and image acquisition are performed. Pupil position detecting device. 2. The pupil position detection according to claim 1, wherein the brightness control unit controls the screen brightness to be equal to or lower than the predetermined value by cutting off power supplied to at least a light source of the display unit. apparatus. 3. The pupil position detecting device according to claim 1, wherein the brightness control unit controls the screen brightness to be equal to or lower than the predetermined value by controlling the brightness of the display unit. 4. The brightness control means controls the screen brightness to be equal to or lower than the predetermined value by controlling image data displayed on the display means.
The described pupil position detecting device. 5. The display means has a barrier mask capable of controlling a pattern, and the brightness control means controls the screen brightness to be equal to or lower than the predetermined value by controlling the pattern of the barrier mask. Claim 1 characterized by
The described pupil position detecting device. 6. The brightness control means comprises : (1) the display hand.
Disconnecting the power supply to at least the light source of the stage, (2)
Controlling the brightness of the display means, (3) the display means
Control two or more of controlling the image data to be displayed
By controlling the brightness of the display means.
Therefore, the screen brightness is controlled below the predetermined value.
The pupil position detecting device according to claim 1, which is a characteristic. 7. The display means can control a pattern.
A barrier mask, wherein the brightness control means is (1)
Disconnecting the power supply to at least the light source of the display means,
(2) The brightness of the display means is controlled, (3) the display
Controlling the image data displayed on the means, (4) the flash
Control two or more of the mask patterns
By controlling the brightness of the display means in combination
Control the screen brightness below the predetermined value.
The pupil position detecting device according to claim 1. 8. The pupil position detecting device according to claim 1, wherein the display unit is a direct-view type stereoscopic image display device. 9. The light irradiating means irradiates infrared light,
The image acquisition means is characterized by acquiring an infrared image of the measured person.
The claim according to any one of claims 1 to 8.
Pupil position detection device. 10. The pupil position detecting device according to claim 1, a visible image acquiring unit that acquires a visible image of the measurement subject, and a visible image acquiring unit. From a visible image, template matching means for creating a template for pattern matching using the pupil position information, and using the template created by the template creating means, perform pattern matching with the visible image acquired by the visible image acquiring means. And a matching unit that detects the viewpoint position of the measurement subject and outputs the result as viewpoint position information. 11. The pupil position detection device according to claim 1, a visible image acquisition unit that acquires a visible image of the measurement subject, and a visible image acquired by the visible image acquisition unit. From the image, template matching means for creating a template for pattern matching using the pupil position information, and pattern matching with the visible image acquired by the visible image acquiring means using the template created by the template creating means, Matching means for detecting the viewpoint position of the measurement subject and outputting the detection result as viewpoint position information, and to create the template again by using the pupil position detecting device and the template creating means when a predetermined condition is satisfied. A viewpoint position detecting device, comprising: 12. The control means evaluates the detection result of the matching means, and when the detection is judged to have failed a predetermined number of times, the pupil position detecting device and the template creating means are used to re-create the template. The visual point position detection device according to claim 11, wherein the visual point position detection device is controlled so as to be created. 13. The viewpoint position according to claim 11, wherein the control unit controls the pupil position detection device and the template creation unit to create the template again at predetermined time intervals. Detection device. 14. The control means evaluates the detection result of the matching means, and when the detection is determined to have failed a predetermined number of times, or at a predetermined time, the pupil position detecting device and the template creating means are activated. The viewpoint position detection device according to claim 11, wherein the viewpoint position detection device is controlled so as to create the template again. 15. A stereoscopic image display system comprising the viewpoint position detection device according to claim 10 and a stereoscopic image display device controllable by the brightness control device, wherein the stereoscopic image display device emits light. A stereoscopic image display device having a mask pattern for controlling a path, wherein the position or pattern of the mask pattern is changed using the viewpoint position information received from the viewpoint position detection device. . 16. The stereoscopic image display system according to claim 15, further comprising image generation means for generating an image to be displayed on the stereoscopic image display means according to the viewpoint position information of the measurement subject. 17. A pupil position detecting method for detecting a pupil position of a measurement subject and outputting pupil position information, wherein a brightness control step for controlling screen brightness of display means for presenting a predetermined image to the measurement subject. A light irradiation step of irradiating the measured person with light, an image acquisition step of acquiring an image of the measured person, and an image acquired in the image acquisition step, detecting the pupil position of the measured person. A detection step of outputting the pupil position information, wherein the brightness control step reduces the screen brightness of the display unit to a predetermined value or less, and then the light irradiation step and the image acquisition step are performed. Pupil position detection method. 18. The pupil position detection according to claim 17, wherein the brightness control step controls the screen brightness to be equal to or lower than the predetermined value by cutting off a power supply to at least the light source of the display means. Method. 19. The pupil position detecting method according to claim 17, wherein the brightness control step controls the screen brightness to be equal to or lower than the predetermined value by controlling the brightness of the display means. 20. The pupil position detecting method according to claim 17, wherein the brightness control step controls the screen brightness to be equal to or lower than the predetermined value by controlling image data displayed on the display means. 21. The display means includes a barrier mask capable of controlling a pattern, and the brightness control step controls the screen brightness to be equal to or lower than the predetermined value by controlling the pattern of the barrier mask. 18. The method for detecting a pupil position according to claim 17, which is characterized in that. 22. The brightness control step comprises : (1)
Disconnecting the power supply to at least the light source of the display means,
(2) The brightness of the display means is controlled, (3) the display
Two or more of controlling the image data displayed on the means
And controlling the brightness of the display means.
To control the screen brightness below the predetermined value.
18. The method for detecting a pupil position according to claim 17, wherein: 23. The display means can control a pattern
A barrier mask, and the brightness control step,
(1) A power supply for at least the light source of the display means
Disconnecting, (2) controlling the brightness of the display means,
(3) controlling the image data displayed on the display means ,
(4) Two of controlling the pattern of the barrier mask
Controlling the brightness of the display means by combining two or more controls
By controlling the screen brightness below the predetermined value
18. The method for detecting the pupil position according to claim 17, wherein
Law. 24. The display device according to claim 17, wherein the display device is a direct-view type stereoscopic image display method.
The method for detecting a pupil position according to any one of 1. 25. The method according to claim 17, wherein the light irradiating step irradiates infrared light, and the image acquiring step acquires an infrared image of the measurement subject. The described pupil position detecting method. 26. The pupil position detecting method according to claim 17, further comprising a visible image acquisition step of acquiring a visible image of the measurement subject, and an acquisition step of the visible image acquisition step. From the visible image, a template creating step for creating a template for pattern matching using the pupil position information, and using the template created in the template creating step, the visible image and the pattern matching obtained by the visible image acquiring step And a matching step of detecting the viewpoint position of the person to be measured and outputting the result as the viewpoint position information. 27. The pupil position detecting method according to claim 17, further comprising a visible image acquisition step of acquiring a visible image of the measurement subject, and the visible image acquisition step. From the visible image, a template creating step of creating a template for pattern matching using the pupil position information, and pattern matching with the visible image acquired by the visible image acquiring step using the template created by the template creating step. A matching step of detecting the viewpoint position of the person to be measured and outputting the detection result as the viewpoint position information, and creating the template again by using the pupil position detecting method and the template creating step when a predetermined condition is satisfied. And a control step for controlling the The viewpoint position detection method. 28. The control step evaluates the detection result of the matching step, and when the detection is determined to have failed a predetermined number of times, the template is detected again using the pupil position detecting method and the template creating step. 28. The viewpoint position detecting method according to claim 27, wherein the viewpoint position detecting method is controlled so that the viewpoint position is created. 29. The viewpoint position according to claim 27, wherein the control step controls to recreate the template by using the pupil position detecting method and the template creating step at predetermined time intervals. Detection method. 30. The control step evaluates the detection result of the matching step, and when the detection is determined to have failed a predetermined number of times, or at a predetermined time, the pupil position detecting method and the template creating step are performed. 28. The viewpoint position detecting method according to claim 27, further comprising: controlling to generate the template again. 31. A computer-readable storage medium in which the pupil position detecting method according to claim 17 is stored as a computer-executable program. 32. A computer-readable storage medium storing the viewpoint position detection method according to claim 26 as a computer-executable program.