JP5071308B2 - Input/Output Devices - Google Patents

Description

The present invention relates to an input/output device that can be used, for example, in a touch panel and is capable of displaying images and sensing objects.

Conventionally, input/output devices have been proposed that can display images in three-dimensional space and allow input operations to be performed in response to the displayed images (hereinafter, this will be referred to as a spatial user interface) (for example, Patent Document 1). In Patent Document 1, a three-dimensional display using the so-called integral method is realized by arranging a microlens array on the display surface side of the display at a distance equal to its focal length. In this method, the number of pixels assigned to one microlens among the multiple pixels that make up the display corresponds to the resolution in the depth direction of the three-dimensional image, and the number of microlenses corresponds to the number of pixels in the two-dimensional plane of the three-dimensional image.

JP 2007-86931 A JP 2005-141102 A

In other words, in the method of Patent Document 1, the resolution in a two-dimensional plane of a three-dimensional image displayed in space is determined by the number of microlenses (lens pitch) in the microlens array. Therefore, it is restricted by the lens pitch, resulting in low resolution.

In response to this, an input/output device has been proposed that displays a two-dimensional image on a plane in space by arranging a microlens array on the display surface side of the display so that the display surface of the display and a plane in space are in an optically conjugate positional relationship (for example, Patent Document 2). With this configuration, the resolution of the image displayed on the plane in space can be adjusted by the pitch of the display pixels on the display and the imaging magnification of the microlenses, making it easier to display high-resolution images.

However, in the configuration of Patent Document 2, various position sensors that detect the position of a finger or other object are placed further outside (toward the viewer) than the image display surface in space, which creates the problem of increasing the thickness of the entire device.

The present invention was made in consideration of these problems, and its purpose is to provide an input/output device that is thin yet capable of realizing a spatial user interface.

The input/output device of the present invention includes a microlens array unit having a plurality of microlenses, a display element disposed on one side of the microlens array unit in each of the unit areas corresponding to each microlens and displaying an image based on an image signal, an imaging element disposed on the same plane as the display element and acquiring imaging data based on light from an object, a position detection unit that detects the position of the object based on the imaging data acquired by the imaging element, and an image processing unit that performs image processing on the imaging data. The imaging elements are disposed at at least two positions within the unit area and are disposed at the same positions between each of the unit areas, and the image processing unit generates a plurality of parallax image data by combining imaging data acquired from the imaging elements at the same positions between each of the unit areas among the plurality of imaging data .

In the input/output device of the present invention, when an image is displayed based on an image signal by the display elements arranged in each unit area corresponding to each microlens, the displayed image is focused on one plane in space by the microlens array section. Meanwhile, light from an object is collected by the microlens array section and then received by an imaging element arranged in the same plane as the display element, thereby obtaining imaging data of the object. Based on this imaging data, the position detection section detects the position of the object.

According to the input/output device of the present invention, a display element that displays an image based on an image signal is arranged on one side of the microlens array section for each unit area corresponding to each microlens, so that the observer can recognize the image displayed on the display surface of the display element as if it were displayed on a plane in space. On the other hand, an imaging element that obtains imaging data of an object based on light collected by the microlens array section is arranged on the same plane as the display element, and the position of the object is detected based on the imaging data obtained by this imaging element. Therefore, the thickness of the entire device can be made smaller than when, for example, a separate position sensor is provided on the other side (observer side) of the microlens array. This makes it possible to realize a spatial user interface while still being thin.

The following describes in detail the embodiments of the present invention with reference to the drawings.

Figure 1 shows a functional block configuration of an input/output device 1 according to one embodiment of the present invention. The input/output device 1 is used, for example, as a touch panel, and is capable of displaying images such as figures and characters, as well as detecting the position (including movement) of a finger or the like. The input/output device 1 includes an input/output panel 10, a display signal driver 21, a display-side scanner 22, an image signal receiver 31, an image-capturing-side scanner 32, an image processing unit 40, a position detection unit 50, and a control unit 60.

The input/output panel 10 is made up of a plurality of pixels arranged two-dimensionally, with a microlens array 12 (described below) installed on the display side, i.e., the observer side. The input/output panel 10 has a mixture of display pixels and imaging pixels arranged so as to realize both an image display function for displaying images and an imaging function for capturing images of objects. The detailed configuration of this input/output panel 10 will be described later.

The display signal driver 21 and the display side scanner 22 drive each display pixel based on the display signal (image data D0) under the control of the control unit 60. The display signal driver 21 drives each display pixel to be driven, and the display side scanner 22 selects the display pixel to be driven.

The imaging signal receiver 31 and the imaging side scanner 32 receive light from the object side at the imaging pixels and drive each imaging pixel to obtain imaging data D1 of the object in response to control from the control unit 60. The imaging signal receiver 31 drives the imaging pixels to be driven for imaging, and the imaging side scanner 32 selects the imaging pixels to be driven for imaging.

The image processing unit 40 performs predetermined image processing on the imaging data D1 supplied from the imaging signal receiver 31 in response to control from the control unit 60. The detailed image processing operation of this image processing unit 40 will be described later.

The position detection unit 50 detects the position (x, y, z) of the finger based on the image processing data D2 input from the image processing unit 40 in response to control from the control unit 60. Information on the finger position detected by the position detection unit 50 is output as position data Dout.

The control unit 60 controls the operations of the display signal driver 21, the display side scanner 22, the imaging side scanner 32, the imaging signal receiver 31, the image processing unit 40, and the position detection unit 50. Specifically, it appropriately controls the display drive operation by the display signal driver 21 and the display side scanner 22, the imaging drive operation by the imaging signal receiver 31 and the imaging side scanner 32, the image processing operation by the image processing unit 40, and the position detection operation by the position detection unit 60. The image processing unit 40, the position detection unit 50, and the control unit 60 are each composed of, for example, a microcomputer.

Next, the detailed configuration of the input/output panel 10 will be described with reference to Figures 2 and 3. Figure 2 shows a schematic cross-sectional configuration of the input/output panel 10. Figure 3 shows a schematic planar configuration of the display unit 11.

The input/output panel 10 has a microlens array 12 installed on the display side of the display unit 11. The microlens array 12 has a plurality of microlenses 12a arranged in a matrix. This microlens array 12 forms a two-dimensional image, which is displayed by the display area 11A described below, on a plane S1 in space, and also functions as an imaging lens for imaging the finger 100. Due to this microlens array 12, the plane S1 in space and the display surface of the display area 11A are optically conjugate.

The display unit 11 has a display area 11A made up of display pixels Pa and imaging areas 11B1 and 11B2 made up of imaging pixels Pb arranged on a drive substrate 111. The drive substrate 111 drives the display pixels Pa and imaging pixels Pb, and is composed of, for example, TFTs (Thin Film Transistors). A backlight 110 for irradiating light toward the display area 11A is provided below the drive substrate 111. The backlight 110 is composed of, for example, CCFLs (Cold Cathode Fluorescent Lamps) and LEDs (Light Emitting Diodes). The display area 11A (display pixels Pa) and the backlight 110 correspond to the display element of the present invention, and the imaging areas 11B1 and 11B2 (imaging pixels Pb) correspond to the imaging element of the present invention.

The display area 11A and the imaging areas 11B1 and 11B2 are arranged in the same plane (in the focal plane of the microlens 12a) for each unit area U1 corresponding to the microlens 12a of the microlens array 12. In this embodiment, as shown in FIG. 3, the display area 11A is provided in the center of the unit area U1, and the imaging areas 11B1 and 11B2 are provided at a distance from each other at the ends of the unit area U1 along the X direction. The display area 11A is composed of, for example, 3×3=9 display pixels Pa, and the imaging areas 11B1 and 11B2 are each composed of one imaging pixel Pb. Furthermore, the imaging areas 11B1 and 11B2 are arranged in the same positions between the unit areas U1.

The display pixel Pa is a display element that displays an image based on the image data D0. This display pixel Pa is, for example, configured with a liquid crystal display element (LCD: Liquid Crystal Display). The liquid crystal display element is configured to include, for example, a liquid crystal cell in which a liquid crystal layer is sealed between a pair of substrates, a pair of polarizing plates attached to the light entrance side and the light exit side of the liquid crystal cell, and a color filter that selectively transmits light of a specific wavelength range from the light emitted from the liquid crystal cell (all not shown). In this configuration, the light emitted from the backlight 110 is modulated and transmitted by applying a voltage between the pair of substrates of the liquid crystal cell.

The imaging pixel Pb acquires imaging data D1 based on the light collected by each microlens of the microlens array 12. The imaging pixel Pb is composed of, for example, a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

Next, we will explain the operation and effects of the input/output device 1.

(Image display operation)
First, the image display operation (output operation) of the input/output device 1 will be described with reference to Fig. 1 and Fig. 4. Fig. 4 is a schematic diagram for explaining the image display operation on one plane in space.

In the input/output device 1, when image data D0 is supplied to the display signal driver 21 under the control of the control unit 60, the display pixels Pa are driven, and an image _I0 is displayed on the display surface S0 in the display area 11A. The image _I0 displayed on the display surface S0 is imaged on a plane S1 at a height H from the microlens array 12 for each unit area U1 by the microlens 12a. However, since the display surface S0 and the plane S1 are in a conjugate relationship by the microlens array 12, the image _I0 is displayed on the display surface S0 as an inverted image of the image _I1 . This allows the observer to recognize it as if the image _I1 is displayed at a position of height H. At this time, the height H of the plane S1 is determined by the imaging magnification of the microlens 12a. The resolution of the image _I1 on the plane S1 is determined by the number (pitch) of the display pixels Pa in the display area 11A and the imaging magnification of the microlens 12a. For example, when the pitch of the display pixels Pa is about 0.1 mm and the imaging magnification of the microlens 12a is 1:2, a highly precise pixel pitch of about 0.2 mm is realized in the image _I1 .

(Sensing operation)
Next, the sensing operation (input operation) of the input/output device 1 will be described with reference to Fig. 1 and Figs. 5 to 8. Fig. 5 is a schematic diagram for explaining the parallax image generating operation in the image processing unit 40. Fig. 6 shows an actual captured image of a finger and a parallax image generated from the captured image. Fig. 7 is a diagram for explaining the position detection operation. Fig. 8 shows a captured image of a stylus.

In the input/output device 1, when a finger 100 is placed above the input/output panel 10, the finger 100 is illuminated by light from the image _I1 described above (hereinafter referred to as image light). The light illuminating the finger 100 is collected by the microlens array 12 in the input/output panel 10 and received by the imaging regions 11B1 and 11B2 (imaging pixels Pb). As a result, imaging data of the finger 100 is acquired and output to the imaging signal receiver 31. The imaging data output to the imaging signal receiver 31 is supplied to the image processing unit 40 as imaging data D1.

Here, the imaging regions 11B1 and 11B2 in each unit region U1 have different directions of travel of the received light rays, and the light is received while retaining information about the travel direction. Furthermore, the imaging pixel data in the imaging regions 11B1 and 11B2 that are arranged at the same positions between the unit regions U1 contain information about the same travel direction.

In the image processing unit 40, a parallax image generation process is performed based on the input imaging data D1, and two parallax image data D11, D12 for the finger 100 are generated. Specifically, first, in the imaging data D1, imaging pixel data in the imaging area 11B1 (pixel data in the black parts in FIG. 5(A)) is extracted for each unit area U1, and then these extracted imaging pixel data are combined (FIG. 5(B)). Similarly, imaging pixel data in the imaging area 11B2 (pixel data in the diagonal lines in FIG. 5(A)) is extracted for each unit area U1, and then these extracted imaging pixel data are combined (FIG. 5(C)). As a result, two parallax image data D11, D12 are generated.

The parallax image data D11 and D12 generated as described above are subjected to other image processing, such as defect correction processing and noise reduction processing, as necessary, and output to the position detection unit 50 as image processing data D2.

In the position detection unit 50, the position (x, y, z) of the finger 100 is detected based on the input image processing data D2. For example, the position (x, y) of the finger 100 is detected by performing edge detection processing on the parallax image data (parallax image data D11 or parallax image data D12) of one piece of the finger 100. On the other hand, the z component of the position of the finger 100 is specified by, for example, the following method. That is, based on the correlation between the parallax image data D11 and D12, the phase difference of the biological image in the image of two parallaxes is calculated, and the z component of the finger 100 is specified based on the phase difference. Here, Figs. 7(A) to (C) show left and right parallax images when the distance from the microlens array 12 to the finger 100 is 0 mm, 5 mm, and 10 mm, respectively. In this way, the phase difference (Δδ ₁ , Δδ ₂ , Δδ ₃ ) of the finger 100 between the left and right parallax images is detected by, for example, edge detection processing. At this time, as the height of the finger 100 decreases, that is, as the finger 100 approaches the cover glass 11, the phase difference between the two parallax images of the finger 100 decreases (Δδ ₃ > Δδ ₂ > Δδ ₁ ). Therefore, the z component of the finger 100 is specified by detecting the phase difference of the finger 100 based on the correlation between the left and right parallax images. This also makes it possible to detect the spatial positional relationship between the plane S1 on which the image _I0 is displayed and the finger 100, for example, the state of z=H, that is, the state in which the finger 100 is in pseudo-contact with the plane S1. Thus, a spatial user interface can be realized.

Movement of the finger 100 in the horizontal plane (XY plane) is detected as follows. For example, multiple pieces of parallax image data of the finger 100 are generated over time, and edge detection processing is performed on each of the multiple pieces of parallax image data to calculate the amount of movement of the finger 100, thereby detecting the movement of the finger 100.

In this way, information about the position (x, y, z) of the finger 100 detected by the position detection unit 50 is output as position data Dout2. By setting in advance the function operation to be executed according to the position of the finger 100, the function is executed according to the output position data Dout. Note that the position data Dout may be input to the display signal driver 21 as the above-mentioned image data D0.

As described above, in this embodiment, the display area 11A consisting of the display pixels Pa is arranged for each unit area U1 corresponding to each microlens, so that the image _I0 displayed by this display area 11A based on the image data D0 can be focused as the image _I1 on the plane S1 in space by the microlens array 12. This allows the observer to recognize as if the image is being displayed on the plane S1 in space. Meanwhile, the light from the finger 100 illuminated by the image light of this image _I1 is collected by the microlens array 12 and received by the imaging areas 11B1 and 11B2 arranged in the same plane as the display area 11A, so that the imaging data D1 of the finger 100 can be acquired. Based on this imaging data D1, the image processing unit 40 generates parallax image data D11, D12 of the finger 100, and based on the parallax image data D11, D12, the position detection unit 50 detects the position (x, y, z) of the finger 100, so there is no need to provide a separate position sensor, for example, above the microlens array 12 (on the observer side). Therefore, it is possible to realize a spatial user interface while being thin.

In the above embodiment, the case where the position of a finger 100 is detected has been described as an example, but the object of position detection is not limited to a finger, and may be other parts of a living body or a non-living object such as a stylus. FIG. 8 shows two parallax images generated based on image data of a stylus obtained. In this case, too, the position (x, y, z) of the stylus can be detected by detecting the phase difference Δδs between the two parallax images using the above-mentioned method. Therefore, input operations using a stylus instead of a finger are also possible.

(Application example)
9A and 9B show a schematic configuration of a mobile phone 2 according to an application example of the above embodiment. The mobile phone 2 has a folding housing (a first housing 23 and a second housing 24), and the first housing 23 is provided with the input/output device 1 according to the above embodiment. Meanwhile, the second housing 24 is provided with an operation unit 25 for performing input operations.

In this application example, as shown in FIG. 9(A), by mounting the input/output device 1, the input/output device 1 can function as, for example, a touch panel in which input is made according to the position of the user's finger 100. Specifically, by detecting the z component of the position of the finger 100, input by a click action is possible. Furthermore, by measuring the frequency of contact of the finger 100 over time, input by a double-click action is also possible. In addition, by detecting the movement of the finger 100, i.e., the change over time in the position (x, y), it is also possible to make the input/output device 1 function as a mouse pointer.

Function input can also be performed to execute login, password substitution, payment processing, and the like. In other words, since the input/output device 1 is capable of distinguishing fingers, function input is possible by pre-registering specific processes such as those described above to be executed according to the order in which fingers are detected. For example, it is possible to register a process such as "open address book" to be executed when the index finger of the right hand and the middle finger of the right hand are detected in that order.

Also, as shown in FIG. 9(B), input is not limited to a finger 100, but can also be performed using a stylus 101.

The input/output device 1 can be used in a variety of mobile devices, such as notebook PCs (personal computers), and is not limited to the above-mentioned mobile phones.

Next, a modified example of the above embodiment will be described. In the following, components similar to those of the input/output device 1 and mobile phone 2 of the above embodiment will be given the same reference numerals, and descriptions thereof will be omitted as appropriate.

(Variation 1)
10 is a schematic diagram showing a planar configuration of a display unit 26 of an input/output device according to a first modified example of the above embodiment. In this modified example, the configuration is the same as that of the input/output panel 10 of the above embodiment, except for the configuration of the display pixels Pa (display area). That is, the display pixels Pa are arranged over almost the entire surface of the drive substrate 111, and the imaging areas 11B1 and 11B2 are arranged at the ends of each unit area U1 along the X direction. With this configuration, an image is displayed over the entire area of the multiple display pixels Pa.

As in this modified example, the arrangement and number of display pixels Pa in the display unit 26 are not particularly limited, and the display pixels Pa may be arranged over almost the entire surface of the drive substrate 111. Even in this configuration, an image can be displayed on a single plane in space by appropriately adjusting the imaging magnification of the microlenses 12a.

(Variation 2)
11 is a functional block diagram showing the overall configuration of an input/output device 3 according to the second modification of the above embodiment. In this modification, the focal length of a microlens array (not shown) in an input/output panel 10 is variable according to an applied voltage, and an MLA drive control unit 70 is provided for applying a voltage to the microlens array. The other configurations are the same as those of the input/output device 1 according to the above embodiment.

The microlens array of this modified example has multiple microlenses arranged in a matrix. Each microlens is, for example, a liquid crystal microlens or a liquid microlens. The MLA drive control unit 70 applies a voltage to each microlens to drive it according to the control of the control unit 60, thereby making it possible to change the focal length of each microlens.

In this configuration, by changing the focal length of each microlens in the microlens array, the height H of the image displayed in space can be changed within the image plane. In other words, a distribution in the height (depth) direction can be formed in the image displayed in space, which allows the viewer to recognize the display as being closer to three-dimensional. Note that the above-mentioned height change may be formed for each microlens, or may be formed for each region in the microlens array.

(Variation 3)
FIG. 12 shows a schematic configuration of a mobile phone 4 according to the third modification of the embodiment. The mobile phone 4 has light sources 27 arranged on both sides of the input/output device 1 in the first housing 23. The light sources 27 are, for example, LEDs, and function as light sources for illuminating the finger 100 when acquiring the image data D1 of the finger 100. As in this modification, the light source for illuminating the finger 100 may be provided separately from the backlight 110. Even in this case, the light source 27 only needs to be arranged on the same plane as the upper surface (microlens array) of the input/output device 1, so the thickness of the input/output device 1 and the mobile phone 4 itself does not increase. Therefore, the same effect as the above embodiment can be obtained.

Although the present invention has been described above with reference to the embodiment and modified examples, the present invention is not limited to these embodiments and various modifications are possible. For example, in the above embodiment, the image processing unit 40 generates two pieces of parallax image data based on the imaging data D1, but the generated parallax image data is not limited to two and may be three or more. In addition, the pixel data extracted from the imaging data D1 may be pixel data for any pixel in the unit region U1. However, it is desirable to extract pixel data for pixels arranged in regions that provide the longest possible baseline length from the left and right directions. As described above, this is to detect the position of the finger 100 based on the phase difference between the parallax images.

In addition, in the above embodiment, two imaging areas 11B1, 11B2 are arranged in unit area U1, and each of these imaging areas 11B1, 11B2 is composed of one imaging pixel Pb. However, the arrangement and number of imaging areas are not limited to this. One imaging area may be composed of multiple imaging pixels Pb. However, each imaging area (imaging pixel) is arranged at the same position between multiple unit areas U1. Similarly, the arrangement and number of display areas, and the number of display pixels Pa that make up the display area are not particularly limited.

In addition, in the above embodiments, the display unit (display section) has been described as having imaging pixels that contribute to sensing and display pixels that contribute to display arranged only in a part of the unit area or all over the unit area, but the configuration of the display section is not limited to this. For example, there may be imaging pixels that do not actually contribute to sensing and display pixels that do not contribute to display, i.e., dummy pixels. In addition, such dummy pixel areas may be configured to have only one or two of the imaging pixels, display pixels, and microlenses formed.

Furthermore, in the above embodiment, a liquid crystal display element has been described as an example of a display element, but other display elements, for example, self-luminous elements such as organic or inorganic EL (Electro Luminescence) elements, may also be used. However, when using the above-mentioned self-luminous elements, a backlight does not necessarily have to be provided.

1 is a functional block diagram illustrating an overall configuration of an input/output device according to an embodiment of the present invention. 2 is a schematic cross-sectional view illustrating a configuration of a main part of the input/output device illustrated in FIG. 1 . FIG. 2 is a schematic diagram showing a two-dimensional configuration of the display unit in FIG. 1 . 2 is a schematic diagram for explaining an image display operation of the input/output device shown in FIG. 1 . 2 is a schematic diagram for explaining a parallax image generating operation of the image processing unit shown in FIG. 1 . (A) is an actual captured image, and (B) and (C) are parallax images. 2 is a schematic diagram for explaining a position detection operation of the position detection unit shown in FIG. 1 . FIG. 1 is a parallax image of a stylus. 2 is a perspective view illustrating a schematic configuration of a mobile phone according to an application example of the input/output device illustrated in FIG. 1. 11 is a schematic diagram illustrating a two-dimensional configuration of a display unit of an input/output device according to a first modified example. FIG. FIG. 11 is a functional block diagram illustrating an overall configuration of an input/output device according to a second modified example. FIG. 11 is a perspective view illustrating a schematic configuration of a mobile phone according to a third modified example.

Explanation of symbols

1, 3...input/output device, 2, 4...mobile phone, 10...input/output panel, 11, 26...display unit, 11A...display area, 11B1, 11B2...imaging area, 12...microlens array, 13, 25...imaging element, 40...image processing unit, 50...position detection unit, 60...control unit, 110...backlight, 111...driving board, 100...finger, U1...unit area, Pa...display pixel, Pb...imaging pixel, _I0 , _I1 ...image, D0...image data, D1...imaging data, D2...image processing data, Dout...position data.

Claims (7)

a microlens array portion having a plurality of microlenses;
a display element disposed on one side of the microlens array in each unit area corresponding to each microlens, the display element displaying an image based on image data;
an image sensor that is disposed on the same plane as the display device and that acquires image data based on light from an object;
a position detection unit that detects a position of the object based on the imaging data acquired by the imaging element ;
an image processing unit that performs image processing on the imaging data,
the imaging elements are disposed at at least two positions within the unit area and at the same positions between the unit areas;
The image processing unit is an input/output device that generates a plurality of pieces of parallax image data by synthesizing imaging data obtained from imaging elements at the same positions between each unit area among a plurality of pieces of imaging data . The position detection unit is
The input/output device according to claim 1 , wherein the position of the object is detected based on a phase difference between the plurality of pieces of parallax image data. The input/output device according to claim 1 , wherein the imaging elements are arranged at positions spaced apart from each other in each unit area. The focal length of the microlens is variable in response to an applied voltage;
The input/output device according to claim 1 , further comprising a microlens driving unit that applies a voltage to the microlens. The input/output device according to claim 4 , wherein the microlens is a liquid crystal microlens or a liquid microlens. The input/output device according to claim 1 , which functions as a touch panel for receiving an input according to the position detected by the position detection unit. The input/output device according to claim 1 , wherein the microlens array section makes the display surface of the display element optically conjugate with a plane on the other side of the microlens array section.

Images