JP7742469B2 - Space-floating image display device - Google Patents

Description

The present invention relates to a space-floating image display device.

As a spatial floating information display system, image display devices that display images directly to the outside and display methods that display images as a spatial screen are already known. Furthermore, a detection system that reduces false detections of operations on the operation surface of a displayed spatial image is also disclosed, for example, in Patent Document 1.

Japanese Patent Application Laid-Open No. 2019-128722

However, touch operations on a floating image in space are not performed on a physical button, touch panel, etc. Therefore, there are cases where the user cannot recognize whether or not a touch operation has been performed.
Therefore, an object of the present invention is to provide a more suitable space floating image display device.

In order to solve the above problem, for example, the configuration described in the claims is adopted. The present application includes multiple means for solving the above problem, but to cite one example, a space-floating image display device includes a display device that displays an image, a retroreflective member that reflects image light from the display device and forms a space-floating image in the air using the reflected light, a sensor that detects the position of a user's finger that performs a touch operation on one or more objects displayed in the space-floating image, and a control unit. The control unit controls image processing for the image displayed on the display device based on the position of the user's finger detected using the sensor, and the space-floating image display device displays a virtual shadow of the user's finger on the display surface of the space-floating image, which has no physical contact surface. The position of the virtual shadow displayed on the display surface of the floating in space image is determined from the positional relationship between the position of the virtual light source and the position of the user's finger detected using the sensor, and the position of the virtual light source is set at an infinite distance from the display surface of the floating in space image. When the position of the tip of the user's finger approaches the display surface of the floating in space image so that the normal distance to the display surface of the floating in space image becomes small on the front side of the display surface of the floating in space image as seen from the user, the virtual shadow of the user's finger continues to be displayed on the display surface of the floating in space image until the normal distance between the position of the tip of the user's finger and the display surface of the floating in space image becomes zero.

This invention makes it possible to realize a more suitable floating-in-space image display device. Other issues, configurations, and advantages will be made clear in the description of the following embodiments.

1 is a diagram showing an example of a usage form of a space floating image display device according to an embodiment of the present invention; 1 is a diagram showing an example of a main part configuration and a retroreflection part configuration of a space floating image display device according to an embodiment of the present invention; 10A and 10B are diagrams illustrating an example of a method for installing a space floating image display device. 10A and 10B are diagrams showing other examples of installation methods for the space floating image display device. FIG. 1 is a diagram illustrating an example of the configuration of a space floating image display device. FIG. 10 is a diagram showing another example of the main part configuration of the space floating image display device according to an embodiment of the present invention. FIG. 10 is an explanatory diagram for explaining the function of a sensing device used in the space floating image display device. 1 is an explanatory diagram of the principle of three-dimensional image display used in a space floating image display device. FIG. 1 is an explanatory diagram of a measurement system for evaluating the characteristics of a reflective polarizing plate. FIG. 10 is a characteristic diagram showing the transmittance characteristics of the transmission axis of a reflective polarizing plate versus the angle of incidence of light. FIG. 10 is a characteristic diagram showing the transmittance characteristics of the reflection axis of a reflective polarizing plate versus the angle of incidence of light. FIG. 10 is a characteristic diagram showing the transmittance characteristics of the transmission axis of a reflective polarizing plate versus the angle of incidence of light. FIG. 10 is a characteristic diagram showing the transmittance characteristics of the reflection axis of a reflective polarizing plate versus the angle of incidence of light. FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. 1 is a layout diagram showing a main part of a space floating image display device according to an embodiment of the present invention; 1 is a cross-sectional view showing a configuration of a display device according to an embodiment of the present invention. FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. FIG. 2 is a cross-sectional view showing an example of a specific configuration of a light source device. 10A and 10B are explanatory diagrams for explaining the light source diffusion characteristics of the image display device. FIG. 10 is an explanatory diagram for explaining the diffusion characteristics of a video display device. FIG. 10 is an explanatory diagram for explaining the diffusion characteristics of a video display device. 1 is a cross-sectional view showing a configuration of a video display device. 1 is an explanatory diagram for explaining the principle of occurrence of a ghost image in the prior art; 1 is a cross-sectional view showing a configuration of a display device according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a display on a display device according to an embodiment of the present invention. 10A and 10B are diagrams illustrating an example of a method for assisting a touch operation using a virtual shadow. 10A and 10B are diagrams illustrating an example of a method for assisting a touch operation using a virtual shadow. 10A and 10B are diagrams illustrating an example of a method for assisting a touch operation using a virtual shadow. 10A and 10B are diagrams illustrating another example of a method for assisting a touch operation using a virtual shadow. 10A and 10B are diagrams illustrating another example of a method for assisting a touch operation using a virtual shadow. 10A and 10B are diagrams illustrating another example of a method for assisting a touch operation using a virtual shadow. FIG. 10 is a diagram illustrating a method for setting a virtual light source. FIG. 10 is a configuration diagram showing an example of a method for detecting the position of a finger. FIG. 10 is a configuration diagram showing another example of a method for detecting the position of a finger. FIG. 10 is a configuration diagram showing another example of a method for detecting the position of a finger. 10A and 10B are diagrams illustrating a method of assisting a touch operation by displaying input contents. 10A and 10B are diagrams illustrating a method of highlighting input content to assist touch operations. 10A and 10B are diagrams illustrating an example of a method for assisting a touch operation by vibration. 10A and 10B are diagrams illustrating another example of a method for assisting a touch operation by vibration. 10A and 10B are diagrams illustrating another example of a method for assisting a touch operation by vibration.

Embodiments of the present invention will be described in detail below with reference to the drawings. It should be noted that the present invention is not limited to the description of the embodiments, and various changes and modifications may be made by those skilled in the art within the scope of the technical concepts disclosed in this specification. Furthermore, in all drawings used to explain the present invention, parts having the same function will be given the same reference numerals, and repeated explanations may be omitted.

The following examples relate to an image display device that can transmit an image generated by image light from an image light source through a transparent member that separates a space, such as glass, and display the image as a floating image outside the transparent member.

The following embodiments enable the realization of image display devices suitable for, for example, bank ATMs, train station ticket machines, digital signage, and the like. For example, while touch panels are currently commonly used in bank ATMs and train station ticket machines, a transparent glass surface or optically transparent plate can be used to display high-resolution image information floating in space on the glass surface or optically transparent plate. By narrowing the divergence angle of the emitted image light, i.e., making it acute, and further aligning it with a specific polarization, only the normal reflected light is efficiently reflected by the retroreflective material, resulting in high light utilization efficiency and suppressing the ghost images that occur in addition to the main floating image, a problem with conventional retroreflective systems, resulting in a clear floating image. Furthermore, a device including the light source of this embodiment can provide a novel, highly usable floating image display device (floating image display system) that can significantly reduce power consumption. Furthermore, a floating image display device for vehicles can be provided that can display a unidirectional floating image that can be viewed both inside and/or outside the vehicle.

On the other hand, in conventional technology, an organic EL panel or a liquid crystal panel is combined with a retroreflective member 151 as a high-resolution color display image source 150. In conventional technology, image light is diffused over a wide angle, so in addition to the light reflected normally by the retroreflective member 151, ghost images 301 and 302 are generated by image light incident obliquely on the retroreflective member 2a as shown in FIG. 24, impairing the image quality of the floating image in space. Furthermore, as shown in FIG. 23, in addition to the normal floating image in space 300, multiple ghost images such as a first ghost image 301 and a second ghost image 302 are generated. As a result, the same floating image in space, which is a ghost image, can be observed by people other than the observer, posing a major security issue.
<Space-floating image display device 1>

Figure 1 is a diagram showing an example of how a space-floating image display device according to one embodiment of the present invention is used, and is a diagram showing the overall configuration of the space-floating image display device according to this embodiment. The specific configuration of the space-floating image display device will be described in detail using Figure 2, etc., but light with a narrow-angle directional characteristic and specific polarization is emitted from image display device 1 as an image beam, first enters retroreflective member 2, and then retroreflects and passes through transparent member 100 (glass, etc.), forming a real aerial image (space-floating image 3) on the outside of the glass surface.

In addition, in stores and other spaces, spaces are divided by show windows (also called "window glass") 105, which are made of a translucent material such as glass. The space-floating image display device of this embodiment makes it possible to transmit floating images through such transparent materials and display them in one direction toward the outside and/or inside of the store (space).

In Figure 1(A), the inside of the window glass 105 (inside the store) is shown in the depth direction, with the outside (e.g., the sidewalk) in the foreground. On the other hand, by providing a means for reflecting specific polarized waves on the window glass 105, it is possible to reflect the waves and form an aerial image at a desired position inside the store.

Figure 1(B) is a schematic block diagram showing the configuration of the above-mentioned video display device 1. The video display device 1 includes a video display unit that displays the original aerial image, a video control unit that converts the input video to match the resolution of the panel, and a video signal receiving unit that receives the video signal. The video signal receiving unit supports wired input signals such as HDMI (High-Definition Multimedia Interface) input, as well as wireless input signals such as Wi-Fi (Wireless Fidelity). It can function as a standalone video receiver and display device and can also display video information from tablets, smartphones, etc. Furthermore, by connecting a stick PC or similar device, it can be equipped with capabilities such as calculation processing and video analysis processing.

Figure 2 is a diagram showing an example of the main components and retroreflection section of a space-floating image display device according to one embodiment of the present invention. The configuration of the space-floating image display device will be explained in more detail using Figure 2. As shown in Figure 2(A), a display device 1 that diverges specific polarized image light at a narrow angle is provided in the diagonal direction of a transparent member 100 such as glass. The display device 1 includes a liquid crystal display panel 11 and a light source device 13 that generates specific polarized light with narrow-angle diffusion characteristics.

Image light of a specific polarization from the display device 1 is reflected by a polarization separation member 101 (shown in the figure as a sheet-like polarization separation member 101 attached to the transparent member 100) that has a film that selectively reflects image light of a specific polarization and is provided on the transparent member 100, and then enters the retroreflective member 2. A λ/4 plate 21 is provided on the image light incident surface of the retroreflective member. The image light is polarized and converted from the specific polarization to the other polarization by passing through the λ/4 plate 21 twice, once upon entering the retroreflective member and once upon exiting. Here, the polarization separation member 101, which selectively reflects image light of a specific polarization, has the property of transmitting the polarized light of the other polarization that has been polarized and converted, so the image light of the specific polarization after polarization conversion passes through the polarization separation member 101. The image light that passes through the polarization separation member 101 forms a real image, a floating image 3, outside the transparent member 100.

The light that forms the floating image 3 is a collection of light rays that converge from the retroreflective member 2 onto the optical image of the floating image 3, and these light rays continue to travel in a straight line even after passing through the optical image of the floating image 3. Therefore, the floating image 3 is a highly directional image, unlike the diffuse image light formed on a screen by a typical projector. Therefore, in the configuration of Figure 2, when a user views the floating image 3 from the direction of arrow A, the floating image 3 appears as a bright image. However, when viewed by another person from the direction of arrow B, the floating image 3 cannot be seen as an image at all. This characteristic is highly suitable for use in systems that display images that require high security or highly confidential images that should be kept secret from people directly facing the user.

Depending on the performance of the retroreflective member 2, the polarization axis of the reflected image light may become misaligned. In this case, some of the image light with misaligned polarization axes is reflected by the polarization separation member 101 described above and returns to the display device 1. This light may be re-reflected by the image display surface of the liquid crystal display panel 11 that constitutes the display device 1, generating ghost images and potentially degrading the image quality of the spatially floating image. Therefore, in this embodiment, an absorbing polarizer 12 is provided on the image display surface of the display device 1. The image light emitted from the display device 1 is transmitted through the absorbing polarizer 12, and the reflected light returning from the polarization separation member 101 is absorbed by the absorbing polarizer 12, thereby suppressing the re-reflection. This prevents degradation of image quality due to ghost images of the spatially floating image.

The polarization separation member 101 described above may be formed, for example, from a reflective polarizing plate or a metal multilayer film that reflects specific polarized light.

Next, Figure 2(B) shows the surface shape of a typical retroreflective member 2 manufactured by Nippon Carbide Industries Co., Ltd., used in this study. Light rays incident on the regularly arranged hexagonal prisms are reflected by the walls and bottom of the hexagonal prisms and exit as retroreflected light in a direction corresponding to the incident light, resulting in a real, floating image based on the image displayed on the display device 1. The resolution of this floating image depends not only on the resolution of the LCD panel 11, but also on the outer diameter D and pitch P of the retroreflective portion of the retroreflective member 2 shown in Figure 2(B). For example, when using a 7-inch WUXGA (1920 x 1200 pixels) LCD panel, even if one pixel (one triplet) is approximately 80 μm, if the diameter D of the retroreflective portion is 240 μm and the pitch is 300 μm, then one pixel of the floating image will be equivalent to 300 μm. As a result, the effective resolution of the floating image is reduced to approximately one-third. Therefore, in order to make the resolution of the spatial floating image equivalent to that of the display device 1, it is desirable to make the diameter and pitch of the retroreflective portion close to that of one pixel of the liquid crystal display panel. On the other hand, to prevent moire from occurring due to the retroreflective material and the pixels of the liquid crystal display panel, it is advisable to design the pitch ratio of each to be a different integer multiple of one pixel. Furthermore, it is advisable to arrange the shape so that none of the sides of the retroreflective portion overlaps with any of the sides of one pixel of the liquid crystal display panel.

On the other hand, to manufacture retroreflective members at low cost, it is recommended to use a roll press method. Specifically, this method aligns the retroreflective portions and forms them on a film. The reverse shape of the shape to be formed is formed on the surface of a roll, and a UV-curable resin is applied to a fixing base material and passed between the rolls to form the required shape. The resin is then irradiated with UV light to cure, resulting in the retroreflective member 2 of the desired shape.
<<How to install the space floating image display device>>

Next, we will explain how to install the space-floating image display device. The installation method for the space-floating image display device can be freely changed depending on the usage form. Figure 3A is a diagram showing an example of how to install a space-floating image display device. The space-floating image display device shown in Figure 3A is installed horizontally so that the surface on which the space-floating image 3 is formed faces upward. That is, in Figure 3A, the space-floating image display device is installed so that the transparent member 100 faces upward, and the space-floating image 3 is formed above the space-floating image display device.

3B is a diagram showing another example of a method for installing a space-floating image display device. The space-floating image display device shown in FIG. 3B is installed vertically so that the surface on which the space-floating image 3 is formed faces sideways (toward the user 230). That is, in FIG. 3B, the space-floating image display device is installed so that the transparent member 100 faces sideways, and the space-floating image 3 is formed on the side of the space-floating image display device (toward the user 230).
<<Configuration of the Space Floating Image Display Device>>

Next, we will explain the configuration of the space-floating image display device 1000. Figure 3C is a block diagram showing an example of the internal configuration of the space-floating image display device 1000.

The space floating image display device 1 includes a retroreflector 1101, an image display unit 1102, a light guide 1104, a light source 1105, a power source 1106, an operation input unit 1107, a non-volatile memory 1108, a memory 1109, a control unit 1110, an image signal input unit 1131, an audio signal input unit 1133, a communication unit 1132, an aerial operation detection sensor 1351, an aerial operation detection unit 1350, an audio output unit 1140, an image control unit 1160, a storage unit 1170, an imaging unit 1180, etc.

All components of the space floating image display device 1000 are housed in a housing 1190. Note that the imaging unit 1180 and mid-air operation detection sensor 1351 shown in Figure 3C may be provided outside the housing 1190.

The retroreflective portion 1101 in Figure 3C corresponds to the retroreflective member 2 in Figure 2. The retroreflective portion 1101 retroreflects light modulated by the image display portion 1102. Of the light reflected from the retroreflective portion 1101, the light output to the outside of the space-floating image display device 1000 forms the space-floating image 3.

The image display unit 1102 in FIG. 3C corresponds to the liquid crystal display panel 11 in FIG. 2. The light source 1105 in FIG. 3C corresponds to the light source device 13 in FIG. 2. The image display unit 1102, light guide 1104, and light source 1105 in FIG. 3C correspond to the display device 1 in FIG. 2.

Video display unit 1102 is a display unit that generates an image by modulating transmitted light based on a video signal input under the control of video control unit 1160 (described below). Video display unit 1102 corresponds to liquid crystal display panel 11 in FIG. 2. For example, a transmissive liquid crystal panel is used as video display unit 1102. Alternatively, for example, a reflective liquid crystal panel that modulates reflected light or a DMD (Digital Micromirror Device: registered trademark) panel may be used as video display unit 1102.

Light source 1105 generates light for image display unit 1102 and is a solid-state light source such as an LED light source or laser light source. Power supply 1106 converts AC current input from the outside into DC current and supplies power to light source 1105. Power supply 1106 also supplies the necessary DC current to each component within the space floating image display device 1000.

The light guide 1104 guides the light generated by the light source 1105 and irradiates it onto the image display unit 1102. The combination of the light guide 1104 and the light source 1105 can also be referred to as the backlight of the image display unit 1102. There are various possible combinations of the light guide 1104 and the light source 1105. Specific configuration examples of combinations of the light guide 1104 and the light source 1105 will be described in detail later.

The aerial operation detection sensor 1351 is a sensor that detects operations on the floating in space image 3 by the user's 230 fingers. The aerial operation detection sensor 1351 senses, for example, the range that overlaps with the entire display range of the floating in space image 3. Note that the aerial operation detection sensor 1351 may also sense only the range that overlaps with at least a portion of the display range of the floating in space image 3.

Specific examples of the aerial operation detection sensor 1351 include distance sensors that use invisible light such as infrared, invisible lasers, ultrasonic waves, etc. The aerial operation detection sensor 1351 may also be configured to combine multiple sensors and detect coordinates on a two-dimensional plane. The aerial operation detection sensor 1351 may also be configured as a ToF (Time of Flight) LiDAR (Light Detection and Ranging) sensor or an image sensor.

The mid-air operation detection sensor 1351 only needs to be capable of sensing to detect touch operations, etc., made by the user with their finger on an object displayed as the floating image 3. Such sensing can be performed using existing technology.

The aerial operation detection unit 1350 acquires a sensing signal from the aerial operation detection sensor 1351, and based on the sensing signal, determines whether or not the user's 230 finger has made contact with an object in the floating-in-space image 3, and calculates the position (contact position) where the user's 230 finger has made contact with the object. The aerial operation detection unit 1350 is configured, for example, with a circuit such as an FPGA (Field Programmable Gate Array). Furthermore, some of the functions of the aerial operation detection unit 1350 may be realized in software by, for example, a spatial operation detection program executed by the control unit 1110.

The aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be configured to be built into the space-floating image display device 1000, or may be provided externally as a separate entity from the space-floating image display device 1000. When provided as a separate entity from the space-floating image display device 1000, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 are configured to be able to transmit information and signals to the space-floating image display device 1000 via a wired or wireless communication connection path or video signal transmission path.

Also, the aerial operation detection sensor 1351 and the aerial operation detection unit 1350 may be provided separately. This makes it possible to build a system in which the space-floating image display device 1000, which does not have aerial operation detection functionality, is the main body, and the aerial operation detection functionality can be added as an option. Alternatively, the aerial operation detection sensor 1351 may be provided separately, and the aerial operation detection unit 1350 may be built into the space-floating image display device 1000. In cases where it is desired to more freely position the aerial operation detection sensor 1351 relative to the installation position of the space-floating image display device 1000, a configuration in which only the aerial operation detection sensor 1351 is provided separately is advantageous.

The imaging unit 1180 is a camera with an image sensor, and captures images of the space near the floating-in-space image 3 and/or the face, arms, fingers, etc. of the user 230. Multiple imaging units 1180 may be provided. Using multiple imaging units 1180, or an imaging unit with a depth sensor, can assist the mid-air operation detection unit 1350 in detecting touch operations on the floating-in-space image 3 by the user 230.

For example, if the aerial operation detection sensor 1351 is configured as an object intrusion sensor that targets a plane (intrusion detection plane) including the display surface of the floating image 3 and detects whether or not an object has intruded into this intrusion detection plane, the aerial operation detection sensor 1351 may not be able to detect information such as how far an object that has not intruded into the intrusion detection plane (for example, a user's finger) is from the intrusion detection plane, or how close the object is to the intrusion detection plane.

In such cases, the distance between the object and the intrusion detection plane can be calculated using information such as object depth calculation information based on images captured by multiple imaging units 1180 and object depth information from a depth sensor. This information, as well as various other information such as the distance between the object and the intrusion detection plane, is then used for various display controls for the Floating in Space Image 3.

Alternatively, without using the mid-air operation detection sensor 1351, the mid-air operation detection unit 1350 may detect touch operations on the floating-in-space image 3 by the user 230 based on the captured image by the imaging unit 1180.

The imaging unit 1180 may also capture an image of the face of the user 230 operating the space-floating image 3, and the control unit 1110 may perform an identification process for the user 230. Furthermore, in order to determine whether or not there is another person standing around or behind the user 230 operating the space-floating image 3 and peeking at the user 230's operation of the space-floating image 3, the imaging unit 1180 may capture an image of the user 230 operating the space-floating image 3 and an area including the user 230's surrounding area.

The operation input unit 1107 is, for example, an operation button or a light receiving unit of a remote controller, and inputs signals for operations other than the aerial operation (touch operation) by the user 230. In addition to the aforementioned user 230 touching the space floating image 3, the operation input unit 1107 may also be used by, for example, an administrator to operate the space floating image display device 1000.

The video signal input unit 1131 connects to an external video output device and inputs video data. The audio signal input unit 1133 connects to an external audio output device and inputs audio data. The audio output unit 1140 is capable of outputting audio based on the audio data input to the audio signal input unit 1133. The audio output unit 1140 may also output built-in operation sounds and error warning sounds.

Non-volatile memory 1108 stores various data used by the space floating image display device 1000. Data stored in non-volatile memory 1108 includes, for example, data for various operations to be displayed on the space floating image 3, display icons, object data for user operation, layout information, etc. Memory 1109 stores image data to be displayed as the space floating image 3, data for controlling the device, etc.

The control unit 1110 controls the operation of each connected unit. The control unit 1110 may also work in conjunction with a program stored in the memory 1109 to perform calculations based on information obtained from each unit within the space floating image display device 1000. The communication unit 1132 communicates with external devices, external servers, etc. via a wired or wireless interface. Various types of data such as video data, image data, and audio data are sent and received through communication via the communication unit 1132.

The storage unit 1170 is a storage device that records various types of data and information, such as video data, image data, and audio data. For example, the storage unit 1170 may be pre-recorded with various types of information, such as video data, image data, and audio data, at the time of product shipment. The storage unit 1170 may also record various types of information, such as video data, image data, and audio data, acquired from external devices, external servers, etc. via the communication unit 1132.

The video data, image data, etc. recorded in the storage unit 1170 is output as the space floating image 3 via the video display unit 1102 and the retroreflection unit 1101. The video data, image data, etc. of the display icons and objects for the user to operate, etc., displayed as the space floating image 3, are also recorded in the storage unit 1170.

Layout information for display icons and objects displayed as the floating image 3, as well as various metadata information related to the objects, is also recorded in the storage unit 1170. Audio data recorded in the storage unit 1170 is output as audio from the audio output unit 1140, for example.

The video control unit 1160 performs various controls related to the video signal input to the video display unit 1102. For example, the video control unit 1160 controls video switching, such as which video signal is input to the video display unit 1102 between the video signal stored in the memory 1109 and the video signal (video data) input to the video signal input unit 1131.

The video control unit 1160 may also generate a superimposed video signal by superimposing the video signal to be stored in the memory 1109 and the video signal input from the video signal input unit 1131, and input the superimposed video signal to the video display unit 1102, thereby controlling the formation of a composite video as the floating-in-space video 3.

The video control unit 1160 may also control image processing of video signals input from the video signal input unit 1131 and video signals to be stored in the memory 1109. Examples of image processing include scaling processing to enlarge, reduce, or deform an image, brightness adjustment processing to change the brightness, contrast adjustment processing to change the contrast curve of an image, and Retinex processing to decompose an image into light components and change the weighting of each component.

The video control unit 1160 may also perform special effect video processing, etc., on the video signal input to the video display unit 1102 to assist the user 230 in performing an aerial operation (touch operation). The special effect video processing is performed, for example, based on the detection result of the user 230's touch operation by the aerial operation detection unit 1350 and the image of the user 230 captured by the imaging unit 1180.

As explained above, various functions are installed in the space floating image display device 1000. However, the space floating image display device 1000 does not need to have all of these functions, and any configuration is acceptable as long as it has the function of forming the space floating image 3.
<Space-floating image display device 2>

Figure 4 shows another example of the main components of a space-floating image display device according to one embodiment of the present invention. The image display element 11 constituting the display device 1 includes a light source device 13 that generates light of a specific polarization with diffusion characteristics that form a narrow angle with the liquid crystal display panel 11. The display device 1 is configured with an LCD panel ranging in screen size from a small one of approximately 5 inches to a large one exceeding 80 inches. A polarization separation member 101 such as a reflective polarizer is provided on the surface of the folding mirror 22, which reflects the image light from the LCD panel 11 toward the retroreflective member 2. The image light of the specific polarization from the display device 1 is reflected by a film (sheet 101 is adhesively attached in the figure) provided on a transparent member 100 that selectively reflects the image light of the specific polarization, and enters the retroreflective member 2.

A λ/4 plate 21 is provided on the light incident surface of the retroreflective member. The image light is polarized and converted from a specific polarization to the other polarization by passing through the polarization separation member 101 twice, resulting in a real image, the floating-in-space image 3, being displayed on the outside of the transparent member 100. An absorptive polarizer is provided on the external light incident surface of the transparent member 100. Because the polarization separation member 101 described above causes the polarization axis to become misaligned due to retroreflection, some of the image light is reflected and returned to the display device 1. This light is then reflected again by the image display surface of the liquid crystal display panel 11 that constitutes the display device 1, generating ghost images and significantly degrading the image quality of the floating-in-space image. Therefore, in this embodiment, an absorptive polarizer 12 is provided on the image display surface of the display device 1, which transmits the image light and absorbs the reflected light, thereby preventing image quality degradation due to ghost images of the floating-in-space image. Furthermore, an absorptive polarizer 12 is preferably provided on the surface of the transparent member 100 to mitigate image quality degradation caused by sunlight and lighting outside the set. The polarization separation member 101 is formed from a reflective polarizing plate or a metal multilayer film that reflects specific polarized waves.

Next, sensors 44 with TOF (Time of Fly) functionality are arranged in multiple layers as shown in Figure 5 to sense the distance and position relationship between the object and the sensors 44 in relation to the floating image obtained by the floating image display device described above. This makes it possible to sense not only the object's coordinates in the plane direction, but also its coordinates in the depth direction and the object's movement direction and speed. To read two-dimensional distance and position, multiple combinations of ultraviolet light emitters and light receivers are arranged in a straight line, and light from the light-emitting point is irradiated onto the object, and the reflected light is received by the light-receiving unit. The distance to the object is determined by the product of the difference between the time of light emission and the time of light reception and the speed of light. Furthermore, coordinates on the plane can be read from the coordinates at the point where the difference between the time of light emission and the time of light reception is smallest using multiple light-emitting units and light-receiving units. As described above, three-dimensional coordinate information can also be obtained by combining the object's coordinates in the plane (two dimensions) and multiple sensors described above.

Furthermore, a method for obtaining a three-dimensional space-floating image using the above-mentioned space-floating image display device will be described with reference to FIG. 6. FIG. 6 is an explanatory diagram of the principle of three-dimensional image display used in the space-floating image display device. A horizontal lenticular lens is arranged to match the pixels of the image display screen of the liquid crystal display panel 11 of the display device 1 shown in FIG. 4. As a result, to display motion parallax from three directions, P1, P2, and P3, in the horizontal direction of the screen as shown in FIG. 6, images from the three directions are grouped into blocks of three pixels, and image information from the three directions is displayed for each pixel. The light emission direction is controlled by the action of the corresponding lenticular lens (shown by vertical lines in FIG. 6) to separate and emit the light in three directions. As a result, a three-parallax stereoscopic image can be displayed.
<Reflective polarizing plate>

In the space-floating image display device of this embodiment, the polarization separation element 101 is used to improve the contrast performance, which determines the image quality, compared to a typical half mirror. The characteristics of a reflective polarizer are explained as an example of the polarization separation element 101 of this embodiment. Figure 7 is an explanatory diagram of a measurement system used to evaluate the characteristics of a reflective polarizer. The transmission and reflection characteristics of the reflective polarizer in Figure 7 relative to the angle of incidence of light perpendicular to the polarization axis are shown in Figures 8 and 9, respectively, as V-AOI. Similarly, the transmission and reflection characteristics of the reflective polarizer relative to the angle of incidence of light horizontal to the polarization axis are shown in Figures 10 and 11, respectively, as H-AOI.

As shown in Figures 8 and 9, a reflective polarizer with a grid structure exhibits reduced characteristics for light coming from a direction perpendicular to the polarization axis. For this reason, specifications aligned with the polarization axis are desirable, and the light source of this embodiment, which can emit the image light from the liquid crystal display panel at a narrow angle, is an ideal light source. Similarly, the horizontal characteristics also deteriorate for light coming from an oblique angle. Taking these characteristics into consideration, the following describes a configuration example of this embodiment, in which a light source capable of emitting the image light from the liquid crystal display panel at a narrower angle is used as the backlight for the liquid crystal display panel. This makes it possible to provide a high-contrast floating image.
<Display device>

Next, the display device 1 of this embodiment will be described using the drawings. The display device 1 of this embodiment includes an image display element 11 (liquid crystal display panel) and a light source device 13 that constitutes its light source. Figure 12 shows the light source device 13 together with the liquid crystal display panel as an exploded perspective view.

As shown by arrow 30 in Figure 12, this liquid crystal display panel (image display element 11) receives light from the light source device 13, which serves as a backlight device, and produces an illumination light beam with narrow-angle diffusion characteristics. This illumination light beam has strong directionality (linearity) and a polarization plane aligned in one direction, similar to that of laser light. The image light is modulated according to the input video signal, reflected by the retroreflective member 2, and transmitted through the transparent member 100 to form a real, floating image (see Figure 1). Figure 12 also shows the liquid crystal display panel 11, which constitutes the display device 1, and further includes a light direction conversion panel 54 that controls the directional characteristics of the light beam emitted from the light source device 13, and an optional narrow-angle diffusion plate (not shown). Polarizing plates are provided on both sides of the liquid crystal display panel 11, and the image light of a specific polarization is emitted with its intensity modulated according to the video signal (see arrow 30 in Figure 12). As a result, the desired image is projected as highly directional (straight-line) light of a specific polarization via the light direction conversion panel 54 toward the retroreflective member 2, where it is reflected and transmitted toward the eyes of an observer outside the store (space), forming a floating image 3. A protective cover 50 (see Figures 13 and 14) may be provided on the surface of the light direction conversion panel 54 described above.

In this embodiment, in order to improve the utilization efficiency of the light beam 30 emitted from the light source device 13 and significantly reduce power consumption, in a display device 1 including the light source device 13 and the liquid crystal display panel 11, light from the light source device 13 (see arrow 30 in FIG. 12 ) is projected toward the retroreflective member 2, and after reflection by the retroreflective member 2, a transparent sheet (not shown) provided on the surface of the transparent member 100 (such as the window glass 105) can be used to control the directivity so that a floating image is formed at a desired position. Specifically, this transparent sheet controls the imaging position of the floating image while providing high directivity using optical components such as a Fresnel lens or a linear Fresnel lens. In this way, the image light from the display device 1 can efficiently reach an observer outside the display window 105 (e.g., on a sidewalk) with high directivity (straightness) like laser light. As a result, it is possible to display high-quality floating images at high resolution and significantly reduce power consumption by the display device 1 including the LED elements 201 of the light source device 13.
<Example 1 of Display Device>

Figure 13 shows an example of a specific configuration of the display device 1. In Figure 13, a liquid crystal display panel 11 and a light direction conversion panel 54 are arranged on top of the light source device 13 of Figure 12. This light source device 13 is formed, for example, from plastic on the case shown in Figure 12, and is configured by housing LED elements 201 and a light guide 203 inside. As also shown in Figure 12, the end surface of the light guide 203 has a shape whose cross-sectional area gradually increases toward the side facing the light receiving section in order to convert the divergent light from each LED element 201 into a substantially parallel beam of light. The end surface has a lens shape that gradually reduces the divergence angle by multiple total reflections as the light propagates inside. The liquid crystal display panel 11 that constitutes the display device 1 is attached to its top surface. Furthermore, LED (Light Emitting Diode) elements 201, which are semiconductor light sources, and an LED board 202, which mounts their control circuits, are attached to one side of the case of the light source device 13 (the left end face in this example), and a heat sink, which is a member for cooling the heat generated by the LED elements and control circuit, may be attached to the outer surface of the LED board 202.

The LCD panel frame (not shown) attached to the top surface of the case of the light source device 13 is configured to include the LCD panel 11 attached to the frame, as well as an FPC (Flexible Printed Circuit) (not shown) electrically connected to the LCD panel. That is, the LCD panel 11, which is an LCD display element, generates a display image by modulating the intensity of transmitted light in conjunction with the LED elements 201, which are solid-state light sources, based on control signals from a control circuit (not shown) that constitutes the electronic device. The generated image light has a narrow diffusion angle and contains only specific polarization components, resulting in a novel image display device similar to a surface-emitting laser image source driven by a video signal. Currently, it is technically and safety-wise impossible to obtain a laser beam of the same size as the image obtained by the display device 1 described above using a laser device. Therefore, in this embodiment, light similar to the surface-emitting laser image light described above is obtained from a beam of light from a general light source, such as an LED element.

Next, the configuration of the optical system housed within the case of the light source device 13 will be described in detail with reference to Figures 13 and 14.

13 and 14 are cross-sectional views, and only one of the LED elements 201 making up the light source is shown. This is converted into approximately collimated light by the shape of the light-receiving end surface 203a of the light guide 203. For this reason, the light-receiving portion of the light guide end surface and the LED element are attached while maintaining a predetermined positional relationship. Each light guide 203 is formed from a translucent resin such as acrylic. The LED light-receiving surface at the end of this light guide has, for example, a convex convex outer surface obtained by rotating a parabolic cross section, and at its apex, a concave portion with a convex portion (i.e., a convex lens surface) formed in its central portion, and the central portion of the flat portion has a convex lens surface protruding outward (or may be a concave lens surface recessed inward) (not shown). The outer shape of the light receiving section of the light guide on which the LED element 201 is mounted is a parabolic shape that forms a conical outer surface, and is set within an angle range that allows for total reflection within the light emitted from the LED element in the peripheral direction, or a reflective surface is formed.

On the other hand, the LED elements 201 are each arranged at a predetermined position on the surface of the LED board 202, which is the circuit board. This LED board 202 is positioned and fixed relative to the LED collimator (light-receiving end surface 203a) so that the LED elements 201 on its surface are each located in the center of the recess mentioned above.

With this configuration, the shape of the light-receiving end surface 203a of the light guide 203 makes it possible to extract the light emitted from the LED element 201 as approximately parallel light, thereby improving the utilization efficiency of the generated light.

As described above, the light source device 13 is configured by attaching a light source unit having an array of LED elements 201 serving as light sources to the light-receiving end surface 203a, which is the light-receiving portion provided on the end surface of the light guide 203. The divergent light beam from the LED elements is converted into approximately parallel light by the lens shape of the light-receiving end surface 203a of the light guide end surface, and as shown by the arrow, this light is guided inside the light guide 203 (in a direction parallel to the drawing), and is emitted by the light beam direction conversion means 204 toward the liquid crystal display panel 11, which is positioned approximately parallel to the light guide (in a direction perpendicular to the front of the drawing). The uniformity of the light beam incident on the liquid crystal display panel 11 can be controlled by optimizing the distribution (density) of this light beam direction conversion means depending on the shape inside or on the surface of the light guide. The light beam direction conversion means 204 described above emits the light beam propagated within the light guide toward the liquid crystal display panel 11, which is positioned approximately parallel to the light guide (in a direction perpendicular to the front of the drawing), by changing the shape of the light guide surface or by providing areas with different refractive indices within the light guide. In this case, when the liquid crystal display panel 11 is faced directly at the center of the screen and the viewpoint is positioned at a position equal to the diagonal dimension of the screen, if the relative brightness ratio between the center and periphery of the screen is 20% or more, there is no practical problem, and if it exceeds 30%, it will be an even better characteristic.

Figure 13 is a cross-sectional layout diagram illustrating the configuration and operation of the polarization-converting light source of this embodiment in the light source device 13 including the light guide 203 and LED element 201 described above. In Figure 13, the light source device 13 is composed of a light guide 203 made of plastic or the like, with a light beam direction conversion means 204 on its surface or inside, an LED element 201 as a light source, a reflective sheet 205, a retardation plate 206, a lenticular lens, etc., and attached to the top surface is a liquid crystal display panel 11 equipped with polarizing plates on the light source light entrance surface and the image light exit surface.

A film or sheet-like reflective polarizer 49 is provided on the light source light incidence surface (bottom surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, selectively reflecting one polarized wave (e.g., P wave) 212 of the natural light beam 210 emitted from the LED element 201, which is then reflected by a reflective sheet 205 provided on one surface (bottom surface in the figure) of the light guide 203, and directed back toward the liquid crystal display panel 52. Therefore, a retardation plate (λ/4 plate) is provided between the reflective sheet 205 and the light guide 203, or between the light guide 203 and the reflective polarizer 49, and the reflected light beam is reflected by the reflective sheet 205 and passed twice, converting it from P-polarized light to S-polarized light, improving the utilization efficiency of the light source light as image light. The image light beam, whose light intensity has been modulated by the video signal on the liquid crystal display panel 11 (arrow 213 in Figure 13), enters the retroreflective member 2, and after reflection, as shown in Figure 1, passes through the window glass 105, creating a real, floating image inside or outside the store (space).

Similar to FIG. 13, FIG. 14 is a cross-sectional layout diagram illustrating the configuration and operation of the polarization-converting light source of this embodiment in a light source device 13 including a light guide 203 and an LED element 201. Similarly, the light source device 13 is composed of a light guide 203 formed of, for example, plastic, on the surface of which or inside which a light beam direction conversion means 204 is provided, an LED element 201 as a light source, a reflective sheet 205, a retardation plate 206, a lenticular lens, etc., and on top of this is attached a liquid crystal display panel 11 as an image display element, which has polarizing plates on the light source light entrance surface and the image light exit surface.

A film or sheet-like reflective polarizer 49 is provided on the light source light incidence surface (bottom surface in the figure) of the liquid crystal display panel 11 corresponding to the light source device 13, selectively reflecting one polarized wave (e.g., S wave) 211 of the natural light beam 210 emitted from the LED light source 201, which is then reflected by a reflective sheet 205 provided on one surface (bottom surface in the figure) of the light guide 203 and directed back toward the liquid crystal display panel 11. A retardation plate (λ/4 plate) is provided between the reflective sheet 205 and the light guide 203 or between the light guide 203 and the reflective polarizer 49, and the reflected light beam is reflected by the reflective sheet 205 and passes through it twice, converting the reflected light beam from S-polarized to P-polarized, improving the efficiency of use of the light source light as image light. The image light beam (arrow 214 in Figure 14) that has been intensity modulated by the video signal on the liquid crystal display panel 11 enters the retroreflective member 2, and after reflection, as shown in Figure 1, passes through the window glass 105, creating a real, floating image inside or outside the store (space).

In the light source devices shown in Figures 13 and 14, in addition to the function of the polarizer provided on the light incident surface of the corresponding liquid crystal display panel 11, the reflective polarizer reflects the polarized light component on one side. Therefore, the theoretically obtainable contrast ratio is the reciprocal of the cross transmittance of the reflective polarizer multiplied by the reciprocal of the cross transmittance obtained by the two polarizers attached to the liquid crystal display panel. This results in high contrast performance. In fact, experiments have confirmed that the contrast performance of the displayed image is improved by more than 10 times. As a result, high-quality images comparable to those of self-luminous organic EL displays are obtained.
<Example 2 of Display Device>

Figure 15 shows another example of the specific configuration of the display device 1. The light source device 13 in Figure 15 is similar to the light source device in Figure 17, etc. This light source device 13 is configured by housing LEDs, a collimator, a composite diffusion block, a light guide, etc. in a case made of, for example, plastic, and has a liquid crystal display panel 11 attached to its upper surface. Also, an LED board 102 mounting LED (Light Emitting Diode) elements 14a and 14b, which are semiconductor light sources, and their control circuits is attached to one side of the case of the light source device 13, and a heat sink 103, which is a member for cooling the heat generated by the LED elements and control circuit, is attached to the outer surface of the LED board 102 (see also Figures 17, 18, etc.).

The liquid crystal display panel frame attached to the top surface of the case is configured to have attached thereto the liquid crystal display panel 11 attached to the frame, and further to have attached thereto an FPC (Flexible Printed Circuits) 403 (see FIG. 7 ) electrically connected to the liquid crystal display panel 11. That is, the liquid crystal display panel 11, which is a liquid crystal display element, generates a display image by modulating the intensity of transmitted light together with the LED elements 14 a and 14 b, which are solid-state light sources, based on a control signal from a control circuit (not shown here) that constitutes the electronic device.
<Display Device Example 3>

Next, another example of the specific configuration of the display device 1 will be described using Figure 16. The light source device of this display device 1 converts the divergent light beam of natural light (a mixture of P-polarized and S-polarized waves) from the LED into a substantially parallel light beam using the LED collimator 18, which is then reflected toward the liquid crystal display panel 11 by the reflective light guide 304. The reflected light is incident on a wave plate and reflective polarizer 49 arranged between the liquid crystal display panel 11 and the reflective light guide 304. A specific polarized wave (e.g., S-polarized wave) is reflected by the reflective polarizer, its phase is converted by the wave plate, it returns to the reflecting surface, passes through the retardation plate again, and is converted into a polarized wave (e.g., P-polarized wave) that transmits through the reflective polarizer.

As a result, the natural light from the LEDs is aligned to a specific polarization (e.g., P polarization) and enters the LCD panel 11. It is then brightness-modulated in accordance with the video signal, displaying an image on the panel surface. As with the previous example, multiple LEDs constituting the light source are shown (however, because of the longitudinal cross section, only one is shown in Figure 16), which are attached to a predetermined position relative to the LED collimator 18. Each LED collimator 18 is formed from a translucent resin such as acrylic or glass. The LED collimator 18 has a convex cone-shaped outer surface obtained by rotating a parabolic cross section, and its apex has a concave portion with a central convex portion (i.e., a convex lens surface). The center of its flat surface also has a convex lens surface (or a concave lens surface) that protrudes outward. The parabolic surface that forms the conical outer surface of the LED collimator 18 is set within an angle range that allows for total reflection of the light emitted from the LEDs in the peripheral direction, or a reflective surface is formed within the parabolic surface.

The above configuration is similar to that of the light source device of the image display device shown in Figures 17 and 18. Furthermore, the light converted into approximately parallel light by the LED collimator 15 shown in Figure 16 is reflected by the reflective light guide 304 and transmitted through the reflective polarizer 49, with the other polarized light passing through the light guide 304 again and reflected by the reflector 271 on the other side of the light guide that is not in contact with the liquid crystal display panel 11. At this time, the polarization is converted by passing twice through the retardation plate (λ/4 plate) 270 positioned between the reflector 271 and the liquid crystal display panel 11, and the light passes through the light guide 304 again and the reflective polarizer 49 on the opposite side, aligning the polarization direction before entering the liquid crystal display panel 11. As a result, all of the light from the light source can be utilized, doubling the light utilization efficiency.

In conventional TV sets, the light emitted from an LCD display panel has similar diffusion characteristics in both the horizontal direction of the screen (shown on the X-axis in Figure 22(a)) and the vertical direction of the screen (shown on the Y-axis in Figure 22(b)). In contrast, the diffusion characteristics of the light beam emitted from the LCD display panel of this embodiment are 1/5 of the conventional 62 degrees when the viewing angle at which the brightness is 50% of that when viewed from the front (0 degree angle) is set to 13 degrees, as shown in Example 1 of Figure 22. Similarly, the vertical viewing angle is made uneven between the top and bottom, and the reflection angle and reflective surface area of the reflective light guide are optimized to keep the upper viewing angle to about 1/3 of the lower viewing angle. As a result, the amount of image light directed in the monitoring direction is significantly improved compared to conventional LCD TVs, with brightness being more than 50 times greater.

Furthermore, assuming the viewing angle characteristics shown in Example 2 in Figure 22, the viewing angle at which brightness is 50% of that when viewed from the front (angle of 0 degrees) is set to 5 degrees, which is 1/12 of the conventional 62 degrees. Similarly, the vertical viewing angle is equal from top to bottom, and the reflection angle and reflective surface area of the reflective light guide are optimized to reduce the viewing angle to approximately 1/12 of the conventional viewing angle. As a result, the amount of image light directed toward the monitoring direction is significantly improved compared to conventional LCD TVs, with brightness being more than 100 times higher. As described above, by setting the viewing angle to a narrow angle, the amount of light flux directed toward the monitoring direction can be concentrated, significantly improving light utilization efficiency. As a result, even when using LCD display panels for conventional TVs, significant brightness improvements can be achieved with similar power consumption by controlling the light diffusion characteristics of the light source device, making it possible to create an image display device that is compatible with information display systems aimed at bright outdoor environments.

When using a large LCD display panel, the overall brightness of the screen can be improved by directing the light from the periphery of the screen inward so that it is directed toward the observer when they are facing the center of the screen. Figure 20 shows the convergence angle between the long and short sides of the panel when the observer's distance from the panel, L, and the panel size (screen ratio 16:10) are used as parameters. When monitoring with the screen in portrait orientation, the convergence angle can be set to match the short side. For example, with a 22" panel used vertically and a monitoring distance of 0.8 m, a convergence angle of 10 degrees will allow the image light from the four corners of the screen to be effectively directed toward the observer.

Similarly, when monitoring with a 15" panel in portrait orientation and the monitoring distance is 0.8m, a convergence angle of 7 degrees will allow the image light from the four corners of the screen to be effectively directed towards the monitor. As mentioned above, depending on the size of the LCD display panel and whether it is used portrait or landscape, the image light from the periphery of the screen can be directed towards the monitor who is in the optimal position to monitor the centre of the screen, improving the overall brightness of the screen.

The basic configuration is as shown in Figure 16, in which a light beam with a narrow angle of directionality is incident on a liquid crystal display panel 11 by a light source device, and the brightness is modulated according to the video signal, and the video information displayed on the screen of the liquid crystal display panel 11 is reflected by a retroreflective member, and the resulting spatially floating image is displayed indoors or outdoors via a transparent member 100.
<Example 1 of Light Source Device>

Next, the configuration of the optical system, including the light source device housed in the case, will be described in detail with reference to Figures 17, 18(a) and 18(b).

17 and 18 show LEDs 14a and 14b, which constitute the light source, attached at predetermined positions relative to the LED collimator 15. Each LED collimator 15 is made of a translucent resin, such as acrylic. As shown in FIG. 18(b), the LED collimator 15 has a cone-shaped outer surface 156 obtained by rotating a parabolic cross section, and a recess 153 at its apex with a central convex portion (i.e., a convex lens surface) 157. The central portion of the flat surface has a convex lens surface 154 (or a concave lens surface) 154 that protrudes outward. The parabolic surface 156 that forms the cone-shaped outer surface of the LED collimator 15 is set within an angle range that allows for total reflection of the light emitted from the LEDs 14a and 14b in the peripheral direction, or a reflective surface is formed thereon.

The LEDs 14a and 14b are each arranged at a predetermined position on the surface of the LED board 102, which is the circuit board. The LED board 102 is positioned and fixed to the LED collimator 15 so that the LED 14a or 14b on its surface is located in the center of the recess 153.

With this configuration, the light emitted from LED 14a or 14b by the LED collimator 15 described above, particularly the light emitted upward from the center (toward the right in the figure), is collected and converted into parallel light by the two convex lens surfaces 157, 154 that form the exterior of the LED collimator 15. Light emitted from other areas toward the periphery is reflected by the parabolic surface that forms the conical outer surface of the LED collimator 15, and is similarly collected and converted into parallel light. In other words, an LED collimator 15 with a convex lens in its center and a parabolic surface around its periphery makes it possible to extract almost all of the light generated by LED 14a or 14b as parallel light, thereby improving the utilization efficiency of the generated light.

A polarization conversion element 21 is provided on the light output side of the LED collimator 15. As is clear from Figure 18, this polarization conversion element 21 is composed of a combination of a cylindrical translucent member with a parallelogram cross section (hereinafter referred to as a parallelogram prism) and a cylindrical translucent member with a triangular cross section (hereinafter referred to as a triangular prism), arranged in an array parallel to a plane perpendicular to the optical axis of the collimated light from the LED collimator 15. Furthermore, polarizing beam splitters (hereinafter referred to as "PBS films") 211 and reflective films 212 are alternately provided at the interfaces between adjacent translucent members arranged in the array, and a λ/2 phase plate 213 is provided on the output surface from which light incident on the polarization conversion element 21 and transmitted through the PBS film 211 exits.

The exit surface of this polarization conversion element 21 is further provided with a rectangular composite diffusion block 16, also shown in Figure 18(a). That is, the light emitted from LED 14a or 14b is converted into parallel light by the action of the LED collimator 15 and enters the composite diffusion block 16. After being diffused by the texture 161 on the exit side, the light reaches the light guide 17.

The light guide 17 is a rod-shaped member made of a translucent resin such as acrylic and having a roughly triangular cross section (see Figure 18(b)). As is clear from Figure 17, it comprises a light guide light incident portion (surface) 171 that faces the exit surface of the composite diffusion block 16 via a first diffuser plate 18a, a light guide light reflecting portion (surface) 172 that forms a slope, and a light guide light exit portion (surface) 173 that faces the liquid crystal display panel 11, which is a liquid crystal display element, via a second diffuser plate 18b.

As shown in Figure 17, which is an enlarged partial view, the light guide light reflecting portion (surface) 172 of this light guide 17 has a sawtooth pattern with numerous alternating reflecting surfaces 172a and connecting surfaces 172b. The reflecting surfaces 172a (the line segments sloping upward to the right in the figure) form an angle αn (n is a natural number, for example, 1 to 130 in this example) with respect to the horizontal plane indicated by the dashed dotted line in the figure, and as an example, αn is set to 43 degrees or less (but 0 degrees or greater).

The light guide light incident surface (surface) 171 is formed in a curved convex shape inclined toward the light source. As a result, parallel light from the exit surface of the composite diffusion block 16 is diffused and incident via the first diffuser plate 18a. As is clear from the figure, the light is slightly bent (deflected) upward by the light guide light incident surface (surface) 171 before reaching the light guide light reflecting surface (surface) 172, where it is reflected and reaches the liquid crystal display panel 11 provided on the exit surface at the top of the figure.

The display device 1 described above in detail not only improves light utilization efficiency and uniform illumination characteristics, but also enables compact, low-cost manufacturing, including of a modularized S-polarized light source device. While the above description describes the polarization conversion element 21 as being attached after the LED collimator 15, the present invention is not limited to this, and similar effects can be achieved by locating it in the optical path leading to the liquid crystal display panel 11.

The light guide light reflection portion (surface) 172 has a number of reflective surfaces 172a and connecting surfaces 172b formed alternately in a sawtooth pattern, and the illumination light beam is totally reflected by each reflective surface 172a and directed upward, and then enters the light direction conversion panel 54, which is provided with a narrow-angle diffuser plate on the light guide light exit portion (surface) 173 to control the directional characteristics as a substantially parallel diffused light beam, and then enters the liquid crystal display panel 11 from an oblique direction. In this embodiment, the light direction conversion panel 54 is provided between the light guide exit portion (surface) 173 and the liquid crystal display panel 11, but the same effect can be obtained by providing the light direction conversion panel 54 on the exit surface of the liquid crystal display panel 11.
<Example 2 of Light Source Device>

Figure 19 shows another example of the optical system configuration of the light source device 13, etc. Similar to the example shown in Figure 18, multiple (two in this example) LEDs 14a and 14b that make up the light source are shown, and these are attached to a predetermined position relative to the LED collimator 15. Each LED collimator 15 is made of a translucent resin, such as acrylic. Similar to the example shown in Figure 18, this LED collimator 15 has a cone-convex outer surface 156 obtained by rotating a parabolic cross section, and at its apex, a recess 153 with a central convex portion (i.e., a convex lens surface) 157. The center of the flat portion also has a convex lens surface 154 that protrudes outward (or may be a concave lens surface that is recessed inward). The parabolic surface 156 that forms the cone-shaped outer surface of the LED collimator 15 is set within an angle range that allows for total reflection within it of the light emitted from the LED 14a in the peripheral direction, or a reflective surface is formed thereon.

The LEDs 14a and 14b are each arranged at a predetermined position on the surface of the LED board 102, which is the circuit board. The LED board 102 is positioned and fixed to the LED collimator 15 so that the LED 14a or 14b on its surface is located in the center of the recess 153.

With this configuration, the light emitted from LED 14a or 14b by the LED collimator 15 described above, particularly the light emitted upward from the center (toward the right in the figure), is collected and converted into parallel light by the two convex lens surfaces 157, 154 that form the exterior of the LED collimator 15. Light emitted from other areas toward the periphery is reflected by the parabolic surface that forms the conical outer surface of the LED collimator 15, and is similarly collected and converted into parallel light. In other words, an LED collimator 15 with a convex lens in its center and a parabolic surface around its periphery makes it possible to extract almost all of the light generated by LED 14a or 14b as parallel light, thereby improving the utilization efficiency of the generated light.

A light guide 170 is provided on the light emission side of the LED collimator 15 via a first diffuser plate 18a. The light guide 170 is a rod-shaped member made of a translucent resin such as acrylic and having a roughly triangular cross section (see Figure 19(a)). As is clear from Figure 19(a), it is equipped with a light guide light incident portion (surface) 171 that faces the emission surface of the diffusion block 16 via the first diffuser plate 18a, a light guide light reflecting portion (surface) 172 that forms a slope, and a light guide light emission portion (surface) 173 that faces the liquid crystal display panel 11, which is a liquid crystal display element, via a reflective polarizer 200.

If this reflective polarizer 200 is selected to have the property of reflecting P-polarized light (transmitting S-polarized light), it will reflect the P-polarized light from the natural light emitted by the LED light source, pass through the λ/4 plate 202 provided in the light guide light reflecting section 172 shown in Figure 19(b), reflect at the reflective surface 201, and then pass through the λ/4 plate 202 again to be converted into S-polarized light, and all light beams incident on the liquid crystal display panel 11 will be unified into S-polarized light.

Similarly, if a material having the property of reflecting S-polarized light (and transmitting P-polarized light) is selected as reflective polarizer 200, the S-polarized light in the natural light emitted from the LED light source will be reflected, pass through λ/4 plate 202 provided in light guide light reflecting section 172 shown in Figure 19(b), be reflected by reflective surface 201, and be converted into P-polarized light by passing through λ/4 plate 202 again, so that all light beams incident on liquid crystal display panel 52 are unified into P-polarized light. Polarization conversion can also be achieved with the configuration described above.
<Light Source Device Example 3>

Another example of the optical system configuration of a light source device, etc., will be described using Figure 16. In the third example, as shown in Figure 16, the divergent beam of natural light (a mixture of P-polarized and S-polarized light) from the LED 102 is converted into a nearly parallel beam by a collimator lens 18 and reflected toward the liquid crystal display panel 11 by a reflective light guide 304. The reflected light is incident on a reflective polarizer 206 arranged between the liquid crystal display panel 11 and the reflective light guide 304. A specific polarized wave (e.g., S-polarized wave) is reflected by the reflective polarizer 206, passes through the surface connecting the reflective surfaces of the light guide 304, is reflected by a reflector 271 arranged facing the opposite surface of the light guide 304, and is polarization-converted by passing twice through a phase plate (λ/4 wave plate) 270. The light passes through the light guide and the reflective polarizer, enters the liquid crystal display panel 11, and is modulated into image light. In this case, by combining the specific polarized wave with the polarization-converted polarization plane, the light utilization efficiency is doubled, and the degree of polarization (extinction ratio) of the reflective polarizer is also included in the extinction ratio of the entire system, so using the light source device of this embodiment significantly improves the contrast ratio of the information display system.

As a result, the natural light from the LEDs is aligned to a specific polarization (e.g., P polarization). Similar to the previous example, multiple LEDs are provided to form the light source (however, because Figure 16 is a longitudinal cross-section, only one is shown), and these are attached to a predetermined position relative to the LED collimator 18. Each LED collimator 18 is made of a translucent resin such as acrylic or glass. The LED collimator 18 has a convex convex outer surface obtained by rotating a parabolic cross-section, and its apex has a concave portion with a central convex portion (i.e., a convex lens surface). The center of the flat portion also has a convex lens surface (or a concave lens surface) that protrudes outward. The parabolic surface that forms the conical outer surface of the LED collimator 18 is set within an angle range that allows for total internal reflection of the light emitted from the LEDs 18 in the peripheral direction, or a reflective surface is formed.

The LEDs are also arranged at predetermined positions on the surface of the LED board 102, which serves as the circuit board. The LED board 102 is positioned and fixed to the LED collimator 18 so that the LEDs on its surface are each located in the center of its recess.

With this configuration, the light emitted from the LEDs by the LED collimator 18, particularly the light emitted from the central portion, is collected and converted into parallel light by the two convex lens surfaces that form the outer shape of the LED collimator 18. Meanwhile, the light emitted from other portions toward the periphery is reflected by the parabolic surface that forms the outer peripheral surface of the conical shape of the LED collimator 18, and is similarly collected and converted into parallel light. In other words, the LED collimator 18, which has a convex lens in its center and a parabolic surface formed on its periphery, can extract almost all of the light generated by the LEDs as parallel light, thereby improving the utilization efficiency of the generated light.
<Light Source Device Example 4>

Another example of the optical system configuration of a light source device, etc., is described using Figure 25. Two optical sheets 207 are used on the light output side of the LED collimator 18 to convert the diffusion characteristics in the vertical and horizontal directions (front and back directions, not shown in the figure), and light from the LED collimator 18 is incident between the two optical sheets 207 (diffusion sheets). When this optical sheet 207 is configured with a single sheet, the vertical and horizontal diffusion characteristics are controlled by the fine shapes on the front and back surfaces. Alternatively, multiple diffusion sheets may be used to share the function. The front and back shapes of the optical sheet 207 can be optimized to match the diffusion angle of light from the LED collimator 18 in the vertical direction to the width of the vertical surface of the diffusion sheet's reflective surface, while the number of LEDs and the divergence angle from the LED substrate (optical element) 102 can be used as design parameters to ensure a uniform surface density of the luminous flux emitted from the LCD display panel 11 in the horizontal direction. In other words, the diffusion characteristics are controlled by the surface shapes of multiple diffusion sheets instead of a light guide. In this embodiment, polarization conversion is performed in the same manner as in Example 3 of the light source device described above. Alternatively, a polarization conversion element 21 may be provided between the LED collimator 18 and the diffusion film 207, and the light from the light source may be incident on the diffusion sheet 207 after polarization conversion has been performed.

If the above-mentioned reflective polarizing plate 206 is selected to have the property of reflecting S-polarized light (and transmitting P-polarized light), it will reflect the S-polarized light of the natural light emitted from the LED light source, pass through the retardation plate 270 shown in Fig. 25, be reflected by the reflecting surface 271, and pass through the retardation plate 270 again to be converted into P-polarized light, which then enters the liquid crystal display panel 11. The optimum value for the thickness of this retardation plate must be selected depending on the angle of incidence of the light beam on the retardation plate, and the optimum value lies in the range of λ/16 to λ/4.
<Lenticular lens>

To control the diffusion distribution of the image light from the liquid crystal display panel 11, a lenticular lens can be installed between the light source device 13 and the liquid crystal display panel 11, or on the surface of the liquid crystal display panel 11, and the lens shape can be optimized to control the emission characteristics in one direction. Furthermore, by arranging a microlens array in a matrix, the emission characteristics of the image light beam from the display device 1 can be controlled in the X-axis and Y-axis directions, resulting in an image display device with the desired diffusion characteristics.

The effect of the lenticular lens will now be explained. By optimizing the lens shape, the lenticular lens can efficiently produce a floating image in space by transmitting or reflecting light emitted from the display device 1 described above through the transparent member 100. That is, by combining two lenticular lenses or by providing a sheet that controls the diffusion characteristics by arranging a microlens array in a matrix, the brightness (relative brightness) of the image light from the display device 1 can be controlled in the X-axis and Y-axis directions according to its reflection angle (vertical direction is 0 degrees). In this embodiment, such a lenticular lens makes the vertical brightness characteristic steeper than conventional ones, as shown in Figure 22(b), and further changes the balance of the directional characteristics in the up and down directions (positive and negative directions on the Y axis) to increase the brightness (relative brightness) of the light due to reflection and diffusion. This results in image light with a narrow diffusion angle (high linearity) and containing only specific polarization components, like image light from a surface-emitting laser image source, and suppresses ghost images that occur in retroreflective materials when using image display devices using conventional technology, making it possible to efficiently control the floating image in space due to retroreflection so that it reaches the observer's eyes.

Furthermore, by using the light source device described above, it is possible to realize an image display device that emits light of a specific polarization that emits an image light beam that is nearly parallel to a specific direction by achieving a directional characteristic that is significantly narrower in both the X-axis and Y-axis directions than the diffusion characteristic of light emitted from a typical liquid crystal display panel shown in Figures 22(a) and 22(b) (labeled "conventional" in the figures).

Figure 21 shows an example of the characteristics of the lenticular lens used in this embodiment. This example particularly shows the characteristics in the X direction (vertical direction), with characteristic O showing a luminance characteristic that is symmetrical up and down, with the peak in the light emission direction at an angle of approximately 30 degrees above the vertical direction (0 degrees). Furthermore, characteristics A and B in Figure 21 show examples of characteristics in which the image light above the peak luminance is concentrated at approximately 30 degrees, thereby increasing the luminance (relative luminance). For this reason, with characteristics A and B, the luminance (relative luminance) of light drops sharply at angles exceeding 30 degrees compared to characteristic O.

That is, with the optical system including the lenticular lens described above, when the image light beam from the display device 1 is incident on the retroreflective member 2, the light source device 13 can control the emission angle and viewing angle of the image light aligned at a narrow angle, significantly improving the flexibility of installation of the retroreflective sheet (retroreflective member 2). As a result, the flexibility of the image position of the space-floating image that is reflected or transmitted through the transparent member 100 and focused at the desired position can be significantly improved. As a result, it is possible to efficiently deliver light with a narrow diffusion angle (high linearity) and only specific polarization components to the eyes of an observer indoors or outdoors. This allows the observer to accurately recognize the image light and obtain information even if the intensity (brightness) of the image light from the image display device is reduced. In other words, by reducing the output of the image display device, it is possible to realize a space-floating image display device with low power consumption.
<Touch operation support function>

Next, we will explain the assistive functions for touch operations for the user. First, we will explain touch operations when the assistive functions are not provided. Note that here we will use an example in which the user selects and touches one of two buttons (objects), but the following content can also be suitably applied to, for example, ATMs at banks, ticket vending machines at stations, digital signage, etc.

Figure 26 is a diagram illustrating an example display and touch operation of the space-floating image display device 1000. The space-floating image 3 shown in Figure 26 includes a first button BUT1 labeled "YES" and a second button BUT2 labeled "NO." The user selects "YES" or "NO" by moving their finger 210 toward the space-floating image 3 and touching the first button BUT1 or the second button BUT2. Note that in the examples of Figures 26 and 27-29, the first button BUT1 and the second button BUT2 are displayed in different colors. The areas of the space-floating image 3 other than the first button BUT1 and the second button BUT2 may be transparent and not display any images. In that case, the area affected by the virtual shadow effect (described below) will only be the displayed button areas (the display area of the first button BUT1 and the display area of the second button BUT2). Therefore, in the following explanation, as a more suitable example, in the area of the floating image 3 other than the first button BUT1 and the second button BUT2, an image of a different color or brightness than that of the first button BUT1 and the second button BUT2 is displayed in a wider area including the display area of the first button BUT1 and the display area of the second button BUT2.

In a typical touch-panel image display device that is not a space-floating image display device, the button selected by the user is an image button displayed on the touch panel surface. Therefore, by visually checking the touch panel surface, the user can recognize the distance between their finger and an object (e.g., a button) displayed on the touch panel surface. However, in a space-floating image display device, because the space-floating image 3 floats in the air, it may not be easy for the user to recognize the depth of the space-floating image 3. Therefore, when touching the space-floating image 3, it may not be easy for the user to recognize the distance between their finger and a button displayed on the space-floating image 3. Furthermore, in a typical touch-panel image display device that is not a space-floating image display device, the user can easily determine whether or not they have touched a button based on the feel they get when they touch it. However, when touching the space-floating image 3, there is no feel when touching an object (e.g., a button), so the user may not be able to determine whether or not they have touched the object. Taking the above situation into consideration, this embodiment provides a function to assist the user in touch operations.

In the following description, processing based on the position of the user's finger will be explained, and a specific method for detecting the position of the user's finger will be described later.
<<Touch operation assistance using virtual shadows (1)>>

27 to 29 are diagrams illustrating an example of a method for assisting a touch operation using a virtual shadow. In the examples of FIGS. 27 to 29, the user touches the first button BUT1 to select "YES." The space-floating image display device 1000 of this embodiment assists the user's touch operation by displaying a virtual shadow on the displayed image of the space-floating image 3. Here, "displaying a virtual shadow on the displayed image of the space-floating image 3" refers to an image display process in which the brightness of the video signal for a portion of the image displayed as the space-floating image 3 is reduced for a portion of the image shaped like a finger, thereby making it appear as if a shadow is being projected onto the image. Specifically, this process can be performed by calculations performed by the image control unit 1160 or the control unit 1110. In the virtual shadow display process, the brightness of the video signal for a portion of the finger-shaped area may be set to completely zero. However, rather than completely setting the brightness of the video signal for that portion of the finger-shaped area to completely zero, it is preferable to display the image at reduced brightness in that portion of the finger-shaped area, as this results in a more natural shadow. In this case, when displaying the virtual shadow, not only may the brightness of the video signal be reduced for a portion of the area shaped like a finger, but the saturation of the video signal may also be reduced.

The floating image 3 exists in the air, where there is no physical contact surface, and in a normal environment, the shadow of a finger would normally be projected. However, with the virtual shadow display process of this embodiment, even in the air, where a finger shadow would not normally be projected, it is possible to make it appear as if a shadow exists within the floating image 3, thereby improving the user's perception of depth and the realism of the floating image 3.

Figure 27 shows the state at a first point in time when the user attempts to touch the first button BUT1 on the display surface 3a of the floating image 3 with the finger 210, Figure 28 shows the state at a second point in time when the finger 210 is closer to the floating image 3 in the space than in Figure 27, and Figure 29 shows the state at a third point in time when the finger 210 has touched the first button BUT1 on the display surface 3a of the floating image 3. In addition, (A) of Figures 27 to 29 shows the state when the display surface 3a of the floating image 3 is viewed from the front (normal direction of the display surface 3a), and (B) of Figures 27 to 29 shows the state when the display surface 3a of the floating image 3 is viewed from the side (direction parallel to the display surface 3a). 27 to 29, the x direction is the horizontal direction on the display surface 3a of the space-floating image 3, the y direction is the direction perpendicular to the x axis within the display surface 3a of the space-floating image 3, and the z direction is the normal direction of the display surface 3a of the space-floating image 3 (the height direction relative to the display surface 3a). In the explanatory diagrams of FIGS. 27 to 33, the space-floating image 3 is shown as having a thickness in the depth direction for ease of explanation, but in reality, if the image display surface of the display device 1 is flat, the space-floating image 3 is also flat and has no thickness in the depth direction. In this case, the space-floating image 3 and the display surface 3a are on the same plane. In the description of this embodiment, the display surface 3a refers to the surface on which the space-floating image 3 can be displayed, and the space-floating image 3 refers to the portion on which the space-floating image is actually displayed.

27, 28, and 29, the detection process for the finger 210 is performed using, for example, an image generated by the imaging unit 1180 and a sensing signal from the aerial operation detection sensor 1351. In the detection process for the finger 210, for example, the position (x coordinate, y coordinate) of the tip of the finger 210 on the display surface 3a of the floating in space image 3, the height position (z coordinate) of the tip of the finger 210 relative to the display surface 3a, etc. are detected. Here, the position (x coordinate, y coordinate) of the tip of the finger 210 on the display surface 3a of the floating in space image 3 is the position coordinate on the display surface 3a of the intersection of a perpendicular line from the tip of the finger 210 to the display surface 3a of the floating in space image 3. Note that the height position of the tip of the finger 210 relative to the display surface 3a is also depth information that represents the depth of the finger 210 relative to the display surface 3a. Note that the arrangement of the imaging unit 1180 and the aerial operation detection sensor 1351 that detect the finger 210 etc. will be explained in detail later.

At the first time point shown in FIG. 27, the finger 210 is at a position farthest from the display surface 3a of the floating-in-space image 3 compared to the second time point shown in FIG. 28 and the third time point shown in FIG. 28. The distance (height position) between the tip of the finger 210 and the display surface 3a of the floating-in-space image 3 at this time is defined as dz1. In other words, the distance dz1 indicates the height of the finger 210 relative to the display surface 3a of the floating-in-space image 3 in the z direction.

Note that distance dz1 shown in FIG. 27 and distance dz2 shown in FIG. 28 (described later) are positive on the user side of display surface 3a of the floating in space image 3, and negative on the side opposite the user from display surface 3a. In other words, if finger 210 is on the user side of display surface 3a, distances dz1 and dz2 will be positive values, and if finger 210 is on the opposite side of display surface 3a from the user, distances dz1 and dz2 will be negative values.

In this embodiment, it is assumed that virtual light source 1500 is located on the user's side relative to display surface 3a of space-floating image 3. The setting of the installation direction of virtual light source 1500 may be actually stored as information in non-volatile memory 1108 or memory 1109 of space-floating image display device 1000. Furthermore, the setting of the installation direction of virtual light source 1500 may be a parameter that exists only in design. Even if the setting of the installation direction of virtual light source 1500 is a parameter that exists only in design, the design installation direction of virtual light source 1500 is uniquely determined based on the relationship between the position of the user's finger and the display position of the virtual shadow, which will be described later. In the examples of Figures 27 to 29, virtual light source 1500 is located on the user's side relative to display surface 3a, on the right side of display surface 3a as seen from the user. A virtual shadow 1510 simulating the shadow of finger 210 formed by light emitted from virtual light source 1500 is displayed on space-floating image 3. In the examples of Figures 27 to 29, a virtual shadow 1510 is displayed to the left of the finger 210. This virtual shadow 1510 assists the user in performing touch operations.

In the state of FIG. 27(B), the tip of finger 210 is farthest in the normal direction from display surface 3a of floating-in-space image 3 compared to the states of FIG. 28(B) and FIG. 29(B). Therefore, in FIG. 27(A), the tip of virtual shadow 1510 is formed at a position farthest in the horizontal direction from first button BUT1 to be touched compared to the states of FIG. 28(A) and FIG. 29(A). Therefore, in FIG. 27(A), the horizontal distance between the tip of finger 210 and the tip of virtual shadow 1510 when viewing display surface 3a of floating-in-space image 3 from the front is greatest compared to the states of FIG. 28(A) and FIG. 29(A). In FIG. 27(A), the distance between the tip of finger 210 and the tip of virtual shadow 1510 in the horizontal direction on display surface 3a of floating-in-space image 3 is dx1.

28(B), finger 210 is closer to space-floating image 3 than in FIG. 27(B). Therefore, in FIG. 28(B), the distance dz2 in the normal direction between the tip of finger 210 and display surface 3a of space-floating image 3 is smaller than dz1. In this case, in FIG. 28(A), virtual shadow 1510 is displayed at a position where the distance between the tip of finger 210 and the tip of virtual shadow 1510 in the horizontal direction of display surface 3a of space-floating image 3 is dx2, which is smaller than dx1. That is, in the example of FIG. 28, virtual light source 1500 is provided on the user's side of display surface 3a, on the right side of display surface 3a as seen from the user. Therefore, the horizontal distance between the tip of finger 210 and the tip of virtual shadow 1510 when display surface 3a of space-floating image 3 is viewed from the front, changes in conjunction with the distance in the normal direction between the tip of finger 210 and display surface 3a of space-floating image 3.

When the tip of finger 210 comes into contact with the tip of virtual shadow 1510, the distance in the normal direction between the tip of finger 210 and display surface 3a of space floating image 3 becomes zero, as shown in FIG. 29. At this time, virtual shadow 1510 is displayed so that the distance between finger 210 and virtual shadow 1510 in the horizontal direction of display surface 3a of space floating image 3 becomes zero. This allows the user to recognize that finger 210 has touched display surface 3a of space floating image 3. At this time, if the tip of finger 210 is touching the area of first button BUT1, the user can recognize that they have touched first button BUT1. That is, in the example of FIG. 29 , virtual light source 1500 is provided on the user side of display surface 3a, to the right of display surface 3a as seen from the user, and therefore the horizontal distance between the tip of finger 210 and the tip of virtual shadow 1510 when display surface 3a of space floating image 3 is viewed from the front changes in conjunction with the distance in the normal direction between the tip of finger 210 and display surface 3a of space floating image 3. That is, the display position of the tip of virtual shadow 1510 is a position specified by the positional relationship between the position of virtual light source 1500 and the position of the tip of user's finger 210, and changes in conjunction with changes in the position of the tip of user's finger 210.

According to the configuration and processing of "Touch Operation Assistance Using Virtual Shadow (1)" explained above, when performing a touch operation, the user can more appropriately recognize the distance (depth) in the normal direction between the finger 210 and the display surface 3a of the Floating-in-Space Image 3 from the horizontal positional relationship between the finger 210 and the virtual shadow 1510 on the display surface 3a of the Floating-in-Space Image 3. Furthermore, when the finger 210 touches an object (e.g., a button) that is the Floating-in-Space Image 3, the user can recognize that he or she has touched the object. This makes it possible to provide a more appropriate Floating-in-Space Image Display Device.
<<Touch operation assistance using virtual shadows (2)>>

Next, as another example of a method for assisting touch operations using a virtual shadow, a case where virtual light source 1500 is provided on the left side of display surface 3a as seen from the user will be described. Figures 30 to 32 are diagrams illustrating another example of a method for assisting touch operations using a virtual shadow. Figure 30 corresponds to Figure 27 and shows the state at a first point in time when the user attempts to touch first button BUT1 on display surface 3a of floating-in-space image 3 with finger 210. Figure 31 corresponds to Figure 28 and shows the state at a second point in time when finger 210 is closer to floating-in-space image 3 than in Figure 30. Figure 32 corresponds to Figure 29 and shows the state when finger 210 touches floating-in-space image 3. Note that for ease of explanation, (B) in Figures 30 to 32 is shown as being viewed from the opposite direction to (B) in Figures 27 to 29.

In Figures 30 to 32, virtual light source 1500 is provided on the user's side of display surface 3a, on the left side of display surface 3a as seen from the user. A virtual shadow 1510 that resembles the shadow of finger 210 formed by light emitted from virtual light source 1500 is displayed in floating-in-space image 3. In Figures 30 to 32, virtual shadow 1510 is displayed to the right of finger 210. This virtual shadow 1510 assists the user in performing touch operations.

In the state of FIG. 30(B), the tip of finger 210 is the furthest away in the normal direction from display surface 3a of space floating image 3 compared to the states of FIG. 31(B) and FIG. 32(B). In FIG. 30(B), the distance in the normal direction between the tip of finger 210 and display surface 3a of space floating image 3 at this time is dz10. Also, in FIG. 30(A), the distance in the horizontal direction between the tip of finger 210 and the tip of virtual shadow 1510 on display surface 3a of space floating image 3 at this time is dx10.

31(B), finger 210 is closer to space-floating image 3 than in FIG. 27(B). Therefore, in FIG. 31(B), the normal distance dz20 between the tip of finger 210 and display surface 3a of space-floating image 3 is smaller than dz10. At this time, in FIG. 31(A), virtual shadow 1510 is displayed at a position where the distance between the tip of finger 210 and the tip of virtual shadow 1510 in the horizontal direction of display surface 3a of space-floating image 3 is dx20, which is smaller than dx10. That is, in the example of FIG. 31, virtual light source 1500 is provided on the user's side of display surface 3a, on the left side of display surface 3a as seen from the user. Therefore, the horizontal distance between the tip of finger 210 and the tip of virtual shadow 1510 when display surface 3a of space-floating image 3 is viewed from the front, changes in conjunction with the normal distance between the tip of finger 210 and display surface 3a of space-floating image 3.

When the tip of finger 210 comes into contact with the tip of virtual shadow 1510, the distance in the normal direction between the tip of finger 210 and display surface 3a of space floating image 3 becomes zero, as shown in FIG. 32. At this time, virtual shadow 1510 is displayed so that the distance between finger 210 and virtual shadow 1510 in the horizontal direction of display surface 3a of space floating image 3 becomes zero. This allows the user to recognize that finger 210 has touched display surface 3a of space floating image 3. At this time, if the tip of finger 210 is touching the area of first button BUT1, the user can recognize that they have touched first button BUT1. That is, in the example of FIG. 32, virtual light source 1500 is provided on the user's side of display surface 3a, to the left of display surface 3a as seen from the user, and therefore the horizontal distance between the tip of finger 210 and the tip of virtual shadow 1510 when display surface 3a of space floating image 3 is viewed from the front changes in conjunction with the distance in the normal direction between the tip of finger 210 and display surface 3a of space floating image 3.

The configuration and processing of "Touch operation assistance using virtual shadow (2)" described above also achieves the same effects as the configurations shown in Figures 27 to 29.

Here, when implementing the above-mentioned "Touch operation assistance using virtual shadow (1)" process and/or "Touch operation assistance using virtual shadow (2)" process in the space floating image display device 1000, there are several possible implementation examples:

A first implementation example is a method of implementing only "Touch Operation Assistance Using Virtual Shadows (1)" in the space-floating image display device 1000. In this case, since the virtual light source 1500 is located on the user's side of the display surface 3a, on the right side of the display surface 3a as seen from the user, the virtual shadow 1510 is displayed to the left of the tip of the user's finger 210 as seen from the user. Therefore, if the user's finger 210 is a finger on the right hand, the visibility of the display of the virtual shadow 1510 is not obstructed by the user's right hand or right arm, which is preferable. Therefore, given the statistical tendency for there to be a large number of right-handed users, even if only "Touch Operation Assistance Using Virtual Shadows (1)" is implemented in the space-floating image display device 1000, the probability that the display of the virtual shadow 1510 will be clearly visible is sufficiently high, and this is preferable.

Also, as a first implementation example, both the "Touch Operation Assistance Using Virtual Shadow (1)" process and the "Touch Operation Assistance Using Virtual Shadow (2)" process may be implemented, with switching between the processes depending on whether the user is performing the touch operation with their right or left hand. In this case, it is possible to further increase the probability that the display of the virtual shadow 1510 can be clearly seen, improving user convenience.

Specifically, when a user performs a touch operation with their right hand, the configurations of Figures 27 to 29 are used to display a virtual shadow 1510 to the left of the finger 210. In this case, the visibility of the virtual shadow 1510 display is not obstructed by the user's right hand or right arm, which is preferable. On the other hand, when a user performs a touch operation with their left hand, the configurations of Figures 30 to 32 are used to display a virtual shadow 1510 to the right of the finger 210. In this case, the visibility of the virtual shadow 1510 display is not obstructed by the user's left hand or left arm, which is preferable. As a result, whether the user performs a touch operation with their right hand or left hand, the virtual shadow 1510 is displayed in a position that is easy for the user to see, improving user convenience.

Here, determining whether the touch operation is being performed with the right hand or the left hand may be made, for example, based on the captured image generated by the imaging unit 1180. For example, the control unit 1110 performs image processing on the captured image and detects the user's face, arms, hands, and fingers from the captured image. The imaging unit 1180 then estimates the user's posture or movement from the arrangement of these detected elements (face, arms, hands, and fingers) and determines whether the user is performing the touch operation with the right hand or the left hand. Note that in this determination, capturing an image of the face is not necessarily required as long as the vicinity of the center of the user's body in the left-right direction can be determined from other parts. Alternatively, the above determination may be made based solely on the arrangement of the arms. Alternatively, the above determination may be made based solely on the arrangement of the hands. Alternatively, the above determination may be made based on a combination of the arrangement of the arms and the arrangement of the hands. Furthermore, the above determination may be made in combination with the arrangement of the face.

Note that Figures 27 to 29 and 30 to 32 show a virtual shadow 1510 extending at an angle corresponding to the extension direction of the actual finger 210. The extension direction of the actual finger 210 can be calculated by capturing an image of the finger using any of the image capturing units already described. Here, it is also possible to display a virtual shadow 1510 with its extension direction fixed at a predetermined angle, without reflecting the angle corresponding to the extension direction of the finger 210. This reduces the load on the image control unit 1160 or control unit 1110, which controls the display of the virtual shadow 1510.

For example, if finger 210 is a finger on the right hand, it would be natural for the user to extend their arm from the front right side of display surface 3a of the floating in space image 3, and attempt to touch display surface 3a of the floating in space image 3 with finger 210 pointing to the upper left as they face display surface 3a of the floating in space image 3. Therefore, if finger 210 is a finger on the right hand, by configuring the shadow of the finger indicated by virtual shadow 1510 to be displayed in a predetermined direction pointing to the upper right as they face display surface 3a of the floating in space image 3, a natural display will be achieved without reflecting the angle corresponding to finger 210.

Furthermore, for example, if finger 210 is a finger on the left hand, it would be natural for the user to extend their arm from the left side in front of display surface 3a of the floating in space image 3, and attempt to touch display surface 3a of the floating in space image 3 with finger 210 pointing to the upper right as they face display surface 3a of the floating in space image 3. Therefore, if finger 210 is a finger on the left hand, by configuring the shadow of the finger indicated by virtual shadow 1510 to be displayed in a predetermined direction pointing to the upper left as they face display surface 3a of the floating in space image 3, a natural display will be achieved without reflecting the angle corresponding to finger 210.

Note that when the user's finger 210 is on the opposite side of the display surface 3a of the space floating image 3 from the user, a display can be made that allows the user to recognize that the finger 210 is behind the space floating image 3 and cannot be touched. For example, a message informing the user that the finger 210 is behind the space floating image 3 and cannot be touched can be displayed on the space floating image 3. Alternatively, for example, the virtual shadow 1510 can be displayed in a different color from normal, such as red. This makes it possible to more suitably prompt the user to return the finger 210 to the appropriate position.
<<Example of virtual light source setting conditions>>

Here, we will explain how to set the virtual light source 1500. Figure 33 is a diagram explaining how to set the virtual light source. Figure 33 shows a situation in which the user performs a touch operation with their left hand, but the content explained below can also be suitably applied when the user performs a touch operation with their right hand.

Figure 33 shows the normal line L1 of the display surface 3a of the floating-in-space image 3, which extends from point C at the center of the display surface 3a toward the user, the line L2 connecting the virtual light source 1500 and point C where the normal line L1 intersects with the display surface 3a, and the virtual light source installation angle α, which is defined by the angle between the normal line L1 and line L2. For ease of explanation, Figure 33 shows the moment when the tip of the user's finger 210 is on line L2.

Here, for simplicity's sake, in Figures 27 to 33, virtual light source 1500 is illustrated as being placed in a position not far from display surface 3a of space-floating image 3 and user's finger 210. While virtual light source 1500 may be set in such a position, the most suitable setting example is as follows. That is, it is desirable to set the distance between virtual light source 1500 and point C, the center of display surface 3a of space-floating image 3, to infinity. The reason for this is as follows. If an object plane with a contact surface is in the same coordinate system as display surface 3a of space-floating image 3 in Figures 27 to 32, and the light source is the sun rather than a virtual light source, the distance of the sun can be approximated as nearly infinity, so the horizontal (x-direction) position of the tip of the user's finger shadow on the real object plane changes linearly with changes in the distance (z-direction) between the tip of the user's finger and the object plane. 27 to 33 of this embodiment, if the distance between virtual light source 1500 and center point C of display surface 3 a of floating-in-space image 3 is set to infinity and the horizontal (x-direction) position of the tip of virtual shadow 1510 in floating-in-space image 3 changes linearly with changes in the distance (z-direction) between the tip of user's finger 210 and display surface 3 a of floating-in-space image 3, a virtual shadow that can be perceived more naturally by the user can be expressed. If virtual light source 1500 is set to be located not far from display surface 3 a of floating-in-space image 3 and user's finger 210, the horizontal (x-direction) position of the tip of virtual shadow 1510 in floating-in-space image 3 changes nonlinearly with changes in the distance (z-direction) between the tip of user's finger 210 and display surface 3 a of floating-in-space image 3, making the calculation for calculating the horizontal (x-direction) position of the tip of virtual shadow 1510 somewhat complicated. On the other hand, if the distance between virtual light source 1500 and center point C of display surface 3a of space floating image 3 is set to infinity, the horizontal (x-direction) position of the tip of virtual shadow 1510 in space floating image 3 will change linearly with changes in the distance (z-direction) between the tip of user's finger 210 and display surface 3a of space floating image 3, which has the effect of simplifying the calculation to calculate the horizontal (x-direction) position of the tip of virtual shadow 1510.

When the virtual light source installation angle α is small, the angle between the normal line L1 and the line connecting virtual light source 1500 and finger 210 cannot be made large from the user's perspective, and as a result, the distance between the tip of finger 210 and the tip of virtual shadow 1510 in the horizontal direction (x direction) of display surface 3a of floating-in-space image 3 becomes short. This makes it difficult for the user to visually recognize changes in the position of virtual shadow 1510 when the tip of finger 210 performs a touch operation, which may reduce the effectiveness of the user's depth perception during touch operations. To avoid this, it is desirable to install virtual light source 1500 so that the angle between line L2 connecting virtual light source 1500 and point C and normal line L1 is, for example, 20° or greater.

On the other hand, if the angle between the normal line L1 and the line connecting virtual light source 1500 and finger 210 approaches 90°, the distance between the tip of finger 210 and the tip of virtual shadow 1510 becomes very long. This increases the probability that the display position of virtual shadow 1510 will be outside the range of floating-in-space image 3, and increases the probability that virtual shadow 1510 will not be displayed in floating-in-space image 3. For this reason, it is desirable that the installation angle α of virtual light source 1500 be 70° or less, so that the angle between line L2 connecting virtual light source 1500 and point C and normal line L1 does not become too close to 90°, for example.

In other words, it is desirable to place the virtual light source 1500 in a position that is not too close to the plane containing the normal passing through the finger 210, and not too close to the plane containing the display surface 3a of the floating-in-space image 3.

The space-floating image display device 1000 of this embodiment can display a virtual shadow as described above. This is an image processing that produces a physically more natural effect than when a predetermined mark is superimposed on an image to assist the user's touch operation. Therefore, the touch operation assistance technology of the space-floating image display device 1000 of this embodiment by displaying the above-mentioned virtual shadow can provide a situation in which the user can recognize the depth of the touch operation more naturally.
<<Finger position detection method>>

Next, a method for detecting the position of the finger 210 will be described. A configuration for detecting the position of the finger 210 of the user 230 will be specifically described below.
<<<<Finger position detection method (1)>>>

Figure 34 is a configuration diagram showing an example of a method for detecting the position of a finger. In the example shown in Figure 34, the position of the finger 210 is detected using one imaging unit 1180 and one aerial operation sensor 1351. Note that all imaging units in this embodiment of the present invention have an imaging sensor.

The first imaging unit 1180a (1180) is installed on the opposite side of the user 230 with respect to the floating image 3. The first imaging unit 1180a may be installed in the housing 1190 as shown in FIG. 34, or may be installed in a location away from the housing 1190.

The imaging area of the first imaging unit 1180a is set to include, for example, the display area of the Floating in Space image 3, as well as the fingers, hands, arms, face, etc. of the user 230. The first imaging unit 1180a images the user 230 performing a touch operation on the Floating in Space image 3, and generates a first captured image. Note that even if the display area of the Floating in Space image 3 is captured by the first imaging unit 1180a, the image will be captured from the opposite side of the direction of travel of the directional light beam of the Floating in Space image 3, and the Floating in Space image 3 itself will not be visible as an image. Here, in the example of finger position detection method (1), the first imaging unit 1180a is not just an imaging unit, but has a built-in depth sensor in addition to an imaging sensor. The configuration and processing of the depth sensor can use existing technology. The depth sensor of the first imaging unit 1180a detects the depth of each part (e.g., the user's fingers, hands, arms, face, etc.) in the captured image of the first imaging unit 1180a and generates depth information.

The aerial operation sensor 1351 is installed in a position where it can sense the display surface 3a of the floating-in-space image 3 as the sensing target surface. In FIG. 34, the aerial operation sensor 1351 is installed below the display surface 3a of the floating-in-space image 3, but it may also be installed to the side or above the display surface 3a. The aerial operation sensor 1351 may be installed on the housing 1190 as shown in FIG. 34, or may be installed in a location away from the housing 1190.

The aerial operation detection sensor 1351 in FIG. 34 is a sensor that detects the position where the finger 210 comes into contact with or overlaps the display surface 3a of the floating image 3. In other words, when the tip of the finger 210 approaches the display surface 3a of the floating image 3 from the user's side of the display surface 3a of the floating image 3, the aerial operation detection sensor 1351 can detect the finger 210 coming into contact with the display surface 3a of the floating image 3.

For example, the control unit 1110 shown in FIG. 3C reads from the non-volatile memory 1108 a program for performing image processing and a program for displaying the virtual shadow 1510. The control unit 1110 performs first image processing on the first captured image generated by the image sensor of the first imaging unit 1180a, detects the finger 210, and calculates the position (x coordinate, y coordinate) of the finger 210. The control unit 1110 calculates the position (z coordinate) of the tip of the finger 210 relative to the floating-in-space image 3 based on the first captured image generated by the image sensor of the first imaging unit 1180a and the depth information generated by the depth sensor of the first imaging unit 1180a.

In the example of FIG. 34, the image sensor and depth sensor of the first imaging unit 1180a, the aerial operation sensor 1351, the aerial operation detection unit 1350, and the control unit 1110 constitute a touch detection unit that detects the position of the user's finger and detects a touch on an object in the floating in space image 3. This calculates the position (x coordinate, y coordinate, z coordinate) of the finger 210. Furthermore, the touch detection result is calculated from the detection result of the aerial operation detection unit 1350 or a combination of the detection result of the aerial operation detection unit 1350 and information generated by the first imaging unit 1180a.

Then, the control unit 1110 calculates the position (display position) at which to display the virtual shadow 1510 based on the position (x coordinate, y coordinate, z coordinate) of the finger 210 and the position of the virtual light source 1500, and generates video data of the virtual shadow 1510 based on the calculated display position.

The control unit 1110 may calculate the display position of the virtual shadow 1510 in the video data each time the position of the finger 210 is calculated. Instead of calculating the display position of the virtual shadow 1510 in the video data each time the position of the finger 210 is calculated, display position map data that calculates the display position of the virtual shadow 1510 corresponding to each of multiple positions of the finger 210 may be stored in advance in the non-volatile memory 1108, and after calculating the position of the finger 210, video data of the virtual shadow 1510 may be generated based on the display position map data stored in the non-volatile memory 1108. The control unit 1110 may also calculate the tip of the finger 210 and the extension direction of the finger 210 in the first image processing, calculate the display position of the tip of the finger 210 and the extension direction of the virtual shadow 1510 that corresponds to the extension direction, and generate video data of the virtual shadow 1510 adjusted to a display angle that corresponds to the actual orientation of the finger 210 based on these.

The control unit 1110 outputs the generated video data of the virtual shadow 1510 to the video control unit 1160. The video control unit 1160 generates video data (superimposed video data) in which the video data of the virtual shadow 1510 is superimposed on other video data such as objects, and outputs the superimposed video data including the video data of the virtual shadow 1510 to the video display unit 1102.

The video display unit 1102 displays a video based on the superimposed video data, including the video data of the virtual shadow 1510, thereby displaying a floating-in-space video 3 in which the virtual shadow 1510 and an object, etc., are superimposed.

Detection of a touch on an object is performed, for example, as follows. The aerial operation detection unit 1350 and aerial operation detection sensor 1351 are configured as described in FIG. 3, and when the finger 210 touches or is superimposed on a plane including the display surface 3a of the space floating image 3, they detect the position and output touch position information indicating the position where the finger 210 touches or is superimposed on the display surface 3a to the control unit 1110. When the touch position information is input, the control unit 1110 determines whether the position (x coordinate, y coordinate) of the finger 210 calculated by the first image processing is included in the display range of each object displayed on the display surface 3a of the space floating image 3. If the position of the finger 210 is included in the display range of any object, the control unit 1110 determines that this object has been touched.

The detection method described above makes it possible to detect the position of the finger 210 and detect touch operations using a simple configuration that combines one imaging unit 1180 (first imaging unit 1180a) that has an image sensor and a depth sensor with one aerial operation detection sensor 1351.

As a variant of the finger position detection method (1), the control unit 1110 may detect a touch operation by the finger 210 solely based on the first captured image generated by the image sensor of the first imaging unit 1180a and the depth information generated by the depth sensor of the first imaging unit 1180a, without using the detection results by the aerial operation detection unit 1350 and the aerial operation detection sensor 1351. For example, during normal operation, the system is configured to enter a mode in which touch operations by the finger 210 are detected by combining the image captured by the image sensor of the first imaging unit 1180a, the detection results of the depth sensor, and the detection results of the aerial operation detection sensor 1351, and if there is any malfunction in the operation of the aerial operation detection sensor 1351 or the aerial operation detection unit 1350, the control unit 1110 may switch to a mode in which touch operations by the finger 210 are detected solely based on the first image captured by the image sensor of the first imaging unit 1180a and the depth information generated by the depth sensor of the first imaging unit 1180a, without using the detection results of the aerial operation detection unit 1350 and the aerial operation detection sensor 1351.
<<Finger position detection method (2)>>

Figure 35 is a configuration diagram showing another example of a method for detecting the position of a finger. In the example shown in Figure 35, the position of the finger 210 is detected using two imaging units. The second imaging unit 1180b (1180) and the third imaging unit 1180c (1180) are both provided on the opposite side of the floating-in-space image 3 from the user 230.

The second imaging unit 1180b is installed, for example, on the right side as seen from the user 230. The imaging area of the second imaging unit 1180b is set to include, for example, the floating in space image 3, the fingers, hands, arms, face, etc. of the user 230. The second imaging unit 1180b images the user 230 performing a touch operation on the floating in space image 3 from the right side of the user 230, and generates a second captured image.

The third imaging unit 1180c is installed, for example, on the left side as seen from the user 230. The imaging area of the third imaging unit 1180c is set to include, for example, the floating in space image 3, the fingers, hands, arms, face, etc. of the user 230. The third imaging unit 1180c images the user 230 performing a touch operation on the floating in space image 3 from the left side of the user 230, and generates a third captured image. Thus, in the example of FIG. 35, the second imaging unit 1180b and the third imaging unit 1180c form a so-called stereo camera.

The second imaging unit 1180b and the third imaging unit 1180c may be installed in the housing 1190 as shown in FIG. 35, or may be installed at a location away from the housing 1190. Alternatively, one imaging unit may be installed in the housing 1190 and the other imaging unit may be installed at a location away from the housing 1190.

The control unit 1110 performs second image processing on the second captured image and third image processing on the third captured image. The control unit 1110 then calculates the position (x coordinate, y coordinate, z coordinate) of the finger 210 based on the results of the second image processing (second image processing result) and the results of the third image processing (third image processing result).

In the example of FIG. 35, the second imaging unit 1180b, the third imaging unit 1180c, and the control unit 1110 constitute a touch detection unit that detects the position of the user's finger and detects a touch on an object in the floating-in-space image 3. Then, the position (x coordinate, y coordinate, z coordinate) of the finger 210 is calculated as the position detection result or touch detection result.

In this way, in the example of Figure 35, a virtual shadow 1510 is generated based on the position of the finger 210 calculated based on the second image processing result and the third image processing result. In addition, whether or not an object has been touched is determined based on the position of the finger 210 calculated based on the second image processing result and the third image processing result.

With this configuration, there is no need to employ an imaging unit with a depth sensor. Furthermore, with this configuration, the second imaging unit 1180b and the third imaging unit 1180c are used as a stereo camera, making it possible to improve the accuracy of detecting the position of the finger 210. In particular, compared to the example in FIG. 34, the accuracy of detecting the x and y coordinates can be improved. This makes it possible to more accurately determine whether an object has been touched.

Furthermore, as a modified example of the finger position detection method (2), the detection of the user's finger position (x coordinate, y coordinate, z coordinate) is performed based on the second captured image by the second imaging unit 1180b and the third captured image by the third imaging unit 1180c, as described above, and the display of the virtual shadow 1510 is controlled thereby, and the presence or absence of a touch on the object in the space floating image 3 can be detected by the aerial operation detection unit 1350 or the control unit 1110 based on the detection result by the aerial operation detection sensor 1351. According to this modified example, the aerial operation sensor 1351 is used, which senses the display surface 3a of the space floating image 3 as the sensing target surface, so that the detection of the contact of the user's finger 210 with the display surface 3a of the space floating image 3 can be detected with higher accuracy than the detection accuracy in the depth direction by the stereo camera using the second imaging unit 1180b and the third imaging unit 1180c.
<<<<Finger position detection method (3)>>>

Figure 36 is a configuration diagram showing another example of a method for detecting the position of a finger. In the example shown in Figure 36, the position of the finger 210 is also detected using two imaging units. The example in Figure 36 differs from the example in Figure 35 in that the fourth imaging unit 1180d (1180), one of the imaging units, is positioned to capture an image of the display surface 3a of the space floating image 3 from the side. Also, as in the example in Figure 34, the first imaging unit 1180a (1180) is installed on the opposite side of the user 230 with respect to the space floating image 3. In the example in Figure 36, the first imaging unit 1180a (1180) only needs to be able to capture an image, and does not need to be equipped with a depth sensor.

Therefore, the fourth imaging unit 1180d is installed around the display surface 3a of the space floating image 3. In FIG. 36, the fourth imaging unit 1180d is installed below the side of the display surface 3a of the space floating image 3, but it may also be installed to the side or above the display surface 3a. The fourth imaging unit 1180d may be installed in the housing 1190 as shown in FIG. 36, or may be installed in a location away from the housing 1190.

The imaging area of the fourth imaging unit 1180d is set to include, for example, the floating in space image 3, the fingers, hands, arms, face, etc. of the user 230. The fourth imaging unit 1180d images the user 230 performing a touch operation on the floating in space image 3 from around the display surface 3a of the floating in space image 3, and generates a fourth captured image.

The control unit 1110 performs fourth image processing on the fourth captured image and calculates the distance (z coordinate) between the display surface 3a of the spatial floating image 3 and the tip of the finger 210. Then, the control unit 1110 performs processing related to the virtual shadow 1510 and determines whether or not an object has been touched, based on the position (x coordinate, y coordinate) of the finger 210 calculated by the first image processing on the first captured image by the first imaging unit 1180a described above, and the position (z coordinate) of the finger 210 calculated by the fourth image processing.

In the example of FIG. 36, the first imaging unit 1180a, the fourth imaging unit 1180d, and the control unit 1110 constitute a touch detection unit that detects the position of the user's finger and detects a touch on an object. The position (x coordinate, y coordinate, z coordinate) of the finger 210 is then calculated as the position detection result or touch detection result.

This configuration makes it possible to improve the accuracy of detecting the distance between the display surface 3a of the floating image 3 and the tip of the finger 210, i.e., the depth of the finger 210 relative to the display surface 3a of the floating image 3, compared to the example stereo camera configuration shown in Figure 35.

Furthermore, as a modified example of the finger position detection method (3), the detection of the user's finger position (x coordinate, y coordinate, z coordinate) is performed based on the first captured image by the first imaging unit 1180a and the fourth captured image by the fourth imaging unit 1180d, as described above, and the display of the virtual shadow 1510 is controlled thereby, and the presence or absence of a touch on the object in the floating in space image 3 can be detected by the aerial operation detection unit 1350 or the control unit 1110 based on the detection result by the aerial operation detection sensor 1351. According to this modified example, the aerial operation sensor 1351 is used, which senses the display surface 3a of the floating in space image 3 as the sensing target surface, so that the touch of the user's finger 210 on the display surface 3a of the floating in space image 3 can be detected with higher accuracy than the detection accuracy using the fourth captured image by the fourth imaging unit 1180d.
<<How to display input content to assist touch operations>>

An example of assisting the user's touch operation in another way will be described below. For example, it is possible to assist the touch operation by displaying the input content. Figure 37 is a diagram illustrating a method of assisting the touch operation by displaying the input content. Figure 37 shows the case where numbers are entered by touch operation.

The floating image 3 in Figure 37 includes a key input UI (user interface) display area 1600 containing multiple objects, including, for example, multiple objects for inputting numbers, etc., an object 1601 for erasing input content, and an object 1603 for confirming input content, as well as an input content display area 1610 for displaying input content.

In the input content display area 1610, the content (e.g., numbers) entered by touch operation is displayed sequentially on the floating image 3 from left to right. The user can check the content entered by touch operation while looking at the input content display area 1610. Once the user has entered all the desired numbers, they touch the object 1603. This registers the input content displayed in the input content display area 1610. Unlike physical contact with the surface of a display device, touching the floating image 3 in space does not give the user a tactile sensation. Therefore, by separately displaying the input content in the input content display area 1610, the user can proceed with the operation while checking whether their touch operation was valid, which is advantageous.

On the other hand, if the user touches the wrong object or otherwise inputs something different from what they intended, they can erase the last input (here, "9") by touching object 1601. Then, the user continues to perform touch operations on objects for inputting numbers, etc. Once the user has input all of the desired numbers, they touch object 1603.

In this way, by displaying the input content in the input content display area 1610, the user can confirm the input content, thereby improving convenience. Furthermore, if the user touches the wrong object, the input content can be corrected, thereby improving convenience.
<<How to highlight input content to assist touch operation>>

Next, it is also possible to highlight the input content to assist with touch operations. Figure 38 is a diagram explaining how to highlight the input content to assist with touch operations.

Figure 38 shows an example in which a number entered by touch operation is highlighted. Referring to Figure 38, when the object corresponding to the number "6" is touched, the touched object is erased, and the entered number "6" is displayed in the area where this object was displayed.

In this way, by displaying a number corresponding to the touched object in place of the object, the user can recognize that they have touched an object, improving convenience. The number corresponding to the touched object may also be called a replacement object, which replaces the touched object.

Other methods for highlighting input content include, for example, brightly lighting the object touched by the user or blinking the object touched by the user. Although not shown here, by recognizing the distance between the finger 210 and the display surface 3a as described in the examples of Figures 27 and 28, the object being touched by the finger may become brighter than the surrounding objects as the finger approaches the display surface, and may reach its highest level of highlighting, light up even brighter, or blink when the finger finally reaches the display surface. Even with such a configuration, it is possible for the user to recognize that they have touched an object, thereby improving convenience.
<<Method for assisting touch operations with vibration (1)>>

Next, a method of assisting touch operations with vibration will be described. Figure 39 is a diagram illustrating an example of a method of assisting touch operations with vibration. Figure 39 shows a case where touch operations are performed using a touch pen (touch input device) 1700 instead of a finger 210. The touch pen 1700 is equipped with a communication unit that sends and receives various information such as signals and data to and from devices such as a space floating image display device, as well as a vibration mechanism that vibrates based on input signals.

Let's say the user operates the touch pen 1700 and touches an object displayed in the key input UI display area 1600 of the floating image 3 with the touch pen 1700. At this time, for example, the control unit 1100 transmits a touch detection signal from the communication unit 1132 indicating that a touch on the object has been detected. When the touch pen 1700 receives the touch detection signal, the vibration mechanism generates vibrations based on the touch detection signal. This causes the touch pen 1700 to vibrate. The vibration of the touch pen 1700 is then transmitted to the user, who recognizes that they have touched the object. In this way, the vibration of the touch pen 1700 assists the touch operation.

This configuration allows the user to recognize that they have touched an object through vibration.

Here, we have described a case where the touch detection signal transmitted from the floating-in-space image device is received by the touch pen 1700, but other configurations are also possible. For example, when a touch on an object is detected, the floating-in-space image display device notifies the host device that a touch on the object has been detected. The host device then transmits a touch detection signal to the touch pen 1700.

Alternatively, the space-floating image display device and the higher-level device may transmit the touch detection signal via a network. In this way, the touch pen 1700 may indirectly receive the touch detection signal from the space-floating image display device.
<<Method for assisting touch operations with vibration (2)>>

Next, we will explain another method of assisting touch operations using vibration. Here, the user is made aware that they have touched an object by vibrating the device they own. Figure 40 is a diagram illustrating another example of a method of assisting touch operations using vibration. In the example of Figure 40, a user 230 wearing a wristwatch-type wearable device 1800 performs a touch operation.

The wearable device 1800 is equipped with a communication unit that sends and receives various information such as signals and data to and from devices such as a floating image display device, as well as a vibration mechanism that vibrates based on input signals.

Let's say the user performs a touch operation with their finger 210 and touches an object displayed in the key input UI display area 1600 of the floating image 3. At this time, for example, the control unit 1100 transmits a touch detection signal from the communication unit 1132 indicating that a touch on the object has been detected. When the wearable device 1800 receives the touch detection signal, the vibration mechanism generates vibrations based on the touch detection signal. This causes the wearable device 1800 to vibrate. The vibrations of the wearable device 1800 are then transmitted to the user, who recognizes that they have touched the object. In this way, the vibrations of the wearable device 1800 assist the touch operation. Here, a wristwatch-type wearable device has been used as an example, but it may also be a smartphone worn by the user.

The wearable terminal 1800 may receive a touch detection signal from a higher-level device, similar to the touch pen 1700 described above. The wearable terminal 1800 may also receive a touch detection signal via a network. In addition to the wearable terminal 1800, it is also possible to assist touch operations using an information processing terminal such as a smartphone owned by the user.

According to this configuration, it is possible to make the user aware that he or she has touched an object via various terminals such as the wearable terminal 1800 owned by the user.
<<Method for assisting touch operations with vibration (3)>>

Next, another method of assisting touch operations with vibration will be described. Figure 41 is a diagram illustrating another example of a method of assisting touch operations with vibration. In the example of Figure 41, the user 230 performs a touch operation while standing on the vibration plate 1900. The vibration plate 1900 is placed in a predetermined position where the user 230 performs the touch operation. In actual use, the vibration plate 1900 is placed, for example, under a mat (not shown), and the user 230 stands on the vibration plate 1900 via the mat.

As shown in FIG. 41 , the diaphragm 1900 is connected to, for example, the communication unit 1132 of the space floating image display device 1000 via cable 1910. When a touch on an object is detected, for example, the control unit 1110 supplies an AC voltage to the diaphragm 1900 for a predetermined period of time via the communication unit 1132. The diaphragm 1900 vibrates while the AC voltage is being supplied. In other words, the AC voltage is a control signal output from the communication unit 1132 for vibrating the diaphragm 1900. The vibrations generated by the diaphragm 1900 are transmitted from the feet to the user 230, allowing the user 230 to recognize that they have touched an object. In this way, the vibration of the diaphragm 1900 assists the touch operation.

The frequency of the AC voltage is set to a value within the range in which the user 230 can sense vibrations. The vibration frequencies that humans can sense are in the range of approximately 0.1 Hz to 500 Hz. For this reason, it is desirable to set the frequency of the AC voltage within this range.

It is also desirable to change the frequency of the AC voltage as appropriate depending on the characteristics of the diaphragm 1900. For example, when the diaphragm 1900 vibrates vertically, people are said to be most sensitive to vibrations of around 410 Hz. When the diaphragm 1900 vibrates horizontally, people are said to be most sensitive to vibrations of around 12 Hz. Furthermore, at frequencies of 34 Hz or higher, people are said to be more sensitive to vibrations in the vertical direction than in the horizontal direction.

Therefore, when the diaphragm 1900 vibrates in the vertical direction, it is desirable that the frequency of the AC voltage be set to a value within a range including, for example, 410 Hz. Furthermore, when the diaphragm 1900 vibrates in the horizontal direction, it is desirable that the frequency of the AC voltage be set to a value within a range including, for example, 12 Hz. Note that the peak voltage and frequency of the AC voltage may be adjusted as appropriate depending on the performance of the diaphragm 1900.

This configuration allows the user 230 to recognize that an object has been touched by feeling a vibration from their feet. Furthermore, with this configuration, it is also possible to set the display of the floating image 3 not to change when an object is touched, reducing the chance that the input content will be revealed even if someone spies on the touch operation, further improving security.

Although various embodiments have been described in detail above, the present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments are detailed descriptions of the entire system in order to clearly explain the present invention, and are not necessarily limited to systems that include all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

1...Display device, 2...Retroreflective member, 3...Spatial image (space-floating image), 105...Window glass, 100...Transparent member, 101...Polarization separation member, 12...Absorptive polarizer, 13...Light source device, 54...Light direction conversion panel, 151...Retroreflective member, 102, 202...LED substrate, 203...Light guide, 205, 271...Reflective sheet, 206, 270...Retardation plate, 300...Space-floating image, 301...Ghost of space-floating image ghost image, 302...ghost image of floating image in space, 230...user, 1000...space floating image display device, 1110...control unit, 1160...image control unit, 1180...imaging unit, 1102...image display unit, 1350...air operation detection unit, 1351...air operation detection sensor, 1500...virtual light source, 1510...virtual shadow, 1610...input content display area, 1700...touch pen, 1800...wearable terminal, 1900...diaphragm.

Claims (10)

A space floating image display device,
a display device for displaying an image;
a retroreflective member that reflects image light from the display device and forms a floating image in the air using the reflected light;
a sensor for detecting the position of a user's finger performing a touch operation on one or more objects displayed on the floating image;
A control unit;
Equipped with
The control unit controls image processing for the image displayed on the display device based on the position of the user's finger detected using the sensor, so that the space-floating image display device displays a virtual shadow of the user's finger on the display surface of the space-floating image, which has no physical contact surface;
a position of the virtual shadow displayed on the display surface of the space floating image is a position specified from a positional relationship between a position of a virtual light source and a position of the user's finger detected using the sensor, and the position of the virtual light source is set at a position at an infinite distance from the display surface of the space floating image;
When the position of the tip of the user's finger approaches the display surface of the space floating image so that the distance in the normal direction to the display surface of the space floating image becomes small on the front side of the display surface of the space floating image as seen from the user, the display of the virtual shadow of the user's finger is continued until the distance in the normal direction between the position of the tip of the user's finger and the display surface of the space floating image becomes 0;
A floating image display device. 2. The space floating image display device according to claim 1,
When the position of the tip of the user's finger changes in the normal direction on the front side of the display surface of the space floating image as seen from the user, the position of the tip of the virtual shadow displayed on the space floating image in the left-right direction on the display surface of the space floating image changes.
A floating image display device. 3. The space floating image display device according to claim 2,
With respect to the change in the normal direction of the position of the tip of the user's finger, a left-right position of the tip of the virtual shadow displayed on the floating image in the space on the display surface of the floating image in the space changes linearly.
A floating image display device. 2. The space floating image display device according to claim 1,
an imaging unit that captures an image of a user's hand or arm;
the position of the virtual light source is changed depending on whether the finger of the user performing the touch operation is of the right hand or the left hand, so that when the finger of the user performing the touch operation on one or more objects displayed in the space floating image is of the right hand, the virtual shadow is displayed in the space floating image at a position to the left of the tip of the finger as seen from the user, and when the finger of the user performing the touch operation on one or more objects displayed in the space floating image is of the left hand, the virtual shadow is displayed in the space floating image at a position to the right of the tip of the finger as seen from the user;
A floating image display device. 2. The space floating image display device according to claim 1,
The control unit detects the position of the tip of the finger on the display surface of the space floating image and the height position of the tip of the finger relative to the display surface using the sensor that detects the position of the user's finger.
A floating image display device. 2. The space floating image display device according to claim 1,
The presence or absence of the user's finger touching the display surface of the floating image in space is detected by a sensor different from a sensor that detects the position of the user's finger.
A floating image display device. 2. The space floating image display device according to claim 1,
The position of the virtual light source is
A position where a virtual light source installation angle defined by the angle between a normal line extending from a center point of the display surface of the space floating image toward the user and a line connecting the virtual light source and the center point of the display surface of the space floating image is 20° or more.
A floating image display device. 2. The space floating image display device according to claim 1,
The angle of the extension direction of the virtual shadow displayed on the display surface of the space floating image changes in conjunction with the angle of the user's finger imaged by the imaging unit of the space floating image display device.
A floating image display device. 2. The space floating image display device according to claim 1,
The angle of the extension direction of the virtual shadow displayed on the display surface of the space floating image is a fixed angle, not linked to the angle of the user's finger imaged by the imaging unit of the space floating image display device,
A floating image display device. 2. The space floating image display device according to claim 1,
When a distance in the normal direction between the position of the tip of the user's finger and the display surface of the space floating image becomes 0, the virtual shadow is displayed on the space floating image so that the tip of the virtual shadow and the tip of the user's finger come into contact with each other in the left-right direction on the display surface of the space floating image.
A floating image display device.