JP7579504B2 - Display control device, head-up display device, and display control method - Google Patents

Description

The present disclosure relates to a display control device for a head-up display device that is used in a vehicle and that superimposes an image on the vehicle's foreground for viewing, as well as a head-up display device and a display control method.

Patent Document 1 describes a head-up display device that is optically designed to distribute display light toward the entire area called the eyebox, which includes most or all of the iris where the eyes of occupants (observers) of various builds seated in the driver's seat of a vehicle are expected to be located. According to this, if the occupant (observer) has their eyes positioned within the eyebox, they can view the image (virtual image) displayed by the head-up display device in the desired display mode (for example, within a desired brightness range).

JP 2012-203176 A

The head-up display device of Patent Document 1 directs display light toward the entire eyebox, so the display light directed toward other areas of the eyebox where the observer's eyes are not present is wasted. In other words, there is a problem in that the utilization efficiency of the display light emitted by the head-up display device (here, utilization efficiency is the ratio of the display light visible to the observer to the emitted display light) is low. From another perspective, there is also a problem in that power consumption is high.

A summary of certain embodiments disclosed herein is provided below. It should be understood that these aspects are presented merely to provide the reader with an overview of these certain embodiments, and are not intended to limit the scope of the disclosure. Indeed, the disclosure may encompass a variety of aspects that are not set forth below.

The present disclosure relates to improving light utilization efficiency. The present disclosure also relates to making it difficult for a viewer to view a low-quality image while improving light utilization efficiency.

Therefore, the display control device, head-up display device, and display control method described in this specification adopt the following means to solve the above problems. In this embodiment, the gist is that the area of the planar illumination unit that illuminates the light modulation element, from which illumination light of a predetermined light intensity or more is emitted, is changed according to the viewer's eye position.

In the display control device in the first aspect described in this specification, a light source element having a plurality of partial regions and capable of selectively emitting illumination light for each partial region, a light modulation element that modulates the illumination light incident from the light source element and emits it as display light, a focusing optical system that focuses the illumination light emitted from different partial regions, modulated by the light modulation element, toward different regions in an eye box for each display light, and a display control device that controls a head-up display device in which an observer views a virtual image from within the eye box by projecting the display light onto a projection portion, the light source element is controlled. The processor acquires information on the forward/backward eye position of the observer facing the projection target from the eye box, or information that allows the observer's forward/backward eye position to be estimated, and executes a light-emitting area change process that includes setting a partial area (hereinafter also referred to as a light-emitting area) that emits illumination light of a predetermined light intensity or more to a first light-emitting area when the observer's forward/backward eye position is at a first position, and setting the light-emitting area to a second light-emitting area that is larger than the first light-emitting area when the eye position is at a second position that is closer to the projection target than the first position.

According to the display control device in the first aspect, when the eye position is close to the projection target, the light-emitting area of the light source element is widened. In other words, the display control device in the first aspect widens the light-emitting area of the light source element when the eye position in the longitudinal direction of the vehicle on which the head-up display device is mounted is located relatively forward, compared to when the eye position is located rearward. According to this, when the eye position is close to the projection target, by widening the light-emitting area of the light source element, it is possible to suppress a decrease in the brightness of the virtual image viewed near the outer edge of the virtual image display area, and to display a virtual image that is easy to view. In addition, when the eye position is far from the projection target, by narrowing the light-emitting area of the light source element, it is possible to reduce the display light directed to areas other than the eye position (improving light utilization efficiency).

FIG. 1 is a diagram illustrating an example of a vehicle application of a vehicular display system according to some embodiments. FIG. 2 is a diagram illustrating a configuration of a head-up display device according to some embodiments. FIG. 3 is a diagram illustrating light paths in a head-up display device, according to some embodiments. FIG. 4A is a diagram illustrating an aspect of partial viewing zones that are virtually divided within an eyebox, and an aspect in which display light is directed to a plurality of partial viewing zones in accordance with the eye position of a viewer, according to some embodiments. FIG. 4B is a diagram illustrating a relationship between partial viewing areas of the eyebox and positions at which display light based on illumination light emitted from each partial region of the directional light source unit is concentrated in some embodiments. FIG. 4C is a diagram showing how an illumination laser beam is irradiated onto a light-emitting area of a light-diffusing optical system to direct the display light into the spot lighting pattern of FIG. 4A. FIG. 5A is a diagram illustrating an example of a distribution of light amount (luminance) when display light is distributed in a spot lighting pattern according to some embodiments. FIG. 5B is a diagram showing a state in which the eye position has moved after the state of FIG. 5A. FIG. 5C shows how, after the state of FIG. 5B, the spot lighting pattern has switched from one corresponding to the previous partial viewing zone to one corresponding to the new partial viewing zone E. FIG. 6 is a block diagram of a vehicle display system according to some embodiments. FIG. 7A is a flow diagram illustrating a method for performing operations to set a spot lighting pattern (the location where display light is distributed) based on an observer's eye position, according to some embodiments. FIG. 7B is a flow diagram illustrating the method that continues from FIG. 7A. FIG. 7C is a flow diagram illustrating the method that continues from FIG. 7B. FIG. 8A is a diagram illustrating an aspect of partial viewing zones that are virtually divided within an eyebox, and an aspect in which display light is directed to a plurality of partial viewing zones in accordance with the eye position of a viewer, according to some embodiments. FIG. 8B is a diagram illustrating a relationship between partial viewing areas of the eyebox and positions at which display light based on illumination light emitted from each partial region of the directional light source unit is concentrated in some embodiments. FIG. 9 is a diagram illustrating light paths in a head-up display device, according to some embodiments. FIG. 10 is a diagram showing an embodiment of the first light-emitting region and the second light-emitting region DM2 in the planar illumination unit. FIG. 11 is a diagram illustrating light paths in a head-up display device, according to some embodiments. FIG. 12 is a diagram showing an example of the light source array shown in FIG. FIG. 13A is a diagram showing one embodiment of a light emission pattern of a light source array. FIG. 13B is a diagram showing one embodiment of a light emission pattern of the light source array. FIG. 14A is a diagram showing a modified example of the relationship between the partial viewing area of the eyebox and the position where the display light based on the illumination light emitted from each partial area of the directional light source unit is concentrated in some embodiments. FIG. 14B is a diagram showing a modified example of the relationship between the partial viewing area of the eyebox and the position where the display light based on the illumination light emitted from each partial area of the directional light source unit is concentrated in some embodiments. FIG. 14C is a diagram showing a modified example of the relationship between the partial viewing area of the eyebox and the position where the display light based on the illumination light emitted from each partial area of the directional light source unit is concentrated in some embodiments. FIG. 14D is a diagram showing a modified example of the relationship between the partial viewing area of the eyebox and the position where the display light based on the illumination light emitted from each partial area of the directional light source unit is concentrated in some embodiments.

Below, in Figs. 1 to 14D, an explanation is provided of the configuration and operation of an exemplary vehicle display system, a head-up display device, and a display control device. Note that the present invention is not limited to the following embodiments (including the contents of the drawings). Of course, modifications (including the deletion of components) can be made to the following embodiments. In addition, in the following explanation, explanations of well-known technical matters will be omitted as appropriate in order to facilitate understanding of the present invention.

The vehicular display system 10 in FIG. 1 is mounted on a vehicle 1 and is composed of a head-up display device (hereinafter also referred to as HUD device) 20, a display control device 30 that controls the HUD device 20, and a projection unit 2 that includes a non-specular reflection element 3 (optional) described below and receives display light K from the HUD device 20. The HUD device 20 projects display light K toward the vehicle's front windshield (an example of the projection unit 2) that is located in front of an observer (typically a driver seated in the driver's seat of the vehicle) (the front can also be rephrased as the forward direction or the traveling direction of the vehicle).

The non-specular reflection element 3 can be composed of, for example, a holographic element or a diffractive optical element (DOE). The holographic element is an optical element made of, for example, a hologram material such as a photopolymer or dichromated gelatin, which diffracts incident light in a desired direction, and is created by recording an interference pattern between an object light and a reference light. The non-specular reflection element 3 is, for example, sheet-shaped and is contained in the projection target portion 2, and diffracts the display light K projected by the HUD device 20 to the eye box 200, which is a predetermined area inside the vehicle 1. For example, the non-specular reflection element 3 described later can diffract the display light K received from the HUD device 20 to increase the size of the image (virtual image) viewed by the observer. By placing the eye 700 in the eye box 200, the observer can view the virtual image V of the display image (not shown) displayed by the HUD device 20 on the far side of the projection target portion 2. As described above, the non-specular reflection element 3 may be omitted. In other words, the vehicle display system 10 does not allow the viewer to see non-specularly reflected light from the projected portion 2 (non-specular reflection element 3), but may allow the viewer to see specularly reflected light from the projected portion 2 (not including the non-specular reflection element 3).

The "eye box" used in the description of this embodiment is: (1) a region in which the entire virtual image V of the image is visible, and at least a part of the virtual image V of the image is not visible outside the region; (2) a region in which at least a part of the virtual image V of the image is visible, and at least a part of the virtual image V of the image is not visible outside the region; (3) a region in which at least a part of the virtual image V of the image is visible with a predetermined luminance or more, and at least a part of the virtual image V of the image is less than the predetermined luminance outside the region; or (4) a region in which at least a part of the virtual image V can be viewed stereoscopically, and at least a part of the virtual image V cannot be viewed stereoscopically outside the region, when the HUD device 20 is capable of displaying a virtual image V that can be viewed stereoscopically. In other words, when the observer places his/her eyes (both eyes) outside the eye box 200, the observer cannot view the entire virtual image V of the image, the visibility of the entire virtual image V of the image is very low and difficult to perceive, or the virtual image V of the image cannot be viewed stereoscopically. The predetermined brightness is, for example, about 1/50 of the brightness of the virtual image of the image viewed at the center of the eyebox. The "eyebox" is set to be the same as the area (also called the iris) where the observer's viewpoint is expected to be located in the vehicle in which the HUD device 20 is installed, or to include most of the iris (for example, 80% or more).

The virtual image display area VS is a flat, curved, or partially curved area where an image generated inside the HUD device 20 is formed as a virtual image V, and is also called an imaging surface. The virtual image display area VS is a position where a display surface (e.g., an exit surface of a liquid crystal light modulation element) 52 of a display 50 of the HUD device 20 described later is formed as a virtual image, that is, the virtual image display area VS corresponds to a display surface 52 of the HUD device 20 described later (in other words, the virtual image display area VS is in a conjugate relationship with the display surface 52 of the display 50 described later), and the virtual image viewed in the virtual image display area VS corresponds to an image displayed on the display surface 52 of the HUD device 20 described later. It is preferable that the virtual image display area VS itself has low visibility to the extent that it is not actually visible to the observer's eyes or is difficult to see.

The virtual image display area VS is set so that the angle (tilt angle θt in Figure 1) between the horizontal direction (XZ plane) and the axis extending along the left-right direction (X-axis direction) of the vehicle 1, the angle between the line segment connecting the center 205 of the eye box 200 and the upper end VS1 of the virtual image display area VS, and the angle between the line segment connecting the center of the eye box and the lower end VS2 of the virtual image display area VS are set as the vertical angle of view of the virtual image display area VS, and the angle (vertical arrangement angle θv in Figure 1) between the bisector of this vertical angle of view and the horizontal direction (XZ plane). The vertical arrangement angle θv is typically set downward from the horizontal direction (XZ plane), for example, +2 degrees (here, the positive direction of the vertical arrangement angle θv can be said to be a depression angle where the bisector of the vertical angle of view is oriented downward from the center 205 of the eye box 200, and the negative direction of the vertical arrangement angle θv can be said to be an elevation angle where the bisector of the vertical angle of view is oriented upward from the center 205 of the eye box 200).

In addition, the virtual image display region VS of this embodiment has a tilt angle θt of approximately 90 degrees so that it faces approximately directly when facing forward (positive direction of the Z axis). However, the tilt angle θt is not limited to this and can be changed within the range of 40≦θt<90 degrees. In this case, for example, the tilt angle θt may be set to 60 degrees, and the virtual image display region VS may be positioned so that the upper region is farther away than the lower region when viewed from the viewer.

2 is a diagram showing the configuration of the HUD device 20 of this embodiment. The HUD device 20 has a display 50 having a display surface 52 for displaying an image, a relay optical system (light-collecting optical system) 40, and a housing 22 that houses the display 50 and the relay optical system 40 and has a light exit window 21 that can emit display light K from the display 50 from the inside to the outside. The display 50 of this embodiment, which will be described later, is a 2D display, but may be a 3D display. That is, the HUD device 20 may display a parallax image corresponding to the eye points (left and right eyes) arranged in the eye box 200, allowing the observer to perceive a virtual image of a stereoscopic image based on the virtual image display area VS (the 3D display does not have to be a binocular parallax type 3D display). Also, a part of the relay optical system 40 may be omitted.

The display 50 in FIG. 2 is composed of a light modulation element 51 and a directional light source unit 60. The display surface 52 is the surface of the light modulation element 51 facing the observer, and emits display light K of an image. The angle of the virtual image display area VS (including the tilt angle θt) can be set by setting the angle of the display surface 52 with respect to the optical axis Kp of the display light K that travels from the center of the display surface 52 through the relay optical system 40 and the projection target 2 toward the eye box 200 (center 205 of the eye box 200). In addition, the directional light source unit 60, which will be described later, can change the position at which the virtual image V can be viewed (easily viewed) within the eye box 200 by changing the direction of the illumination light.

The relay optical system 40 is arranged on the optical path of the display light K (light traveling from the display 50 toward the eye box 200) emitted from the display 50, and is composed of one or more optical components that project the display light K from the display 50 onto the front windshield 2 outside the HUD device 20. The relay optical system 40 in FIG. 2 includes one concave first mirror 41 and one flat second mirror 42.

The first mirror 41 is, for example, a free-form surface shape having positive optical power. In other words, the first mirror 41 may be a curved surface shape with different optical powers for each region, that is, the optical power added to the display light K may differ depending on the region (optical path) through which the display light K passes. Specifically, the optical power added by the relay optical system 40 may differ between the first display light K1, the second display light K2, and the third display light K3 (see FIG. 2) traveling from each region of the display surface 52 toward the eye box 200.

The second mirror 42 is, for example, a flat mirror, but is not limited thereto, and may be a curved surface having optical power. That is, the relay optical system 40 may combine multiple mirrors (for example, the first mirror 41 and the second mirror 42 in this embodiment) to add different optical power depending on the area (optical path) through which the display light K passes. The second mirror 42 may be omitted. That is, the display light K emitted from the display 50 may be reflected by the first mirror 41 to the projection target portion (front windshield) 2.

In addition, in this embodiment, the relay optical system 40 includes two mirrors, but is not limited to this, and may include, in addition to or instead of these, one or more refractive optical elements such as lenses, diffractive optical elements such as holograms, reflective optical elements, or combinations of these.

The relay optical system 40 of this embodiment has the function of setting the distance to the virtual image display area VS and the function of generating a virtual image that is an enlarged image of the image displayed on the display surface 52 by using this curved surface shape (an example of optical power), but in addition to this, it may also have the function of suppressing (correcting) distortion of the virtual image that may occur due to the curved shape of the front windshield 2.

The relay optical system 40 may also be fitted with one or more actuators (not shown) that are controlled by the display control device 30, allowing it to rotate (and/or move).

The light modulation element 51 is, for example, an LCD, which is a transmissive display panel, and receives illumination light from the directional light source unit 60 and emits spatially light-modulated display light K toward the relay optical system 40 (second mirror 42). The light modulation element 51 is, for example, rectangular in shape, with the short side being the direction in which pixels corresponding to the vertical direction (Y-axis direction) of the virtual image V seen by the observer are arranged, and the long side being the direction in which pixels corresponding to the horizontal direction (X-axis direction) of the virtual image V seen by the observer that is perpendicular to the short side are arranged. The observer visually recognizes the transmitted light of the light modulation element 51 (reflected light, diffracted light, etc. may be considered depending on the type of the light modulation element 51) through the virtual image optical system 90. The virtual image optical system 90 is a combination of the relay optical system 40 shown in FIG. 2 and the front windshield 2 (non-regular reflection element 3 included therein). The light modulation element 51 (display 50) may be a reflective display panel using LCoS or DMD.

(First embodiment)
Please refer to FIG. 3. FIG. 3 is a diagram showing an example of a main configuration of the HUD device 20 (the directional light source unit 60 of the first embodiment) having a function of performing the viewpoint tracking spotlighting control. In the HUD device 20 (the directional light source unit 60) shown in FIG. 3, the eye box 200 is divided into n (25 in this embodiment) partial viewing zones E (E(1,1) to E(5,5) in FIG. 4A) which are divided into five in the left-right direction (X-axis direction) when the observer (driver of the vehicle) faces the front of the vehicle and five in the up-down direction (Y-axis direction). The HUD device 20 (the directional light source unit 60) performs viewpoint tracking spotlighting control in which the light emission pattern is changed for each partial viewing zone E where the eye position of the driver (user) is detected, and the display light K is locally directed toward the partial viewing zone E where the eye position of the driver (user) is detected.

The display control device 30, which will be described later, detects the eye position of the observer using the eye position detection unit 411, which will be described later, and controls the directional light source unit 60 so that a bright image (virtual image V) is visible only at the observer's eye position and its surroundings. These will be described in detail later.

The directional light source unit (light source element) 60 of the first embodiment is composed of a light source 70, a scanning device (scanner) 75, a first field lens (illumination optical system) 76, a second field lens (illumination optical system) 77, and a light diffusion optical system 80. The scanning device 75 can scan the laser light emitted from the light source 70 over m (m>n) partial areas D (121 partial areas D in this embodiment from D(1,11) to D(11,11)) of the light diffusion optical system 80, and the display control device 30 controls the light source 70 to irradiate strong laser light onto the partial area D (light-emitting area DM) corresponding to the partial viewing area E where the eye position detected by the eye position detection unit 411 exists, thereby allowing the observer to view an image (virtual image V). The light-emitting area DM is a partial area D that emits illumination light of a predetermined light intensity toward the light modulation element 51.

The light source 70 is, for example, a plurality of laser diodes each capable of emitting three different colors (e.g., R, G, B), and the plurality of laser beams are mixed (e.g., white) using a dichroic mirror or the like, and directed to the scanning device 75 as a single illumination laser beam 74. The light source 70 is capable of adjusting the intensity of the illumination laser beam 74 (including turning it off) under the control of the display control device 30 described below.

The scanning device (scanner) 75 is an element that two-dimensionally deflects the illumination laser beam 74 emitted from the light source 70, and is an optical scanning device such as a polygon mirror, a galvanometer mirror, or a MEMS mirror device using MEMS (Micro Electro Mechanical System) technology. The scanning device 75 is a tiny mirror that is configured to be able to oscillate using two axes that are perpendicular to each other. The scanning device 75, for example, scans at high speed in the main scanning direction and at low speed in the sub-scanning direction that is perpendicular to the main scanning direction.

The first field lens 76 is disposed on the optical path of the illumination laser beam 74 scanned by the scanning device 75 between the scanning device 75 and the optical diffusion optical system 80. The first field lens 76 has the function of collimating the received illumination laser beam 74 in a direction parallel to the optical axis of the first field lens 76. The first field lens 76 is not an essential component and may be omitted. However, by providing the first field lens 76, the display light K obtained by modulating the illumination laser beam 74 emitted from each light-emitting region DM of the optical diffusion optical system 80 can be directed accurately to each partial viewing area E of the eye box 200. The first field lens 76 is a refractive optical system, but instead of this, one or more other refractive optical systems, reflective optical systems, or diffractive optical systems may be provided on the optical path of the illumination laser beam 74 between the scanning device 75 and the optical diffusion optical system 80.

The second field lens (light-collecting optical system) 77 is disposed on the optical path of the illumination laser beam 74 diffused by each partial region D (light-emitting region DM) between the light diffusion optical system 80 and the light modulation element 51, and functions as a light-collecting optical system that collects the illumination light emitted from the different (light-emitting region DM) toward different regions in the eye box 200 for each display light K modulated by the light modulation element 51. The second field lens 77 of this embodiment is a front-stage optical member (backlight optical member) composed of one or more field lenses, and is designed so that the illumination laser beam 74 diffused by each partial region D (light-emitting region DM) of the light diffusion optical system 80 transmits and illuminates the entire surface of the light modulation element 51. The second field lens 77 is also designed so that the display light K emitted from the light modulation element 51 reaches a partial viewing area E at any position in the eye box 200 via the virtual image optical system 90 (HUD optical member composed of the relay optical system 40 and the front windshield 2). The focusing optical system that focuses the display light K emitted from the different partial regions D (light-emitting regions DM) and modulated by the light modulation element 51 toward different partial viewing zones E in the eye box 200 may include various optical systems such as one or more refractive optical systems, reflective optical systems, and diffractive optical systems instead of (or in addition to) the field lens. Note that the display light K passes through the relay optical system 40 and the projection target 2 before reaching the eye box 200 from the light modulation element 51. At this time, the display light K is deflected to a certain extent by the relay optical system 40 and the projection target 2 before being focused toward different regions in the eye box 200. Therefore, it can be said that the focusing optical system includes the relay optical system 40 and the projection target 2 in addition to the second field lens (focusing optical system) 77. Also, the focusing optical system may be configured with the relay optical system 40 and the projection target 2 without the second field lens (focusing optical system) 77.

The diffuser plate 78 is disposed on the optical path of the illumination laser beam 74 between the light modulation element 51 and the second field lens 77. The diffuser plate 78 diffuses the received illumination laser beam 74 and emits it to the light modulation element 51. This reduces uneven brightness in the display light K generated by the illumination laser beam 74 emitted from the light-emitting region DM that is viewed in each partial viewing zone E of the eye box 200. The diffuser plate 78 may be any optical member that has the function of diffusing light, and may have a surface that is made of a bead member, a fine uneven structure, or a rough surface, for example. It may also be a dot sheet or a transparent milky white sheet.

First, an example in which a lens array 81 is used as the light diffusion optical system 80 will be described. The lens array 81 of this embodiment is divided into m partial regions D, and a micro lens having a curvature is formed in each partial region D. When an illumination laser beam 74 is incident on each partial region D, the illumination laser beam 74 refracted by the micro lens is emitted as diffused light. More specifically, the illumination laser beam 74 incident on any of the m partial regions D of the lens array 81 (light diffusion optical system 80) from the light source 70 is refracted by the micro lens, and after passing through the second field lens 77 and the diffusion plate 78, is irradiated on the entire surface of the light modulation element 51, and the optically modulated display light K passes through the virtual image optical system 90 and is irradiated on a predetermined region of the eye box 200 including at least the partial region D where the eye position 700 is present.

Next, an example in which a hologram recording medium 82 is used as the light diffusion optical system 80 will be described. The hologram recording medium 82 of this embodiment is divided into m partial regions D, and interference fringes are formed in each partial region D. When an illumination laser beam 74 is incident on each partial region D, the illumination laser beam 74 diffracted by the interference fringes is emitted as diffused light. More specifically, the illumination laser beam 74 incident on any of the m partial regions D of the hologram recording medium 82 (light diffusion optical system 80) from the light source 70 is diffracted by the interference fringes, and after passing through the second field lens 77 and the diffusion plate 78, is irradiated onto the entire surface of the light modulation element 51, and the optically modulated display light K passes through the virtual image optical system 90 and is irradiated onto a predetermined region of the eye box 200 corresponding to the predetermined partial region D. The light diffusion optical system 80 may be a Fresnel lens.

Figure 4C is a diagram showing how the scanning device (scanner) 75 scans the light diffusion optical system 80. In this Figure 4C, the first field lens 76 is omitted. The light diffusion optical system 80 has a third direction (α axis) corresponding to the first direction (X axis) in which the partial viewing areas E of the eye box 200 are arranged horizontally, and a fourth direction (β axis) corresponding to the second direction (Y axis) in which the partial viewing areas E are arranged vertically. The scanning device 75 performs main scanning along the third direction (α axis) and sub-scanning along the fourth direction (β axis). In the example of Figure 4C, the scanning device 75 passes through different partial areas D on the outward and return paths of the main scanning, and irradiates a predetermined partial area D with the illumination laser beam 74 under the control of the display control device 30. Specifically, the scanning device 75 irradiates the illumination laser beam 74 on a predetermined partial area D (light-emitting area DM) selected by the display control device 30 while scanning at high speed from partial area D (11,1) to D (1,1) on the forward path, and does not irradiate the illumination laser beam 74 on a partial area D not selected by the display control device 30, and then, on the return path, irradiates the illumination laser beam 74 on the predetermined partial area D (light-emitting area DM) selected by the display control device 30 while scanning from partial area D (1,2) to D (11,2). However, the manner in which the scanning device (scanner) 75 scans the light diffusion optical system 80 is not limited to this.

In another example, the scanning device 75 may irradiate the illumination laser beam 74 only on the forward path (or, alternatively, only on the return path). Specifically, the scanning device 75 may irradiate the illumination laser beam 74 on a predetermined partial area D (light-emitting area DM) selected by the display control device 30 while scanning from partial area D (11,1) to D (1,1) on the forward path, and then scan to partial area D (11,2) without irradiating the illumination laser beam 74, and then irradiate the illumination laser beam 74 on the predetermined partial area D (light-emitting area DM) selected by the display control device 30 while scanning from partial area D (11,2) to D (2,2) on the forward path again.

In another example, the scanning device 75 may irradiate the illumination laser beam 74 to the same partial area D on the forward and return paths. Specifically, the scanning device 75 irradiates the illumination laser beam 74 to a predetermined partial area D (light-emitting area DM) selected by the display control device 30 while scanning from partial area D (11,1) to D (1,1) on the forward path, and then irradiates the illumination laser beam 74 to the predetermined partial area D (light-emitting area DM) selected by the display control device 30 while scanning again from partial area D (1,1) to D (11,1) on the return path. Note that the position in the partial area D to which the illumination laser beam 74 is irradiated differs between the forward and return paths. In other words, the illumination laser beam 74 may be irradiated to an upper position (negative direction of β) of the predetermined partial area D on the forward path, and irradiated to a lower position (positive direction of β) of the predetermined partial area D on the return path.

4A is a diagram illustrating partial viewing zones that are virtually divided within the eye box, and a state in which display light is directed to a plurality of partial viewing zones corresponding to the eye positions of the observer. Here, it is assumed that the observer's right eye 700R is detected in partial viewing zone E(2,3), and the observer's left eye 700L is detected in partial viewing zone E(4,3). The partial viewing area E (hereinafter also referred to as the right-eye spotlighting pattern SPR) to which the right-eye display light K is directed is a horizontally elongated elliptical area centered on the partial viewing area E(2,3) and spanning nine partial viewing areas E: the upper part of the upper partial viewing area E(2,2), the lower part of the lower partial viewing area E(2,4), the right part of the right partial viewing area E(1,3), the left part of the left partial viewing area E(3,3), the upper left part of the upper left partial viewing area E(3,2), the lower left part of the lower left partial viewing area E(3,4), the lower right part of the lower right partial viewing area E(1,4), and the upper right part of the upper right partial viewing area E(1,2). Similarly, the partial viewing area E to which the left-eye display light K is directed (hereinafter also referred to as the left-eye spotlighting pattern SPL) is a horizontally elongated elliptical area that is centered on the partial viewing area E(4,3) and spans nine partial viewing areas E, including the upper part of the upper partial viewing area E(4,2), the lower part of the lower partial viewing area E(4,4), the right part of the right partial viewing area E(3,3), the left part of the left partial viewing area E(5,3), the upper left part of the upper left partial viewing area E(5,2), the lower left part of the lower left partial viewing area E(5,4), the lower right part of the lower right partial viewing area E(3,4), and the upper right part of the upper right partial viewing area E44(3,2).

Figure 4B is a diagram that shows the relationship between the state of the partial viewing zone E of the eye box 200 and the position where the display light K based on the illumination light emitted from each partial region D of the directional light source unit 60 is condensed. In FIG. 4B, the position where the display light K based on the illumination light emitted from each partial region D is condensed is drawn with a dot (black circle), but it shows the position where the main ray of the display light K is directed, and the display light K is also irradiated around the dot (black circle). In the example of FIG. 4B, the directional light source unit 60 arranges the partial regions D so that the display light K can be irradiated at nine different points in one partial viewing zone E. Specifically, there is one point in the center of the partial viewing zone E, four points at the four corners of the partial viewing zone E, and four points at the midpoints of each of the four corners of the partial viewing zone E, for a total of nine points. In one aspect shown in FIG. 4B, the display control device 30 controls the directional light source unit 60 so that the display light K is irradiated at all nine different points in the partial viewing zone E where the eye position 700 exists. The light-emitting region DM includes a main partial region DA that emits illumination light that is the source of display light K that is condensed at the center of the partial viewing region E where the eye position 700 is located, and a sub-partial region DB that emits illumination light that is the source of display light K that is condensed at a position farther from the center of the partial viewing region E where the eye position 700 is located than the main partial region DA. That is, when the left eye position 700L is located in the partial viewing region E(4,3), the main partial region DA is the partial region D(8,6), and the sub-partial regions DB are the partial regions D(7,5), D(8,5), D(9,5), D(7,6), D(9,6), D(7,7), D(8,7), and D(9,7). On the other hand, when the right eye position 700R is in the partial viewing zone E(2,3), the main partial area DA is the partial area D(4,6), and the sub partial areas DB are the partial areas D(3,5), D(4,5), D(5,5), D(3,6), D(5,6), D(3,7), D(4,7), and D(5,7).

Figure 4C is a diagram showing how the illumination laser beam 74 is irradiated onto the light-emitting region DM of the light diffusion optical system 80 in order to direct the display light K to the spot lighting pattern SP of Figure 4A. The display control device 30 drives the light source 70 in accordance with the timing at which the scanning device 75 scans the light-emitting region DM that emits illumination light of a predetermined light intensity among the m partial regions D. As a result, as shown in Figure 4A, the display light K of the light modulation element 51 is directed to the spot lighting pattern SP (spot lighting pattern SPR for the right eye, spot lighting pattern SPL for the left eye) of the above-mentioned eye box 200 corresponding to the light-emitting region DM of the light diffusion optical system 80. Note that the light-emitting region DM corresponding to the above-mentioned right-eye spot lighting pattern SPR is referred to as the right-eye light-emitting region DMR, and the light-emitting region DM corresponding to the above-mentioned left-eye spot lighting pattern SPL is referred to as the left-eye light-emitting region DML. Therefore, the display light K obtained by modulating the illumination light emitted from the right-eye light-emitting region DMR by the light modulation element 51 is irradiated onto the right-eye spotlighting pattern SPR, and the display light K obtained by modulating the illumination light emitted from the left-eye light-emitting region DML by the light modulation element 51 is irradiated onto the left-eye spotlighting pattern SPL. In addition, the display control device 30 does not irradiate the illumination laser beam 74, or irradiates an illumination laser beam 74 with a very low light intensity, onto partial regions D other than the right-eye light-emitting region DMR and the left-eye light-emitting region DML. This allows the display light K to be efficiently irradiated onto the viewing zone E where the observer's eyes are located, and power consumption to be kept low.

FIG. 5A is a diagram showing an example of the distribution of light quantity (luminance) when display light K is distributed in the spot lighting pattern SP. When the eye position 700 is estimated to be in the partial viewing zone E (2, 3), the display control device 30 drives the light source 70 in accordance with the timing at which the scanning device 75 scans the light emitting region DM so that the illumination laser beam 74 is irradiated onto the light emitting region DM (partial regions D (3, 5), D (4, 5), D (5, 5), D (3, 6), D (4, 6), D (5, 6), D (3, 7), D (4, 7), D (5, 7)) of the light diffusion optical system 80 to form a predetermined spot lighting pattern SP associated with the partial viewing zone E (2, 3). As a result, as shown in FIG. 4A, the display light K of the light modulation element 51 is directed to the predetermined spot lighting pattern SP of the eye box 200 corresponding to the light emitting region DM of the light diffusion optical system 80. The symbol DB1 (DB) in FIG. 5A indicates the light amount distribution in the left-right direction (X-axis direction) of the eye box 200 (the same applies to FIGS. 5B and 5C described later). The light quantity distribution Ld1 is a combination of the display light based on the illumination light emitted from the light emitting region DM (partial regions D(3,5), D(4,5), D(5,5), D(3,6), D(4,6), D(5,6), D(3,7), D(4,7), D(5,7), and a light quantity of a predetermined threshold Lth (for example, 50% of the maximum light quantity (brightness) in the spot lighting pattern SP) or more is ensured within the boundary of the spot lighting pattern SP. In addition, the light quantity distribution Ld extends beyond the boundary of the spot lighting pattern SP of the partial viewing area E and spreads, and the light intensity does not decrease sharply but gradually (with a gradient). Therefore, even when the viewpoint moves outside the boundary, the display (virtual image) becomes faint and disappears, and does not disappear suddenly. This suppresses (reduces) discomfort and anxiety.

In this embodiment, the sub-partial region DB emits illumination light that is the source of the display light K that is condensed at a position away from the center of the partial viewing zone E (2, 3). In particular, in the aspects shown in Figures 5A, 5B, and 5C, the sub-partial region DB is arranged so that all of the display light K is condensed at the boundary of the partial viewing zone E (2, 3), so that even if the eyes move near (or outside) the partial viewing zone E, the image (virtual image V) with relatively high brightness due to the sub-partial region DB can be viewed. Figure 5B shows the case where the eye position 700 moves from the partial viewing zone E (2, 3) to E (3, 3) after the state of Figure 5A. Even if system latency occurs and the spot lighting pattern SP does not switch (remains as the spot lighting pattern SP corresponding to the partial viewing area E(2,3)) at the moment of moving from the partial viewing area E(2,3) to E(3,3), the display light K with a relatively high brightness can be viewed by the light from the sub-partial area DB that emits the illumination light that is the source of the display light K that is condensed at a position away from the center of the partial viewing area E(2,3). FIG. 5C is a diagram showing the state in which the spot lighting pattern SP has switched from the one corresponding to the previous partial viewing area E(2,3) to the one corresponding to the new partial viewing area E(3,3) after the state of FIG. 5B.

In another preferred example, uniform (including substantially uniform) light is irradiated within the spot lighting pattern SP to achieve a predetermined brightness level. Specifically, the light emitting regions DM are arranged, the emitted light intensity is adjusted, and/or the entire or partial optical adjustment of the focusing optical system is performed so that the light quantity distribution Ld has a top hat shape. This allows the image (virtual image V) to be viewed with a substantially uniform brightness within the illuminated partial viewing zone E, even if the eye position moves. However, in this embodiment, the light quantity distribution Ld may be a Gaussian distribution.

In some embodiments, the display control device 30 (processor 33) may set the emission intensity of the sub-partial region DB to be greater than 0.7 times the emission intensity of the main partial region DA. In this way, since the emission intensity of the sub-partial region DB is relatively strong, even when the eye position 700 is near the boundary of the partial viewing zone E, the light from the sub-partial region DB can cause the relatively bright display light K to be visible.

In addition, in some embodiments, the display control device 30 (processor 33) may set the emission intensity of the sub-partial region (DB) to be higher than the emission intensity of the main partial region (DA). In this way, since the emission intensity of the sub-partial region DB is relatively strong, even when the eye position 700 is near the boundary of the partial viewing zone E, the light from the sub-partial region DB can cause the relatively bright display light K to be visible.

Figure 6 is a block diagram of a vehicle display system 10 according to some embodiments. The display controller 30 includes one or more I/O interfaces 31, one or more processors 33, one or more image processing circuits 35, and one or more memories 37. The various functional blocks described in Figure 6 may be configured in hardware, software, or a combination of both. Figure 6 is only one embodiment, and the components shown may be combined into fewer components or there may be additional components. For example, the image processing circuitry 35 (e.g., a graphics processing unit) may be included in one or more processors 33.

As shown, the processor 33 and the image processing circuit 35 are operably coupled to the memory 37. More specifically, the processor 33 and the image processing circuit 35 execute the programs stored in the memory 37 to operate the vehicle display system 10 (HUD device 20, directional light source unit 60), for example, by controlling the directional light source unit 60 (the control of the directional light source unit 60 includes, for example, control of the illumination laser beam 74 (illumination light) emitted by the light source 70, or control of the light source array 80A described later), generating and/or transmitting image data, etc. The processor 33 and/or the image processing circuit 35 may include at least one general-purpose microprocessor (e.g., a central processing unit (CPU)), at least one application specific integrated circuit (ASIC), at least one field programmable gate array (FPGA), or any combination thereof. The memory 37 may include any type of magnetic medium such as a hard disk, any type of optical medium such as a CD and a DVD, any type of semiconductor memory such as a volatile memory, and a non-volatile memory. Volatile memory may include DRAM and SRAM, and non-volatile memory may include ROM and NVRAM.

As shown in the figure, the processor 33 is operably connected to the I/O interface 31. The I/O interface 31 communicates (also referred to as CAN communication) with, for example, a vehicle ECU 401 (described later) or other electronic devices (reference numerals 403 to 419 described later) provided in the vehicle according to the CAN (Controller Area Network) standard. Note that the communication standard adopted by the I/O interface 31 is not limited to CAN, and includes, for example, wired communication interfaces such as CANFD (CAN with Flexible Data Rate), LIN (Local Interconnect Network), Ethernet (registered trademark), MOST (Media Oriented Systems Transport: MOST is a registered trademark), UART, or USB, or in-vehicle communication (internal communication) interfaces that are short-range wireless communication interfaces within several tens of meters, such as personal area networks (PANs) such as Bluetooth (registered trademark) networks and local area networks (LANs) such as 802.11x Wi-Fi (registered trademark) networks. The I/O interface 31 may also include an external communication (external communication) interface such as a wide area communication network (e.g., an Internet communication network) based on a wireless wide area network (WWAN0, IEEE 802.16-2004 (WiMAX: Worldwide Interoperability for Microwave Access)), IEEE 802.16e-based (Mobile WiMAX), 4G, 4G-LTE, LTE Advanced, 5G, or other cellular communication standards.

As shown in the figure, the processor 33 is interoperably connected to the I/O interface 31, thereby enabling information to be exchanged with various other electronic devices connected to the vehicle display system 10 (I/O interface 31). The I/O interface 31 is operably connected to, for example, a vehicle ECU 401, a road information database 403, a vehicle position detection unit 405, an external sensor 407, an operation detection unit 409, an eye position detection unit (gaze direction detection unit) 411, an IMU 413, a brightness sensor 415, a mobile information terminal 417, and an external communication device 419. The I/O interface 31 may include a function for processing (converting, calculating, analyzing) information received from other electronic devices connected to the vehicle display system 10.

The light modulation element 51 is, for example, a liquid crystal light modulation element, and is operably coupled to the processor 33 and the image processing circuitry 35. Thus, the image displayed by the light modulation element 51 may be based on image data received from the processor 33 and/or the image processing circuitry 35. The processor 33 and the image processing circuitry 35 control the image displayed by the light modulation element 51 based on information obtained from the I/O interface 31.

The vehicle ECU 401 obtains the state of the vehicle 1 (e.g., mileage, vehicle speed, accelerator pedal opening, brake pedal opening, engine throttle opening, injector fuel injection amount, engine RPM, motor RPM, steering angle, shift position, drive mode, various warning states, posture (including roll angle and/or pitch angle), vehicle vibration (including vibration magnitude, frequency, and/or frequency)) from sensors and switches provided on the vehicle 1, and collects and manages (which may also include control) said state of the vehicle 1, and as part of its function, can output a signal indicating a numerical value of said state of the vehicle 1 (e.g., the vehicle speed of the vehicle 1) to the processor 33 of the display control device 30. In addition to or instead of simply transmitting a value detected by a sensor or the like (e.g., the pitching angle is 3 degrees in the forward tilt direction) to the processor 33, the vehicle ECU 401 may transmit to the processor 33 a determination result based on one or more states of the vehicle 1 including the value detected by the sensor (e.g., that the vehicle 1 satisfies a predetermined condition for a forward tilt state) and/or an analysis result (e.g., that the vehicle has been tilted forward by braking in combination with information on the brake pedal opening degree). For example, the vehicle ECU 401 may output to the display control device 30 a signal indicating a determination result that the vehicle 1 satisfies a predetermined condition previously stored in a memory (not shown) of the vehicle ECU 401. The I/O interface 31 may acquire the above-mentioned information from a sensor or switch provided in the vehicle 1 without going through the vehicle ECU 401.

In addition, the vehicle ECU 401 may output an instruction signal to the display control device 30 to indicate the image to be displayed by the vehicle display system 10, and in this case, the coordinates, size, type, display mode, notification necessity of the image, and/or necessity-related information that is the basis for determining the notification necessity of the image may be added to the instruction signal and transmitted.

The road information database 403 is included in a navigation device (not shown) provided in the vehicle 1, or an external server connected to the vehicle 1 via an external communication interface (I/O interface 31), and may read out information about the surroundings of the vehicle 1 (information related to real objects around the vehicle 1), such as road information (lanes, white lines, stop lines, crosswalks, road widths, number of lanes, intersections, curves, forks, traffic regulations, etc.) on which the vehicle 1 is traveling, and information about features (buildings, bridges, rivers, etc.), including their presence or absence, position (including distance to the vehicle 1), direction, shape, type, detailed information, etc., based on the position of the vehicle 1 acquired from the vehicle position detection unit 405 described later, and transmit this information to the processor 33. The road information database 403 may also calculate an appropriate route (navigation information) from the departure point to the destination, and output a signal indicating the navigation information or image data indicating the route to the processor 33.

The vehicle position detection unit 405 is a GNSS (Global Navigation Satellite System) or the like provided in the vehicle 1, which detects the current position and orientation of the vehicle 1 and outputs a signal indicating the detection result to the road information database 403, the mobile information terminal 417 described later, and/or the external communication device 419 via the processor 33 or directly. The road information database 403, the mobile information terminal 417 described later, and/or the external communication device 419 may select/generate information about the surroundings of the vehicle 1 by acquiring the position information of the vehicle 1 from the vehicle position detection unit 405 continuously, intermittently, or for each predetermined event, and output the information to the processor 33.

The external sensor 407 detects real objects present around the vehicle 1 (in front, to the sides, and behind). The real objects detected by the external sensor 407 may include, for example, obstacles (pedestrians, bicycles, motorcycles, other vehicles, etc.), the road surface of the driving lane described below, dividing lines, roadside objects, and/or features (buildings, etc.). The external sensor is composed of, for example, a detection unit consisting of a radar sensor such as a millimeter wave radar, ultrasonic radar, or laser radar, a camera, or a combination of these, and a processing device that processes (data fuses) the detection data from the one or more detection units. Conventional well-known methods are applied to object detection by these radar sensors and camera sensors. By object detection by these sensors, the presence or absence of a real object in a three-dimensional space, and if a real object exists, the position of the real object (relative distance from the vehicle 1, position in the left-right direction when the traveling direction of the vehicle 1 is the forward-rear direction, position in the up-down direction, etc.), size (size in the lateral direction (left-right direction), height direction (up-down direction), etc.), movement direction (lateral direction (left-right direction), depth direction (front-rear direction)), movement speed (lateral direction (left-right direction), depth direction (front-rear direction)), and/or type, etc. may be detected. One or more external sensors 407 can detect real objects in front of the vehicle 1 at each detection period of each sensor, and output real object information (presence or absence of a real object, and if a real object exists, information on the position, size, and/or type of each real object, etc.) which is an example of real object information, to the processor 33. Note that the real object information may be transmitted to the processor 33 via another device (for example, the vehicle ECU 401). In addition, when a camera is used as a sensor, an infrared camera or a near-infrared camera is preferable so that a real object can be detected even when the surroundings are dark, such as at night. Also, if a camera is used as a sensor, a stereo camera that can obtain distance and other information using parallax is desirable.

The operation detection unit 409 is, for example, a hardware switch provided on the CID (Center Information Display) or instrument panel of the vehicle 1, or a software switch that combines an image and a touch sensor, and outputs operation information based on an operation by an occupant of the vehicle 1 (a user sitting in the driver's seat and/or a user sitting in the passenger seat) to the processor 33. For example, the operation detection unit 409 outputs to the processor 33 display area setting information based on an operation to move the virtual image display area VS, eye box setting information based on an operation to move the eye box 200, image brightness setting information based on an operation to change the brightness of the virtual image, information based on an operation to set the observer's eye position 700, and the like, by the user's operation.

The eye position detection unit 411 may include a camera such as an infrared camera that detects the eye position 700 of an observer seated in the driver's seat of the vehicle 1, and may output the captured image to the processor 33. The processor 33 may obtain the captured image (an example of information that can estimate the eye position 700) from the eye position detection unit 411, and may analyze the captured image using a method such as pattern matching to detect the coordinates of the observer's eye position 700, and may output a signal indicating the coordinates of the detected eye position 700 to the processor 33.

The eye position detection unit 411 may also output the analysis results of the image captured by the camera (e.g., a signal indicating to which spatial region the observer's eye position 700 belongs to in relation to a plurality of pre-set display parameters 600 (described later)). Note that the method of acquiring the observer's eye position 700 in the vehicle 1 or information that can estimate the observer's eye position 700 is not limited to these, and may be acquired using known eye position detection (estimation) techniques.

The eye position detection unit 411 may also detect the speed and/or direction of movement of the observer's eye position 700 and output a signal indicating the speed and/or direction of movement of the observer's eye position 700 to the processor 33.

In addition, when the eye position detection unit 411 detects (10) a signal indicating that the observer's eye position 700 is outside the eye box 200, (20) a signal that estimates that the observer's eye position 700 is outside the eye box 200, or (30) a signal that predicts that the observer's eye position 700 will be outside the eye box 200, it may determine that a predetermined condition is met and output a signal indicating that state to the processor 33.

(20) Signals indicating that the observer's eye position 700 is estimated to be outside the eye box 200 include (21) a signal indicating that the observer's eye position 700 cannot be detected, (22) a signal indicating that the observer's eye position 700 cannot be detected after movement of the observer's eye position 700 has been detected, and/or (23) a signal indicating that either the observer's eye position 700R, 700L is in the vicinity of the boundary 200A of the eye box 200 (the vicinity includes, for example, within a predetermined coordinate range from the boundary 200A).

(30) Signals predicting that the observer's eye position 700 will be outside the eye box 200 include (31) a signal indicating that the newly detected eye position 700 is equal to or greater than the eye position movement distance threshold pre-stored in memory 37 relative to the previously detected eye position 700 (the movement of the eye position within a specified unit time is greater than a specified range), and (32) a signal indicating that the movement speed of the eye position is equal to or greater than the eye position movement speed threshold pre-stored in memory 37.

The eye position detection unit 411 may also have a function as a gaze direction detection unit 411. The gaze direction detection unit 411 may include an infrared camera or a visible light camera that captures an image of the face of an observer sitting in the driver's seat of the vehicle 1, and output the captured image to the processor 33. The processor 33 can obtain an image (an example of information that can estimate the gaze direction) from the gaze direction detection unit 411 and identify the observer's gaze direction (and/or the gaze position) by analyzing the captured image. The gaze direction detection unit 411 may analyze the captured image from the camera and output a signal indicating the observer's gaze direction (and/or the gaze position) as the analysis result to the processor 33. Note that the method of acquiring information that can estimate the gaze direction of an observer of vehicle 1 is not limited to these, and may be acquired using other known gaze direction detection (estimation) techniques, such as the EOG (Electro-oculogram) method, the corneal reflex method, the scleral reflex method, the Purkinje image detection method, the search coil method, and the infrared fundus camera method.

The IMU 413 may include a combination of one or more sensors (e.g., an accelerometer and a gyroscope) configured to detect the position, orientation, and changes therein (speed of change, acceleration of change) of the vehicle 1 based on inertial acceleration. The IMU 413 may output the detected values (the detected values include signals indicating the position, orientation, and changes therein (speed of change, acceleration of change) of the vehicle 1) and the results of analyzing the detected values to the processor 33. The results of the analysis may be a signal indicating a determination result as to whether or not the detected values satisfy a predetermined condition, and may be, for example, a signal indicating that the behavior (vibration) of the vehicle 1 is small, based on a value regarding the change (speed of change, acceleration of change) of the position or orientation of the vehicle 1.

The brightness sensor 415 detects the illuminance or brightness of a predetermined range of the foreground in front of the cabin of the vehicle 1 as the external brightness (an example of brightness information), or the illuminance or brightness inside the cabin as the interior brightness (an example of brightness information). The brightness sensor 415 is, for example, a phototransistor or a photodiode, and is mounted on the instrument panel, rearview mirror, or HUD device 20 of the vehicle 1 shown in FIG. 1.

The mobile information terminal 417 is a smartphone, a laptop computer, a smartwatch, or other information device that can be carried by the observer (or other occupants of the vehicle 1). The I/O interface 31 can communicate with the mobile information terminal 417 by pairing with the mobile information terminal 417, and acquires data recorded in the mobile information terminal 417 (or a server via the mobile information terminal). The mobile information terminal 417 has, for example, the same functions as the above-mentioned road information database 403 and vehicle position detection unit 405, and may acquire the road information (an example of real object related information) and transmit it to the processor 33. The mobile information terminal 417 may also acquire commercial information (an example of real object related information) related to commercial facilities in the vicinity of the vehicle 1 and transmit it to the processor 33. The mobile information terminal 417 may transmit schedule information of the user of the mobile information terminal 417 (e.g., the observer), information on incoming calls to the mobile information terminal 417, information on received e-mails, etc. to the processor 33, and the processor 33 and the image processing circuit 35 may generate and/or transmit image data relating to these.

The external communication device 419 is a communication device that exchanges information with the vehicle 1, and is, for example, another vehicle connected to the vehicle 1 by vehicle-to-vehicle communication (V2V: Vehicle To Vehicle), a pedestrian (a mobile information terminal carried by a pedestrian) connected to the vehicle 1 by pedestrian-to-pedestrian communication (V2P: Vehicle To Pedestrian), and a network communication device connected by road-to-vehicle communication (V2I: Vehicle To Roadside Infrastructure), and in a broad sense, includes everything connected to the vehicle 1 by communication (V2X: Vehicle To Everything). The external communication device 419 may acquire the positions of, for example, pedestrians, bicycles, motorcycles, other vehicles (such as a preceding vehicle), road surfaces, dividing lines, roadside objects, and/or features (such as buildings) and output them to the processor 33. In addition, the external communication device 419 may have the same function as the vehicle position detection unit 405 described above, and may acquire position information of the vehicle 1 and transmit it to the processor 33, and may also have the function of the road information database 403 described above, and may acquire the road information (an example of real object related information) and transmit it to the processor 33. Note that the information acquired from the external communication device 419 is not limited to the above.

The software components stored in memory 37 include an eye position detection module 502, an eye position estimation module 504, an eye position prediction module 506, an eye position state determination module 508, a spot lighting pattern setting module 510, a display parameter setting module 512, a graphics module 514, a light source driving module 516, an actuator driving module 518, and a brightness detection module 520.

7A, 7B, and 7C are flow diagrams illustrating a method S100 for performing operations to set a spot illumination pattern based on an observer's eye position, according to some embodiments. The method S100 is performed in a HUD device 20 including a display, and a display control device 30 that controls the HUD device 20. Some operations of the method S100 described below are optionally combined, the sequence of some operations is optionally changed, and some operations are optionally omitted.

The eye position detection module 502 in FIG. 6 detects the observer's eye position 700 (S110). The eye position detection module 502 includes various software components for performing various operations related to detecting coordinates indicating the observer's eye position 700 (positions in the X and Y axes, which are an example of a signal indicating the eye position 700), detecting coordinates indicating the observer's eye height (positions in the Y axis, which are an example of a signal indicating the eye position 700), detecting coordinates indicating the observer's eye height and depth position (positions in the Y and Z axes, which are an example of a signal indicating the eye position 700), and/or detecting coordinates indicating the observer's eye position 700 (positions in the X, Y, and Z axes, which are an example of a signal indicating the eye position 700).

The eye position 700 detected by the eye position detection module 502 includes the right eye and left eye positions 700R, 700L, a predetermined one of the right eye position 700R and the left eye position 700L, one of the right eye position 700R and the left eye position 700L that is detectable (easy to detect), or a position calculated from the right eye position 700R and the left eye position 700L (for example, the midpoint between the right eye position and the left eye position). For example, the eye position detection module 502 determines the eye position 700 based on the observed position acquired from the eye position detection unit 411 immediately before the timing of updating the display settings.

The eye position detection unit 411 may also detect the movement direction and/or movement speed of the observer's eye position 700 based on multiple observation positions with different detection timings of the observer's eyes acquired from the eye position detection unit 411, and output a signal indicating the movement direction and/or movement speed of the observer's eye position 700 to the processor 33.

The eye position estimation module 504 acquires information that allows the eye position to be estimated (S114). The information that allows the eye position to be estimated is, for example, the captured image acquired from the eye position detection unit 411, the position of the driver's seat of the vehicle 1, the position of the observer's face, the height of the seat, or the observed positions of the eyes of multiple observers. The eye position estimation module 504 estimates the eye position 700 of the observer of the vehicle 1 from the information that allows the eye position to be estimated. The eye position estimation module 504 includes various software components for executing various operations related to estimating the observer's eye position 700, such as estimating the observer's eye position 700 from the captured image acquired from the eye position detection unit 411, the position of the driver's seat of the vehicle 1, the position of the observer's face, the height of the seat, or the observed positions of the eyes of multiple observers. That is, the eye position estimation module 504 may include table data, arithmetic expressions, and the like for estimating the observer's eye position 700 from the information that allows the eye position to be estimated.

The eye position estimation module 504 may calculate the eye position 700 based on the eye observation position acquired from the eye position detection unit 411 immediately before the timing for updating the spot illumination pattern and one or more previously acquired eye observation positions, for example, using a method such as a weighted average.

The eye position prediction module 506 acquires information that can predict the observer's eye position 700 (S116). The information that can predict the observer's eye position 700 is, for example, the latest observation position acquired from the eye position detection unit 411, or one or more observation positions acquired in the past. The eye position prediction module 506 includes various software components for performing various operations related to predicting the eye position 700 based on the information that can predict the observer's eye position 700. Specifically, for example, the eye position prediction module 506 predicts the observer's eye position 700 at the time when the image to which the new display settings are applied is viewed by the observer. The eye position prediction module 506 may predict the next value using one or more past observation positions using a prediction algorithm such as the least squares method, a Kalman filter, an α-β filter, or a particle filter.

The spot lighting pattern setting module 510 sets an area to which the display light K is directed based on the eye position acquired in S110 (S130). The spot lighting pattern setting module 510 sets an area including the partial viewing zone E where the observer's left eye position 700L is located and some or all of the surrounding partial viewing zones E as a left eye spot lighting pattern SPL, and sets an area including the partial viewing zone E where the observer's right eye position 700R is located and some or all of the adjacent partial viewing zones E as a right eye spot lighting pattern SPR (S132). Specifically, as shown in FIG. 4B , the spot lighting pattern setting module 510 directs the display light K to a right-eye spot lighting pattern SPR including a partial viewing area E(2,3) in which the observer's right eye 700L is present and parts of the surrounding partial viewing areas E(1,2), E(2,2), E(2,3), E1,3), E(3,3), E(1,4), E(2,4), and E(2,5), and to a right-eye spot lighting pattern SPR including a partial viewing area E(4,3) in which the observer's left eye 700L is present and parts of the surrounding partial viewing areas E(3,2), E(4,2), E(5,2), E3,3), E(5,3), E(3,4), E(4,4), and E(5,4). In this case, the spot illumination pattern setting module 510 has a main partial area DA (D(4,6)) that directs the display light K to the center of the partial viewing area E(2,3) where the right eye 700R of the observer is located, and sub partial areas DB (D(3,5), D(4,5), D(5,5), D(3,6), D(5,6), D(3,7), D(4,7), and D(5,7)) that direct the display light K to a non-center area in the partial viewing area E(2,3) where the right eye 700R is located. The light source 510 emits light at a predetermined light intensity or more in the main partial area DA (D(8,6)) which directs the display light K to the center of the partial viewing area E(4,3) where the viewer's left eye 700L is located, and the sub partial areas DB (D(7,5), D(8,5), D(9,5), D(7,6), D(9,6), D(7,7), D(8,7), and D(9,7)) which direct the display light K to an area that is not the center of the partial viewing area E(2,3) where the left eye 700L is located. The spot illumination pattern setting module 510 does not irradiate the illumination laser beam 74 (or irradiates the illumination laser beam 74 with a very low light intensity) to the partial areas D other than the right eye light emitting area DMR (the main partial area DA + the sub partial area DB for the right eye) and the left eye light emitting area DML (the main partial area DA + the sub partial area DB for the left eye).

The spot lighting pattern setting module 510 also sets a single binocular spot lighting pattern SPC including the partial viewing area E where the observer's left eye position 700L is located and the partial viewing area E where the observer's right eye position 700R is located (S134). Specifically, as shown in FIG. 8A, the spot lighting pattern setting module 510 directs the display light K to the binocular spot lighting pattern SPC including the partial viewing area E (2,3) where the observer's right eye 700L is located, the partial viewing area E (4,3) where the left eye 700L is located, and the partial viewing areas E (1,2), E (2,2), E (3,2), E (4,2), E (5,2), E (1,3), E (3,3), E (3,5), E (1,4), E (2,4), E (3,4), E (4,4), and a part of E (5,4). In this case, the spot illumination pattern setting module 510 causes a main partial area DA (D(4,6)) that directs the display light K to the center of the partial viewing area E(2,3) where the observer's right eye 700R is located, a main partial area DA (D(8,6)) that directs the display light K to the center of the partial viewing area E(4,3) where the left eye 700L is located, and sub-partial areas DB (D(5,4) to D(7,4), D(2,5) to, D(9,5), D(2,6), D(3,6), D(5,6) to D(7,6), D(9,6), D(2,7) to D(9,7), and D(5,8) to D(7,8)) surrounding the main partial area DA (D(4,6)) and the main partial area DA (D(8,6)) to emit light at a predetermined light intensity or more. The spot illumination pattern setting module 510 does not irradiate the illumination laser beam 74 (or irradiates the illumination laser beam 74 with a very low light intensity) to the partial areas D other than these light-emitting areas DMC for both eyes (main partial area DA for the right eye + main partial area DA for the right eye + sub partial area DB).

In addition, the spot lighting pattern setting module 510 sets the width of the light emitting area DM in the planar lighting unit SE based on the longitudinal eye position (Z-axis coordinate) in the vehicle 1 acquired in S110 (executes the light emitting area change process S136). Figure 13 is a diagram explaining the difference in the light path in the head-up display device 20 due to the difference in the longitudinal eye position in the vehicle 1. Note that the above-mentioned lighting optical system is omitted in this diagram. The lighting light emitted from a specific light emitting area DM of the planar lighting unit SE is modulated into display light K that shows an image on the light modulation element 51, and this display light K is focused by the focusing optical system into a spot pattern SP corresponding to the light emitting area DM. The spot illumination pattern setting module 510 sets a first light-emitting region DM1 for the eye at a first position 701 that is a first distance D1 away from the projection target 2, and sets a second light-emitting region DM2 that is wider than the first light-emitting region DM1 for the eye at a second position 702 that is closer to the projection target 2 than the first distance D1 (in other words, a position forward of the first distance D1).

Figure 10 is a diagram showing one aspect of the first light-emitting region DM1 (first light-emitting region for the left eye DML1, first light-emitting region for the right eye DMR1) and the second light-emitting region DM2 (second light-emitting region for the left eye DML2, second light-emitting region for the right eye DMR2) in the planar illumination unit SE. If the vertical length of the light-emitting region DM is designated by the symbol PV and the horizontal length by the symbol PH, the first light-emitting region DM1 is shorter in both the vertical length PV and the horizontal length PH than the second light-emitting region DM2. Furthermore, the second light-emitting region DM2 includes the first light-emitting region DM1 and its surrounding light-emitting region DM. In this way, when the eye position 700 is positioned forward (second position 702 based on the first position 701), by widening the light-emitting area DM of the light source element 60 (light source element 60A described later), the virtual image V of the image that can be displayed on the entire display surface 52 of the light modulation element 51 (here, "the entire" includes 100% of the displayable area of the light modulation element 51 and a predetermined display area in the displayable area (e.g., 70% of the entire area)) can be viewed by the observer at the desired brightness (or higher). In the example of FIG. 14, the first left-eye light-emitting region DML1 (second left-eye light-emitting region DML2) for the left eye 700L and the first right-eye light-emitting region DMR1 (second right-eye light-emitting region DMR2) for the right eye 700R are provided, but this is not limited to this. The spot lighting pattern setting module 510 of another example may change the width of only the light-emitting region DM directed toward one of the eyes based on the eye position (Z-axis coordinate) in the longitudinal direction of the vehicle 1. Also, the spot lighting pattern setting module 510 of another example may change the width of one connected binocular light-emitting region DMC directed toward one connected binocular spot lighting pattern SPC including the partial viewing zone E where the left eye position 700L exists and the partial viewing zone E where the right eye position 700R exists based on the eye position (Z-axis coordinate) in the longitudinal direction of the vehicle 1.

The spot lighting pattern setting module 510 controls the light quantity distribution Ld in the light emitting area DM of the planar lighting unit SE based on the longitudinal eye position (Z-axis coordinate) of the vehicle 1 acquired in S110 (S138).

In some embodiments, the spot illumination pattern setting module 510 may set the emission intensity of the sub-part region DB to be greater than 0.7 times the emission intensity of the main part region DA. In this way, since the emission intensity of the sub-part region DB is relatively strong, even when the eye position 700 is near the boundary of the partial viewing zone E, the light from the sub-part region DB can cause the relatively bright display light K to be visible.

In addition, in some embodiments, the spot illumination pattern setting module 510 may set the emission intensity of the sub-partial region (DB) to be higher than the emission intensity of the main partial region (DA). In this way, since the emission intensity of the sub-partial region DB is relatively strong, even when the eye position 700 is near the boundary of the partial viewing zone E, the display light K with a relatively high brightness can be viewed by the light from the sub-partial region DB. In addition, the uniformity of the brightness of the virtual image V that can be displayed in the entire virtual image display region VS can be improved.

6 determines to which of a plurality of predetermined partial viewing zones E the eye position 700 detected by the eye position detection module 502, the eye position 700 estimated by the eye position estimation module 504, or the eye position 700 predicted by the eye position prediction module 506 belongs, and sets the display parameters 600 corresponding to the eye position 700. The display parameter setting module 512 includes various software components for executing various operations related to determining to which of a plurality of spatial regions (partial viewing zones E) the X-axis coordinate, Y-axis coordinate, Z-axis coordinate, or combination thereof indicated by the eye position 700 belongs, and setting the display parameters 600 corresponding to the partial viewing zone E to which the eye position 700 belongs. That is, the display parameter setting module 512 may include table data, arithmetic expressions, etc. for identifying the display parameters 600 from the X-axis coordinate, Y-axis coordinate, Z-axis coordinate, or combination thereof indicated by the eye position 700.

The display parameter setting module 512 may apply corresponding display parameters to each partial viewing area E (S152). For example, when the light emitting regions DM emit light in a time-division manner, the display parameter setting module 512 may switch to different display parameters 600 for each timing at which one light emitting region DM emits light. In the example of FIG. 4B, the display parameters 600 corresponding to the light emitting region DM (7,6) may be applied to the light modulation element 51 at the timing at which the light emitting region DM (7,6) emits light, and the display parameters 600 corresponding to the light emitting region DM (8,6) may be applied to the light modulation element 51 at the timing at which the light emitting region DM (8,6) emits light.

The other display parameter setting module 512 may apply one common display parameter to a plurality of light-emitting regions DM (S154). For example, the display parameter setting module 512 applies a common display parameter 600 to the light modulation element 51 at each timing of emitting light from the right-eye light-emitting region DMR (partial regions D(8,6), D(7,5), D(8,5), D(9,5), D(7,6), D(9,6), D(7,7), D(8,7), D(9,7)) in order to direct the display light K to the spotlighting pattern SPR for the right eye shown in FIG. 4B. Specifically, the display parameter setting module 512 may apply the display parameter 600 corresponding to the main partial region DA (D(8,6) in the example of FIG. 4B) that directs the display light K to the center of the spotlighting pattern SP to the light modulation element 51.

In some embodiments, the display parameter setting module 512 may switch the display parameters 600 between the timing at which the right-eye light-emitting region DMR is illuminated and the timing at which the left-eye light-emitting region DML is illuminated. Here, if the display parameters 600 are parameters that cause different images to be displayed, the display parameter setting module 512 can cause the right-eye image to be displayed at the timing at which the right-eye light-emitting region DMR is illuminated, and cause the left-eye image to be displayed at the timing at which the left-eye light-emitting region DML is illuminated, thereby allowing the viewer to perceive a stereoscopic image (virtual image) due to left-right parallax.

The other display parameter setting module 512 may apply two or more display parameters to one spot lighting pattern SP (S154). In this case, the display parameter setting module 512 sets a common display parameter 600 for multiple partial regions D in advance. For example, in the example of FIG. 4B, if a common first display parameter is set for the partial regions D(7,5), D(8,5), and D(9,5), a common second display parameter (different from the first display parameter) is set for the partial regions D(7,6), D(8,6), and D(9,6), and a common third display parameter (different from the first and second display parameters) is set for the partial regions D(7,7), D(8,7), and D(9,7), the display parameter setting module 512 applies the first display parameter to the light modulation element 51 when the partial regions D(7,5), D(8,5), and D(9,5) emit light, respectively, applies the second display parameter to the light modulation element 51 when the partial regions D(7,6), D(8,6), and D(9,6) emit light, respectively, and applies the third display parameter to the light modulation element 51 when the partial regions D(7,7), D(8,7), and D(9,7) emit light, respectively.

The types of display parameters 600 include (1) an image arrangement that, as viewed from the eye position 700, has a predetermined positional relationship with a real object located outside the vehicle 1 (parameters for controlling the display device 50 to control the image arrangement, and/or parameters for controlling an actuator); (2) parameters for pre-distorting an image to reduce image distortion that may occur due to the virtual image optical system 90 as viewed from the eye position 700 (parameters for controlling the display device 50 to pre-distort an image displayed on the display device 50, also called warping parameters); and (3) parameters for visually recognizing a desired stereoscopic image as viewed from the eye position (parameters for controlling the display device 50, parameters for controlling the light source of the display device 50, parameters for controlling an actuator, or parameters including a combination of these). However, the display parameters 600 set (selected) by the display parameter setting module 512 may be parameters that are preferably changed according to the observer's eye position 700, and the types of display parameters 600 are not limited to these.

For each type of display parameter, the display parameter setting module 512 selects one or more display parameters 600 corresponding to the eye position 700. If the eye position 700 includes, for example, a right eye position 700R and a left eye position 700L, the display parameter setting module 512 may select two display parameters: a display parameter 600 corresponding to the right eye position 700R, and a display parameter 600 corresponding to the left eye position 700L. On the other hand, when the eye position 700 is, for example, a predetermined one of the right eye position 700R and the left eye position 700L, a detectable (easy to detect) one of the right eye position 700R and the left eye position 700L, or a position calculated from the right eye position 700R and the left eye position 700L (for example, the midpoint between the right eye position 700R and the left eye position 700L), the display parameter setting module 512 may select one display parameter 600 corresponding to the eye position 700. The display parameter setting module 512 may also select the display parameter 600 corresponding to the eye position 700 and the display parameters 600 set around the eye position 700. That is, the display parameter setting module 512 may select three or more display parameters 600 corresponding to the eye position 700.

The eye position state determination module 508 in FIG. 6 determines whether the observer's eye position 700 is in an unstable state (S170). The eye position state determination module 508 (10) determines whether the observer's eye position 700 is moving, and if movement of the eye position is detected, determines that the state is unstable; (20) determines whether the observer's eye position 700 is moving at a speed equal to or greater than a predetermined threshold, and if movement at a speed equal to or greater than the predetermined threshold is detected, determines that the state is unstable; (30) determines whether the observer's eye position 700 can be detected, and if the eye position 700 cannot be detected, determines that the state is unstable; and (40) determines whether the observer's eye position 700 is outside the eye box 200. , (50) determining whether the observer's eye position 700 is outside the eye box 200 and determining that the state is unstable if the observer's eye position 700 is outside the eye box 200, or (60) determining whether the observer's eye position 700 is predicted to be outside the eye box 200 and determining that the state is unstable if the observer's eye position 700 is predicted to be outside the eye box 200. That is, the eye position state determination module 508 may include table data, arithmetic expressions, etc. for determining whether the eye position is moving, the eye position is moving at high speed, and/or other unstable states from the detection information, estimation information, or prediction information of the eye position 700.

(S176) The eye position detection module 502 calculates the variance of the position data for each of the multiple observation positions acquired from the eye position detection unit 411 within a predetermined measurement time, and the eye position state determination module 508 may be configured to determine that the stability of the observer's eye position is low (unstable) if the variance calculated by the eye position detection module 502 is greater than a predetermined threshold value.

(S178) The eye position detection module 502 calculates the deviation of the position data for each of the multiple observation positions acquired from the eye position detection unit 411 within a predetermined measurement time, and the eye position state determination module 508 may be configured to determine that the stability of the observer's eye position is low (unstable) (not in an unstable state) if the deviation calculated by the eye position detection module 502 is greater than a predetermined threshold value.

In addition, without using the variance of method S176 or the deviation of method S178, the eye position detection module 502 may be configured to determine that the stability of the observer's eye position is low (unstable) when the number of partial viewing zones to which the eye position 700 has moved per specified unit time becomes greater than a specified threshold (not in an unstable state).In addition, the eye position detection module 502 may be configured to determine that the stability of the observer's eye position is low (unstable) when the total distance moved by the eye position 700 per specified unit time (the sum of the distances between multiple observation positions acquired multiple times per unit time) becomes longer than a specified threshold (not in an unstable state).

(S176) Methods for determining whether the observer's eye position 700 can be detected include (1) a part (e.g., a predetermined number of times or more) or all of the observed positions of the observer's eyes acquired from the eye position detection unit 411 within a predetermined period cannot be detected, (2) the eye position detection module 502 cannot detect the observer's eye position 700 during normal operation, (3) the eye position estimation module 504 cannot estimate the observer's eye position 700 during normal operation, (4) the eye position prediction module 506 cannot predict the observer's eye position 700 during normal operation, or a combination of these, determining that the observer's eye position 700 cannot be detected (the observer's eye position 700 is in an unstable state) (note that the above-mentioned determination methods are not limited to these).

(S178) Methods for determining whether the observer's eye position 700 is outside the eye box 200 include (1) acquiring some (e.g., a predetermined number of times or more) or all of the observation positions of the observer's eyes acquired from the eye position detection unit 411 within a predetermined period of time outside the eye box 200, (2) the eye position detection module 502 detecting the observer's eye position 700 outside the eye box 200, or determining that the observer's eye position 700 is outside the eye box 200 (the observer's eye position 700 is in an unstable state) by a combination of these (note that the above-mentioned determination methods are not limited to these).

(S180) Methods for determining whether the observer's eye position 700 can be estimated to be outside the eye box 200 include (1) the eye position detection unit 411 detects the movement of the observer's eye position 700, and then the observer's eye position 700 becomes undetectable, (2) the eye position detection module 502 detects the observer's eye position 700 near the boundary 200A of the eye box 200, (3) the eye position detection module 502 detects either the observer's right eye position 700R or the left eye position 700L near the boundary 200A of the eye box 200, or a combination of these, and determining that the observer's eye position 700 can be estimated to be outside the eye box 200 (the observer's eye position 700 is in an unstable state) (note that the above-mentioned determination methods are not limited to these).

(S182) Methods for determining whether the observer's eye position 700 is predicted to be outside the eye box 200 include (1) the eye position prediction module 506 predicting that the observer's eye position 700 will be outside the eye box 200 after a predetermined time, (2) the eye position detection module 502 determining that the newly detected eye position 700 is equal to or greater than the eye position movement distance threshold pre-stored in memory 37 compared to the previously detected eye position 700 (the movement speed of the eye position 700 is equal to or greater than the eye position movement speed threshold pre-stored in memory 37), or a combination of these, that the observer's eye position 700 can be predicted to be outside the eye box 200 (the observer's eye position 700 is in an unstable state) (note that the above-mentioned determination methods are not limited to these).

When the eye position state determination module 508 determines that the state is not unstable, the spot lighting pattern setting module 510 sets the first spot lighting pattern SP1 (S192). On the other hand, when the eye position state determination module 508 determines that the state is unstable, the display parameter setting module 512 may set the second spot lighting pattern SP2, which has a wider area for directing the display light K than the first spot lighting pattern SP1 (S194).

Referring again to FIG. 6 . The graphics module 514 includes various known software components for performing image processing such as rendering to generate image data and drive the display 50. The graphics module 514 may also include various known software components for changing the type, arrangement (position coordinates, angle), size, display distance (in the case of 3D), and visual effects (e.g., brightness, transparency, saturation, contrast, or other visual characteristics) of the image to be displayed. The graphics module 514 may generate image data to be viewed by the observer in terms of the type of image (one of the display parameters), the position coordinates of the image (one of the display parameters), the angle of the image (such as a pitching angle with the X-direction as its axis, a yaw rate angle with the Y-direction as its axis, and a rolling angle with the Z-direction as its axis, which is one of the display parameters), and the size of the image (one of the display parameters), and drive the HUD device 20.

The light source driving module 516 includes various known software components for executing the driving of the directional light source unit 60. The light source driving module 516 may drive the directional light source unit 60 based on the set spot lighting pattern and display parameters 600.

The actuator drive module 518 includes various known software components for executing the operation of driving the first actuator 28 and/or the second actuator 29. The actuator drive module 518 can drive the first actuator 28 and the second actuator 29 based on the set display parameters 600.

The brightness detection module 520 in FIG. 6 detects, by the brightness sensor 415, the illuminance or brightness of a predetermined range of the foreground in front of the cabin of the vehicle 1 as the outside brightness (an example of brightness information), or the illuminance or brightness inside the cabin as the interior brightness (an example of brightness information). The brightness detection module 520 includes various software components for performing various operations related to acquiring (detecting) brightness information, comparing the acquired brightness information with a predetermined threshold value, and outputting the comparison result.

The brightness detection module 520 also acquires information that can predict brightness, such as external brightness, which is the illuminance or brightness of a predetermined range of the foreground in front of the cabin of the vehicle 1, or interior brightness, which is the illuminance or brightness inside the cabin. Information that can predict brightness is, for example, information from the road information database 403 that indicates positions where the brightness becomes dark. Based on this, the brightness detection module 520 may, for example, predict whether it will become brighter or darker after a predetermined time and output the prediction result.

In some embodiments, the processor 33 may obtain information on the surrounding brightness obtained from the brightness detection module 520, or information that can predict the surrounding brightness, and execute the spot lighting pattern setting module 510 to increase the ratio of the light intensity of the sub-part area DB to the light intensity of the main part area DA when the surrounding brightness is estimated to be dark. Specifically, for example, the spot lighting pattern setting module 510 may set the ratio of the light intensity of the main part area DA to the sub-part area DB to 1:0.7 when the surroundings are bright, and set it to 1:0.8 when the surroundings are dark. In addition, in some embodiments, the spot lighting pattern setting module 510 may set the ratio of the light intensity of the main part area DA to the sub-part area DB to 1:1.3 when the surroundings are bright, and set it to 1:1.5 when the surroundings are dark. In a dark environment, the intensity of the illumination light emitted from each light-emitting region DM is reduced overall, which reduces the effect of the combination of the light amounts from the surrounding light-emitting regions DM (increase in the amount of light); however, in a dark environment, by increasing the ratio of the light-emitting intensity of the sub-subregion DB to the light-emitting intensity of the main subregion DA, it is possible to suppress a decrease in the brightness of the image (virtual image) near the boundary of the partial viewing zone E.

The operations of the above-described processing processes can be implemented by executing one or more functional modules of an information processing device, such as a general-purpose processor or an application-specific chip. All of these modules, combinations of these modules, and/or combinations with known hardware that can replace their functions, are within the scope of protection of the present invention.

The functional blocks of the vehicle display system 10 are optionally implemented by hardware, software, or a combination of hardware and software to implement the principles of the various embodiments described. It will be understood by those skilled in the art that the functional blocks described in FIG. 6 may be optionally combined or a functional block may be separated into two or more sub-blocks to implement the principles of the embodiments described. Thus, the description herein optionally supports any possible combination or division of the functional blocks described herein.

Second Embodiment
FIG. 8 is a diagram showing the configuration of the directional light source unit 60A in the second embodiment. FIG. 12 is a diagram showing one aspect of the light source array 80A shown in FIG. 8. The directional light source unit 60 is composed of a field lens 77A and a light source array 80A in which a plurality of light sources (81 in this embodiment as shown in FIG. 12) are arranged in an array. The light source array 80A (surface illumination unit SE) is arranged in m (m>n) partial regions D (the partial regions D in this embodiment are 121 from D(1,1) to D(11,11).) The display control device 30 partially drives (partially drives) the light source so that strong display light K is directed to the spot lighting pattern SP corresponding to the partial viewing area E in which the eye position detected by the eye position detection unit 411 exists, and as in the case of the directional light source unit 60A described above, the observer is made to view an image (virtual image V), so that the light is not directed to the partial viewing area E in which the observer's eyes are not present.

The field lens 77A is disposed on the optical path of the illumination light emitted from the light source disposed in each partial region D between the light source array 80A and the light modulation element 51. The second field lens 77 in this embodiment is a front-stage optical component (backlight optical component) that is composed of one or more field lenses and is designed so that the illumination laser beam 74 diffused by each partial region D of the light diffusion optical system 80 transmits and illuminates the entire surface of the light modulation element 51. In addition, the field lens 77A is designed so that the display light K emitted from the light modulation element 51 passes through the virtual image optical system 90 (HUD optical component composed of the relay optical system 40 and the front windshield 2) and reaches the partial viewing area E located at any position in the eye box 200.

The diffuser 78 is disposed on the optical path of the illumination light between the light modulation element 51 and the field lens 77A. The diffuser 78 diffuses the received illumination light and emits it to the light modulation element 51. This reduces uneven brightness in the display light K generated by the illumination light emitted from the multiple light-emitting regions DM visible within the partial viewing area E of the eye box 200. The diffuser 78 may be any optical member that has the function of diffusing light, and may have a surface that is made of a bead member, a fine uneven structure, or a rough surface, for example. It may also be a dot sheet or a transparent milky white sheet.

In the light source array 80A of this embodiment, one light source such as an LED is arranged in each of the m partial regions D (multiple light sources may be arranged), and the illumination light emitted by the light source arranged in any of the m partial regions D is irradiated onto the entire surface of the light modulation element 51 after passing through the field lens 77 and the diffuser plate 78, and the optically modulated display light K passes through the virtual image optical system 90 and is then irradiated onto a predetermined region in a predetermined eye box 200 corresponding to the predetermined partial region D.

The relationship between the vertical length PV and the horizontal length PH of the light-emitting region DM will be described with reference to Figures 12, 13A, and 13B. Although the light-emitting patterns are different in Figures 12, 13A, and 13B, if a bounding box including all of a group of partial regions D that are illuminated with a predetermined light intensity or more is defined as the light-emitting region DM, it can be said that the size of the light-emitting region DM in Figures 12, 13A, and 13B is the same. The horizontal direction of the light-emitting region DM corresponds to the left-right direction (X-axis direction) of the eye box 200, and the vertical direction of the light-emitting region DM corresponds to the up-down direction (Y-axis direction) of the eye box 200. In this embodiment, the horizontal length PH of the light-emitting region DM is set to be twice or more the vertical length PV. As a result, the horizontal length of the spot lighting pattern SP corresponding to the light-emitting region DM is set to be approximately twice or more the vertical length. This allows the vertical size of the spot lighting pattern SP to be reduced to reduce the power of the light source, while the horizontal size of the spot lighting pattern SP can be increased to allow the image (virtual image) to be viewed at the desired brightness even when the eye position 700 moves horizontally.

That is, in some embodiments, the width of the spot lighting pattern SP may be set to be wider than the partial viewing area E. Furthermore, in some embodiments, the left-right length of the spot lighting pattern SP may be set to be more than twice the length of the partial viewing area E. Specifically, the light emitting area DM is arranged, the light intensity emitted by the light emitting area DM is adjusted (balanced), and/or the entire or partial optical adjustment of the focusing optical system is performed so that the vertical length of the spot lighting pattern SP is 52 mm and the horizontal length is 20 mm, while the vertical length of the partial viewing area E is 26 mm and the horizontal length is 10 mm.

Figures 14A to 14D are diagrams showing modified examples of the relationship between the partial viewing area of the eyebox and the position where the display light based on the illumination light emitted from each partial area of the directional light source unit is focused, according to some embodiments.

In the example of FIG. 14A, the directional light source unit 60 arranges the partial regions D so that the light can be irradiated to five different locations within one partial viewing zone E. Specifically, there is one location at the center of the partial viewing zone E and four locations at the four corners of the partial viewing zone E, for a total of five locations. In one aspect shown in FIG. 14A, the display control device 30 controls the directional light source unit 60 so that the display light K is irradiated to all five different locations within the partial viewing zone E where the eye position 700 is present. The light-emitting region DM includes a main partial region DA that emits illumination light that is the source of the display light K that is condensed at the center of the partial viewing zone E where the eye position 700 is present, and a sub-partial region DB that emits illumination light that is the source of the display light K that is condensed at a position farther away from the center of the partial viewing zone E where the eye position 700 is present than the main partial region DA. That is, when the left eye position 700L is in the partial viewing zone E(4,3), the main partial area DA is the partial area D(8,6), and the sub partial areas DB are the partial areas D(7,5), D(9,5), D(7,7), and D(9,7). On the other hand, when the right eye position 700R is in the partial viewing zone E(2,3), the main partial area DA is the partial area D(4,6), and the sub partial areas DB are the partial areas D(3,5), D(5,5), D(3,7), and D(5,7).

In the example of FIG. 14B, the directional light source unit 60 arranges the partial regions D so that the light can be irradiated to three different locations within one partial viewing zone E. Specifically, there is one location at the center of the partial viewing zone E and two locations at the four corners of the partial viewing zone E, for a total of three locations. In one aspect shown in FIG. 14B, the display control device 30 controls the directional light source unit 60 so that the display light K is irradiated to all three different locations within the partial viewing zone E where the eye position 700 is present. The light-emitting region DM includes a main partial region DA that emits illumination light that is the source of the display light K that is condensed at the center of the partial viewing zone E where the eye position 700 is present, and a sub-partial region DB that emits illumination light that is the source of the display light K that is condensed at a position farther away from the center of the partial viewing zone E where the eye position 700 is present than the main partial region DA. That is, when the left eye position 700L is in the partial viewing zone E(4,3), the main partial area DA is the partial area D(8,6), and the sub partial area DB is the partial area D(7,5) and D(9,7). On the other hand, when the right eye position 700R is in the partial viewing zone E(2,3), the main partial area DA is the partial area D(4,6), and the sub partial area DB is the partial area D(3,5) and D(5,7).

In the example of FIG. 14C, the directional light source unit 60 arranges the partial regions D so that the partial region D can be irradiated at four different locations within one partial viewing zone E. Specifically, the four locations are at the four corners of the partial viewing zone E. In one aspect shown in FIG. 14C, the display control device 30 controls the directional light source unit 60 so that the display light K is irradiated at all four different locations within the partial viewing zone E where the eye position 700 is present. The light-emitting region DM does not include the main partial region DA that emits illumination light that is the source of the display light K that is condensed at the center of the partial viewing zone E where the eye position 700 is present, but only includes the sub-partial region DB that emits illumination light that is the source of the display light K that is condensed at a position farther away from the center of the partial viewing zone E where the eye position 700 is present than the main partial region DA. That is, when the left eye position 700L is in the partial viewing zone E(4,3), there is no main partial area DA, and the sub partial areas DB are the sub areas D(7,5), D(9,5), D(7,7), and D(9,7). On the other hand, when the right eye position 700R is in the partial viewing zone E(2,3), there is no main partial area DA, and the sub partial areas DB are the sub areas D(3,5), D(5,5), D(3,7), and D(5,7).

In the example of FIG. 14D, the directional light source unit 60 arranges the partial regions D so that it can be irradiated to two different locations within one partial viewing zone E. Specifically, the two locations are other than the four corners of the partial viewing zone E. In one aspect shown in FIG. 14D, the display control device 30 controls the directional light source unit 60 so that the display light K is irradiated to all two different locations within the partial viewing zone E where the eye position 700 is present. The light-emitting region DM does not include the main partial region DA that emits illumination light that is the source of the display light K that is condensed at the center of the partial viewing zone E where the eye position 700 is present, but only includes a sub-partial region DB that emits illumination light that is the source of the display light K that is condensed at a position that is farther from the center of the partial viewing zone E where the eye position 700 is present than the main partial region DA and is not a boundary of the partial viewing zone D. That is, when the left eye position 700L is in the partial viewing zone E(4,3), there is no main partial area DA, and the sub partial area DB is the partial areas D(7.5,6) and D(8.5,6) that are not on the boundaries of the partial viewing zone D. On the other hand, when the right eye position 700R is in the partial viewing zone E(2,3), there is no main partial area DA, and the sub partial area DB is the partial areas D(3.5,6) and D(4.5,6).

As shown in Figures 4B, 14A, and 14B, the light source element 60 (60A) in some embodiments may have a main partial area DA that emits illumination light for directing the display light K to the center of the partial viewing area E. In this case, the processor 33 may set the light emission pattern so that the light emitting area DM includes the main partial area DA that emits illumination light for directing the display light to the center of the partial viewing area E.

In addition, as shown in Figures 4B, 14A, 14B, 14C, and 14D, the light source element 60 (60A) in some embodiments has two or more sub-partial regions DB that are point-symmetrical with respect to positions at which illumination light can be emitted so that the display light is directed to the center of the partial viewing area E. In this case, the processor 33 sets the light emission pattern so that the light-emitting area DM includes two or more sub-partial regions DB that are point-symmetrical with respect to positions at which illumination light can be emitted so that the display light is directed to the center of the partial viewing area E.

In addition, in some embodiments, the light source element 60 (60A) may be arranged in positions where the above-mentioned two or more sub-partial regions DB, which are point-symmetrical with respect to positions where illumination light can be emitted so that the display light is directed to the center of the partial viewing area E, can be emitted so that the display light is directed to the boundary of the partial viewing area E, as shown in Figures 4B, 14A, 14B, and 14C.

In addition, in some embodiments, the light source element 60 (60A) may be arranged in positions where the above-mentioned two or more sub-partial regions DB, which are point-symmetrical with respect to positions at which illumination light can be emitted so that the display light is directed to the center of the partial viewing area E, can be emitted to two or all of the four point-symmetric corners of the partial viewing area E, as shown in Figures 4B, 14A, 14B, and 14C.

As shown in Figures 14C and 14D, the light source element 60 (60A) in some embodiments may not have a main partial area DA that emits illumination light so that the display light is directed to the center of the partial viewing area E, but may only have two or more sub-partial areas DB described above that are point-symmetrical with respect to a position at which illumination light can be emitted so that the display light is directed to the center of the partial viewing area E.

The arrangement of the partial regions D in the light source element 60 (60A) does not need to be regularly arranged overall, and may be different between a position corresponding to the center (and its periphery) of the eye box 200 and a position corresponding to the periphery of the eye box 200, for example. The arrangement of the partial regions D in the light source element 60 (60A) may be different between a position corresponding to the center (and its periphery) in the left-right direction of the eye box 200 and a position corresponding to the periphery in the left-right direction of the eye box 200. The arrangement of the partial regions D in the light source element 60 (60A) may be different between a position corresponding to the center (and its periphery) in the up-down direction of the eye box 200 and a position corresponding to the periphery in the up-down direction of the eye box 200.

In the above embodiment, the light source element 60 (60A) has m partial regions D, which is more than n partial viewing regions E, but this is not limited to this. In other words, the number of partial viewing regions E (n) and the number of partial regions D (m) may be equal.

In addition, in the above embodiment, the display control device 30 (processor 33) changes the width of the light-emitting area DM in the planar illumination unit SE based on the eye position (Z-axis coordinate) in the forward/backward direction of the vehicle 1, but this is not limited to this. The display control device 30 (processor 33) may also change the width of the light-emitting area DM in the planar illumination unit SE based on (1) the eye position (X-coordinate) in the left/right direction of the vehicle 1, (2) the eye position (Y-coordinate) in the up/down direction of the vehicle 1, or (3) any combination of these eye positions (X-coordinate, Y-coordinate), in addition to the eye position (Z-axis coordinate) in the forward/backward direction of the vehicle 1.

1: Vehicle 2: Front windshield (projected area)
3: Non-specular reflection element 10: Vehicle display system 20: HUD device 28: First actuator 29: Second actuator 30: Display control device 31: I/O interface 33: Processor 35: Image processing circuit 37: Memory 40: Relay optical system 41: First mirror 42: Second mirror 50: Display device 51: Light modulation element 52: Display surface 60: Directional light source unit (light source element)
60A: Directional light source unit (light source element)
70: Light source 74: Illumination laser beam 75: Scanning device 76: First field lens 77: Second field lens 77A: Field lens 78: Diffusion plate 80: Light diffusion optical system (surface illumination unit)
80A: Light source array (surface illumination unit)
81: Lens array 82: Hologram recording medium 90: Virtual image optical system 100: Virtual image display area 101: Upper end 102: Lower end 200: Eye box 200A: Boundary 205: Center 401: Vehicle ECU
403: Road information database 405: Vehicle position detection unit 407: Vehicle exterior sensor 409: Operation detection unit 411: Eye position detection unit (gaze direction detection unit)
413: IMU
415: Brightness sensor 417: Portable information terminal 419: External communication device 502: Eye position detection module 504: Eye position estimation module 506: Eye position prediction module 508: Eye position state determination module 510: Spot lighting pattern setting module 512: Display parameter setting module 514: Graphic module 516: Light source driving module 518: Actuator driving module 520: Brightness detection module 600: Display parameters 700: Eye position 700L: Left eye 700R: Right eye D: Partial viewing area DA: Main partial area DB: Sub partial area DM: Light emitting area DML: Light emitting area for left eye DMR: Light emitting area for right eye E: Partial viewing area SP: Spot lighting pattern SP1: First spot lighting pattern SP2: Second spot lighting pattern SPL: Spot lighting pattern for left eye SPR: Spot lighting pattern for right eye V : Virtual image θt : Tilt angle θv : Vertical arrangement angle

Claims (10)

a light source element (60, 60A) having a plurality of partial regions (D) and capable of selectively emitting illumination light for each of the partial regions (D);
a light modulation element (51) that modulates the illumination light incident from the light source element (60, 60A) and emits the modulated light as display light;
a focusing optical system (2, 40, 77, 77A) that focuses the display light obtained by modulating the illumination light emitted from the different partial regions (D) by the light modulation element (51) toward a different region in an eye box (200);
A display control device (30) controls a head-up display device (20) in which a viewer visually recognizes a virtual image from within the range of the eye box by projecting the display light onto a projection target,
A processor (33) for controlling the light source elements (60, 60A),
The processor (33)
Acquire information about an eye position in a front-back direction of the observer facing the projection unit from the eye box (200), or information that allows the eye position in the front-back direction of the observer to be estimated;
When the eye position in the front-rear direction of the observer is at a first position (Z1), the partial region (D) (hereinafter referred to as a light-emitting region (DM)) that emits the illumination light having a predetermined light intensity or more is set to a first light-emitting region (DM1);
and when the eye position is at a second position (Z2) closer to the projection target portion than the first position (Z1), setting the light-emitting region (DM) to a second light-emitting region (DM2) wider than the first light-emitting region (DM1).
A display control device (30). The information that enables estimation of the eye position includes information regarding a position of the driver's seat in a front-rear direction,
The light emitting area change process is based on the position of the driver's seat in a front-rear direction,
When the eye position in the front-rear direction of the observer is estimated to be at the first position (Z1), setting the first light-emitting area (DM1);
When it is estimated that the eye position is at a second position (Z2) closer to the projection target than the first position (Z1), setting the eye position to the second light-emitting region (DM2).
A display control device (30) according to claim 1. The processor (33)
The second light-emitting region (DM2) is set wider than the first light-emitting region (DM1) at least in a direction along the vertical direction of the light modulation element (51) corresponding to the vertical direction of the virtual image and in a direction along the horizontal direction of the light modulation element (51) corresponding to the horizontal direction of the virtual image.
A display control device (30) according to claim 1 or 2. The processor (33)
The average light intensity in the second light-emitting region (DM2) is set to be lower than the average light intensity in the first light-emitting region (DM1).
A display control device (30) according to any one of claims 1 to 3. The processor (33)
obtaining information about the right and left eye positions of the observer in the eye box (200) or information from which the right and left eye positions can be estimated;
The light emitting region (DM) is
A right-eye light-emitting region (DMR) that is set according to a partial viewing area (E) in which the right eye position is located;
A left-eye light-emitting region (DML) is set according to the partial viewing region (E) in which the left eye position is located.
A display control device (30) according to any one of the preceding claims. The processor (33)
Switching the light emission pattern of the right eye light emitting region (DMR) and the light emission pattern of the left eye light emitting region (DML) in a time division manner;
A display control device (30) according to claim 5. The processor (33)
the light emitting region (DM) is set to include a partial region (D) (hereinafter, a main partial region (DA)) that emits the illumination light so that the peak of the light intensity of the display light is directed to the eye position, and a partial region (D) (hereinafter, a sub partial region (DB)) that emits the illumination light so that the peak of the light intensity of the display light is directed to a periphery of the eye position,
The emission intensity of the sub-region (DB) is set to be more than 0.7 times the emission intensity of the main region (DA).
A display control device (30) according to any one of the preceding claims. The processor (33)
the light emitting region (DM) is set to include a partial region (D) (hereinafter, a main partial region (DA)) that emits the illumination light so that the peak of the light intensity of the display light is directed to the eye position, and a partial region (D) (hereinafter, a sub partial region (DB)) that emits the illumination light so that the peak of the light intensity of the display light is directed to a periphery of the eye position,
The emission intensity of the sub-region (DB) is made higher than the emission intensity of the main region (DA).
A display control device (30) according to any one of the preceding claims. A head-up display device (20) in which a viewer visually recognizes a virtual image from within an eye box (200) by projecting display light onto a projection surface,
a light source element (60, 60A) having a plurality of partial regions (D) and capable of selectively emitting illumination light for each of the partial regions (D);
a light modulation element (51) that modulates the illumination light incident from the light source element (60, 60A) and emits the modulated light as the display light;
a focusing optical system (2, 40, 77, 77A) that focuses the display light obtained by modulating the illumination light emitted from the different partial regions (D) by the light modulation element (51) toward a different region in an eye box (200);
A processor (33) that controls the light source elements (60, 60A),
The processor (33)
Acquire information on a front-back eye position of an observer facing the projection unit from the eye box (200), or information that allows the eye position of the observer in the front-back direction to be estimated;
When the eye position in the front-rear direction of the observer is at a first position (Z1), the partial region (D) (hereinafter referred to as a light-emitting region (DM)) that emits the illumination light having a predetermined light intensity or more is set to a first light-emitting region (DM1);
and when the eye position is at a second position (Z2) closer to the projection target portion than the first position (Z1), setting the light-emitting region (DM) to a second light-emitting region (DM2) wider than the first light-emitting region (DM1).
A head-up display device (20). a light source element (60, 60A) having a plurality of partial regions (D) and capable of selectively emitting illumination light for each of the partial regions (D);
a light modulation element (51) that modulates the illumination light incident from the light source element (60, 60A) and emits the modulated light as display light;
a focusing optical system (2, 40, 77, 77A) that focuses the display light obtained by modulating the illumination light emitted from the different partial regions (D) by the light modulation element (51) toward a different region in an eye box (200);
A display control method for a head-up display device (20) in which a viewer visually recognizes a virtual image from within the range of the eye box by projecting the display light onto a projection target, comprising:
Acquiring information about an eye position in a front-back direction of the observer facing the projection unit from the eye box (200) or information that allows the eye position in the front-back direction of the observer to be estimated;
When the eye position in the front-rear direction of the observer is at a first position (Z1), the partial region (D) (hereinafter referred to as a light-emitting region (DM)) that emits the illumination light having a predetermined light intensity or more is set to a first light-emitting region (DM1);
and executing a light emitting area change process, the light emitting area being changed when the eye position is at a second position (Z2) closer to the projection target than the first position (Z1). The light emitting area is set to a second light emitting area (DM2) wider than the first light emitting area (DM1).
Display control method.

Images