JP7655103B2 - Display control device, head-up display device, and display control method - Google Patents

Description

The present disclosure relates to a display control device, a head-up display device, a display control method, and the like, which are used in a moving body such as a vehicle and allow an image to be superimposed on the foreground of the moving body (the actual view in the forward direction of the moving body as seen by the vehicle occupants) and made visible.

Patent Document 1 describes a head-up display device (one example of a virtual image display device) in which display light projected onto a projection target such as the vehicle's front windshield is reflected toward a vehicle occupant (observer) inside the vehicle, allowing the observer to view a virtual image superimposed on the vehicle's foreground. In particular, the head-up display device described in Patent Document 1 allows the observer to virtually perceive a display object (virtual image) at a predetermined position in the depth or vertical/horizontal directions (here, the position is referred to as a target position) in the real space of the foreground, and controls the image displayed inside the head-up display device as if the display object were present at the target position in the foreground, even if the attitude of the vehicle changes or the observer's eye position changes. In other words, such a head-up display device creates an augmented reality display by adding virtual objects to the real scenery (foreground), and even when the vehicle's attitude changes (which also leads to a change in the observer's eye position relative to the real scenery) or the observer's eye position changes inside the vehicle, the position of the image displayed inside the head-up display device is corrected in response to the change in the observer's eye position detected by an eye position detection unit such as a camera, thereby imparting motion parallax to the virtual object and allowing the observer to perceive it as if the virtual object were located at the target position in the foreground (real scenery).

JP 2010-156608 A

Meanwhile, an eye position detection unit such as a camera detects the eye position (left and right eye positions) of the observer using a captured image and other data through a complex algorithm. However, even if the observer's eye position has only moved a specified amount left and right, it may be detected (misdetected) as having moved an amount greater than the specified amount due to detection errors or erroneous detections depending on the detection environment. The motion parallax image of the virtual object may then be corrected according to the amount of movement that has been erroneously detected. As a result, the position correction of the image left and right will be greater than the amount of movement of the observer's eye position, resulting in an amount of movement greater than the observer's actual perception, which may cause the observer to feel uncomfortable.

A summary of certain embodiments disclosed herein is provided below. It should be understood that these aspects are presented merely to provide the reader with an overview of these certain embodiments, and are not intended to limit the scope of the disclosure. Indeed, the disclosure may encompass a variety of aspects that are not set forth below.

The present disclosure relates to providing a display control device, a head-up display device, a display control method, and the like that provide a natural motion parallax to the observer.

Therefore, in order to solve the above problems, the display control device, the head-up display device, the display control method, etc. described in this specification employ the following means. The present embodiment relates to a display control device (30) that executes display control in a head-up display device (20) that is configured at least with a display (40) that displays an image and a relay optical system (80) that projects light of the image displayed by the display (40) onto a projection target member, and that allows a vehicle user to visually recognize a virtual image of the image superimposed on a foreground, the display control device (30) comprising one or more processors (33), a memory (37), one or more computer programs that are stored in the memory (37) and configured to be executed by the one or more processors (33), and an outside-vehicle sensor (411) that detects real objects present around the vehicle, the one or more processors (33) acquiring the user's eye position or head position (Py) in a vertical direction of the vehicle and the eye position or head position (Px) in a horizontal direction of the vehicle, and calculating a change (ΔPy) in the eye position or head position (Py) in the vertical direction and a change ( ΔPx ) in the eye position or head position (Px) in the horizontal direction. and a second image correction process (S170) for correcting the position of the image to be displayed on the display (40) with a second correction amount (Cxq1, Cyr1) obtained by multiplying the first correction amount (Cx, Cy) by a first predetermined coefficient (Q1, R1) that is equal to or less than 1. The method is characterized in that the first image correction process is executed when an object distance (D4) that is a distance from the vehicle to the real object in the foreground that is superimposed on the virtual image is equal to or less than a first distance threshold, and the second image correction process (S170) is executed when the object distance (D4) is greater than the first distance threshold.

FIG. 1 is a diagram showing an example of application of a virtual image display system for a vehicle to a vehicle. FIG. 2 is a diagram showing the configuration of the head-up display device. FIG. 3 is a diagram showing an example of a foreground viewed by an observer while the host vehicle is traveling, and a virtual image displayed superimposed on the foreground. FIG. 4 is a conceptual diagram showing the positional relationship between a left viewpoint virtual image and a right viewpoint virtual image displayed on the virtual image formation surface and a perceived image perceived by an observer from these left viewpoint virtual image and right viewpoint virtual image in an embodiment in which the HUD device is a 3D-HUD device. FIG. 5 is a diagram conceptually showing a virtual object placed at a target position in the real world, and an image displayed in the virtual image display area so that the virtual object is visually recognized at the target position in the real world. FIG. 6 is a diagram for explaining a method of motion parallax adding processing in this embodiment. FIG. 7A is a comparative example showing a virtual image viewed from the position Px12 shown in FIG. 6 when the motion parallax adding process of the present embodiment is not performed. FIG. 7B is a diagram showing a virtual image viewed from the position Px12 shown in FIG. 6 when the motion parallax adding process of the present embodiment is performed. FIG. 8 is a diagram for explaining a method of adding motion parallax by moving the eye position (head position) in the vertical direction in this embodiment. FIG. 9 is a diagram showing an example of a foreground viewed by an observer while the host vehicle is traveling, and a virtual image displayed superimposed on the foreground. FIG. 10 is a block diagram of a vehicle virtual image display system according to some embodiments. FIG. 11A is a flow diagram illustrating a method S100 for performing operations to correct an image based on an observer's eye position. FIG. 11B is a diagram for explaining a part of the flow diagram shown in FIG. 11A. FIG. 11C is a diagram for explaining a part of the flow diagram shown in FIG. 11A. FIG. 12 is an image diagram showing the eye position detected at each predetermined period of time, the amount of change in the eye position, the moving speed of the eye position, etc. FIG. 13 is a diagram for explaining the relationship between the object distance and the correction amount ratio for the amount of change in eye position in the left-right and up-down directions.

Below, in Figs. 1 to 6 and Figs. 7B to 13, an explanation is provided of the configuration and operation of an exemplary vehicle display system. Note that the present invention is not limited to the following embodiments (including the contents of the drawings). Of course, modifications (including the deletion of components) can be made to the following embodiments. In addition, in the following explanation, explanations of well-known technical matters will be omitted as appropriate in order to facilitate understanding of the present invention.

Refer to FIG. 1. FIG. 1 is a diagram showing an example of the configuration of a vehicle virtual image display system including a parallax-type 3D-HUD device. In FIG. 1, the left-right direction of a vehicle (an example of a moving body) 1 (in other words, the width direction of the vehicle 1) is the X-axis (the positive direction of the X-axis is the left direction when the vehicle 1 faces forward), the up-down direction (in other words, the height direction of the vehicle 1) along a line segment that is perpendicular to the left-right direction and perpendicular to the ground or a surface equivalent to the ground (here, road surface 6) is the Y-axis (the positive direction of the Y-axis is the upward direction), and the front-rear direction along a line segment that is perpendicular to each of the left-right direction and the up-down direction is the Z-axis (the positive direction of the Z-axis is the straight-ahead direction of the vehicle 1). This point is the same in other drawings.

As shown in the figure, the vehicle display system 10 provided in the vehicle (host vehicle) 1 is composed of an eye position detection unit 409 for pupil (or face) detection that detects the position and gaze direction of the left eye 700L and right eye 700R of the observer (typically the driver sitting in the driver's seat of the host vehicle 1), an outside sensor 411 composed of a camera (e.g. a stereo camera) that captures an image in front of the host vehicle 1 (broadly speaking, the surroundings), a head-up display device (hereinafter also referred to as a HUD device) 20, and a display control device 30 that controls the HUD device 20.

Figure 2 is a diagram showing one embodiment of the configuration of a head-up display device. The HUD device 20 is placed, for example, in a dashboard (reference numeral 5 in Figure 1). This HUD device 20 has a stereoscopic display device (an example of a display device) 40, a relay optical system 80, and a housing 22 that houses the stereoscopic display device 40 and the relay optical system 80 and has a light exit window 21 that can emit display light K from the stereoscopic display device 40 from the inside to the outside.

Here, the stereoscopic display device 40 is a parallax 3D display device. This stereoscopic display device (parallax 3D display device) 40 is composed of a light modulation element 50, which is a naked-eye stereoscopic display device that uses a multi-viewpoint image display method that can control the depth representation by visualizing a left viewpoint image and a right viewpoint image, and a light source unit 60 that functions as a backlight.

The light modulation element 50 includes a light modulation element 51 that modulates the illumination light from the light source unit 60 to generate an image, and an optical layer (an example of a light beam separation unit) 52 that has, for example, a lenticular lens or a parallax barrier and separates the light emitted from the light modulation element 51 into left-eye display light (symbol K10 in FIG. 1) such as light beams K11, K12, and K13 for the left eye, and right-eye display light (symbol K20 in FIG. 1) such as light beams K21, K22, and K23 for the right eye. The optical layer 52 includes optical filters such as a lenticular lens, a parallax barrier, a lens array, and a microlens array. However, this is an example and is not limited to this. The embodiment of the optical layer 52 is not limited to the optical filter described above, but includes all forms of optical layers disposed on the front or rear surface of the light modulation element 51, as long as they generate the left eye display light (reference symbol K10 in FIG. 1) and the right eye display light (reference symbol K20 in FIG. 1) from the light emitted from the light modulation element 51. Some embodiments of the optical layer 52 may be electrically controlled to generate the left eye display light (reference symbol K10 in FIG. 1) and the right eye display light (reference symbol K20 in FIG. 1) from the light emitted from the light modulation element 51, such as a liquid crystal lens. That is, the embodiment of the optical layer 52 may include an electrically controlled one and an electrically non-controlled one.

In addition, the stereoscopic display device 40 may emit display light for the left eye (symbol K10 in FIG. 1), such as light rays K11, K12, and K13 for the left eye, and display light for the right eye (symbol K20 in FIG. 1), such as light rays K21, K22, and K23 for the right eye, by configuring the light source unit 60 with a directional backlight unit (an example of a light beam separating unit) instead of or in addition to the optical layer 52 (an example of a light beam separating unit). Specifically, for example, when the directional backlight unit irradiates illumination light toward the left eye 700L, the display control device 30 (described later) causes the light modulation element 51 to display a left viewpoint image, thereby directing left-eye display light K10 such as light rays K11, K12, and K13 for the left eye to the left eye 700L of the observer, and when the directional backlight unit irradiates illumination light toward the right eye 700R, causes the light modulation element 51 to display a right viewpoint image, thereby directing right-eye display light K20 such as light rays K21, K22, and K23 for the right eye to the right eye 700R of the observer. However, this is just one example and is not limited to the present invention.

The display control device 30, which will be described later, can control the appearance of the perceived virtual image FU displayed by the HUD device 20 (perceived by the observer) by, for example, directing the left eye display light K10 of the left viewpoint image V10 to the observer's left eye 700L and the right eye display light K20 of the right viewpoint image V20 to the right eye 700R by performing image rendering processing (graphics processing), display drive processing, etc., and adjusting the left viewpoint image V10 and the right viewpoint image V20. The display control device 30, which will be described later, may control the display (light modulation element 50) to generate a light field that reproduces (approximately) the light rays output in various directions from a point existing in a certain space.

The relay optical system 80 has curved mirrors (concave mirrors, etc.) 81, 82 that reflect light from the stereoscopic display device 40 and project image display light K10, K20 onto the windshield (projected member) 2. However, it may further have other optical elements (which may include refractive optical elements such as lenses, diffractive optical elements such as holograms, reflective optical elements, or combinations of these).

In FIG. 1, the stereoscopic display device 40 of the HUD device 20 displays images with parallax (parallax images) for the left and right eyes. As shown in FIG. 1, each parallax image is displayed as V10 and V20 formed on a virtual image display surface (virtual image forming surface) VS. The focus of each eye of the observer (person) is adjusted to match the position of the virtual image display area VS. The position of the virtual image display area VS is referred to as the "adjustment position (or imaging position)," and the distance from a predetermined reference position (for example, the center 205 of the eye box 200 of the HUD device 20, the observer's viewpoint position, the observer's head position, or a specific position of the vehicle 1) to the virtual image display area VS (see reference symbol D10 in FIG. 4) is referred to as the adjustment distance (imaging distance).

However, in reality, the human brain fuses each image (virtual image), so the human recognizes that the perceived image (here, a graphic of an arrowhead for navigation) FU is displayed at a position further back than the adjustment position (for example, a position determined by the convergence angle between the left viewpoint image V10 and the right viewpoint image V20, and the smaller the convergence angle, the farther the position is perceived to be from the observer). The perceived virtual image FU may be called a "stereoscopic virtual image", or, if "image" is taken in a broad sense to include virtual images, it may also be called a "stereoscopic image". It may also be called a "stereoscopic image", "3D display", etc. The HUD device 20 may display the left viewpoint image V10 and the right viewpoint image V20 so that the perceived virtual image FU is visually recognized at a position closer to the observer than the adjustment position.

Next, reference is made to Figures 3 and 4. Figure 3 is a diagram showing an example of the foreground viewed by the observer while the host vehicle 1 is traveling, and a perceived image displayed superimposed on the foreground. Figure 4 is a diagram conceptually showing the positional relationship between the left viewpoint virtual image and the right viewpoint virtual image displayed on the virtual image formation surface, and the perceived image perceived by the observer from these left viewpoint virtual image and right viewpoint virtual image.

In FIG. 3, the vehicle 1 is traveling on a straight road (road surface) 6. The HUD device 20 is disposed in the dashboard 5. Display light K (K10, K20) is projected from a light exit window 21 of the HUD device 20 onto a projection target (the front windshield of the vehicle 1) 2. In the example of FIG. 3, a first content image FU1 is displayed, which is superimposed on the road surface 6 and indicates the route of the vehicle 1 (indicating straight ahead here), and a second content image FU2 is displayed, which also indicates the route of the vehicle 1 (indicating straight ahead here) and is perceived to be farther away than the first content image FU1.

As shown in the left diagram of Figure 4, the HUD device 20 (1) emits left-eye display light K10 to the projection unit 2 at a position and angle such that it is reflected by the projection unit 2 to the left eye 700L detected by the eye position detection unit 409, and forms a first left-viewpoint content image V11 at a predetermined position in the virtual image display area VS as seen from the left eye 700L, and (2) emits right-eye display light K20 to the projection unit 2 at a position and angle such that it is reflected by the projection unit 2 to the right eye 700R, and forms a first right-viewpoint content image V21 at a predetermined position in the virtual image display area VS as seen from the right eye 700R. The first content image FU1 perceived by the first left viewpoint content image V11 and the first right viewpoint content image V21, which have parallax, is viewed at a position that is a distance D21 behind the virtual image display area VS (a position that is a distance D31 away from the above-mentioned reference position).

Similarly, as shown in the right diagram of Figure 4, the HUD device 20 (1) emits left-eye display light K10 to the projection unit 2 at a position and angle such that it is reflected by the projection unit 2 to the left eye 700L detected by the eye position detection unit 409, and forms a second left-viewpoint content image V12 at a predetermined position in the virtual image display area VS as seen from the left eye 700L, and (2) emits right-eye display light K20 to the projection unit 2 at a position and angle such that it is reflected by the projection unit 2 to the right eye 700R, and forms a second right-viewpoint content image V22 at a predetermined position in the virtual image display area VS as seen from the right eye 700R. The second content image FU2 perceived by the second left viewpoint content image V12 and the second right viewpoint content image V22, which have parallax, is viewed at a position that is a distance D22 behind the virtual image display area VS (a position that is a distance D32 away from the above-mentioned reference position).

Specifically, the distance from the above-mentioned reference position to the virtual image display area VS (imaging distance (first distance threshold) D10) is set to, for example, 4 m, the distance from the above-mentioned reference position to the first content image FU1 shown in the left diagram of FIG. 4 (first perceptual distance D31) is set to, for example, 7 m, and the distance from the above-mentioned reference position to the second content image FU2 shown in the right diagram of FIG. 4 (second perceptual distance D32) is set to, for example, 10 m. However, this is just an example and is not limiting.

FIG. 5 is a conceptual diagram showing a virtual object placed at a target position in the real world and an image displayed in the virtual image display area so that the virtual object is visually recognized at the target position in the real world. The HUD device 20 shown in FIG. 5 shows an example of 2D display, not 3D display. That is, the display 40 of the HUD device 20 shown in FIG. 5 is a 2D display device that is not a stereoscopic display device (note that 2D display is also possible with a stereoscopic display device). As shown in FIG. 5, the depth direction from the viewer's eye position 700 is the Z-axis direction, the left-right direction (width direction of the vehicle 1) is the X-axis direction, and the up-down direction (up-down direction of the vehicle 1) is the Y-axis direction. Note that the direction away from the viewer is the positive direction of the Z-axis, the left direction from the viewer is the positive direction of the X-axis, and the up direction from the viewer is the positive direction of the Y-axis.

The viewer's eye position 700 perceives a virtual object FU at a predetermined target position PT in the real scene by viewing the virtual image V formed (imaged) in the virtual image display area VS via the projection unit 2. The viewer views the virtual image V of the image of the display light K reflected by the projection unit 2. At this time, if the virtual image V is, for example, an arrow indicating a course, the arrow of the virtual image V is displayed in the virtual image display area VS so that the virtual object FU is positioned and viewed at a predetermined target position PT in the foreground of the vehicle 1. Specifically, the HUD device 20 (display control device 30) renders an image to be displayed on the display 40 so that a virtual object FU of a predetermined size and shape placed at the target position PT is projected and transformed into the virtual image display area VS, with the center between the observer's left eye 700L and right eye 700R as the origin of the projection transformation. The HUD device 20 (display control device 30) changes the position of the virtual image V displayed in the virtual image display area VS so that the virtual object FU is perceived at the same target position PT as before the eye position was moved, even if the virtual object FU (virtual image V) is displayed at a position (virtual image display area VS) away from the target position PT, so that the virtual object FU (virtual image V) can be recognized as if it is at the target position PT, even if the virtual object FU (virtual image V) is displayed at a position (virtual image display area VS) away from the target position PT. In other words, the HUD device 20 (display control device 30) expresses natural motion parallax by changing the position (this may include changing the size or shape) of the image (virtual image V in the virtual image display area VS) on the display 40 based on the movement of the eye position (in other words, the HUD device 20 adds motion parallax to the virtual image (image) through image correction associated with the movement of the eye position, making it easier to sense depth perception).

In the description of this embodiment, the correction of the image position to express motion parallax in response to such changes in eye position is called motion parallax addition processing (an example of eye tracking image correction processing). The motion parallax addition processing is not limited to image position correction to completely reproduce natural motion parallax, but may also include image position correction to approximate natural motion parallax. Note that the HUD device 20 (display control device 30) may execute not only the motion parallax addition processing (an example of eye tracking image correction processing) in response to changes in eye position 700, but also the motion parallax addition processing (an example of eye tracking image correction processing) based on the observer's head position 710 instead of the eye position 700.

Figure 6 is a diagram for explaining the method of motion parallax addition processing in this embodiment. The display control device 30 (processor 33) of this embodiment controls the HUD device 20 and displays virtual images V41, V42, and V43 formed (imaged) in the virtual image display area VS via the projection unit 2. The virtual image V41 is set to a target position PT11 at a perceived distance D33 (a position that is a distance D23 further back than the virtual image display area VS), the virtual image V42 is set to a target position PT12 at a perceived distance D34 (a position that is a distance D24 (>D23) further back than the virtual image display area VS) that is longer than the perceived distance D33 of the virtual image V41, and the virtual image V43 is set to a target position PT13 at a perceived distance D35 (a position that is a distance D25 (>D24) further back than the virtual image display area VS) that is longer than the perceived distance D34 of the virtual image V42. Note that the amount of correction of the image on the display 40 corresponds to the amount of correction of the virtual image in the virtual image display area VS, so in FIG. 6, the amounts of correction of the virtual image corresponding to the amounts of correction C1, C2, and C3 of the image on the display 40 also use the same symbols C1, C2, and C3 (the same applies to the symbols Cy11 (Cy) and Cy21 (Cy) in FIG. 8).

When the observer's head position 710 (eye position 700) moves from the position of Px11 to the right (negative direction of the X-axis) by ΔPx10, the display control device 30 (processor 33) executes the motion parallax addition process to correct the positions where the virtual images V41, V42, and V43 displayed in the virtual image display area VS are displayed by the correction amounts C1, C2 (>C1), and C3 (>C2), respectively, in the same direction as the movement of the observer's head position 710 (eye position 700). Figure 7A is a comparative example showing the virtual images V41, V42, and V43 viewed from the position Px12 shown in Figure 6 when the motion parallax addition process of this embodiment is not performed, and Figure 7B is a diagram showing the virtual images V44, V45, and V46 viewed from the position Px12 shown in Figure 6 when the motion parallax addition process of this embodiment is performed. Please note that in FIG. 7B, the difference in the positions of the virtual images V44, V45, and V46 is exaggerated to make the difference in the correction amount easier to understand. That is, the display control device 30 (processor 33) can make the observer perceive motion parallax only between the virtual images V41 (V44), V42 (V45), and V43 (V46) by changing the correction amount of the positions of the virtual images V41, V42, and V43 associated with the movement of the eye position due to the difference in the perception distances D33, D34, and D35 of the virtual images V41, V42, and V43. More specifically, the display control device 30 (processor 33) adds motion parallax to the virtual images V41 (V44), V42 (V45), and V43 (V46) by increasing the correction amount in the motion parallax addition process as the set perception distance D30 becomes longer.

Figure 8 is a diagram for explaining a method of motion parallax addition processing when the eye position (head position) moves in the vertical direction in this embodiment. When the observer's head position 710 (eye position 700) moves upward (positive Y-axis direction) from the position marked Py12, the display control device 30 (processor 33) executes motion parallax addition processing to correct the position at which the virtual image V displayed in the virtual image display area VS is displayed in the same direction (upward (positive Y-axis direction)) as the observer's head position 710 (eye position 700) moved, as shown in Figure 8 (a), by the correction amount Cy11 (changing the position of the virtual image V from the position marked V48 to the position marked V47). In addition, when the observer's head position 710 (eye position 700) moves downward (negative Y-axis direction) from the position of Py12, the display control device 30 (processor 33) executes a motion parallax addition process to correct the position where the virtual image V displayed in the virtual image display area VS is displayed in the same direction (downward (negative Y-axis direction)) as the observer's head position 710 (eye position 700) moved, as shown in FIG. 8(c), by the correction amount Cy21 (changing the position of the virtual image V from the position of V48 to V49). This allows the virtual object FU (virtual image V) to be recognized as if it is at the target position PT, even though it is displayed at a position (virtual image display area VS) away from the target position PT (the sensation that the virtual object FU (virtual image V) is at the target position PT can be enhanced).

9 is a diagram showing a real object 300 present in the foreground, which is visually recognized when the observer faces forward from the driver's seat of the vehicle 1, and a virtual image V displayed by the HUD device 20 of this embodiment. The virtual image V shown in FIG. 9 includes an AR virtual image V60 whose display position can be changed depending on the position of the real object 300, and a non-AR virtual image V70, which is a virtual image whose display position is set regardless of the position of the real object 300. The AR virtual image V60 is displayed at a position (target position PT) corresponding to the position of the real object 300 present in the real scene. The AR virtual image V60 is displayed, for example, at a position superimposed on the real object 300 or in the vicinity of the real object 300, and the presence of the real object 300 is emphasized and notified. In other words, the "position corresponding to the position of the real object 300 (target position PT)" is not limited to a position superimposed on the real object 300 and visually recognized by the observer, but may be a position in the vicinity of the real object 300. The AR virtual image V60 can be any shape as long as it does not interfere with the visibility of the real object 300.

9 includes navigation virtual images V61 and V62 indicating the guide route, enhanced virtual images V64 and V65 that emphasize and notify of objects of attention, and a POI virtual image V65 that indicates a target object or a specified building. The position (target position PT) corresponding to the position of the real object 300 is the position of the road surface 311 (an example of the real object 300) on which the navigation virtual images V61 and V62 are superimposed, the position around the person 313 (an example of the real object 300) in the enhanced virtual image V63, the position near another vehicle 314 (an example of the real object 300) in the enhanced virtual image V64, and the position around the building 315 (an example of the real object 300) in the POI virtual image V65. As described above, the display control device 30 (processor 33) increases the correction amount C associated with the movement of the observer's eye position in the motion parallax adding process as the perceived distance D30 set for the virtual image V becomes longer. That is, if the perceived distances D30 set for the virtual images V shown in FIG. 9 are V65 → V64 → V63 → V62 → V61 in order from the longest, the display control device 30 (processor 33) sets the correction amount C associated with the movement of the observer's eye position as follows: correction amount of V65>correction amount of V64>correction amount of V63>correction amount of V62>correction amount of V61. Note that since the virtual images V62 and V61 are the same type of virtual images and are displayed close to each other, the display control device 30 (processor 33) may set the correction amount of V62 and the correction amount of V61 associated with the movement of the observer's eye position to be the same.

In addition, in some embodiments, the display control device 30 (processor 33) may set the correction amount C associated with movement of the observer's eye position in the non-AR virtual image V70 to zero (no correction may be required associated with movement of the observer's eye position).

In addition, the display control device 30 (processor 33) of some embodiments may correct the non-AR virtual image V70 in accordance with the movement of the observer's eye position. In the example shown in Fig. 9, the non-AR virtual image V70 (V71, V72) is disposed below the virtual image display area VS, and the area of the road surface 311, which is the real object 300, that overlaps with them is closer to the host vehicle 1 than the area of the road surface 311 that overlaps with the navigation virtual image V61 in Fig. 9. That is, in some embodiments, the display control device 30 (processor 33) may set the perception distance D30 of the non-AR virtual image V70 (V71, V72) shorter than the perception distance D30 of the AR virtual image V60 (narrowly speaking, the navigation virtual image V61 located at the lowest position among the AR virtual images V60), and set the correction amount C of the non-AR virtual image V70 associated with the movement of the observer's eye position to be smaller than the correction amount C of the AR virtual image V60 (narrowly speaking, the navigation virtual image V61 located at the lowest position among the AR virtual images V60) associated with the movement of the observer's eye position.

Figure 10 is a block diagram of a vehicle virtual image display system according to some embodiments. The display controller 30 includes one or more I/O interfaces 31, one or more processors 33, one or more image processing circuits 35, and one or more memories 37. The various functional blocks described in Figure 3 may be configured in hardware, software, or a combination of both. Figure 10 is only one embodiment, and the components shown may be combined into fewer components, or there may be additional components. For example, the image processing circuitry 35 (e.g., a graphics processing unit) may be included in one or more processors 33.

As shown, the processor 33 and the image processing circuit 35 are operatively coupled to the memory 37. More specifically, the processor 33 and the image processing circuit 35 can execute programs stored in the memory 37 to control the vehicle display system 10 (display 40), for example, to generate and/or transmit image data. The processor 33 and/or the image processing circuit 35 can include at least one general-purpose microprocessor (e.g., a central processing unit (CPU)), at least one application specific integrated circuit (ASIC), at least one field programmable gate array (FPGA), or any combination thereof. The memory 37 can include any type of magnetic media, such as a hard disk, any type of optical media, such as CDs and DVDs, any type of semiconductor memory, such as volatile memory, and non-volatile memory. Volatile memory can include DRAM and SRAM, and non-volatile memory can include ROM and NVRAM.

As shown in the figure, the processor 33 is operably connected to the I/O interface 31. The I/O interface 31 communicates (also referred to as CAN communication) with, for example, a vehicle ECU 401 (described later) and/or other electronic devices (reference numerals 403 to 419 described later) provided in the vehicle according to the CAN (Controller Area Network) standard. Note that the communication standard adopted by the I/O interface 31 is not limited to CAN, and includes, for example, wired communication interfaces such as CANFD (CAN with Flexible Data Rate), LIN (Local Interconnect Network), Ethernet (registered trademark), MOST (Media Oriented Systems Transport: MOST is a registered trademark), UART, or USB, or in-vehicle communication (internal communication) interfaces that are short-range wireless communication interfaces within several tens of meters, such as personal area networks (PANs) such as Bluetooth (registered trademark) networks and local area networks (LANs) such as 802.11x Wi-Fi (registered trademark) networks. The I/O interface 31 may also include an external communication (external communication) interface such as a wide area communication network (e.g., an Internet communication network) based on a wireless wide area network (WWAN0, IEEE 802.16-2004 (WiMAX: Worldwide Interoperability for Microwave Access)), IEEE 802.16e-based (Mobile WiMAX), 4G, 4G-LTE, LTE Advanced, 5G, or other cellular communication standards.

As shown in the figure, the processor 33 is interoperably connected to the I/O interface 31, thereby enabling information to be exchanged with various other electronic devices connected to the vehicle display system 10 (I/O interface 31). The I/O interface 31 is operably connected to, for example, a vehicle ECU 401, a road information database 403, a vehicle position detection unit 405, an operation detection unit 407, an eye position detection unit 409, an outside vehicle sensor 411, a brightness detection unit 413, an IMU 415, a mobile information terminal 417, and an external communication device 419. The I/O interface 31 may include a function for processing (converting, calculating, analyzing) information received from other electronic devices connected to the vehicle display system 10.

The display 40 is operatively coupled to the processor 33 and the image processing circuitry 35. Thus, the image displayed by the light modulation element 51 may be based on image data received from the processor 33 and/or the image processing circuitry 35. The processor 33 and the image processing circuitry 35 control the image displayed by the light modulation element 51 based on information obtained from the I/O interface 31.

The vehicle ECU 401 acquires information about the state of the vehicle 1 (e.g., the ON/OFF state (one example of startup information) of a startup switch (e.g., an accessory switch: ACC or an ignition switch: IGN), mileage, vehicle speed, accelerator pedal opening, brake pedal opening, engine throttle opening, injector fuel injection amount, engine RPM, motor RPM, steering angle, shift position, drive mode, various warning states, attitude (including roll angle and/or pitch angle), vehicle vibration (including vibration magnitude, frequency, and/or frequency)) from sensors and switches provided on the vehicle 1, and collects and manages (which may also include control) the state of the vehicle 1, and as part of its function, can output a signal indicating a numerical value of the state of the vehicle 1 (e.g., the vehicle speed of the vehicle 1) to the processor 33 of the display control device 30. In addition to or instead of simply transmitting a value detected by a sensor or the like (e.g., the pitching angle is 3 degrees in the forward tilt direction) to the processor 33, the vehicle ECU 401 may transmit to the processor 33 a determination result based on one or more states of the vehicle 1 including the value detected by the sensor (e.g., that the vehicle 1 satisfies a predetermined condition for a forward tilt state) and/or an analysis result (e.g., that the vehicle has been tilted forward by braking in combination with information on the brake pedal opening degree). For example, the vehicle ECU 401 may output to the display control device 30 a signal indicating a determination result that the vehicle 1 satisfies a predetermined condition previously stored in a memory (not shown) of the vehicle ECU 401. The I/O interface 31 may acquire the above-mentioned information from a sensor or switch provided on the vehicle 1 without going through the vehicle ECU 401.

In addition, the vehicle ECU 401 may output an instruction signal to the display control device 30 to indicate the image to be displayed by the vehicle display system 10, and in this case, the coordinates, size, type, display mode, notification necessity of the image, and/or necessity-related information that is the basis for determining the notification necessity of the image may be added to the instruction signal and transmitted.

The road information database 403 is included in a navigation device (not shown) provided in the vehicle 1, or an external server connected to the vehicle 1 via an external communication interface (I/O interface 31), and may read out information about the surroundings of the vehicle 1 (information related to real objects around the vehicle 1), such as road information (lanes, white lines, stop lines, crosswalks, road widths, number of lanes, intersections, curves, forks, traffic regulations, etc.) on which the vehicle 1 is traveling, and information about features (buildings, bridges, rivers, etc.), including their presence or absence, position (including distance to the vehicle 1), direction, shape, type, detailed information, etc., based on the position of the vehicle 1 acquired from the vehicle position detection unit 405 described later, and transmit the information to the processor 33. The road information database 403 may also calculate an appropriate route (navigation information) from the departure point to the destination, and output a signal indicating the navigation information or image data indicating the route to the processor 33.

The vehicle position detection unit 405 is a GNSS (Global Navigation Satellite System) or the like provided in the vehicle 1, which detects the current position and orientation of the vehicle 1, and outputs a signal indicating the detection result to the road information database 403, the mobile information terminal 417 described later, and/or the external communication device 419 via the processor 33 or directly. The road information database 403, the mobile information terminal 417 described later, and/or the external communication device 419 may select/generate information about the surroundings of the vehicle 1 by acquiring the position information of the vehicle 1 from the vehicle position detection unit 405 continuously, intermittently, or for each predetermined event, and output the information to the processor 33.

The operation detection unit 407 is, for example, a hardware switch provided on the CID (Center Information Display) or instrument panel of the vehicle 1, or a software switch that combines an image and a touch sensor, and outputs operation information based on an operation by an occupant of the vehicle 1 (a user sitting in the driver's seat and/or a user sitting in the passenger seat) to the processor 33. For example, the operation detection unit 407 outputs to the processor 33 display area setting information based on an operation to move the virtual image display area 100 by a user's operation, eye box setting information based on an operation to move the eye box 200, information based on an operation to set the observer's eye position 700, and the like.

The eye position detection unit 409 may include a camera such as an infrared camera that detects the eye position 700 (see FIG. 1) of an observer sitting in the driver's seat of the vehicle 1, and may output the captured image to the processor 33. The processor 33 may obtain a captured image (one example of information that can estimate the eye position 700) from the eye position detection unit 409, and may analyze the captured image using a method such as pattern matching to detect the coordinates of the observer's eye position 700, and output a signal indicating the coordinates of the detected eye position 700 to the processor 33.

The eye position detection unit 409 may also output the analysis results of the image captured by the camera (e.g., a signal indicating to which spatial region the observer's eye position 700 belongs in which a plurality of pre-set display parameters correspond) to the processor 33. Note that the method of acquiring the observer's eye position 700 of the subject vehicle 1 or information that can estimate the observer's eye position 700 is not limited to these, and may be acquired using known eye position detection (estimation) techniques.

The eye position detection unit 409 may also detect the speed and/or direction of movement of the observer's eye position 700, and output a signal indicating the speed and/or direction of movement of the observer's eye position 700 to the processor 33.

The eye position detection unit 409 may also have a function as a gaze direction detection unit 409. The gaze direction detection unit 409 may include an infrared camera or a visible light camera that captures an image of the face of an observer sitting in the driver's seat of the vehicle 1, and output the captured image to the processor 33. The processor 33 can obtain an image (an example of information that can estimate the gaze direction) from the gaze direction detection unit 409 and identify the observer's gaze direction (and/or the gaze position) by analyzing the captured image. The gaze direction detection unit 409 may analyze the captured image from the camera and output a signal indicating the observer's gaze direction (and/or the gaze position) as the analysis result to the processor 33. Note that the method of acquiring information that can estimate the gaze direction of the observer of the vehicle 1 is not limited to these, and may be acquired using other known gaze direction detection (estimation) techniques such as the EOG (Electro-oculogram) method, the corneal reflex method, the scleral reflex method, the Purkinje image detection method, the search coil method, and the infrared fundus camera method.

The external sensor 411 detects real objects present around the vehicle 1 (in front, to the sides, and behind). The real objects detected by the external sensor 411 may include, for example, obstacles (pedestrians, bicycles, motorcycles, other vehicles, etc.), the road surface of the driving lane described below, dividing lines, roadside objects, and/or features (buildings, etc.). The external sensor is composed of, for example, a detection unit consisting of a radar sensor such as a millimeter wave radar, ultrasonic radar, or laser radar, a camera, or a combination of these, and a processing device that processes (data fuses) the detection data from the one or more detection units. Conventional well-known methods are applied to object detection by these radar sensors and camera sensors. Object detection by these sensors may detect the presence or absence of a real object in three-dimensional space, and if a real object is present, the position of the real object (relative distance from the vehicle 1, left/right position when the traveling direction of the vehicle 1 is the forward/backward direction, up/down position, etc.), size (size in the lateral direction (left/right), height direction (up/down), etc.), direction of movement (lateral direction (left/right), depth direction (front/back)), speed of movement (lateral direction (left/right), depth direction (front/back)), and/or type, etc.).

One or more external sensors 411 can detect a real object in front of the vehicle 1 at each detection period of the sensor, and output real object information (presence or absence of a real object, and if a real object exists, information on the position, size, and/or type of each real object, etc.) which is an example of real object information to the processor 33. Note that this real object information may be transmitted to the processor 33 via other devices (e.g., the vehicle ECU 401). In addition, when a camera is used as a sensor, an infrared camera or a near-infrared camera is preferable so that a real object can be detected even when the surroundings are dark, such as at night. In addition, when a camera is used as a sensor, a stereo camera that can also obtain distance, etc. by parallax is preferable. Here, the relative distance of the detected real object 300 from the vehicle 1 is defined as the object distance (distance to the real object) D4.

The brightness detection unit 413 detects the illuminance or brightness of a predetermined range of the foreground in front of the vehicle cabin of the vehicle 1 as the external brightness (an example of brightness information), or the illuminance or brightness inside the vehicle cabin as the interior brightness (an example of brightness information). The brightness detection unit 413 is, for example, a phototransistor or a photodiode, and is mounted on the instrument panel, rearview mirror, or HUD device 20 of the vehicle 1 shown in FIG. 1.

The IMU 415 may include a combination of one or more sensors (e.g., an accelerometer and a gyroscope) configured to detect the position, orientation, and changes therein (speed of change, acceleration of change) of the vehicle 1 based on inertial acceleration. The IMU 415 may output the detected values (the detected values include signals indicating the position, orientation, and changes therein (speed of change, acceleration of change) of the vehicle 1) and the results of analyzing the detected values to the processor 33. The results of the analysis may be a signal indicating a determination result as to whether or not the detected values satisfy a predetermined condition, and may be, for example, a signal indicating that the behavior (vibration) of the vehicle 1 is small based on a value regarding the change (speed of change, acceleration of change) in the position or orientation of the vehicle 1.

The mobile information terminal 417 is a smartphone, a laptop computer, a smartwatch, or other information device that can be carried by the observer (or other passengers of the vehicle 1). The I/O interface 31 can communicate with the mobile information terminal 417 by pairing with the mobile information terminal 417, and acquires data recorded in the mobile information terminal 417 (or a server via the mobile information terminal). The mobile information terminal 417 has, for example, the same functions as the above-mentioned road information database 403 and vehicle position detection unit 405, and may acquire the road information (an example of real object related information) and transmit it to the processor 33. The mobile information terminal 417 may also acquire commercial information (an example of real object related information) related to commercial facilities in the vicinity of the vehicle 1 and transmit it to the processor 33. The mobile information terminal 417 may transmit schedule information of the user of the mobile information terminal 417 (e.g., the observer), information on incoming calls to the mobile information terminal 417, information on received e-mails, etc. to the processor 33, and the processor 33 and the image processing circuit 35 may generate and/or transmit image data relating to these.

The external communication device 419 is a communication device that exchanges information with the vehicle 1, and includes, for example, other vehicles connected to the vehicle 1 by vehicle-to-vehicle communication (V2V: Vehicle To Vehicle), pedestrians (mobile information terminals carried by pedestrians) connected to the vehicle 1 by pedestrian-to-pedestrian communication (V2P: Vehicle To Pedestrian), and network communication devices connected by road-to-vehicle communication (V2I: Vehicle To Roadside Infrastructure), and in a broad sense includes everything connected to the vehicle 1 by communication (V2X: Vehicle To Everything). The external communication device 419 may acquire the positions of, for example, pedestrians, bicycles, motorcycles, other vehicles (preceding vehicles, etc.), road surfaces, dividing lines, roadside objects, and/or features (buildings, etc.) and output them to the processor 33. In addition, the external communication device 419 has the same function as the vehicle position detection unit 405 described above, and may acquire position information of the vehicle 1 and transmit it to the processor 33, and may also have the function of the road information database 403 described above, and may acquire the road information (an example of real object related information) and transmit it to the processor 33. Note that the information acquired from the external communication device 419 is not limited to the above.

The software components stored in memory 37 include an eye position detection module 502, an eye position estimation module 504, an eye position prediction module 506, an eye position state determination module 508, a vehicle state determination module 510, an eye tracking image processing module 512, a graphics module 514, a light source driving module 516, and an actuator driving module 518.

FIGS. 11A, 11B, and 11C are flow diagrams showing a method S100 for performing an operation of correcting an image based on the eye position of an observer. The method S100 is performed in a HUD device 20 including a display, and a display control device 30 that controls the HUD device 20. Some operations of the method S100 shown below are optionally combined, some operation procedures are optionally changed, and some operations are optionally omitted.

First, the display control device 30 (processor 33) detects the viewer's eye position 700 (step S110).

(Step S112)
In step S110 in some embodiments, the display control device 30 (processor 33) detects the eye position 700 of the observer via the eye position detection unit 409 by executing the eye position detection module 502 in FIG. 10 (obtains information indicating the eye position 700). The eye position detection module 502 includes various software components for executing various operations related to detecting coordinates indicating the observer's eye position 700 (positions in the X and Y axes, which are an example of a signal indicating the eye position 700), detecting coordinates indicating the height of the observer's eyes (positions in the Y axis direction, which are an example of a signal indicating the eye position 700), detecting coordinates indicating the height and depth direction position of the observer's eyes (positions in the Y and Z axes, which are an example of a signal indicating the eye position 700), and/or detecting coordinates indicating the observer's eye position 700 (positions in the X, Y, and Z axes, which are an example of a signal indicating the eye position 700).

The eye position 700 detected by the eye position detection module 502 may include the right eye and left eye positions 700R, 700L, a predetermined one of the right eye position 700R and the left eye position 700L, one of the right eye position 700R and the left eye position 700L that is detectable (easy to detect), or a position calculated from the right eye position 700R and the left eye position 700L (for example, the midpoint between the right eye position and the left eye position). For example, the eye position detection module 502 determines the eye position 700 based on the observed position acquired from the eye position detection unit 409 immediately before the timing of updating the display settings.

The eye position detection unit 409 may also detect the movement direction and/or movement speed of the observer's eye position 700 based on multiple observation positions with different detection timings of the observer's eyes acquired from the eye position detection unit 409, and output a signal indicating the movement direction and/or movement speed of the observer's eye position 700 to the processor 33.

(Step S114)
Also, in step S110 in some embodiments, the display control device 30 (processor 33) may acquire information that allows the eye position to be estimated by executing the eye position estimation module 504 (step S114). The information that allows the eye position to be estimated is, for example, the captured image acquired from the eye position detection unit 409, the position of the driver's seat of the vehicle 1, the position of the observer's face, the height of the seat, or the observation positions of the eyes of multiple observers. The eye position estimation module 504 estimates the eye position 700 of the observer of the vehicle 1 from the information that allows the eye position to be estimated. The eye position estimation module 504 includes various software components for executing various operations related to estimating the observer's eye position 700, such as estimating the observer's eye position 700 from the captured image acquired from the eye position detection unit 409, the position of the driver's seat of the vehicle 1, the position of the observer's face, the height of the seat, or the observation positions of the eyes of multiple observers. That is, the eye position estimation module 504 may include table data, arithmetic expressions, and the like for estimating the viewer's eye positions 700 from information that enables the estimation of the eye positions.

(Step S116)
Also, in step S110 in some embodiments, the display control device 30 (processor 33) may execute the eye position prediction module 506 to obtain information capable of predicting the viewer's eye position 700 (step S116). The information capable of predicting the viewer's eye position 700 may be, for example, the latest observation position obtained from the eye position detection unit 409, or one or more observation positions obtained in the past. The eye position prediction module 506 includes various software components for performing various operations related to predicting the eye position 700 based on the information capable of predicting the viewer's eye position 700. Specifically, for example, the eye position prediction module 506 predicts the viewer's eye position 700 at the timing when the image to which the new display settings are applied is viewed by the viewer. The eye position prediction module 506 may predict the next value using one or more past observation positions using a prediction algorithm such as, for example, the least squares method, a Kalman filter, an α-β filter, or a particle filter.

(Step S120)
Next, the display control device 30 (processor 33) determines whether a predetermined condition is satisfied (step S120).

Figure 11B is a diagram explaining the operation of the eye position state determination module 508 in step S130.

(Step S130)
In step S120 in some embodiments, the display control device 30 (processor 33) executes the eye position state determination module 508 of FIG. 10 to determine whether the eye position 700 (movement of the eye position 700) satisfies a specified condition based on the information indicating the eye position obtained in step S110, information that allows the eye position to be estimated, or information that allows the eye position to be predicted.

FIG. 12 shows (11) vertical eye position (or head position) Py (Y1, Y2, Y3, . . . Y10), (12) change amount ΔPy of the vertical eye position (or head position) (Py1 (=Y2-Y1), Py2 (=Y3-Y2), Py3 (=Y4-Y3), . . . Py9 (=Y10-Y9)), and (13) movement speed Vy of the vertical eye position (or head position) (Vy1 (=Py1/Δt), Vy2 (=Py2/Δt), Vy3 (=Py3/Δt), . . . 9 is a table showing (21) the left- right eye position (or head position) Px (X1, X2, X3, ... X10), (22) the amount of change ΔPx of the left- right eye position (or head position) (Px1 (=X2-X1), Px2 (=X3-X2), Px3 (=X4-X3), ... Px9 (=X10-X9)), and (23) the movement speed Vx of the left- right eye position (or head position) (Vx1 (=Px1/Δt), Vx2 (=Px2/Δt), Vx3 (=Px3/Δt), ... Vx9 (=Px9/Δt)).

(Step S131)
In step S130 in some embodiments, the display control device 30 (processor 33) may determine that a predetermined condition is satisfied when the eye position in the left-right direction changes continuously in one direction by executing the eye position state determination module 508 in Fig. 10. The eye position state determination module 508 may determine that the predetermined condition is satisfied when it is detected that the change amount ΔPx of the eye position in the left-right direction shown in Fig. 12 changes continuously in one direction (here, the right direction), such as Px3 (right direction) → Px4 (right direction).

(Step S132)
In addition, in step S130 in some embodiments, the display control device 30 (processor 33) may execute the eye position state determination module 508 in Fig. 10 to determine that a predetermined condition is satisfied when the vertical eye position changes continuously in one direction. For example, the eye position state determination module 508 may determine that the predetermined condition is satisfied when it is detected that the vertical eye position change amount ΔPy shown in Fig. 12 changes continuously in one direction (here, downward), such as Py3 (downward) -> Py4 (downward).

(Step S133)
In addition, in step S130 in some embodiments, the display control device 30 (processor 33) can execute the distance determination module 511 of Figure 10 to, for example, compare the object distance D4 with the imaging distance D10, and if the object distance D4 is greater than the imaging distance D10, it may determine that a specified condition is satisfied.

(Step S134)
In addition, in step S130 in some embodiments, the display control device 30 (processor 33) may execute the distance determination module 511 in Fig. 10 to compare the object distance D4 with a predetermined second distance threshold (e.g., 20 m) stored in advance in the memory 37, and determine that the predetermined condition is satisfied. Here, the second distance threshold is set to about five times the imaging distance D10.

Figure 11C is a diagram explaining the operation of the vehicle state determination module 510 in step S140.

(Step S140)
In step S120 in some embodiments, the display control device 30 (processor 33) may execute the vehicle state determination module 510 in FIG. 10 to determine whether the state of the vehicle satisfies a predetermined condition.

(Step S141)
In step S140 in some embodiments, the display control device 30 (processor 33) executes the vehicle state determination module 510 of FIG. 10 to estimate whether or not the vehicle 1 is moving based on, for example, information such as mileage, vehicle speed, and shift position obtained from the vehicle ECU 401, information indicating the vehicle position obtained from the vehicle position detection unit 405, and information obtained from the IMU 415, and if it is estimated that the vehicle 1 is moving, it may determine that the specified condition is satisfied.

(Step S150)
11A again, after it is determined in step S120 whether the predetermined condition is satisfied, the display control device 30 (processor 33) executes the eye tracking image processing module 512 to correct the position of the image to be displayed on the display 40 based on the observer's eye position 700 (specifically, the eye position 700 is the eye position Py in the vertical direction and the eye position Px in the horizontal direction in FIG. 12, and the eye position in the front-back direction may also be added) (eye tracking image processing).

The eye tracking image processing module 512 in FIG. 10 switches between a first image correction process S160 (step S160), a second image correction process S170 (step S170), and a third image correction process S180 (step S180) based on the determination result in step S120.

(First image correction process S160)
When it is determined in step S120 that the predetermined condition is satisfied (step S131), the eye tracking image processing module 512 in Fig. 10 corrects the left-right position of the virtual image V by a first left-right correction amount (first correction amount) Cx according to the left-right eye position change amount ΔPx. The first left-right correction amount Cx (the same applies to the second left-right correction amount Cxq1 and the third left-right correction amount Cxq2 described later) is a parameter that gradually increases as the left-right eye position change amount ΔPx increases. In addition, the first left-right correction amount Cx (the same applies to the second left-right correction amount Cxq1 and the third left-right correction amount Cxq2 described later) is a parameter that gradually increases as the perception distance D30 set for the virtual image V becomes longer. The first image correction process S160 includes image position correction to completely reproduce natural motion parallax as if the virtual image V were fixed at the set target position PT even when viewed from each left-right eye position Px, and may also include image position correction to approximate natural motion parallax in a broad sense. That is, the first image correction process S160 aligns the display position of the virtual image V with the position of the intersection of the straight line connecting the target position PT set on the virtual image V and the observer's eye position 700 with the virtual image display area VS (brings the display position of the virtual image V closer).

(First image correction process S160)
When it is determined in step S120 that the predetermined condition is satisfied (step S132), the eye tracking image processing module 512 in Fig. 10 corrects the vertical position of the virtual image V by a first vertical correction amount (first correction amount) Cy according to the vertical eye position change amount ΔPy. The first vertical correction amount Cy (the same applies to the second vertical correction amount Cyr1 and the third vertical correction amount Cyr2 described later) is a parameter that gradually increases as the vertical eye position change amount ΔPy increases. In addition, the first vertical correction amount Cy (the same applies to the second vertical correction amount Cyr1 and the third vertical correction amount Cyr2 described later) is a parameter that gradually increases as the perception distance D30 set for the virtual image V becomes longer. The first image correction process S160 includes image position correction to completely reproduce natural motion parallax as if the virtual image V were fixed at the set target position PT even when viewed from each vertical eye position Py, and may also include image position correction to approximate natural motion parallax in a broad sense. That is, the first image correction process S160 aligns the display position of the virtual image V with the intersection position of the virtual image display area VS and the straight line connecting the target position PT set on the virtual image V and the observer's eye position 700 (brings the display position of the virtual image V closer).

(Second image correction process S170)
When it is determined in step S120 that the predetermined condition is satisfied (step S133), the eye tracking image processing module 512 in Fig. 10 corrects the left-right position of the virtual image V by a second left-right correction amount (second correction amount) Cxq1 according to the left-right eye position change amount ΔPx. As shown in Fig. 13(a), the second left-right correction amount Cxq1 is a value obtained by multiplying the first left-right correction amount Cx for the left-right eye position change amount ΔPx in the first image correction process S160 by a first left-right coefficient (first predetermined coefficient) Q1 of 0.9, which is equal to or less than 1. Specifically, for example, when the first left-right correction amount Cx for the left-right eye position change amount ΔPx is 100%, the second left-right correction amount Cxq1 for the same up-down eye position change amount ΔPx is 90%. In a broad sense, the second left-right correction amount Cxq1 only needs to be smaller than the first left-right correction amount Cx so as to suppress the first left-right correction amount Cx, and therefore it is sufficient for it to be less than 100% of the first left-right correction amount Cx, but preferably it is less than 90% of the first left-right correction amount Cx.

(Second image correction process S170)
When it is determined in step S120 that the predetermined condition is satisfied (step S133), the eye tracking image processing module 512 in FIG. 10 corrects the vertical position of the virtual image V by a second vertical correction amount (second correction amount) Cyr1 according to the vertical eye position change amount ΔPy. As shown in FIG. 13B, the second vertical correction amount Cyr1 is a value obtained by multiplying the first vertical correction amount Cy for the vertical eye position change amount ΔPy in the first image correction process S160 by a first vertical coefficient (first predetermined coefficient) R1 of 0.8, which is equal to or less than 1. Specifically, for example, when the first vertical correction amount Cy for the vertical eye position change amount ΔPy is 100%, the second vertical correction amount Cyr1 for the same vertical eye position change amount ΔPy is 80%. In a broad sense, the second up-down correction amount Cyr1 should be smaller than the first up-down correction amount Cy so as to suppress the first up-down correction amount Cy , and therefore it is sufficient for it to be less than 100% of the first up-down correction amount Cy, but preferably it is less than 80% of the first up-down correction amount Cy.

(An example of step S170)
It is preferable that the first left-right coefficient Q1 be greater than the first up-down coefficient R1.

(An example of step S170)
In step S170 in some embodiments, the display control device 30 (processor 33) may, after transitioning to the second image correction process as shown in Fig. 13A, gradually decrease the second left-right correction amount Cxq1 for the left-right eye position change amount ΔPx over time as it becomes farther away from the first left-right correction amount Cx for the left-right eye position change amount ΔPx (see Cxq11). For example, assuming that the first left-right correction amount Cx for the left-right eye position change amount ΔPx is 100% and the second left-right correction amount Cxq1 is 90%, the display control device 30 (processor 33) may gradually decrease the second left-right correction amount Cxq1 over time as it becomes farther away from the first left-right correction amount Cx, such as 100% -> 98% -> 96% ... -> 92%. Similarly, the second vertical correction amount Cyr1 for the vertical eye position change amount ΔPy may be gradually decreased over time as it becomes farther away from the first vertical correction amount Cy for the vertical eye position change amount ΔPy.

This correction amount Cxq11, which changes over time, gradually decreases over time as the object distance increases, and conversely, gradually increases over time as the object distance decreases, but the rate of change may be different when the object distance increases and when the object distance decreases.

(Third image correction process S180)
When it is determined in step S120 that the predetermined condition is satisfied (step S134), the eye tracking image processing module 512 in FIG. 10 corrects the left-right position of the virtual image V by a third left-right correction amount (third correction amount) Cxq2 corresponding to the left-right eye position change amount ΔPx. As shown in FIG. 13A, the third left-right correction amount Cxq2 is a value obtained by multiplying the first left-right correction amount Cx for the left-right eye position change amount ΔPx in the first image correction process S160 by a second left-right coefficient (second predetermined coefficient) Q2 of 0.5, which is smaller than the first left-right coefficient Q1. Specifically, for example, when the first left-right correction amount Cx for the left-right eye position change amount ΔPx is 100%, the third left-right correction amount Cxq2 for the same left- right eye position change amount ΔPx is 50%. In a broad sense, the third left-right correction amount Cxq2 only needs to be smaller than the first left-right correction amount Cx so as to suppress the first left-right correction amount Cx, and therefore it is sufficient if the third left-right correction amount Cxq2 is less than 100% of the first left-right correction amount Cx, but preferably, it is less than 50% of the first left-right correction amount Cx.

(Third image correction process S180)
When it is determined in step S120 that the predetermined condition is satisfied (step S134), the eye tracking image processing module 512 in FIG. 10 corrects the vertical position of the virtual image V by a third vertical correction amount (third correction amount) Cyr2 according to the vertical eye position change amount ΔPy. As shown in FIG. 13B, the third vertical correction amount Cyr2 is a value obtained by multiplying the first vertical correction amount Cy for the vertical eye position change amount ΔPy in the first image correction process S160 by a second vertical coefficient (second predetermined coefficient) R2 that is smaller than the first vertical coefficient R1 of 0.4. Specifically, for example, when the first vertical correction amount Cy for the vertical eye position change amount ΔPy is 100%, the third vertical correction amount Cyr2 for the same vertical eye position change amount ΔPy is 40%. In a broad sense, the third vertical correction amount Cyr2 should be smaller than the first vertical correction amount Cy so as to suppress the first left-right correction amount Cx, and therefore it is sufficient for it to be less than 100% of the first vertical correction amount Cy, but preferably it is less than 40% of the first vertical correction amount Cy.

(An example of step S180)
In some embodiments, in step S180, the display control device 30 (processor 33) may, after proceeding to the third image correction process as shown in Fig. 13A, gradually reduce the third left-right correction amount Cxq2 for the left-right eye position change amount ΔPx over time so as to move away from the second left-right correction amount Cxq1 for the left-right eye position change amount ΔPx (see Cxq21). For example, if the second left-right correction amount Cxq1 for the left-right eye position change amount ΔPx is 90% and the third left-right correction amount Cxq2 is 50%, the display control device 30 (processor 33) may gradually reduce the third left-right correction amount Cxq2 over time so as to move away from the second left-right correction amount Cxq1, such as 90% → 80% → 70% → ... → 50%. Similarly, the third vertical correction amount Cyr2 for the vertical eye position change amount ΔPy may be gradually decreased over time so as to move away from the second vertical correction amount Cyr1 for the vertical eye position change amount ΔPy.

This correction amount Cxq21, which changes over time, gradually decreases over time as the object distance increases, and conversely, gradually increases over time as the object distance decreases, but the rate of change may be different when the object distance increases and when the object distance decreases.

(Step S190)
In some embodiments, in step S160, if the display control device 30 (processor 33) determines that a predetermined release condition is satisfied, the process proceeds from the second image correction process S170 to the first image correction process S160.

The predetermined release condition includes the passage of a predetermined time (e.g., 20 seconds) after transition to the second image correction process S170. The eye tracking image processing module 512 may perform timing after transition to the second image correction process S170, and determine that the release condition is satisfied when the predetermined time previously stored in the memory 37 (or set by the operation detection unit 407) has elapsed.

(Step S190)
In some embodiments, in step S160, if the display control device 30 (processor 33) determines that a predetermined release condition is satisfied, the process proceeds from the third image correction process S180 to the second image correction process S170.

The predetermined release condition includes the passage of a predetermined time (e.g., 20 seconds) after transition to the third image correction process S180. The eye tracking image processing module 512 may perform timing after transition to the third image correction process S180, and determine that the release condition is satisfied when the predetermined time previously stored in the memory 37 (or set by the operation detection unit 407) has elapsed.

The predetermined release condition may include the predetermined condition no longer being satisfied in step S120. In other words, the predetermined release condition may include detecting that at least one of steps S131 to S134 and step S141 has transitioned from a state in which the predetermined condition is satisfied to a state in which the predetermined condition is no longer satisfied. The predetermined release condition may also include the passage of a predetermined time (e.g., 20 seconds) since the predetermined condition no longer being satisfied in step S120.

Referring again to FIG. 10, the graphics module 514 in FIG. 10 includes various known software components for performing image processing such as rendering to generate image data and drive the display 40. The graphics module 514 may also include various known software components for changing the type (moving image, still image, shape), arrangement (position coordinates, angle), size, display distance (in the case of 3D), and visual effects (e.g., brightness, transparency, saturation, contrast, or other visual characteristics) of the image to be displayed. The graphics module 514 generates image data so that the type of image (one example of a display parameter), the position coordinates of the image (one example of a display parameter), the angle of the image (one example of a display parameter, such as a pitching angle with the X-axis as its axis, a yaw rate angle with the Y-axis as its axis, and a rolling angle with the Z-axis as its axis), the size of the image (one example of a display parameter), the color of the image (one example of a display parameter set by hue, saturation, brightness, etc.), and the intensity of the perspective representation of the image (one display parameter set by the position of the vanishing point, etc.) can be visually recognized by the observer, and can drive the light modulation element 50.

The light source driving module 516 includes various known software components for executing the driving of the light source unit 24. The light source driving module 516 may drive the light source unit 24 based on the set display parameters.

The actuator drive module 518 includes various known software components for executing the operation of driving the first actuator 28 and/or the second actuator 29. The actuator drive module 518 can drive the first actuator 28 and the second actuator 29 based on the set display parameters.

As described above, the display control device 30 of this embodiment is a display control device (30) that is composed of at least a display (40) that displays an image and a relay optical system (80) that projects light of the image displayed by the display (40) onto a projection target member, and executes display control in a head-up display device (20) that allows a vehicle user to visually recognize a virtual image of the image superimposed on a foreground, the display control device (30) comprising one or more processors (33), a memory (37), one or more computer programs that are stored in the memory (37) and configured to be executed by the one or more processors (33), and an outside-vehicle sensor (411) that detects real objects present around the vehicle, the one or more processors (33) acquiring the user's eye position or head position (Py) in a vertical direction of the vehicle and the eye position or head position (Px) in a horizontal direction of the vehicle, and calculating a change (ΔPy) in the eye position or head position (Py) in the vertical direction and a change ( ΔPx ) in the eye position or head position (Px) in the horizontal direction. and a second image correction process (S170) for correcting the position of the image to be displayed on the display (40) with a second correction amount (Cxq1, Cyr1) obtained by multiplying the first correction amount (Cx, Cy) by a first predetermined coefficient (Q1, R1) that is equal to or less than 1. When an object distance (D4) that is a distance from the vehicle to the real object in the foreground that is superimposed on the virtual image is equal to or less than a first distance threshold, the first image correction process is executed, and when the object distance (D4) is greater than the first distance threshold, the second image correction process (S170) is executed.
As a result, even if the observer's eye position is detected (misdetected) as having moved an amount greater than a predetermined amount, the position correction of the image can be suppressed, thereby reducing the sense of discomfort felt by the observer.

In some embodiments, the first distance threshold may be the distance (D10) from the vehicle to the area in which the virtual image is displayed.

In addition, in some embodiments, the method may further include a third image correction process (S180) for correcting the position of the image to be displayed on the display (40) with a third correction amount (Cxq2, Cyr2) obtained by multiplying the first correction amount (Cx, Cy) by a second predetermined coefficient (Q2, R2), and the third image correction process (S180) may be executed when the object distance (D4) is greater than a second distance threshold.

In some embodiments, the second distance threshold may be greater than the first distance threshold.

In some embodiments, the second predetermined coefficient (Q2, R2) may be smaller than the first predetermined coefficient (Q1, R1).

In addition, in some embodiments, the first predetermined coefficient (Q1) when correcting the position of the image to be displayed on the display (40) based on the change ( ΔPx ) in the left-right eye position or head position (Px) may be greater than the first predetermined coefficient (R1) when correcting the position of the image to be displayed on the display (40) based on the change (ΔPy) in the up-down eye position or head position (Py).

In addition, in some embodiments, the first predetermined coefficient (Q1 , R1 ) may change over time in the second image correction process as the second correction amount (Cxq1, Cyr1) becomes farther away from the first correction amount (Cx, Cy) in the first image correction process.

In some embodiments, a head-up display device 20 is a display control device (30) that executes display control in a head-up display device (20) that is composed of at least a display (40) that displays an image and a relay optical system (80) that projects light of the image displayed by the display (40) onto a projection target member, and causes a user of a vehicle to visually recognize a virtual image of the image superimposed on a foreground. The display control device (30) includes one or more processors (33), a memory (37), one or more computer programs that are stored in the memory (37) and configured to be executed by the one or more processors (33), and an outside-vehicle sensor (411) that detects real objects present around the vehicle. The one or more processors (33) acquire the user's eye position or head position (Py) in a vertical direction of the vehicle and the eye position or head position (Px) in a horizontal direction of the vehicle, and calculates a change (ΔPy) in the eye position or head position (Py) in the vertical direction and a change ( ΔPx ) in the eye position or head position (Px) in the horizontal direction. and a second image correction process (S170) for correcting the position of the image to be displayed on the display (40) with a second correction amount (Cxq1, Cyr1) obtained by multiplying the first correction amount (Cx, Cy) by a first predetermined coefficient (Q1, R1) that is equal to or less than 1. When an object distance (D4) that is a distance from the vehicle to the real object in the foreground that is superimposed on a virtual image is equal to or less than a first distance threshold, the first image correction process is executed, and when the object distance (D4) is greater than the first distance threshold, the second image correction process (S170) is executed.

The operations of the above processing steps can be implemented by executing one or more functional modules of an information processing device, such as a general-purpose processor or an application-specific chip. All of these modules, combinations of these modules, and/or combinations with known hardware that can replace their functions, are within the scope of protection of the present invention.

The functional blocks of the vehicle display system 10 are optionally implemented by hardware, software, or a combination of hardware and software to implement the principles of the various embodiments described. It will be understood by those skilled in the art that the functional blocks described in FIG. 10 may be optionally combined or a functional block may be separated into two or more sub-blocks to implement the principles of the embodiments described. Thus, the description herein optionally supports any possible combination or division of the functional blocks described herein.

1: Vehicle 2: Projected part 5: Dashboard 6: Road surface 10: Vehicle display system 20: HUD device 21: Light exit window 22: Housing 24: Light source unit 28: First actuator 29: Second actuator 30: Display control device 31: I/O interface 33: Processor 35: Image processing circuit 37: Memory 40: Display device 205: Center 300: Real object 311: Road surface 313: Person 314: Other vehicle 315: Building 401: Vehicle ECU
403: Road information database 405: Vehicle position detection unit 407: Operation detection unit 409: Eye position detection unit 411: Vehicle exterior sensor 413: Brightness detection unit 417: Portable information terminal 419: External communication device 502: Eye position detection module 504: Eye position estimation module 506: Eye position prediction module 508: Eye position state determination module 510: Vehicle state determination module 511: Distance determination module 512: Eye tracking image processing module 514: Graphics module 516: Light source driving module 518: Actuator driving module 710: Head position Cx: First left/right correction amount (first correction amount)
Cxq1: second left/right correction amount (second correction amount)
Cxq2: Third left/right correction amount (third correction amount)
Cy: First up/down correction amount (first correction amount)
Cyr1: Second up/down correction amount (second correction amount)
Cyr2: Third up/down correction amount (third correction amount)
Q1: First left-right coefficient (first predetermined coefficient)
Q2: Second left-right coefficient (second predetermined coefficient)
R1: First up/down coefficient (first predetermined coefficient)
R2: Second up/down coefficient (second predetermined coefficient)
D10: Imaging distance (first distance threshold)
D30: Perceived distance D4: Object distance (distance to real object)
FU: Virtual object (perceptual virtual image)
K: Display light PS: Virtual image display area PT: Target position Px: Eye position (head position) in the left-right direction
Py: Eye position in the vertical direction (head position)
V: Virtual image V60: AR virtual image (upper virtual image)
V61: Navigation virtual image V62: Navigation virtual image V63: Enhanced virtual image V64: Enhanced virtual image V65: Enhanced virtual image V65: POI virtual image V70: Non-AR virtual image (lower virtual image)
VS: Virtual image display area Vx: Movement speed Vy: Movement speed t: Period time ΔPx: Amount of change ΔPy: Amount of change

Claims (8)

A display control device (30) that executes display control in a head-up display device (20) that is configured at least with a display device (40) that displays an image and a relay optical system (80) that projects light of the image displayed by the display device (40) onto a projection target member, and that allows a vehicle user to visually recognize a virtual image of the image superimposed on a foreground,
one or more processors (33);
A memory (37);
one or more computer programs stored in said memory (37) and configured to be executed by said one or more processors (33);
an outside sensor (411) for detecting real objects present around the vehicle;
a first image correction process (S160) in which the one or more processors (33) acquire the user's eye position or head position (Py) in a vertical direction of the vehicle and the user's eye position or head position (Px) in a horizontal direction of the vehicle, and correct the position of the image to be displayed on the display (40) with a first correction amount (Cx, Cy) based at least on a change ( ΔPy ) in the eye position or head position (Py) in the vertical direction and a change (ΔPx) in the eye position or head position (Px) in the horizontal direction;
a second image correction process (S170) for correcting a position of the image to be displayed on the display (40) with a second correction amount (Cxq1, Cyr1) obtained by multiplying the first correction amount (Cx, Cy) by a first predetermined coefficient (Q1, R1) that is equal to or smaller than 1;
When an object distance (D4) which is a distance from the vehicle to the real object in the foreground which is superimposed on the virtual image is equal to or less than a first distance threshold, the first image correction process is executed;
When the object distance (D4) is greater than the first distance threshold, the second image correction process (S170) is executed.
A display control device (30). The first distance threshold is a distance (D10) from the vehicle to an area where the virtual image is displayed.
A display control device (30) according to claim 1. a third image correction process (S180) for correcting a position of the image to be displayed on the display (40) with a third correction amount (Cxq2, Cyr2) obtained by multiplying the first correction amount (Cx, Cy) by a second predetermined coefficient (Q2, R2);
When the object distance (D4) is greater than a second distance threshold, the third image correction process (S180) is executed.
A display control device (30) according to claim 1. The second distance threshold is greater than the first distance threshold.
A display control device (30) according to claim 3. The second predetermined coefficient (Q2, R2) is smaller than the first predetermined coefficient (Q1, R1);
A display control device (30) according to claim 3. the first predetermined coefficient (Q1) when correcting the position of the image to be displayed on the display (40) based on the change ( ΔPx ) in the eye position or head position (Px) in the left-right direction is greater than the first predetermined coefficient (R1) when correcting the position of the image to be displayed on the display (40) based on the change (ΔPy) in the eye position or head position (Py) in the up-down direction.
A display control device (30) according to claim 1. The first predetermined coefficient (Q1 , R1 ) is determined in the second image correction process as follows:
the second correction amount (Cxq1, Cyr1) changes over time as it becomes farther from the first correction amount (Cx, Cy) in the first image correction process;
A display control device (30) according to claim 1. A display control device (30) that executes display control in a head-up display device (20) that is configured at least with a display device (40) that displays an image and a relay optical system (80) that projects light of the image displayed by the display device (40) onto a projection target member, and that allows a vehicle user to visually recognize a virtual image of the image superimposed on a foreground,
one or more processors (33);
A memory (37);
one or more computer programs stored in said memory (37) and configured to be executed by said one or more processors (33);
an outside sensor (411) for detecting real objects present around the vehicle;
a first image correction process (S160) in which the one or more processors (33) acquire the user's eye position or head position (Py) in a vertical direction of the vehicle and the user's eye position or head position (Px) in a horizontal direction of the vehicle, and correct the position of the image to be displayed on the display (40) with a first correction amount (Cx, Cy) based at least on a change ( ΔPy ) in the eye position or head position (Py) in the vertical direction and a change (ΔPx) in the eye position or head position (Px) in the horizontal direction;
a second image correction process (S170) for correcting a position of the image to be displayed on the display (40) with a second correction amount (Cxq1, Cyr1) obtained by multiplying the first correction amount (Cx, Cy) by a first predetermined coefficient (Q1, R1) that is equal to or smaller than 1;
When an object distance (D4) which is a distance from the vehicle to the real object in the foreground which is superimposed on the virtual image is equal to or less than a first distance threshold, the first image correction process is executed;
When the object distance (D4) is greater than the first distance threshold, the second image correction process (S170) is executed.
A display control device (30) that executes display control in a head-up display device (20) .

Images