JP7619007B2 - Display control device, head-up display device, and display control method - Google Patents

Description

The present disclosure relates to a display control device, a head-up display device, a display control method, and the like, which are used in a moving body such as a vehicle and allow an image to be superimposed and viewed on the foreground of the moving body (the actual view in the forward direction of the moving body as seen by the vehicle occupants).

Patent Document 1 describes a head-up display device (one example of a virtual image display device) in which display light projected onto a projection target such as the vehicle's front windshield is reflected toward a vehicle occupant (observer) inside the vehicle, allowing the observer to view a virtual image superimposed on the vehicle's foreground. In particular, the head-up display device described in Patent Document 1 virtually places a display object (virtual image) at a predetermined position (here, the position is referred to as a target position) in the depth or vertical and horizontal directions within the foreground space, and controls the image displayed inside the head-up display device as if the display object were present at the target position in the foreground, even if the attitude of the vehicle changes or the observer's eye position changes. In other words, such a head-up display device forms an augmented reality in which a virtual object is added to and displayed in a real landscape (foreground).

In the example of FIG. 4 of Patent Document 1, a virtual object in the shape of an arrow pointing forward (in other words, an arrow extending from the rear to the front) is displayed at a position slightly above the road surface on which the vehicle is traveling (a position above the road surface). In such a case, the viewer perceives the sense of perspective as the width of the rear end of the arrow (typically the width in the left-right direction) gradually narrows toward the tip, as in the example of FIG. 5 of Patent Document 1. In other words, the viewer perceives the rear end of the arrow as being close and the tip of the arrow as being far away. A technique such as linear perspective (perspective projection) is typically used to create this sense of perspective.

JP 2010-156608 A

In the meantime, in the head-up display device as described above, the area in which the virtual image can be displayed is typically a flat or curved area (here, this area will be referred to as the virtual image display area), and the virtual image display area is positioned so as to face directly toward the observer's line of sight. In other words, when viewed from the left or right of the vehicle, the angle between the virtual image display area in which the virtual image is displayed and the road surface on which the vehicle is traveling is set to approximately 90 degrees (the virtual image display area is not positioned so as to be approximately parallel to the road surface).

10 is a diagram showing the foreground viewed by an observer through the front windshield WS of the vehicle while the vehicle is traveling, and the perspective image 1100 displayed in the virtual image display area 1000 overlapping the foreground. FIG. 10 shows the aspect of the perspective image expressed as being arranged parallel to the road surface using linear perspective, with the left line of the vehicle's driving lane 1010 indicated by reference numeral 1011 and the right line indicated by reference numeral 1012, and the intersection of the extension of the left line 1011 and the extension of the right line 1012 is the vanishing point 1020. The perspective image 1100 in FIG. 10 is a trapezoid, and is expressed by linear perspective (one-point perspective) so that the intersection of the extension of the left side 1101 and the extension of the right side 1102 of the trapezoid coincides with the vanishing point 1020. This gives a pictorial representation of the perspective image 1100 being placed parallel to the plane of the driving lane 1010.

However, as described above, even if the perspective image 1100 shown in FIG. 10 is displayed in a virtual image display area that is not arranged so as to be approximately parallel to the road surface, i.e., is raised toward the observer by approximately 90 degrees from the road surface on which the vehicle is traveling, the distance between the far end 1110 of the perspective image 1100 and the driving lane 1010 is longer than the distance between the near end 1120 and the driving lane 1010, and the far end 1110 of the perspective image 1100 is perceived as raised toward the observer relative to the parallel surface of the driving lane 1010, making it difficult to sense that it is parallel to the surface of the driving lane 1010, and there is room for improvement.

A summary of certain embodiments disclosed herein is provided below. It should be understood that these aspects are presented merely to provide the reader with an overview of these certain embodiments, and are not intended to limit the scope of the disclosure. Indeed, the disclosure may encompass a variety of aspects that are not set forth below.

The present disclosure relates to providing a display control device, a head-up display device, a display control method, and the like that display an image that is easily perceived as being parallel to the plane of the driving lane.

Therefore, the display control device, head-up display device, and display control method described in this specification employ the following means to solve the above problems. The gist of this embodiment is to execute a first display process that displays a perspective virtual image in a first display mode having a first vanishing point that is located below the vanishing point of the foreground as seen by the observer, and a second display process that displays a perspective virtual image in a second display mode having a second vanishing point that is located above the first vanishing point as seen by the observer.

Therefore, the display control device described in this specification is a display control device that executes display control in a head-up display device that projects an image onto a projection target member, thereby allowing the observer to view a virtual image of the image superimposed on the foreground, and includes one or more processors, a memory, and one or more computer programs stored in the memory and configured to be executed by the one or more processors, and the processor executes a first display process that displays a perspective virtual image of a first display mode having a first vanishing point that is located below the vanishing point of the foreground as seen by the observer, and a second display process that displays a perspective virtual image of a second display mode having a second vanishing point that is located above the first vanishing point as seen by the observer.

FIG. 1 is a diagram showing an example of application of a virtual image display system for a vehicle to a vehicle. FIG. 2 is a diagram showing the configuration of the head-up display device. FIG. 3 is a diagram conceptually showing a virtual object placed at a target position in the real world, and an image displayed in the virtual image display area so that the virtual object is visually recognized at the target position in the real world. FIG. 4 is a diagram showing an example of a foreground viewed by an observer while the host vehicle is traveling, and a virtual image of the first display mode displayed so as to be superimposed on the foreground. FIG. 5A is a diagram showing a modified example of a perspective image. FIG. 5B is a diagram showing a modified example of a perspective image. FIG. 5C is a diagram showing a modified example of a perspective image. FIG. 5D is a diagram showing a modified example of a perspective image. FIG. 6 is a diagram showing an example of the foreground viewed by the observer while the host vehicle is traveling, and a virtual image of the second display manner displayed so as to be superimposed on the foreground. FIG. 7A is a diagram showing a perspective virtual image V of a second display mode having a second vanishing point. FIG. 7B is a diagram showing a perspective virtual image V of a second display mode having a second vanishing point. FIG. 8 is a block diagram of a vehicle virtual image display system according to some embodiments. FIG. 9 is a flowchart showing a display control method for executing a display control process for displaying a perspective virtual image in the first display mode or the second display mode. FIG. 10 is a diagram relating to a conventional example, showing an example of a foreground viewed by an observer while the host vehicle is traveling, and a virtual image displayed superimposed on the foreground.

Below, in Figures 1 to 8, an explanation is provided of the configuration and operation of an exemplary vehicle virtual image display system. Note that the present invention is not limited to the following embodiments (including the contents of the drawings). Of course, modifications (including the deletion of components) can be made to the following embodiments. In addition, in the following explanation, explanations of well-known technical matters will be omitted as appropriate in order to facilitate understanding of the present invention.

Refer to FIG. 1. FIG. 1 is a diagram showing an example of application of a vehicle virtual image display system to a vehicle. In FIG. 1, the left-right direction (in other words, the width direction of the vehicle 1) of the vehicle (an example of a moving body, hereinafter also referred to as the subject vehicle) 1 is the X-axis (the positive direction of the X-axis is the left direction when the vehicle 1 faces forward), the up-down direction (in other words, the height direction of the vehicle 1) along a line segment that is perpendicular to the left-right direction and perpendicular to the ground or a surface equivalent to the ground (here, the road surface 6) is the Y-axis (the positive direction of the Y-axis is the upward direction), and the front-rear direction along a line segment that is perpendicular to each of the left-right direction and the up-down direction is the Z-axis (the positive direction of the Z-axis is the straight-ahead direction of the vehicle 1). This point is the same in other drawings.

The head-up display device (an example of a virtual image display device) 20 in the vehicle virtual image display system 10 is a head-up display (HUD: Head-Up Display) device provided in the dashboard 5 of the vehicle (an example of a moving body) 1. The HUD device 20 emits display light K toward the front windshield 2 (an example of a projection target), and the front windshield 2 reflects the display light K of the image displayed by the HUD device 20 toward the eye box 200. The HUD device 20 not only notifies the occupant of the vehicle 1 (the occupant is typically the driver of the vehicle 1) of information related to the vehicle 1 (hereinafter referred to as vehicle information) but also of information other than the vehicle information in an integrated manner. Note that the vehicle information includes not only information about the vehicle 1 itself but also information outside the vehicle 1 related to the operation of the vehicle 1.

As shown in the figure, the vehicle virtual image display system 10 provided in the vehicle (own vehicle) 1 has an eye position detection unit (gaze detection unit) 409 for pupil (or face) detection that detects the position and gaze direction of the left eye 700L and right eye 700R of the observer (typically the driver sitting in the driver's seat of the vehicle 1), an outside sensor 411 consisting of a camera (e.g., a stereo camera) that captures an image in front of the vehicle 1 (broadly speaking, the surroundings), a head-up display device (hereinafter also referred to as a HUD device) 20, and a display control device 30 that controls the HUD device 20.

In addition, the "eye box" used in the description of this embodiment is a region in which (1) the entire virtual image V of the image can be seen, and at least a part of the virtual image V of the image cannot be seen outside the region, (2) at least a part of the virtual image V of the image can be seen within the region, and at least a part of the virtual image V of the image cannot be seen outside the region, (3) at least a part of the virtual image V of the image can be seen with a predetermined luminance or more within the region, and at least a part of the virtual image V of the image is less than the predetermined luminance outside the region, or (4) when the HUD device 20 is capable of displaying a virtual image V that can be viewed stereoscopically, at least a part of the virtual image V can be viewed stereoscopically, and at least a part of the virtual image V cannot be viewed stereoscopically outside the region. In other words, when the observer places his/her eyes (both eyes) outside the eye box 200, the observer cannot view the entire virtual image V of the image, the visibility of the entire virtual image V of the image is very low and difficult to perceive, or the virtual image V of the image cannot be viewed stereoscopically. The predetermined brightness is, for example, about 1/50 of the brightness of the virtual image of the image viewed at the center of the eyebox. The "eyebox" is set to be the same as the area (also called the iris) where the observer's viewpoint is expected to be located in the vehicle in which the HUD device 20 is installed, or to include most of the iris (for example, 80% or more).

The virtual image display area VS is a flat, curved, or partially curved area where an image generated inside the HUD device 20 is formed as a virtual image V, and is also called an imaging surface. The virtual image display area VS is a position where a display surface (e.g., an exit surface of a liquid crystal light modulation element) 51 of the display 40 of the HUD device 20 described later is formed as a virtual image, that is, the virtual image display area VS corresponds to the display surface 51 of the HUD device 20 described later (in other words, the virtual image display area VS is in a conjugate relationship with the display surface 51 of the display 40 described later), and it can be said that the virtual image viewed in the virtual image display area VS corresponds to an image displayed on the display surface 51 of the HUD device 20 described later. It is preferable that the virtual image display area VS itself has low visibility to the extent that it is not actually visible to the observer's eyes or is difficult to see.

The virtual image display area VS is set so that the angle (tilt angle θt in Figure 1) between the horizontal direction (XZ plane) and the axis extending along the left-right direction (X-axis direction) of the vehicle 1, the angle between the line segment connecting the center 205 of the eye box 200 and the upper end VS1 of the virtual image display area VS, and the angle between the line segment connecting the center of the eye box and the lower end VS2 of the virtual image display area VS are set as the vertical angle of view of the virtual image display area VS, and the angle (vertical arrangement angle θd in Figure 1) between the bisector of this vertical angle of view and the horizontal direction (XZ plane). The vertical arrangement angle θd is typically set downward from the horizontal direction (XZ plane), for example, +2 degrees (here, the positive direction of the vertical arrangement angle θd is the depression angle where the bisector of the vertical angle of view points downward from the center 205 of the eye box 200, and the negative direction of the vertical arrangement angle θd is the elevation angle where the bisector of the vertical angle of view points upward from the center 205 of the eye box 200).

In addition, the virtual image display region VS of this embodiment has a tilt angle θt of approximately 90 degrees so that it faces approximately directly when facing forward (positive direction of the Z axis). However, the tilt angle θt is not limited to this and can be changed within the range of -10≦θt<90 degrees. The tilt angle θt may be set to, for example, 60 degrees, and the virtual image display region VS may be positioned so that the upper region is farther away than the lower region as seen by the viewer. However, the range of the tilt angle θt is not limited to this.

Figure 2 is a diagram showing one embodiment of the configuration of a head-up display device. The HUD device 20 is installed, for example, in a dashboard (reference numeral 5 in Figure 1). This HUD device 20 has a display 40, a relay optical system 80, and a housing 22 that houses the display 40 and the relay optical system 80 and has a light exit window 21 that can emit display light K from the display 40 from the inside to the outside.

The display 40 in FIG. 2 is typically mainly composed of a liquid crystal display panel (light modulation element) 50 and a light source unit 60. The display surface 51 is the surface on the viewing side of the liquid crystal display panel 50, and emits display light K of an image. The angle of the virtual image display area VS (including the tilt angle θt) can be set by setting the angle of the display surface 51 with respect to the optical axis Kp of the display light K that travels from the center of the display surface 51 through the relay optical system 80 and the projected portion toward the eye box 200 (center 205 of the eye box 200).

The liquid crystal display panel (light modulation element) 50 receives light from the light source unit 60 and emits the display light K, which is spatially light modulated into a 2D image, toward the relay optical system 80 (second mirror 82). The liquid crystal display panel 50 is, for example, rectangular in shape, with the short side being the direction in which the pixels corresponding to the vertical direction (Y-axis direction) of the virtual image V seen by the observer are arranged. The observer visually recognizes the transmitted light of the liquid crystal display panel 50 through the virtual image optical system 90. The virtual image optical system 90 is a combination of the relay optical system 80 and the front windshield 2 shown in FIG. 2. The light modulation element 50 (display 40) may be a self-luminous display or a projection type display (for example, a reflective display panel using LCoS or DMD) that projects an image onto a screen, but is not limited to these. In this case, the display surface 51 is the screen of the projection type display.

The light source unit 60 is composed of a light source (not shown) and an illumination optical system (not shown).

The light source (not shown) is, for example, a number of chip-type LEDs, and emits illumination light to the liquid crystal display panel (an example of a light modulation element) 22. The light source unit 60 is, for example, composed of four light sources, which are arranged in a row along the long side of the liquid crystal display panel 50. The light source unit 60 emits illumination light toward the liquid crystal display panel 50 under the control of the display control device 30. The configuration of the light source unit 60 and the arrangement of the light sources are not limited to this.

The illumination optical system (not shown) is composed of, for example, one or more lenses (not shown) arranged in the emission direction of the illumination light from the light source unit 60, and a diffusion plate (not shown) arranged in the emission direction of the one or more lenses.

The light source unit 60 may be configured to be capable of local dimming, and may change the degree of illumination for each area of the display surface 51 under the control of the display control device 30. This allows the light source unit 60 to adjust the brightness of the image displayed on the display surface 51 for each area.

The relay optical system 80 is arranged on the optical path of the display light K (light traveling from the display 40 toward the eye box 200) emitted from the display 40, and is composed of one or more optical components that project the display light K from the display 40 onto the front windshield 2 outside the HUD device 20. The relay optical system 80 in FIG. 2 includes one concave first mirror 81 and one flat second mirror 82.

The first mirror 81 is, for example, a free-form surface shape having positive optical power. In other words, the first mirror 81 may be a curved surface shape with different optical powers for each region, that is, the optical power added to the display light K may differ depending on the region (optical path) through which the display light K passes. Specifically, the optical power added by the relay optical system 80 may differ between the first display light K11, the second display light K12, and the third display light K13 (see FIG. 2) traveling from each region of the display surface 51 toward the eye box 200.

The second mirror 82 is, for example, a flat mirror, but is not limited thereto, and may be a curved surface having optical power. That is, the relay optical system 80 may combine multiple mirrors (for example, the first mirror 81 and the second mirror 82 in this embodiment) to add different optical power depending on the area (optical path) through which the display light K passes. The second mirror 82 may be omitted. That is, the display light K emitted from the display 40 may be reflected by the first mirror 81 to the projection target portion (front windshield) 2.

In addition, while the relay optical system 80 includes two mirrors (reflective optical system) in this embodiment, the present invention is not limited to this. The relay optical system 80 may include one or more refractive optical elements such as lenses, diffractive optical elements such as holograms, reflective optical elements different from those described above, or combinations of these, which may be added to the reflective optical system described above, or may replace part or all of the reflective optical system described above.

The relay optical system 80 has the function of setting the distance to the virtual image display area VS and the function of generating a virtual image that is an enlarged image of the image displayed on the display surface 51 by using this curved shape (an example of optical power), but in addition to this, it may also have the function of suppressing (correcting) distortion of the virtual image that may occur due to the curved shape of the front windshield 2, or other known optical functions.

The relay optical system 80 may also be rotatable by being fitted with one or more actuators 86, 87 controlled by the display control device 30. For example, the actuator 86 may rotate (and/or move) the first mirror 81 about the first rotation axis AX1. The actuator 87 may rotate (and/or move) the second mirror 82 about the first rotation axis AX1.

The head-up display device 20, based on the control of the display control device 30 described later, displays images in the vicinity of objects such as the road surface of the driving lane, a fork in the road, a road sign, an obstacle (pedestrian, bicycle, motorcycle, other vehicle, etc.), and a feature (building, bridge, etc.) that exist in the foreground, which is the real space (real scene) visually recognized through the front windshield 2 of the vehicle 1, a position overlapping the object, or a position set based on the object, thereby allowing an observer (typically an observer seated in the driver's seat of the vehicle 1) to perceive visual augmented reality (AR). In the description of this embodiment, an image whose display position can be changed according to a real object existing in the real scene or a depth or a target position in the up, down, left, and right directions in a predetermined space in the real scene determined by the display control device 30 is defined as an AR image, and an image whose display position is set regardless of the position of the real object is defined as a non-AR image. In addition, in the description of this embodiment, a perspective image is defined as an image consisting of one or more images expressed using a perspective projection method that creates a sense of depth, and an image that is not expressed using a perspective projection method is defined as a non-perspective image. A perspective image is typically an AR image.

The display control device 30, which will be described later, can control the appearance of the image V displayed by the HUD device 20 (perceived by the observer) by, for example, executing image rendering processing (graphics processing), display drive processing, etc.

Figure 3 is a conceptual diagram showing a virtual object placed at a target position in the real scene, and an image displayed in the virtual image display area so that the virtual object is visually recognized at the target position in the real scene. As shown in Figure 3, the depth direction as viewed from the viewer 700 is the Z-axis direction, the left-right direction (width direction of the host vehicle 1) is the X-axis direction, and the up-down direction (up-down direction of the host vehicle 1) is the Y-axis direction. Note that the direction moving away from the viewer is the positive direction of the Z-axis, the left direction as viewed from the viewer is the positive direction of the X-axis, and the upward direction as viewed from the viewer is the positive direction of the Y-axis.

The viewer 700 perceives a virtual object FU at a predetermined target position PT in the real scene by viewing the virtual image V formed (focused) in the virtual image display area VS via the projection unit 2. The viewer views the virtual image V of the image of the display light K reflected by the projection unit 2. At this time, if the virtual image V is, for example, an arrow indicating the course, the arrow of the virtual image V is displayed in the virtual image display area VS so that the virtual object FU is positioned and viewed at the predetermined target position PT in the foreground of the vehicle 1.

(Perspective virtual image V10 of the first display mode)
4 is a diagram showing an example of a foreground viewed by an observer while the vehicle 1 is traveling, and a virtual image of a first display mode displayed superimposed on the foreground. A virtual image V10 of the first display mode (hereinafter also referred to as a perspective virtual image V10) has a vanishing point VP1 (first vanishing point VP1). That is, the figure of the perspective virtual image V10 has a shape with a vanishing point VP1 that gives a sense of depth, and for example, an extension line V1A of the left side of the stem of the "arrow" and an extension line V1B of the right side intersect at the vanishing point VP1.

As shown in the figure, when the perspective virtual image V10 is positioned below the viewer (at a position where the vertical arrangement angle θd is in a positive direction), the vanishing point VP1 is positioned above the position of the virtual image V10. On the other hand, when the perspective virtual image V10 is positioned above the viewer (at a position where the vertical arrangement angle θd is in a negative direction), the vanishing point VP1 is positioned below the position of the virtual image.

The viewer gets a sense of depth from the perspective virtual image V10 as seen by the viewer based on the relationship between the positions of the vanishing point VP1 and the perspective virtual image V10. In this way, the perspective virtual image V10 has a vanishing point VP1, making it easier for the viewer to estimate the depth position of the perspective virtual image V10.

On the other hand, as shown in FIG. 4, the driving lane 6 of the vehicle 1 has a vanishing point VP0 (actual scene vanishing point VP0). In this specific example, there is a driving lane 6 going straight ahead as seen by the viewer, and the extensions of the boundaries 7 and 8 on both sides of the driving lane 6 essentially intersect (become a point) at the vanishing point VP0. In this way, because the driving lane (road surface) 6 has a vanishing point VP0, the viewer gets a sense of depth in the actual scene (driving lane 6).

In the vehicle virtual image display system 10 according to this embodiment, as shown in FIG. 4, the vanishing point VP1 of the perspective virtual image V10 is located in front of (below when viewed from the observer) the vanishing point VP0 of the actual scene (the driving lane 6).

In general, when creating images, including paintings, using linear perspective (perspective projection), vanishing points are used to position various objects in the image at their respective depth positions. For example, lines are virtually drawn radially from a specified vanishing point, and the contours of each object are aligned with these lines, so that each object is perceived as being fixed at a specified depth position. Note that multiple vanishing points can be set, but for simplicity's sake, the following explanation will be given assuming that there is one vanishing point in one image.

When using general linear perspective (perspective projection), the virtual image is displayed so that the extension lines of each boundary line forming the contour of the virtual object perceived by the observer intersect at the vanishing point VP0 in the foreground (driving lane 6). In other words, the virtual image is generated so that the position of the vanishing point VP1 of the virtual image coincides with the position of the vanishing point VP0 in the foreground (driving lane 6). In this embodiment, however, the perspective virtual image V10 is generated so that the position of the vanishing point VP1 of the perspective virtual image V10 is closer to the viewer than the position of the vanishing point VP0 in the foreground (driving lane 6) (first display process).

This makes it easier for the observer to sense that the perceived placement angle of the virtual object FU10 (the angle with respect to the road surface with the X2 axis in FIG. 3 as the axis of rotation) is roughly parallel to the driving lane 6.

The virtual object FU10 is an "arrow", and the target position PT at which the virtual object FU10 is placed (set) is usually set to match the surface height of the foreground (driving lane 6) (i.e., the set height is set to 0 m). However, this is just one example and is not limiting. In another example, the target position PT may be set to a position higher than the surface height l of the foreground (driving lane 6) (i.e., the set height may be set to 0.5 m or 1 m). In another example, the target position PT may be set to a position lower than the surface height of the foreground (driving lane 6) (i.e., the set height may be set to -1 m or -2 m).

5A, 5B, 5C, and 5D are diagrams showing modified examples of perspective images. The perspective virtual image V10 in the above embodiment is composed of a single straight "arrow", but is not limited to this. The perspective virtual image V20 shown in FIG. 5A may be bent halfway through the image ("arrow"). In this case, the left perspective line V1A can be defined as an extension of the left side of the perspective virtual image V20 until it bends, and the right perspective line V1B can be defined as an extension of the right side of the perspective virtual image V20 until it bends.

The perspective virtual image V30 shown in FIG. 5B is composed of a first perspective virtual image V31 and a second perspective virtual image V32 that is disposed above the first perspective virtual image V31 and is set to be smaller in size than the first perspective virtual image V31. In this case, the left perspective line V1A can be defined as a straight line connecting the left end of the first perspective V31 and the left end of the second perspective V32, and the right perspective line V1B can be defined as a straight line connecting the right end of the first perspective V31 and the right end of the second perspective V32.

The perspective virtual image V40 shown in FIG. 5C may be composed of multiple perspective virtual images V41 to V49 divided into left and right. In this case, the left perspective line V1A can be defined as a straight line (approximate straight line) connecting the left ends of the left perspectives V41 to V44, and the right perspective line V1B can be defined as a straight line (approximate straight line) connecting the right ends of the right perspectives V45 to V48.

The vanishing point VP1 (first vanishing point VP1) of the perspective virtual image V50 shown in FIG. 5D may be set at a position directed toward vanishing point VP0' that is shifted to the left or right of vanishing point VP0, which is the intersection of boundary lines 7, 8 of the travel lane 6, as seen by the observer (vanishing point VP0' shown in FIG. 5D is positioned to the left of vanishing point VP0). Even in this case, vanishing point VP1 (first vanishing point VP1) of the perspective virtual image V50 can be positioned closer to the viewer than vanishing point VP0' (real scene vanishing point VP0).

(Perspective virtual image V60 of the second display mode)
6 is a diagram showing an example of a foreground viewed by an observer while the vehicle 1 is traveling, and a virtual image of a second display mode displayed superimposed on the foreground. A virtual image V60 of the second display mode (hereinafter also referred to as a perspective virtual image V60) has a vanishing point VP2 (second vanishing point VP2) on the farther side than the first vanishing point VP1 of the perspective virtual image V10 of the first display mode. That is, the figure of the perspective virtual image V60 has a shape having a second vanishing point VP2 that gives a sense of depth, and for example, an extension line V1B of the left side of the stem of the "arrow" and an extension line V2B of the right side intersect at the second vanishing point VP2.

As shown in the figure, when the perspective virtual image V60 of the second display mode is positioned downward (at a position where the vertical arrangement angle θd is in a positive direction) as seen by the viewer, the second vanishing point VP2 is positioned above the position of the first vanishing point VP1. On the other hand, when the perspective virtual image V60 of the second display mode is positioned upward (at a position where the vertical arrangement angle θd is in a negative direction) as seen by the viewer, the second vanishing point VP2 is positioned below the position of the first vanishing point VP1.

The HUD device 20 (display control device 30) of some embodiments changes the display mode of the perspective virtual image V to the above-mentioned first display mode having the first vanishing point VP1 or the above-mentioned second display mode having the second vanishing point VP2 set on the far side of the first vanishing point VP1, depending on the type of the perspective virtual image V. The HUD device 20 (display control device 30) of some embodiments (10) displays the perspective virtual image V in the second display mode having the second vanishing point VP2 (an example of a second display process) if the perspective virtual image V is a moving image that moves or expands in the perspective direction, and (15) displays the perspective virtual image V in the first display mode having the first vanishing point VP1 if the perspective virtual image V is a still image, a moving image that does not move much in the perspective direction, or a moving image that does not expand much in the perspective direction.

7A and 7B are diagrams showing a perspective virtual image V in a second display mode having a second vanishing point VP2. In some embodiments, the HUD device 20 (display control device 30) displays the perspective virtual image V in a second display mode having a second vanishing point VP2 when the perspective virtual image V is (11) a moving image that is perceived as gradually expanding and contracting in the perspective direction (in the example of FIG. 7A, a moving image expressed in a perspective projection that gradually extends from the near side toward the second vanishing point VP2 on the far side), or (12) a moving image that is perceived as gradually moving in the perspective direction (in the example of FIG. 7B, a moving image expressed in a perspective projection that gradually moves from the near side toward the second vanishing point VP2 on the far side) (an example of a second display process). On the other hand, when the perspective virtual image V is (16) a still image, (17) a moving image that is not perceived as expanding or contracting much in the perspective direction, or (18) a moving image that is not perceived as moving much in the perspective direction, the HUD device 20 (display control device 30) displays it in a first display mode having a first vanishing point VP1 (an example of a first display process). "(17) A moving image that does not expand or contract much in the perspective direction, or (18) A moving image that does not move much in the perspective direction" specifically includes, for example, a moving image that moves in the left-right direction (X-axis direction), a moving image that moves in the up-down direction (Y-axis direction) by a minute distance less than a predetermined threshold in a direction that can also be perceived by humans as the depth direction (Z-axis direction) (more specifically, a moving image in which the distance in the up-down direction (Y-axis direction) is 1/10 or less of the distance in the left-right direction (X-axis direction)), etc.

In addition, the HUD device 20 (display control device 30) of some embodiments changes the display mode of the perspective virtual image V to the above-mentioned first display mode having the first vanishing point VP1 or the above-mentioned second display mode having the second vanishing point VP2 set on the far side of the first vanishing point VP1, depending on the moving speed (deformation speed) of the moving image of the perspective virtual image V. For example, the HUD device 20 (display control device 30) of some embodiments (20) displays the perspective virtual image V in the second display mode having the second vanishing point VP2 if the moving speed (deformation speed) of the moving image is fast (an example of a second display process), and (25) displays the perspective virtual image V in the first display mode having the first vanishing point VP1 if the moving speed of the moving image is slow (an example of a first display process).

In the above embodiment, the HUD device 20 (display control device 30) displays the perspective virtual image V in a second display mode having a second vanishing point VP2 when the speed at which the image (21) gradually expands and contracts in the perspective direction or gradually moves in the perspective direction is equal to or greater than a predetermined image speed threshold and is determined to be fast (an example of a second display process). On the other hand, the HUD device 20 (display control device 30) displays the perspective virtual image V in a first display mode having a first vanishing point VP1 when the speed at which the image (26) gradually expands and contracts in the perspective direction or gradually moves in the perspective direction is less than a predetermined image speed threshold and is determined to be slow (an example of a first display process).

In addition, the HUD device 20 (display control device 30) of some embodiments changes the display mode of the perspective virtual image V to the first display mode having the first vanishing point VP1 or the second display mode having the second vanishing point VP2 set farther away than the first vanishing point VP1 depending on the brightness of the area ahead (surroundings) of the vehicle 1 (an example of the second display process). For example, the HUD device 20 (display control device 30) of some embodiments (30) displays the image in the second display mode having the second vanishing point VP2 when it is determined that the area ahead (surroundings) of the vehicle 1 is bright, and (35) displays the image in the first display mode having the first vanishing point VP1 when it is determined that the area ahead (surroundings) of the vehicle 1 is dark (an example of the first display process).

In addition, in some embodiments, the HUD device 20 (display control device 30) changes the display mode of the perspective virtual image V to the above-mentioned first display mode having a first vanishing point VP1, or the above-mentioned second display mode having a second vanishing point VP2 set farther away than the first vanishing point VP1, depending on the length of the perspective virtual image V in the up-down direction (Y-axis direction), which is the depth direction (Z-axis direction) as perceived by humans. For example, in some embodiments, the HUD device 20 (display control device 30) displays (40) the perspective virtual image V in a second display mode having a second vanishing point VP2 when the length of the perspective virtual image V in the up-down direction (Y-axis direction), which is perceived by humans as a distance in the depth direction (Z-axis direction), is equal to or longer than a predetermined threshold (an example of a second display process), and (45) displays the perspective virtual image V in a first display mode having a first vanishing point VP1 when the length of the perspective virtual image V in the up-down direction (Y-axis direction), which is perceived by humans as a distance in the depth direction (Z-axis direction), is shorter than a predetermined threshold (an example of a first display process).

In addition, the HUD device 20 (display control device 30) of some embodiments changes the display mode of the perspective virtual image V to the first display mode having the first vanishing point VP1 or the second display mode having the second vanishing point VP2 set on the far side of the first vanishing point VP1, depending on the length of the period during which the display continues. For example, the HUD device 20 (display control device 30) of some embodiments (50) displays the perspective virtual image V of a moving image or still image in the second display mode having the second vanishing point VP2 when the period from the start to the end of the display is equal to or longer than a predetermined threshold value (an example of a second display process), and (55) displays the perspective virtual image V of a moving image or still image in the first display mode having the first vanishing point VP1 when the period from the start to the end of the display is shorter than a predetermined threshold value (an example of a first display process).

Figure 8 is a block diagram of a vehicle virtual image display system according to some embodiments. The display controller 30 includes one or more I/O interfaces 31, one or more processors 33, one or more image processing circuits 35, and one or more memories 37. The various functional blocks described in Figure 8 may be configured in hardware, software, or a combination of both. Figure 8 is only one embodiment, and the components shown may be combined into fewer components, or there may be additional components. For example, the image processing circuitry 35 (e.g., a graphics processing unit) may be included in one or more processors 33.

As shown, the processor 33 and the image processing circuit 35 are operably coupled to the memory 37. More specifically, the processor 33 and the image processing circuit 35 can execute a program stored in the memory 37 to control the vehicle virtual image display system 10 (display 40), for example, to generate and/or transmit image data. The processor 33 and/or the image processing circuit 35 can include at least one general-purpose microprocessor (e.g., a central processing unit (CPU)), at least one application specific integrated circuit (ASIC), at least one field programmable gate array (FPGA), or any combination thereof. The memory 37 can include any type of magnetic medium, such as a hard disk, any type of optical medium, such as a CD and a DVD, any type of semiconductor memory, such as a volatile memory, and a non-volatile memory. Volatile memory can include DRAM and SRAM, and non-volatile memory can include ROM and NVRAM.

As shown in the figure, the processor 33 is operably connected to the I/O interface 31. The I/O interface 31 communicates (also referred to as CAN communication) with, for example, a vehicle ECU 401 (described later) and/or other electronic devices (reference numerals 403 to 419 described later) provided in the vehicle according to the CAN (Controller Area Network) standard. Note that the communication standard adopted by the I/O interface 31 is not limited to CAN, and includes, for example, wired communication interfaces such as CANFD (CAN with Flexible Data Rate), LIN (Local Interconnect Network), Ethernet (registered trademark), MOST (Media Oriented Systems Transport: MOST is a registered trademark), UART, or USB, or in-vehicle communication (internal communication) interfaces that are short-range wireless communication interfaces within several tens of meters, such as personal area networks (PANs) such as Bluetooth (registered trademark) networks and local area networks (LANs) such as 802.11x Wi-Fi (registered trademark) networks. The I/O interface 31 may also include an external communication (external communication) interface such as a wide area communication network (e.g., an Internet communication network) based on a wireless wide area network (WWAN0, IEEE 802.16-2004 (WiMAX: Worldwide Interoperability for Microwave Access)), IEEE 802.16e-based (Mobile WiMAX), 4G, 4G-LTE, LTE Advanced, 5G, or other cellular communication standards.

As shown in the figure, the processor 33 is interoperably connected to the I/O interface 31, thereby enabling information to be exchanged with various other electronic devices connected to the vehicle virtual image display system 10 (I/O interface 31). The I/O interface 31 is operably connected to, for example, a vehicle ECU 401, a road information database 403, a vehicle position detection unit 405, an operation detection unit 407, an eye position detection unit 409, an outside vehicle sensor 411, a brightness detection unit 413, an IMU 415, a mobile information terminal 417, and an external communication device 419. The I/O interface 31 may include a function for processing (converting, calculating, analyzing) information received from other electronic devices connected to the vehicle virtual image display system 10.

The display 40 is operatively coupled to the processor 33 and the image processing circuitry 35. Thus, the image displayed by the light modulation element 50 may be based on image data received from the processor 33 and/or the image processing circuitry 35. The processor 33 and the image processing circuitry 35 control the image displayed by the light modulation element 50 based on information obtained from the I/O interface 31.

The vehicle ECU 401 obtains the state of the vehicle 1 (e.g., mileage, vehicle speed, accelerator pedal opening, brake pedal opening, engine throttle opening, injector fuel injection amount, engine RPM, motor RPM, steering angle, shift position, drive mode, various warning states, posture (including roll angle and/or pitch angle), vehicle vibration (including vibration magnitude, frequency, and/or frequency)) from sensors and switches provided on the vehicle 1, and collects and manages (which may also include control) said state of the vehicle 1, and as part of its function, can output a signal indicating a numerical value of said state of the vehicle 1 (e.g., the vehicle speed of the vehicle 1) to the processor 33 of the display control device 30. In addition to or instead of simply transmitting a value detected by a sensor or the like (e.g., the pitching angle is 3 degrees in the forward tilt direction) to the processor 33, the vehicle ECU 401 may transmit to the processor 33 a determination result based on one or more states of the vehicle 1 including the value detected by the sensor (e.g., that the vehicle 1 satisfies a predetermined condition for a forward tilt state) and/or an analysis result (e.g., that the vehicle has been tilted forward by braking in combination with information on the brake pedal opening degree). For example, the vehicle ECU 401 may output to the display control device 30 a signal indicating a determination result that the vehicle 1 satisfies a predetermined condition previously stored in a memory (not shown) of the vehicle ECU 401. The I/O interface 31 may acquire the above-mentioned information from a sensor or switch provided in the vehicle 1 without going through the vehicle ECU 401.

The vehicle ECU 401 may also output an instruction signal to the display control device 30 to indicate the image to be displayed by the vehicle virtual image display system 10, and in this case, the coordinates, size, type, display mode, notification necessity of the image, and/or necessity-related information that is the basis for determining the notification necessity of the image may be added to the instruction signal and transmitted.

The road information database 403 is included in a navigation device (not shown) provided in the vehicle 1, or an external server connected to the vehicle 1 via an external communication interface (I/O interface 31), and may read out information about the surroundings of the vehicle 1 (information related to real objects around the vehicle 1), such as road information (lanes, white lines, stop lines, crosswalks, road widths, number of lanes, intersections, curves, forks, traffic regulations, etc.) on which the vehicle 1 is traveling, and information about features (buildings, bridges, rivers, etc.), including their presence or absence, position (including distance to the vehicle 1), direction, shape, type, detailed information, etc., based on the position of the vehicle 1 acquired from the vehicle position detection unit 405 described later, and transmit this information to the processor 33. The road information database 403 may also calculate an appropriate route (navigation information) from the departure point to the destination, and output a signal indicating the navigation information or image data indicating the route to the processor 33.

The vehicle position detection unit 405 is a GNSS (Global Navigation Satellite System) or the like provided in the vehicle 1, which detects the current position and orientation of the vehicle 1 and outputs a signal indicating the detection result to the road information database 403, the mobile information terminal 417 described later, and/or the external communication device 419 via the processor 33 or directly. The road information database 403, the mobile information terminal 417 described later, and/or the external communication device 419 may select/generate information about the surroundings of the vehicle 1 by acquiring the position information of the vehicle 1 from the vehicle position detection unit 405 continuously, intermittently, or for each predetermined event, and output the information to the processor 33.

The operation detection unit 407 is, for example, a hardware switch provided on the CID (Center Information Display) or instrument panel of the vehicle 1, or a software switch that combines an image and a touch sensor, and outputs operation information based on an operation by an occupant of the vehicle 1 (a user sitting in the driver's seat and/or a user sitting in the passenger seat) to the processor 33. For example, the operation detection unit 407 outputs to the processor 33 display area setting information based on an operation to move the virtual image display area 100 by a user's operation, eye box setting information based on an operation to move the eye box 200, information based on an operation to set the observer's eye position 700, and the like.

The eye position detection unit 409 may include a camera such as an infrared camera that detects the eye position 700 (see FIG. 1) of an observer sitting in the driver's seat of the vehicle 1, and may output the captured image to the processor 33. The processor 33 may obtain a captured image (one example of information that can estimate the eye position 700) from the eye position detection unit 409, and may analyze the captured image using a method such as pattern matching to detect the coordinates of the observer's eye position 700, and output a signal indicating the coordinates of the detected eye position 700 to the processor 33.

The eye position detection unit 409 may also output the analysis results of the image captured by the camera (e.g., a signal indicating to which spatial region the observer's eye position 700 belongs in which a plurality of pre-set display parameters correspond) to the processor 33. Note that the method of acquiring the observer's eye position 700 in the vehicle 1 or information that can estimate the observer's eye position 700 is not limited to these, and may be acquired using known eye position detection (estimation) techniques.

The eye position detection unit 409 may also detect the speed and/or direction of movement of the observer's eye position 700 and output a signal indicating the speed and/or direction of movement of the observer's eye position 700 to the processor 33.

In addition, when the eye position detection unit 409 detects (10) a signal indicating that the observer's eye position 700 is outside the eye box 200, (20) a signal that estimates that the observer's eye position 700 is outside the eye box 200, or (30) a signal that predicts that the observer's eye position 700 will be outside the eye box 200, it may determine that a predetermined condition is met and output a signal indicating that state to the processor 33.

(20) Signals indicating that the observer's eye position 700 is estimated to be outside the eye box 200 include (21) a signal indicating that the observer's eye position 700 cannot be detected, (22) a signal indicating that the observer's eye position 700 cannot be detected after movement of the observer's eye position 700 has been detected, and/or (23) a signal indicating that either the observer's eye position 700R, 700L is in the vicinity of the boundary 200A of the eye box 200 (the vicinity includes, for example, within a predetermined coordinate range from the boundary 200A).

(30) Signals predicting that the observer's eye position 700 will be outside the eye box 200 include (31) a signal indicating that the newly detected eye position 700 is equal to or greater than the eye position movement distance threshold pre-stored in memory 37 relative to the previously detected eye position 700 (the movement of the eye position within a specified unit time is greater than a specified range), and (32) a signal indicating that the movement speed of the eye position is equal to or greater than the eye position movement speed threshold pre-stored in memory 37.

The eye position detection unit 409 may also have a function as a gaze direction detection unit 409. The gaze direction detection unit 409 may include an infrared camera or a visible light camera that captures an image of the face of an observer sitting in the driver's seat of the vehicle 1, and output the captured image to the processor 33. The processor 33 can obtain an image (an example of information that can estimate the gaze direction) from the gaze direction detection unit 409 and identify the observer's gaze direction (and/or the gaze position) by analyzing the captured image. The gaze direction detection unit 409 may analyze the captured image from the camera and output a signal indicating the observer's gaze direction (and/or the gaze position) as the analysis result to the processor 33. Note that the method of acquiring information that can estimate the gaze direction of an observer of vehicle 1 is not limited to these, and may be acquired using other known gaze direction detection (estimation) techniques, such as the EOG (Electro-oculogram) method, the corneal reflex method, the scleral reflex method, the Purkinje image detection method, the search coil method, and the infrared fundus camera method.

The external sensor 411 detects real objects present around the vehicle 1 (in front, to the sides, and behind). The real objects detected by the external sensor 411 may include, for example, obstacles (pedestrians, bicycles, motorcycles, other vehicles, etc.), the road surface of the driving lane described below, dividing lines, roadside objects, and/or features (buildings, etc.). The external sensor is composed of, for example, a detection unit consisting of a radar sensor such as a millimeter wave radar, ultrasonic radar, or laser radar, a camera, or a combination of these, and a processing device that processes (data fuses) the detection data from the one or more detection units. Conventional well-known methods are applied to object detection by these radar sensors and camera sensors. By object detection by these sensors, the presence or absence of a real object in a three-dimensional space, and if a real object exists, the position of the real object (relative distance from the vehicle 1, position in the left-right direction when the traveling direction of the vehicle 1 is the forward-rear direction, position in the up-down direction, etc.), size (size in the lateral direction (left-right direction), height direction (up-down direction), etc.), movement direction (lateral direction (left-right direction), depth direction (front-rear direction)), movement speed (lateral direction (left-right direction), depth direction (front-rear direction)), and/or type, etc. may be detected. One or more external sensors 411 can detect a real object in front of the vehicle 1 at each detection period of each sensor, and output real object information (presence or absence of a real object, and if a real object exists, information on the position, size, and/or type of each real object, etc.) which is an example of real object information, to the processor 33. Note that the real object information may be transmitted to the processor 33 via another device (for example, the vehicle ECU 401). In addition, when a camera is used as a sensor, an infrared camera or a near-infrared camera is preferable so that a real object can be detected even when the surroundings are dark, such as at night. Also, if a camera is used as a sensor, a stereo camera that can obtain distance and other information using parallax is desirable.

The brightness detection unit 413 detects the illuminance or brightness of a predetermined range of the foreground in front of the cabin of the vehicle 1 as the external brightness (an example of brightness information), or the illuminance or brightness inside the cabin as the interior brightness (an example of brightness information). The brightness detection unit 413 is, for example, a phototransistor or a photodiode, and is mounted on the instrument panel, rearview mirror, or HUD device 20 of the vehicle 1 shown in FIG. 1.

The IMU 415 may include a combination of one or more sensors (e.g., an accelerometer and a gyroscope) configured to detect the position, orientation, and changes therein (speed of change, acceleration of change) of the vehicle 1 based on inertial acceleration. The IMU 415 may output the detected values (the detected values include signals indicating the position, orientation, and changes therein (speed of change, acceleration of change) of the vehicle 1) and the results of analyzing the detected values to the processor 33. The results of the analysis may be a signal indicating a determination result as to whether or not the detected values satisfy a predetermined condition, and may be, for example, a signal indicating that the behavior (vibration) of the vehicle 1 is small, based on a value regarding the change (speed of change, acceleration of change) of the position or orientation of the vehicle 1.

The mobile information terminal 417 is a smartphone, a laptop computer, a smartwatch, or other information device that can be carried by the observer (or other occupants of the vehicle 1). The I/O interface 31 is capable of communicating with the mobile information terminal 417 by pairing with the mobile information terminal 417, and acquires data recorded in the mobile information terminal 417 (or a server via the mobile information terminal). The mobile information terminal 417 has, for example, the same functions as the above-mentioned road information database 403 and vehicle position detection unit 405, and may acquire the road information (an example of real object related information) and transmit it to the processor 33. The mobile information terminal 417 may also acquire commercial information (an example of real object related information) related to commercial facilities in the vicinity of the vehicle 1 and transmit it to the processor 33. The mobile information terminal 417 may transmit schedule information of the user of the mobile information terminal 417 (e.g., the observer), information on incoming calls to the mobile information terminal 417, information on received e-mails, etc. to the processor 33, and the processor 33 and the image processing circuit 35 may generate and/or transmit image data relating to these.

The external communication device 419 is a communication device that exchanges information with the vehicle 1, and is, for example, another vehicle connected to the vehicle 1 by vehicle-to-vehicle communication (V2V: Vehicle To Vehicle), a pedestrian (a mobile information terminal carried by a pedestrian) connected to the vehicle 1 by pedestrian-to-pedestrian communication (V2P: Vehicle To Pedestrian), and a network communication device connected by road-to-vehicle communication (V2I: Vehicle To Roadside Infrastructure), and in a broad sense, includes everything connected to the vehicle 1 by communication (V2X: Vehicle To Everything). The external communication device 419 may acquire the positions of, for example, pedestrians, bicycles, motorcycles, other vehicles (such as a preceding vehicle), road surfaces, dividing lines, roadside objects, and/or features (such as buildings) and output them to the processor 33. In addition, the external communication device 419 may have the same function as the vehicle position detection unit 405 described above, and may acquire position information of the vehicle 1 and transmit it to the processor 33, and may also have the function of the road information database 403 described above, and may acquire the road information (an example of real object related information) and transmit it to the processor 33. Note that the information acquired from the external communication device 419 is not limited to the above.

The software components stored in memory 37 include a display parameter setting module 502, a graphics module 504, a light source driving module 506, and an actuator driving module 508.

FIG. 9 is a flow diagram showing a method S100 for executing a display control process for displaying a perspective virtual image in a first display mode or a second display mode. The method S100 is executed in a HUD device 20 including a display and a display control device 30 that controls the HUD device 20. Some operations in the method S100 are optionally combined, the procedures of some operations are optionally changed, and some operations are optionally omitted.

The display control device 30 (graphics module 504, described later) acquires, for example, an appropriate route from the departure point to the destination (navigation information) from the road information database 403, real object information from the external sensor 411 (presence or absence of real objects, and if real objects exist, information such as the position, size, and/or type of each real object), and selects and sets the virtual object to be displayed (S110).

As shown in S131, the display parameter setting module 502 changes the display mode of the perspective virtual image V based on one or more pieces of information acquired from the I/O interface 31. Based on one or more pieces of information acquired from the I/O interface 31, the display parameter setting module 502 (10) sets the perspective virtual image V to a second display mode having a second vanishing point VP2 when the perspective virtual image V is set to a moving image that moves or expands in the perspective direction (an example of a second display process), and (15) sets the perspective virtual image V to a first display mode having a first vanishing point VP1 when the perspective virtual image V is set to a still image, a moving image that does not move much in the perspective direction, or a moving image that does not expand much in the perspective direction (an example of a first display process).

Furthermore, as shown in S132, the display parameter setting module 502, based on one or more pieces of information acquired from the I/O interface 31, (20) sets the moving speed of the moving image of the perspective virtual image V to a second display mode having a second vanishing point VP2 (an example of a second display process) when the moving speed of the moving image of the perspective virtual image V is set to be faster than a predetermined threshold, and (25) sets the moving speed of the moving image of the perspective virtual image V to a first display mode having a first vanishing point VP1 (an example of a first display process).

Furthermore, as shown in S133, based on information (but not limited to this) obtained from the brightness detection unit 413, the display parameter setting module 502 (30) sets the perspective virtual image V to a second display mode having a second vanishing point VP2 if it is determined that the area ahead (surroundings) of the vehicle 1 is bright (an example of a second display process), and (35) sets the perspective virtual image V to a first display mode having a first vanishing point VP1 if it is determined that the area ahead (surroundings) of the vehicle 1 is dark (an example of a first display process).

Also, as shown in S134, based on the road information acquired from the road information database 403 and the position information (but not limited to) of the real object in the foreground acquired from the outside sensor 411, the display parameter setting module 502 sets the perspective virtual image V to a second display mode having a second vanishing point VP2 if the vertical length (Y axis direction) of the perspective virtual image V indicating the guide route or the real object in the foreground, which is the distance in the depth direction (Z axis direction) as perceived by humans, is equal to or longer than a predetermined threshold value pre-stored in the memory 37 (an example of a second display process), and sets the perspective virtual image V to a first display mode having a first vanishing point VP1 if the vertical length (Y axis direction) of the perspective virtual image V, which is the distance in the depth direction (Z axis direction) as perceived by humans, is shorter than a predetermined threshold value (an example of a first display process).

Furthermore, as shown in S135, based on one or more pieces of information acquired from the I/O interface 31, the display parameter setting module 502 sets the perspective virtual image V to a second display mode having a second vanishing point VP2 (an example of a second display process) if the period from the start to the end of displaying the perspective virtual image V of a moving image or still image is equal to or longer than a predetermined threshold value previously stored in the memory 37, and sets the perspective virtual image V to a first display mode having a first vanishing point VP1 (an example of a first display process) if the period from the start to the end of displaying the perspective virtual image V of a moving image or still image is shorter than a predetermined threshold value previously stored in the memory 37.

The graphics module 504 includes various known software components for generating image data by performing image processing such as rendering, and driving the display 40. The graphics module 504 may also include various known software components for changing the type (moving image, still image, shape), arrangement (position coordinates, angle), size, display distance (in the case of 3D), and visual effects (e.g., brightness, transparency, saturation, contrast, or other visual characteristics) of the image to be displayed. The graphics module 504 generates image data so that the type of image (one of the examples of display parameters), the position coordinates of the image (one of the examples of display parameters), the angle of the image (one of the examples of display parameters such as pitching angle with the X-direction as the axis, yaw rate angle with the Y-direction as the axis, and rolling angle with the Z-direction as the axis, etc.), the size of the image (one of the examples of display parameters), the color of the image (one of the examples of display parameters set by hue, saturation, brightness, etc.), and the intensity of the perspective representation of the image (one of the display parameters set by the position of the vanishing point, etc.) are visually recognized by the observer, and can drive the light modulation element 50.

The light source driving module 506 includes various known software components for executing the driving of the light source unit 24. The light source driving module 506 may drive the light source unit 24 based on the set display parameters.

The actuator drive module 508 includes various known software components for executing the operation of driving the first actuator 28 and/or the second actuator 29. The actuator drive module 508 can drive the first actuator 28 and the second actuator 29 based on the set display parameters.

The eye position detection module 510 detects the observer's eye position 700. The eye position detection module 510 includes various software components for performing various operations related to detecting coordinates indicating the observer's eye position 700 (positions in the X and Y axes, which are an example of a signal indicating the eye position 700), coordinates indicating the observer's eye height (position in the Y axis direction, which is an example of a signal indicating the eye position 700), detecting coordinates indicating the observer's eye height and depth position (positions in the Y and Z axes, which are an example of a signal indicating the eye position 700), and/or detecting coordinates indicating the observer's eye position 700 (positions in the X, Y, and Z axes, which are an example of a signal indicating the eye position 700).

The positions of the first vanishing point VP1 and the second vanishing point VP2 may be adjusted by the eye position 700 of the observer detected by the eye position detection unit 409, or the attitude of the vehicle 1 detected by the IMU 415. For example, the display control device 30 (display parameter setting module 502) may adjust the positions of the first vanishing point VP1 and the second vanishing point VP2 in response to a change in the attitude (pitching angle (angle around the X-axis)) of the vehicle 1. When the pitching angle is such that the front of the vehicle 1 is lowered, the display control device 30 (display parameter setting module 502) adjusts the relative positions of the first vanishing point VP1 and the second vanishing point VP2 with respect to the virtual image display area VS to be on the upper side as seen by the observer, and when the pitching angle is such that the front of the vehicle 1 is raised, the display control device 30 (display parameter setting module 502) adjusts the relative positions of the first vanishing point VP1 and the second vanishing point VP2 with respect to the virtual image display area VS to be on the lower side as seen by the observer.

The operations of the above-described processing processes can be implemented by executing one or more functional modules of an information processing device, such as a general-purpose processor or an application-specific chip. All of these modules, combinations of these modules, and/or combinations with known hardware that can replace their functions, are within the scope of protection of the present invention.

The functional blocks of the vehicle virtual image display system 10 are optionally implemented by hardware, software, or a combination of hardware and software to implement the principles of the various embodiments described. It will be understood by those skilled in the art that the functional blocks described in FIG. 8 may be optionally combined or a functional block may be separated into two or more sub-blocks to implement the principles of the embodiments described. Thus, the description herein optionally supports any possible combination or division of the functional blocks described herein.

1: Vehicle 2: Projected part (front windshield)
5: Dashboard 6: Driving lane (road surface)
7: Boundary 8: Boundary 10: Vehicle virtual image display system 20: HUD device (head-up display device)
21: Light exit window 22: Housing 24: Light source unit 28: First actuator 29: Second actuator 30: Display control device 31: I/O interface 33: Processor 35: Image processing circuit 37: Memory 40: Display device 50: Light modulation element 51: Display surface 60: Light source unit 80: Relay optical system 81: First mirror 82: Second mirror 86: Actuator 87: Actuator 90: Virtual image optical system 100: Virtual image display area 200: Eye box 200A: Boundary 205: Center 401: Vehicle ECU
403: Road information database 405: Vehicle position detection unit 407: Operation detection unit 409: Eye position detection unit (gaze direction detection unit)
411: Vehicle exterior sensor 413: Brightness detection unit 417: Portable information terminal 419: External communication device 502: Display parameter setting module 504: Graphic module 506: Light source driving module 508: Actuator driving module 510: Eye position detection module 700: Eye position (viewer)
700L: Left eye (eye position)
700R: Right eye (eye position)
AX1: First rotation axis FU: Virtual object FU10: Virtual object K: Display light Kp: Optical axis PT: Target position V: Perspective virtual image V10: Perspective virtual image V1A: Left perspective line V20: Perspective virtual image V1B: Right perspective line V30: Perspective virtual image V31: First perspective virtual image V32: Second perspective virtual image VP0: Real scene vanishing point VP0': Vanishing point VP1: First vanishing point VP2: Second vanishing point VS: Virtual image display area VS1: Upper end VS2: Lower end θd: Vertical arrangement angle θt: Tilt angle

Claims (8)

A display control device (30) that performs display control in a head-up display device that projects a perspective image, which is expressed as being arranged parallel to a road surface , onto a projection target member, thereby allowing an observer to visually recognize a virtual image of the perspective image superimposed on a foreground, comprising:
one or more processors (33);
A memory (37);
one or more computer programs stored in the memory (37) and configured to be executed by the one or more processors (33);
The processor (33)
a first display process for displaying a perspective virtual image in a first display mode having a first vanishing point (VP1) disposed below a vanishing point (VP0) of the foreground as viewed from the observer;
a second display process for displaying a perspective virtual image in a second display mode having a second vanishing point (VP2) disposed above the first vanishing point (VP1) as seen by the observer ;
The processor (33)
If the perspective virtual image is a still image, the first display process is executed;
If the perspective virtual image is a moving image, the second display process is executed .
A display control device (30). A display control device (30) that performs display control in a head-up display device that projects a perspective image, which is expressed as being arranged parallel to a road surface , onto a projection target member, thereby allowing an observer to visually recognize a virtual image of the perspective image superimposed on a foreground, comprising:
one or more processors (33);
A memory (37);
one or more computer programs stored in the memory (37) and configured to be executed by the one or more processors (33);
The processor (33)
a first display process for displaying a perspective virtual image in a first display mode having a first vanishing point (VP1) disposed below a vanishing point (VP0) of the foreground as viewed from the observer;
a second display process for displaying a perspective virtual image in a second display mode having a second vanishing point (VP2) disposed above the first vanishing point (VP1) as seen by the observer ;
The processor (33)
If the perspective virtual image is a moving image that does not expand or contract in a perspective direction as viewed by an observer, or a moving image that does not move in a perspective direction, the first display process is executed;
executes the second display process if the perspective virtual image is a moving image that is perceived as expanding or contracting in a perspective direction or moving in a perspective direction when viewed by an observer.
A display control device (30). A display control device (30) that performs display control in a head-up display device that projects a perspective image, which is expressed as being arranged parallel to a road surface , onto a projection target member, thereby allowing an observer to visually recognize a virtual image of the perspective image superimposed on a foreground, comprising:
one or more processors (33);
A memory (37);
one or more computer programs stored in the memory (37) and configured to be executed by the one or more processors (33);
The processor (33)
a first display process for displaying a perspective virtual image in a first display mode having a first vanishing point (VP1) disposed below a vanishing point (VP0) of the foreground as viewed from the observer;
a second display process for displaying a perspective virtual image in a second display mode having a second vanishing point (VP2) disposed above the first vanishing point (VP1) as seen by the observer ;
The processor (33)
When the surroundings are dark, the first display process is executed;
When the surroundings are bright, the second display process is executed.
A display control device (30). A display control device (30) that performs display control in a head-up display device that projects a perspective image, which is expressed as being arranged parallel to a road surface , onto a projection target member, thereby allowing an observer to visually recognize a virtual image of the perspective image superimposed on a foreground, comprising:
one or more processors (33);
A memory (37);
one or more computer programs stored in the memory (37) and configured to be executed by the one or more processors (33);
The processor (33)
a first display process for displaying a perspective virtual image in a first display mode having a first vanishing point (VP1) disposed below a vanishing point (VP0) of the foreground as viewed from the observer;
a second display process for displaying a perspective virtual image in a second display mode having a second vanishing point (VP2) disposed above the first vanishing point (VP1) as seen by the observer ;
The processor (33)
When the moving speed or the deformation speed of the perspective virtual image is slow, the first display process is executed;
When the moving speed or the deformation speed of the perspective virtual image is fast, the second display process is executed.
A display control device (30). A display control device (30) that performs display control in a head-up display device that projects a perspective image, which is expressed as being arranged parallel to a road surface , onto a projection target member, thereby allowing an observer to visually recognize a virtual image of the perspective image superimposed on a foreground, comprising:
one or more processors (33);
A memory (37);
one or more computer programs stored in the memory (37) and configured to be executed by the one or more processors (33);
The processor (33)
a first display process for displaying a perspective virtual image in a first display mode having a first vanishing point (VP1) disposed below a vanishing point (VP0) of the foreground as viewed from the observer;
a second display process for displaying a perspective virtual image in a second display mode having a second vanishing point (VP2) disposed above the first vanishing point (VP1) as seen by the observer ;
The processor (33)
executes the first display process when a period from when the perspective virtual image starts to when the perspective virtual image ends is shorter than a predetermined threshold value;
executing the second display process when a period from when the display of the perspective virtual image starts to when the display of the perspective virtual image ends is longer than the predetermined threshold value;
A display control device (30). In the second display process,
The second vanishing point (VP2) is located below the foreground vanishing point (VP0);
A display control device (30) according to any one of the preceding claims. A display control device (30) according to any one of claims 1 to 6,
A light modulation element (50) that emits display light;
A head-up display device (20) comprising: a relay optical system (80) that directs the display light from the light modulation element (50) to a projection target. A display control method for performing display control in a head-up display device that projects a perspective image, which is expressed as being arranged parallel to a road surface, onto a projection target member, thereby allowing an observer to visually recognize a virtual image of the perspective image superimposed on a foreground , comprising:
a first display process for displaying a perspective virtual image in a first display mode having a first vanishing point (VP1) disposed below a vanishing point (VP0) in the foreground as seen by an observer;
a second display process for displaying a perspective virtual image in a second display mode having a second vanishing point (VP2) disposed above the first vanishing point (VP1) as seen by the observer ;
When the surroundings are dark, the first display process is executed;
When the surroundings are bright, the second display process is executed.
Display control method.

Images