JP7797352B2 - Head-up display device and video data processing method - Google Patents

Description

The present invention relates to a head-up display device and a method for processing video data, and relates to technology for a head-up display device that uses AR (Augmented Reality), for example.

Patent Document 1 discloses a display system that extracts an object to be displayed in AR based on imaging data, generates AR image data for the object, and sets the frame rate of the AR image data based on the importance of the object. For example, if the overall frame rate is 60 fps and three AR image data are displayed in a time-division manner, the frame rates of the three AR image data are set to 30 fps, 20 fps, and 10 fps, respectively, depending on their importance. This makes it possible to suppress flickering more effectively for AR image data with higher importance.

Japanese Patent Application Laid-Open No. 2019-6164

A head-up display device prepares various image data based on information obtained from ADAS (Advanced Driver Assistance Systems) and other systems, and projects image light based on that image data onto a display area to be viewed as a virtual image. The preparation time required to prepare the image data varies depending on the type and number of images, the size of the image based on the virtual image distance (distance), the display format (2D/3D), the presence or absence of graphics effects, and other factors. In this specification, a head-up display device is also referred to as a HUD device.

Here, in a HUD device, processing performance such as maximum frame rate is typically fixed and determined primarily based on the hardware specifications. However, there are cases where a desire arises to change the display specifications of the HUD device by updating the software without changing the hardware itself. In such cases, the required display specifications of the HUD device often involve an increase in processing load. As a result, there is a risk that the hardware's processing performance may be insufficient, resulting in a failure to meet the required display specifications, such as dropped frames during AR display.

The present invention was made in light of these circumstances, and one of its objectives is to provide a head-up display device and a video data processing method that can flexibly accommodate display specifications while satisfying the required display specifications as much as possible within the scope of the hardware's processing performance.

The above and other objects and novel features of the present invention will become apparent from the description in this specification and the accompanying drawings.

A brief summary of the representative inventions disclosed in this application is as follows:

A typical head-up display device includes an image display unit that displays an image and emits image light of the displayed image; an image light projection unit that projects the emitted image light onto a display area, causing the projected image light to be viewed as a virtual image; and a control unit that determines display content based on acquired vehicle information, prepares image data based on the determined display content, and displays an image based on the prepared image data on the image display unit. If the preparation time required to prepare the image data for the provisionally determined display content before determining the display content is longer than a predetermined processing cycle, the control unit changes the provisionally determined display content.

To briefly explain the effect obtained by a representative invention among those disclosed in this application, it enables a head-up display device to flexibly accommodate display specifications.

1 is a schematic diagram showing a configuration example of a vehicle equipped with a head-up display device according to a first embodiment. FIG. 2 is a schematic diagram illustrating a configuration example of a main part of the HUD device in FIG. 1 . 2B is a schematic diagram showing an example of the configuration of a main part of the HUD device in FIG. 1 that is different from that in FIG. 2A. FIG. 3 is a block diagram showing an example of the configuration of a main part of a control system that is responsible for control in the HUD device shown in FIGS. 2A and 2B. FIG. 3B is a block diagram showing an example of the configuration of a main part of a control system that is responsible for control in the HUD device shown in FIGS. 2A and 2B, which is different from that shown in FIG. 3A. 3A and 3B are block diagrams showing examples of the configuration of parts related to the control unit. FIG. 2 is a schematic diagram showing an example of display content of the HUD device shown in FIG. 1 . FIG. 3B is a flowchart showing an example of a processing procedure when displaying an image in the HUD device shown in FIG. 3A. 7 is a timing chart schematically illustrating an example of a problem that occurs when displaying video using the flow shown in FIG. 6. FIG. 10 is a schematic diagram showing how the display content changes when no dropped frames occur. 10A and 10B are schematic diagrams showing how display content changes when a frame is dropped. 3B is a diagram showing an example of an internal state of the control unit shown in FIG. 3A. FIG. 3B is a diagram showing an example of a result of monitoring the preparation time by the control unit shown in FIG. 3A. FIG. FIG. 10 is a schematic diagram showing an example of the display content of the HUD device in the normal state shown in FIG. 9 . FIG. 10 is a schematic diagram showing an example of the display content of the HUD device in the suppression transition state shown in FIG. FIG. 10 is a schematic diagram showing an example of the display content of the HUD device in the suppressed state shown in FIG. 9 . 3B is a flowchart showing an example of processing performed by a control unit shown in FIG. 3A. FIG. FIG. 3C is a schematic diagram showing an example of the relationship between video data written to a frame buffer and display content displayed on a video display unit in the HUD device shown in FIGS. 3A and 3B. FIG. 13B is a schematic diagram showing an example of a relationship different from that shown in FIG. 13A. FIG. 6 is a diagram showing an example of items that affect the preparation time for video data. FIG. 10 is a flowchart showing an example of a processing procedure for displaying an image in the HUD device according to the second embodiment. FIG. 16 is a flowchart showing an example of detailed processing content of the display content adjustment processing in FIG. 15 . FIG. 17 is a diagram showing a specific example of a method for predicting preparation time (step S412) in FIG. 16. FIG. 17B is a diagram showing a specific example different from FIG. 17A. FIG. 17 is a diagram illustrating an example of a method for changing the display content in FIG. 16 (step S414). FIG. 19 is a schematic diagram showing an example of the display content of the HUD device obtained as a result of changing the display content in FIG. 18. 17 is a timing chart for displaying video using the flow shown in FIGS. 15 and 16. FIG. 11 is a schematic diagram showing an example of main processing contents of a control unit in a HUD device according to a third embodiment. FIG. 21B is a supplementary figure to FIG. 21A. FIG. 21B is a schematic diagram showing an example different from FIG. 21A. FIG. 22B is a supplementary figure to FIG. 22A. FIG. 11 is a diagram showing an example of the configuration of a display setting table stored in a control unit in the HUD device according to the third embodiment. FIG. 11 is a flowchart showing an example of processing details of a control unit in the HUD device according to the third embodiment. FIG. 24B is a flowchart showing an example of processing content different from that shown in FIG. 24A.

Embodiments of the present invention will be described in detail below with reference to the drawings. In all drawings used to explain the embodiments, identical components will generally be designated by the same reference numerals, and repeated explanations will be omitted.

(Embodiment 1)
<Overview of HUD device>
FIG. 1 is a schematic diagram showing an example of the configuration of a vehicle equipped with a head-up display device according to a first embodiment. The head-up display (HUD) device 1 shown in FIG. 1 is mounted on a vehicle 2, which is a type of vehicle. The vehicle 2 is typically an automobile, but is not necessarily limited to this and may be a railway vehicle or the like. The vehicle is not limited to a car and may be an aircraft or the like. The vehicle 2 is also equipped with a control unit 21, for example, called an ECU (Electronic Control Unit).

The control unit 21 acquires vehicle information 4, for example, from various sensors installed in various parts of the vehicle 2, as well as from a navigation device, etc. The various sensors detect, for example, various events that occur in the vehicle 2 and various parameter values related to the driving conditions. The HUD device 1 acquires the vehicle information 4 acquired by the control unit 21, for example, using CAN (Controller Area Network) communication, etc.

Vehicle information 4 includes, for example, vehicle 2 speed information, gear information, steering wheel angle information, lamp illumination information, external light information, distance information, infrared information, engine ON/OFF information, camera image information inside and outside the vehicle, acceleration gyro information, GPS (Global Positioning System) information, navigation information, vehicle-to-vehicle communication information, and road-to-vehicle communication information. GPS information also includes information such as the current time. Vehicle information 4 also includes various warning information. Based on this vehicle information 4, HUD device 1 projects image light onto a display area such as windshield 3. As a result, HUD device 1 allows a user such as the driver to visually recognize the image light projected onto the display area as a virtual image, specifically as a virtual image superimposed on the scenery ahead of vehicle 2.

Figure 2A is a schematic diagram showing an example configuration of the main parts of the HUD device in Figure 1. The HUD device 1 shown in Figure 2A includes, for example, an image display unit 11 housed in a housing 12, mirrors M1 and M2, and a mirror driver 14. The image display unit 11 is, for example, a display panel such as an LCD (Liquid Crystal Display) or a projector, and displays an image based on input image data and emits image light of the displayed image.

Mirror M2 reflects the image light from the image display unit 11 toward mirror M1. Mirror M2 is effective in saving space and ensuring a long optical path length. Depending on the space within the HUD housing and the required optical path length, mirror M2 may not be installed, or multiple mirrors M2 may be installed. Mirror M1 functions as an image light projection unit. Mirror M1, which is the image light projection unit, projects the image light emitted from the image display unit 11 and reflected by mirror M2 onto the display area 5 of the windshield 3 through an opening 7 provided in the dashboard 10. In this way, the image light projection unit allows the user 6 to view the projected image light as a virtual image.

More specifically, mirror M1 is, for example, a concave mirror (magnifying mirror) that reflects and magnifies the image light reflected by mirror M2 and projects it onto display area 5 through opening 7. The image light projected onto display area 5 is reflected by display area 5 and enters the eyes of user 6. As a result, user 6 visually recognizes the image light projected onto display area 5 as a virtual image 9 that exists beyond transparent windshield 3 and is superimposed on the scenery outside the vehicle (roads, buildings, people, etc.). The information represented by virtual image 9 includes a variety of information, such as road signs, the vehicle's current speed, and various information attached to objects in the scenery, i.e., AR information.

Mirrors M1 and M2 may be, for example, free-form mirrors or mirrors with an asymmetrical optical axis. Here, mirror M2 has a fixed installation angle. On the other hand, mirror M1 is equipped with a mirror driver 14. Mirror driver 14 variably adjusts the installation angle of mirror M1. Specifically, mirror driver 14 includes, for example, a motor, and rotates mirror M1 by the rotation of the motor.

By adjusting the installation angle of mirror M1, the position of display area 5 on windshield 3, i.e., the vertical position of the virtual image viewed by user 6, can be adjusted. Furthermore, by adjusting the installation angle of mirror M1, it is possible to protect image display unit 11 from sunlight. Specifically, sunlight can travel in the opposite direction along the optical path of the image light and enter image display unit 11. If the incidence of sunlight increases the possibility of damaging image display unit 11, the installation angle of mirror M1 can be changed to prevent sunlight from reaching image display unit 11.

Figure 2B is a schematic diagram showing an example configuration of the main parts of the HUD device in Figure 1, different from Figure 2A. Unlike the configuration shown in Figure 2A, the HUD device 1 shown in Figure 2B has a lens LS instead of a mirror M2 inside the housing 12. Image light from the image display unit 11 is incident on the mirror M1 via the lens LS. As in the case of Figure 2A, the mirror M1 projects the incident image light onto the display area 5 through the opening 7. As in the case of Figure 2A, a mirror driver may be installed on the mirror M1. The configuration shown in Figure 2B can be applied, for example, to vehicles such as minivans and trucks, where the windshield 3 is installed at an angle close to vertical.

Figure 3A is a block diagram showing an example configuration of the main components of the control system responsible for control in the HUD device shown in Figures 2A and 2B. The HUD device 1 shown in Figure 3A includes a mirror driver 14, a display driver 15, a communication unit 16, a memory 17, a frame buffer 18, and a control unit 20, all of which are connected to one another via a bus 13.

The communication unit 16 receives and transmits vehicle information and is realized, for example, by a communication interface circuit or the like, and functions as an information acquisition unit. The communication unit 16 acquires or receives information about the vehicle from the control unit 21 using CAN communication or the like, and transmits the received information about the vehicle to the control unit 20. The control unit 20 controls the mirror drive unit 14 and display drive unit 15 based on the information from the communication unit 16. The mirror drive unit 14 adjusts the installation angle of the mirror M1, for example, as described in FIG. 2A, in response to commands from the control unit 20. The mirror drive unit 14 can be realized by a motor such as the one described in FIG. 2A, as well as a motor driver circuit that drives the motor.

The frame buffer 18 is, for example, composed of volatile memory, and stores video data. The display driver 15 reads the video data stored in the frame buffer 18 via the bus 13 and drives the video display unit 11 based on the video data. The video display unit 11 is, for example, a liquid crystal display equipped with a light source and a display panel. The display panel displays images by modulating the backlight emitted from the light source for each pixel based on the video data. In this case, the display driver 15 can be realized by an LCD driver circuit, etc.

Memory 17 is, for example, a combination of volatile and non-volatile memory, and stores programs, data, and the like used by control unit 20. Control unit 20 is realized by a processor such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit), and controls the entire HUD device 1 by executing the programs stored in memory 17. As part of this, control unit 20 prepares video data, including creating the video data, based on information about the vehicle acquired by communication unit 16, i.e., the information acquisition unit, and displays an image based on the prepared video data on video display unit 11.

The communication unit 16, memory 17, frame buffer 18, and control unit 20 shown in FIG. 3A may be mounted on a microcontroller or the like. However, this implementation is not limited to this, and implementations may also be made by appropriately combining, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), etc.

Figure 3B is a block diagram showing an example configuration of the main components of the control system that are responsible for control in the HUD device shown in Figures 2A and 2B, different from those shown in Figure 3A. Unlike the example configuration shown in Figure 3A, the HUD device 1 shown in Figure 3B does not have a memory 17 or control unit 20. In this case, the control unit 21 creates video data in place of the control unit 20 shown in Figure 3A, and writes the created video data to the frame buffer 18 via the communication processing unit 16a.

In the configuration example shown in Figure 3B, the control unit 20 described in Figure 3A may function as the communication processing unit 16a. Alternatively, some functions may be shared between the control unit 21 and the communication processing unit 16a. The configuration example in Figure 3B is provided with a mirror driving unit 14, a display driving unit 15, and a communication processing unit 16a. Unlike the communication unit 16 in Figure 3A, the communication processing unit 16a receives information about the vehicle from the control unit 21 using CAN communication or the like, processes the received information, and adjusts the operation of the mirror driving unit 14 and the display driving unit 15 based on the processing results.

Figure 4 is a block diagram showing an example configuration of the parts related to the control unit in Figures 3A and 3B. The control unit 21 acquires vehicle information 4 as described in Figure 1. As shown in Figure 4, the vehicle information 4 is generated by information acquisition devices such as various sensors connected to the control unit 21. Figure 4 shows an example of such an information acquisition device.

In Figure 4, for example, vehicle speed sensor 101 detects the speed of vehicle 2 in Figure 1 and generates speed information as the detection result. Shift position sensor 102 detects the current gear and generates gear information as the detection result. Steering wheel steering angle sensor 103 detects the current steering wheel steering angle and generates steering wheel steering angle information as the detection result. Headlight sensor 104 detects whether the headlights are on or off and generates lamp illumination information as the detection result. Illuminance sensor 105 and chromaticity sensor 106 detect external light and generate external light information as the detection result.

The distance sensor 107 detects the distance between the vehicle 2 and external objects and generates distance information as the detection result. The infrared sensor 108 detects the presence and distance of objects in the vicinity of the vehicle 2 and generates infrared information as the detection result. The engine start sensor 109 detects whether the engine is on or off and generates on or off information as the detection result. The acceleration sensor 110 and gyro sensor 111 detect the acceleration and angular velocity of the vehicle 2, respectively, and generate acceleration gyro information representing the attitude and behavior of the vehicle 2 as the detection result. The temperature sensor 112 detects the temperature inside and outside the vehicle and generates temperature information as the detection result.

The road-to-vehicle communication wireless receiver 113 generates road-to-vehicle communication information through road-to-vehicle communication between the vehicle 2 and roads, signs, traffic lights, etc. The vehicle-to-vehicle communication wireless receiver 114 generates vehicle-to-vehicle communication information through vehicle-to-vehicle communication between the vehicle 2 and other vehicles in the vicinity. The interior camera 115 and exterior camera 116 capture images of the interior and exterior of the vehicle, respectively, to generate interior camera image information and exterior camera image information. The interior camera 115 is, for example, a camera for a DMS (Driver Monitoring System) that captures the posture, eye position, movement, etc. of the user 6, as shown in Figure 2A, etc. In this case, by analyzing the captured image, the user 6's fatigue level, line of sight, etc. can be determined.

On the other hand, the exterior vehicle camera 116 captures images of the surrounding environment, such as the area in front of and behind the vehicle 2. In this case, by analyzing the captured images, it is possible to determine the presence or absence of obstacles such as other vehicles or people in the vicinity, buildings, terrain, road conditions such as rain, snow, ice, unevenness, etc., and road signs. The exterior vehicle camera 116 may also include, for example, a drive recorder that records images of the situation while driving.

The GPS receiver 117 generates GPS information by receiving GPS signals. For example, the current time can be obtained using the GPS receiver 117. The VICS (Vehicle Information and Communication System, registered trademark) receiver 118 generates VICS information by receiving VICS signals. The GPS receiver 117 and VICS receiver 118 may be provided as part of a navigation device. Note that the various information acquisition devices shown in FIG. 4 can be deleted, or other types of devices can be added or replaced with other types of devices, as appropriate.

<About the HUD device display>
Fig. 5 is a schematic diagram showing an example of the display content of the HUD device shown in Fig. 1. The display content shows an example of an AR display, and is an example of the virtual image 9 shown in Fig. 2A etc. In the example shown in Fig. 5, five images VDa to VDe are displayed. In the specification, the multiple images VDa to VDe are collectively referred to as an image VD, an image VD, or an image object VD.

The image VDa is a piece of scenery and is displayed superimposed on an object OB, which is the subject of AR, in this case a person. The image VDa indicates that the object OB has been detected by the various information acquisition devices shown in Figure 4. In other words, the HUD device 1 acquires information indicating the detection results of the object OB from the control unit 21. The image VDa also displays warning information to alert the user 6, for example, the driver, to the object OB.

Video VDb is displayed on a road, which is one type of scenery, and shows the direction of travel of vehicle 2. Video VDc shows navigation information. Video VDd shows, for example, road signs, which are one type of road-to-vehicle communication information. Video VDe shows vehicle 2's speed information. Note that videos VDa and VDb are 3D graphics, while videos VDc to VDe are 2D graphics.

Figure 6 is a flow diagram showing an example of the processing steps for displaying an image in the HUD device shown in Figure 3A. In Figure 6, first, the communication unit 16, i.e., the information acquisition unit, acquires information about the vehicle from the control unit 21 (step S11). Next, the control unit 20 performs a preparation process for the image data, including steps S121 to S124 (step S12). For example, the processor executes the image processing program in the memory 17 to perform the processes of steps S121 to S123.

In step S121, the control unit 20 determines the display content based on the information about the vehicle acquired in step S11, for example, the vehicle information 4 shown in FIG. 1. Specifically, the control unit 20 selects the vehicle information 4 that corresponds to the display, and determines the position, size, and layout in which the video VD representing the selected vehicle information 4 will be displayed. In the example shown in FIG. 5, the control unit 20 determines that five videos VDa to VDe will be displayed as the display content. In step S122, the control unit 20 creates video data for each video VD based on the display content determined in step S121.

In step S123, the control unit 20 writes each of the multiple pieces of video data created in step S122 to a storage area in the frame buffer 18 corresponding to the position where it should be displayed. In step S124, the control unit 20 performs distortion correction on all of the video data in the frame buffer 18, for example, in accordance with the curvature of the windshield 3. Note that distortion correction may be achieved, for example, by the processor executing a distortion correction program in the memory 17, or by dedicated hardware.

After completing this video data preparation process (step S12), the video display process is performed (step S13). In step S13, the display drive unit 15 reads the video data stored in the frame buffer 18 and drives the video display unit 11 based on the video data, causing the video display unit 11 to display the video VD. The processing procedure shown in Figure 6 is executed in synchronization with a processing cycle determined based on the frame rate. For example, when the frame rate is 60 fps, the processing cycle is 16.6 ms. The processing procedure for displaying the video VD is not limited to that shown in Figure 6, and can be replaced with various commonly known procedures.

Figure 7 is a timing chart that schematically illustrates an example of a problem that occurs when displaying video using the flow shown in Figure 6. Figure 7 shows the operations performed in the first through fifth processing or control cycles Tc[1] through Tc[5]. In the specification, the processing or control cycles Tc[1] through Tc[5] are collectively referred to as the processing cycle or control cycle Tc. The control cycle Tc is, for example, 16.6 ms. Figure 7 also illustrates the operations that occur when the processes of steps S11 and S12 and step S13 in Figure 6 are executed in a pipeline.

In Figure 7, in the first control cycle Tc[1], information about the vehicle is acquired (step S11), and video data is prepared based on that information (step S12). The video data preparation time Tp[1] required for processing steps S11 and S12 is shorter than the control cycle Tc[1]. This is the case, for example, when the number of AR objects and, therefore, the number of video VDs to be displayed is small, and the preparation time Tp[1] fits within the control cycle Tc[1]. Similarly, in the second control cycle Tc[2], information about the vehicle is acquired (step S11), and video data is prepared based on that information (step S12). In parallel with this, in the second control cycle Tc[2], video VD display processing is performed based on the video data prepared in the first control cycle Tc[1] (step S13).

Here, in the second control cycle Tc[2], the preparation time Tp[2] for the video data required for processing steps S11 and S12 is longer than the control cycle Tc[2]. The preparation time Tp[2] can be lengthened, for example, due to an increase in the number of AR objects and therefore the video VD to be displayed. As a result, this video data is not reflected in the display processing in the third control cycle Tc[3], but is reflected in the display processing in the fourth control cycle Tc[4]. As a result, dropped frames occur in the third control cycle Tc[3].

Furthermore, in the fourth control cycle Tc[4], the preparation time Tp[3] for the video data required for processing steps S11 and S12 is shorter than that of control cycle Tc[4]. As a result, this video data is reflected in the display process in the fifth control cycle Tc[5]. However, depending on the method, if a frame is dropped, the video data prepared in preparation time Tp[3] may actually be the video data that should have been prepared in the third control cycle Tc[3]. Note that in this specification, the multiple preparation times Tp[1] to Tp[3] are collectively referred to as preparation time Tp.

Figure 8A is a schematic diagram showing how the display content changes when no frames are dropped. Figure 8B is a schematic diagram showing how the display content changes when frames are dropped. For example, as shown in Figures 8A and 8B, consider a case in which an object OB, in this case a person, moves from time t1 to time t2. When no frames are dropped, as shown in Figure 8A, an image VDa(t1) is displayed superimposed on the position of object OB(t1) at time t1, and an image VDa(t2) is displayed superimposed on the position of object OB(t2) at time t2.

In this way, when no dropped frames occur, it is possible to display video VD with high real-time performance that tracks the AR object. On the other hand, when dropped frames occur, particularly when dropped frames occur consecutively over successive control cycles Tc, as shown in Figure 8B, video VDa (t2) is displayed at a position shifted toward object OB (t1) from the position of object OB (t2) at time t2. In this case, video VDa (t2) is not completely superimposed on object OB (t2). In this way, when dropped frames occur, it is possible that video VD with low real-time performance that does not track the AR object will be displayed.

<Outline of operation of the control unit>
Therefore, in order to display a video VD with high real-time performance, the control unit 20 generally performs control to reduce the processing load required for preparing the video data so that the preparation of the video data is completed within a predetermined processing cycle or control period Tc. That is, the control unit 20 prepares the video data based on information about the vehicle received by the communication unit 16, and if the preparation conditions for the video data do not satisfy predetermined conditions within the predetermined processing cycle, the control unit 20 modifies some of the content of the video data to be prepared. More specifically, the control unit 20 monitors the preparation time Tp required to prepare the video data, and when the preparation time Tp satisfies predetermined conditions, it initiates control to reduce the processing load. Furthermore, the predetermined processing cycle or control period Tc is determined based on the frame rate.

Preparation time Tp is the time required for processing steps S11 and S12 shown in Figure 6, assuming an operating method such as that described in Figure 7. On the other hand, for example, assuming an operating method in which the processing of step S11 and the processing of step S12 are executed in a pipeline, preparation time Tp may be the time required for processing step S12. However, to improve real-time performance, it is preferable to use the operating method described in Figure 7.

The control unit 20 continuously monitors the preparation time Tp. For example, the preparation time can be monitored in conjunction with the vehicle time or using a timer or the like. Assuming the operating method described in Figure 7, as shown in Figure 3A, the control unit 20 uses a timer or the like to monitor the preparation time Tp required from the start of acquiring information about the vehicle using the communication unit 16 until the creation of the video data is completely written to the frame buffer 18. When using the configuration shown in Figure 3B, unlike the case of Figure 3A, the communication processing unit 16a acquires information from the control unit 21, processes the acquired information, and monitors the preparation time Tp required from the end of writing the processed results to the frame buffer 18 using a timer or the like.

Figure 9 is a diagram showing an example of the internal state of the control unit shown in Figure 3A. Figure 10 is a diagram showing an example of the results of monitoring the preparation time by the control unit shown in Figure 3A. As shown in Figure 9, the control unit 20 has the following internal states: normal state ST0, return transition state ST1, suppression transition state ST2, and suppression state ST3. In other words, the control unit 20 has the following operating modes: normal mode ST0, return transition mode ST1, suppression transition mode ST2, and suppression mode ST3. Having four states is just one example; in another example, the control unit 20 may have two states: normal state ST0 and suppression state ST3.

In the normal state ST0, the control unit 20 prepares video data as usual based on information acquired using the communication unit 16, and displays the video VD based on this data on the video display unit 11. Furthermore, in the normal state ST0, the control unit 20 transitions to the suppression transition state ST2 if (A) the preparation time Tp, which is the monitoring result, is longer than the control period Tc, or (B) the preparation time Tp is longer than the first threshold time Tth1 multiple times in a row. The number of consecutive times is set to a value between 2 and 10, for example.

As a specific example, in Figure 10, the preparation time Tp4 obtained at monitoring time tm4 is longer than the control period Tc. Furthermore, the preparation times Tp2 and Tp3 obtained at successive monitoring times tm2 and tm3 are both shorter than the control period Tc, but longer than the first threshold time Tth1. The first threshold time Tth1 is shorter than the control period Tc of 16.6 ms, and is, for example, 15.0 ms.

In normal state ST0, the control unit 20 transitions to suppression transition state ST2 when (A) a monitoring result such as preparation time Tp4 is obtained, or (B) a monitoring result such as preparation times Tp2 or Tp3 is obtained multiple times in succession, for example, twice in succession. Condition (A) is intended to quickly resolve dropped frames. On the other hand, condition (B) is intended to prevent a situation in the near future in which preparation time Tp becomes longer than control cycle Tc, thereby causing dropped frames.

After starting control to reduce the processing load in the suppression transition state ST2, the control unit 20 gradually increases the amount of processing load reduction for each control cycle Tc within a predetermined suppression transition period. The suppression transition period is set to, for example, 5 seconds. As will be described in detail later, the control unit 20 reduces the amount of video data to be prepared and the processing load by, for example, not creating some video data or simplifying some video data. In this case, if the amount of video data were to be suddenly reduced, the display content would also suddenly decrease, which is undesirable from the perspective of the user 6. Therefore, the control unit 20 gradually increases the amount of data to be reduced. Then, after the suppression transition period, for example, 5 seconds, has elapsed, the control unit 20 transitions to the suppression state ST3.

In the suppression state ST3, the control unit 20 prepares video data with the reduced processing load. Furthermore, when the state in which the preparation time Tp is shorter than the second threshold time Tth2 continues for a predetermined threshold duration TthD or longer in the suppression state ST3, the control unit 20 transitions to the return transition state ST1. The second threshold time Tth2 is shorter than the control period Tc, and is set, for example, to the same time as the first threshold time Tth1 or to a time shorter than the first threshold time Tth1. The threshold duration TthD is a period that is at least multiple times the control period Tc, and is set, for example, to 5 seconds.

As a specific example, in Figure 10, the second threshold time Tth2 is set to a time shorter than the first threshold time Tth1. The preparation times Tp5 and Tp6 obtained at monitoring times tm5 and tm6 are both shorter than the second threshold time Tth2. The control unit 20 transitions to the return transition state ST1 when monitoring results such as preparation times Tp5 and Tp6 are obtained continuously for the threshold duration TthD or longer.

In this way, if the state of "preparation time Tp < second threshold time Tth2" is occurring stably, it is expected that the above-mentioned conditions (A) and (B) will no longer be met even if the normal state ST0 is restored. Note that in some cases, the control unit 20 may not perform such a condition determination, and may simply transition to the restoration transition state ST1 after a period of time such as 5 seconds has passed, anticipating that the processing load will naturally be reduced after a predetermined period of time has passed.

In the return transition state ST1, the control unit 20 gradually reduces the amount of processing load reduction for each control cycle Tc within a predetermined return transition period. The return transition period is set to, for example, 5 seconds. Then, after the return transition period, for example, 5 seconds, has elapsed, the control unit 20 transitions to the normal state ST0. By providing the return transition state ST1, it is possible to avoid situations that are undesirable from the perspective of the user 6, as in the case of the suppression transition state ST2.

<Method for reducing processing load>
Fig. 11A is a schematic diagram showing an example of the display contents of the HUD device in the normal state shown in Fig. 9. In the normal state ST0, for example, as shown in Fig. 11A, seven images VDa1 to VDa3 and VDb to VDe are displayed. The images VDa1 and VDa2 are displayed so as to be superimposed on objects OB1 and OB2, respectively (people in this case). The image VDa3 is displayed so as to be superimposed on object OB3, a vehicle in this case. The images VDb to VDe represent navigation information, direction of travel, road signs, and speed information, respectively, as in Fig. 5.

Figure 11B is a schematic diagram showing an example of the display content of the HUD device in the suppression transition state shown in Figure 9. In suppression transition state ST2, the display content shown in Figure 11A is used as the reference, and as indicated by reference numerals 201 and 203 in Figure 11B, two images VDa2 and VDd in Figure 11A have been deleted. In other words, image VDa2 superimposed on object OB2, which is farther away than object OB1, and image VDd representing a road sign have been deleted. In addition, as indicated by reference numeral 202 in Figure 11B, image VDa3 in Figure 11A has been simplified by, for example, making it uncolored.

Furthermore, in FIG. 11B, an image VDm1, such as a mark, is displayed to notify the user 6 that the suppression transition state ST2, in other words, the suppression transition period, is in effect. In the suppression transition state ST2, the control unit 20 writes the image data of the image VDm1 as a template to a fixed storage area in the frame buffer 18. By displaying such a mark, the user 6 can recognize that the change in display content is not due to a malfunction, but rather is due to the activation of the display suppression function.

When performing the display shown in FIG. 11B, the control unit 20 reduces the amount of video data prepared in step S12 by not displaying videos VDa2 and VDd and applying simplified display to video VDa3 in step S121 shown in FIG. 6. This allows the control unit 20 to reduce the processing load required to prepare the video data. In other words, the control unit 20 reduces the time required for processing in step S122 and the time required for processing in step S123. The control unit 20 also reduces the amount of data in this manner in stages.

Figure 11C is a schematic diagram showing an example of the display content of the HUD device in the suppressed state shown in Figure 9. In the suppressed state ST3, the display content shown in Figure 11B is used as a reference, and the two images VDa1 and VDb in Figure 11B are further simplified as indicated by reference numerals 301 and 302 in Figure 11C. In other words, the two images VDa1 and VDb are simplified by removing color and reducing their size. Furthermore, Figure 11C displays an image VDm2, such as a mark, to notify the user 6 that the suppressed state ST3, or in other words, the suppressed period, is in effect.

As an example of specific processing, the control unit 20 stores in advance in the memory 17 a suppression table that defines the correspondence between the type of vehicle information 4 and its priority. For example, in the suppression transition state ST2, the control unit 20 selects vehicle information 4 in descending order of priority based on the suppression table, and gradually increases the number of selected vehicle information 4. Then, the control unit 20 hides the video VD representing the selected vehicle information 4, or simplifies it in a predetermined manner.

Here, the priority in the suppression table is determined, for example, according to the following criteria: First, the greater the contribution of vehicle information 4 to safe driving, the higher the priority. Furthermore, if vehicle information 4 indicates a warning and the subject of the warning is a person or a vehicle, the person is given a higher priority. However, in this case, the priority may be weighted according to the distance between the vehicle and the object. For example, if the person is far away and the vehicle is extremely close, the vehicle may be given priority. Furthermore, a person approaching the vehicle may be given a higher priority than a person moving away from the vehicle.

Furthermore, with regard to the image VDb showing the direction of travel shown in Figures 11B and 11C, for example, the priority may be lowered if the road is a series of straight lines based on navigation information, and the priority may be higher if the timing for a right or left turn is approaching. Furthermore, with regard to the image VDb, the priority may be lowered based on past driving history if the vehicle is traveling on a frequently used road, and the priority may be higher if the vehicle is traveling on a road that has not been used before. Furthermore, with regard to the road sign shown in Figures 11A and 11B, i.e., the image VDd showing the speed limit, the priority may be changed based on the difference from the vehicle's driving speed.

The control unit 20 may reduce the processing load by, for example, displaying the video VD once every multiple control cycles Tc, rather than completely deleting the video VD. Furthermore, the control unit 20 is not limited to such methods of reducing the amount of video data. For example, the control unit 20 may reduce the processing load by simplifying the distortion correction process in step S124 in FIG. 6, thereby reducing the accuracy.

<Detailed operation of the control unit>
Fig. 12 is a flow diagram showing an example of the processing contents of the control unit shown in Fig. 3A. The control unit 20 executes the flow shown in Fig. 12, for example, when the adjustment of the installation angle of the mirror M1 shown in Fig. 2A is completed and the environment for image projection is ready. In Fig. 12, the control unit 20 starts monitoring the preparation time Tp using a timer or the like (step S20). Next, the control unit 20 waits for the occurrence of a start trigger (step S21). The start trigger is generated every control period Tc.

When a start trigger occurs in step S21, the control unit 20 acquires information about the vehicle using the communication unit 16, i.e., the information acquisition unit (step S22). Next, the control unit 20 checks the current internal state described in FIG. 9 (step S23). Then, the control unit 20 performs the video data preparation process described in step S12 of FIG. 6 (step S24). At this time, if the internal state is any state other than the normal state ST0, the control unit 20 performs control to reduce the processing load required for preparing the video data, as described in FIGS. 9, 11B, and 11C.

When the control unit 20 completes the video data preparation process in step S24, it generates a preparation complete signal (step S25). For example, in response to a start trigger after generating the preparation complete signal, the control unit 20 outputs a display start command to the display drive unit 15. In response to the display start command, the display drive unit 15 performs display processing of the video VD as described in step S13 in FIG. 6 and in FIG. 7. Furthermore, upon completing the video data preparation process, the control unit 20 evaluates the preparation time Tp, which is the monitoring result obtained in step S21 (step S26).

Then, the control unit 20 determines whether the state transition described in FIG. 9 is necessary based on the preparation time Tp, which is the monitoring result (step S27). If a state transition is necessary (step S27: Yes), the control unit 20 determines the transition destination (step S29), transitions the state to the determined transition destination, and updates the internal state (step S30). On the other hand, if a state transition is not necessary (step S27: No), the control unit 20 returns to step S21 and waits for the next start trigger, repeating the same process until a request to end the HUD display is issued (step S28).

<Major Effects of First Embodiment>
As described above, the method of the first embodiment performs control to reduce the processing load required for preparing the video data so that the preparation of the video data is completed within the control period Tc, thereby suppressing the occurrence of dropped frames and ensuring a minimum display quality. Specifically, even if the HUD device 1 is designed to prevent dropped frames, a software update or the like that involves a change in the display specifications may increase the processing load and cause dropped frames. By using the method of the first embodiment, dropped frames and the like can be suppressed even in such cases. As a result, it becomes possible to satisfy the required display specifications as much as possible within the processing performance limits of the hardware and flexibly accommodate the display specifications.

(Embodiment 2)
The method of the first embodiment is a method of actually monitoring the preparation time Tp required to prepare video data or image data, thereby reducing the processing load so that preparation of the video data or image data is completed within the control period Tc, in other words, the processing period. In the second embodiment, a method of predicting the preparation time Tp will be described, thereby reducing the processing load so that preparation of the video data or image data is completed within the control period Tc, in other words, the processing period. The following description will be given using video data.

<Video data preparation process>
Fig. 13A is a schematic diagram showing an example of the relationship between video data written to the frame buffer and the display content displayed on the video display unit in the HUD device shown in Fig. 3A and Fig. 3B. Fig. 13B is a schematic diagram showing an example of a relationship different from that shown in Fig. 13A. For example, as described in Fig. 6, in the video data preparation process (step S12), video data based on the determined display content is written to the frame buffer 18 (steps S122 and S123).

In Figure 13A, six images VDa1 to VDa3, VDb, VDc, and VDe are each written as video data in a storage area within the frame buffer 18 corresponding to the position where they should be displayed. As described in Figure 5, images VDa1 to VDa3 are warning images superimposed on detected objects. Image VDb represents the vehicle's direction of travel, image VDc represents navigation information, and image VDe represents vehicle speed information. Furthermore, for example, images VDa1 to VDa3 and VDb are 3D graphics, while images VDc and VDe are 2D graphics.

In the example shown in Figure 13A, the size of the frame buffer 18 is equal to the size of the video display unit 11. In this case, each video VD written to the frame buffer 18, in other words, all video objects VD, are displayed in their original positions on the video display unit 11. On the other hand, in the example shown in Figure 13B, the size of the frame buffer 18 is larger than the size of the video display unit 11. In this case, only a portion of each video VD written to the frame buffer 18 is displayed on the video display unit 11.

In Figure 13B, for example, to accommodate cases where the display moves up and down due to pitching correction, video VDs are also placed in areas of the frame buffer 18 outside the display area of the video display unit 11. In this case, the magnitude of the processing load required for the preparation process is determined by how many video VDs are placed in the entire frame buffer 18, including areas outside the display area of the video display unit 11.

Figure 14 shows an example of items in Figure 6 that affect the preparation time for video data. Items that affect the preparation time include the number of video VDs and the display format for each video VD. Regarding the number of video VDs, the greater the number, the greater the processing load, and ultimately the longer the preparation time Tp. Items for the display format for each video VD include, for example, size, display position, design type, distortion correction, etc.

Regarding size, the larger the size, the longer the preparation time Tp. Regarding display position, the display position is usually closer to the front; in other words, the closer the display position, the larger the size, and therefore the longer the preparation time Tp. Design type includes, for example, types such as 2D/3D graphics and types such as whether or not there is a gradation. For example, the more complex the design, such as using 3D graphics with a large number of polygons and gradations, the longer the preparation time Tp. Furthermore, as mentioned in Figure 6, the preparation time Tp associated with distortion correction is shorter when distortion correction is performed using hardware and longer when distortion correction is performed using software.

Figure 15 is a flow diagram showing an example of the processing procedure for displaying an image in a HUD device according to embodiment 2. The HUD device 1 according to embodiment 2 is realized with the configuration shown in Figure 3A or 3B described above. Figure 15 shows a flow similar to that shown in Figure 6. That is, as in Figure 6, the control unit 20 determines the display content based on information about the vehicle acquired by the communication unit 16, i.e., the information acquisition unit, prepares video data based on the determined display content, and causes the video display unit 11 to display the image based on the prepared video data.

However, unlike the case of FIG. 6, in FIG. 15, the control unit 20 adjusts the display content as appropriate before determining the display content. Therefore, the video data preparation process (step S12A) in FIG. 15 is slightly different from that in FIG. 6. That is, in step S12A shown in FIG. 15, the control unit 20 first determines the display content by adjusting the display content (step S41A), and then creates video data based on the adjusted display content (step S42A). Thereafter, as in the case of FIG. 6, the control unit 20 writes the created video data to the frame buffer 18 (step S123A) and performs distortion correction using hardware or software processing (step S124A).

Figure 16 is a flow diagram showing an example of detailed processing details of the display content adjustment process (step S41A) in Figure 15. Details will be described later, but as a premise, the control unit 20 stores in advance in memory 17 or the like predicted time information that defines the relationship between the display content, for example, differences in display content, and the predicted time required to prepare the video data. Based on this premise, the control unit 20 first provisionally determines the display content based on the information about the vehicle acquired in step S11 in Figure 15 before determining the display content (step S411). Next, the control unit 20 predicts the preparation time Tp required to prepare the video data based on the provisionally determined display content (also referred to as provisional display content) based on the predicted time information (step S412).

Next, the control unit 20 determines whether the preparation time Tp predicted in step S412 is shorter than the predetermined control period Tc, in other words, the processing period (step S413). If the preparation time Tp is longer than the control period Tc (step S413: No), the control unit 20 changes the provisionally determined display content and then returns to step S412 to repeat the same process (step S414). As a result, the control unit 20 changes the provisionally determined display content, i.e., the provisional display content, so that the preparation time Tp is shorter than the control period Tc. On the other hand, if the preparation time Tp is shorter than the control period Tc (step S413: Yes), the control unit 20 determines the provisionally determined display content, i.e., the current provisional display content, as the final display content (step S415).

<How to estimate preparation time>
Fig. 17A is a diagram showing a specific example of the preparation time prediction method (step S412) in Fig. 16. Fig. 17B is a diagram showing a specific example different from Fig. 17A. As shown in Figs. 17A and 17B, control unit 20 pre-stores predicted time information 400 that defines the relationship between differences in display content and the predicted time required to prepare video data. Specifically, predicted time information 400 includes a basic time, 0.8 msec in this example, and coefficients corresponding to each item included in the display format for each video shown in Fig. 14.

That is, the predicted time information 400 includes a size coefficient C1, a display position coefficient C2, and a design type coefficient C3. The design type coefficient C3 includes a polygon count coefficient C31 and a gradation coefficient C32. The size coefficient C1 is a coefficient proportional to the size of the video VD. The size coefficient C1 is fixedly determined for each type of video VD, such as the video VDa1 showing a warning, the video VDb showing the direction of travel, and the video VDc showing navigation information shown in FIG. 13A.

The display position coefficient C2 is set to a larger value as the display position of the video VD gets closer. In this example, the display position is divided into three levels: close, medium, and far, and the display position coefficient C2 is set to 1.5 when the display position is close, 1.0 when it is medium, and 0.5 when it is close. The polygon count coefficient C31 is set to a larger value as the number of polygons used to render the video VD increases. In this example, the polygon count is divided into two levels: above a reference value and below the reference value, i.e., high and standard, and the polygon count coefficient C31 is set to 1.2 when the number of polygons is high and 1.0 when it is standard.

The gradation coefficient C32 is a coefficient that depends on whether or not a gradation is present. In this example, the gradation coefficient C32 is set to 1.1 if a gradation is present, and 1.0 if it is not present. The basic time and the values of each coefficient are determined taking into consideration the time required for the video data creation process (in other words, the drawing process) in step S42A shown in FIG. 15, the time required for the writing process to the frame buffer 18 in step S123A, and the time required for distortion correction in step S124A. Specific methods for determining these values include, for example, a method based on simulations or a method based on actual measurements.

The control unit 20 uses this predicted time information 400 to predict the preparation time Tp by multiplying the basic time by each coefficient. Specifically, the control unit 20 calculates the predicted time Tr required to prepare video data for each video VD by "basic time x C1 x C2 x C31 x C32". Note that the method for calculating the predicted time Tr is not limited to this, and in some cases it may be calculated by "basic time x C1 x C2 x C31". Furthermore, if there is another parameter that affects the predicted time Tr, that parameter may be taken into consideration when calculating the predicted time Tr.

Furthermore, when calculating the predicted time Tr, rather than calculating it each time from the basic time and coefficients, information indicating, for example, how much time it will take to prepare a particular video, at what size, in what position, and with what design, can be stored in advance as predicted time information. In this case, the time required for preparation must be stored for each video VD, but the predicted time Tr does not need to be calculated each time. The control unit 20 predicts the preparation time Tp by adding up the predicted times Tr for each video VD calculated using the above method.

As a specific example, assume that in step S411 shown in Figure 16, the display content is provisionally determined to display three videos VD1 to VD3 as shown in Figure 17A. Video VD1 has a size coefficient C1 of 5 and is displayed in a closer display position using 3D graphics with gradation. Video VD2 has a size coefficient C1 of 2 and is displayed in an intermediate display position using 2D graphics. Video VD3 has a size coefficient C1 of 10 and is displayed in a farther display position using 2D graphics with gradation.

In this case, the control unit 20 calculates the estimated time Tr required to prepare the video data for video VD1 as 7.9 msec = 0.8 x C1 (= 5) x C2 (= 1.5) x C31 (= 1.2) x C32 (= 1.1). Similarly, the control unit 20 calculates the estimated time Tr required to prepare the video data for video VD2 as 1.6 msec, and the estimated time Tr required to prepare the video data for video VD3 as 4.4 msec.

The control unit 20 then sums the calculated predicted times Tr for each of the videos VD1 to VD3 to predict the preparation time Tp as 13.9 [msec] = 7.9 + 1.6 + 4.4. For example, if the control period Tc is 16.6 ms, the predicted preparation time Tp will be shorter than the control period Tc. Therefore, in step S415 shown in FIG. 16, the control unit 20 determines the three videos VD1 to VD3, which have the same display format, as the display content.

As another specific example, as shown in Figure 17B, assume that the display content has been provisionally determined to display two videos VD4 and VD5 in addition to the three videos VD1 to VD3 shown in Figure 17A. Video VD4 has a size coefficient C1 of 8 and is displayed in the middle display position using 2D graphics. Video VD5 has a size coefficient C1 of 2 and is displayed in a nearby display position using 3D graphics.

In this case, the control unit 20 calculates the predicted time Tr required to prepare the video data for video VD4 as 6.4 [msec] = 0.8 x C1 (= 8) x C2 (= 1.0) x C31 (= 1.0) x C32 (= 1.0). Similarly, the control unit 20 calculates the predicted time Tr required to prepare the video data for video VD5 as 2.9 [msec]. The control unit 20 then sums the calculated predicted times Tr for each of videos VD1 to VD5 to predict the preparation time Tp as 23.2 [msec]. The predicted preparation time Tp is longer than the control cycle Tc. Therefore, the control unit 20 changes the display content, i.e., the provisionally determined display content, in step S414 shown in FIG. 16.

<How to change the display content>
18 is a diagram illustrating an example of a method for changing the display content (step S414) in FIG. 16 . The control unit 20 changes the number of video VDs or changes at least one of the display format of each video VD, i.e., size, display position, and design type, so that the preparation time Tp becomes shorter than the control period Tc. In this case, the control unit 20 stores, in advance in the memory 17, priority information 410 that defines the priority of each item to be changed, as shown in FIG. 18 . The control unit 20 then changes the display content while increasing the number of items to be changed based on the priority information 410 until the preparation time Tp becomes shorter than the control period Tc.

In the priority information 410 shown in FIG. 18, the number of video VDs is set to a lower priority than the display format of each video VD. In the example shown in FIG. 18, the priority when changing the display format of each video VD is, in descending order, design type, size, display position, and distortion correction. The control unit 20 first changes the design type of each video VD based on the priority information 410. Specifically, the control unit 20 changes, for example, 3D graphics to 2D graphics, and changes gradation to no gradation.

As a result, in FIG. 16, if the re-estimated preparation time Tp is shorter than the control period Tc (steps S412 and S413), the control unit 20 determines the display content using the changed video VD (step S415). On the other hand, if the re-estimated preparation time Tp is still longer than the control period Tc (steps S412 and S413), the control unit 20 reduces the size of each video VD in addition to the design type of each video VD. Specifically, the control unit 20, for example, predefines a lower limit value for the size of each video VD, and sequentially reduces the size of each video VD until the lower limit value is reached.

As a specific example, assume that the lower limit of the size for video VD3 in Figure 17B, specifically the lower limit of the size coefficient C1, is set to 8. In this case, the control unit 20 first changes the size of video VD3 to a size such that the size coefficient C1 is 9, and if Tp > Tc still holds, changes the size to a size such that the size coefficient C1 is 8. Note that in this case, depending on the type of video VD, it is also possible to set a video VD in which changes to the size coefficient C1 are not permitted, i.e., no lower limit is set.

In the same manner, the control unit 20 sequentially adds display position and distortion correction to the items to be changed based on the priority information 410 until Tp<Tc. When changing the display position, the control unit 20 reduces the size by moving the display position to the back. At this time, depending on the type of video VD, it may be possible to set video VDs whose display position is not allowed to be changed. Furthermore, when changing the distortion correction item, the control unit 20 may, for example, apply a predetermined simplified distortion correction, or may not perform distortion correction at all.

If Tp > Tc still holds even after changing the display format for each video VD in this way, the control unit 20 reduces the number of video VDs as a last resort. In doing so, the control unit 20 determines the order in which to reduce the number of video VDs, for example, based on the suppression table described in Figure 11C. As described above, the suppression table sets priorities so that the vehicle information 4 that contributes more to safe driving, in other words, the type of video VD, is not subject to deletion.

Figure 19 is a schematic diagram showing an example of the display content of the HUD device obtained as a result of changing the display content in Figure 18. The upper part of Figure 19 shows the display content before the change, which is the same as that in Figure 11A. That is, seven images VDa1 to VDa3 and VDb to VDe are displayed here. Images VDa1 and VDa2 are displayed so as to be superimposed on objects OB1 and OB2, respectively (people in this case). Image VDa3 is displayed so as to be superimposed on object OB3, a vehicle in this case. Images VDb, VDc, VDd, and VDe represent the direction of travel, navigation information, road signs, and speed information, respectively. Images VDa1 to VDa3 are assumed to use 3D graphics with gradation.

The bottom row in Figure 19 shows the display content after the change. Compared to the display content before the change, the images VDa1 to VDa3 have been replaced with images VDa1x to VDa3x, respectively. Images VDa1x to VDa3x use 2D graphics without gradation. Furthermore, the size of images VDa1x to VDa3x is slightly smaller than that of images VDa1 to VDa3.

Furthermore, in the example shown in FIG. 19, even when images VDa1x to VDa3x are used, the preparation time Tp is still longer than the control period Tc, so the image VDd representing the speed information is deleted, as shown at 405. In other words, in this example, the image VDd representing the speed information is deleted on the assumption that it contributes less to safe driving than the other images VD. By using this modification method, it is possible to modify the display content in a way that does not cause significant discomfort to users such as drivers, while maintaining the display content before the modification as much as possible from the perspective of safe driving.

Figure 20 is a timing chart for displaying video using the flows shown in Figures 15 and 16. Similar to Figure 7, Figure 20 shows the operations executed in the first through fourth processing or control cycles Tc[1] through Tc[4]. The control cycle Tc is, for example, 16.6 ms. As shown in Figure 20, if the display content is not changed, for example, the preparation time Tp[2a] in control cycle Tc[2] may be longer than the control cycle Tc[2]. In this case, as described in Figure 7, dropped frames will occur.

On the other hand, if the display content is changed, for example, the preparation time Tp[2b] in the control cycle Tc[2] can be made shorter than the control cycle Tc[2]. As a result, it is possible to prevent dropped frames. Note that the preparation time Tp also includes the time required for the display content adjustment process (step S41A) shown in Figures 15 and 16. For this reason, the time required for step S41A constitutes overhead time within the preparation time Tp, but is usually sufficiently small compared to the time required for steps S42, S123, and S124 that it can be considered negligible.

Furthermore, the method of the first embodiment described above monitors the actual preparation time Tp and performs control based on the monitoring results, i.e., feedback control, so there is a possibility that a situation like the preparation time Tp[2a] shown in Figure 20 may occur momentarily. On the other hand, the method of the second embodiment predicts the preparation time Tp and performs control, i.e., feedforward control, so ideally, a situation like the preparation time Tp[2a] shown in Figure 20 will not occur.

<Major Effects of Second Embodiment>
As described above, the method of the second embodiment can also achieve the same effects as those described in the first embodiment. That is, it is possible to suppress or prevent the occurrence of dropped frames and ensure a minimum display quality. Furthermore, it is possible to satisfy the required display specifications as much as possible within the range of the processing performance of the hardware and to flexibly respond to the display specifications. Furthermore, by predicting the preparation time Tp, it is less likely that the display content will be excessively changed or suppressed compared to the method of the first embodiment, thereby making it possible to further improve the display quality.

The method of embodiment 1 is particularly useful when there are many variables in the display specifications, such as with an initial product, making it difficult to create predicted time information 400 such as that shown in Figures 17A and 17B. On the other hand, the method of embodiment 2 is particularly useful when there are few variables in the display specifications, such as with a relatively mature product. Furthermore, since the method of embodiment 2 requires the creation of predicted time information 400, the method of embodiment 1 may be more useful in terms of flexibility or versatility when dealing with display specifications that vary greatly.

(Embodiment 3)
<Details of the control unit>
FIG. 21A is a schematic diagram showing an example of main processing contents of a control unit in a HUD device according to embodiment 3, and FIG. 21B is a supplementary diagram of FIG. 21A. FIG. 22A is a schematic diagram showing an example different from FIG. 21A, and FIG. 22B is a supplementary diagram of FIG. 22A. The HUD device 1 according to embodiment 3 is realized by the configuration shown in FIG. 3A or 3B described above. The control unit 20 superimposes a warning image on an object OB3, here a vehicle, as shown in image VDa3a in FIG. 21A and image VDa3b in FIG. 22A.

At this time, the control unit 20 determines the color or shape of the warning images VDa3a and VDa3b according to the distance to the object OB3. In the example of Figures 21A and 21B, the distance between the host vehicle 420 equipped with the HUD device 1 and the object OB3 located in front of the host vehicle 420 is 50 m. In this case, the control unit 20 determines the color of the image VDa3a to be green, for example. On the other hand, in the example of Figures 22A and 22B, the distance between the host vehicle 420 and the object OB3 is 10 m. In this case, the control unit 20 determines the color of the image VDa3b to be red, for example.

Figure 23 is a diagram showing an example of the configuration of a display setting table stored by the control unit in the HUD device according to embodiment 3. The control unit 20 stores, for example, a display setting table 415 as shown in Figure 23 in advance in the memory 17 or the like. The display setting table 415 defines the correspondence between the distance to the forward object OB3 and the color or shape of the warning image (color in this case). In this example, the color of the warning image is set to red when the distance to the object OB3 is less than 12 m, green when 32 m or more, and yellow when the distance is between 17 m and 27 m. Furthermore, the color for the intermediate distance ranges of 12 m to 17 m and 27 m to 32 m is set to an intermediate color as appropriate.

The display setting table 415 is set, for example, so that the closer the distance to the object OB3, the more the color or shape attracts the user's (e.g., the driver's) attention. This can contribute to safe driving. However, how a color or shape is perceived depends on the user's subjective opinion. For this reason, the display setting table 415 may be configured so that the user can select the color or shape in the table from multiple options through initial setup. Furthermore, the display setting table 415 may be configured so that the distance range can be set arbitrarily.

Figure 24A is a flow diagram showing an example of the processing content of the control unit in the HUD device according to embodiment 3. In Figure 24A, the control unit 20 executes the processing of steps S411 to S415 described in Figure 16 for embodiment 2. Prior to this processing, the communication unit 16, i.e., the information acquisition unit, acquires the distance between the host vehicle 420 and an object OB3 located in front of the host vehicle 420 as one piece of information related to the vehicle in step S11 shown in Figure 15.

By performing steps S411 to S415 shown in FIG. 24A, the control unit 20 determines the display content so that the preparation time Tp is shorter than the control cycle Tc, in other words, the processing cycle. The control unit 20 then determines whether the display content determined in step S415 includes a warning image to be superimposed on the object OB3 (step S416A). If the determined display content includes a warning image (step S416A: Yes), the control unit 20 references the distance information from the object OB3 acquired by the communication unit 16 (step S417) and updates the color or shape of the warning image based on the display setting table 415 (step S418A). Note that if multiple warning images are included, the update process is performed for each warning image.

In this way, in the flow shown in FIG. 24A, if the determined display content includes a warning image, the control unit 20 updates the color or shape of the warning image (step S418A) without re-estimating the preparation time Tp, i.e., without going through processing such as step S412. For this reason, it is desirable that the color or shape of the warning image for each distance be determined in advance so that the preparation time Tp remains the same as before the update, even when the color or shape of the warning image is updated.

Figure 24B is a flow diagram showing an example of processing content different from that shown in Figure 24A. Unlike the flow shown in Figure 24A, the flow shown in Figure 24B incorporates processing similar to steps S416A, S417, and S418A described in Figure 24A within the processing of steps S411 to S415 described in Figure 16, rather than after the processing of steps S411 to S415.

In FIG. 24B, after the control unit 20 tentatively determines the display content (step S411), it determines whether the tentatively determined display content includes a warning image to be superimposed on the object OB3 (step S416B). If the tentatively determined display content includes a warning image (step S416B: Yes), the control unit 20 references the distance information from the object OB3 (step S417) and determines the color or shape of the warning image based on the display setting table 415 (step S418B).

Then, the control unit 20 predicts the preparation time Tp required to prepare video data based on the provisionally determined display content (step S412). Note that if the provisionally determined display content does not include a warning video (step S416B: No), the control unit 20 proceeds directly to step S412. After step S412, the control unit 20 changes the provisionally determined display content by processing steps S413 to S415, as in the case of FIG. 16, so that the preparation time Tp is shorter than the control cycle Tc.

However, unlike the case in Figure 16, the control unit 20 changes the provisionally determined display content and then proceeds to step S416B (step S414). As a result, even if the color or shape of the warning image is changed by the processing of step S414, it can be restored to the correct state by the processing of steps S417 and S418B, and the preparation time Tp can be predicted (step S412).

Using the flow shown in Figure 24B allows the preparation time Tp to be predicted while reflecting changes in color or shape in the warning video, which may improve prediction accuracy compared to using the flow shown in Figure 24A. However, using the flow shown in Figure 24B may increase processing overhead because steps S416B, S417, and S418B must be processed each time a loop from step S414 to step S416B occurs. Therefore, from this perspective, it is more beneficial to use the flow shown in Figure 24A.

Furthermore, for example, the processing based on the display settings table 415 as described in steps S416A, S417, and S418A in FIG. 24A is not limited to the method of embodiment 2, and may be applied to the method of embodiment 1. In this case, the control unit 20 may execute the processing as described in steps S416A, S417, and S418A at the final stage when determining the display content based on the internal state in step S121 shown in FIG. 6, for example.

<Major Effects of Third Embodiment>
As described above, by using the method of embodiment 3, in addition to the various effects described in embodiments 1 and 2, it is possible to contribute to safer driving by changing the color or shape of the warning image depending on the distance from the object.

Furthermore, by using the methods of each embodiment, the user 6 can view various information necessary for driving, such as navigation information such as destination and speed, as well as alert information when an oncoming vehicle or pedestrian is detected, as an image through the windshield 3. Even if the display specifications change, the image can be viewed with a minimum display quality ensured. This makes it possible to provide a HUD device 1 that reduces the need for the user 6 to move their viewpoint and contributes to supporting safe driving. As a result, it becomes possible to prevent traffic accidents. Furthermore, it becomes possible to contribute to "3. Good health and well-being for all" of the Sustainable Development Goals (SDGs) advocated by the United Nations.

The invention made by the inventor has been specifically described above based on the embodiments, but the present invention is not limited to the above embodiments and can be modified in various ways without departing from the spirit of the invention. For example, the above-mentioned embodiments have been described in detail to clearly explain the invention, and are not necessarily limited to those that include all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

1...Head-up display (HUD) device, 2...Vehicle, 4...Vehicle information, 5...Display area, 6...User, 11...Video display unit, 16...Communication unit (information acquisition unit), 20...Control unit, 400...Predicted time information, 410...Priority information, 415...Display setting table, C1-C3, C31, C32...Coefficients, M1...Mirror (video light projection unit), ST0...Normal state, ST1...Return transition state, ST2...Suppression transition state, ST3...Suppression state, Tc...Control period, Tp...Preparation time, Tr...Predicted time, Tth1...First threshold time, Tth2...Second threshold time, TthD...Threshold duration, VD...Video

Claims (14)

A head-up display device mounted on a vehicle,
an information acquisition unit that acquires information about the vehicle;
an image display unit that displays an image and emits image light of the displayed image;
an image light projection unit that projects the image light emitted from the image display unit onto a display area, thereby allowing the projected image light to be viewed as a virtual image;
a control unit that determines display content based on information about the vehicle acquired by the information acquisition unit, prepares video data based on the determined display content, and displays video based on the prepared video data on the video display unit; and
Equipped with
the control unit, before determining the display content, predicts a preparation time required to prepare the video data based on the provisionally determined display content based on predicted time information that defines a relationship between the display content and a predicted time required to prepare the video data, and, if the preparation time required to prepare the video data of the provisionally determined display content before determining the display content is longer than a predetermined processing cycle, changes the provisionally determined display content so that the preparation time is shorter than the processing cycle .
Head-up display device. The head-up display device according to claim 1 ,
The items for changing the provisionally determined display content include the number of images or the display format for each image,
The items for changing the display format for each image include at least one of size, display position, and design type.
Head-up display device. The head-up display device according to claim 2 ,
the control unit pre-stores priority information that defines a priority for each of the items, and changes the provisionally determined display content while increasing the number of items to be changed based on the priority information until the preparation time becomes shorter than the processing cycle.
Head-up display device. The head-up display device according to claim 3 ,
the items when changing the provisionally determined display content include the number of images and a display format for each image;
In the priority information, the number of images is set to a lower priority than the display format of each image.
Head-up display device. The head-up display device according to claim 2 ,
the predicted time information includes a basic time and a coefficient according to the item included in the display format of each of the videos;
the control unit predicts the preparation time by multiplying the basic time by the coefficient.
Head-up display device. The head-up display device according to claim 1,
the information acquisition unit acquires a distance between the vehicle equipped with the head-up display device and an object present in front of the vehicle;
When the control unit superimposes a warning image on the object, the control unit determines a color or a shape of the warning image depending on a distance from the object.
Head-up display device. The head-up display device according to claim 6 ,
the control unit determines the display content so that the preparation time is shorter than the processing cycle, and then, if the determined display content includes the warning video, updates the color or shape of the warning video without re-estimating the preparation time.
Head-up display device. A method for processing video data for a head-up display device mounted on a vehicle, comprising:
obtaining information about the vehicle;
determining display content based on the acquired information about the vehicle, preparing video data based on the determined display content, and displaying a video based on the prepared video data on a video display unit of the head-up display device;
predicting a preparation time required for preparing the video data based on the provisionally determined display content based on predicted time information that defines a relationship between the display content and a predicted time required for preparing the video data, before determining the display content; and, if the preparation time required for preparing the video data of the provisionally determined display content before determining the display content is longer than a predetermined processing cycle, changing the provisionally determined display content so that the preparation time is shorter than the processing cycle .
How video data is processed. 9. The video data processing method according to claim 8 ,
The items for changing the provisionally determined display content include the number of images or the display format for each image,
The items for changing the display format for each image include at least one of size, display position, and design type.
How video data is processed. 10. The video data processing method according to claim 9 ,
storing in advance priority information that defines a priority for each of the items, and changing the provisionally determined display content while increasing the number of items to be changed based on the priority information until the preparation time becomes shorter than the processing cycle;
How video data is processed. 11. The video data processing method according to claim 10 ,
the items when changing the provisionally determined display content include the number of images and a display format for each image;
In the priority information, the number of images is set to a lower priority than the display format of each image.
How video data is processed. 10. The video data processing method according to claim 9 ,
the predicted time information includes a basic time and a coefficient according to the item included in the display format of each of the videos;
predicting the preparation time by multiplying the basic time by the coefficient;
How video data is processed. 9. The video data processing method according to claim 8 ,
acquire a distance between the vehicle equipped with the head-up display device and an object present in front of the vehicle;
When a warning image is superimposed on the object, a color or a shape of the warning image is determined according to a distance from the object.
How video data is processed. 14. The video data processing method according to claim 13 ,
after determining the display contents so that the preparation time is shorter than the processing cycle, if the warning image is included in the determined display contents, updating the color or shape of the warning image without re-estimating the preparation time;
How video data is processed.