JP7532053B2 - Object presentation device and object presentation method - Google Patents

Description

The present invention relates to an object presentation device and an object presentation method.

There is known a vehicle information display device that, when an obstacle is detected ahead of the vehicle, estimates the visibility of the obstacle and displays a display image created according to the visibility of the obstacle in the driver's field of vision, thereby providing the driver with appropriate information display (for example, Patent Document 1).

JP 2009-248721 A

However, the technology described in Patent Document 1 has the problem that even if an image is displayed to notify the occupant of the presence of an obstacle, if the occupant's attention is not directed to the displayed image, the presence of the obstacle may not be recognized.

The problem that this invention aims to solve is to provide an object presentation device that can prevent occupants from noticing objects that require their attention.

The present invention solves the above problem by detecting an object that a vehicle occupant needs to pay attention to, transmitting an object image showing the object to a display device, acquiring occupant posture information including directional information of the occupant's line of sight or directional information of the occupant's face orientation, estimating whether or not the occupant recognizes the object image based on the occupant posture information and the display position at which the object is displayed on the display screen of the display device, and if it is estimated that the occupant does not recognize the object image, executing image display control to highlight the object image, and if it is estimated that the occupant recognizes the object image after executing the image display control, determining whether or not the object is behind an obstruction as seen by the occupant, and if it is determined that the object is behind an obstruction as seen by the occupant, hiding the object image and displaying a transparent image with adjusted transparency of a real-life image of the object, and if it is determined that the object is not behind an obstruction as seen by the occupant, hiding the object image and displaying the real-life image of the object, and the real-life image of the object is acquired via a communication device from an external information acquisition device that is located outside the vehicle and acquires an image including the object by a camera .

The present invention can prevent occupants from noticing objects that require their attention.

FIG. 1 is a block diagram showing an embodiment of an object presentation device according to the present embodiment. 5 is a flowchart showing a procedure of display control of an object image executed in the object presentation device in the present embodiment. FIG. 13 is a diagram illustrating an example of control for highlighting an object image. 13A and 13B are diagrams illustrating an example of display image control after a user recognizes an object image.

The following describes an embodiment of the present invention with reference to the drawings.

In this embodiment, as shown in FIG. 1, the object presentation device 12 is mounted on the vehicle 10 and is communicatively connected to the display terminal 11 (e.g., an AR head-mounted display), the in-vehicle sensor 13, and the external information acquisition device 30 (other vehicles and road facilities). The display terminal 11 is, for example, an AR head-mounted display worn on the head of a user 40 who is an occupant of the vehicle 10, and the user 40 can view an augmented reality (AR) in which an image is displayed on a real scene via the display terminal 11. The user 40 is, for example, the driver of the vehicle 10. In this embodiment, the object presentation device 12 is mounted on the vehicle 10 and is connected to the display terminal 11 and the in-vehicle sensor 13 by wires, but is not limited thereto. The object presentation device 12 may be installed outside the vehicle 10 and connected to the display terminal 11 and the in-vehicle sensor 13 by wireless.

The display terminal 11 includes at least a sensor group 100 and an output unit 110. The sensor group 100 includes a camera 101, an acceleration sensor 102, and a gyro sensor 103, and detects the head direction or line of sight of the user 40. For example, the camera 101 captures an image of the user 40's eyeball, and the captured image is analyzed to detect the line of sight. The acceleration sensor 102 and the gyro sensor 103 measure the movement of the head to detect the direction of the head. The attitude information regarding the head direction and line of sight of the user 40 acquired by the sensor group 100 is output to the passenger attitude information acquisition unit 123 of the object presentation device 12. The output unit 110 also includes a monitor 111 and a speaker 112, and displays an image on the monitor 111 and outputs sound from the speaker. The monitor 111 displays a display image based on the information of the display image output from the display image transmission unit 124 of the object presentation device 12. For example, the monitor 111 is a transmissive display that displays a display image superimposed on the user's field of vision.

The object presentation device 12 includes a communication device 120 and a controller 121. The communication device 120 is wirelessly connected to an external information acquisition device 30 (other vehicles and road facilities) that exists outside the vehicle 10, and acquires external environment information. The external environment information includes objects on the driving route of the vehicle 10 that the user 40 needs to be careful of. The objects may affect the driving of the vehicle 10, and include obstacles that may approach on the driving route of the vehicle 10 and guidance displays that provide necessary information to the occupants of the vehicle 10. For example, the objects include pedestrians, other vehicles, bicycles, walls (including guardrails, tunnel walls, buildings, utility poles, and toll booths), road signs, road markings, and traffic signals. The objects may also include guidance information that is useful to the occupants. For example, a guide board showing the distance to a parking area.

The controller 121 acquires user posture information and external environment information, generates a display image to be displayed to the user 40, and transmits it to the display terminal 11. The controller 121 is equipped with a computer having hardware and software, and this computer includes a ROM (Read Only Memory) that stores programs, a CPU (Central Processing Unit) that executes the programs stored in the ROM, and a RAM (Random Access Memory) that functions as an accessible storage device. The controller 121 has functional blocks including an object detection unit 122, an occupant posture information acquisition unit 123, a display image transmission unit 124, and an attention calling control unit 125, and executes each function by cooperation between the software for realizing each of the above functions or executing each process and the hardware.

The controller 121 acquires the external environment information acquired by the external information acquisition device 30 through the communication device 120 by the object detection unit 122, and acquires the external environment information acquired by the on-board sensor 13 from the on-board sensor 13. From the acquired external environment information, the controller 121 detects whether or not there is an object that the user 40 needs to pay attention to within a predetermined range from the vehicle 10. As a method for detecting the object, for example, the object is detected by image recognition. The controller 121 extracts the characteristics of the object from the image data acquired by the camera 131 of the on-board sensor 13 or the camera 320 of the external information acquisition device 30 by the object detection unit 122, and identifies the type of the object. The detection of the object may be performed by detecting multiple objects at once.

The controller 121 may detect, by the object detection unit 122, an object with low visibility, for example, an object with low visibility due to the external environment or an object that is partially or completely blocked from the occupant's field of vision by an obstruction. An object with low visibility is an object with a small visible area or an object whose characteristics are difficult to see in relation to the surrounding environment. An object with a small visible area is an object that is small in size, is far from the vehicle 10, or has a characteristic part blocked by an obstruction. An object whose characteristics are difficult to see in relation to the surrounding environment is an object whose color is similar to the surrounding environment, or an object that is in an external environment that reduces visibility. Since an object with low visibility is particularly likely to be overlooked by the occupant, detecting an object with low visibility and displaying an object image can prevent the occupant from not recognizing the object. Examples of external environments that reduce visibility include external environments such as night, fog, and rain. Methods for detecting an object with low visibility include, for example, a method of detecting an object with low visibility from the difference between an image captured by a visible light camera and an image captured by a far-infrared camera, and a method of detecting an object as an object with low visibility when an object is detected that can be recognized by the camera 320 of the external information acquisition device 30 but cannot be recognized by the camera 131 of the in-vehicle sensor 13. The controller 121 also detects an object as having low visibility when the object is partially or completely blocked from the viewpoint of the user 40 by an obstruction and visibility is low. For example, when an object behind the obstruction cannot be detected from the image acquired by the camera 131 of the in-vehicle sensor 13 because the obstruction is in front, if the object can be detected by the camera 320 of the external information acquisition device 30, the object is detected as an object with low visibility.

The controller 121 acquires posture information related to the line of sight and head orientation of the user 40 acquired by the sensor group 100 from the display terminal 11 via the occupant posture information acquisition unit 123.

The controller 121 also generates an object image by the display image transmission unit 124 and transmits the generated object image to the display terminal 11. At this time, the object image includes information on the display position of the object image displayed on the monitor 111 of the display terminal 11. In this embodiment, the type of image representing the object is selected according to the type of object, and the size, color, brightness, etc. of the selected image are determined to generate the object image. For example, when a light vehicle is detected, the controller 121 recognizes the light vehicle as a "vehicle", selects an image representing the "vehicle", and generates an object image of the light vehicle. In addition, the method of acquiring the display position of the object image displayed on the monitor 111 of the display terminal 11 is to first identify the actual position of the object. Next, the position on the monitor 111 corresponding to the identified position of the object is acquired as the display position. Specifically, the display position on the monitor 111 is acquired by associating three-dimensional coordinates representing the actual position of the object with two-dimensional coordinates on the monitor 111.

The controller 121 also estimates whether the user 40 recognizes the object image based on the posture information of the user 40 acquired by the attention calling control unit 125, that is, posture information related to the direction of the user's line of sight or the direction of the head, and position information of the display position of the object image displayed on the monitor 111. For example, the controller 121 determines, by the attention calling control unit 125, whether the direction match time during which the direction of the user's line of sight or the direction of the head matches the direction of the position at which the object image is displayed is less than a predetermined time. If it is determined that the time is less than the predetermined time, it estimates that the user 40 does not recognize the object image. If it is determined that the time is not less than the predetermined time, that is, that the time is equal to or longer than the predetermined time, it estimates that the user 40 recognizes the object image.

Furthermore, when the attention alert control unit 125 estimates that the user 40 recognizes the object image, the controller 121 maintains the estimation result that the user 40 recognizes the object image. Specifically, after the estimation, if the direction of the user 40's gaze or the direction of the head changes to a direction different from the direction of the display position of the object image, even if the direction match time becomes less than a predetermined time, the controller 121 does not estimate that the user 40 does not recognize the object image, and maintains the estimation result that the user 40 recognizes the object image. This is because, once the user 40 recognizes the object image, it is considered that the user 40 notices and recognizes the object even if the user 40 is looking elsewhere.

Furthermore, when the attention calling control unit 125 estimates that the user 40 does not recognize the object image, the controller 121 executes control to call the user 40's attention to the object image. Specifically, the controller 121 executes control to highlight the object image displayed to the user 40 via the display terminal 11. For example, the controller 121 outputs a control signal to the display image transmission unit 124 to generate a highlighted image. The highlighted image includes an image in which the color of the object image is changed to a conspicuous color such as red, an image with increased brightness, an image in which the display position is changed, and an image displayed in a superimposed manner. The controller 121 may also transmit a control signal to the display terminal 11 to cause the object image to blink using the attention calling control unit 125. Note that, as a control to highlight the object image, audio data may be generated to guide the direction of the line of sight or the direction of the face of the user 40 toward the display position of the object image and transmitted to the display terminal 11. The controller 121 executes control to call attention to the object image using at least one of these controls.

The on-board sensor 13 is a group of sensors mounted on the vehicle 10, and acquires information about the driving environment including the presence of objects around the vehicle 10 (front, side, and rear). The on-board sensor 13 includes a camera 131, an ultrasonic sensor 132, a radar 133, and a GPS 134. The camera 131 is a camera equipped with an image sensor such as a CCD. The camera of this embodiment is installed on the vehicle 10, captures an image of the driving environment around the vehicle 10, and acquires image data including vehicles and pedestrians around the vehicle 10. The camera 131 includes an ultrasonic camera, an infrared camera, and the like. The ultrasonic sensor 132 and the radar 133 acquire the relative distance between the vehicle 10 and the object. The radar 133 can be a laser radar, a millimeter wave radar (LRF, etc.), or a LiDAR (Light Detection and Ranging) unit. The on-board sensor 13 may employ one or more cameras, an ultrasonic sensor 132, and a radar 133. The GPS 134 is a GPS device that detects the current position of the vehicle 10. The GPS 134 acquires the vehicle's position information by receiving radio waves transmitted from multiple satellite communications with a receiver. The GPS 134 can also detect changes in the vehicle's position information by periodically receiving radio waves transmitted from multiple satellite communications.

Things that the on-board sensor 13 detects include lane boundaries, center lines, road markings, medians, guard rails, curbs, highway sidewalls, road signs, traffic lights, crosswalks, construction sites, accident sites, traffic restrictions, automobiles (other vehicles) other than the vehicle itself, motorcycles, bicycles, and pedestrians. In addition, things that the on-board sensor 13 detects include objects that the user 40 needs to pay attention to while driving.

The external information acquisition device 30 is a device provided on other vehicles or roadside facilities, and includes a communication device 31 and an external sensor group 32. The external sensor group 32 includes a camera 320, an ultrasonic sensor 321, a radar 322, and a GPS 323, similar to the in-vehicle sensor 13, and acquires information on the surrounding driving environment including objects. The external sensor group 32 constituting the external information acquisition device 30 may be configured differently from the in-vehicle sensor 13. The communication device 31 transmits information acquired by the external sensor group 32 to the communication device 120 of the object presentation device 12. For example, the other vehicle acquires surrounding environmental information by a sensor while moving, or the roadside facilities installed on or near the road acquire surrounding environmental information by a sensor. This allows environmental information that cannot be acquired by the in-vehicle sensor 13 of the vehicle 10 to be transmitted to the object presentation device 12. For example, even if the vehicle's 10 onboard sensor 13 cannot detect an object behind a wall at a corner, the controller 121 can obtain the location of the invisible object by having the external information acquisition device 30 detect the object.

Next, the highlighting control of the display image displayed to the user 40 according to this embodiment will be described with reference to FIG. 2. FIG. 2 is a flowchart for executing the highlighting control of the display image displayed to the user 40. In this embodiment, when the user 40 turns on the activation switch to start using the in-car AR system, the highlighting control starts from step S201 based on the flowchart.

In step S201, the controller 121 acquires external environment information about the surroundings of the vehicle 10. Specifically, the controller 121 acquires external environment information acquired from the on-board sensor 13 and the external information acquisition device 30, respectively.

In step S202, the controller 121 detects whether there is an object that the user 40 needs to pay attention to based on the external environment information acquired in step S201. Specifically, the controller 121 extracts the characteristics of the object from the image data acquired by the camera by image recognition, and identifies the type of the object. Examples of the type include automobiles, pedestrians, bicycles, buildings, signs, and traffic lights. The object includes obstacles (pedestrians, automobiles, etc.) that exist within a predetermined range from the vehicle 10 and may approach the vehicle, and traffic lights at the intersection if the object is near an intersection. In addition, the controller 121 may detect objects that are low in visibility due to the external environment, and objects that are partially or completely blocked from the field of view of the user 40 by an obstruction. If an object that the user 40 needs to pay attention to is detected, the process proceeds to step S203. If an object that the user 40 needs to pay attention to is not detected, the process returns to step S201 and acquires external environment information again. In this embodiment, while the vehicle 10 is traveling, external environment information is constantly acquired and object detection is repeated.

In step S203, the controller 121 displays an object image showing the object. Specifically, the controller 121 selects a display image set in advance for each type of object identified in step S202, and generates the display image as an object image. Next, the controller 121 transmits the generated object image to the display terminal 11 worn by the user 40 together with position information regarding the position of the object image displayed on the monitor 111. At this time, the position at which the object image is displayed is acquired by determining the corresponding two-dimensional coordinate position from the actual position of the object. Then, when the display terminal 11 acquires the object image and the position information of the display position at which the object image is displayed from the controller 121, the object image is displayed at the display position on the monitor 111 of the output unit 110. At this time, for example, even if the object is present behind a wall or the like and is not directly visible in the field of view of the user 40, if there is no wall or the like, the object is displayed at a position where it can be seen in the field of view of the user 40.

In step S204, the controller 121 acquires posture information of the user 40. Specifically, the controller 121 acquires posture information including the line of sight and face orientation of the user 40 acquired by the sensor group 100 of the display terminal 11 worn by the user 40.

In step S205, the controller 121 estimates whether the user 40 recognizes the object image. First, the controller 121 compares the direction of the object image position with the line of sight or head direction of the user 40 based on the posture information acquired in step S204 and the position information of the display position where the object image is displayed. Then, it is determined whether the direction match time during which the line of sight or head direction of the user 40 matches the direction of the object image position is less than a predetermined time. Then, if it is determined that the direction match time is not less than the predetermined time, that is, is equal to or longer than the predetermined time, it is estimated that the user 40 recognizes the object image. Also, if it is determined that the direction match time is less than the predetermined time, it is estimated that the user 40 does not recognize the object image. If it is estimated that the user 40 recognizes the object image, the process proceeds to step S209. If it is estimated that the user 40 does not recognize the object image, the process proceeds to step S206.

The predetermined time, which is set as a threshold for the direction match time during which the direction of the user's 40 gaze or head direction matches the direction of the object image position, may be changed depending on the type of object. For example, the predetermined time is set longer when the object type is another vehicle or a pedestrian than when the object type is other types. Also, the predetermined time is set shorter when the object type is a guide sign than when the object type is other types.

In step S206, the controller 121 executes control to highlight the object image. For example, an emphasis image that highlights the display of the object image is generated and displayed. Examples of emphasis images include an image in which the size of the object image is increased, an image in which the object image is changed to a conspicuous color such as red, and an image in which the brightness of the object image is increased. When the controller 121 generates the emphasis image, it outputs the emphasis image to the output unit 110 of the display terminal 11. The display terminal 11 that has acquired the emphasis image displays the emphasis image at the position on the monitor 111 where the object image is displayed. In addition to generating an emphasis image, the object image may be displayed in a blinking manner. In this case, the controller 121 outputs a control signal for executing the blinking display of the object image to the output unit 110. Then, the output unit 110 displays the object image in a blinking manner based on the control signal.

3 is a diagram showing an example of executing the highlighting control of the object image. FIG. 3 shows an example of a scene seen in the field of view of the user 40 through the display terminal 11 in a scene where another vehicle 302, which is an object, exists behind the wall 301. As shown in FIG. 3, the other vehicle 302 is partially blocked by the wall 301. When the other vehicle 302 is detected as an object, the object image 303 is displayed at the position of the object on the monitor 111. At this time, as shown in FIG. 3, if the other vehicle 302, which is the object, is blocked by a wall or the like, the object image 303 is superimposed on the wall or the like. Furthermore, in a state where the object image 303 is displayed, an estimation is made as to whether the user 40 recognizes the object image 303, and if it is estimated that the user 40 does not recognize the object image 303, the highlight image 304 is displayed at the position of the object image on the monitor. At this time, it is not limited to displaying the highlight image 304, and the object image 303 may be displayed in a blinking manner. Note that in FIG. 3, a scene in which the other vehicle 302 is blocked by a wall 301 is given as an example, but this is not limited thereto, and the other vehicle, which is the target object, may be in a location where there is no blocking object.

In step S207, the controller 121 acquires posture information of the user 40. Specifically, the controller 121 acquires posture information including the line of sight and face orientation of the user 40 acquired by the display terminal 11 worn by the user 40, in the same manner as in step S204.

In step S208, the controller 121 estimates whether the user 40 recognizes the object image. The estimation method is the same as in step S205. In step S208, in order to determine whether the direction of the line of sight or the direction of the face of the user 40 has changed toward the position of the object image as a result of highlighting the object image in step S206, it is estimated again whether the user 40 recognizes the object image. If it is estimated that the user 40 recognizes the object image, the process proceeds to step S209. If it is estimated that the user 40 does not recognize the object image, the process returns to step S206. In step S206, the controller 121 again highlights the object image using another highlighting method, or guides the direction of the line of sight or the direction of the head of the user 40 toward the position of the object image by voice.

In step S209, the controller 121 notifies the user 40 that it has confirmed the attention state of the user 40. The attention state is a state in which the user is aware of an object that requires attention. The controller 121 indicates the attention state, i.e., that the user 40 has confirmed that he or she is aware of the object, for example, by illuminating the area around the object or the image of the object.

In step S210, the controller 121 determines whether or not the object is behind the obstruction as viewed by the user 40, based on the external environment information acquired by the in-vehicle sensor 13 and the external information acquisition device 30. At this time, the controller 121 determines that the object is behind the obstruction, regardless of whether the part of the object that is obstructed by the obstruction as viewed by the user 40 is part or all of the object. If it is determined that the object is behind the obstruction, the process proceeds to step S211. If it is not determined that the object is behind the obstruction, the process proceeds to step S212.

In step S211, the controller 121 acquires a live-action image of the object, and displays a see-through image in which the transparency of the live-action image of the object has been adjusted on the monitor 111. That is, in step S211, the see-through image is displayed in a manner that complements part or all of the object that is not visible to the user 40, thereby improving the visibility of the object.

In step S212, controller 121 hides the object image. Specifically, controller 121 outputs a control signal to output unit 110 to stop displaying the object image, and output unit 110 hides the object image based on the control signal. In step S212, user 40 recognizes the presence of the object and the object is in a position where it can be directly viewed, so by hiding the displayed object image, user 40 can directly view the object.

Figure 4 is a diagram showing an example of the presentation control of the confirmation of the user's attention state and the display control of the object image from step S209 to step S212. In Figure 4, a scene is shown in which it is estimated that the user 40 recognizes the object image after the object image is superimposed and displayed at the position of the object on the monitor 111. When it is estimated that the user 40 recognizes the object, the controller 121 presents to the user 40 that the object presentation device 12 has confirmed that the user 40 recognizes the object. For example, as shown by the light emission 305 in STEP 209 (1) and STEP 209 (2), the periphery of the object image 303 is illuminated. In Figure 4, STEP 209 (1) is an example of S209 in Figure 2, and shows a situation in which the periphery of the object image 303 is illuminated in a scene in which another vehicle, which is the object, is blocked by a wall 301. STEP 209 (2) is an example of S209 in FIG. 2, and shows a situation in which the surroundings of the object image 303 are illuminated in a scene where another vehicle, which is the object, is present in a place without any obstructions. Next, if there is a wall 301 in front of the object (STEP 209 (1)), the display image is changed from the object image 303 to the transparent image 306, and the transparent image is displayed at the position of the object on the monitor 111 (STEP 211). The transparent image 306 is an image that complementarily displays a part that the user 40 cannot see directly due to an obstruction such as a wall. Also, if there is no wall 301 in front of the object (STEP 209 (2)), the object image 303 is hidden, and the actual object or a live-action image is displayed (STEP 212).

In step S213, the controller 121 determines whether or not use of the in-vehicle AR system has been terminated. For example, if the user 40 turns off the activation switch to terminate use of the in-vehicle AR system, the highlighting control is terminated. If it is not determined that use of the in-vehicle AR system has been terminated, the process returns to step S201, and the flow is repeated thereafter until use of the in-vehicle AR system is terminated.

In addition, in this embodiment, when object images of multiple objects detected by the object detection unit 122 are displayed, the controller 121 may select an object with a high importance from among the multiple objects according to the importance of the object, and determine whether the user 40 recognizes the selected object. The importance is the degree of need for the user to pay attention to it. The importance is set in advance for each type of object. For example, when the type of object is a "pedestrian," "bicycle," or "other vehicle," the importance is set to "high," and when the type of object is a "signboard," etc., the importance is set to "low." The setting of the importance is not limited to two levels, "high" and "low," and may be divided into two or more levels, such as three levels, "high," "medium," and "low." In this embodiment, the controller 121 first identifies the type of object and determines the importance of the object based on the importance set for each type of object. Next, the controller 121 selects the object with the highest importance based on the importance of each of the multiple objects whose object images are displayed, and estimates whether the user 40 recognizes the object image of that object. If the controller 121 estimates that the user 40 does not recognize the object, it highlights the object. In this way, if the user does not recognize an object that requires high attention, the object is highlighted, thereby preventing the occupant from not recognizing the object that requires high attention.

In this embodiment, when multiple objects are detected by the object detection unit 122, the controller 121 may determine the importance of each of the multiple objects and select an object for which an object image is to be generated according to the importance. Specifically, when multiple objects are detected, the controller 121 identifies the type of object and determines the importance of the object based on the importance set for each type of object. Then, the controller 121 generates an object image of an object belonging to the type with the highest importance among the detected objects according to the importance of each of the multiple objects, and transmits the object image to the display terminal 11. The controller 121 may also select an object for which an object image is to be generated in order of importance according to the importance of each of the multiple objects. For example, the controller 121 may be set in advance to generate object images of a predetermined number of objects, and when multiple objects are detected, generate object images for the predetermined number of objects in order of importance.

As described above, in this embodiment, an object that is within a predetermined range from the vehicle and that the vehicle occupant needs to pay attention to is detected, an object image showing the detected object is transmitted to the display device, occupant posture information including directional information of the occupant's line of sight or directional information of the occupant's face orientation is acquired, and based on the acquired occupant posture information and the display position at which the object image is displayed on the display screen of the display device, it is estimated whether the occupant recognizes the object image on the display screen, and if it is estimated that the occupant does not recognize the object image, image display control is executed to highlight the object image. This makes it possible to prevent the occupant from not recognizing objects that require attention.

In addition, in this embodiment, if it is detected based on the occupant posture information that the direction of the occupant's line of sight or the direction of the occupant's face matches the direction of the display position for a period of time that is less than a predetermined period of time, it is presumed that the occupant does not recognize the object image. This makes it possible to determine that the object image is not recognized by the occupant.

In addition, in this embodiment, if the direction match time is detected to be equal to or longer than a predetermined time, it is presumed that the occupant recognizes the object image, and even if the direction of the occupant's line of sight or the direction of the occupant's face changes to a direction different from the direction of the display position after it is presumed that the occupant recognizes the object image, the presumption that the occupant recognizes the object image is maintained. As a result, once the occupant recognizes the object image, it can be determined that the occupant recognizes the object image even if the direction of the occupant's line of sight or the direction of the face changes to a direction different from the direction of the display position of the object image.

In addition, in this embodiment, the image display control includes control to generate and display an emphasis image that emphasizes the display of the object image, or control to blink the object image. This makes it possible to guide the direction of the passenger's line of sight or the direction of the face in the direction of the display position of the object image.

In addition, in this embodiment, the object includes at least one of an object that is low in visibility due to the external environment of the vehicle and an object that is partially or completely blocked from the occupant's field of vision by an obstruction. This allows the occupant to recognize even an object that is low in visibility and difficult to recognize without overlooking it.

Furthermore, in this embodiment, audio data is generated to guide the direction of the occupant's line of sight or the direction of the occupant's face in the direction of the display position, and the audio data is transmitted to the display device. This makes it possible to guide the direction of the occupant's line of sight or the direction of the face in the direction of the display position of the object image, even if the object is not visually recognized by the occupant.

In this embodiment, an object image is displayed on the monitor 111, and it is estimated whether or not the user recognizes the displayed object image. However, this is not limited to the above, and for example, it may be possible to estimate whether or not the user recognizes the object after the object is detected in step S202. Then, when it is estimated that the user does not recognize the object, the object image may be displayed.

The above-described embodiments are described to facilitate understanding of the present invention, and are not described to limit the present invention. Therefore, each element disclosed in the above embodiments is intended to include all design modifications and equivalents that fall within the technical scope of the present invention.

REFERENCE SIGNS LIST 11 display terminal 100 sensor group 110 output unit 12 object presentation device 120 communication device 121 controller 122 object detection unit 123 occupant posture information acquisition unit 124 display image transmission unit 125 attention call control unit 13 in-vehicle sensor 30 external information acquisition device 31 communication device 32 external sensor group

Claims (7)

an object detection unit that detects an object that is present within a predetermined range from the vehicle and that requires the attention of an occupant of the vehicle;
a display image transmission unit that transmits an object image showing the object detected by the object detection unit to a display device;
an occupant posture information acquisition unit that acquires occupant posture information including gaze direction information of the occupant or face direction information of the occupant;
an attention drawing control unit that draws the occupant's attention to the object image displayed by the display device,
The attention control unit is
estimating whether or not the occupant recognizes the object image on the display screen based on the occupant posture information acquired by the occupant posture information acquisition unit and a display position at which the object image is displayed on the display screen of the display device;
When it is estimated that the occupant does not recognize the object image, an image display control is executed to highlight the object image;
After executing the image display control, if it is estimated that the occupant recognizes the object image, it is determined whether or not the object is behind an obstruction as viewed from the occupant;
When it is determined that the object is behind the obstruction as viewed from the occupant, the object image is hidden and a transparent image in which the transparency of the actual image of the object is adjusted is displayed;
When it is determined that the object is not behind the obstruction as viewed from the occupant, the image of the object is not displayed and an actual image of the object is displayed ;
An object presentation device in which the actual image of the object is acquired via a communication device from an external information acquisition device that is located outside the vehicle and acquires an image including the object using a camera . The object presentation device according to claim 1 ,
The attention control unit is
An object presentation device that estimates that the occupant does not recognize the object image when it detects, based on the occupant posture information, that the direction of the occupant's line of sight or the direction of the occupant's face matches the direction of the display position for a directional agreement time that is less than a predetermined time. The object presentation device according to claim 2,
The attention control unit is
When the direction matching time is detected to be equal to or longer than a predetermined time, it is estimated that the occupant recognizes the object image;
An object presentation device that maintains the estimation result that the occupant recognizes the object image even if the direction of the occupant's line of sight or the direction of the occupant's face changes to a direction different from the direction of the display position after it is estimated that the occupant recognizes the object image. The object presentation device according to claim 1 ,
The image display control includes control of generating and displaying an emphasis image that emphasizes the display of the object image, or control of blinking the object image. The object presentation device according to claim 1 ,
The object presentation device includes at least one of an object that is poorly visible due to an external environment of the vehicle and an object that is partially or completely blocked from the occupant's field of vision by an obstruction. The object presentation device according to claim 1 ,
The attention control unit is
generating voice data for guiding the direction of the line of sight of the occupant or the direction of the face of the occupant toward the direction of the display position;
The object presentation device transmits the audio data to the display device. An object presentation method performed using an object presentation device, comprising:
The object presentation device includes:
Detecting an object that is within a predetermined range of the vehicle and requires the attention of an occupant of the vehicle;
transmitting an object image indicative of the detected object to a display device;
Acquire occupant posture information including line of sight direction information of the occupant or face direction information of the occupant,
estimating whether or not the occupant recognizes the object image on the display screen based on the acquired occupant posture information and a display position at which the object image is displayed on the display screen of the display device;
When it is estimated that the occupant does not recognize the object image, an image display control is executed to highlight the object image;
After executing the image display control, if it is estimated that the occupant recognizes the object image, it is determined whether or not the object is behind an obstruction as viewed from the occupant;
When it is determined that the object is behind the obstruction as viewed from the occupant, the object image is hidden and a transparent image in which the transparency of the actual image of the object is adjusted is displayed;
When it is determined that the object is not behind the obstruction as viewed from the occupant, the image of the object is not displayed and an actual image of the object is displayed ;
The object presentation method , in which the actual image of the object is acquired via a communication device from an external information acquisition device that is located outside the vehicle and acquires an image including the object using a camera .

Images