JP7600767B2 - Display control device, display control method and program - Google Patents

Description

The present invention relates to a display control device, a display control method, and a program.

Patent document 1 discloses a perimeter monitoring device that has a distance measurement detection unit that detects the distance from a vehicle to an object. The perimeter monitoring device of Patent document 1 has a camera that captures an image of an object from an imaging unit installed in the vehicle. The perimeter monitoring device displays the color of the object superimposed on a color corresponding to the height of the object.

Patent document 2 discloses a vehicle surroundings visual recognition device that includes an obstacle detection means that detects the distance between the vehicle and an obstacle, and an imaging unit that captures the scenery outside the vehicle. When an obstacle is detected, the device of Patent document 2 displays a figure in the captured image that allows the direction in which the obstacle is located to be recognized. This device changes the display color or size of the figure depending on the distance to the obstacle.

JP 2016-97843 A JP 2006-311222 A

In such a device, it is desirable to appropriately warn the driver. For example, when multiple obstacles are involved, it is preferable to warn the driver about objects that pose a greater risk. However, when multiple objects are present around the vehicle, changing the display color according to distance or height can lead to a cluttered display. For example, if objects with a lower risk are emphasized, it becomes difficult to effectively warn the driver about objects with a higher risk.

In view of the above problems, the present invention aims to provide a display control device, a display control method, and a program that can effectively alert users by appropriately setting a risk level.

The display control device according to this embodiment includes an image data acquisition unit that acquires image data of an image captured outside the vehicle, an object detection unit that detects an object contained in the image, a distance measurement data acquisition unit that acquires distance measurement data relating to the distance from the vehicle to the object, a risk level setting unit that sets a risk level for the object according to the size of the object when the distance is equal to or less than a threshold, an emphasis display processing unit that performs emphasis display processing on the image data according to the risk level, and an output unit that outputs an emphasis display image on which emphasis display processing has been performed.

The display control method according to this embodiment includes the steps of acquiring image data of an image taken of the outside of the vehicle, detecting an object contained in the image, acquiring distance measurement data relating to the distance from the vehicle to the object, setting a danger level for the object according to the size of the object when the distance is equal to or less than a threshold, performing highlighting processing on the image data according to the danger level, and outputting a highlighting image where the highlighting processing has been performed.

The program according to this embodiment causes a computer to execute the steps of acquiring image data of an image taken of the outside of the vehicle, detecting an object contained in the image, acquiring distance measurement data regarding the distance from the vehicle to the object, setting a danger level for the object according to the size of the object when the distance is equal to or less than a threshold, performing highlighting processing on the image data according to the danger level, and outputting a highlighting image after the highlighting processing.

The present invention provides a display control device, a display control method, and a program that can effectively alert users by setting an appropriate risk level.

1 is a diagram showing a vehicle according to an embodiment; FIG. 1 illustrates a display control device according to an embodiment. A color bar shows a display color according to the degree of danger. 4 is a flowchart showing a display control method according to the first embodiment; FIG. 2 is a schematic diagram for explaining display control according to the first embodiment. FIG. 2 is a schematic diagram for explaining display control according to the first embodiment. FIG. 2 is a schematic diagram for explaining display control according to the first embodiment. 13 is a flowchart showing a display control method according to a second embodiment. FIG. 11 is a schematic diagram for explaining display control according to the second embodiment. FIG. 11 is a schematic diagram for explaining another display control method.

The present invention will be described below through embodiments of the invention, but the invention according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems. For clarity of explanation, the following description and drawings have been omitted and simplified as appropriate. In addition, the same elements are given the same reference numerals in each drawing, and duplicate explanations have been omitted as necessary.

<First embodiment>
1 is a diagram showing a vehicle 1 according to a first embodiment. The vehicle 1 includes a sensor unit 2 and a display control device 100. In the following description, in order to distinguish the vehicle 1 from surrounding vehicles, the vehicle 1 may be referred to as the host vehicle and the surrounding vehicles may be referred to as surrounding vehicles.

The sensor unit 2 has at least one imaging device and captures an image of the outside of the vehicle. The sensor unit 2 outputs image data captured by the imaging device to the display control device 100. The sensor unit 2 has a distance measurement sensor that detects obstacles. The sensor unit 2 outputs distance measurement data measured by the distance measurement sensor to the display control device 100.

The display control device 100 may be installed at any position in the vehicle 1. The display control device 100 may be connected to a CAN (Controller Area Network). The display control device 100 performs image processing to detect objects (obstacles) in the images captured by the sensor unit 2. The display control device 100 performs highlighting to alert the user. For example, the display control device 100 applies a highlighting color such as red to objects included in the image. This makes it easier for users such as the driver to recognize the presence of the object and to avoid contact. The display control device 100 will be described later.

Note that, hereinafter, the term "video" also means "video data showing a video" as an object of information processing. Similarly, hereinafter, the term "image" also means "image data showing an image" as an object of information processing. Note that images and videos may be moving images or successive still images.

FIG. 2 is a block diagram showing the configuration of a display control device 100 according to the first embodiment and a display control system 10 having the display control device 100. The display control system 10 has a sensor unit 2, a display unit 30, an ECU (Electronic Control Unit) 40, and the display control device 100. The display control device 100 is connected to each of the sensor unit 2, the display unit 30, and the ECU 40 so as to be able to communicate with them.

The display unit 30 is, for example, a display. The display unit 30 displays an image under the control of the display control device 100. The display unit 30 may include a speaker. In this case, the display unit 30 may output sound from the speaker. In addition, when the display control device 100 is mounted on the vehicle 1, the display unit 30 may be provided inside the vehicle 1 at a position where the driver of the vehicle can see while driving. In this case, the display unit 30 may be realized by the interface unit 108 described later. The display unit 30 may be a monitor of a car navigation system. In addition, the display unit 30 does not need to be an in-vehicle device permanently installed in the vehicle 1. The display unit 30 may be configured, for example, by a display of an information terminal such as a smartphone or tablet terminal owned by the user. Furthermore, part of the display control process may be performed by a smartphone having a display unit.

The sensor unit 2 includes a front camera 21F and a rear camera 21R. When the vehicle 1 moves forward, the front camera 21F captures a front image of the vehicle 1. When the vehicle 1 moves backward, the rear camera 21R captures a rear image of the vehicle 1. The front camera 21F and the rear camera 21R generate image data at, for example, 30 frames per second (30 fps) and supply the generated image data to the display control device 100 every 1/30th of a second. The image data may be generated using, for example, a method such as H.264 or H.265. Alternatively, the sensor unit 2 may be a 360° camera that captures the entire surroundings of the vehicle 1.

The sensor unit 2 has a distance measurement sensor 22 for detecting the distance to an object in the vicinity of the vehicle 1. The distance measurement sensor 22 detects the distance from the vehicle to the object. The distance measurement sensor 22 is, for example, a LIDAR (Laser Imaging Detection and Ranging). Alternatively, the distance measurement sensor 22 may be an infrared camera, a stereo camera, a millimeter wave radar, or the like. Furthermore, the distance measurement sensor 22 may be configured by combining these sensors. Furthermore, the distance measurement sensor 22 is not limited to a single physical device. For example, the distance measurement sensor 22 may be configured by LIDARs arranged at multiple locations on the vehicle 1.

The distance measurement sensor 22 supplies distance measurement data relating to the distance from the vehicle to the object to the display control device 100. Furthermore, the distance measurement sensor 22 may detect the direction of an obstacle as well as the distance to the obstacle. In other words, the distance measurement data may correspond to the distance to the object and the direction of the object. Here, the direction of the object is the direction based on the vehicle. Furthermore, the forward/backward/left/right directions of the object may be indicated by azimuth angles, and the up/down directions may be indicated by elevation/depression angles.

The ECU 40 is included in the vehicle 1 and is part of the configuration that controls the vehicle 1. The display control device 100 and the ECU 40 are connected to each other so that they can communicate with each other via an in-vehicle network such as a CAN.

The display control device 100 includes a control unit 102, a storage unit 104, a communication unit 106, and an interface unit 108 (IF: Interface). The display control device 100 includes a driving information acquisition unit 110, an image data acquisition unit 121, an object detection unit 122, a distance measurement data acquisition unit 125, a risk setting unit 127, an emphasis display processing unit 128, and an output unit 129. The driving information acquisition unit 110 includes a traveling direction acquisition unit 111 and a speed information acquisition unit 113.

The control unit 102 is a processor such as a CPU (Central Processing Unit). The control unit 102 has a function as a calculation device that performs control processing and calculation processing. The storage unit 104 is a storage device such as a memory or a hard disk. The storage unit 104 is, for example, a ROM (Read Only Memory) or a RAM (Random Access Memory). The storage unit 104 has a function for storing control programs and calculation programs executed by the control unit 102 or as a function of the control unit 102. The storage unit 104 also has a function for temporarily storing processing data, etc.

The communication unit 106 performs processing necessary to communicate with the sensor unit 2 and the display unit 30. The communication unit 106 may also perform processing necessary to communicate with a CAN (not shown). The communication unit 106 may include a communication port. The interface unit 108 is, for example, a user interface (UI). The interface unit 108 has an input device such as a keyboard, a touch panel, or a mouse, and an output device such as a speaker.

Therefore, the display control device 100 has a function as a computer. In addition, the display control device 100 realizes components such as the driving information acquisition unit 110, the image data acquisition unit 121, the object detection unit 122, the distance measurement data acquisition unit 125, the risk setting unit 127, the highlighting processing unit 128, and the output unit 129 by the processor of the control unit 102 executing the program stored in the storage unit 104. In addition, each component of the display control device 100 is not limited to being realized by software by a program, but may be realized by any combination of hardware and software. In addition, each component of the display control device 100 may be realized by using an integrated circuit that can be programmed by the user, such as an FPGA (field-programmable gate array), a microcomputer, or an SOC (system on a chip). In this case, the integrated circuit may be used to realize a program composed of each of the above components. This is also true in other embodiments described later.

The driving information acquisition unit 110 acquires driving information related to the driving of the vehicle 1. The driving information acquisition unit 110 acquires various information from the ECU 40. The driving information acquisition unit 110 includes a traveling direction acquisition unit 111 and a speed information acquisition unit 113.

The traveling direction acquisition unit 111 acquires information indicating the traveling direction of the vehicle 1. For example, the traveling direction acquisition unit 111 detects whether the vehicle 1 is moving forward or backward based on the shift position of the vehicle 1. When the shift position of the vehicle 1 is in the reverse (R) position, it detects that the vehicle 1 is moving backward. Alternatively, when the shift lever of the vehicle 1 is in a position such as drive (D), it detects that the vehicle 1 is moving forward.

Furthermore, the traveling direction acquisition unit 111 may acquire steering angle information as the traveling direction. The traveling direction acquisition unit 111 acquires a signal from a CAN or the like to acquire steering angle information indicating the steering angle of the wheels of the vehicle 1. Note that the steering angle information includes information indicating the steering direction, such as right or left, in addition to information indicating the steering angle.

The speed information acquisition unit 113 acquires speed information indicating the speed of the vehicle 1. For example, the speed information acquisition unit 113 detects the vehicle speed based on a vehicle speed pulse from the ECU 40. The speed information acquisition unit 113 acquires speed information indicating a vehicle speed of, for example, 40 kilometers per hour. Alternatively, the speed information may be information in stages such as low speed, medium speed, high speed, etc.

The image data acquisition unit 121 acquires image data captured by the front camera 21F and the rear camera 21R. The image data acquisition unit 121 may switch the image to be acquired depending on the traveling direction acquired by the traveling direction acquisition unit 111. For example, when the vehicle 1 is moving forward, the image data acquisition unit 121 acquires a front image from the front camera 21F. When the vehicle 1 is moving backward, the image data acquisition unit 121 acquires a rear image from the rear camera 21R. Of course, the image data acquisition unit 121 may acquire both a front image and a rear image. In this case, the display control device 100 can switch the image to be processed depending on the traveling direction.

The object detection unit 122 detects objects contained in the image. The object detection unit 122 recognizes objects in the image by performing image analysis on the image data. For example, the object detection unit 122 can perform object recognition by comparing the image with a pre-registered pattern. Furthermore, when an image contains multiple objects, the object detection unit 122 detects each of the objects. Furthermore, the object detection unit 122 may identify the position coordinates and size of the objects in the image.

The objects here include nearby vehicles, walls, wheel stops, overpasses, bicycles, etc. The objects may also include people such as pedestrians. Note that the object detection process in the object detection unit 122 can use known methods, so a detailed description is omitted.

The distance measurement data acquisition unit 125 acquires distance measurement data relating to the distance to an object. Furthermore, if an image contains images of multiple objects, the distance measurement data acquisition unit 125 determines the distance to each object. In other words, the distance from the vehicle to each object is calculated based on the distance measurement data from the distance measurement sensor 22.

The risk setting unit 127 sets a risk level for an object. Furthermore, if an image contains images of multiple objects, the risk setting unit 127 sets a risk level for each object. In other words, a value indicating a risk level is assigned to each object.

Specifically, the risk setting unit 127 sets the risk level according to the possibility of the vehicle coming into contact with an object and the impact of contact. The risk setting unit 127 sets the risk level according to the distance to the object. The risk setting unit 127 expresses the risk level value, for example, on a scale of 10, with 10 being the highest risk level and 0 being the lowest risk level. The risk level is higher for closer objects and lower for farther objects. Furthermore, for objects with an extremely low possibility of contact, the risk setting unit 127 may set the risk level to 0, and for objects that are significantly off the path of the vehicle 1, for example, the risk level may be set to 0.

Alternatively, the risk setting unit 127 sets the risk level according to the size of the object. The risk setting unit 127 sets a higher risk level for larger objects and a lower risk level for smaller objects. For example, the risk setting unit 127 can calculate the size of an object according to the size of the object image in the image and the distance to the object. The processing in the risk setting unit 127 will be described later. Here, the size of an object can be one or both of the size in the horizontal direction (left-right direction) and the size in the vertical direction (up-down direction).

The highlighting processing unit 128 performs highlighting processing to highlight objects in the image according to the level of danger. For example, the highlighting processing unit 128 superimposes an emphasis color such as red on objects in the image that are highly dangerous. The highlighting processing unit 128 superimposes a non-emphasis color such as blue on objects in the image that are low in danger. The highlighting processing unit 128 does not perform highlighting processing on objects that are not dangerous, that is, objects that are unlikely to come into contact with the vehicle. The highlighting processing unit 128 superimposes a color that is emphasized more the more dangerous the object is, in order to display a warning to the user, on the objects in the image.

The danger level can be represented by a color bar with a color distribution as shown in FIG. 3. The higher the danger level, the redder the color, i.e., the longer the wavelength, and the lower the danger level, the bluer the color, i.e., the shorter the wavelength. The highlighting processing unit 128 superimposes a color corresponding to the danger level on an object in the image. For example, in the image, red is superimposed on an object with a high danger level, and blue is superimposed on an object with a low danger level. For an object with an intermediate danger level, green is superimposed on the object in the image. The highlighting processing unit 128 then generates a superimposed image in which a color corresponding to the danger level is superimposed on the object in the image. The output unit 129 outputs the image data so that the superimposed image is displayed on the display unit 30 as a highlighting image.

In this way, by changing the overlay color according to the level of danger, it is possible to appropriately alert the user. In other words, the higher the level of danger, the more emphasized the display is, so the user can easily recognize highly dangerous objects. A warning display can be performed by overlaying an emphasis color on an object in an image. The user can drive while paying more attention to highly dangerous objects than to objects with a low level of danger. It is possible to appropriately alert the user to the object, which contributes to safer driving.

The highlighting is not limited to the color distribution shown in FIG. 3. For example, the danger level may be indicated by shading. For example, the same color may be applied to the image of the object, and only the shading may be changed according to the danger level. The highlighting processing unit 128 may perform highlighting processing by changing the color tone, shading, brightness, transparency, etc. of the object image. The highlighting processing unit 128 may also perform highlighting processing by applying a contour line to the object. In this case, the contour line is made thicker for objects with a higher danger level. In this way, the highlighting processing unit 128 performs highlighting processing such that the higher the danger level, the more emphasized the object is.

Using Figs. 4 to 7, examples of the risk level setting process and highlighting process will be described. Fig. 4 is a flowchart showing the display control process according to this embodiment. Figs. 5 to 7 are diagrams explaining the risk level when reversing and the highlighting process. Figs. 5 to 7 show an example of the vehicle 1 reversing into parking space 300. Figs. 5 to 7 show, from the top, a superimposed image, a color scheme according to the risk level, a camera image, and a top view. As the vehicle 1 reverses at a low speed, the state changes in the order of Figs. 5, 6, and 7.

A rear wall 301 is disposed at the rear of the parking space 300, and side walls 302 are disposed on the left and right. In other words, the parking space 300 is surrounded on three sides by the rear wall 301 and the side walls 302. The vehicle then retreats into the parking space 300, which is partitioned by the rear wall 301 and the side walls 302. Wheel stops 304 are disposed on the road surface of the parking space 300. The rear camera 21R captures an image of the parking space 300. The distance measurement sensor 22 detects the distance to the rear wall 301 and the side walls 302 as the distance to the object.

First, the image data acquisition unit 121 acquires image data of an image captured by the rear camera 21R (S401). The object detection unit 122 detects objects contained in the image (S402). For example, the object detection unit 122 detects the rear wall 301, the side wall 302, and the wheel stopper 304 contained in the image. Distance measurement data measured by the distance measurement sensor 22 is acquired (S403). This allows the distance from the vehicle 1 to each of the rear wall 301, the side wall 302, and the wheel stopper 304 to be calculated. The risk level setting unit 127 detects the size of the object based on the distance measurement data and image data (S404). This allows the distance and size of each object to be detected.

Next, the risk setting unit 127 determines whether the distance from the vehicle to the object is equal to or less than a threshold distance (S405). For example, the risk setting unit 127 compares the distance to the closest object with the threshold distance. When multiple objects are included in the image, the risk setting unit 127 determines whether the distance to the closest object is equal to or less than the threshold distance. Here, the distance to the rear wall 301 is compared with the threshold distance.

If the distance to the object is not equal to or less than the threshold distance (NO in S405), the risk setting unit 127 sets the risk level according to the distance (S406). In other words, the closer the distance to the object, the higher the risk level set by the risk setting unit 127.

If the distance to the object is equal to or less than the threshold distance (YES in S405), the risk setting unit 127 sets the risk level according to the size of the object (S407). In other words, the larger the object, the higher the risk level set by the risk setting unit 127.

Then, the highlighting processing unit 128 performs highlighting processing according to the degree of danger (S408). That is, the highlighting processing unit 128 generates a superimposed image by superimposing a red color on objects in the image that are highly dangerous. The output unit 129 outputs the superimposed image that has been subjected to the highlighting processing to the display unit 30 (S409). The processing is repeated while the vehicle 1 is reversing. As a result, the superimposed images shown in Figures 5 to 7 are displayed.

For example, in Figs. 5 and 6, the distance to the rear wall 301 is farther than the threshold distance. Therefore, as in S406, the risk setting unit 127 sets the risk level according to the distance to the object. As shown in Fig. 5, when the distance to the rear wall 301 is far, the risk level is low. Then, as the vehicle 1 moves backward, the vehicle 1 moves closer to the rear wall 301 and the side wall 302. Therefore, the risk level gradually increases. In the state shown in Fig. 6, the vehicle 1 is closer to the rear wall 301 and the side wall 302 than in the state shown in Fig. 5, so the risk level of the rear wall 301 and the side wall 302 is higher. Therefore, as the vehicle 1 moves backward from the position shown in Fig. 5 to the position shown in Fig. 6, the color of the superimposed image changes.

On the other hand, in FIG. 7, the distance to the rear wall 301 is less than the threshold distance. Therefore, as in S407, the risk level setting unit 127 sets the risk level according to the size of the object. Since the rear wall 301 and the side wall 302 are larger than the predetermined size, the highest risk level is set. In the superimposed image in FIG. 7, the color red, which indicates the highest risk level, is superimposed on the rear wall 301 and the side wall 302.

In this way, when the distance to the rear wall 301 is equal to or less than the threshold distance, the risk setting unit 127 sets the risk level according to the size of the object. This allows the user to easily recognize large objects with a high risk level. For example, if the risk level changes only according to the distance, the same risk level is set regardless of the size of the object. Therefore, it is difficult to draw attention to large objects that are at a distance away from the vehicle 1. In contrast, according to this embodiment, when an object is closer than the threshold distance, the risk setting unit 127 sets the risk level according to the size of the object. If the vehicle 1 comes into contact with a large object, the impact is large. In this embodiment, the risk level of large objects can be set high, so that attention can be drawn to large objects that are at a distance away from the vehicle 1.

In this way, when the vehicle 1 approaches the object to within a threshold distance, the risk setting unit 127 sets the risk level according to the size of the object. Therefore, the user can appropriately recognize large objects that are highly dangerous. The user, that is, the driver, can decelerate and park appropriately. This can effectively assist driving. Furthermore, when the vehicle 1 is away from the object by more than the threshold distance, the risk setting unit 127 sets the risk level according to the distance to the object. Therefore, the risk level changes according to the approach to the object, so the user can recognize the distance to the object. In this way, the risk level can be appropriately set according to the situation, so that the user can be effectively alerted.

Note that in step S407, if the size of an object closer than the threshold distance is smaller than a predetermined size, the risk setting unit 127 may set the risk level according to the distance, as in step S405. In other words, as the distance becomes closer, the risk setting unit 127 increases the risk level of small-sized objects. In other words, if the size of an object closer than the threshold distance is smaller than a predetermined size, the risk level may be set to a medium level. Furthermore, the size of an object may be divided into three or more stages, and the risk level may be set in multiple stages.

The risk level setting unit 127 may also exclude objects smaller than the predetermined size from the comparison with the threshold distance in S405. In other words, the risk level setting unit 127 compares the distance to the object closest to the vehicle 1 among objects of a certain size or larger with the threshold distance. Specifically, the wheel stopper 304 is an object smaller than the predetermined size, and is therefore not subject to comparison in step S404.

Furthermore, the risk level may be set to 0 for objects that are unlikely or impossible to come into contact with vehicle 1. For example, the risk level setting unit 127 calculates height information indicating the height of the object as the size of the object. If the height of the object is less than the minimum ground clearance of vehicle 1, the possibility of contacting vehicle 1 is low. For example, the height of wheel stopper 304 is lower than the minimum ground clearance of vehicle 1, and therefore the risk level is low. Therefore, wheel stopper 304 may be excluded from the comparison process in step S404.

The threshold distance may be determined according to the size of the vehicle 1. For example, a 4-ton truck has a long overall length of approximately 8 m. Therefore, the threshold distance from the object to the vehicle is set to approximately 4 m. Also, for example, a compact or ordinary passenger car has a short overall length of approximately 4.70 m or less. Therefore, the threshold distance from the object to the vehicle can be set to approximately 2 m. Alternatively, if the depth of the parking space 300 is approximately 10% longer than the overall length of the vehicle, the threshold distance may be set to less than the overall length of the vehicle. The threshold distance may also be determined according to the width of the vehicle 1.

<Embodiment 2>
In the second embodiment, the risk setting unit 127 performs risk setting processing in accordance with the driving information. More specifically, the setting processing performed by the risk setting unit 127 is switched depending on the traveling direction. For example, when the vehicle 1 is moving forward, the risk setting unit 127 performs a first setting processing. When the vehicle 1 is moving backward, the risk setting unit 127 performs a second setting processing different from the first setting processing. Note that the configurations of the vehicle 1 and the display control system 10 are similar to those in Figs. 1 and 2, etc., and therefore description thereof will be omitted.

Figure 8 is a flowchart showing the display control process in this embodiment. First, the driving information acquisition unit 110 acquires driving information from the ECU 40 (S801). That is, the traveling direction acquisition unit 111 acquires the traveling direction. Furthermore, the speed information acquisition unit 113 may acquire speed information.

The image data acquisition unit 121 acquires image data of the image captured by the rear camera 21R (S802). The object detection unit 122 detects an object contained in the image (S803). Distance measurement data measured by the distance measurement sensor 22 is acquired (S804). The risk level setting unit 127 detects the size of the object based on the distance measurement data and image data (S805). The processing of steps S802 to S805 is basically similar to the processing of steps S401 to S404 in the first embodiment, and therefore a description thereof will be omitted. However, step S802 differs from step S401 in that the image data acquisition unit 121 acquires image data of the forward image captured by the front camera 21F and the rear image captured by the rear camera 21R.

Next, it is determined whether the vehicle 1 is moving forward (S806). If the vehicle 1 is moving forward (YES in S806), the risk level setting unit 127 sets the risk level by a first setting process (S807). The first setting process will be described later.

If the vehicle 1 is not moving forward (NO in S806), the risk setting unit 127 sets the risk level by a second setting process (S808). Note that the second setting process is different from the first setting process. If the vehicle 1 is moving backward, the risk setting unit 127 sets the risk level by a second setting process that is different from the first setting process. For the second setting process, for example, the method shown in the first embodiment can be used. In other words, in the second setting process, the risk setting unit 127 sets the risk level according to the size and distance of the object.

Then, the highlighting processing unit 128 performs highlighting processing according to the degree of danger (S809). That is, the highlighting processing unit 128 generates a superimposed image by superimposing a red color on objects in the image that are highly dangerous. The output unit 129 outputs the superimposed image that has been subjected to the highlighting processing to the display unit 30 (S810).

An example of the first setting process will be described below with reference to FIG. 9. FIG. 9 shows a camera image of a road 310 ahead of the vehicle and a superimposed image in which the camera image has been subjected to highlighting processing. An viaduct 311 is provided ahead of the vehicle. In other words, the vehicle 1 is traveling on the road 310 that passes under the viaduct 311.

Here, assume that the vehicle height of vehicle 1 is such that it cannot pass under viaduct 311. For example, assume that viaduct 311 is 1.9 m above the ground, and the vehicle height is greater than 1.9 m.

The object detection unit 122 detects the viaduct 311 ahead in the direction of travel as an object. The risk setting unit 127 detects the ground height of the viaduct 311. In other words, it detects the clearance height from the road to the viaduct 311. The risk setting unit 127 compares the height of the vehicle 1 with the ground height of the viaduct 311. Then, depending on the comparison result, the risk setting unit 127 sets the risk level.

If the vehicle 1 is at a height where it cannot pass under the viaduct 311, the danger level setting unit 127 sets a high danger level for the viaduct 311. The highlighting processing unit 128 performs highlighting according to the danger level. The highlighting processing unit 128 superimposes a red highlight color, which corresponds to a high danger level, on the viaduct 311 in the image as a warning display 312. Since there is a high possibility that the vehicle 1 will come into contact with the viaduct 311, the display unit 30 can display the warning display 312 to the user. The warning display 312 in the figure is superimposed only on the viaduct 311 at the top in the direction of travel, but it may be superimposed on the entire surface of the viaduct 311.

In general, the vehicle speed when moving forward is faster than the vehicle speed when moving backward. For example, when moving backward, the vehicle 1 moves at a relatively slow speed, for example, when parking in a garage. When moving backward to park, the vehicle speed is slow, so the user is likely to notice the height restriction. In contrast, when moving forward, the vehicle speed is fast, so the user may overlook the height restriction. Therefore, when moving forward, the risk setting unit 127 detects the height of the object and compares the vehicle height with the height of the object. In this way, by switching the risk setting process depending on the direction of travel, the risk can be set appropriately. The user, who is the driver, can be more appropriately warned.

In the above explanation, an example of passing through the road 310 under the viaduct 311 has been described, but the processing of this embodiment can also be used for things other than the viaduct 311. For example, parking lots such as multi-story parking lots, indoor parking lots, and mechanical parking lots may have height restrictions. When entering such parking lots with height restrictions, the processing of this embodiment can be applied. The risk setting unit 127 sets a risk level for the object according to the clearance limit height of the viaduct 311 and the height inside the building. The risk setting unit 127 compares the height of the object with the height of the vehicle 1 to determine whether the vehicle 1 can pass under the object. The risk setting unit 127 sets a risk level according to the determination result.

A margin may be provided for the comparison result between the vehicle height of vehicle 1 and the clearance height of the object. For example, in the above explanation, a warning is displayed when the vehicle height of vehicle 1 is higher than the clearance height limit of viaduct 311, but a warning may be displayed if the vehicle height of vehicle 1 is approximately the same as the clearance height limit of viaduct 311. The risk setting unit 127 may increase the risk when the difference between the vehicle height of vehicle 1 and the ground height of the object is equal to or less than a predetermined value. Specifically, it is possible to provide a height margin taking into account road surface steps, etc.

In the first setting process, the danger level may be set according to the distance to the object. For example, weighting may be performed so that the closer the distance, the higher the danger level. Furthermore, if the distance to the object is farther than a threshold distance, the danger level may be set according to the distance as in embodiment 1. Then, the height may be compared when the distance to the object becomes equal to or less than the threshold distance. In this way, the user can be more appropriately warned. For example, when the distance is far, the accuracy of height detection is low.

The risk level setting unit 127 may also set the risk level based on speed information. For example, the risk level setting unit 127 may adjust the timing of displaying a warning depending on the speed information and the distance to the object. When the vehicle 1 is traveling at high speed, the time from object detection to arrival at the viaduct 311 is short. Therefore, it is preferable to display a warning to the user early. The faster the speed, the earlier the timing of displaying the highlight color is preferable. It becomes possible to display a warning at an appropriate timing.

The risk setting unit 127 may measure at least one of the height of the object from the road surface and the left-right position of the object. Based on the measurement results, the risk setting unit 127 determines whether there is a possibility that the object will collide with the vehicle 1. The risk setting unit 127 sets a risk level for the object depending on the determination result. This process will be described with reference to FIG. 10. Here, an example of parking forward or backward into the parking space 320 will be described.

Parking space 320 is provided with wheel stops 324. Furthermore, nearby vehicle 321 is parked to the side of parking space 320. Also, nearby vehicle 322 is parked at the back of parking space 320. The distances to nearby vehicles 321 and 322 are closer to vehicle 1 than the threshold distance. Their size in the left-right direction (horizontal direction) is approximately the same as that of vehicle 1. Furthermore, pedestrian 325 is located at the far end of parking space 320. Note that the distance to pedestrian 325 is farther than the threshold distance.

The risk level setting unit 127 measures the left-right positions of the surrounding vehicles 321, 322 based on the distance measurement data. Based on the measurement results of the left-right positions, the risk level setting unit 127 determines whether there is a possibility that the vehicle 1 will collide with the surrounding vehicles 321, 322. The risk level setting unit 127 sets the risk level of the surrounding vehicles 321, 322 according to the determination results.

For example, since nearby vehicle 321 is located immediately next to parking space 320, the risk setting unit 127 determines that there is a possibility of collision. Since nearby vehicle 322 is located ahead of parking space 320 in the direction of travel, the risk setting unit 127 determines that there is a possibility of collision. Therefore, the risk setting unit 127 sets a high risk level for nearby vehicles 321 and 322. Note that the forward direction of travel means behind vehicle 1 when vehicle 1 is reversing, and means in front of vehicle 1 when vehicle 1 is moving forward.

In the superimposed image, warning displays 331 and 332 are superimposed on the surrounding vehicles 321 and 322, respectively. In other words, a highlighting color such as red is superimposed on the surrounding vehicles 321 and 322 as the warning displays 331 and 332.

In the left-right direction (horizontal direction), pedestrian 325 is located away from the front of vehicle 1 in the traveling direction. Based on the distance measurement data, risk level setting unit 127 measures the left-right position of pedestrian 325. Based on the measurement result of the left-right position, risk level setting unit 127 determines whether there is a possibility that vehicle 1 will collide with pedestrian 325. Based on the determination result, risk level setting unit 127 sets the risk level of pedestrian 325. Risk level setting unit 127 determines that the possibility of colliding with pedestrian 325 is extremely low. A low risk level is set for pedestrian 325. For example, risk level setting unit 127 sets the risk level of pedestrian 325 to 0. Therefore, no warning is displayed for pedestrian 325 in the superimposed image.

In this way, the risk setting unit 127 may set the risk level according to the left-right position and size (width) of the object. Also, it may determine whether there is a possibility of a collision with the vehicle 1 according to the steering angle, the width of the object, and the left-right (lateral) position. Then, if there is a low possibility of contact with the vehicle 1, the risk setting unit 127 may set the risk level low. Also, if there is no possibility of contact, the risk setting unit 127 may set the risk level to 0. Note that when determining whether or not there will be a collision with the vehicle 1 based on the left-right position, steering angle information of the vehicle 1 may be taken into consideration. In other words, the risk setting unit 127 may determine whether or not there is an object ahead in the traveling direction according to the steering angle, for example, in the front left or front right.

The risk setting unit 127 measures the height of the wheel stopper 324 based on the distance measurement data. Based on the height measurement result, the risk setting unit 127 determines whether or not there will be a collision with the wheel stopper 324. The risk setting unit 127 sets a risk level for the object according to the determination result. The wheel stopper 324 is lower than the minimum ground clearance of the vehicle 1. Therefore, the risk setting unit 127 determines that there is no possibility of a collision with the wheel stopper 324, and sets the risk level to 0. Therefore, no warning is displayed for the wheel stopper 324 in the superimposed image.

In this way, the danger level setting unit 127 can set the danger level more appropriately. This makes it possible to effectively alert the user. Furthermore, the display control device 100 may output a voice message or a warning sound from the speaker to alert the user based on the danger level. In other words, the output unit 129 may generate voice data according to the danger level.

The program for executing the above display control method can be stored and supplied to the computer using various types of non-transitory computer readable media. The non-transitory computer readable media includes various types of tangible storage media. Examples of the non-transitory computer readable media include magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R/Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (random access memories)). The program may also be supplied to the computer by various types of transitory computer readable media. Examples of the transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer readable media can supply the program to the computer via wired communication paths such as electric wires and optical fibers, or wireless communication paths.

REFERENCE SIGNS LIST 1 vehicle 2 sensor unit 21F front camera 21R rear camera 22 distance measurement sensor 30 display unit 40 ECU
REFERENCE SIGNS LIST 100 Display control device 110 Traveling information acquisition unit 111 Traveling direction acquisition unit 113 Speed information acquisition unit 121 Image data acquisition unit 122 Object detection unit 125 Distance measurement data acquisition unit 127 Risk level setting unit 128 Highlighting processing unit 129 Output unit 301 Rear wall 302 Side wall 304 Wheel stopper 310 Road 311 Viaduct 312 Warning display 320 Parking space 321, 322 Nearby vehicles 324 Wheel stopper 325 Pedestrians 331, 332 Warning display

Claims (5)

an image data acquisition unit that acquires image data of an image captured outside the vehicle;
an object detection unit that detects an object included in the image;
a distance measurement data acquisition unit that acquires distance measurement data relating to a distance from the host vehicle to the object;
a danger level setting unit that sets a danger level for the object in accordance with a size of the object when the distance becomes equal to or less than a threshold;
an emphasis display processing unit that performs emphasis display processing on image data according to the degree of risk;
an output unit that outputs a highlighted image that has been subjected to a highlighting process;
Equipped with
The threshold distance is set according to the size of the host vehicle.
Display control device. The display control device according to claim 1, wherein the risk level setting unit sets a risk level for the object according to the distance to the object when the distance is greater than a threshold value. The risk level setting unit is
measuring at least one of a height of the object from a road surface and a lateral position of the object;
determining whether the object will collide with the host vehicle based on the measurement result;
The danger level of the object is set according to the judgment result.
The display control device according to claim 1 or 2. acquiring image data of an image captured outside the vehicle;
detecting an object in the image;
acquiring distance measurement data relating to a distance from the host vehicle to the object;
When the distance is equal to or less than a threshold, setting a danger level for the object according to a size of the object;
performing a highlighting process on the image data according to the degree of risk;
outputting a highlighted image on which the highlighting process has been performed;
Equipped with
The threshold distance is set according to the size of the host vehicle.
Display control method. acquiring image data of an image captured outside the vehicle;
detecting an object in the image;
acquiring distance measurement data relating to a distance from the host vehicle to the object;
When the distance is equal to or less than a threshold, setting a danger level for the object according to a size of the object;
performing a highlighting process on the image data according to the degree of risk;
outputting a highlighted image on which the highlighting process has been performed;
A program for causing a computer to execute the following:
The threshold distance is set according to the size of the host vehicle.
program.

Images