JP7746912B2 - Image display control system - Google Patents

Description

The present invention relates to technology for displaying captured images of the area around a vehicle.

Patent Document 1 discloses a surroundings monitoring device that includes an acquisition unit that acquires a first surroundings image of a vehicle, a first display control unit that displays a selection screen on a display device for selecting a second surroundings image that has been subjected to image processing for visibility of the first surroundings image, a touch panel that accepts the selection of the second surroundings image, and a second display control unit that displays the selected second surroundings image on the display device.

Japanese Patent Application Laid-Open No. 2018-201115

In the technology described in Patent Document 1, the situations in which occupants want to display images of the vehicle's surroundings vary depending on the driving conditions and individual differences, but if occupants are always required to perform an operation to display the surroundings image, this can be inconvenient for occupants while driving.

The object of the present invention is to provide technology that displays captured images of the vehicle's surroundings at appropriate times.

To solve the above problem, an image display control system according to one embodiment of the present invention includes an acquisition unit that acquires captured images of the vehicle's surroundings and information about the vehicle's driving conditions; a display determination unit that determines whether to start displaying the captured images on an in-vehicle display in response to a start condition input by an occupant or a set start condition being satisfied; a learning unit that learns the display start timing based on the occupant's input based on information about the driving conditions at the time the captured images are started to be displayed by the occupant's input; and a condition setting unit that sets the learned display start timing as a start condition or suggests to the occupant that the start condition be set. If the start condition is satisfied, the display determination unit starts displaying the captured images from a non-displayed state. If the learned timing starts displaying the captured images earlier than the set start condition, the condition setting unit suggests to the occupant that the start condition be set.

This invention provides technology that displays captured images of the vehicle's surroundings at appropriate times.

FIG. 1 is a diagram showing an image of the surroundings of a vehicle displayed on an in-vehicle display. FIG. 1 is a diagram illustrating a functional configuration of an image display control system according to an embodiment. 10 is a flowchart of a process for displaying a captured image of the periphery of a vehicle.

1 is a diagram showing an image of the vehicle's surroundings displayed on an in-vehicle display. A first surroundings image 18a and a second surroundings image 18b (when not distinguished from each other, simply referred to as surroundings images) are displayed separately on an output unit 18 functioning as an in-vehicle display. The first surroundings image 18a is generated as a plan view by combining multiple captured images, and is displayed as if viewed from above the vehicle 2. The second surroundings image 18b displays the area ahead of the vehicle 2. The first surroundings image 18a and the second surroundings image 18b are images captured at the same time.
The image of the vehicle's surroundings may be a processed image of the area around the vehicle, or it may be the image itself; in either case, although it includes an image that has been processed, it is essentially a captured image.

Figure 1 shows the host vehicle 2 and an oncoming vehicle 6 passing each other on a narrow road. The second peripheral image 18b shows that the oncoming vehicle 6 is approaching, and a road ditch 4 is located to the left of the host vehicle 2. The driver drives while looking at the first peripheral image 18a to prevent the host vehicle 2 from falling into the road ditch 4. In this way, the output unit 18 can provide driving assistance by displaying an image of the surroundings of the host vehicle 2.

Captured images of the area around the vehicle are displayed on the output unit 18 when the occupant inputs a command to start display, or when a predetermined start condition is met. For example, in the initial setting, the predetermined start condition is met only when parking is detected, for example, when the vehicle speed is 10 kilometers per hour or less. On the other hand, occupants may want to display captured images not only when parking, but also when passing an oncoming vehicle 6 on a narrow road as shown in Figure 1, or when turning a sharp corner, etc.

The image display control system of this embodiment learns the driving conditions under which the occupant started displaying the peripheral image, and resets the specified start conditions to display the peripheral image under the learned driving conditions. This allows the system to set the peripheral image to be displayed if a driving condition similar to that under which the occupant input to start display is detected.

Figure 2 is a diagram showing the functional configuration of the image display control system 1 of the embodiment. Each function of the image display control system 1 can be configured in hardware using circuit blocks, memory, and other LSIs, and in software, can be realized by system software or application programs loaded into memory. Therefore, those skilled in the art will understand that each function of the image display control system 1 can be realized in various ways using hardware alone, software alone, or a combination of both, and is not limited to any one of these.

The image display control system 1 is installed in a vehicle and includes an image display control device 10, a camera 12, an input unit 14, an on-board sensor 16, and an output unit 18. The camera 12 captures images of the area around the vehicle. There may be multiple cameras 12, each capable of capturing images of the front, rear, left, and right sides of the vehicle. The camera 12 sends the captured images of the area around the vehicle to the image display control device 10.

The input unit 14 is a touch panel provided on the in-vehicle display and/or a microphone that detects the occupant's voice. The touch panel accepts operational input from the occupant, and the microphone accepts voice input from the occupant. This allows the occupant to input information to display a peripheral image. The input unit 14 sends the occupant's input results to the image display control device 10. The occupant's input results include inputs made by operation and inputs made by voice.

The on-board sensor 16 detects the vehicle's driving conditions and transmits driving condition information to the image display control device 10. The on-board sensor 16 may be, for example, a vehicle speed sensor, a shift position detection sensor, a steering angle sensor, a millimeter-wave radar, an ultrasonic sensor, or a combination of these. Like the on-board sensor 16, the camera 12 also functions as a sensor that detects driving conditions. These sensors that detect driving conditions can detect the vehicle's driving state and driving environment.

The output unit 18 includes an in-vehicle display and a speaker. The in-vehicle display may be integrated with the touch panel of the input unit 14. The output unit 18 displays surrounding images under the control of the image display control device 10 and outputs audio to the occupants.

The image display control device 10 includes an acquisition unit 20, a display determination unit 22, a display control unit 24, a driving situation analysis unit 26, a condition storage unit 28, a learning unit 30, and a condition setting unit 32.

The acquisition unit 20 acquires information via the network from the camera 12, the input unit 14, and the on-board sensor 16. The acquisition unit 20 acquires captured images of the area around the vehicle from the camera 12, acquires input information entered by the occupant from the input unit 14, and acquires information related to the vehicle's driving conditions from the on-board sensor 16.

The driving situation analysis unit 26 analyzes the driving situation based on the detection results of the camera 12 and the on-board sensor 16, and calculates information related to the driving situation. The information related to the driving situation includes multiple parameters such as vehicle speed, the relative positions of the vehicle and other vehicles, the relative positions of the vehicle and obstacles, road width, the presence or absence of parking lots, and the presence or absence of left or right turns. The driving situation information is used to determine the start conditions and for the learning process in the learning unit 30.

The display decision unit 22 decides to start displaying captured images of the vehicle's surroundings on the output unit 18 in response to occupant input or when a predetermined start condition is met. The captured images of the vehicle's surroundings may be processed and displayed, as in the first surroundings image 18a in Figure 1. The occupant may input a voice command to display the captured image into a microphone, or an operation command to display the captured image on a touch panel. When the input unit 14 detects the occupant's input, the display decision unit 22 decides to display the captured image. After deciding to display, the display decision unit 22 also sends the time when the occupant input instructing display was made to the learning unit 30.

The display determination unit 22 determines to display a captured image of the vehicle's surroundings when a predetermined start condition is met. The display determination unit 22 determines whether the predetermined start condition is met based on the driving situation information calculated by the driving situation analysis unit 26. The predetermined start condition is stored in the condition storage unit 28. The predetermined start condition is changeable and updated from the initial setting. The initial setting start condition is met, for example, when the vehicle speed is 10 kilometers per hour or less and the shift position is reverse. The start condition stored in the condition storage unit 28 is updated based on the learning results of the learning unit 30.

The display control unit 24 controls the output unit 18 to display the captured image of the vehicle's surroundings in accordance with the decision made by the display decision unit 22. The display control unit 24 processes the captured image of the vehicle's surroundings to resemble the first surroundings image 18a shown in Figure 1, adds an image that highlights obstacles to the captured image, or adds an image that indicates the vehicle's predicted traveling direction to the captured image. The display control unit 24 displays the processed captured image on the in-vehicle display of the output unit 18.

The learning unit 30 learns the timing of display start due to occupant input, based on information about the driving conditions when display of captured images of the vehicle's surroundings is started due to occupant input. This derives a parameter indicating the driving conditions when the occupant input to start display. The timing when display of captured images of the vehicle's surroundings is started due to occupant input may be the timing when the occupant inputs using voice or touch operation, or the timing when that input is detected by the input unit 14.

Display start timings based on occupant input can include slowing down in parking lots, slowing down on narrow roads, passing oncoming vehicles on narrow roads, avoiding obstacles, and turning right or left at corners where there are obstacles. Slowing down in parking lots and slowing down on narrow roads are detected by the camera 12 and vehicle speed sensor and are determined by parameters such as the presence or absence of a parking lot, road width, and vehicle speed. Passing oncoming vehicles on narrow roads and avoiding obstacles are detected by the camera 12, millimeter-wave radar, and ultrasonic sensor and are determined by parameters such as the relative positions of the vehicle and other vehicles, the relative positions of the vehicle and obstacles, and road width. Turning right or left at corners where there are obstacles is detected by the camera 12, ultrasonic sensor, and steering angle sensor and is determined by parameters such as the relative positions of the vehicle and obstacles, and steering angle.

The learning unit 30 learns the driving conditions at the time of input by the occupant and derives parameters that enable detection of similar driving conditions. The parameters derived by the learning unit 30 may consist of a single parameter included in the information about the driving conditions calculated by the driving condition analysis unit 26, or may consist of multiple types of parameters, and in either case indicate the display start timing obtained through learning. The learning unit 30 uses machine learning techniques to derive optimal parameters for re-detecting driving conditions similar to those at the time of input.

The condition setting unit 32 suggests to the occupant that the learned display start timing be set as a predetermined start condition. If the occupant agrees, the condition setting unit 32 sets the proposed display start timing as an additional predetermined start condition. The condition setting unit 32 may set the learned display start timing as a predetermined start condition, or may set it as is without suggesting it to the occupant. The learned display start timing is determined by parameters that indicate the driving conditions.

The condition setting unit 32 determines whether the learned display start timing will start displaying the captured image earlier than the currently set start condition, and if the display starts earlier, suggests to the occupant that they set the start condition; if the display does not start earlier, the condition setting unit 32 does not suggest to the occupant. For example, if the currently set start condition is 10 kilometers per hour or less and the learning unit 30 learns that the display should start at 12 kilometers per hour, the learning result will be an earlier display start than the currently set start condition, so the condition setting unit 32 suggests to the occupant that the start condition be set to 12 kilometers per hour or less. Furthermore, the farther the positional relationship between the subject vehicle and other vehicles indicated in the set condition is, the earlier the display will start.

By setting conditions in the condition setting unit 32, captured images of the vehicle's surroundings can be displayed on the in-vehicle display when driving conditions similar to those that occurred when the occupant input the display start command occur. This allows start conditions to be set according to the occupant's needs.

Figure 3 is a flowchart of the process for displaying captured images of the vehicle's surroundings. The acquisition unit 20 acquires information about the driving conditions from the camera 12 and the on-board sensor 16 (S10), and the driving condition analysis unit 26 calculates various parameters corresponding to the information about the driving conditions.

The display determination unit 22 determines whether the start conditions stored in the condition storage unit 28 are met based on the parameters indicating the driving conditions calculated by the driving condition analysis unit 26 (S12). If the parameters indicating the driving conditions meet the start conditions (Y in S12), the display determination unit 22 decides to start display, and the display control unit 24 causes the output unit 18 to display captured images of the area around the vehicle (S14), and this process ends.

If the start conditions are not met (N in S12), the display determination unit 22 determines whether an instruction to start display has been input by the occupant based on the detection results of the input unit 14 (S16). If no input instruction has been input (N in S16), this process ends without displaying.

If an input instruction is received (Y in S16), the display determination unit 22 determines to start display, and the display control unit 24 causes the output unit 18 to display captured images of the vehicle's surroundings (S18). The learning unit 30 learns the display start timing based on the occupant's input, based on information about the driving conditions when the display of captured images of the vehicle's surroundings is started based on the occupant's input, and derives a parameter indicating the display start timing (S20).

The condition setting unit 32 compares the learned display start timing with the currently set start condition (S22) and determines whether the learned display start timing will start the display earlier than the currently set start condition (S24). If the learned display start timing will not start the display earlier than the currently set start condition (N in S24), the process ends. If the learned display start timing will start the display earlier than the currently set start condition (Y in S24), the condition setting unit 32 proposes a change to the start condition via the output unit 18 (S26). If the occupant does not approve the proposed change (N in S28), the process ends.

If the occupant approves the proposed change (Y in S28), the currently set start conditions are reset to new start conditions using the learned parameters indicating the display start timing (S30), and the process ends.

The present disclosure has been described above based on examples. The present disclosure is not limited to the above examples, and various design changes and other modifications may be made based on the knowledge of those skilled in the art.

For example, in the embodiment, the image display control device 10 is provided in a vehicle, but this is not limited to this. For example, some processing may be performed by an external server device, and the analysis processing by the driving situation analysis unit 26 may be performed by the server device. In other words, the image display control device 10 may be configured as a combination of an in-vehicle device and a server device.

1 Image display control system, 2 Vehicle, 4 Road drain, 6 Oncoming vehicle, 10 Image display control device, 12 Camera, 14 Input unit, 16 On-board sensor, 18 Output unit, 18a First peripheral image, 18b Second peripheral image, 20 Acquisition unit, 22 Display determination unit, 24 Display control unit, 26 Driving situation analysis unit, 28 Condition storage unit, 30 Learning unit, 32 Condition setting unit.

Claims (1)

an acquisition unit that acquires captured images of the surroundings of the vehicle and information about the traveling conditions of the vehicle;
a display determination unit that determines whether to start displaying the captured image on the in-vehicle display in response to a start condition input by an occupant or set by the occupant being satisfied;
a learning unit that learns the timing at which the display of the captured image is started by an input from the occupant, based on information about a driving situation at the time when the display of the captured image is started by an input from the occupant;
a condition setting unit that sets the learned timing as the start condition or suggests to an occupant that the timing be set;
the display determination unit starts displaying the captured image from a non-displayed state when the start condition is satisfied,
The image display control system is characterized in that the condition setting unit suggests to the occupant that the start condition be set if the learned timing starts displaying the captured image earlier than the start condition being set .