JP7820144B2 - In-vehicle device and traffic light violation determination method - Google Patents

Description

The disclosed embodiments relate to an in-vehicle device and a method for determining whether a vehicle has run a red light.

Dashboard video recorders, which are in-vehicle video recording devices, are well known. For example, when an acceleration sensor detects an impact due to an accident, a dashboard can perform event recording, recording a predetermined amount of video footage before and after the detection without overwriting it. This makes it possible to reliably record, for example, the circumstances surrounding an accident without overwriting it.

One driving behavior that carries a high risk of causing an accident is running a red light. To detect such red light running as an event, technology has been proposed that determines whether a vehicle has run a red light based on, for example, the illumination status of traffic lights detected from information received by optical beacons or images captured by cameras, as well as various vehicle operation statuses such as braking (see, for example, Patent Document 1).

JP 2012-069051 A

However, the above-mentioned conventional technology leaves room for further improvement in terms of more easily determining whether the vehicle is running a red light.

One aspect of the embodiment has been made in consideration of the above, and aims to provide an in-vehicle device and a method for determining whether a vehicle has run a red light more easily.

According to one aspect of the embodiment, an in-vehicle device is mounted on a vehicle and includes a control unit that performs image recognition of a lighting state of a main traffic signal and a movement of the main traffic signal in an image captured by a camera, determines a traveling direction of the vehicle based on a direction of movement of the main traffic signal, and determines that the vehicle has run a red light if the lighting state of the main traffic signal indicates that traffic in the traveling direction of the vehicle does not have priority and if the speed of the vehicle based on the moving speed of the main traffic signal in the image remains at or above a predetermined speed for a predetermined time .

According to one aspect of the embodiment, it is possible to more easily determine whether the vehicle has run a red light.

FIG. 1 is a diagram illustrating an outline of a traffic light ignoring determination method according to an embodiment. FIG. 2 is a diagram illustrating an example of the configuration of an information processing system according to the embodiment. FIG. 3 is a block diagram illustrating an example of the configuration of the drive recorder according to the embodiment. FIG. 4 is an explanatory diagram of the image recognition AI according to the embodiment. FIG. 5 is an explanatory diagram for the case of turning left. FIG. 6 is an explanatory diagram for the case of turning right. FIG. 7 is an explanatory diagram for determining whether a vehicle has ignored a green light. FIG. 8 is a flowchart showing a processing procedure executed by the drive recorder according to the embodiment.

Embodiments of the in-vehicle device and red light running detection method disclosed herein are described in detail below with reference to the accompanying drawings. Note that the present invention is not limited to the embodiments described below.

Furthermore, the following description will be given using an example in which the in-vehicle device according to the embodiment is a drive recorder 10 installed in a vehicle.

First, an overview of the traffic light running detection method according to the embodiment will be explained using Figure 1. Figure 1 is an explanatory diagram of the traffic light running detection method according to the embodiment.

As shown in Figure 1, the drive recorder 10 has a camera 3 and a control unit 13 (see Figure 3). The drive recorder 10 can continuously record video footage captured by the camera 3 inside and outside the vehicle, repeatedly overwriting the footage.

When the drive recorder 10 detects an impact during an accident as an event, for example using an acceleration sensor, it protects the recording for a predetermined period of time before and after the detection time as event recording, ensuring a reliable record of the situation at the time the accident occurred. Event recording is performed using the event recording method, which is non-volatile recording that cannot be overwritten.

The drive recorder 10 uses images acquired from the camera 3 to execute the red light disregarding method according to the embodiment with the control unit 13. The drive recorder 10 detects the determined red light disregarding by the subject vehicle V-s as an event and performs event recording. In the red light disregarding method according to the embodiment, the control unit 13 determines the illumination status of the main traffic light recognized in the image, determines the movement of the main traffic light in the image, and determines whether the subject vehicle has disregarded the red light based on the determined illumination status of the main traffic light and the movement of the main traffic light in the image.

In other words, the red light running detection method according to the embodiment determines whether the host vehicle V-s has run a red light based on image recognition. Using Figure 1, we will explain an example situation in which the host vehicle V-s is about to enter an intersection.

In the traffic light violation determination method according to the embodiment, the control unit 13 performs image recognition of the main traffic light 300 based on the image captured by the camera 3. The main traffic light 300 is a traffic light that indicates whether or not the subject vehicle V-s has priority in the direction of travel at the intersection where the subject vehicle V-s is entering. The control unit 13 also performs image recognition using image recognition AI (Artificial Intelligence), which will be described later.

The control unit 13 then determines the lighting state of the main traffic light 300 that has been image-recognized (step S1). The control unit 13 makes this determination based on the recognition results of the image recognition AI.

Next, in the traffic light violation determination method according to the embodiment, the control unit 13 determines the movement of the main traffic light 300 on the image used for image recognition (step S2). This determination is made by performing image analysis on multiple consecutive frames and analyzing the differences between frames and the movement of optical flow between frames.

Using Figure 1, the movement of the main signal light 300 when the host vehicle V-s is traveling straight ahead will be explained. In the image captured by camera 3, the main signal light 300 moves upward and out of the frame. By determining this movement of the main signal light 300, the control unit 13 can recognize that the host vehicle V-s is traveling straight ahead in the example of Figure 1. Note that "upward" means, for example, that the movement of the main signal light 300 on the image is directed upward rather than left or right. The control unit 13 compares the directional components of the movement direction vector of the main signal light 300 on the image and determines whether the movement of the main signal light 300 corresponds to the movement when the host vehicle V-s is traveling straight ahead.

The control unit 13 may also determine the speed and direction of travel of the vehicle V-s based on the moving speed of the main traffic light 300 on the image. Specifically, the speed of the vehicle V-s can be determined by pre-storing a coefficient indicating how far the moving distance on the image in a given time corresponds when converted to an actual distance, and converting the moving distance on the image to an actual distance.

Then, the control unit 13 determines whether the vehicle V-s has run a red light based on the results of the determinations made in steps S1 and S2 (step S3). Specifically, if the main traffic light 300 is lit red, there is no priority for straight-ahead traffic, and the movement of the main traffic light 300 corresponds to the vehicle V-s being in a straight-ahead state, the control unit 13 determines that the vehicle V-s has run a red light.

Note that if the control unit 13 determines the speed and direction of travel of the host vehicle V-s in step S2, the control unit 13 may make the following determination. If the main traffic light 300 is lit red, there is no priority for straight-ahead traffic, the host vehicle V-s is traveling straight at a predetermined speed or faster, and a predetermined time has passed, the control unit 13 determines that the host vehicle V-s has run a red light.

Then, when the control unit 13 detects that the host vehicle V-s has run a red light, it treats this as an event and performs event recording.

In this way, the red light running determination method according to the embodiment makes it possible to more easily determine whether the subject vehicle V-s has run a red light based on image recognition. Furthermore, the drive recorder 10 can easily and reliably record the circumstances under which the subject vehicle V-s ran a red light.

Furthermore, if the control unit 13 detects that the vehicle V-s has run a red light, it may notify a server device that collects driving status data for each vehicle. The server device can use the driving status data collected in this way to calculate a driving score for each vehicle or perform big data analysis.

Below, we will explain in more detail an example configuration of an information processing system 1 that includes a drive recorder 10 to which the red light running detection method according to the above-mentioned embodiment is applied.

Figure 2 is a diagram showing an example configuration of an information processing system 1 according to an embodiment. The information processing system 1 includes one or more drive recorders 10 and a server device 100.

Each drive recorder 10 and the server device 100 are connected to each other so that they can communicate with each other via a network N, such as the Internet or a mobile phone network.

As described above, the drive recorder 10 is a device that records various events, including red light ignition. The drive recorder 10 can also transmit various vehicle status data at the time an event occurs, including the recorded event video, to the server device 100.

The server device 100 is a device that collects, stores, and manages vehicle status data transmitted from each drive recorder 10. Based on the collected vehicle status data, the server device 100 can also perform big data analysis and calculate a driving score for each vehicle, as described above.

Next, Figure 3 is a block diagram showing an example configuration of a drive recorder 10 according to an embodiment. Note that Figure 3 shows only the components necessary to explain the features of this embodiment, and omits descriptions of general components.

In other words, the components shown in Figure 3 are conceptual functional components and do not necessarily have to be physically configured as shown. For example, the specific form of distribution and integration of each block is not limited to that shown, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

Furthermore, in the explanation using Figure 3, the explanation of components that have already been explained may be simplified or omitted.

As shown in FIG. 3, the drive recorder 10 according to this embodiment includes a camera 3, a communication unit 11, a memory unit 12, and a control unit 13. The camera 3 is attached to various locations on the vehicle, such as the windshield, rear windshield, side mirrors, and dashboard, and captures images of a predetermined imaging area around the vehicle.

The communication unit 11 is implemented by a network adapter or the like. The communication unit 11 is wirelessly connected to the network N and transmits and receives information to and from the server device 100 via the network N.

The storage unit 12 is realized by a storage device such as RAM (Random Access Memory) or flash memory, and in the example of Figure 3, it stores image recognition AI 12a, processing parameter information 12b, and recorded video information 12c.

Image recognition AI 12a is an AI model for image recognition. Specifically, image recognition AI 12a is a DNN (Deep Neural Network) model trained using a machine learning algorithm. After being loaded as a DNN model into image recognition unit 13a (described below), image recognition AI 12a is configured to be able to distinguish various objects contained in an image captured by camera 3 when the image is input to image recognition unit 13a. Furthermore, image recognition AI 12a is further configured to be able to distinguish the lighting state of a traffic light when a traffic light is identified as an object.

Here, Figure 4 is an explanatory diagram of the image recognition AI 12a according to the embodiment. As shown in Figure 4, the image recognition AI 12a is trained in advance in a learning device 500 including a GPU (Graphics Processing Unit) and AI platform (not shown) to be able to distinguish between various traffic lights, such as standard three-light traffic lights, various arrow-type traffic lights, and single-light traffic lights, using images of these traffic lights as training data.

The image recognition AI 12a is then loaded as a DNN model into the image recognition unit 13a in the drive recorder 10. When an image captured by the camera 3 is input to the image recognition unit 13a, the image recognition AI 12a determines the main traffic light 300 and its illumination status relative to the vehicle V-s in the image.

Returning to the explanation of Figure 3, the processing parameter information 12b is information including parameters such as thresholds used in the determination process executed by the determination unit 13b, which will be described later. The recorded video information 12c is information including a group of recorded videos captured by the camera 3 and recorded by the video recording unit 13d, which will be described later.

The control unit 13 is a controller that is implemented by a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing various programs stored in a storage device within the drive recorder 10 using RAM as a working area. The control unit 13 can also be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

The control unit 13 has an image recognition unit 13a, a judgment unit 13b, a recording control unit 13c, a video recording unit 13d, and a transmission unit 13e, and realizes or executes the information processing functions and actions described below.

The image recognition unit 13a acquires the captured image captured by the camera 3 and inputs the captured image to the image recognition AI 12a that has read it. Furthermore, the image recognition unit 13a acquires the image recognition result output from the image recognition AI 12a as a result of inputting the captured image. Furthermore, the image recognition unit 13a outputs the image recognition result acquired from the image recognition AI 12a to the determination unit 13b.

The determination unit 13b determines the lighting state of the main traffic light 300 of the host vehicle V-s based on the image recognition results input from the image recognition unit 13a. The determination unit 13b also determines the movement of the main traffic light 300 on the image based on the image recognition results.

The determination unit 13b also determines whether the vehicle V-s has ignored a traffic signal based on the determined lighting status of the main traffic signal 300 and its movement on the image.

The determination unit 13b determines that the vehicle V-s has run a red light when the main traffic light 300 is lit red, indicating that there is no priority for straight-ahead traffic, and the vehicle V-s has been traveling straight at a predetermined speed (e.g., 30 km/h) or greater for a predetermined period of time (e.g., 5 seconds) or greater.

Here, we will explain how to determine whether a vehicle is turning left or right. Figure 5 is an explanatory diagram for a left turn. Figure 6 is an explanatory diagram for a right turn.

For example, in the case of a left turn, as shown in Figure 5, the main traffic light 300 will move out of the frame to the right on the image. By determining this movement of the main traffic light 300, the determination unit 13b determines that the vehicle V-s is turning left in the example of Figure 5. Note that "to the right" means, for example, that the rightward component of the directional component of the movement vector of the main traffic light 300 on the image is larger than the components other than the rightward component.

In such a case, if the illumination state of the main traffic light 300 indicates that there is no right of way for traffic turning left, and the host vehicle V-s has been traveling at a predetermined speed (e.g., 30 km/h) or more for a predetermined period of time (e.g., 5 seconds) or more, the determination unit 13b determines that the host vehicle V-s has run a red light.

On the other hand, in the case of a right turn, for example, as shown in Figure 6, the main traffic light 300 will move out of the frame to the left on the image. By determining this movement of the main traffic light 300, the determination unit 13b determines that the vehicle V-s is turning right in the example of Figure 6. Note that "to the left" means, for example, that the leftward component of the directional component of the movement vector of the main traffic light 300 on the image is larger than the components other than the leftward component.

In such a case, if the illumination state of the main traffic light 300 indicates that there is no right of way for traffic turning right, and the host vehicle V-s has been traveling at a predetermined speed (e.g., 30 km/h) or higher for a predetermined period of time (e.g., 5 seconds) or longer, the determination unit 13b determines that the host vehicle V-s has run a red light.

The determination method described using Figures 5 and 6 makes it possible to determine whether the host vehicle V-s has run a red light through image recognition, even in the right or left turn direction.

Returning to the explanation of Figure 3, the determination unit 13b outputs the determination result to the recording control unit 13c. The recording control unit 13c controls the recording method of the video recorded by the video recording unit 13d according to the determination result of the determination unit 13b. Specifically, the recording control unit 13c controls the access rights to the recorded video.

More specifically, when the determination unit 13b determines that the host vehicle V-s has run a red light, the recording control unit 13c causes the video recording unit 13d to perform event recording that prohibits overwriting. After a predetermined period of event recording has been performed, the recording control unit 13c stops the event recording and then causes the video recording unit 13d to resume continuous recording that allows overwriting.

Furthermore, for events designated in advance as events to be transmitted, the recording control unit 13c causes the transmission unit 13e to transmit vehicle status data including images captured by the camera 3 corresponding to the event to the server device 100. If red light running is designated as an event to be transmitted, and the determination unit 13b determines that the vehicle V-s has run a red light, the recording control unit 13c causes the transmission unit 13e to transmit vehicle status data at the time of the red light running to the server device 100.

The video recording unit 13d records the video captured by the camera 3 in the recorded video information 12c using a recording method controlled by the recording control unit 13c. Based on instructions from the recording control unit 13c, the transmission unit 13e transmits vehicle status data corresponding to the instructions to the server device 100 via the communication unit 11.

Up until now, we have explained how to determine whether the vehicle V-s has run a red light, but the method for determining whether the vehicle has run a red light according to the present embodiment can also be applied to cases where the vehicle has run a green light. Next, we will explain how to determine whether the vehicle has run a green light. Figure 7 is an explanatory diagram for determining whether the vehicle has run a green light.

Specifically, consider a situation in which the host vehicle V-s is stopped at an intersection with a red light. In such a case, the image recognition unit 13a performs image recognition of the main traffic light 300 based on the image captured by the camera 3.

Then, the image recognition unit 13a detects that the illumination state of the image-recognized main traffic light 300 has changed from red to green while the vehicle is stopped at the traffic light (step S11). Based on the image recognition results of the image recognition unit 13a, the determination unit 13b determines that the vehicle V-s has run a green light if the vehicle V-s is the leading vehicle and the speed of the vehicle V-s remains at 0 km/h for a predetermined time (e.g., 5 seconds) (step S12).

When the determination unit 13b detects such a green light violation, it treats this as an event and causes the recording control unit 13c to perform event recording.

This means that if a problem occurs due to the vehicle V-s ignoring a green light, i.e., ignoring the start of the vehicle, the circumstances surrounding the problem can be reliably recorded.

Next, the processing procedure executed by the drive recorder 10 according to the embodiment will be described using Figure 8. Figure 8 is a flowchart showing the processing procedure executed by the drive recorder 10 according to the embodiment.

As shown in FIG. 8, the control unit 13 determines the lighting state of the main traffic light 300 that has been recognized through image recognition (step S101). The control unit 13 also determines the movement of the main traffic light 300 on the image that has been recognized through image recognition (step S102).

The control unit 13 then determines whether the vehicle has run a red light based on the illumination status of the main traffic light 300 and the movement on the image (step S103). If a red light has been detected (step S104, Yes), the control unit 13 performs event recording (step S105). The control unit 13 also transmits vehicle status data to the server device 100 (step S106).

Then, the process is repeated from step S101. Also, if no red light violation is detected (step S104, No), the process is repeated from step S101.

Up until now, red light ignition has been determined using image recognition, but it is of course also possible to combine sensor data from various sensors installed in the vehicle. For example, the direction of travel of the host vehicle V-s may be obtained using sensor values from a steering sensor or acceleration sensor, and the vehicle speed of the host vehicle V-s may be obtained using sensor values from a speed sensor.

As described above, the drive recorder 10 according to the embodiment is an on-board device installed in the host vehicle V-s (an example of a "vehicle") and includes a control unit 13. The control unit 13 determines whether the host vehicle V-s has run a red light based on the illumination state of the main traffic light 300, which is recognized from the image captured by the camera 3, and the movement of the main traffic light 300 in the captured image.

Therefore, the drive recorder 10 according to this embodiment makes it easier to determine whether the host vehicle V-s has ignored a red light.

The control unit 13 also determines the movement of the host vehicle V-s based on the movement of the main traffic light 300 in the captured image.

Therefore, the drive recorder 10 according to the embodiment can determine the movement of the host vehicle V-s based on the movement of the main traffic light 300 detected by image recognition.

In addition, the control unit 13 determines that the host vehicle V-s has run a red light if the illuminated state of the main traffic signal 300 indicates that the host vehicle V-s does not have priority in the direction of travel and the host vehicle V-s has been traveling at a predetermined speed or higher for a predetermined period of time or longer.

Therefore, the drive recorder 10 according to this embodiment can determine whether the host vehicle V-s has run a red light through image recognition.

Furthermore, if the main traffic light 300 moves upward and out of the frame in the captured image, the control unit 13 determines that the vehicle V-s is traveling straight ahead.

Therefore, the drive recorder 10 according to this embodiment can use image recognition to determine whether the host vehicle V-s is running a red light when traveling straight ahead.

Furthermore, if the main traffic light 300 is moving out of the frame to the right in the captured image, the control unit 13 determines that the vehicle V-s is heading left.

Therefore, the drive recorder 10 according to this embodiment can use image recognition to determine whether the host vehicle V-s is running a red light when making a left turn.

Furthermore, if the main traffic light 300 is moving out of the frame to the left in the captured image, the control unit 13 determines that the vehicle V-s is heading in a right turn direction.

Therefore, the drive recorder 10 according to this embodiment can use image recognition to determine whether the host vehicle V-s will run a red light when turning right.

Furthermore, the control unit 13 determines that the host vehicle V-s has run a green light if the main traffic light 300 is lit while the host vehicle V-s is stopped at a traffic light, indicating that the stop has been lifted, and the host vehicle V-s, which is the lead vehicle, remains stopped for a predetermined period of time.

Therefore, the drive recorder 10 according to this embodiment can determine whether the host vehicle V-s has ignored a green light through image recognition.

In the above-described embodiment, the drive recorder 10 performs image recognition and a red light running judgment based on the results of the image recognition. However, some or all of the functions of the image recognition unit 13a and/or the judgment unit 13b may be performed by a device other than the drive recorder 10, such as the server device 100.

Further advantages and modifications may readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents.

1 Information processing system 3 Camera 10 Drive recorder 11 Communication unit 12 Storage unit 12a Image recognition AI
12b Processing parameter information 12c Recorded video information 13 Control unit 13a Image recognition unit 13b Determination unit 13c Recording control unit 13d Video recording unit 13e Transmission unit 100 Server device 300 Main signal Vs Vehicle

Claims (7)

An in-vehicle device mounted on a vehicle,
image recognition of the lighting state of the main traffic signal and the movement of the main traffic signal on the captured image;
determining a direction of travel of the vehicle based on a direction of movement of the main traffic light;
An in-vehicle device comprising: a control unit that determines that the vehicle has run a red light when the lighting state of the main traffic light indicates that the vehicle does not have priority in the direction of travel, and when the speed of the vehicle based on the moving speed of the main traffic light in the captured image remains at or above a predetermined speed for a predetermined time . The control unit
determining the movement of the vehicle, including the traveling direction of the vehicle, based on the movement of the main traffic light on the captured image;
The in-vehicle device according to claim 1 . The control unit
When the main traffic light moves upward and out of the frame in the captured image, it is determined that the vehicle is traveling in a straight direction.
The in-vehicle device according to claim 1 or 2 . The control unit
When the main traffic light is moving out of the frame to the right on the captured image, it is determined that the vehicle is turning left.
The in-vehicle device according to claim 1 , 2 or 3 . The control unit
When the main traffic light is moving out of the frame to the left on the captured image, it is determined that the vehicle is turning right.
The in-vehicle device according to any one of claims 1 to 4 . The control unit
When the lighting state of the main traffic signal indicates that the vehicle has been stopped at a traffic light and image recognition determines that the vehicle is the lead vehicle and has remained stopped for a predetermined time, the vehicle is determined to have ignored a green light.
The in-vehicle device according to any one of claims 1 to 5 . A method for determining whether a vehicle is running a red light, the method comprising:
image recognition of the lighting state of the main traffic signal and the movement of the main traffic signal on the captured image from the camera;
determining a direction of travel of the vehicle based on a direction of movement of the main traffic light;
determining that the vehicle has run a red light when the lighting state of the main traffic light indicates that the vehicle does not have priority in the direction of travel and when the speed of the vehicle based on the moving speed of the main traffic light in the captured image remains at or above a predetermined speed for a predetermined time .