JP7762527B2 - Traffic light information determination device - Google Patents

Description

The present invention relates to a traffic light information determination device.

Patent Document 1 describes a vehicle driving assistance device that extracts red traffic lights from an image of the area ahead of the vehicle.

Japanese Patent Application Laid-Open No. 2005-170154

If there is an obstacle such as a truck between the traffic light and the vehicle, or if the sun is behind the traffic light, it becomes difficult to determine the traffic light's display from the image in front of the vehicle.

The objective of the present invention is to provide a traffic light information determination device that can determine the display of a traffic light even when the image ahead of the vehicle is obstructed.

(1)
One aspect of the present invention is
A traffic light information determination device mounted on a vehicle,
a photographing unit that acquires video;
a controller that determines the display of a target traffic light that displays signal information for the lane in which the vehicle is traveling, and
Equipped with
The photographing unit acquires an image of a front of the vehicle and an image of a reflecting object located around the vehicle,
The controller is characterized in that when a normal judgment result of the display of the target traffic light cannot be obtained based on the image of the front, it judges the display of the target traffic light based on the reflected image of the target traffic light contained in the image of the reflecting object.
(2)
Another aspect of the present invention is a method for producing a
A traffic light information determination device mounted on a vehicle,
a photographing unit that acquires video;
a controller that determines the display of a target traffic light that displays signal information for the lane in which the vehicle is traveling, and
Equipped with
The photographing unit acquires an image of a reflecting object located around the vehicle,
The controller estimates the reflection position of the target traffic light based on the position of the target traffic light and the position of the reflecting object, and determines the display of the target traffic light based on the reflected image of the target traffic light included in the image of the reflection position of the reflecting object.
(3)
Another aspect of the present invention is a method for producing a
A traffic light information determination device mounted on a vehicle,
a photographing unit that acquires video;
a controller that determines the display of a target traffic light that displays signal information for the lane in which the vehicle is traveling, and
Equipped with
The photographing unit acquires an image of a reflecting object located around the vehicle,
The controller
determining the display of the target traffic signal based on a reflected image of the target traffic signal included in the image of the reflecting object;
Furthermore, a validation process is performed to determine whether the reflected image represents the target traffic light;
The validation process includes:
A process of predicting a distorted shape of the target traffic light due to reflection;
comparing the reflected image with the predicted distorted shape;
The present invention is characterized by comprising:

According to the present invention, the controller can determine the display status of a target traffic signal based on the image of the target traffic signal reflected by reflective objects around the vehicle. Therefore, even if the target traffic signal is located ahead of the vehicle and obstructs the image ahead of the vehicle, it is possible to determine the display status of the target traffic signal.

1 is a block diagram showing a traffic light information determination device and a vehicle according to an embodiment of the present invention; FIG. 10 is a diagram illustrating a process for acquiring the position of a target traffic light. 10A and 10B are diagrams illustrating a process for acquiring the position of a reflecting object. 10A and 10B are diagrams illustrating a process of estimating the reflection position of a target traffic light. 10A and 10B are diagrams illustrating a signal display determination process and a validity confirmation process. 4 is a flowchart illustrating an example of a control process of a traffic light information determination device.

Embodiments of the present invention will now be described in detail with reference to the drawings. Figure 1 is a block diagram showing a traffic light information determination device and a vehicle according to an embodiment of the present invention.

The traffic light information determination device 1 of this embodiment is mounted on a vehicle 100 and is capable of determining the display of a target traffic light even when the image ahead of the vehicle 100 is obstructed. The target traffic light refers to a traffic light that displays signal information for the lane in which the vehicle 100 is traveling.

Vehicle 100 is equipped with wheels 102, 103, a power source 104 that generates power transmitted to wheel 102, and a brake mechanism 105 that generates braking force on wheels 102, 103. Vehicle 100 also includes a driving operation unit 106 that receives driving operations from the driver, and a vehicle controller 110 that controls the operation of power source 104 and brake mechanism 105. Vehicle 100 is also equipped with a positioning device 113 that performs positioning, and a notification device 120 (e.g., a display panel) that notifies the driver of information.

The power source 104 may include, for example, an internal combustion engine, an electric motor, or both. The driving operation unit 106 includes an accelerator operation unit (such as an accelerator pedal), a brake operation unit (such as a brake pedal), and a steering unit (such as a steering wheel). The brake mechanism 105 may include a mechanism that generates braking force through friction, a device that generates regenerative braking by regeneratively operating the electric motor, or both. The vehicle controller 110 is composed of one or more EUCs (Electronic Control Units). The vehicle controller 110 receives operation signals from the accelerator operation unit, operation signals from the brake operation unit, and driving requests from the controller 20, which will be described later. Then, in response to these operation signals and driving requests, the vehicle controller 110 controls the driving of the power source 104 and the driving of the brake mechanism 105.

The traffic light information determination device 1 includes a photographing unit 40 that captures video of the area around the vehicle 100, and a controller 20 that determines the display of a target traffic light based on the reflected image of the target traffic light contained in the video of reflecting objects captured by the photographing unit 40. The target traffic light refers to a traffic light that displays signal information for the lane in which the vehicle 100 is traveling.

The image capturing unit 40 is a digital camera or the like that has a lens and an image sensor. The image capturing unit 40 sends a digital video signal to the controller 20. The image capturing unit 40 is attached so that the side of the vehicle 100 (the side having width in the front, back, top and bottom directions) is the capturing direction.

The controller 20 is composed of one ECU or multiple ECUs that communicate with each other and work together. A control program is stored in the controller 20. By executing this control program, the controller 20 extracts the reflected image of the target traffic light from the image of the reflecting object and determines the display of the target traffic light based on the reflected image.

The traffic light information determination device 1 further includes a front image capture unit 45 that captures video of the area ahead of the vehicle 100. If there are no obstacles in the video of the area ahead, the controller 20 may determine the display of the target traffic light from that video. The front image capture unit 45 is configured similarly to the image capture unit 40, and is attached so that the front of the vehicle 100 (the front having width in all directions) is the capture direction.

The traffic light information determination device 1 further includes a first detector 11 capable of detecting the location of structures located in front of the vehicle 100, and a second detector 12 capable of detecting the location of structures located to the side of the vehicle 100. The detection information from the first detector 11 and the detection information from the second detector 12 are sent to the controller 20.

The traffic light information determination device 1 may include an information acquisition unit 15 instead of, or in addition to, the first detector 11 and the second detector 12. The information acquisition unit 15 acquires information from outside the traffic light information determination device 1 and transfers it to the controller 20. The information acquisition unit 15 corresponds to the first information acquisition unit and second information acquisition unit according to the present invention.

The first detector 11 includes, for example, an imaging unit that captures video in the detection direction, and a ranging device that can detect the distance to each part in the captured video. A LiDAR (Light Detection and Ranging) or similar device can be used as the ranging device. The first detector 11 is attached to the vehicle 100 so that the detection direction is the front of the vehicle 100 (the front having width in all directions). From the video information sent from the first detector 11, the controller 20 can calculate the direction of a structure included in the video relative to the vehicle 100 from its position in the video. Furthermore, the controller 20 can obtain the distance to the structure from the distance information sent from the first detector 11. From these angles and distances, the controller 20 can determine the relative position of the structure with respect to the vehicle 100.

Note that the first detector 11 is not limited to the above configuration. For example, the first detector 11 may be a compound eye camera that can acquire images and detect the distance to each part in the image using parallax. Alternatively, the first detector 11 may be a distance measuring device that performs 3D scanning. From the profile of distance information from the 3D scan, it is possible to detect where and what type of structure is located within the scan range, and from the distance information, it is possible to detect the distance to that structure. In addition, various devices that can detect the location of structures located in front of the vehicle 100 can be used as the first detector 11.

The second detector 12 can have the same configuration as the first detector 11, but the detection direction is different. The second detector 12 is installed so that the detection direction is the side of the vehicle 100 (the side having width in the front-to-back and top-to-bottom directions).

Note that if the first detector 11 is capable of detecting the sides of the vehicle 100 in addition to the front of the vehicle 100, the first detector 11 may be used as the second detector 12. In other words, the first detector 11 and the second detector 12 may be combined into a single detector. Furthermore, if the second detector 12 is configured to be able to acquire images, the second detector 12 may be used as the imaging unit 40. Similarly, if the first detector 11 is configured to be able to acquire images, the first detector 11 may be used as the forward imaging unit 45.

The information acquisition unit 15 is a communication device capable of communicating with a system having a database 111 and receiving information from the database 111. The database 111 may be provided on the vehicle 100 or on a server device outside the vehicle. Furthermore, the information acquisition unit 15 is a communication device capable of sequentially receiving positioning information from the positioning device 113 of the vehicle 100. The information acquired by the information acquisition unit 15 includes map information D1 relating to the placement of traffic lights, map information D2 relating to the placement of buildings around the road, and position information D3 of the vehicle 100.

Map information D1 related to the placement of traffic lights may be map information that directly indicates the position (latitude, longitude, and altitude) of traffic lights, or map information that indirectly indicates the position of traffic lights. Map information refers to information that indicates the map position (latitude, longitude, etc.) of fixed objects such as roads and buildings. Map information that indirectly indicates the position of traffic lights is, for example, map information that indicates the horizontal position (latitude and longitude) of traffic lights. Traffic light heights are specified, and standard heights are roughly determined. Therefore, the position of traffic lights, including their height, can be determined from the above map information. Other map information that indirectly indicates the position of traffic lights may be map information that indicates the positions of intersections and crosswalks. The horizontal position (latitude and longitude) of traffic lights can be roughly determined from the layout of intersections or crosswalks relative to roads, and the position (latitude, longitude, and altitude) of traffic lights can be roughly determined from this information and the standard height of traffic lights.

Map information D2 relating to the location of buildings may be map information that can directly or indirectly identify the planar position of the side of a building. The buildings may primarily include buildings facing roads. The sides of the buildings may be sides facing the road. Side information may include information indicating whether the material is reflective, such as glass, and position information (height, orientation, etc.) indicating the location of the reflective material.

The vehicle 100 may be equipped with a navigation system 115, and the database 111 containing the map information D1 and D2 may be the database of the navigation system 115. The information acquisition unit 15 may read the map information D1 and D2 of the area around the road on which the vehicle 100 is traveling from the database 111. Alternatively, the navigation system 115 may supply the map information D1 and D2 of the area around the road on which the vehicle 100 is traveling, or the map information D1 and D2 of the area around the road on which the vehicle 100 is scheduled to travel, to the information acquisition unit 15.

Location information D3 is a measurement of the position of vehicle 100 measured by positioning device 113 provided on vehicle 100. Positioning device 113 has a receiver that receives signals from positioning satellites, an acceleration sensor, a gyro sensor, etc., and sequentially measures the current position (latitude and longitude) of vehicle 100 from the detection results of these. Positioning device 113 may be a component of navigation system 115.

<Operation>
Next, an example of specific processing by the controller 20 will be described. The controller 20 executes processing to acquire the position of a target traffic light, processing to acquire the position of a reflecting object around the vehicle 100, and processing to estimate the reflection position of the target traffic light. The controller 20 then determines the display of the target traffic light from the image of the estimated reflection position. Furthermore, to increase the accuracy of the above determination, the controller 20 performs processing to verify whether the reflected image included in the image of the reflecting object represents the target traffic light. Furthermore, the controller 20 performs driving assistance processing based on the determination result of the display of the target traffic light. Next, each of these processes will be described in detail.

<Process for acquiring the position of the target traffic light>
2 is a diagram illustrating the process of acquiring the position of a target traffic light. The controller 20 acquires the positions of the target traffic lights F1 and F2 based on the detection information of the first detector 11, the acquired information of the information acquisition unit 15 (map information D1 and position information D3 of the vehicle 100), or both. The target traffic lights F1 and F2 are traffic lights that display signal information for the lane on which the vehicle 100 is traveling. The signal information includes information such as stop or go by displaying red, yellow, and green lights. The target traffic lights F1 and F2 are often located ahead of the vehicle 100. The positions acquired by the controller 20 correspond to positions relative to the vehicle 100.

To obtain the positions of target traffic lights F1 and F2 at intersection Rx, the controller 20 first recognizes the intersection Rx and each of the intersecting roads r1 and r2 based on the detection information from the first detector 11. This recognition can be achieved, for example, by performing a pattern matching process using image patterns specific to various intersecting roads to determine whether the image patterns are included in the image obtained from the first detector 11. Next, the controller 20 calculates the distance L to the intersection Rx and the widths x and y of each of the intersecting roads r1 and r2 based on the detection information from the first detector 11. Through this calculation, the controller 20 can identify the layout of the intersection Rx and the relative positions of each part of the intersection Rx from the vehicle 100. The controller 20 stores data indicating the standard positions and standard heights of traffic lights at intersections with various layouts. Based on this data, the controller 20 calculates the relative positions of each of the traffic lights F1 to F8 from the vehicle 100 by applying standard positions of traffic lights to the identified layout of the intersection Rx. The controller 20 can then extract information on the relative positions of the target traffic lights F1 and F2 from this.

This method has the advantage of being able to obtain the positions of the target traffic lights F1 and F2 even when the first detector 11 cannot directly detect the target traffic lights F1 and F2 due to an obstacle, such as when there is a tall vehicle in front of the vehicle 100.

Note that the method for acquiring the positions of the target traffic lights F1 and F2 is not limited to the above example. For example, the controller 20 may perform pattern recognition on the video or three-dimensional scan data of the first detector 11 to directly identify the target traffic lights F1 and F2. When performing pattern recognition, the controller 20 may narrow the range of pattern recognition based on traffic light map information acquired from the information acquisition unit 15. Then, once the target traffic lights F1 and F2 have been identified, the controller 20 may determine the relative positions of the target traffic lights F1 and F2 from the distance and orientation detection information of the first detector 11.

The controller 20 may also determine the positions of the target traffic lights F1 and F2 from map information D1 regarding the location of traffic lights acquired from the information acquisition unit 15 and vehicle 100 position information D3. In this case, the controller 20 extracts map information D1 regarding the location of the target traffic lights F1 and F2 located on the path of the vehicle 100 from the map information in the database 111 based on the vehicle 100 position information D3. The controller 20 then determines the map positions (latitude, longitude, and altitude) of the target traffic lights F1 and F2 from the map information D1. Furthermore, the controller 20 can determine the relative positions of the target traffic lights F1 and F2 from the vehicle 100 by calculating the difference between the map positions of the target traffic lights F1 and F2 and the current position of the vehicle 100.

The controller 20 can obtain the position of the target traffic light by performing the same process as above at intersections other than intersection Rx, such as roads with crosswalks and traffic lights.

As the vehicle 100 travels, the relative positions of the target traffic signals F1 and F2 change in real time. The controller 20 can determine the positions (relative positions) of the target traffic signals F1 and F2 in real time by repeating one of the position acquisition processes described above. Alternatively, once the positions of the target traffic signals F1 and F2 have been determined using one of the methods described above, the controller 20 may then determine the real-time positions of the target traffic signals F1 and F2 from changes in the position of the vehicle 100. That is, the controller 20 represents the determined positions of the target traffic signals F1 and F2 as position vectors. The controller 20 can then determine the real-time positions of the target traffic signals F1 and F2 by sequentially subtracting the displacement vector (amount and direction of movement) of the vehicle 100 from this position vector. The displacement vector of the vehicle 100 can be calculated from the positioning information of the positioning device 113.

<Process for acquiring the position of a reflecting object>
3 is a diagram illustrating the process of acquiring the position of a reflecting object. The controller 20 acquires the position of a reflecting object located around the vehicle 100 based on the detection information of the second detector 12, the acquired information of the information acquisition unit 15 (map information D2 and position information D3 of the vehicle 100), or both of these. The reflecting object located around the vehicle 100 includes windows w1 to w4 of a building H around the road, another vehicle 200 located around the vehicle 100, etc. The position acquired by the controller 20 corresponds to the relative position from the vehicle 100.

When acquiring the position of a reflecting object, the controller 20 first performs pattern recognition on the image or 3D scan data obtained from the second detector 12 to recognize the building H facing the road and the surrounding vehicles 200. Next, the controller 20 extracts the distance and direction to the recognized building H and the distance and direction to the recognized surrounding vehicles 200 from the detection information of the second detector 12 to acquire the relative position to the building H and the relative position to the surrounding vehicles 200. Then, the controller 20 assumes that there is a reflective surface such as windows w1 to w4 on the side of the building H facing the road, and acquires the position of that side as the position of the reflecting object. Furthermore, the controller 20 acquires the position of the outer surface of the surrounding vehicles 200 as the position of the reflecting object.

Note that during pattern recognition, the controller 20 may determine whether there are reflective objects such as windows w1 to w4 on the side of the building H, and acquire the positions of only those areas recognized as reflective objects as the positions of the reflective objects. Furthermore, during pattern recognition, the controller 20 may recognize the windows, bodies, and body colors of the surrounding vehicles 200, and acquire the positions of the reflective objects by regarding only the windows and bodies that can reflect the color of the traffic light displays as reflective objects.

Note that the method for acquiring the position of a reflecting object is not limited to the above example. For example, the controller 20 may acquire the position of the side of a building H facing the road as the position of a reflecting object from the vehicle 100's position information D3 obtained from the information acquisition unit 15 and map information D2 regarding the layout of buildings. The controller 20 can acquire the relative position of the vehicle 100 from the side of the building by calculating the difference between the position information (latitude, longitude, orientation) of the side of the building H obtained from the map information D2 and the vehicle 100's position information D3 (latitude, longitude, orientation).

As the vehicle 100 travels, the relative position of the structure H changes in real time. Furthermore, as time passes, the relative position of the vehicle 100 to the surrounding vehicles 200 also changes in real time. The controller 20 can determine the position (relative position) of the reflecting object in real time by repeating the above-described position acquisition process. Alternatively, once the position of the side of the structure H is determined, the controller 20 may then determine the real-time position of the side of the structure H from changes in the position of the vehicle 100. That is, the controller 20 represents the determined position of the structure H as a position vector. The controller 20 can then determine the real-time position of the side of the structure H by sequentially subtracting the displacement vector (amount and direction of movement) of the vehicle 100 from this position vector. The displacement vector of the vehicle 100 can be calculated from the positioning information of the positioning device 113.

<Processing to estimate the reflection position of the target traffic light>
4 is a diagram illustrating the process of estimating the reflection position of the target traffic signal. The controller 20 estimates the reflection position c1 of the target traffic signal F1, F2 based on the acquired positions of the target traffic signals F1, F2 and the position of the reflecting object.

For example, the controller 20 calculates a candidate reflection position c1 from the midpoint p1 of the line segment q1 connecting the target traffic light F1 and the vehicle 100 in a direction perpendicular to the line segment q1, at a height half the height of the target traffic light F1 plus the height of the vehicle 100's camera unit 40. Then, if a candidate reflector is found among the reflectors whose positions have been acquired, the controller 20 estimates that position as the reflection position c1 of the target traffic light F1. This estimation method makes it easy to determine the accurate reflection position c1 of the target traffic light F1 when the vehicle 100 and the target traffic light F1 are located in a straight line along the road and the reflector has a reflective surface parallel to the road. That is, at the candidate reflection position c1, the light of the target traffic light F1 is reflected so that the angle of incidence α = the angle of reflection β, and the reflected light reaches the vehicle 100. When using this method, the controller 20 may exclude the sides of buildings that are not parallel to the road from the list of reflectors, and may only include the sides of buildings that are parallel to the road as reflectors.

Alternatively, the controller 20 may estimate the reflection position c1 of the target traffic light F1 as follows. First, the controller 20 sets multiple candidate points for all surrounding reflecting objects acquired by the controller 20. The multiple candidate points should be set so that they are evenly distributed on the reflecting objects. The multiple candidate points may be set in large numbers so that the calculation load does not become excessive. Next, the controller 20 calculates the line segments connecting the vehicle 100's image capture unit 40 to each candidate point, as well as the line segments of reflected light that reflect through the candidate points when these line segments are considered to be light rays. The reflection direction can be calculated based on the orientation of the reflecting object acquired by the controller 20 so that the reflection angle β = the incident angle α. Furthermore, the controller 20 determines whether the line segments of the reflected light overlap the target traffic light F1, and if so, estimates that the candidate point is the reflection position c1 of the target traffic light F1. By performing this calculation for all candidate points, the controller 20 can estimate one or more reflection positions c1 for the target traffic light F1.

If there are multiple target traffic lights F1, F2, the controller 20 performs the above-mentioned reflection position c1 estimation process for each of the multiple target traffic lights F1, F2.

<Signal display determination process and validity confirmation process>
5 is a diagram illustrating the signal display determination process and the validity confirmation process. In the following, a case where a display determination is made for one of the two target signals F1 will be described, but the controller 20 can also make a display determination for the other target signal F2 in the same way.

The image capture unit 40 captures an image of the side of the vehicle 100 and sends the image data to the controller 20. This image includes the image v1 of the reflection position c1 estimated by the estimation process described above. If, as expected, there is a reflecting object at the reflection position c1 and the acquired position of the target traffic light F1 is normal, the image v1 of the reflection position c1 will include the reflected image Fa of the target traffic light F1. The controller 20 performs traffic light pattern recognition on the image v1 of the reflection position c1. If the result matches the traffic light pattern, the controller 20 determines the display of the target traffic light F1 from the display state (such as the lighting color or lighting position) of the matching reflection image Fa.

Even if the image ahead is obstructed, for example, because a tall vehicle 220 is located in front of the vehicle 100 or the sun K is overlapping behind the target traffic light F1, the controller 20 can determine the display of the target traffic light F1 by using the reflected image Fa.

If the reflected image is recognized as a traffic light in the above pattern recognition process, the controller 20 then performs a validation process to determine whether the above reflected image Fa represents the target traffic light F1. The validation process includes a process of predicting the distorted shape of the target traffic light F1 caused by reflection from a reflecting object, and a process of comparing the predicted distorted shape with the reflected image Fa of the traffic light contained in the image v1 at the above reflection position c1. The controller 20 confirms validity by matching the shape of the reflected image Fa through the above comparison process.

Next, we will explain a specific example of the validation process.

The controller 20 stores data on the standard aspect ratio of traffic lights or the standard shape of traffic lights in advance. Alternatively, the controller 20 may extract data on the shape and orientation of the target traffic light F1 from the detection information of the first detector 11 in advance and store that data. Even if an obstruction occurs in the image ahead of the vehicle 100, there is often no obstruction shortly before the obstruction occurs, and the controller 20 can extract data on the shape and orientation of the target traffic light F1 from the detection information of the first detector 11 during the period when there is no obstruction.

The reflected image Fa of the traffic light is an image of the traffic light viewed from an oblique direction through a reflecting object. Therefore, the aspect ratio of the reflected image Fa of the traffic light changes from the aspect ratio of the actual traffic light. The controller 20 predicts the change in aspect ratio, for example, as a process for predicting the distorted shape of the target traffic light F1. The controller 20 calculates the change in aspect ratio as the distorted shape based on the actual shape and orientation of the target traffic light F1 and the relative positions of the vehicle 100, the reflecting object, and the target traffic light F1. The controller 20 then compares the aspect ratio of the traffic light pattern-recognized in the reflected image Fa with the calculated distorted shape and determines whether the difference between these is within a predetermined allowable error. If the difference is within the allowable error, the controller 20 determines that the validity of the judgment result for the display of the target traffic light F1 is high. On the other hand, if the difference exceeds the allowable error, the controller 20 determines that the validity of the judgment result for the display of the target traffic light F1 is low.

The validity confirmation process is not limited to the process based on comparing the shape of the reflected image Fa described above. Instead of or in addition to this process, the following validity confirmation process may be performed. For example, the controller 20 detects the driving conditions of surrounding vehicles (whether they are driving or not, direction of travel, etc.) and compares whether these driving conditions match the judgment results of the display of the target traffic light F1. In this case, if all conditions match, the controller 20 determines that the validity of the judgment results of the display of the target traffic light F1 is high. On the other hand, the more points that do not match, the less validity the controller 20 determines that the judgment results of the display of the target traffic light F1 are.

Alternatively, as a validity confirmation process, the controller 20 may compare the shape of the traffic light's reflected image Fa not only with the target traffic lights F1 and F2, but also with the other surrounding traffic lights F3 to F8. The controller 20 may then compare the traffic light shape of the reflected image Fa with the distorted shape predicted when the other traffic lights F3 to F8 are reflected, and determine whether the two are similar (for example, whether the aspect ratio is within a predetermined error). In this case, if they are similar, the controller 20 determines that the validity of the judgment result for the display of the target traffic light F1 is low. On the other hand, if they are not similar, the controller 20 determines that the validity of the judgment result for the display of the target traffic light F1 is high.

By performing the above-described validity confirmation process, even if the image v1 of the reflecting object contains a reflected image of another traffic light F7, the controller 20 can reduce the chance of erroneously determining the display of the target traffic light F1 from that reflected image.

<Driving assistance processing>
The controller 20 may perform driving assistance processing based on the determination result of the display of the target traffic light F1. When performing driving assistance processing, the controller 20 may be provided with detection information from various detectors that detect the driving conditions of the vehicle 100 (vehicle speed, steering angle, accelerator operation amount, brake operation amount, etc.). In addition, the controller 20 may be provided with monitoring result information from a device that monitors the situation around the vehicle 100 (presence or absence of people, vehicles, other objects, etc.). Furthermore, the controller 20 may be capable of controlling a notification device 120 (e.g., a display panel) that notifies the driver of information. Then, based on the determination result of the display of the target traffic light F1 and the above detection information and monitoring result information, the controller 20 sends a driving request corresponding to these to the vehicle controller 110 as driving assistance. Furthermore, the controller 20 outputs a notification corresponding to the driving situation from the notification device 120 as driving assistance. Specifically, when the target traffic light F1 indicates a stop and the vehicle 100 is traveling, the controller 20 may send a driving request to the vehicle controller 110 to decelerate the vehicle 100. In addition, the controller 20 may output the driving request sent to the vehicle controller 110, the determination result of the display of the target traffic light F1, etc. from the notification device 120.

<Control flow>
Next, a series of control flows performed by the traffic light information determination device 1 will be described. Fig. 6 is a flowchart showing an example of control processing performed by the traffic light information determination device. The control processing is initiated, for example, when the driver turns on a driving assistance function. When the control processing is initiated, the controller 20 operates the first detector 11 and the second detector 12 to obtain detection information about the areas ahead and to the sides of the vehicle 100 (step S1).

Furthermore, the controller 20 determines whether there is an intersection near the front of the vehicle 100 (step S2). Specifically, the controller 20 determines whether the distance to the upcoming intersection is equal to or less than a threshold. The determination in step S2 may be made by the controller 20 analyzing the detection information from the first detector 11, or by the controller 20 analyzing the vehicle 100's position information D3 and road map information. Alternatively, the determination in step S2 may be made by the controller 20 based on a notification from the vehicle 100's navigation system (notification that the vehicle is approaching an intersection). Note that the determination process in step S2 is not limited to an intersection, and may also be a process for determining proximity to a road location with a traffic light.

If the determination result in step S2 is NO, the controller 20 returns the process to step S1 and repeats the processes of steps S1 and S2. If the determination result in step S2 is YES, the controller 20 proceeds with the process.

As a result, as the process progresses, the controller 20 performs a display determination of the target traffic light from the image captured by the forward-facing image capture unit 45 (step S3). Next, the controller 20 determines whether the result of the determination in step S3 is normal (OK) or poor (NG) (step S4). If an obstruction occurs in the forward-facing image, the determination result is determined to be poor (NG).

If the result of the determination in step S4 is OK, the controller 20 performs one cycle of driving assistance processing using the display determination result of the target traffic light (step S12) and returns to step S1. The driving assistance processing in step S12 corresponds to a portion of the driving assistance processing series. For example, if driving assistance is provided throughout a series of periods from when the vehicle 100 approaches an intersection until it passes the intersection, the processing in step S5 corresponds to one cycle of processing into which the series of periods is divided. Therefore, if there is no obstruction to the forward image from when the vehicle 100 approaches the intersection until it passes the intersection, the loop processing of steps S1 to S4 and S12 continues, and the driving assistance processing of step S12 is repeatedly executed within this loop processing. This repeated execution realizes a series of driving assistance according to the display of the target traffic light. The driving assistance processing is as described above.

On the other hand, if the result of the determination in step S4 is NG, the controller 20 executes the process of acquiring the position of the target traffic light (step S5), the process of acquiring the position of surrounding reflecting objects (step S6), and the process of estimating the reflection position of the target traffic light (step S7). Next, the controller 20 executes the process of determining the display of the target traffic light from the image of the reflection position (step S8), and determines whether the display determination was normal (OK) or defective (NG) (step S9). If the result is OK, the controller 20 executes the process of confirming the validity of the determination result (step S10), and determines whether the validity is above a threshold (step S11). The processes in steps S5 to S8 and S10 are as described above.

In the process of acquiring the position of the target traffic light in step S5, it may be difficult to recognize the target traffic light from the detection information of the first detector 11 due to an obstruction in the forward image. In this case, the controller 20 may calculate the position of the target traffic light using the detection information of the first detector 11 received in step S1 before the obstruction in the forward image occurs and the subsequent displacement vector of the vehicle 100.

If the determination result in step S9 is OK and the determination result in step S11 is YES, the controller 20 executes one cycle of driving assistance processing using the display determination result of the target traffic light (step S12). The controller 20 then returns to step S1. Therefore, even if an obstruction occurs in the forward image, as long as a reflected image of the target traffic light is obtained from surrounding reflectors, the loop processing of steps S1 to S12 continues, and the driving assistance processing of step S12 is repeatedly executed within this loop processing. This repetition then realizes a series of driving assistance processes according to the display of the target traffic light.

On the other hand, if the determination result in step S9 is NG or the determination result in step S11 is NO, the controller 20 stops the driving assistance function and notifies the driver of the stoppage (step S13). Then, the control process ends. The driver can resume the control process by turning on the driving assistance function again. Alternatively, after the process of step S12, if a predetermined condition is met, the controller 20 may return the process to step S1.

The control program of FIG. 6 is stored in a non-transitory computer-readable medium, such as a ROM included in the controller 20. The controller 20 may be configured to read and execute a program stored in a portable, non-transitory recording medium. The portable, non-transitory storage medium may store the control program described above.

As described above, the traffic light information determination device 1 of this embodiment is equipped with a photographing unit 40 that acquires video and a controller 20 that determines the display of the target traffic light. The photographing unit 40 acquires video of reflecting objects located around the vehicle 100. The controller 20 then determines the display of the target traffic light based on the reflected image of the target traffic light contained in the video of the reflecting objects. Therefore, even if the target traffic light is located ahead of the vehicle 100 and obstructs the video ahead of the vehicle 100, the controller 20 can determine the display of the target traffic light.

Furthermore, with the traffic light information determination device 1 of this embodiment, the controller 20 estimates the reflection position of the target traffic light based on the position of the target traffic light and the position of the reflecting object, and determines the display of the target traffic light based on the image of the estimated reflection position. Therefore, the controller 20 can omit processing such as searching for the target traffic light's reflection in areas where the target traffic light's reflection is not located, thereby reducing the load on the controller 20 and improving the speed and accuracy of recognizing the target traffic light's reflection.

In the above embodiment, an example was shown in which a process for estimating the reflection position of a target traffic light was performed. However, the controller 20 may be configured to find the reflected image of the target traffic light without performing a process for estimating the reflection position. For example, the controller 20 can find the reflected image of the target traffic light without estimating the reflection position by performing pattern recognition processing of the reflected image of the traffic light over a wide area of the image captured by the image capture unit 40. Alternatively, the controller 20 can find the reflected image of the target traffic light by acquiring the positions of reflecting objects around the vehicle 100 and performing pattern recognition processing of the reflected image of the traffic light over the entire area in which the reflecting objects are located. For traffic lights on a road without branching off, the traffic light facing the display side corresponds to the target traffic light. Therefore, even without estimating the reflection position, if the controller uses pattern recognition to search for the reflected image of a traffic light facing the display side, this traffic light corresponds to the target traffic light, making it possible to distinguish the target traffic light from a traffic light in another lane.

Furthermore, the traffic light information determination device 1 of this embodiment is equipped with a first detector 11 capable of detecting the location of structures ahead, and a second detector 12 capable of detecting the location of structures to the side. Furthermore, the traffic light information determination device 1 is equipped with an information acquisition unit 15 capable of acquiring map information D1 related to the location of traffic lights, map information D2 related to the location of buildings, and position information D3 of the vehicle 100. The controller 20 acquires the position of the target traffic light based on the detection information of the first detector 11, the map information D1 and position information D3 acquired by the information acquisition unit 15, or both. Furthermore, the controller 20 acquires the position of the reflecting object based on the detection information of the second detector 12, the map information D2 and position information D3 acquired by the information acquisition unit 15, or both. Therefore, the accuracy of the process for acquiring the position of the target traffic light and the process for acquiring the position of the reflecting object can be improved.

Furthermore, according to the traffic light information determination device 1 of this embodiment, the controller 20 further performs a validation process to determine whether the image recognized as the reflected image of the target traffic light actually represents the target traffic light. By referencing the results of the validation process, if the determination result of the display of the target traffic light is incorrect, the controller 20 can reduce the possibility of using the incorrect determination result for other control.

Furthermore, according to the traffic light information determination device 1 of this embodiment, the validity confirmation process includes a process of predicting the distorted shape of the target traffic light due to reflection, and a process of comparing the image recognized as the reflected image of the target traffic light with the predicted distorted shape. This validity confirmation process allows the controller 20 to efficiently reduce the chance of confusing the target traffic light in the vehicle 100's lane at an intersection or the like with a traffic light on another intersecting road.

The above describes an embodiment of the present invention. However, the present invention is not limited to the above embodiment. For example, in the above embodiment, an example was shown in which the traffic light information determination device 1 performs driving assistance processing. However, the traffic light information determination device 1 may not perform driving assistance processing, but may instead supply the signal display determination results to another driving assistance device. When performing driving assistance processing, the traffic light information determination device 1 may also be called a driving assistance device. Other details shown in the embodiment may be modified as appropriate without departing from the spirit of the invention.

REFERENCE SIGNS LIST 1 Traffic light information determination device 11 First detector 12 Second detector 15 Information acquisition unit 20 Controller 40 Photography unit 45 Front photography unit 100 Vehicle 102, 103 Wheels 104 Power source 105 Brake mechanism 106 Driving operation unit 110 Vehicle controller 111 Database 113 Positioning device 115 Navigation system 120 Notification device F1, F2 Target traffic light H Building c1 Reflection position Fa Reflection image

Claims (4)

A traffic light information determination device mounted on a vehicle,
a photographing unit that acquires video;
a controller that determines the display of a target traffic light that displays signal information for the lane in which the vehicle is traveling, and
Equipped with
The photographing unit acquires an image of a front of the vehicle and an image of a reflecting object located around the vehicle,
The traffic light information judgment device is characterized in that when a normal judgment result of the display of the target traffic light cannot be obtained based on the image of the front, the controller judges the display of the target traffic light based on the reflected image of the target traffic light contained in the image of the reflecting object. A traffic light information determination device mounted on a vehicle,
a photographing unit that acquires video;
a controller that determines the display of a target traffic light that displays signal information for the lane in which the vehicle is traveling, and
Equipped with
The photographing unit acquires an image of a reflecting object located around the vehicle,
The controller estimates the reflection position of the target traffic light based on the position of the target traffic light and the position of the reflecting object, and determines the display of the target traffic light based on the reflected image of the target traffic light included in the image of the reflection position of the reflecting object. A traffic light information determination device mounted on a vehicle,
a photographing unit that acquires video;
a controller that determines the display of a target traffic light that displays signal information for the lane in which the vehicle is traveling, and
Equipped with
The photographing unit acquires an image of a reflecting object located around the vehicle,
The controller
determining the display of the target traffic signal based on a reflected image of the target traffic signal included in the image of the reflecting object ;
Furthermore, a validation process is performed to determine whether the reflected image represents the target traffic light;
The validation process includes:
A process of predicting a distorted shape of the target traffic light due to reflection;
comparing the reflected image with the predicted distorted shape;
A traffic light information determination device comprising : a first detector capable of detecting the location of a structure located in front of the vehicle, a first information acquisition unit capable of acquiring map information relating to the location of traffic signals and location information of the vehicle, or both of these;
a second detector capable of detecting the location of a structure located on the side of the vehicle, a second information acquisition unit capable of acquiring map information relating to the location of the structure and location information of the vehicle, or both of these;
Equipped with
A traffic light information determination device as described in any one of claims 1 to 3, characterized in that the controller acquires the position of the target traffic light based on the detection information of the first detector, the acquired information of the first information acquisition unit, or both, and acquires the position of the reflecting object based on the detection information of the second detector, the acquired information of the second information acquisition unit, or both .