JP7535459B2 - Traffic light recognition method and traffic light recognition device - Google Patents

Description

The present invention relates to a traffic light recognition method and a traffic light recognition device.

Patent Document 1 discloses a driving assistance device that recognizes the direction of the arrow lights on a traffic light. This driving assistance device compares a first direction, which is the direction of the arrow lights, with a second direction in which the vehicle is moving, based on an image captured in front of the vehicle. If the driving assistance device determines that the first direction and the second direction do not match, it provides predetermined driving assistance to the driver of the vehicle.

JP 2019-16000 A

In an actual traffic environment, an image captured of the area ahead of a vehicle may contain not only the actual image of a traffic light in front of the vehicle, but also a mirror image of the traffic light reflected on a reflective surface such as the window glass of a building or the side of a large truck. For this reason, it was necessary to accurately recognize the traffic light arrow by determining whether the arrow light was a mirror image reflected on a reflective surface or a real image.

The present invention was made in consideration of these problems, and its purpose is to provide a traffic light recognition method and a traffic light recognition device that can accurately recognize the arrow lights of traffic lights.

In one aspect of the present invention, a traffic light recognition method involves a controller that processes an image in front of a vehicle, which detects the direction of the arrow light of a traffic light present in the image and the position of the arrow light in the image, sets a reference area in the image in which the red light of the traffic light should be present based on the direction and position of the arrow light, determines whether a red light is present in the reference area, and if it is determined that a red light is present in the reference area, recognizes the direction of the detected arrow light as the current indication of the traffic light in front of the vehicle.

The present invention makes it possible to accurately recognize the arrow lights on traffic signals.

FIG. 1 is a block diagram showing the configuration of a traffic light recognition device according to an embodiment. FIG. 2A is an explanatory diagram showing a horizontal traffic light. FIG. 2B is an explanatory diagram showing a vertical traffic light. FIG. 3 is a flowchart showing the process flow of the traffic light recognition method according to the embodiment. FIG. 4 is a flowchart showing the process flow of the traffic light recognition method according to the embodiment. FIG. 5A is an explanatory diagram illustrating a process of setting a reference region based on the light of a left-pointing arrow. FIG. 5B is an explanatory diagram illustrating a process of setting a reference region based on the light of a right-pointing arrow.

Embodiments of the present invention will be described below with reference to the drawings. In the description of the drawings, identical parts are given the same reference numerals and the description will be omitted.

The configuration of the traffic light recognition device according to this embodiment will be described with reference to FIG. 1. The traffic light recognition device includes an imaging unit 10 and a controller 30. The controller 30 is connected to the imaging unit 10 via wired or wireless communication. The controller 30 is also connected to the driving assistance device 50 via wired or wireless communication.

The imaging unit 10, the controller 30, and the driving assistance device 50 are mounted on a vehicle (not shown). The controller 30 may be installed outside the vehicle.

The imaging unit 10 captures images in the direction of travel of the vehicle, for example, in front of the vehicle. The imaging unit 10 is a digital camera equipped with a solid-state imaging element such as a CCD or CMOS, and captures images of the surroundings of the vehicle to obtain image data of the surrounding area. The imaging unit 10 captures an image of a specified range in front of the vehicle by setting the focal length, the lens angle of view, the vertical and horizontal angles of the camera, etc.

The images captured by the imaging unit 10 are input to the controller 30 and stored in a storage unit (not shown). The imaging unit 10 captures images at a predetermined cycle, and the images are input to the controller 30 sequentially.

The controller 30 processes the images input from the imaging unit 10. The controller 30 is a general-purpose microcomputer equipped with a CPU (Central Processing Unit), memory, and input/output units, and is an example of a control device or processing device. A computer program for causing the controller 30 to function as part of the traffic light recognition device is installed in the controller 30. By executing the computer program, the controller 30 functions as multiple information processing circuits equipped in the traffic light recognition device.

The controller 30 includes multiple information processing circuits, including an arrow light detection unit 31, a driving route acquisition unit 32, a reference area setting unit 33, a shape determination unit 34, and an arrow light recognition unit 35.

The arrow light detection unit 31 detects the arrow lights of traffic lights present in the image, and detects the direction of the arrow lights and the position of the arrow lights within the image. The driving route acquisition unit 32 acquires the vehicle's driving route.

The traffic light arrow is illuminated together with the red light. The reference area setting unit 33 sets a reference area in the image where the red light of the traffic light should be present, based on the direction and position of the arrow light. The reference area setting unit 33 then determines whether or not a red light is present in the reference area, i.e., whether or not a pixel corresponding to red is present in the reference area.

When the shape determination unit 34 determines that a red light is present in the reference area, it determines whether the shape of the red light in the reference area is similar to the shape of a red light in a traffic light.

When a red light is present in the reference area and the shape of the red light in the reference area is similar to the shape of a traffic light, the arrow light recognition unit 35 recognizes the direction of the arrow light detected by the arrow light detection unit 31 as the current indication of the traffic light ahead of the vehicle. The arrow light recognition unit 35 then outputs the direction of the arrow light recognized by the arrow light recognition unit 35 to the driving assistance device 50.

The driving assistance device 50 performs driving assistance to assist the driving of a vehicle (not shown) based on the traffic light recognition results obtained by the controller 30. The driving assistance may be, for example, driving the vehicle by automatic driving along a predetermined driving route, or may be assisting the driving operation of the vehicle occupant. The driving assistance may also be controlling a human interface (display device and speaker) to notify the vehicle occupant of the traffic light recognition results.

The driving assistance device 50 does not need to be configured as hardware independent of the controller 30. The functions executed by the driving assistance device 50 may be realized as one of a plurality of information processing circuits included in the controller 30.

The traffic light will be described with reference to FIG. 2A. The traffic light 100 shown in FIG. 2A is a so-called horizontal traffic light, and is equipped with a three-light traffic light main body 110 and an arrow-type traffic light main body 120.

The three-light traffic light main body 110 is a device that lights up three colors, for example, colors representing a green light, a yellow light, and a red light. The three-light traffic light main body 110 is equipped with a blue light unit 111, a yellow light unit 112, and a red light unit 113. The three color lights 111, 112, and 113 are lined up horizontally. The blue light unit 111 is located at the left end, the yellow light unit 112 is located in the center, and the red light unit 113 is located at the right end. The arrangement of the color lights 111, 112, and 113 is predetermined by traffic laws and regulations, etc.

The blue light unit 111 is a light unit that contains a light source and emits a blue light. The light emitted by the blue light unit 111 is round. The light emitted by the blue light unit 111 indicates a green traffic light.

Like blue light unit 111, yellow light unit 112 and red light unit 113 are lights that house a light source and emit yellow and red lights. The light emitted by yellow light unit 112 and red light unit 113 is round. The lights emitted by yellow light unit 112 and red light unit 113 indicate yellow and red lights.

The meaning of the lights on the three-light traffic light main body 110 is determined by traffic laws and regulations that vehicles must follow. For example, a green light (green signal) means "you may proceed," and a red light (red signal) means "stop at the stop position." A yellow light (yellow signal) means "stop at the stop position, unless you are too close to the stop position to stop safely."

The term "green" light refers to the three colors of light emitted by a three-light traffic light main body 110, excluding yellow and red. In other words, a "green" light does not have to be blue in actual chromaticity, but may be a chromaticity equivalent to green.

The arrow-type traffic light body 120 is a device that lights up to indicate arrow signals in specific directions, such as a left arrow signal, an up arrow signal, and a right arrow signal. The arrow-type traffic light body 120 is disposed below the three-light traffic light body 110.

The arrow-type traffic light body 120 is equipped with a left arrow light 121, an up arrow light 122, and a right arrow light 123. The three arrow lights 121, 122, and 123 are lined up horizontally. The left arrow light 121 is located at the left end, the up arrow light 122 is located in the center, and the right arrow light 123 is located at the right end.

That is, the left arrow light 121 is located below the blue light 111. If it were to match the traffic light 100, the left arrow light 121 would be located two lights to the left and one light below the red light 113. The up arrow light 122 is located below the yellow light 112. If it were to match the traffic light 100, the up arrow light 122 would be located one light to the left and one light below the red light 113. The right arrow light 123 is located below the red light 113. If it were to match the traffic light 100, the right arrow light 123 would be located one light below the red light 113. The relative positions of the arrow lights 121, 122, and 123 in relation to the three color lights 111, 112, and 113 are determined in advance by traffic regulations, etc.

The left arrow light 121 is a light that contains a light source and lights a left-pointing arrow. The light of the left arrow light 121 indicates a left-pointing arrow signal. Like the left arrow light 121, the up arrow light 122 and the right arrow light 123 are lights that contain a light source and light an up arrow and a right arrow. The light of the up arrow light 122 and the right arrow light 123 indicates an up arrow signal and a right arrow signal.

The meaning of the lights on the arrow-type traffic light main body 120 is determined by traffic laws and regulations that vehicles must follow. For example, a left-pointing arrow (left-pointing arrow signal) means "you may proceed left regardless of the red light," and an upward-pointing arrow (upward-pointing arrow signal) means "you may proceed straight ahead regardless of the red light." A right-pointing arrow (right-pointing arrow signal) means "you may proceed right regardless of the red light."

The traffic light 100 is not limited to a so-called horizontal traffic light, but may be a vertical traffic light as shown in FIG. 2B. The vertical traffic light 100 includes a three-light traffic light main body 110 and an arrow-shaped traffic light main body 120 located to the right of the three-light traffic light main body 110.

In the three-light traffic light body 110, the three colored lights 111, 112, and 113 are lined up vertically. The blue light 111 is located at the bottom, the yellow light 112 is located in the center, and the red light 113 is located at the top.

In the arrow-type traffic light body 120, the three arrow lights 121, 122, and 123 are lined up in the vertical direction. The left arrow light 121 is located at the bottom, the up arrow light 122 is located in the center, and the right arrow light 123 is located at the top. In other words, the left arrow light 121 is located to the right of the green light 111. When viewed as a traffic light 100, the left arrow light 121 is located one light unit to the right and two lights units downward from the red light 113 as the base point. The up arrow light 122 is located to the right of the yellow light 112. When viewed as a traffic light 100, the up arrow light 122 is located one light unit to the right and one light unit downward from the red light 113 as the base point. The right arrow light 123 is located to the right of the red light 113. When viewed from the perspective of traffic light 100, right arrow light 123 is located one light to the right of red light 113. The relative positions of arrow lights 121, 122, and 123 with respect to the three color lights 111, 112, and 113 are determined in advance by traffic regulations, etc.

Note that the arrow-type traffic light body 120 does not need to have all three arrow lights 121, 122, and 123. The arrow-type traffic light body 120 only needs to have at least one arrow light 121, 122, and 123 according to the right of way that the traffic light 100 gives to traffic.

A series of processes executed by the traffic light recognition device, i.e., the flow of the traffic light recognition method, will be described with reference to Figures 3 and 4. The flowcharts shown in Figures 3 and 4 are called in response to the imaging timing of the imaging unit 10 and are executed by the controller 30.

First, the arrow light detection unit 31 acquires the image input from the imaging unit 10 (step S10). The driving route acquisition unit 32 acquires the vehicle's driving route. The driving route can be acquired from information from a car navigation system that provides guidance on the vehicle's driving route, information such as the direction indicated by the turn signal, etc. Then, the driving route acquisition unit identifies the vehicle's traveling direction based on the vehicle's driving route (step S11). If there is a traffic light ahead of the vehicle, the vehicle's traveling direction corresponds to the vehicle's traveling direction at an intersection where the traffic light is installed. The vehicle's traveling direction can be any one of a left turn, straight ahead, and right turn.

The arrow light detection unit 31 detects the arrow lights of traffic lights present in the image 200, for example, by template matching (step S12). The arrow light detection unit 31 holds an image of a standard arrow light as a template. The arrow light detection unit 31 scans the entire image 200 while shifting the template by one pixel at a time, and calculates, for example, the correlation of the luminance distribution. The arrow light detection unit 31 detects the arrow light (a light that is turned on) at the position of the template where the correlation value is the highest.

The arrow light detection unit 31 may refer to map information including the vehicle's position and a map of the area around the vehicle, or may refer to the detection results of an object detection sensor that detects objects around the vehicle. The arrow light detection unit 31 can quickly detect traffic lights in the image by using different templates depending on the distance between the vehicle and the traffic light.

The method of detecting traffic lights is not limited to template matching. Traffic lights may be detected based on image features by an image processing means that performs processing based on machine learning, such as AdaBoost or SVM (Support Vector Machine). Examples of image features that can be used include Haar-like features that indicate the sum of luminance changes between multiple regions in an image, LBP (Local Binary Pattern) features that can extract intensity textures, and HOG (Histogram of Gradient) that expresses the intensity of image gradients as a directional histogram.

If the arrow light is not detected (NO in step S13), the arrow light detection unit 31 ends this process. On the other hand, if the arrow light is detected (YES in step S13), the arrow light detection unit 31 identifies the characteristics of the arrow light (step S14). Specifically, the arrow light detection unit 31 identifies the direction of the arrow light. Also, as shown in Figures 5A and 5B, the arrow light detection unit 31 sets a rectangular frame RF of approximately the same size as the arrow light so as to surround the periphery of the arrow light. Then, the arrow light detection unit 31 identifies the position (x, y) of the upper left corner of the rectangular frame RF, as well as the vertical size h and horizontal size w of the rectangular frame RF. The arrow light detection unit 31 identifies the position and vertical and horizontal sizes of the arrow light through the position and vertical and horizontal sizes of the rectangular frame RF.

As shown in FIG. 3, if the direction of the arrow light is different from the traveling direction of the vehicle (NO in step S15), the arrow light detection unit 31 ends this process. On the other hand, if the direction of the arrow light is the same as the traveling direction of the vehicle (YES in step S15), the arrow light detection unit 31 determines whether the detected arrow light is an upward arrow light (step S16).

If the arrow light detection unit 31 determines that the arrow light is an upward arrow light (YES in step S16), the arrow light recognition unit 35 recognizes the direction of the arrow light detected by the arrow light detection unit 31, i.e., the upward arrow, as the current indication of the traffic light ahead of the vehicle (step S17). The arrow light recognition unit 35 then outputs the upward arrow to the driving assistance device 50.

On the other hand, if it is determined that the arrow light is not an upward arrow light (NO in step S16), the arrow light detection unit 31 determines whether the detected arrow light is a leftward arrow light (step S18).

If the arrow light is pointing left (YES in step S18), the reference area setting unit 33 sets reference areas RA1 and RA2 (step S19). As shown in FIG. 5A, the reference area setting unit 33 sets a reference area RA1 with a vertical size h and a horizontal size w, based on a first position (x+2w, y-h) determined from the position (x, y) of the arrow light. Similarly, the reference area setting unit 33 sets a reference area RA2 with a vertical size h and a horizontal size w, based on a second position (x-w, y-2h) determined from the position (x, y) of the arrow light.

If it is a right-pointing arrow light (NO in step S18), the reference area setting unit 33 sets reference areas RA1 and RA2 (step S20). As shown in FIG. 5B, the reference area setting unit 33 sets a reference area RA1 with a vertical size h and a horizontal size w, based on a first position (x, y-h) determined from the position (x, y) of the arrow light. Similarly, the reference area setting unit 33 sets a reference area RA2 with a vertical size h and a horizontal size w, based on a second position (x-w, y) determined from the position (x, y) of the arrow light.

The reference area RA1 set in step S19 or step S20 corresponds to the area where a red light should be present if the arrow light detected in the image is that of a real traffic light (a real image traffic light) and this traffic light is a horizontal traffic light. Similarly, the reference area RA2 set in step S19 or step S20 corresponds to the area where a red light should be present if the arrow light detected in the image is that of a real traffic light and this traffic light is a vertical traffic light. As described above, the left-pointing arrow light or the right-pointing arrow light is in a predetermined positional relationship with respect to the red light. The reference area setting unit 33 sets the reference areas RA1 and RA2 based on this positional relationship.

In FIG. 4, the reference area setting unit 33 extracts red lights that are turned on for each of the reference areas RA1 and RA2 (step S21). The reference area RA1 will be used as an example, but the process for the reference area RA2 is the same.

The reference area setting unit 33 extracts pixels having a chromaticity corresponding to a red light based on the chromaticity of each of the multiple pixels present in the reference area RA1. When each pixel of the image is expressed by an RGB value, the light color determination unit 36 converts the RGB value from RGB space to HSV space for each pixel to obtain a hue. As a result, the chromaticity of each pixel is expressed by a hue ranging from, for example, 0 degrees (red) through 180 degrees (blue) to 360 degrees (red). The reference area setting unit 33 has a predetermined hue range with the hue representing the light of the red light device 113 as the median, that is, a hue range corresponding to red. The reference area setting unit 33 evaluates the hue for each pixel, and if the hue of the evaluated pixel is included in the hue range corresponding to red, the pixel is extracted as a red light.

If a red light is present in either of the reference areas RA1 and RA2 (YES in step S22), the shape determination unit 34 calculates the circularity of the red light (step S23). Specifically, the shape determination unit 34 recognizes a group of multiple pixels extracted as a red light as a group pixel, and identifies the shape of the group pixel. The shape is identified by, for example, calculating the circularity, as shown in the following formula 1.
(Formula 1)
Circularity = 4π × (area of pixel group) ÷ (perimeter of pixel group) ²

The circularity shown in Equation 1 approaches 1 the closer the shape of the pixel group is to a circle. In other words, this circularity indicates the similarity between the shape of the group pixel and the light shape of the red lamp 113.

If the circularity of the set pixel, i.e., the red light, is equal to or greater than a predetermined value (YES in step S24), the shape determination unit 34 determines that the red light is similar to a circle and outputs the result of the shape determination to the arrow light recognition unit 35. When a red light is present in the reference area and the result of the determination that the shape of the red light is similar to a circle, the arrow light recognition unit 35 recognizes the direction of the detected arrow light, i.e., a left-pointing arrow or a right-pointing arrow, as the current indication of the traffic light ahead of the vehicle (step S17). The arrow light recognition unit 35 then outputs the left-pointing arrow or the right-pointing arrow to the driving assistance device 50.

When the arrow light is on, the red light is usually on at the same time, so the red light should be extracted in either the reference area RA1 or RA2 in the processing of step S22. However, if the red light is not extracted in the processing of step S22 (NO in step S24), the detected arrow light is determined to be noise and is ignored (step S25).

In this manner, in the traffic light recognition method of this embodiment, the controller 30 detects the direction of the arrow light of the traffic light present in the image and the position of the arrow light within the image, and sets reference areas RA1, RA2 in the image where the red light of the traffic light should be present based on the direction and position of the arrow light. The controller 30 then determines whether or not a red light is present in the reference areas RA1, RA2, and if it determines that a red light is present in the reference areas RA1, RA2, it recognizes the direction of the detected arrow light as the current indication of the traffic light ahead of the vehicle.

The arrow light that emits the arrow light is arranged near the three-light traffic light body 110 according to a predetermined rule for each direction of the arrow light. Therefore, the relative position of the arrow light to the red light can be defined for each direction of the arrow light. Therefore, by setting reference areas RA1 and RA2 according to the direction and position of the arrow light and determining whether a red light exists in the reference areas RA1 and RA2, it is possible to recognize whether the detected arrow light corresponds to a real traffic light. As a result, it is possible to suppress the recognition of a mirror image (virtual image) of a traffic light that is reflected on the window glass of a building or the side of a large truck and is reflected in a left-right inverted manner as an image (real image) of a real traffic light. As a result, even when a traffic light is recognized by image recognition, the arrow light that exists as a virtual image of the traffic light is not recognized as the actual display of the traffic light in front of the vehicle. As a result, the arrow light of the traffic light can be recognized with high accuracy.

In the traffic light recognition method of this embodiment, the controller 30 may set the size of the reference areas RA1 and RA2 based on the size of the arrow light.

The vertical and horizontal sizes of the arrow lights in the image are proportional to the relative distance between the vehicle and the traffic light. Therefore, by determining the size of the reference areas RA1 and RA2 based on the size of the arrow lights, it is possible to set a reference area of appropriate size within the required range.

In the traffic light recognition method of this embodiment, when the direction of the detected arrow light is upward, the controller 30 recognizes the direction of the detected arrow light as the current state of the traffic light without setting reference areas RA1 and RA2. On the other hand, when the direction of the detected arrow light is rightward or leftward, the controller 30 may determine whether or not there is a red light in the reference areas RA1 and RA2, and then recognize the current state of the traffic light ahead of the vehicle.

If the arrow light is pointing upwards, the direction of the arrow light will remain the same even if it is reversed left to right. Therefore, if the arrow light is pointing straight ahead, the direction can be trusted without determining whether or not there is a red light. This reduces the computational load.

In the traffic light recognition method of this embodiment, if the controller 30 determines that there are multiple pixels in the reference area that have a chromaticity corresponding to a red light, the controller 30 may determine that there is a red light in the reference area.

By determining multiple pixels that have a specified range of chromaticity corresponding to red lights, it is possible to accurately determine whether or not a red light is present in the reference area.

In the traffic light recognition method of this embodiment, the controller 30 determines the shape of a set pixel, which is a set of multiple pixels, and if the similarity between the shape of the set pixel and the shape of a red light on a traffic light is equal to or greater than a predetermined judgment value, the direction of the detected arrow light may be recognized as the current indication of the traffic light ahead of the vehicle.

With this configuration, the shape of the red light can be taken into account in addition to the color, making it possible to prevent red objects from being mistaken for red traffic lights. This allows the arrow lights of traffic lights to be recognized with high accuracy.

The above describes the traffic light recognition method and traffic light recognition device according to the present embodiment. The traffic light recognition device according to the present embodiment executes the traffic light recognition method described in the embodiment, and provides the same functions and effects as the traffic light recognition method.

In the first or second embodiment described above, the traffic light recognition device includes a driving assistance device 50. However, the traffic light recognition device may be configured with only the imaging unit 10 and the controller 30.

In the first and second embodiments described above, the light shape of each color light that constitutes the main body of the three-light traffic light is circular. However, the light shape of each color light may be polygonal, such as rectangular.

In the first and second embodiments, an example is shown in which the multiple information processing circuits provided in the controller 30 are realized by software, but it is of course possible to configure the information processing circuits by preparing dedicated hardware for realizing the functions of each information processing circuit. Also, the multiple information processing circuits may be configured by individual hardware.

As described above, an embodiment of the present invention has been described, but the descriptions and drawings that form part of this disclosure should not be understood as limiting this invention. Various alternative embodiments, examples, and operating techniques will become apparent to those skilled in the art from this disclosure.

REFERENCE SIGNS LIST 10 Imaging unit 30 Controller 31 Arrow light detection unit 32 Travel route acquisition unit 33 Reference area setting unit 34 Shape determination unit 35 Arrow light recognition unit 50 Driving support device

Claims (6)

An image of an area in front of the vehicle is captured using an imaging unit mounted on the vehicle;
A controller that processes the image input from the imaging unit,
Detecting a direction of a traffic light arrow present in the image and a position of the traffic light arrow in the image based on the image;
setting a reference region in the image where a red light of the traffic signal should be present based on the direction and position of the arrow light;
determining whether the red light is present in the reference region;
when it is determined that the red light is present in the reference area, the direction of the detected arrow light is recognized as the current indication of a traffic light ahead of the vehicle. The controller:
The traffic light recognition method according to claim 1 , further comprising: setting a size of the reference region to be set in the image based on a size of the arrow light in the image. The controller:
When the direction of the detected arrow light is upward, the direction of the detected arrow light is recognized as the current indication of a traffic light ahead of the vehicle without setting the reference area;
3. The traffic light recognition method according to claim 1, further comprising: determining whether or not the red light is present in the reference area when the detected arrow light is pointing to the right or to the left, and then recognizing the arrow light as the current indication of a traffic light ahead of the vehicle. The controller:
4. The traffic light recognition method according to claim 1, further comprising: determining that the red light is present in the reference area when it is determined that a plurality of pixels having a chromaticity corresponding to the red light are present in the reference area. The controller:
determining a shape of a set of pixels that is a set of the plurality of pixels;
5. The traffic light recognition method according to claim 4, further comprising the step of: recognizing the direction of the detected arrow light as the current indication of the traffic light ahead of the vehicle when a similarity between a shape of the group pixel and a shape of the red light of the traffic light is equal to or greater than a predetermined judgment value. An imaging unit mounted on a vehicle to capture an image of an area in front of the vehicle;
a controller for processing the image input from the imaging unit;
The controller:
Detecting a direction of a traffic light arrow present in the image and a position of the traffic light arrow in the image based on the image;
setting a reference region in the image where a red light of the traffic signal should be present based on the direction and position of the arrow light;
determining whether the red light is present in the reference region;
When it is determined that the red light is present in the reference area, the traffic light recognition device recognizes the direction of the detected arrow light as the current indication of a traffic light ahead of the vehicle.

Images