JP7784912B2 - Image processing device, image processing method and program - Google Patents

Description

The disclosed embodiments relate to an image processing device, an image processing method, and a program.

Technology has been proposed to determine whether a vehicle has run a red light, for example, based on the green, yellow, or red light status of a traffic light detected from information received by an optical beacon or from camera images, as well as vehicle operation status such as braking (see, for example, Patent Document 1).

When using such technology to determine the illumination status of a traffic light using an image captured by a camera, it is necessary to detect the traffic light itself in the image. In such cases, a known method is to use a machine learning algorithm such as deep learning to extract the traffic light to be detected as a rectangular area called a bounding box.

The shape of the luminous area (circle, ellipse) is detected within the extracted rectangular area. Then, for example, by identifying the position of the detected shape within the rectangular area, the luminous color of the traffic light can be determined. For example, if the luminous area is located to the left of the rectangular area, it can be determined that the light is green.

JP 2012-069051 A

However, there is room for further improvement in conventional technology in order to improve the accuracy of determining the illumination status of traffic lights at low cost.

For example, when detecting the shape of the aforementioned luminous area, the Hough transform is a standard method for detecting a "circular" shape, but it is known to have high processing costs, making it difficult to implement in environments with limited computing resources, such as in-vehicle devices.

Furthermore, if the accuracy of the extracted position of the rectangular area itself is low, determining the light emission state of the traffic light based solely on the position of the light-emitting area within that rectangular area may result in an incorrect determination.

One aspect of the embodiment has been made in consideration of the above, and aims to provide an image processing device, image processing method, and program that can improve the accuracy of determining the illumination state of traffic lights at low cost.

An image processing device according to one aspect of the embodiment is an image processing device mounted on a vehicle and includes a control unit. The control unit projects the center coordinates of a rectangular area extracted as a traffic light by image recognition of a camera image onto a virtual plane in real space. The control unit then generates a traffic light image by sampling pixels of the camera image corresponding to sampling points of a virtual rectangular area of a predetermined size centered on the projected center coordinates. The control unit also performs a predetermined color space adjustment on the traffic light image and extracts pixels corresponding to each color component indicating the light emission state of the traffic light from the traffic light image after the color space adjustment. The control unit also determines the light emission state of the traffic light based on the positional relationship of each extracted pixel.

According to one aspect of the embodiment, it is possible to improve the accuracy of determining the illumination state of traffic lights at low cost.

FIG. 1 is a diagram (part 1) outlining an image processing method according to an embodiment. FIG. 2 is a diagram (part 2) outlining the image processing method according to the embodiment. FIG. 3 is a block diagram illustrating an example of the configuration of an image processing apparatus according to the embodiment. FIG. 4 is an explanatory diagram (part 1) of the virtual rectangular area generation process. FIG. 5 is an explanatory diagram (part 2) of the virtual rectangular area generation process. FIG. 6 is an explanatory diagram (part 3) of the virtual rectangular area generation process. FIG. 7 is an explanatory diagram of the signal image generation process. FIG. 8 is an explanatory diagram of the color space adjustment process. FIG. 9 is an explanatory diagram (part 1) of the color feature extraction process. FIG. 10 is an explanatory diagram (part 2) of the color feature extraction process. FIG. 11 is an explanatory diagram (part 1) of the position feature extraction process. FIG. 12 is an explanatory diagram (part 2) of the position feature extraction process. FIG. 13 is an explanatory diagram (part 3) of the position feature extraction process. FIG. 14 is an explanatory diagram (part 4) of the position feature extraction process. FIG. 15 is an explanatory diagram (part 5) of the position feature extraction process. FIG. 16 is an explanatory diagram (part 1) of the determination process. FIG. 17 is an explanatory diagram (part 2) of the determination process. FIG. 18 is an explanatory diagram (part 3) of the determination process. FIG. 19 is a flowchart showing a processing procedure executed by the image processing apparatus according to the embodiment.

Embodiments of the image processing device, image processing method, and program disclosed herein will be described in detail below with reference to the accompanying drawings. Note that the present invention is not limited to the embodiments described below.

Furthermore, in the following description, the image processing device 10 according to the embodiment is assumed to be an in-vehicle device mounted on a vehicle V. The image processing device 10 is a device capable of detecting an object in an image based on an image captured by a camera 3 mounted on the vehicle V, such as a red light violation determination device or a drive recorder. In this embodiment, the object is assumed to be a traffic light 300.

First, an overview of the image processing method according to the embodiment will be described using Figures 1 and 2. Figure 1 is a diagram (part 1) outlining the image processing method according to the embodiment. Figure 2 is a diagram (part 2) outlining the image processing method according to the embodiment.

As already mentioned, when determining the light emission state of a traffic light 300 using an image captured by camera 3, it is necessary to detect the traffic light 300 itself in the image. In such cases, for example, a machine learning algorithm such as deep learning is used to extract the traffic light 300 to be detected as a rectangular area called a bounding box.

The shape of the light-emitting area is detected within the extracted rectangular area. Then, for example, by identifying the position of the detected shape within the rectangular area, the light color of the traffic light 300 can be determined.

As an example, as shown in the left diagram of Figure 1, if a light-emitting area is detected near the left side of rectangular area BB1, it can be correctly determined to be a green light.

When detecting the shape of a light-emitting area, the Hough transform is a standard method for detecting a "circular" shape. However, this method is known to have high processing costs, making it difficult to implement in environments with limited computing resources, such as in-vehicle devices.

Furthermore, if the accuracy of the extracted position of the rectangular area itself is low, determining the light emission state of the traffic light 300 based solely on the position of the light-emitting area within that rectangular area may result in an erroneous determination.

As an example, as shown in the right diagram of Figure 1, if a light-emitting area is detected near the center of rectangular area BB2 extracted at a position shifted to the left of traffic light 300, there is a possibility that the light will be erroneously determined to be yellow even though it is actually green.

In the image processing method according to the embodiment, the control unit 12 (see FIG. 3) of the image processing device 10 projects the center coordinates of a rectangular area BB extracted as a traffic light 300 by image recognition of the camera image onto a virtual plane in real space, samples pixels of the camera image corresponding to sampling points of a virtual rectangular area VB of a predetermined size centered on the projected center coordinates, generates a traffic light image, performs a predetermined color space adjustment on the traffic light image, extracts pixels corresponding to each color component indicating the light emission state of the traffic light 300 from the traffic light image after the color space adjustment, and determines the light emission state of the traffic light 300 based on the positional relationship of each extracted pixel.

Specifically, as shown in FIG. 2, the control unit 12 receives a camera image captured by the camera 3 as input and extracts a rectangular area (step S1), generates a virtual rectangular area (step S2), generates a signal image (step S3), and adjusts the color space (step S4). Based on the signal image after color space adjustment, the control unit 12 then extracts a first feature amount (step S5), extracts a second feature amount (step S6), and determines the light emission state (step S7), and outputs the determination result. The first feature amount is a color feature amount. The second feature amount is a position feature amount.

Step S1, which is outside the dashed rectangle, is an existing technology. In step S1, the control unit 12 extracts the traffic light 300 in the image as a rectangular area BB using a DNN (Deep Neural Network) model trained using a machine learning algorithm.

In step S2, the control unit 12 also converts the center coordinates of the rectangular area BB into world coordinates. World coordinates are points projected onto a virtual plane at the height of the traffic light (for example, 5 m). These world coordinates are regarded as the center point of the traffic light 300 in real space, and a rectangle with a fixed width in the vertical and horizontal directions from this center point in real space is generated as a "virtual rectangular area." Specific examples of step S2 will be described later using Figures 4 to 6.

In step S3, the control unit 12 generates a signal image corresponding to the virtual rectangular area. The control unit 12 samples the virtual rectangular area as an image of a fixed size (e.g., 25 x 25 pixels) and generates the signal image by referencing the pixels on the camera image corresponding to each sampling point. Because the size of the signal image is always the same, a nearby traffic light 300 and a distant traffic light 300 will appear the same, including in terms of scale. By performing steps S2 and S3, the traffic light 300 being judged can always be treated as image information of the same size, regardless of its size, position, or vertical and horizontal orientation in the camera image. This helps to reduce processing costs. A specific example of step S3 will be described later using Figure 7.

Furthermore, in step S4, the control unit 12 adjusts (converts) the signal image to a predetermined color space in accordance with the extraction of the first feature amount in the subsequent step S5. In this embodiment, color space adjustment is performed using the so-called YUV format, but the representation format of color information is not limited. For example, color space adjustment may be performed using the HSV format. The differences in characteristics when using the YUV format and the HSV format will be described later using Figure 8.

In step S5, the control unit 12 inputs the signal image after color space adjustment, sets color flags (hereinafter referred to as "color flags") corresponding to each color component of the traffic light 300 (blue, yellow, red, arrow, and black) for pixels that satisfy the corresponding conditions indicated by the predetermined YUV thresholds (lower and upper limits), and extracts them as color information. Specific examples of step S5 will be described later using Figures 9 and 10.

Note that the term "flag" here refers to an area that stores a truth value indicating the state of each pixel in image processing and its equivalent value. In this embodiment, "setting" a flag means storing a value equivalent to a "true value" in the flag. On the other hand, "lowering" a flag means storing a value equivalent to a "false value" in the flag. "Setting" can also be referred to as "turning on." "Lowering" can also be referred to as "turning off."

In addition, in step S6, the control unit 12 combines color flags, and if the color flags are in a specific positional relationship with each other, sets a position flag (hereinafter referred to as a "position flag" as appropriate) for that pixel and extracts it as position information. Specific examples of step S6 will be described later using Figures 11 to 15.

In step S7, the control unit 12 then determines the illumination state of the traffic light 300 based on the processing result of step S6. In making this determination, the control unit 12 first determines whether each position flag is concentrated within a predetermined range, and if so, sets a corresponding area flag (hereinafter referred to as "area flag" as appropriate). The control unit 12 then calculates the sum of each area flag within the signal image, and determines the illumination state of the traffic light 300 based on this sum. By making a determination on an area-by-area basis based on the area flag, it is possible to reduce erroneous determinations due to spike-like noise. Specific examples of step S7 will be described later using Figures 16 to 18.

As described above, the image processing method according to the embodiment is an image processing method executed by the image processing device 10 mounted on the vehicle V, and includes the steps of: projecting the center coordinates of a rectangular area BB extracted as a traffic light 300 by image recognition of a camera image onto a virtual plane in real space; generating a traffic light image by sampling pixels of the camera image corresponding to sampling points of a virtual rectangular area VB of a predetermined size centered on the projected center coordinates; performing a predetermined color space adjustment on the traffic light image; extracting pixels corresponding to each color component indicating the light emission state of the traffic light 300 from the traffic light image after the color space adjustment; and determining the light emission state of the traffic light 300 based on the positional relationship of each extracted pixel.

Therefore, the image processing method according to the embodiment can improve the accuracy of determining the light emission state of the traffic light 300 at low cost.

Below, we will explain in more detail an example configuration of an image processing device 10 that applies the image processing method according to the above-mentioned embodiment.

Figure 3 is a block diagram showing an example configuration of an image processing device 10 according to an embodiment. Note that Figure 3 shows only the components necessary to explain the features of this embodiment, and omits a description of general components.

In other words, the components shown in Figure 3 are conceptual functional components and do not necessarily have to be physically configured as shown. For example, the specific form of distribution and integration of each block is not limited to that shown, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

Furthermore, in the explanation using Figure 3, the explanation of components that have already been explained may be simplified or omitted.

As shown in FIG. 3, the image processing device 10 includes a memory unit 11 and a control unit 12. The image processing device 10 is also connected to a camera 3 and an output unit 5.

Camera 3 is mounted on vehicle V and is capable of capturing images of the area in front of vehicle V. Note that camera 3 may be, for example, a 360-degree camera capable of capturing images of the entire periphery of vehicle V.

The output unit 5 is an output device for outputting data. The output unit 5 is realized, for example, by a display, a speaker, etc. The output unit 5 may also be an external device that performs information processing based on the data output by the image processing device 10.

The camera 3 and output unit 5 are connected to the image processing device 10 via an in-vehicle network such as a Controller Area Network (CAN). The camera 3 and output unit 5 may also be connected to the image processing device 10 wirelessly via Bluetooth (registered trademark), Wi-Fi (registered trademark), UWB (Ultra Wide Band), etc.

The image processing device 10 is a computer installed in the vehicle V, and as described above, is, for example, a red light running determination device. The image processing device 10 executes at least steps S1 to S7 described using Figure 2.

The storage unit 11 is realized by a storage device such as RAM (Random Access Memory) or flash memory, and in the example of Figure 3, it stores image recognition AI 11a and processing parameter information 11b.

Image recognition AI11a is an AI (Artificial Intelligence) model for image recognition. Specifically, image recognition AI11a is a DNN model or the like trained using a machine learning algorithm. After being loaded as a DNN model into image recognition unit 12a (described below), image recognition AI11a is configured to be able to detect various objects contained in images captured by camera 3 when the images are input to image recognition unit 12a. In this embodiment, image recognition AI11a is configured to be able to detect at least traffic lights 300 as objects.

The processing parameter information 11b is information about various parameters used in the processes executed by the virtual rectangular area generation unit 12b, signal image generation unit 12c, color space adjustment unit 12d, color feature extraction unit 12e, position feature extraction unit 12f, and determination unit 12g, which will be described later.

The processing parameter information 11b includes the various flags mentioned above. The flags may be implemented by registers or represented by program variables. Furthermore, the stored values of flags do not have to be limited to "0" or "1." Therefore, the size of a flag does not have to be limited to 1 bit.

The control unit 12 is a controller, and is implemented by a CPU (Central Processing Unit), MPU (Micro Processing Unit), or the like, executing a program according to an embodiment stored in the storage unit 11 using RAM as a working area. The control unit 12 can also be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

The control unit 12 has an image recognition unit 12a, a virtual rectangular area generation unit 12b, a signal image generation unit 12c, a color space adjustment unit 12d, a color feature extraction unit 12e, a position feature extraction unit 12f, and a determination unit 12g, and realizes or executes the information processing functions and actions described below.

The image recognition unit 12a acquires the captured image captured by the camera 3 and inputs it to the image recognition AI 11a that has read the captured image. Furthermore, the image recognition unit 12a acquires the image recognition result output from the image recognition AI 11a as a result of inputting the captured image, and outputs the acquired image recognition result to the virtual rectangular area generation unit 12b. The image recognition result includes the traffic light 300 extracted as rectangular area BB. In other words, the image recognition unit 12a executes step S1 described above. The camera 3 captures moving images. In response, the image recognition unit 12a extracts rectangular area BB from each frame of the moving images.

The virtual rectangular area generation unit 12b executes step S2 described above. The virtual rectangular area generation process executed by the virtual rectangular area generation unit 12b will now be described in more detail using Figures 4 to 6. Figure 4 is an explanatory diagram (part 1) of the virtual rectangular area generation process. Figure 5 is an explanatory diagram (part 2) of the virtual rectangular area generation process. Figure 6 is an explanatory diagram (part 3) of the virtual rectangular area generation process.

The virtual rectangular area generation unit 12b converts the center coordinates of the rectangular area BB into world coordinates, regards the world coordinates as the center point of the traffic light 300 in real space, and generates a virtual rectangular area, a rectangle with a fixed width in the vertical and horizontal directions from this center point in real space.

As shown in Figure 4, the virtual rectangular area generation unit 12b first assumes that the height of the traffic light 300 is relatively constant, and then uses camera parameters to convert the camera coordinates indicating the center of the rectangular area BB into world coordinates on a virtual plane (hereinafter referred to as the "virtual plane") that assumes the height of the traffic light 300.

Specifically, the virtual rectangular area generation unit 12b calculates a virtual vehicle position that indicates the position of the vehicle V projected onto a virtual plane at the same height in time-series space, based on a rectangular area BB that indicates the position of the traffic light 300 in the image 201 captured by the camera 3 of the vehicle V.

Then, in a coordinate system represented by three mutually orthogonal axes, including a vertical axis, the virtual rectangular area generation unit 12b calculates, as world coordinates, the coordinates of the traffic light 300 on a virtual plane obtained by fixing the coordinate value of the vertical axis.

As shown in Figure 4, a position in real space is represented by coordinate values on the X, Y, and Z axes. The X, Y, and Z axes are perpendicular to each other. The Z axis is a vertical axis. The Y axis is parallel to the direction of travel of the vehicle V in Figure 4. The X axis is perpendicular to the Z and Y axes.

Here, the center of rectangular area BB is defined as center point 221. The virtual rectangular area generation unit 12b uses camera parameters to convert the two-dimensional coordinates (camera coordinates) of center point 221 in image 201 into three-dimensional world coordinates. Then, the distance to center point 221 in these world coordinates is calculated as the distance (Dist) from vehicle V to traffic light 300.

Camera parameters include internal and external parameters of the camera 3. Internal parameters include dot pitch, focal length, distortion (elevation image height TBL), etc. External parameters include the X-, Y-, and Z-axis coordinate values of the camera 3's shooting position, and values representing the camera 3's attitude (PAN, ROLL, TILT).

In addition to the camera parameters, the virtual rectangular area generation unit 12b also references the predetermined height of the traffic light 300 from the road surface. The height of the traffic light 300 is set, for example, in the range of 5.0 m to 5.5 m.

The camera parameters and the height of the traffic light 300 are stored, for example, in the processing parameter information 11b mentioned above.

In this way, by converting to world coordinates, the distance to the traffic light 300 can be calculated inexpensively using a monocular camera.

The virtual rectangular area generation unit 12b then regards the world coordinates as the center point of the traffic light 300 in real space, and generates a rectangle with a fixed width in the vertical and horizontal directions from this center point in real space as the virtual rectangular area VB.

As a result, as shown in Figure 5, in the case of rectangular area BB, the rectangular area BB1 of the closer traffic light 300 is larger and the rectangular area BB2 of the farther traffic light 300 is smaller, but in the case of virtual rectangular area VB, the sizes of virtual rectangular areas VB1 and VB2 are the same regardless of the distance between the traffic lights 300.

The virtual rectangular area generation unit 12b determines the size of the virtual rectangular area VB based on the size of a standard traffic light 300 in real space (e.g., 1250 mm x 450 mm). In this embodiment, the virtual rectangular area generation unit 12b determines both horizontal and vertical traffic lights, and therefore generates the virtual rectangular area VB using a square as the reference shape, with one side being the long side of the traffic light 300 in real space (i.e., 1250 mm, for example), as shown in FIG. 6. Also, as shown in the same figure, the virtual rectangular area generation unit 12b uses the above-mentioned reference shape expanded by a predetermined expansion rate (e.g., 1.1 times) to accommodate variations in the size and position of the input rectangular area BB.

Returning to the explanation of Figure 3, the signal image generation unit 12c executes step S3 described above. Here, the signal image generation process executed by the signal image generation unit 12c will be explained in more detail using Figure 7. Figure 7 is an explanatory diagram of the signal image generation process.

The signal image generation unit 12c generates a signal image corresponding to the virtual rectangular area VB. As already mentioned, the signal image generation unit 12c samples the virtual rectangular area VB as an image of the same size (e.g., 25 x 25 pixels) and generates the signal image by referencing the pixels on the camera image that correspond to each sampling point.

When referencing pixels on the camera image, the inverse conversion from camera coordinates to world coordinates described above is performed. As for the sampling direction, as shown in Figure 7, horizontal traffic lights are scanned from the top left to the bottom right, and vertical traffic lights are scanned from the bottom left to the top right. In other words, the traffic light image generator 12c generates a traffic light image by sampling in a sampling direction in which the pixels of the camera image are sequentially scanned along the longitudinal direction of the traffic light 300 in the camera image. As a result, as shown in the figure, the traffic light image after sampling will look the same whether it is a horizontal traffic light or a vertical traffic light, including the sense of scale and orientation within the image.

Therefore, the traffic light 300 being judged can always be treated as image information of the same size, regardless of its size, position, or vertical/horizontal orientation in the camera image, which helps reduce processing costs.

Furthermore, by reducing the size of the signal image (i.e., increasing the sampling interval in real space), the processing costs of each subsequent process can be reduced.

Returning to the explanation of Figure 3, the color space adjustment unit 12d executes step S4 described above. The color space adjustment process executed by the color space adjustment unit 12d will now be explained with reference to Figure 8. Figure 8 is an explanatory diagram of the color space adjustment process.

The color space adjustment unit 12d adjusts the generated signal image to a predetermined color space in accordance with the color feature extraction process performed by the subsequent color feature extraction unit 12e. In this embodiment, the color space adjustment unit 12d performs color space adjustment in YUV format, as already described.

As shown in Figure 8, the color space adjustment unit 12d can perform color space adjustment using the HSV format in addition to the YUV format. As shown in the figure, the YUV format uses a U-V orthogonal coordinate system to represent colors, and although color separation can be difficult due to the rectangular representation, it has the characteristic of low processing costs. On the other hand, the HSV format uses a H-S polar coordinate system to represent colors, and although color separation is easier than with the YUV format due to the sector representation, it has the characteristic of high processing costs.

In this embodiment, the YUV format is used, with emphasis on processing cost, but it is possible to select the HSV format or another format as appropriate depending on processing cost, such as the allowable computational resources of the image processing device 10.

Returning to the explanation of Figure 3, the color feature extraction unit 12e executes step S5 described above. The color feature extraction process executed by the color feature extraction unit 12e will now be explained in more detail using Figures 9 and 10. Figure 9 is an explanatory diagram (part 1) of the color feature extraction process. Also, Figure 10 is an explanatory diagram (part 2) of the color feature extraction process.

The color feature extraction unit 12e inputs the signal image after color space adjustment, and extracts color information by setting color flags corresponding to each color component for pixels that satisfy the corresponding conditions for each color component of the traffic light 300 indicated by the predetermined YUV thresholds.

Specifically, for example, in the processing parameter information 11b, as shown in Figure 9, the relevant conditions for each of the color components of the traffic light 300 - blue, yellow, red, arrow, and black - are specified by predetermined lower and upper limits of Y, U, and V.

The color feature extraction unit 12e extracts pixels that satisfy the relevant conditions for each color component of the traffic light 300 in the signal image after color space adjustment. Then, for each extracted pixel, the color feature extraction unit 12e sets a color flag corresponding to the relevant color component.

As shown in Figure 10, for example, in the case of a blue light, at least each pixel that satisfies the corresponding condition for the blue component and each pixel that satisfies the corresponding condition for the black component are extracted from the signal image, and a green light color flag is set for the former, and a non-emission signal color flag is set for the latter.

Similarly, for example, in the case of a yellow traffic light, at least each pixel that meets the corresponding conditions for the yellow component and each pixel that meets the corresponding conditions for the black component are extracted from the signal image, and a yellow signal color flag is set for the former, and a non-emission signal color flag is set for the latter.

Similarly, for example, in the case of a red traffic light, at least each pixel that meets the corresponding conditions for the red component and each pixel that meets the corresponding conditions for the black component are extracted from the traffic light image, and a red signal color flag is set for the former, and a non-emission signal color flag is set for the latter.

Similarly, for example, in the case of an arrow signal, at least each pixel that meets the corresponding conditions for the red component and each pixel that meets the corresponding conditions for the arrow component are extracted from the signal image, and a red signal color flag is set for the former, and an arrow signal color flag is set for the latter.

Similarly, for example, in the case of a non-emission signal, at least each pixel in the signal image that meets the corresponding conditions for the black component is extracted, and a non-emission signal color flag is set for these.

Returning to the explanation of Figure 3, the position feature extraction unit 12f executes step S6 described above. Here, the position feature extraction process executed by the position feature extraction unit 12f will be explained in more detail using Figures 11 to 15.

Figure 11 is an explanatory diagram (part 1) of the position feature extraction process. Figure 12 is an explanatory diagram (part 2) of the position feature extraction process. Figure 13 is an explanatory diagram (part 3) of the position feature extraction process. Figure 14 is an explanatory diagram (part 4) of the position feature extraction process. Figure 15 is an explanatory diagram (part 5) of the position feature extraction process.

The position feature extraction unit 12f combines the color flags extracted by the position feature extraction unit 12f, and if the color flags are in a specific positional relationship with each other, it sets a position flag for that pixel and extracts it as position information.

As shown in Figure 11, if there is a black component indicated by the non-emission signal color flag to the right of the blue component indicated by the green signal color flag, and if there is also a black component further to the right, the position feature extraction unit 12f estimates it to be a green signal and sets a green signal position flag for each pixel for which such a green signal color flag is set.

Furthermore, as shown in FIG. 12, if there is a black component to the right of the yellow component indicated by the yellow signal color flag and also to the left, the position feature extraction unit 12f estimates it as a yellow signal and sets a yellow signal position flag for each pixel for which such a yellow signal color flag has been set.

Furthermore, as shown in FIG. 13, if there is a black component to the left of the red component indicated by the red light color flag, and if there is also a black component further to the left, the position feature extraction unit 12f estimates it to be a red light, and sets a red light position flag for each pixel for which such a red light color flag has been set.

Furthermore, as shown in Figure 14, if there is a black component to the right of the black component indicated by the non-emission signal color flag, and if there is another black component further to the right, the position feature extraction unit 12f estimates it to be a non-emission signal and sets a non-emission signal position flag for each pixel for which such a non-emission signal color flag has been set.

Furthermore, as shown in FIG. 15, for arrow signals, if there is an arrow component indicated by the arrow signal color flag within a predetermined range below the center of gravity of the collection of red light position flags extracted in FIG. 13, the position feature extraction unit 12f estimates that this is the position of an arrow signal, and sets an arrow signal position flag for each pixel for which such arrow signal color flag is set.

Returning to the explanation of Figure 3, the determination unit 12g executes step S7 described above. Here, the determination process executed by the determination unit 12g will be explained in more detail using Figures 16 to 18.

Figure 16 is an explanatory diagram (part 1) of the determination process. Figure 17 is an explanatory diagram (part 2) of the determination process. Figure 18 is an explanatory diagram (part 3) of the determination process. Note that Figure 17 is a flowchart showing the processing steps of the first determination process executed by the determination unit 12g, and Figure 18 is a flowchart showing the processing steps of the second determination process executed by the determination unit 12g.

The determination unit 12g determines the light emission state of the traffic light 300 based on the location information extracted by the location feature extraction unit 12f.

When determining the light emission state of the traffic light 300, the determination unit 12g first determines whether the position flags are clustered within a predetermined range, and if so, sets an area flag accordingly.

As shown in FIG. 16, for the green light position flag, yellow light position flag, red light value flag, and arrow light position flag, the determination unit 12g sets a confirmation range of, for example, 2 x 2 pixels of the signal image, and sets each signal area flag if the number of flags in the confirmation range is 3 or more.

In addition, for the non-emission signal position flag, the determination unit 12g checks an area of, for example, 3 x 3 pixels of the signal image, and if the number of flags in the check area is 6 or more, it sets a non-emission signal area flag.

Then, the determination unit 12g calculates the sum of the signal image for each area flag and determines the illumination state of the traffic light 300 based on this sum.

Specifically, for green, yellow, red, and non-light emitting signals, the determination unit 12g determines whether the sum of the green light area flag, yellow light area flag, or red light area flag is greater than or equal to a predetermined value (here, for example, 1) as shown in FIG. 17 (step S11).

Here, if any of the sums of the green light area flags, yellow light area flags, or red light area flags is 1 or greater (Step S11, Yes), the determination unit 12g determines the light emission state (green light, yellow light, or red light) corresponding to the flag with the largest sum among the green light area flags, yellow light area flags, and red light area flags as the determination result (Step S12), and ends the first determination process.

Furthermore, if the sum of the green light area flags, yellow light area flags, or red light area flags is less than 1 (step S11, No), the determination unit 12g determines whether the sum of the non-emission light area flags is equal to or greater than a predetermined value (here, for example, 2) (step S13).

Here, if the sum of the non-luminous signal area flags is 2 or greater (Step S13, Yes), the judgment unit 12g judges it to be a non-luminous signal (Step S14) and ends the first judgment process. Also, if the sum of the non-luminous signal area flags is less than 2 (Step S13, No), the judgment unit 12g judges it to be impossible to judge (Step S15) and ends the first judgment process.

Furthermore, for the arrow signal, the determination unit 12g determines whether the sum of the arrow signal area flags is equal to or greater than a predetermined value (here, for example, 1) (step S21), as shown in FIG. 18, independently of the first determination process in FIG. 17.

Here, if the sum of the arrow signal area flags is 1 or greater (Step S21, Yes), the determination unit 12g sets the arrow signal flag (Step S22) and terminates the second determination process. On the other hand, if the sum of the arrow signal area flags is less than 1 (Step S21, No), the determination unit 12g does not set the arrow signal flag (Step S23) and terminates the second determination process.

Returning to the explanation of Figure 3, the determination unit 12g then outputs the determination results of the first determination process and the second determination process to the output unit 5. If the image processing device 10 is a red light running determination device, the determination unit 12g determines whether or not the vehicle V has run a red light based on the image recognition result of the image recognition unit 12a and the determination results of the first determination process and the second determination process.

In such a case, the determination unit 12g estimates the behavior of the vehicle V based on, for example, the trajectories of each object image-recognized by the image recognition unit 12a. The determination unit 12g also estimates the trajectory of the traffic light 300 relative to the vehicle V based on the image-recognized traffic light 300 and the distance to the traffic light 300 calculated by the virtual rectangular area generation unit 12b.

The determination unit 12g also determines whether the vehicle V has run a red light based on the estimated behavior of the vehicle V, the trajectory of the traffic light 300, and the illumination state of the traffic light 300, which is the determination result of the first determination process and the second determination process.

For example, if vehicle V continues to pass through traffic at a speed greater than or equal to a predetermined time despite the traffic light 300 indicating whether or not there is priority of passage in the direction of travel of vehicle V being red, the determination unit 12g determines that vehicle V has run a red light. Then, the determination unit 12g outputs the determination result to the output unit 5.

In addition, if the image processing device 10 constitutes part of the functionality of a drive recorder, the determination unit 12g may notify the drive recorder of the determination result that the vehicle V has run a red light as an event and cause the drive recorder to record the event.

Next, the processing procedure executed by the image processing device 10 according to the embodiment will be described with reference to FIG. 19. FIG. 19 is a flowchart showing the processing procedure executed by the image processing device 10 according to the embodiment. Note that the processing procedure shown in FIG. 19 is repeated, for example, while the vehicle V is running.

As shown in FIG. 19, the control unit 12 of the image processing device 10 acquires a camera image from the camera 3 (step S101). Then, the control unit 12 extracts a rectangular area BB of the traffic light 300 by image recognition of the acquired camera image (step S102).

Then, the control unit 12 generates a virtual rectangular area VB based on the rectangular area BB (step S103), and generates a signal image based on the virtual rectangular area VB (step S104).

Then, the control unit 12 adjusts the generated signal image to a predetermined color space (step S105) and extracts a first feature (color) from the signal image after color space adjustment (step S106).

The control unit 12 then extracts a second feature (position) based on the extracted first feature (step S107). The control unit 12 then executes the process of determining the light emission state of the traffic light 300 shown in Figures 17 and 18 based on the extracted second feature (step S108). The control unit 12 then outputs the determination result (step S109) and ends the process.

As described above, the image processing device 10 according to the embodiment is an image processing device mounted on a vehicle V and includes a control unit 12. The control unit 12 projects the center coordinates of a rectangular area BB extracted as a traffic light 300 through image recognition of a camera image onto a virtual plane in real space. The control unit 12 then generates a traffic light image by sampling pixels in the camera image that correspond to sampling points in a virtual rectangular area VB of a predetermined size centered on the projected center coordinates. The control unit 12 then performs a predetermined color space adjustment on the traffic light image and extracts pixels corresponding to each color component that indicates the light emission state of the traffic light 300 from the traffic light image after the color space adjustment. The control unit 12 then determines the light emission state of the traffic light 300 based on the positional relationship of each extracted pixel.

Therefore, the image processing device 10 according to the embodiment can improve the accuracy of determining the light emission state of the traffic light 300 at low cost.

The control unit 12 also generates a virtual rectangular area VB using a square as the base shape, with the long side of a standard traffic light 300 in real space as one side.

Therefore, the image processing device 10 according to the embodiment can detect both horizontal and vertical traffic lights.

The control unit 12 also generates a virtual rectangular area VB by expanding the reference shape at a predetermined expansion rate.

Therefore, the image processing device 10 according to the embodiment can accommodate variations in the size and position of the input rectangular area BB.

In addition, the control unit 12 generates the signal image by sampling in a sampling direction in which the pixels of the camera image are sequentially scanned along the longitudinal direction of the traffic light 300.

Accordingly, with the image processing device 10 according to the embodiment, the signal image after sampling will appear the same whether it is a horizontal or vertical traffic light, including the sense of scale and orientation within the image. This allows the traffic light 300 being judged to be treated as image information that always appears the same size, regardless of its size, position, or vertical/horizontal orientation within the camera image, which helps reduce processing costs.

In addition, the control unit 12 extracts pixels from the signal image after the color space adjustment that satisfy the corresponding conditions for each color component of the traffic light 300 indicated by the predetermined threshold values in the color expression format of the color space adjustment.

Therefore, the image processing device 10 according to the embodiment makes it possible to extract pixels that satisfy the corresponding conditions for each color component of the traffic light 300 according to the color expression format of the color space adjustment.

In addition, the control unit 12 uses the YUV format as the color representation format.

Therefore, the image processing device 10 according to the embodiment can perform image processing with reduced processing costs, even in environments with limited computing resources, such as in-vehicle devices.

In addition, the control unit 12 estimates the position of the arrow signal as the position of a pixel that satisfies the corresponding conditions for the color component of the arrow signal within a specified range below the center of gravity of each pixel that satisfies the corresponding conditions for the red component of the traffic light 300.

Therefore, with the image processing device 10 according to the embodiment, the position of the arrow signal can be estimated based on the relative positional relationship with the center of gravity position of each pixel that satisfies the corresponding condition for the red component.

In addition, when determining the light emission state of the traffic light 300 based on the positional relationship of each pixel extracted as corresponding to each color component of the traffic light 300, the control unit 12 determines whether each pixel for each color component is concentrated within a predetermined range, and if so, sets an area flag for each color component, and determines the light emission state of the traffic light 300 based on the sum of each of the area flags set within the signal image.

Therefore, the image processing device 10 according to the embodiment can suppress erroneous determinations caused by spike noise.

An image processing method according to an embodiment is an image processing method executed by an image processing device 10 mounted on a vehicle V, and includes projecting the center coordinates of a rectangular area BB extracted as a traffic light 300 by image recognition of a camera image onto a virtual plane in real space. The image processing method according to an embodiment also includes generating a traffic light image by sampling pixels of the camera image corresponding to sampling points of a virtual rectangular area VB of a predetermined size centered on the projected center coordinates. The image processing method according to an embodiment also includes performing a predetermined color space adjustment on the traffic light image, and extracting pixels corresponding to each color component indicating the light emission state of the traffic light 300 from the traffic light image after the color space adjustment. The image processing method according to an embodiment also includes determining the light emission state of the traffic light 300 based on the positional relationship of each extracted pixel.

Therefore, the image processing method according to the embodiment can improve the accuracy of determining the light emission state of the traffic light 300 at low cost.

The program according to the embodiment also causes the image processing device 10 (equivalent to an example of a "computer") to project the center coordinates of a rectangular area BB extracted as a traffic light 300 by image recognition of a camera image onto a virtual plane in real space. The program according to the embodiment also causes the image processing device 10 to generate a traffic light image by sampling pixels of the camera image corresponding to sampling points of a virtual rectangular area VB of a predetermined size centered on the projected center coordinates. The program according to the embodiment also causes the image processing device 10 to perform a predetermined color space adjustment on the traffic light image and extract pixels corresponding to each color component indicating the light emission state of the traffic light 300 from the traffic light image after the color space adjustment. The program according to the embodiment also causes the image processing device 10 to determine the light emission state of the traffic light 300 based on the positional relationship of each extracted pixel.

Therefore, the program according to the embodiment can improve the accuracy of determining the light emission state of the traffic light 300 at low cost.

In the above-described embodiment, the illumination state of the traffic light 300 and whether the traffic light has been ignored are determined based on image recognition, but it is of course possible to appropriately combine sensor data from various sensors mounted on the vehicle V. For example, the behavior of the vehicle V may be estimated using sensor values from a steering sensor or acceleration sensor, and the vehicle speed may be obtained using sensor values from a speed sensor.

Further advantages and modifications may readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents.

3 Camera 5 Output unit 10 Image processing device 11 Storage unit 11a Image recognition AI
11b Processing parameter information 12 Control unit 12a Image recognition unit 12b Virtual rectangular area generation unit 12c Signal image generation unit 12d Color space adjustment unit 12e Color feature amount extraction unit 12f Position feature amount extraction unit 12g Determination unit BB Rectangular area V Vehicle VB Virtual rectangular area

Claims (10)

An image processing device mounted on a vehicle, the image processing device having a control unit,
The control unit
The center coordinates of a rectangular area extracted as a traffic light by image recognition of the camera image are projected onto a virtual plane in real space.
generating a signal image by sampling pixels of the camera image corresponding to sampling points of a virtual rectangular area of a determined size centered on the projected central coordinates;
performing a predetermined color space adjustment on the signal image, and extracting pixels corresponding to each color component indicating the light emission state of the traffic light from the signal image after the color space adjustment;
determining the light emission state of the traffic light based on the positional relationship of each extracted pixel;
Image processing device. The control unit
The virtual rectangular area is generated using a square as a reference shape, the square having the long side of a standard traffic light in real space as one side.
The image processing device according to claim 1 . The control unit
generating the virtual rectangular area by expanding the reference shape at a predetermined expansion rate;
The image processing device according to claim 2 . The control unit
generating the signal image by sampling in a sampling direction in which pixels of the camera image are sequentially scanned along the longitudinal direction of the traffic light;
4. The image processing device according to claim 1, 2 or 3. The control unit
extracting, from the signal image after the color space adjustment, pixels that satisfy a corresponding condition for each color component of the traffic light indicated by a predetermined threshold value in a color representation format of the color space adjustment;
5. The image processing device according to claim 1. The control unit
The YUV format is used as the color representation format.
The image processing device according to claim 5 . The control unit
The position of a pixel that satisfies the corresponding condition of the color component of an arrow signal within a predetermined range below the center of gravity of each pixel that satisfies the corresponding condition of the red component of the traffic light is estimated to be the position of the arrow signal.
7. The image processing device according to claim 5 or 6. The control unit
When determining the light emission state of the traffic light based on the positional relationship of each pixel extracted as corresponding to each color component of the traffic light, it is determined whether or not each pixel for each color component is concentrated within a predetermined range, and if so, an area flag for each color component is set, and the light emission state of the traffic light is determined based on the sum of each of the set area flags within the signal image.
The image processing device according to any one of claims 1 to 7. An image processing method executed by an image processing device mounted on a vehicle, comprising:
projecting the center coordinates of a rectangular area extracted as a traffic light by image recognition of the camera image onto a virtual plane in real space;
generating a signal image by sampling pixels of the camera image corresponding to sampling points of a virtual rectangular area of a determined size centered on the projected central coordinates;
performing a predetermined color space adjustment on the signal image, and extracting pixels corresponding to each color component indicating the light emission state of the traffic light from the signal image after the color space adjustment;
determining a light emission state of the traffic light based on the positional relationship of each extracted pixel;
An image processing method comprising: Projecting the center coordinates of a rectangular area extracted as a traffic light by image recognition of the camera image onto a virtual plane in real space;
generating a signal image by sampling pixels of the camera image corresponding to sampling points of a virtual rectangular area of a determined size centered on the projected central coordinates;
performing a predetermined color space adjustment on the signal image, and extracting pixels corresponding to each color component indicating the light emission state of the traffic light from the signal image after the color space adjustment;
determining the light emission state of the traffic light based on the positional relationship of each extracted pixel;
A program that causes a computer to execute the following.