JP7634382B2 - Road marking recognition method and road marking recognition device - Google Patents

Description

The present invention relates to a road marking recognition method and a road marking recognition device.

The invention described in Patent Document 1 has been known as an invention for recognizing road markings. The invention described in Patent Document 1 synthesizes images taken at different points in time based on the vehicle speed, generates an image in which lane markings (lane dividing lines) such as white lines are elongated, and detects the lane markings from the generated image.

U.S. Patent No. 7,415,134 B2

However, with the invention described in Patent Document 1, even if the actual lane markings are dashed, the lane markings in the synthesized image may appear solid, and when applied to road markings other than lane markings, the arrows may appear double.

The present invention was made in consideration of the above problems, and its purpose is to provide a road marking recognition method and a road marking recognition device that can accurately recognize road markings.

A road marking recognition method according to one aspect of the present invention calculates an optical flow using a plurality of images, and when the calculated optical flow is long, selects more images from the plurality of images compared to when the calculated optical flow is short, and when the images are decomposed into a plurality of small regions corresponding to each other across the plurality of images, selects from the selected images, for each of the small regions, an image in which the brightness of pixels in the small region is equal to or higher than a predetermined value, generates a single composite image using the images selected for each of the small regions, and recognizes road markings using the generated composite image.

The present invention makes it possible to accurately recognize road markings.

FIG. 1 is a configuration diagram of a road marking recognition device 1 according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a stationary object region. FIG. 3 is a diagram illustrating an example of an image selection method. FIG. 4 is a diagram for explaining an example of a method for calculating the luminance in the small region 41. In FIG. FIG. 5 is a diagram illustrating an example of an image with the highest brightness. FIG. 6 is a diagram illustrating an example of a composite image. FIG. 7 is a flowchart illustrating an example of the operation of the road marking recognition device 1.

Embodiments of the present invention will be described below with reference to the drawings. In the description of the drawings, identical parts are given the same reference numerals and the description will be omitted.

An example of the configuration of the road sign recognition device 1 will be described with reference to FIG. 1. As shown in FIG. 1, the road sign recognition device 1 includes a camera 10, a yaw rate sensor 11, a controller 20, a steering actuator 12, an accelerator pedal actuator 13, and a brake actuator 14.

The road sign recognition device 1 may be mounted on a vehicle with an automatic driving function, or on a vehicle without an automatic driving function. The road sign recognition device 1 may also be mounted on a vehicle that is capable of switching between automatic driving and manual driving. The automatic driving function may also be a driving assistance function that automatically controls only some of the vehicle control functions, such as steering control, braking force control, and driving force control, to assist the driver in driving. In this embodiment, the road sign recognition device 1 will be described as being mounted on a vehicle with an automatic driving function.

The camera 10 has an imaging element such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The location where the camera 10 is installed is not particularly limited, but as an example, the camera 10 is installed in front, on the side, or behind the vehicle. The camera 10 continuously captures images of the surroundings of the vehicle at a predetermined cycle. In other words, the camera 10 acquires multiple images captured at different times. The camera 10 detects objects (pedestrians, bicycles, motorbikes, other vehicles, etc.) present around the vehicle, and information in front of the vehicle (division lines, traffic lights, signs, crosswalks, intersections, etc.). The images captured by the camera 10 are output to the controller 20.

The yaw rate sensor 11 detects the yaw rate of the vehicle (the rate of change of the rotation angle in the turning direction). The yaw rate sensor 11 outputs the detected yaw rate to the controller 20.

The controller 20 is a general-purpose microcomputer equipped with a CPU (Central Processing Unit), memory, and input/output units. A computer program for functioning as the road marking recognition device 1 is installed in the microcomputer. By executing the computer program, the microcomputer functions as the multiple information processing circuits equipped in the road marking recognition device 1. Note that here, an example is shown in which the multiple information processing circuits equipped in the road marking recognition device 1 are realized by software, but it is of course possible to configure the information processing circuits by preparing dedicated hardware for executing each information process shown below. The multiple information processing circuits may also be configured by individual hardware. The controller 20 includes an optical flow calculation unit 21, a distance calculation unit 22, a behavior calculation unit 23, a stationary object area setting unit 24, an image selection unit 25, a brightness calculation unit 26, an image synthesis unit 27, a road structure recognition unit 28, and a vehicle control unit 29 as examples of the multiple information processing circuits.

The optical flow calculation unit 21 uses past images and the current image to calculate how much an object in an image has moved. Optical flow is a vector that represents the movement of an object in an image. An example of a calculation method will be described. The image captured at time T1 is image I1, and the image captured at time T2 is image I2. Time T1 is < time T2. The optical flow calculation unit 21 divides image I1 into arbitrary sizes. There is no particular limit to the arbitrary size, but here the horizontal size of the image is N pixels, and the vertical size of the image is M pixels. The arbitrary size is N x M. The optical flow calculation unit 21 compares the area of image I1 (N x M) with the area of image I2 (N x M). Known comparison methods include Sum of Squared Difference (SSD), Sum of Absolute Difference (SAD), and Zero-mean Normalized Cross-Correlation (ZNCC). These methods are well-known techniques and will not be described here. The optical flow calculation unit 21 detects, as related feature points, feature points on the current image that correspond to the same object in real space as an object corresponding to a feature point on a past image. The optical flow calculation unit 21 calculates a combination of related feature points on the past image and feature points on the current image as an optical flow.

The distance calculation unit 22 calculates the distance from the camera 10 to the object corresponding to the feature point from the parallax (shift amount) between the feature points on the past image and the feature points on the current image that are related to each other. In other words, the distance calculation unit 22 calculates the distance from the vehicle to the object corresponding to the feature point. The distance calculation unit 22 calculates the three-dimensional coordinates in real space that correspond to these feature points from the parallax between the feature points on the past image and the feature points on the current image that are related to each other. The distance calculation unit 22 may calculate the distance to the object by combining a radar or a lidar with the camera 10.

The behavior calculation unit 23 calculates the three-axis translational motion and three-axis rotational motion of the vehicle. The three-axis translational motion is the motion of the vehicle in the longitudinal direction, width direction, and up and down direction. The three-axis rotational motion is the rotational motion around three axes including the roll axis, pitch axis, and yaw axis. These motions are calculated using wheel speed sensors, acceleration sensors, rotation sensors, etc. Alternatively, these motions are calculated using images. A known calculation method using images is described in Non-Patent Document 1: "KITT, Bernd; GEIGER, Andreas; LATEGAHN, Henning. Visual odometry based on stereo image sequences with ransac-based outlier rejection scheme. In: 2010 ieee intelligent vehicles symposium. IEEE, 2010. p. 486-492."

The stationary object area setting unit 24 combines the optical flow from the optical flow calculation unit 21, the calculation results from the distance calculation unit 22, and the calculation results from the behavior calculation unit 23 to calculate a three-dimensional movement amount (three-dimensional motion vector) excluding the behavior of the vehicle itself. The stationary object area setting unit 24 compares the calculated absolute movement amount with a threshold value. If the movement amount is greater than the threshold value, the area is determined to be an area where a moving object such as another vehicle exists. The stationary object area setting unit 24 excludes areas where moving objects exist from the image. This sets a stationary object area where no moving objects exist.

The image selection unit 25 selects an image to be used for synthesis from among multiple images according to the magnitude of the optical flow calculated by the optical flow calculation unit 21.

The luminance calculation unit 26 calculates the luminance in a direction perpendicular to the optical flow for a specified region.

The image synthesis unit 27 generates a single image using the pixel with the highest brightness calculated by the brightness calculation unit 26. The image synthesis unit 27 outputs the generated image to the road structure recognition unit 28.

The road structure recognition unit 28 recognizes road structures using the images input from the image synthesis unit 27. In this embodiment, road structures refer to road markings. Road markings are markings stipulated by laws, regulations, or rules, and representative examples include lane markings (so-called white lines), arrows indicating the direction of travel, and lines indicating that the vehicle is approaching a safety zone or an obstacle on the road.

The vehicle control unit 29 controls the steering actuator 12, the accelerator pedal actuator 13, and the brake actuator 14 based on the road markings recognized by the road structure recognition unit 28.

Next, an example of how to set a stationary object area will be explained with reference to Figure 2.

Image 30 shown in FIG. 2 is an image captured by camera 10 while the host vehicle is traveling. In the scene shown in FIG. 2, the host vehicle is traveling in the right lane of a two-lane road. From the host vehicle's perspective, there is another vehicle ahead on the lane in which the host vehicle is traveling. There is also another vehicle in the adjacent oncoming lane. The stationary object area setting unit 24 sets the stationary object area by excluding area 40 in which the other vehicle (moving object) exists from image 30 using the method described above.

Next, an example of an image selection method will be described with reference to Figure 3.

The image selection unit 25 selects an image to be used for synthesis from among the multiple images according to the magnitude of the optical flow. In the example of FIG. 3, the number of multiple images is 16, including images 30, 30a to 30o. As shown in image 30 of FIG. 3, the road markings around the vehicle are faded. Specifically, the lane dividing line separating the right lane from the left lane is faded, the arrow line on the lane in which the vehicle is traveling is also faded, and the line on the right side of the vehicle (the line indicating that the vehicle is approaching a safety zone or an obstacle on the road) is also faded. In this way, the road markings on the road may be faded. It goes without saying that road markings are important in autonomous driving, and therefore, when the road markings are faded, it is required that the image be processed to the extent that the controller 20 can recognize the road markings.

When the same scenery is captured at different times at high speed while driving, the scenery flows in the opposite direction to the traveling direction of the vehicle, so that at the same observation point, the road markings may be blurred in one image while being clear in another image. Therefore, in this embodiment, an image in which the road markings are clearly visible is selected from among multiple images. Here, the road markings are usually clearly visible in a part of an image. Consider a case where the image 30 is finely decomposed into a grid pattern (for example, decomposed into each pixel or into an area of several pixels vertically and horizontally). In this case, it is preferable to select an image in which the road markings are most clearly visible in each of the decomposed small areas. Therefore, in this embodiment, an image in which the road markings are most clearly visible in a certain small area is selected. However, what is used for synthesis is not the entire image, but only the pixel data of the part in which the road markings are most clearly visible. Therefore, selecting an image in which the road markings are most clearly visible means, in more detail, selecting an image in which the road markings are most clearly visible in a certain part from among 16 images. There are no particular limitations on the method for breaking down the small regions, but it is sufficient to break them down into areas large enough that part of the road marking can be recognized.

The optical flow is usually longer the closer to the vehicle and shorter the farther from the vehicle. This is because the scenery changes faster the closer to the vehicle and slower the farther from the vehicle. The image selection unit 25 selects more images as the optical flow is longer. On the other hand, the image selection unit 25 selects fewer images as the optical flow is shorter. As shown in FIG. 3, consider a case where the image 30 is divided into an upper half region R1 and a lower half region R2. In this case, the optical flow of region R1 is shorter than the optical flow of region R2. Therefore, as shown in FIG. 3, in region R1, images are selected by thinning out every other image. On the other hand, in region R2, all 16 images are selected. The reason for thinning out in region R1 is that the scenery changes slowly, making it difficult to detect parts where the road markings are clearly reflected, and to reduce the computational processing load. On the other hand, the reason for selecting all 16 images in region R2 is that the scenery changes quickly, making it easy to detect parts where the road markings are clearly reflected. In region R1, the image in which the road markings are most clearly visible in each small region is selected from the eight images. In region R2, the image in which the road markings are most clearly visible in each small region is selected from the sixteen images. Note that eight and sixteen images are merely examples and are not limiting. It is sufficient to select more images for regions with longer optical flows.

Referring to Figure 4 below, an example of a method for selecting an image in which road markings are most clearly visible in a certain area is described.

The small area 41 indicated by the reference numeral 41 in FIG. 4 corresponds to the small area described above. The luminance calculation unit 26 calculates the luminance of the small area 41 in the direction perpendicular to the optical flow. More specifically, the luminance calculation unit 26 calculates the luminance of the small area 41 of all 16 images of the images 30, 30a to 30o. The luminance increases as the road marking is more clearly reflected. As shown in FIG. 5, it is assumed that the image with the highest luminance in the small area 41 of the images 30, 30a to 30o is the image 30e. In this case, the image synthesis unit 27 uses pixel 50 of the image 30e for the small area 41. The same process is repeated for the other small areas. For each small area in the area R1, the image synthesis unit 27 selects the pixel with the highest luminance from the eight images, and for each small area in the area R2, selects the pixel with the highest luminance from the 16 images. The image synthesis unit 27 then generates one image using the pixels selected for each small area. The image generated in this manner is composed of the pixel with the highest luminance in each small area. The original image from which the brightest pixels in each small region are synthesized is the image captured earliest among the 16 images, i.e., image 30. As shown in FIG. 6, the brightest pixels in each small region of image 30 are synthesized to generate synthesized image 60. As can be seen from synthesized image 60, compared to image 30, the lane markings separating the right lane from the left lane, the arrow markings on the lane in which the vehicle is traveling, and the lines to the right of the vehicle (lines indicating that the vehicle is approaching a safety zone or an obstacle on the road) are clearer. This improves the accuracy of road marking recognition.

In the above example, it is assumed that the vehicle is traveling in a straight line. While traveling, the vehicle may turn right or left. When the vehicle turns right or left, a large change occurs in the image before and after the turn. For this reason, when applying the present invention to a scene in which the vehicle turns right or left, it is sufficient to use the images up until the vehicle turns right or left. For example, in FIG. 3, if the vehicle turns right at the timing when image 30g is captured, it is sufficient to use images 30, 30a to 30g. Note that whether the vehicle has turned right or left can be determined using the yaw rate acquired from yaw rate sensor 11.

In addition, although it has been described that the image with the highest brightness is selected in each small region, this is not limited to the above. An image with a brightness equal to or higher than a predetermined level may be selected. The brightness equal to or higher than a predetermined level may be any brightness that can be recognized by the controller 20.

Next, an example of the operation of the road marking recognition device 1 will be described with reference to the flowchart in FIG. 7.

In step S101, the camera 10 mounted on the vehicle captures images of the surroundings of the vehicle at different times. The process proceeds to step S103, where the controller 20 acquires the yaw rate of the vehicle from the yaw rate sensor 11.

The process proceeds to step S105, where the stationary object region setting unit 24 excludes the region 40 in which moving objects exist from the image 30 (see FIG. 2). This sets a stationary object region in which no moving objects exist.

The process proceeds to step S107, where the image selection unit 25 selects more images as the optical flow is longer. On the other hand, the image selection unit 25 selects fewer images as the optical flow is shorter. The process proceeds to step S109, where the luminance calculation unit 26 calculates the luminance in the direction perpendicular to the optical flow for this small region 41. The process proceeds to step S111, where the image synthesis unit 27 selects the brightest pixel from the eight images for each small region in region R1, and selects the brightest pixel from the 16 images for each small region in region R2 (see Figures 3 to 5). The image synthesis unit 27 then generates one image using the pixels selected for each small region (see Figure 6).

The process proceeds to step S113, where the road structure recognition unit 28 recognizes the road markings using the images input from the image synthesis unit 27. The process proceeds to step S115, where the vehicle control unit 29 controls the steering actuator 12, the accelerator pedal actuator 13, and the brake actuator 14 based on the road markings recognized by the road structure recognition unit 28.

(Action and Effect)
As described above, the road marking recognition device 1 according to this embodiment provides the following advantageous effects.

The road marking recognition device 1 includes a camera 10 that captures images of the surroundings of the vehicle multiple times at different times, and a controller 20 that processes the multiple images captured by the camera 10 at different times. The controller 20 calculates an optical flow using multiple images, and selects more images from the multiple images when the calculated optical flow is long compared to when the calculated optical flow is short. This means that, for example, as shown in FIG. 3, in the region R1 where the optical flow is short, images are selected in a thinned-out manner, while in the region R2 where the optical flow is long, all 16 images are selected. The controller 20 selects pixels having a predetermined brightness or higher from the selected images, and generates one composite image using the selected pixels. The controller 20 then recognizes road markings using the generated composite image. As shown in FIG. 6, in the composite image 60, the lane markings separating the right lane from the left lane, the arrow line on the lane on which the vehicle is traveling, and the line on the right side of the vehicle (the line indicating that the vehicle is approaching a safety zone or an obstacle on the road) are clearer than in the image 30 before synthesis. In this way, this embodiment improves the recognition accuracy of road markings.

The controller 20 also uses optical flow to remove areas 40 in which moving objects exist from the image (see FIG. 2). This eliminates the influence of moving objects.

The controller 20 also calculates the brightness of the image in a direction perpendicular to the optical flow. This makes it possible to include road markings that affect the vehicle's driving in the synthesis.

The controller 20 may also calculate the brightness of the image in a direction perpendicular to the traveling direction of the vehicle. This makes it possible to include road markings that affect the traveling of the vehicle in the synthesis.

The controller 20 also selects the pixel with the highest brightness from the selected images. By using such a pixel with the highest brightness, the road markings in the combined image become clearer.

The controller 20 may select an image from among a plurality of images based on the yaw rate. Specifically, when the controller 20 determines that the host vehicle has turned right or left, it may select images up until the host vehicle has turned right or left, and ignore images thereafter. This can reduce the effect of changes in images when the host vehicle turns right or left.

Each of the functions described in the above embodiments may be implemented by one or more processing circuits. Processing circuits include programmed processing devices, such as processors that include electrical circuitry. Processing circuits also include devices, such as application specific integrated circuits (ASICs) or circuit components, arranged to perform the described functions.

As described above, an embodiment of the present invention has been described, but the descriptions and drawings that form part of this disclosure should not be understood as limiting this invention. Various alternative embodiments, examples, and operating techniques will become apparent to those skilled in the art from this disclosure.

REFERENCE SIGNS LIST 1 Road marking recognition device 10 Camera 11 Yaw rate sensor 12 Steering actuator 13 Accelerator pedal actuator 14 Brake actuator 20 Controller 21 Optical flow calculation unit 22 Distance calculation unit 23 Behavior calculation unit 24 Stationary object area setting unit 25 Image selection unit 26 Brightness calculation unit 27 Image synthesis unit 28 Road structure recognition unit 29 Vehicle control unit

Claims (7)

A road sign recognition method for a road sign recognition device including a camera that captures an image of an area around a vehicle multiple times at different times and a controller that processes the multiple images captured by the camera at different times,
The controller:
Calculating an optical flow using the plurality of images;
selecting more images from the plurality of images when the calculated optical flow is long compared to when the calculated optical flow is short;
selecting, for each of the small regions obtained by decomposing the image into a plurality of small regions corresponding to each other among the plurality of images, an image in which the luminance of pixels of the small region is equal to or greater than a predetermined value from among the selected images;
generating a composite image using the images selected for each of the sub-regions ;
A road marking recognition method, comprising: recognizing a road marking using the generated synthetic image. 2. The method according to claim 1, wherein the controller uses the optical flow to exclude an area in which a moving object exists from the image. 3. The road marking recognition method according to claim 1, wherein the controller calculates a luminance of the image in a direction perpendicular to the optical flow. 3. The road marking recognition method according to claim 1, wherein the controller calculates the luminance of the image in a direction perpendicular to the traveling direction of the host vehicle. 5. The road sign recognition method according to claim 1, wherein the controller selects, for each of the small regions, from the selected images, an image in which a pixel of the small region has the highest brightness . 6. The road marking recognition method according to claim 1, wherein the controller selects an image from the plurality of images based on a yaw rate acquired from a sensor that detects a yaw rate of the host vehicle. A camera that captures images of the surroundings of the vehicle multiple times at different times;
a controller for processing a plurality of images captured by the camera at different times;
The controller:
Calculating an optical flow using the plurality of images;
selecting more images from the plurality of images when the calculated optical flow is long compared to when the calculated optical flow is short;
selecting, for each of the small regions obtained by decomposing the image into a plurality of small regions corresponding to each other among the plurality of images, an image in which the luminance of pixels of the small region is equal to or greater than a predetermined value from among the selected images;
generating a composite image using the images selected for each of the sub-regions ;
A road marking recognition device, comprising: a road marking recognition unit that recognizes a road marking using the generated synthetic image.

Images