JP6935977B2 - Lane mark detector - Google Patents

Description

The present invention relates to a device for detecting a lane mark (white line), and more particularly to a method for determining whether or not the lane mark information detected from a camera image can be used.

The detection of the lane mark is usually performed by analyzing the image data captured by the camera mounted on the vehicle. For example, the lane mark detection device of Patent Document 1 uses the average detection rate of lane marks in a plurality of cycles, stops the detection of lane marks when this value becomes equal to or less than a certain value, and becomes equal to or more than a certain value. When the detection is resumed. Further, the traveling section line recognition device of Patent Document 2 stops the lane recognition process when the random value of the camera image (random factors such as snow, gravel, and puddle in the road image) exceeds the threshold value.

JP-A-2009-205608 Japanese Unexamined Patent Publication No. 2016-046556

The lane mark detection device of Patent Document 1 uses the average detection rate of lane marks, and stops the detection when the average detection rate of the lane marks falls below a certain value. It does not reflect the state. For example, when raindrops or water droplets are attached to the lens of the camera, even if a part of the lane mark is not reflected, if it is detected as a lane mark, it does not affect the judgment of canceling the lane mark detection. Originally, the accurate lane mark state cannot be captured by the camera, so it is desirable to stop the lane mark detection.

FIG. 1 is a graph showing the transition of the detection rate of the lane mark when an image with raindrops is captured, with the vertical axis representing the detection rate and the horizontal axis representing time. When the detection rate is less than 25%, it is judged that the lane mark information cannot be used (detection is stopped), and when the detection rate becomes 50% or more, the lane mark information can be used (resumption of detection). Judgment shall be made. This detection rate is calculated, for example, by averaging for 8 seconds.

In the section (1), the lane mark cannot be detected due to the influence of raindrops, and the detection rate decreases, but since the detection rate is 25% or more, the detection of the lane mark is continued. Then, in section (2), one side of the lane mark is detected, and the detection rate increases. In the section (3), the lane mark cannot be detected again, and when the detection rate becomes less than 25%, the lane mark detection is stopped. Originally, it is desirable to stop the detection of the lane mark at the time of the section (1) where the lane mark cannot be detected, but if the detection rate increases after that, the detection of the lane mark is stopped. It becomes difficult to judge accurately.

The present invention solves such a conventional problem, and an object of the present invention is to provide a lane mark detection device capable of determining whether or not lane mark information can be used in consideration of an actual lane mark situation. do.

The white line detection device according to the present invention relates to an imaging means that images a road on which the own vehicle travels and outputs the captured image data, an extraction means that extracts information about a white line candidate based on the image data, and the white line candidate. A calculation means for calculating the reliability of information, a detection means for detecting a white line based on the information about the white line candidate and outputting white line information for the detected white line, and a white line detected by the detection means based on the reliability. It has a judgment means for judging whether or not the information can be used.

Preferably, the information regarding the white line candidate includes a plurality of line segments whose brightness is equal to or higher than a threshold value when the image data is scanned in a certain direction, and the calculation means calculates the reliability of the plurality of line segments. Preferably, the calculation means has a reliability based on at least one evaluation value of the number of line segments, the start position of the line segment, the straightness of the line segment, the contrast of the line segment, the width of the line segment, and the fluctuation of the width of the line segment. Is calculated. Preferably, the calculation means calculates the average of a plurality of reliabilitys calculated within a certain period, and the determination means determines whether or not the white line information can be used based on the average of the reliabilitys. Preferably, the determination means determines that the white line information cannot be used when the reliability becomes equal to or lower than the first threshold value, and after the white line information becomes unusable, the reliability becomes the first threshold value. When the second threshold value larger than is reached, it is determined that the white line information can be used. Preferably, the determination means outputs the availability of the white line information.

According to the present invention, the reliability of the information regarding the white line candidate extracted based on the image data is calculated, and the availability of the white line information is determined based on the reliability, so that the actual white line situation is reflected. It becomes possible to determine whether or not the white line information can be used, and as a result, the accuracy of white line detection can be improved. For example, if raindrops or the like adhere to the camera, a part of the white line captured from the image data will be lost or blurred, the reliability of the information regarding the white line candidate will decrease, and the judgment as to whether or not the white line information can be used will be determined. Affects.

It is a graph explaining the problem of the conventional lane mark detection apparatus. It is a block diagram which shows the structure of the lane mark detection apparatus which concerns on embodiment of this invention. It is a figure which shows the structure and process of the lane mark candidate extraction means which concerns on Example of this invention. It is a figure explaining the structure and operation of acquisition of the horizontal line which concerns on embodiment of this invention. It is a figure explaining the structure and operation of the grouping of the horizontal line which concerns on embodiment of this invention. It is a figure explaining the structure and operation of the line segment extraction which concerns on embodiment of this invention. It is a figure explaining the calculation method of the contrast which concerns on embodiment of this invention. It is a figure which shows the storage example of the information of the lane mark candidate which concerns on embodiment of this invention. It is a figure which shows the storage example of the lane mark information which concerns on embodiment of this invention. It is a figure which shows the processing flow when the lane mark detection apparatus which concerns on embodiment of this invention is activated. It is a figure which shows the processing flow of the lane mark information useability determination which concerns on embodiment of this invention. It is a figure which shows the processing flow of the lane mark information useability determination which concerns on embodiment of this invention. This is an example of storing the determination result of determining whether or not the lane mark information can be used according to the embodiment of the present invention. It is a table explaining the relationship between each index of the lane mark candidate which concerns on embodiment of this invention, and an evaluation value. It is a graph explaining the effect of the lane mark detection apparatus which concerns on embodiment of this invention. It is a block diagram which shows the electronic device which includes the lane mark detection function which concerns on this Example.

Next, an embodiment of the present invention will be described in detail with reference to the drawings. In a preferred practice, the lane mark detection device according to the present invention includes an image pickup device such as a camera, and has a function of detecting a lane mark (white line) by analyzing the captured image data. The lane mark detection device of the present invention may be implemented by itself, or may be implemented in cooperation with an electronic device having other functions such as a navigation device. For example, the lane mark detection device can be linked with a device having a driving assist function, a function of playing audio / video data, a function of receiving TV / radio broadcasts, a function of executing application software, and the like. ..

In a preferred embodiment, the lane mark detection device uses the lane mark reliability or the average of the lane mark reliability to determine whether or not the lane mark information extracted from the image data captured by the camera can be used. The average lane mark reliability is the lane mark detection rate plus the certainty of the lane mark. The certainty of the lane mark indicates whether or not it has the characteristics as a lane mark by a numerical value (for example, in the range of 0 to 10000 at the maximum).

Next, examples of the present invention will be described. FIG. 2 is a block diagram showing a configuration of a lane mark (white line) detecting device according to an embodiment of the present invention. As shown in the figure, the lane mark detection device 10 of the present embodiment extracts lane mark candidates from the image input means 1 for inputting image data captured by an imaging device such as a camera and the input image data. Based on the lane mark candidate extraction means 2, the lane mark detection means 3 that detects the next lane mark from the extracted lane mark candidate and the estimated position of the lane mark and outputs the lane mark information, and the output lane mark information. Whether or not the lane mark information can be used is determined by the lane mark position estimating means 4 that estimates the next lane mark position and the lane mark information output from the lane mark detecting means 3 based on the reliability of the information about the lane mark candidate. It is configured to include the determination means 5. The lane mark detecting means 3 outputs the lane mark information to the lane mark position estimating means 4, and also outputs the lane mark information to, for example, a driving support function unit that requires the lane mark information. Further, the lane mark information availability determination means 5 outputs the availability of the lane mark information output from the lane mark detection means 3 to, for example, the driving support function unit.

The image input means 1 includes a camera mounted on the vehicle and for taking an image of the road on which the own vehicle travels. When the camera has a wide-angle lens such as a fisheye lens, the image input means 1 has a function of correcting distortion of the captured image data.

Next, the lane mark candidate extraction means 2 extracts lane mark candidates from the image data output from the image input means 1. FIG. 3 shows the configuration and processing flow of the lane mark candidate extraction means 2.

The luminance image acquisition 21 extracts grayscale luminance image data from the image data captured by the image input means 1. As shown in FIG. 4, the acquisition 22 of the horizontal line scans the luminance image data in the horizontal direction (X direction), and extracts the horizontal portion whose luminance is equal to or higher than the threshold value and continues at regular intervals as a line. Do this for the scan range.

The horizontal line grouping 23 groups the horizontal lines that overlap in the X direction with respect to the extracted horizontal lines. For example, as shown in FIG. 5, a plurality of horizontal rains overlapping in the X direction are grouped as group 1, group 2, and group 3.

The line segment extraction 24 projects the horizontal line of each group onto the road surface (Z = 0) with three-dimensional coordinates, and from the midpoint of the line, the linear Y of the line segment that becomes a lane mark candidate by the minimum square method. The slope a of = aX + b (Z = 0) and the Y-intercept b are calculated. The origin is the center position of the car, the front on the X-axis is positive, the left on the Y-axis is positive, and the right is negative. At this time, in the calculated linear equation, if the values of the slopes a and the Y-intercept b are within a certain range, a plurality of groups are combined into one. For example, in FIG. 6, group 2 and group 3 are combined into one, and the calculation is performed again from the information of the horizontal line, and as a result, Y = a2'X + b2'is obtained.

The line width is calculated as follows. Draw a straight line at the left and right ends of the horizontal line using the least squares method, as with the midpoint. With respect to the midpoint of the horizontal line, a straight line passing through this midpoint and orthogonal to the straight line Y = aX + b is drawn, the intersection with the straight line drawn at the left and right ends is calculated, and the distance between the left and right ends is obtained. This is done for the midpoint of each horizontal line, and the distance between the left and right ends at each midpoint is calculated. Next, the width of the line is calculated from the average value of these distances.

The reliability calculation 25 calculates the reliability with respect to the extracted line segment. The reliability is calculated from the following six indexes. Each index is a value in the range 0-100.
1. 1. Number of horizontal lines:
Using the number of horizontal lines that make up the line segment, the evaluation value is calculated as follows.
・ 100 when the number of horizontal lines ≥ 50
-When the number of horizontal lines <50, 50 + the number of horizontal lines For example, in FIG. 6, since the number of horizontal lines is 10 for the line segment of group 1, the evaluation value is 50 + 10 = 60.

2. Line segment start position:
Using the distance to the start position of the line segment, the evaluation value is calculated as follows.
The distance of the start position is the distance between the midpoint of the horizontal line closest to the Y-axis in FIG. 6 and the Y-axis.
In the line segment of group 1 in FIG. 6, 61 corresponds to the distance of the start position.
・ When the distance between the start positions is ≤ 1.0 m, 100
-When 1.0 m <distance of start position <7.0 m, (7.0-distance of start position) x 50 / (7.0-1.0) +50
・ When the distance is 7.0 m ≤ the start position, 50
For example, when the distance of the start position is 2.5 m, it is (7.0-2.5) × 50 / 6.0 + 50 = 87.5, and the value after the decimal point is rounded down to 87.

3. 3. Straightness:
The formula Y = aX + b of the line segment, the midpoint of each horizontal line, and the distance (m) are calculated, and the evaluation value of each midpoint is calculated as follows.
・ Evaluation value of midpoint = 100-distance from midpoint x 1000
However, when the evaluation value of the midpoint <0, it is 0.
After finding the evaluation value of the midpoint of each horizontal line, calculate the average value of these. The evaluation value of the linearity is calculated by the following formula.
_{-Log e} (average of midpoint evaluation values +17.0625) /2.8369-1.0) x 147.31
This replaces the average value of the evaluation values of the midpoints, 0 to 100, with a natural logarithm of 0 to 100.

4. contrast:
FIG. 7 shows a horizontal line on one side of FIG. The calculation of contrast will be described with reference to FIG. 7. The following luminance differences are calculated for the left and right sides of each horizontal line.
-Brightness value at the midpoint of the horizontal line-Brightness average of 1/2 pixel of the horizontal line from the outside 2 pixels from the horizontal line Obtain the average value on both the left and right sides, and use this as the contrast of the horizontal line.
Next, the average value of the contrasts of all the horizontal lines is calculated.
The evaluation value of contrast is calculated as follows using this average.
・ 100 when the average contrast of all horizontal lines is> 50
-When the contrast average of all horizontal lines is ≤50, the contrast average of all horizontal lines x 20/50 + 80

5. Line width:
Whether or not the line width calculated in the line segment extraction 24 is within the appropriate range of the width of the lane mark is expressed as an evaluation value. The evaluation value is calculated as follows.
When 0 ≤ line width <0.10 m, 20 x line width / 0.10 + 80
When 0.10 m ≤ line width <0.50 m, 100
When 0.50 m ≤ line width <1.30 m, 25 x (1.30-line width) / (1.30-0.50) +75
・ 60 when 1.30 m ≤ line width

6. Line width fluctuation:
First, the evaluation value of the fluctuation of the line width of the horizontal line is calculated as follows. Here, the distance of the horizontal line is the distance between the left and right ends of the horizontal line calculated in the line segment extraction 24.
-Variation of horizontal line width = 10000- | Line width-distance of horizontal line | × 10000 / line width However, when the fluctuation of horizontal line width <0, it is set to 0.
This is calculated for all horizontal lines. The fluctuation of the line width is calculated as follows.
・ Average of line width fluctuation of all horizontal lines / 100
FIG. 13 shows a table explaining the relationship between the indicators 1 to 6 and the evaluation value.

The reliability is calculated from these indexes as follows.
・ (Number of horizontal lines x Start position of line segment x Straight line-likeness x Contrast x Line width x Line width fluctuation) / (100 x 100 x 100 x 100)
The value of each index is in the range of 0 to 100, and the reliability value is in the range of 0 to 10000.
For example, the evaluation value of the horizontal line is 60, the evaluation value of the start position of the line segment is 87, the evaluation value of straightness is 90, the evaluation value of contrast is 100, the evaluation value of line width is 100, and the fluctuation of line width is 90. At this time, the reliability is (60 × 87 × 90 × 100 × 100 × 90) / (100 × 100 × 100 × 100) = 4228.

After calculating the reliability for each line segment, the process of the lane mark candidate extraction means 2 ends. At this time, each line segment becomes a lane mark candidate, and the information is stored in the variables candidate [i], i = 0, 1, ..., numCandidates-1 shown in FIG. The number of extracted lane mark candidates is stored in the variable numCandidates, and the information of each lane mark candidate is stored in candidate [i]. The lane mark candidate information candidate [i] stores the coefficients a and b of the linear expression Y = aX + b of the line segment in candidate [i] .a and candidate [i] .b. In addition, the reliability calculated as the lane mark width and the line width in candidate [i] .width is stored in candidate [i] .confidence.

Next, the lane mark detecting means 3 will be described. The lane mark detecting means 3 detects the lane mark from the information of the lane mark candidate shown in FIG. 8 output from the lane mark candidate extracting means 2 and the estimated position of the lane mark output from the lane mark position estimating means 4, and lanes. Output as mark information. The detection of such a lane mark is described in, for example, Japanese Patent Application Laid-Open No. 2013-186660.

The detected lane mark is stored in the lane mark information shown in FIG. The left lane mark is stored in leftLM and the right lane mark is stored in rightLM. These can store the detection results for the past 10 times at the maximum.

laneMark [i] stores the detected lane mark information. Stores the coefficients a and b, the lane mark width width, and the reliability confidence of the linear expression Y = aX + b of the line segment.
The storage position of the latest lane mark information is indicated by useNo. If there is no ray mark information, set useNo to -1. When the lane mark detection device is activated, there is no lane mark information for both leftLM and rightLM, and useNo = -1. If there is lane mark information and the lane mark information corresponding to the lane mark candidate candidate [i], i = 0, 1,…, numCandidates-1 cannot be detected, laneMark [i] is used as the lane mark estimation means. The coefficients a and b of the estimated line segment are stored, the lane mark width of the lane mark information stored immediately before is stored in the lane mark width width, and the reliability confidence stores 0. When the reliability of the stored lane mark information laneMark [i] .confidence is all 0, no lane mark information is set.

When adding lane mark information, if it is full, delete the oldest data and add it. Although not specifically described here, laneMark [i], i = 0, 1,…, 9 are managed as ring buffers.

Next, the lane mark position estimation means 4 will be described. As shown in, for example, Japanese Patent Application Laid-Open No. 2013-186660, the lane mark position estimating means 4 obtains the line segment data next to the lane mark from the detected line segment data of the lane mark and the line segment data of the past lane mark. presume.

Next, the lane mark information availability determination means 5 will be described. The process of the lane mark information availability determination means 5 will be described with reference to FIGS. 10, 11, and 12. FIG. 12 stores the information mngConf used by the lane mark information availability determination means 5. mngConf.state stores the availability judgment result. -1 indicates indefinite, 0 indicates unusable, and 1 indicates usable. mngConf.data [i], i = 0, 1, ..., 89 are areas for storing the reliability of the lane mark detected by the lane mark detecting means 3. Up to 90 can be stored. 90 images correspond to 6 seconds of data when processing 15 images per second, for example. mngConf.curPos indicates the position where the reliability was last stored (the latest reliability storage position). mngConf.numUseds indicates the number of stored reliability data.

FIG. 10 is a processing flow executed when the lane mark detection device is activated. Set the number of stored reliability data mngConf.numUseds to 0 (501), and set the position mngConf.curPos where the reliability was stored last to -1 because it has not been stored yet (502). Result mngConf.state is set to indefinite-1 (503).

FIG. 11 is a flowchart showing the processing of the lane mark information availability determination means 5. First, the left lane mark information determines whether or not there is lane mark information (510). If there is no left lane mark information, the variable confidence is set to 0 (511). If there is, the reliability is acquired from the storage position of the latest lane mark information and set in confidence (512). Similarly, with respect to the right lane mark information, it is determined whether or not there is lane mark information (513), and if there is, the reliability is acquired from the storage position of the latest lane mark information and added to confidence (514).

Next, the variable confidence is divided by 2 to obtain the average of the latest reliability of the left and right lane mark information (515). 1 is added to the position mngConf.curPos for storing reliability data, and the remainder obtained by dividing it by the number of areas 90 is set in mngConf.curPos (516). This is done to use mngConf.data [mngConf.curPos] as a ring buffer.

When the storage position is obtained, the average confidence of the reliability of the left and right lane marks is stored at that position (517). It is determined whether the number of stored reliability data mngConf.numUseds is 90 or more including the current data (518). If it is less than 90, the number of stored reliability data mngConf.numUseds is added by 1 (519), the availability judgment result mngConf.state is set to indefinite -1, and the process ends (520). It is undefined until 90 reliability data can be obtained. If the number is 90 or more, the average value of the stored 90 reliability data is obtained and set in the variable average (521). Usability determination result It is determined whether or not mngConf.state is usable (522). When it is in the usable state, it is determined whether or not the average value average of the reliability data is less than 2000 (523), and when it is less than 2000, it is disabled (524). When it is not usable, it is determined whether or not it is in an unusable state (525), and when it is not available, it is determined whether or not the average value average of the reliability data is 4000 or more (526). When it is 4000 or more, the availability judgment result mngConf.state is set to enable to indicate that the state has returned to the enable state (527).

Next, the effect of this embodiment will be described. FIG. 14 (A) shows the determination of whether or not the lane mark information can be used based on the conventional lane mark detection rate (same as the graph shown in FIG. 1), and FIG. 14 (B) shows the lane mark candidates according to the present embodiment. It shows the judgment of whether or not the lane mark information that reflects the reliability of (line segment) can be used. In this embodiment, it is determined that the lane mark information cannot be used when the average reliability of the line segments is less than 2000, and then it is determined that the lane mark information can be used when the average reliability is 4000 or more. As a result, in the case of this embodiment, it becomes possible to determine whether the lane mark information cannot be used in the section (1) where a part of the lane mark cannot be detected due to raindrops, and the actual situation is reflected. It is possible to judge whether or not the provided lane mark information can be used.

FIG. 15 is a block diagram showing a configuration of an electronic device including a lane mark detection function according to the present embodiment. The electronic device 100 includes an imaging unit 110 that images the periphery of the vehicle, an input unit 120 that receives instructions from the user, an output unit 130 that outputs audio and video, and a storage unit 140 that stores various data and the like. , A control unit 150 that controls each unit. The control unit 150 includes, for example, a ROM / RAM, a microcontroller, a microprocessor, and the like, executes application software, a program, and the like stored in the storage unit 140, and in addition to the lane mark detection function, a navigation function and a driving assist. It is possible to provide a function, a function of playing back audio / video data, and the like.

Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the specific embodiments, and various modifications and modifications are made within the scope of the gist of the invention described in the claims. It can be changed.

1: Image input means 2: Lane mark candidate extraction means 3: Lane mark detection means 4: Lane mark position estimation means 5: Lane mark information availability determination means

Claims (6)

An imaging means that captures the road on which the vehicle is traveling and outputs the captured image data.
An extraction means for extracting information on white line candidates based on the image data, and
A calculation means for calculating the reliability of information regarding the white line candidate, and
A detection means that detects a white line based on the information about the white line candidate and outputs the white line information about the detected white line.
It has a determination means for determining whether or not the white line information detected by the detection means can be used based on the reliability.
The calculation means sequentially accumulates information on the white line candidates output from the extraction means, and is accumulated when the information on the white line candidates is accumulated for a certain period of time or when the information on the white line candidates reaches a certain number. Calculate the average of the reliability of each of the information regarding the white line candidate.
The determination means is a white line detection device that determines whether or not the white line information can be used based on the average of the reliability. The calculation means uses the evaluation values of six indexes of the number of line segments, the start position of the line segment, the straightness of the line segment, the contrast of the line segment, the width of the line segment, and the fluctuation of the width of the line segment. The white line detection device according to claim 1, wherein the white line segment detection device is calculated. The larger the number of the line segments, the higher the evaluation value, the smaller the distance to the start position of the line segment, the higher the evaluation value, and the smaller the variation in the linearity of the line segment, the higher the evaluation value. The higher the contrast, the higher the evaluation value, the higher the evaluation value is if the width of the line segment is within the reference range, and the smaller the fluctuation of the width of the line segment, the higher the evaluation value.
The white line according to claim 2, wherein the calculation means calculates the reliability according to the evaluation values of the six indexes from a mathematical formula prepared in advance, and the higher the evaluation values of the six indexes, the higher the reliability. Detection device. The information regarding the white line candidate includes a plurality of line segments whose brightness is equal to or higher than a threshold value when the image data is scanned in a certain direction, and the calculation means calculates the reliability of the plurality of line segments. 3. The white line detection device according to any one of the three. The determination means determines that the white line information cannot be used when the reliability becomes equal to or lower than the first threshold value, and after the white line information becomes unusable, the reliability becomes higher than the first threshold value. The white line detection device according to any one of claims 1 to 4, wherein the white line information is determined to be usable when a large second threshold value is reached. The white line detection device according to claim 1, wherein the determination means outputs the availability of the white line information.

Images