JP7838680B2 - Information processing device, information processing method, and program - Google Patents

Description

This disclosure relates to an information processing device, an information processing method, and a non-temporary computer-readable medium on which a program is stored.

Patent Document 1 discloses a technology for recognizing a lane marking closest to a vehicle when there are multiple candidate lines whose brightness is determined to be above a brightness threshold. This technology addresses the problem that when multiple candidate lines with different brightness levels, such as white and yellow lines, are detected, only the white line may always be recognized.

Japanese Patent Publication No. 2019-20957

However, the technology described in Patent Document 1 may not be able to properly detect lane markings depending on their shape, such as when the lane markings indicating one side of the lane in which a vehicle is traveling (e.g., a center line, a lane boundary line) are dashed lines.

The purpose of this disclosure is, in view of the above-mentioned problems, to provide an information processing device, an information processing method, and a non-temporary computer-readable medium on which a program is stored that can appropriately detect lane markings.

In a first aspect of the present disclosure, an information processing device is provided that includes: a recognition unit that recognizes the area of the road markings from a first image of the road surrounding a vehicle; a generation unit that generates a second image in which the recognized area of the road markings is represented on a coordinate plane with the longitudinal and lateral directions of the vehicle as two axes; and an estimation unit that estimates a linear figure representing the road markings based on the second image.

Furthermore, in a second aspect of the present disclosure, an information processing method is provided which recognizes the area of the road markings from a first image of the road surrounding a vehicle, generates a second image in which the recognized area of the road markings is represented on a coordinate plane with the vehicle's longitudinal and lateral directions as two axes, and estimates a linear figure representing the road markings based on the second image.

Furthermore, in a third aspect of the present disclosure, a non-temporary computer-readable medium is provided which stores a program that causes a computer to perform the following processing: recognize the area of the road markings from a first image of the road surrounding a vehicle; generate a second image representing the recognized area of the road markings on a coordinate plane with the vehicle's longitudinal and lateral directions as two axes; and estimate linear figures representing the road markings based on the second image.

From one perspective, it can accurately detect lane markings.

This figure shows an example of the configuration of an information processing device according to the embodiment. This figure shows an example of the hardware configuration of the information processing device according to the embodiment. A flowchart showing an example of processing performed by the information processing apparatus according to the present invention. This figure shows an example of a captured image according to the embodiment. This figure shows an example of an image captured after thresholding according to the embodiment. This figure shows an example of a binarized image in which the dashed line portion of the road markings according to the embodiment has been detected. This figure shows an example of dashed lines for road boundaries in a bird's-eye view according to the embodiment. This figure shows an example of lanes in a bird's-eye view according to the embodiment. This figure shows an example of an output image according to the embodiment.

The principles of this disclosure will be described with reference to several exemplary embodiments. These embodiments are described for illustrative purposes only and should be understood as helping those skilled in the art to understand and implement this disclosure without implying any limitation on the scope of this disclosure. The disclosures described herein may be implemented in various ways other than those described below.
In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meanings as those generally understood by those skilled in the art to which this disclosure belongs.
Embodiments of this disclosure will be described below with reference to the drawings.

(Embodiment 1)
<Structure>
Referring to Figure 1, the configuration of the information processing device 10 according to the embodiment will be described. Figure 1 is a diagram showing an example of the configuration of the information processing device 10 according to the embodiment. The information processing device 10 has a recognition unit 11, a generation unit 12, and an estimation unit 13. Each of these units may be realized through the cooperation of one or more programs installed in the information processing device 10 and hardware such as the processor and memory of the information processing device 10.

The recognition unit 11 recognizes the area of the road markings from the first image, which is a photograph of the road surrounding the vehicle. The generation unit 12 generates a second image in which the area of the road markings recognized by the recognition unit 11 is represented on a coordinate plane with the vehicle's longitudinal direction (e.g., forward direction) and lateral direction as two axes. The estimation unit 13 estimates linear figures representing the road markings based on the second image.

(Embodiment 2)
<Hardware Configuration>
Figure 2 shows an example of the hardware configuration of an information processing device 10 according to an embodiment. In the example in Figure 2, the information processing device 10 (computer 100) includes a processor 101, a memory 102, and a communication interface 103. These parts may be connected by a bus or the like. The memory 102 stores at least a portion of the program 104. The communication interface 103 includes an interface necessary for communication with other network elements.

When program 104 is executed in cooperation with the processor 101 and memory 102, etc., the computer 100 performs at least some of the processing of embodiments of this disclosure. Memory 102 may be of any type. Memory 102 may, in non-limiting examples, be a non-temporary computer-readable storage medium. Memory 102 may also be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. Although only one memory 102 is shown for computer 100, computer 100 may have several physically different memory modules. Processor 101 may be of any type. Processor 101 may include one or more general-purpose computers, dedicated computers, microprocessors, digital signal processors (DSPs), and, in non-limiting examples, processors based on multicore processor architectures. Computer 100 may have multiple processors, such as application-specific integrated circuit chips that are time-dependent to a clock that synchronizes the main processor.

Embodiments of the present disclosure may be implemented in hardware or in dedicated circuitry, software, logic, or any combination thereof. Some embodiments may be implemented in hardware, while others may be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device.

This disclosure also provides at least one computer program product tangibly stored on a non-temporary computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions contained in a program module, and is executed on a device on a target real or virtual processor to perform the processes or methods of this disclosure. The program module includes routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. The functionality of the program modules may be combined or divided among the program modules as desired in various embodiments. The machine-executable instructions of the program modules can be executed locally or within a distributed device. In a distributed device, the program modules can reside on both local and remote storage media.

The program code for performing the methods of this disclosure may be written in any combination of one or more programming languages. These program codes are provided to a processor or controller of a general-purpose computer, a dedicated computer, or other programmable data processing device. When the program code is executed by the processor or controller, the functions/operations in the flowchart and/or block diagrams it implements are performed. The program code may run entirely on a machine, partially on a machine, partially as a standalone software package, partially on a machine, partially on a remote machine, or entirely on a remote machine or server.

Programs can be stored and supplied to a computer using various types of non-temporary computer-readable media. Non-temporary computer-readable media include various types of tangible recording media. Examples of non-temporary computer-readable media include magnetic recording media, magneto-optical recording media, optical disc media, and semiconductor memory. Magnetic recording media include, for example, flexible disks, magnetic tapes, and hard disk drives. Magneto-optical recording media include, for example, magneto-optical disks. Optical disc media include, for example, Blu-ray discs, CD (Compact Disc)-ROM (Read Only Memory), CD-R (Recordable), and CD-RW (ReWritable). Semiconductor memory includes, for example, solid-state drives, mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (random access memory). Programs may also be supplied to a computer using various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. Temporary computer-readable media can supply programs to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

<Processing>
Next, an example of processing by the information processing device 10 according to the embodiment will be described with reference to Figures 3 to 9. Figure 3 is a flowchart showing an example of processing by the information processing device 10 according to the embodiment. Figure 4 is a diagram showing an example of a captured image according to the embodiment. Figure 5 is a diagram showing an example of a captured image after threshold processing according to the embodiment. Figure 6 is a diagram showing an example of a binarized image in which the dashed line portion of the lane markings according to the embodiment has been detected. Figure 7 is a diagram showing an example of a dashed line of lane markings on a bird's-eye view according to the embodiment. Figure 8 is a diagram showing an example of lanes on a bird's-eye view according to the embodiment. Figure 9 is a diagram showing an example of an output image according to the embodiment.

In step S101, the recognition unit 11 acquires a still image (first image) captured by the camera. Here, the recognition unit 11 may, for example, acquire a frame from a video captured by a monocular camera mounted on the vehicle. In the example in Figure 4, the recognition unit 11 acquires a captured image 401 of the road in front of the vehicle (hereinafter also referred to as "the vehicle") that is equipped with the camera (direction of travel when moving straight).

Next, the recognition unit 11 recognizes the dashed area, which is a lane marking (e.g., a center line, lane boundary line) indicating one side of the road lane, within the region of the first image in which the road in front of the vehicle has been photographed (step S102). Here, the recognition unit 11 may, for example, perform thresholding based on the brightness of each pixel. In this case, the recognition unit 11 may, for example, convert the brightness values of pixels within a specific range to a specific color. This makes it possible to, for example, remove unnecessary background information and emphasize contours. The recognition unit 11 may then recognize the dashed area, which is a lane marking, based, for example, the contour of the region in the image.

In the example in Figure 5, the captured image 401 in Figure 4 is subjected to thresholding to generate image 501 in which the contours of the dashed lines, which are the dividing lines, are emphasized. In the example in Figure 6, based on image 501 in Figure 5, a binarized image 601 is generated in which the dashed lines of the dividing lines are distinguished from other areas.

Next, the generation unit 12 generates a bird's-eye view (bird's-eye view; "second image") with the recognized dashed lines mapped, using the forward and horizontal directions of the vehicle as two axes, based on the recognition results from the recognition unit 11 (step S103). Here, the generation unit 12 generates a bird's-eye view in which the values of each of the two axes are set to values corresponding to the distance from the vehicle in each direction of the two axes in real space. This bird's-eye view may be, for example, a view looking vertically downwards, or a view looking diagonally downwards from above the vehicle.

Here, the generation unit 12 may generate the bird's-eye view by, for example, converting the coordinates of pixels in the captured image within the dashed area of the lane markings to coordinates in the bird's-eye view. In this case, for example, the camera may be mounted by the operator so that the optical axis direction of the camera (center of the image) coincides with the forward direction of the vehicle when it is installed on the vehicle. Also, information on shooting conditions such as the camera's field of view may be registered in the information processing device by the operator. Then, the generation unit 12 may, for example, determine a mathematical formula that converts the coordinates of each pixel in the captured image to the forward position and horizontal position of the vehicle in real space, based on the shooting condition information such as the camera's field of view.

Furthermore, when the system is installed on the vehicle by the operator, for example, the forward and horizontal positions of the vehicle in real space may be registered for each coordinate of multiple points in the captured image. In this case, these multiple points may be, for example, the four corners of a trapezoid where the upper base is shorter than the lower base when the far side in the forward direction of the vehicle is the upper base and the near side is the lower base. The generation unit 12 may then determine a mathematical formula to convert the coordinates of each pixel in the captured image to the forward and horizontal positions of the vehicle in real space, based on information that associates the coordinates of each point in the captured image with the forward and horizontal positions of the vehicle in real space.

For example, when detecting lane markings using coordinates in a captured image as shown in Figure 6, lane markings located relatively far from the vehicle (e.g., segment 611) are depicted as points. Therefore, when detecting lane markings using coordinates in a captured image, for example, by a Hough transform, there may be problems with the detection accuracy of lane markings located relatively far from the vehicle. On the other hand, according to this disclosure, since the coordinates in the captured image are converted to coordinates in a bird's-eye view before detecting the lane markings using a Hough transform, the detection accuracy of lane markings located relatively far from the vehicle can be improved.

Next, the estimation unit 13 divides the bird's-eye view into multiple areas according to the distance in the forward direction of the vehicle (step S104). Here, the estimation unit 13 may, for example, divide the bird's-eye view into a first area including a range less than a specific distance in the forward direction of the vehicle, and a second area including a range greater than or equal to the said specific distance in the forward direction of the vehicle.

In the example shown in Figure 7, the estimation unit 13 divides the bird's-eye view 701 into areas 711, 712, 713, 714, and 715 in order of proximity to the vehicle, at specific distances (for example, 10 m) in the forward direction of the vehicle.

Next, the generation unit 12 detects line segments of the boundary lines for each area of the bird's-eye view (step S105). Here, the generation unit 12 may, for example, detect the line segments of the boundary lines for each area by performing a Hough transform for each area. Note that the generation unit 12 is not limited to the Hough transform and may detect line segments using other known methods.

Next, the estimation unit 13 estimates a linear figure representing the road lane markings based on the line segments detected in each area (step S106). Here, the estimation unit 13 may, for example, estimate a line passing through a position on the first line segment detected in the first area and a position on the second line segment detected in the second area as one side of the road lane. This improves the detection accuracy of the linear figure representing the curved lane markings, even in cases where, for example, the area in front of the vehicle is curved.

In this case, the estimation unit 13 may, for example, detect a straight line passing through a specific point on the line segment detected in each area as one side of the road lane. This specific point may, for example, be a point on the line segment closest to the midpoint in the forward direction of the vehicle in each area. In this case, in the example of Figure 7, the straight lines passing through points 721, 722, 723, 724, and 725 on the line segment at the midpoint closest to the midpoint in the forward direction (vertical direction in Figure 7) of the vehicle in each area 711, 712, 713, 714, and 715 are detected as one side of the road lane. As a result, as shown in the bird's-eye view 801 of Figure 8, for example, the leftmost line 811 and the rightmost line 812 of the vehicle's lane are detected.

The estimation unit 13 may estimate a linear figure representing a road lane marking based on a line segment among the detected multiple line segments that satisfies a predetermined angle condition. In this case, the estimation unit 13 may determine a line segment among the multiple line segments detected in the second area that satisfies a predetermined angle condition from the direction in which the first line segment detected in the first area extends toward the second area as the second line segment forming one side of the same lane as the first line segment. This improves the detection accuracy of curved lane markings, for example, even when the road in front of the vehicle is curved. In this case, the estimation unit 13 may determine a line segment among the multiple line segments detected in the second area that lies within a specific angle (e.g., 15°) centered on the direction in which the first line segment extends toward the second area, from the end of the first line segment closer to the second area, as the second line segment. Furthermore, if other line segments satisfying a predetermined angle condition are detected for a particular line segment, the estimation unit 13 may determine a higher confidence level for the estimation of that particular line segment.

Furthermore, if the width of the detected road lane is less than a threshold, the estimation unit 13 may detect (correct, modify) a line of a specific width from one side of the lane, based on the line segment on the side of the lane with higher reliability in line segment detection by Hough transform, as the line on the other side of the lane. This can improve the detection accuracy of lanes located relatively far from the vehicle, for example. In this case, the estimation unit 13 may, for example, in the Hough transform, first project (transform) the coordinates (x, y) of each pixel on the line segment of the lane markings in the bird's-eye view onto a point (ρ, θ) in a two-dimensional polar coordinate space with distance ρ and angle θ. Here, distance ρ may be, for example, the length when a perpendicular is drawn from the origin to a line passing through coordinates (x, y). Also, angle θ may be, for example, the angle made with the x-axis when a perpendicular is drawn from the origin to a line passing through coordinates (x, y). The estimation unit 13 may, for example, detect line segments in the xy coordinate system of the bird's-eye view based on the projected points (ρ, θ). The estimation unit 13 may, for example, determine a higher reliability for the line segments in the xy coordinate system of the bird's-eye view detected based on the points (ρ, θ) as the number of projected points (ρ, θ) increases.

Furthermore, the estimation unit 13 may calculate the confidence level of the estimation for the linear figures representing the estimated road markings, and identify the two linear figures representing the markings that form both ends of the road lanes based on the calculated confidence level. Also, for example, when a camera mounted on a vehicle continuously captures images, the estimation unit 13 may determine a higher confidence level for the linear figure estimated in the current frame if the distance between the linear figure representing the markings estimated in past frames and the linear figure estimated in the current frame is small. Furthermore, the estimation unit 13 may calculate the width between the linear figures representing the estimated road markings, and calculate the confidence level based on a comparison of the calculated width with a predetermined reference value representing the width (width) of the road lanes. In this case, the estimation unit 13 may determine a higher confidence level for a particular line segment if, for example, another parallel line segment is detected at a distance of the predetermined reference value away from a specific detected line segment. Furthermore, the estimation unit 13 may estimate the lane markings forming the left edge of the road lane in each area based on the linear figure with the highest reliability among the linear figures that are within a first lateral distance (for example, within 3m to the left of the center of the vehicle). Similarly, the estimation unit 13 may estimate the lane markings forming the right edge of the road lane in each area based on the linear figure with the highest reliability among the linear figures that are within a second lateral distance (for example, within 3m to the right of the center of the vehicle).

Next, the estimation unit 13 outputs information based on the estimated road markings (step S107). Here, the estimation unit 13 may output information indicating the distance from one end of the vehicle's lane to the vehicle itself. Alternatively, the estimation unit 13 may output an image in which the lines on both sides of the vehicle's lane are superimposed on the captured image, as shown in Figure 9. In the example in Figure 9, the lines 911 and 912, which are the leftmost line 811 and the rightmost line 812 of the vehicle's lane in Figure 8, respectively, converted from the coordinates of the bird's-eye view to the coordinates of the captured image, are superimposed on the captured image 401 in Figure 4, resulting in the output image 901.

This allows for the generation of simulation data for the development of autonomous driving systems or driver assistance systems that mimic the actual swaying of a vehicle during driving. It also supports the verification of the operation of a vehicle equipped with an autonomous driving system or driver assistance system during actual driving. Furthermore, it enables real-time detection of the vehicle's lane while it is in motion.

<Different example>
The information processing device 10 may be a device contained in a single enclosure, but the information processing device 10 of this disclosure is not limited to this. Each part of the information processing device 10 may be implemented by cloud computing, which is composed of, for example, one or more computers. Such an information processing device 10 is also included as an example of the "information processing device" of this disclosure.

Furthermore, this disclosure is not limited to the embodiments described above, and may be modified as appropriate without departing from its spirit.

Some or all of the above embodiments may also be described as follows, but are not limited to the following:
(Note 1)
A recognition unit that recognizes the area of the road markings from a first image of the road surrounding the vehicle,
A generation unit generates a second image in which the recognized area of the road lane markings is represented on a coordinate plane with the longitudinal and lateral directions of the vehicle as two axes,
Based on the second image, an estimation unit estimates linear figures representing the road markings,
An information processing device equipped with the following features.
(Note 2)
The estimation unit detects a plurality of line segments based on the area of the road markings shown in the second image, and estimates a linear figure representing the road markings based on the detected plurality of line segments.
The information processing device described in Appendix 1.
(Note 3)
The estimation unit divides the second image into a first area including a range less than a predetermined distance in the longitudinal direction of the vehicle and a second area including a range greater than or equal to the predetermined distance in the longitudinal direction of the vehicle, and estimates a linear figure representing the road markings based on the first line segment detected in the first area and the second line segment detected in the second area.
The information processing device described in Appendix 2.
(Note 4)
The information processing device according to Appendix 2 or 3, wherein the estimation unit estimates a linear figure representing the road markings based on the line segments among the detected plurality of line segments that satisfy predetermined angle conditions.
(Note 5)
The estimation unit determines, based on the direction in which the first line segment extends toward the second area, a second line segment from among the multiple line segments detected in the second area that forms a linear figure representing the same demarcation line as the first line segment.
The information processing device described in Appendix 3.
(Note 6)
The estimation unit calculates the confidence level of the estimation for the linear figures representing the estimated road lane markings, and identifies the linear figures representing the two lane markings that form both ends of the road lane based on the calculated confidence level.
The information processing device described in Appendix 2 or 3.
(Note 7)
The estimation unit calculates the width between linear figures representing the estimated road markings, and calculates the reliability based on a comparison of the calculated width with a predetermined reference value representing the width of the road lanes.
The information processing device described in Appendix 6.
(Note 8)
From the first image, which captures the road surrounding the vehicle, the area of the road's lane markings is recognized.
A second image is generated in which the recognized area of the road lane markings is represented on a coordinate plane with the vehicle's longitudinal and lateral directions as two axes.
Based on the second image, a linear figure representing the road markings is estimated.
Information processing methods.
(Note 9)
From the first image, which captures the road surrounding the vehicle, the area of the road's lane markings is recognized.
A second image is generated in which the recognized area of the road lane markings is represented on a coordinate plane with the vehicle's longitudinal and lateral directions as two axes.
A non-temporary computer-readable medium containing a program that causes a computer to perform a process of estimating linear figures representing the road markings based on the second image.

1. Monitoring system 10. Information processing device 11. Recognition unit 12. Generation unit 13. Estimation unit.

Claims (6)

A recognition unit that recognizes the area of the road markings from a first image of the road surrounding the vehicle,
A generation unit generates a second image in which the recognized area of the road lane markings is represented on a coordinate plane with the longitudinal and lateral directions of the vehicle as two axes,
An estimation unit that detects multiple line segments based on the area of the road markings shown in the second image and estimates a linear figure representing the road markings,
Equipped with ,
The estimation unit,
The confidence level of the estimation is calculated for the linear figures representing the estimated road markings.
Based on the calculated confidence level, a linear figure representing the two lane markings forming both ends of the road lane is identified.
The width between the two linear figures representing the two identified lane lines is calculated, and the calculated width is compared with a predetermined standard value representing the width of the road lane.
An information processing device that, if the calculated width is less than a predetermined standard value representing the width of the road lanes, detects the other linear figure at the position of the predetermined standard value from the linear figure representing one of the lane lines, which has a higher reliability, as the lane line of the other lane . The estimation unit divides the second image into a first area including a range less than a predetermined distance in the longitudinal direction of the vehicle and a second area including a range greater than or equal to the predetermined distance in the longitudinal direction of the vehicle, and estimates a linear figure representing the road markings based on the first line segment detected in the first area and the second line segment detected in the second area.
The information processing apparatus according to claim 1 . The information processing apparatus according to claim 2 , wherein the estimation unit estimates a linear figure representing the road markings based on the line segments among the plurality of detected line segments that satisfy predetermined angle conditions. The estimation unit determines, based on the direction in which the first line segment extends toward the second area, a second line segment from among the multiple line segments detected in the second area that forms a linear figure representing the same demarcation line as the first line segment.
The information processing apparatus according to claim 2 . Computers
From the first image, which captures the road surrounding the vehicle, the area of the road's lane markings is recognized.
A second image is generated in which the recognized area of the road lane markings is represented on a coordinate plane with the vehicle's longitudinal and lateral directions as two axes.
Multiple line segments are detected based on the area of the road markings shown in the second image, and a linear figure representing the road markings is estimated .
The confidence level of the estimation is calculated for the linear figures representing the estimated road markings.
Based on the calculated confidence level, a linear figure representing the two lane markings forming both ends of the road lane is identified.
The width between the two linear figures representing the two identified lane lines is calculated, and the calculated width is compared with a predetermined standard value representing the width of the road lane.
An information processing method in which, if the calculated width is less than a predetermined standard value representing the width of the road lanes, the other linear figure at the position of the predetermined standard value is detected as the other lane line from the linear figure representing one of the lane lines with a higher reliability . From the first image, which captures the road surrounding the vehicle, the area of the road's lane markings is recognized.
A second image is generated in which the recognized area of the road lane markings is represented on a coordinate plane with the vehicle's longitudinal and lateral directions as two axes.
Multiple line segments are detected based on the area of the road markings shown in the second image, and a linear figure representing the road markings is estimated .
The confidence level of the estimation is calculated for the linear figures representing the estimated road markings.
Based on the calculated confidence level, a linear figure representing the two lane markings forming both ends of the road lane is identified.
The width between the two linear figures representing the two identified lane lines is calculated, and the calculated width is compared with a predetermined standard value representing the width of the road lane.
A program that causes a computer to perform a process in which, if the calculated width is less than a predetermined standard value representing the width of the road lanes, it detects the other linear figure at the position of the predetermined standard value from the more reliable linear figure representing one of the lane lines as the other lane line .