JP7792986B2 - Control device, control method, and program - Google Patents

Description

The present invention relates to a control device, a control method, and a program.

In recent years, efforts to provide sustainable transportation systems that take various circumstances into consideration have become more active. Toward this goal, efforts are being focused on R&D into driver assistance technologies to further improve traffic safety and convenience. A lane mark detection device has been disclosed that includes a shape evaluation value calculation unit that calculates a shape evaluation value relating to whether a subject has the shape of a lane mark, a distance evaluation value calculation unit that calculates a distance evaluation value relating to the field distance, and a reliability calculation unit that calculates the reliability of the lane mark based on the shape evaluation value and the distance evaluation value. By calculating the reliability based on the evaluation value relating to the field distance in addition to the evaluation value relating to the shape of the lane mark, it is possible to calculate the reliability according to the field distance, which is a factor that affects the accuracy of the detected position of the lane mark, and the reliability calculation load is reduced compared to a method that divides a captured image into regions and changes the reliability for each region (see, for example, Patent Document 1).

JP 2013-191040 A

Previous technology did not sufficiently consider the reliability of road marking detection.

The present invention was made in consideration of these circumstances, and one of its objectives is to provide a control device, control method, and program that can accurately estimate the reliability of road marking detection. For example, by utilizing this reliability, vehicles can be controlled with greater precision, which will ultimately contribute to the development of sustainable transportation systems.

The control device, control method, and program according to the present invention employ the following configuration.
(1): A control device according to one embodiment of the present invention includes a first processing unit that detects a first element corresponding to a road dividing line and a second element that is an element outside the road dividing line from an image of the vehicle's surroundings, and projects a sequence of points corresponding to the first element onto a two-dimensional space at a predetermined distance interval corresponding to the actual distance; and a second processing unit that uses the sequence of points to identify an inflection point and estimates the reliability of the road dividing line detected from the image based on whether the horizontal distance between the first element and the second element that are the same vertical distance away from the inflection point is below a threshold.

(2): In the aspect (1) above, the second processing unit derives each angle between adjacent point sequences and finds the inflection point where the change in angle over distance crosses zero, or the inflection point where the change in angle obtained by quadratically differentiating the change in angle with respect to distance crosses zero.

(3): In the aspect (2) above, the second processing unit estimates the reliability of the road dividing line detected from the image if the difference between the two largest angles before and after the inflection point is equal to or greater than a threshold.

(4): In any of the above aspects (1) to (3), the second processing unit estimates that when the inflection point does not exist, the reliability of the road dividing line detected from the image in a distant area is higher than when the inflection point exists.

(5): In any of the above aspects (1) to (3), the system further includes a vehicle control unit that controls the vehicle by referring to the reliability.

(6): In any of the above aspects (1) to (3), the vehicle control unit further includes a vehicle control unit that changes the level of control of the vehicle by referring to the reliability.

(7): In another aspect of the present invention, a control method includes a computer that detects a first element corresponding to a road dividing line and a second element that is an element located outside the road dividing line from an image of the vehicle's surroundings, projects a sequence of points corresponding to the first element onto a two-dimensional space at a predetermined distance interval corresponding to the actual distance, identifies an inflection point using the sequence of points, and estimates the reliability of the road dividing line detected from the image based on whether the horizontal distance between the first element and the second element that are located at the same vertical distance farther from the inflection point is equal to or less than a threshold value.

(8): A program according to another aspect of the present invention causes a computer to execute the following processes: detect a first element corresponding to a road dividing line and a second element that is an element outside the road dividing line from an image of the vehicle's surroundings; project a sequence of points corresponding to the first element onto a two-dimensional space at a predetermined distance interval corresponding to the actual distance; identify an inflection point using the sequence of points; and estimate the reliability of the road dividing line detected from the image based on whether the horizontal distance between the first element and the second element that are the same vertical distance away from the inflection point is equal to or less than a threshold value.

According to aspects (1)-(8), the reliability of road marking detection can be estimated with high accuracy.

According to aspect (5), the control unit can control the vehicle according to the reliability.

1 is a configuration diagram of a vehicle system 1 that uses a vehicle control system according to an embodiment. FIG. 2 is a diagram illustrating an example of the functional configuration of a determination processing unit 120. 10 is a diagram for explaining the processing of a first conversion processing unit 122 and a second conversion processing unit 124. FIG. FIG. 10 is a diagram for explaining processing using a projected point sequence. FIG. 10 is a diagram showing changes in angle θ for each vertical distance. FIG. 10 is a diagram showing the change in horizontal distance for each vertical distance. 10 is a diagram showing the error between the position of a road dividing line that is actually measured and the position of a road dividing line that is obtained by image processing. FIG. FIG. 10 is a diagram illustrating another example of an inflection point. 4 is a flowchart showing an example of a flow of processing executed by the driving assistance device 100. FIG. 10 is a diagram showing an example of an image IM2 in which a road that is not a curved road is captured. FIG. 10 is a diagram showing a change in angle θ in image IM2. FIG. 10 is a diagram showing changes in lateral distance in image IM2. FIG. 10 is a diagram showing the error between the position of a road dividing line on a non-curved road and the position of an actually measured road dividing line. 10 is a flowchart showing another example of the flow of processing executed by the driving assistance device 100.

[Overall configuration]
FIG. 1 is a configuration diagram of a vehicle system 1 that uses a vehicle control system according to an embodiment. The vehicle on which the vehicle system 1 is mounted may be, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source may be an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using power generated by a generator connected to the internal combustion engine, or power discharged from a secondary battery or a fuel cell. This embodiment will be described as being applied to a vehicle, but the system may also be applied to other moving objects instead of a vehicle.

The vehicle system 1 includes, for example, a camera 10, a radar device 12, a LIDAR (Light Detection and Ranging) device 14, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, vehicle sensors 40, a navigation device 50, an MPU 60, an operator 80, a turn signal 90, a driving assistance device 100, a driving force output device 200, a braking device 210, and a steering device 220. These devices and equipment are connected to each other via multiplexed communication lines such as a Controller Area Network (CAN) communication line, serial communication lines, a wireless communication network, etc. The configuration shown in FIG. 1 is merely an example, and some of the configuration may be omitted, or additional components may be added. The driving assistance device 100 is an example of a "controller."

Camera 10 is a digital camera that uses a solid-state imaging element such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). Camera 10 is attached to any location on the vehicle (hereinafter referred to as vehicle M) in which vehicle system 1 is installed. When capturing images of the front, camera 10 is attached to the top of the front windshield, the back of the rearview mirror, or the like. Camera 10, for example, periodically captures images of the surroundings of vehicle M. Camera 10 may also be a stereo camera.

The radar device 12 emits radio waves such as millimeter waves around the vehicle M and detects radio waves reflected by objects (reflected waves) to detect at least the object's position (distance and direction). The radar device 12 may be mounted at any location on the vehicle M. The radar device 12 may also detect the object's position and speed using the FM-CW (Frequency Modulated Continuous Wave) method.

The LIDAR 14 irradiates the area around the vehicle M with light (or electromagnetic waves with wavelengths similar to light) and measures the scattered light. The LIDAR 14 detects the distance to the target based on the time between light emission and light reception. The irradiated light is, for example, pulsed laser light. The LIDAR 14 can be attached to any location on the vehicle M.

The object recognition device 16 performs sensor fusion processing on the detection results from some or all of the camera 10, radar device 12, and LIDAR 14 to recognize the position, type, speed, etc. of the object. The object recognition device 16 outputs the recognition results to the driving assistance device 100. The object recognition device 16 may output the detection results from the camera 10, radar device 12, and LIDAR 14 directly to the driving assistance device 100. The object recognition device 16 may be omitted from the vehicle system 1.

The communication device 20 communicates with other vehicles in the vicinity of vehicle M using, for example, a cellular network, a Wi-Fi network, Bluetooth (registered trademark), or DSRC (Dedicated Short Range Communication), or with various server devices via a wireless base station.

The HMI 30 presents various information to the occupants of the vehicle M and accepts input operations by the occupants. The HMI 30 includes various display devices, speakers, buzzers, touch panels, switches, keys, etc. The HMI 30 is equipped with a display device. The display device is, for example, a so-called multi-information display, provided in the center of the instrument panel of the vehicle M, and displays various information about the vehicle M, such as a speedometer that indicates the traveling speed of the vehicle M or a tachometer that indicates the rotation speed (rotational speed) of the internal combustion engine equipped in the vehicle M.

The vehicle sensors 40 include a vehicle speed sensor that detects the speed of the vehicle M, an acceleration sensor that detects acceleration, a yaw rate sensor that detects the angular velocity around a vertical axis, and a direction sensor that detects the orientation of the vehicle M.

The navigation device 50 includes, for example, a GNSS (Global Navigation Satellite System) receiver 51, a navigation HMI 52, and a route determination unit 53. The navigation device 50 stores first map information 54 in a storage device such as a hard disk drive (HDD) or flash memory. The GNSS receiver 51 determines the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M may be determined or supplemented by an inertial navigation system (INS) that uses the output of the vehicle sensors 40. The navigation HMI 52 includes a display device, speaker, touch panel, keys, etc. The navigation HMI 52 may share some or all of the components with the HMI 30 described above. The route determination unit 53 determines a route (hereinafter, a map route) from the position of the vehicle M determined by the GNSS receiver 51 (or any input position) to a destination input by the occupant using the navigation HMI 52, for example, by referring to the first map information 54. The first map information 54 is information that represents road shapes using, for example, links indicating roads and nodes connected by the links. The first map information 54 may also include information such as road curvature and POI (Point of Interest) information. The route on the map is output to the MPU 60. The navigation device 50 may provide route guidance using the navigation HMI 52 based on the route on the map. The navigation device 50 may be implemented, for example, by the functions of a terminal device such as a smartphone or tablet device owned by the occupant. The navigation device 50 may transmit the current position and destination to a navigation server via the communication device 20 and obtain a route equivalent to the route on the map from the navigation server.

The MPU 60 includes, for example, a recommended lane determination unit 61 and stores second map information 62 in a storage device such as an HDD or flash memory. The recommended lane determination unit 61 divides the route on the map provided by the navigation device 50 into multiple blocks (for example, every 100 meters in the vehicle's direction of travel) and determines a recommended lane for each block by referring to the second map information 62. The recommended lane determination unit 61 determines, for example, which lane from the left to use. When a branch point is present on the route on the map, the recommended lane determination unit 61 determines a recommended lane so that the vehicle M can travel on a reasonable route to the branch point. For example, when the vehicle M arrives a predetermined distance before the branch point on which the vehicle M is traveling, the recommended lane determination unit 61 determines the lane connecting to the branch point as the recommended lane. The recommended lane determination unit 61 and the second map information 62 may be functional units or information included in another device, such as the driving assistance device 100.

The second map information 62 is map information with higher accuracy than the first map information 54. The second map information 62 includes, for example, information on the center of lanes or information on lane boundaries. The second map information 62 may include road information, traffic regulation information, address information (address and postal code), facility information, telephone number information, and the like. The second map information 62 may be updated as needed by the communication device 20 communicating with other devices.

The operators 80 include, for example, a steering wheel 82, as well as an accelerator pedal, brake pedal, shift lever, and other operators. The operators 80 are fitted with sensors that detect the amount of operation or whether or not an operation is being performed, and the detection results are output to the driving assistance device 100 or some or all of the driving force output device 200, brake device 210, and steering device 220. The steering wheel 82 does not necessarily have to be annular, and may take the form of an irregularly shaped steering wheel, joystick, button, or the like. The operators 80 include a first operator 84. The direction indicator 90 turns on or off in response to operation of the first operator 84.

The first operator 84 is, for example, a turn signal lever switch. For example, when the driver operates the first operator 84, the turn signal 90 lights up in response to the operation. When the driver performs a predetermined operation on the first operator 84, the control unit 150 activates the ALC (Auto Lane Change) function and causes the vehicle M to perform an automatic lane change. The predetermined operation is an operation that triggers activation of the ALC function. Examples of the predetermined operation include operating the turn signal lever switch in the direction of the desired lane change for a predetermined period of time, or pushing the turn signal lever switch to a predetermined position. More specifically, the predetermined operation is operating the turn signal lever switch in the direction of the desired lane change while maintaining it in a predetermined position for a predetermined period of time. Instead of a turn signal lever switch, the first operator 84 may be another type, such as a button. The ALC function may also be activated by operating another operator, such as a predetermined button.

The driving assistance device 100 includes, for example, a recognition unit 110, a determination processing unit 120, and a control unit 150. The recognition unit 110 and the control unit 150 are implemented by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be implemented by hardware (including circuitry) such as an LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), or SOC (System on Chip), or may be implemented by a combination of software and hardware. The program may be stored in advance in a storage device (storage device with a non-transitory storage medium) such as an HDD or flash memory of the driving assistance device 100, or may be stored on a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the driving assistance device 100 by inserting the storage medium (non-transitory storage medium) into a drive device.

The recognition unit 110 recognizes the position, speed, acceleration, and other status of objects around the vehicle M based on information input from the camera 10, radar device 12, and LIDAR 14 via the object recognition device 16. The position of an object is recognized as a position on an absolute coordinate system with a representative point of the vehicle M (such as the center of gravity or the center of the drive shaft) as the origin, and is used for control. The position of an object may be represented by a representative point such as the center of gravity or a corner of the object, or by an area. The "state" of an object may include the acceleration or jerk of the object, or its "behavioral state" (for example, whether or not the vehicle is changing lanes or is about to change lanes).

The recognition unit 110 recognizes, for example, the lane in which the vehicle M is traveling (driving lane). For example, the recognition unit 110 recognizes the driving lane by comparing the pattern of road dividing lines (e.g., an arrangement of solid and dashed lines) obtained from the second map information 62 with the pattern of road dividing lines around the vehicle M recognized from the image captured by the camera 10. The recognition unit 110 may recognize the driving lane by recognizing road boundaries (road boundaries) including not only road dividing lines but also road dividing lines, shoulders, curbs, medians, guardrails, etc. This recognition may take into account the position of the vehicle M obtained from the navigation device 50 and the processing results from the INS. The recognition unit 110 recognizes stop lines, obstacles, red lights, toll booths, and other road phenomena.

When recognizing the driving lane, the recognition unit 110 recognizes the position and orientation of vehicle M relative to the driving lane. For example, the recognition unit 110 may recognize the deviation of vehicle M's reference point from the center of the lane and the angle it makes with a line connecting the centers of the lanes in the vehicle M's direction of travel as the relative position and orientation of vehicle M relative to the driving lane. Alternatively, the recognition unit 110 may recognize the position of vehicle M's reference point relative to either side edge of the driving lane (a road dividing line or road boundary) as the relative position of vehicle M relative to the driving lane.

The determination processing unit 120 will be described later.

The control unit 150 executes driving assistance control. For example, the control unit 150 automatically controls the driving force output device 200 and the brake device 210 without relying on the driver's operation, thereby automatically controlling the speed of the vehicle M. The control unit 150 executes what is known as ACC (Adaptive Cruise Control). The control unit 150 controls the vehicle M to travel at a set speed, or to follow a vehicle ahead at a predetermined distance from the vehicle ahead.

The control unit 150 controls the steering device 220 so that the vehicle M does not deviate from the driving lane. For example, the control unit 150 controls the steering device 220 so that the vehicle M travels in the center or near the center of the driving lane recognized by the recognition unit 110. Hereinafter, this control may be referred to as "lane keeping control." The control unit 150 performs hands-on lane keeping control and hands-off lane keeping control.

Hands-on lane keeping control is a control that is executed when the driver is holding the steering wheel (when a steering wheel grip sensor (not shown) detects that the driver is holding the steering wheel). The conditions under which hands-on lane keeping control can be executed are more lenient than the conditions under which hands-off lane keeping control can be executed.

Hands-off lane keeping control is executed when the driver is not gripping the steering wheel (when a steering grip sensor, not shown, does not detect that the driver is gripping the steering wheel). Hands-off lane keeping control can be executed, for example, when the following conditions are met: the speed of vehicle M is equal to or greater than a predetermined speed, vehicle M is traveling on a predetermined road (for example, a road or type of road that has been set in advance as being capable of implementing hands-off lane keeping control), and the driver is monitoring the road ahead. Hands-off lane keeping control is executed when the driver is monitoring the road ahead, and is not executed or is stopped when the driver is not monitoring the road ahead.

The conditions under which the above-mentioned hands-on lane keeping control and hands-off lane keeping control can be implemented are examples, and other conditions (for example, vehicle M following a vehicle ahead) may be included, or some conditions may be omitted. The conditions under which the hands-on lane keeping control can be implemented may be less stringent than the conditions under which hands-off lane keeping can be implemented (the conditions under which the hands-off lane keeping control can be implemented may be stricter than the conditions under which hands-on lane keeping can be implemented). Whether the driver is monitoring the road ahead is determined by the driving assistance device 100 based on an image captured by a camera (not shown) that images the driver.

The driving force output device 200 outputs driving force (torque) to the drive wheels to drive the vehicle M. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission, as well as an ECU that controls these. The ECU controls the above components according to information input from the driving assistance device 100 or information input from the operator 80.

The brake device 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to information input from the driving assistance device 100 or information input from the operator 80, so that a brake torque corresponding to the braking operation is output to each wheel.

The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor applies force to a rack and pinion mechanism to change the direction of the steered wheels. The steering ECU drives the electric motor to change the direction of the steered wheels in accordance with information input from the driving assistance device 100 or information input from the operator 80.

[Determination processing unit]
2 is a diagram showing an example of the functional configuration of the determination processing unit 120. The determination processing unit 120 includes, for example, a first conversion processing unit 122, a second conversion processing unit 124, an angle processing unit 126, and a distance processing unit 128. The first conversion processing unit 122 is an example of a "first processing unit." The combined functional configuration of the angle processing unit 126 and the distance processing unit 128 is an example of a "second processing unit."

The first conversion processing unit 122 acquires an image of the periphery of the vehicle M captured by the camera 10. The first conversion processing unit 122 detects first elements corresponding to the road dividing lines and second elements, which are elements outside the road dividing lines (objects such as exterior walls and curbs), from the image of the periphery of the vehicle M. The first conversion processing unit 122 detects the first elements corresponding to the road dividing lines and second elements, which are elements outside the road dividing lines, based on feature information extracted from the image. The second conversion processing unit 124 approximates the first elements to a predetermined curve.

Figure 3 is a diagram illustrating the processing of the first conversion processing unit 122 and the second conversion processing unit 124. The first conversion processing unit 122 acquires an image IM capturing an image of the area around the vehicle M. The first conversion processing unit 122 generates two-dimensional information IF1 by projecting road dividing lines, exterior walls, and the like from the image onto a two-dimensional plane. The two-dimensional information IF1 is an image that includes information indicating, for example, the road dividing line LR on the right side of the vehicle M (camera 10), the road dividing line LL on the left side, and the side wall C.

The first conversion processing unit 122, for example, performs image processing on the image to obtain feature information (e.g., brightness information) of the image, and identifies elements on the road (first elements corresponding to road dividing lines, and second elements such as side walls and curbs outside the road dividing lines) based on the feature information. The first conversion processing unit 122 extracts parts of the image where the horizontal brightness changes by more than a threshold. For example, a differential filter (Sobel filter) is used on the image to extract features.

The transformation for projecting the information of the image actually captured onto a two-dimensional plane is performed, for example, based on the following equations (1) and (2). The projected point is represented as (d _i , d _j ). Any point after transformation is (X, Y). "f" is the focal length between the camera 10, the image sensor, and the lens. "H" is the height of the camera 10. "λ" is the pixel size. "Z" is the position in the direction perpendicular to the XY plane.
d _i = Yf/Zλ formula (1)
d _j =Hf/Xλ formula...(2)

The second conversion processing unit 124 converts the above features into three-dimensional distances, for example, using the above-mentioned equations (1) and (2), and fits (regresses) the converted features to a numerical model of the road lane markings. For example, regression is performed using least squares of a cubic equation. This generates three-dimensional information IF2.

For example, in the above processing, there is an area AR1 (an area corresponding to a position far from the camera 10) where road-dividing lines can be detected with high accuracy, and an area AR2 (an area corresponding to a position close to the camera 10) where road-dividing lines cannot be detected with high accuracy. However, it was sometimes impossible to determine whether accuracy was guaranteed in the area between areas AR1 and AR2. In this embodiment, the reliability of road-dividing line recognition can be determined by performing the processing described below.

[Overview of reliability determination process]
The first conversion processing unit 122 projects a sequence of points corresponding to a first element (a road dividing line) onto a two-dimensional space at a predetermined distance interval (e.g., an arbitrary distance such as 0.5 m, 1 m, or 2 m) according to the actual distance. The first conversion processing unit 122 projects a sequence of points corresponding to a second element (an object such as an exterior wall or a curb) onto a two-dimensional space at a predetermined distance interval according to the actual distance. The angle processing unit 126 identifies an inflection point using the sequence of points of the first element (or the second element). FIG. 4 is a diagram for explaining processing using the projected sequence of points.

The angle processing unit 126 calculates the angle θ between the sequence of points on the road-dividing line on the image (see FIG. 4 ) and determines the inflection point where the change in the angle θ for each longitudinal distance crosses zero. For example, the angle θ is calculated using equation (3). The angle processing unit 126 calculates θ between each sequence of points on the road-dividing line. (X _NL , Y _NL ) and (X _L , Y _L ) will be described later.
θ=tan ^-1 (dy/dx)...Formula (3)

Figure 5 is a diagram showing the change in angle θ for each longitudinal distance. The longitudinal distance is the distance based on the point ahead or the point behind the point sequence for which the angle θ was determined. The angle processing unit 126 identifies the inflection point (zero crossing point) where the progression of the change in angle θ crosses zero in Figure 5. If an inflection point is present, it means that the road dividing line is a road dividing line placed on a curved road. The area after the inflection point is an area where it may not be possible to accurately recognize the road dividing line from the image. The inflection point may also be determined from Figure 8, which will be described later.

The distance processing unit 128 estimates the reliability of the road dividing line detected from the image based on whether the horizontal distance between a first element and a second element that are the same vertical distance away from the inflection point is equal to or less than a threshold. The distance processing unit 128 derives the horizontal distance between the first element (road dividing line) and the second element (an object such as an exterior wall or curb) for each distance away from the inflection point, and estimates the reliability of the road dividing line detected from the image based on the distance. The distance is the horizontal distance between a point on the first element and a point on the second element that are the same vertical distance. Note that the projection of the point sequence of the second element may be omitted, and the distance between the first element and the second element may be calculated based on the position of the feature of the second element that is the same vertical distance as the first element. The distance processing unit 128 calculates the horizontal distance d using equation (4).

Figure 6 shows the change in horizontal distance for each vertical distance. A threshold is set in advance for the distance. The distance processing unit 128 classifies areas where the horizontal distance is equal to or greater than the threshold as areas with high reliability, and areas where the horizontal distance is less than the threshold as areas with low reliability. The threshold may be set, for example, for each type (pattern) of distance change progression, or may be set according to the curvature of the curved road obtained from map information.

As described above, the judgment processing unit 120 can estimate (or judge) that the reliability of detected road-dividing lines in the area after the inflection point where the horizontal distance is equal to or greater than the threshold is high. Figure 7 shows the error between the actually measured position of the road-dividing line and the position of the road-dividing line obtained from image processing. The error increases as the vertical distance increases, but the error is small up to the area where the horizontal distance is equal to or greater than the threshold. Therefore, it can be confirmed that the reliability of the road-dividing line detection results up to the area where the horizontal distance is equal to or greater than the threshold is high.

Figure 8 is a diagram showing another example of an inflection point. The angle processing unit 126 may determine an inflection point where the transition of the angle θ crosses zero after quadratically differentiating it with respect to distance. The determination processing unit 120 may perform the above-described processing using the inflection point IP obtained as a result of the quadratically differentiated operation.

Furthermore, the determination processing unit 120 may calculate a value d1 that maximizes the difference between the two largest angles (the difference between a positive angle and a negative angle) before and after the inflection point IP, and if this value d1 is equal to or greater than a threshold, determine that the road dividing line in the image is a road dividing line on a curved road. If the road dividing line in the image is a road dividing line on a curved road, the determination processing unit 120 may perform a process to estimate the reliability of the road dividing line detected from the image using the lateral distance.

[flowchart]
9 is a flowchart showing an example of the flow of processing executed by the driving assistance device 100. First, the determination processing unit 120 acquires an image (step S100). Next, the determination processing unit 120 projects road-dividing lines and exterior walls onto a two-dimensional space based on the image (step S102). Next, the determination processing unit 120 sets a sequence of points for the projected road-dividing lines and exterior walls (step S104). Note that step S102 may be omitted, and the sequence of points may be set directly from the characteristics of the image.

Next, the determination processing unit 120 derives the angle θ between the two sequences of points on the road-dividing line (step S106). Next, the determination processing unit 120 derives the lateral distance between the exterior wall and the road-dividing line (step S108). Next, the determination processing unit 120 identifies areas with low reliability based on the angle θ derived in step S106, a threshold value, and the lateral distance (step S110). This completes processing for one routine of this flowchart.

Here, we will explain the angle θ, lateral distance, and error of road dividing lines on non-curved roads. Figure 10 is a diagram showing an example of image IM2 captured of a road that is not a curved road. Figure 11 is a diagram showing changes in angle θ in image IM2. When the road is not a curved road, there is no inflection point. Figure 12 is a diagram showing changes in lateral distance in image IM2. When the road is not a curved road, no threshold is set and the reliability of all areas is estimated to be high. A threshold may be set at a position farther away than the lateral distance threshold for curved roads. In other words, when there is no inflection point, the determination processing unit 120 estimates that the reliability of road dividing lines detected from images of distant areas is higher than when an inflection point is present.

Figure 13 shows the error between the position of road dividing lines on non-curved roads and the measured position of road dividing lines. The longer the vertical distance, the larger the error, but even with longer vertical distances, the error remains small. Therefore, it can be confirmed that the reliability of the road dividing line detection results is high for all areas or for areas with a vertical distance longer than that of curved roads.

[flowchart]
14 is a flowchart showing another example of the flow of processing executed by the driving assistance device 100. The processing from step S100 to step S106, step S108, and step S110 is the same as the processing of the same step numbers in FIG.

After processing in step S106, the determination processing unit 120 determines whether the image is a target image for which reliability is to be calculated (step S108). The determination processing unit 120 makes this determination based on the change in angle θ. Specifically, as described above, if (1) or both (1) and (2) are satisfied, the image is determined to be a target image and processing proceeds to step S108. If not satisfied, the image is determined to not be a target image, and processing of one routine of this flowchart ends.

(1) above means that an inflection point exists. The inflection point may be an inflection point in the progression of the change in angle θ, or an inflection point in the progression of the result of quadratic differentiation. (2) above means that the difference in the result of quadratic differentiation of angle θ before and after the inflection point is greater than or equal to a threshold.

As described above, the determination processing unit 120 can easily determine whether or not a target is one for which reliability is to be calculated by using changes in angle θ.

Here, for example, in driving assistance control and autonomous driving control, the results of self-position estimation using map information and GNSS satellites are used to estimate the position of vehicle M on the road and control vehicle M. At this time, to correct errors in the position of vehicle M, information about road dividing lines obtained from images captured by camera 10 and the shape characteristics of the road dividing lines in map information are used to correct the errors.

The information about road-dividing lines in the image may not accurately capture the shape of the road-dividing lines because distant features cannot be accurately detected due to obstructions such as side walls or the performance of the image sensor. This can lead to errors in matching the road-dividing line information obtained from the image captured by camera 10 with the shape features of the road-dividing lines in the map information.

In this embodiment, as described above, the driving assistance device 100 projects a sequence of points in two dimensions at a predetermined distance interval based on the first element, which is the road-dividing line obtained from the image, and identifies an inflection point using the sequence of points. It can then accurately estimate the reliability of the road-dividing line detected from the image based on whether the horizontal distance between the first element and the second element, which are the same vertical distance away from the inflection point, is equal to or less than a threshold value.

[Utilizing reliability]
The control unit (vehicle control unit) 150 may control the vehicle by referring to the reliability. The reliability is used in driving assistance, matching with map information, and autonomous driving. The driving assistance device 100, for example, suppresses driving assistance in areas with low reliability or suppresses the generation of a driving assistance plan in areas with low reliability. For example, it suppresses lane keeping control or lane change control in areas with low reliability. For example, it postpones the generation of a driving assistance plan in areas with low reliability. The driving assistance device 100 suppresses the matching process between road dividing lines in areas with low reliability and road dividing lines in map information, or changes the content of the matching process. For example, in areas with low reliability, the driving assistance device 100 prioritizes the position of the road dividing lines in the map information, or performs the matching process assuming low reliability (by lowering the influence of information on road dividing lines obtained from images in identifying the position of the road dividing lines and identifying the vehicle's own location compared to when reliability is high). In this case, for example, the control unit 150 controls the traveling position of the vehicle M based on the results of the matching process. In this way, the control unit 150 may change the content of the control of the vehicle M by referring to the reliability.

The control unit 150 may change the level of control of the vehicle M by referring to the reliability. The control level refers to the degree of autonomous driving control or the degree of driving assistance control. For example, when the reliability is high, the control unit 150 increases the degree of autonomous driving control or the degree of driving assistance control compared to when the reliability is low. The degree of autonomous driving control or the degree of driving assistance control refers to the degree to which the control unit 150 controls the vehicle M itself. A high degree of autonomous driving control or the degree of driving assistance control means that the control unit 150 controls the vehicle M itself to a high degree, and a low degree of autonomous driving control or the degree of driving assistance control means that the control unit 150 controls the vehicle M itself to a low degree and the driver controls the vehicle M by operating it to a high degree. For example, if the vehicle M is capable of autonomous driving, the autonomous driving level may be lowered in areas of low reliability. For example, in areas of low reliability, the driver may be required to monitor the road ahead or keep a hands-on attitude, and these obligations may be lifted in other areas. Additionally, the generation of action plans for autonomous driving in areas with low reliability may be suppressed. The same applies to driving assistance control.

According to the embodiment described above, the determination processing unit 120 can more accurately estimate the reliability of road-dividing lines detected from an image based on whether the horizontal distance between a first element and a second element that are the same vertical distance away from the inflection point is less than or equal to a threshold value.

The above-described embodiment can be expressed as follows.
a storage device storing a program;
a hardware processor;
The hardware processor executes the program stored in the storage device,
detecting a first element corresponding to a road dividing line and a second element that is an element outside the road dividing line from an image of the surroundings of the vehicle, and projecting a sequence of points corresponding to the first element onto a two-dimensional space at a predetermined distance interval according to the actual distance;
Identifying an inflection point using the sequence of points;
a reliability of the road-dividing line detected from the image is estimated based on whether a horizontal distance between the first element and the second element that are the same vertical distance away from the inflection point is equal to or less than a threshold value;
The control device is configured as follows.

The above describes the form for carrying out the present invention using an embodiment, but the present invention is in no way limited to such an embodiment, and various modifications and substitutions can be made without departing from the spirit of the present invention.

REFERENCE SIGNS LIST 1 Vehicle system 10 Camera 80 Operator 100 Driving assistance device 110 Recognition unit 120 Determination processing unit 122 First conversion processing unit 124 Second conversion processing unit 126 Angle processing unit 128 Distance processing unit 150 Control unit

Claims (7)

a first processing unit that detects a first element corresponding to a road dividing line and a second element that is an element outside the road dividing line from an image of the periphery of the vehicle, and projects a sequence of points corresponding to the first element onto a two-dimensional space at a predetermined distance interval according to an actual distance;
Identifying an inflection point using the sequence of points;
a second processing unit that estimates an area where the horizontal distance between the first element and the second element that are the same vertical distance away from the inflection point is less than a threshold as an area where the reliability of the road-dividing line detected from the image is low,
The second processing unit is
deriving each angle between adjacent points in the sequence, and determining the inflection point where the transition of the angle for each distance crosses zero, or the inflection point where the transition of the result of quadratically differentiating the transition of the angle with respect to the distance crosses zero;
Control device. the second processing unit estimates the reliability of the road-dividing line detected from the image when a difference between the two largest angles before and after the inflection point is equal to or greater than a threshold.
The control device according to claim 1 . the second processing unit estimates that when the inflection point is not present, the reliability of the road-dividing line detected from the image in a distant area is higher than when the inflection point is present.
The control device according to claim 1 or 2. a vehicle control unit that controls the vehicle by referring to the reliability;
The control device according to claim 1 or 2. a vehicle control unit that changes a level of control of the vehicle by referring to the reliability;
The control device according to claim 1 or 2. The computer
a process of detecting a first element corresponding to a road dividing line and a second element that is an element outside the road dividing line from an image of the periphery of the vehicle, and projecting a sequence of points corresponding to the first element onto a two-dimensional space at a predetermined distance interval according to the actual distance;
A process of identifying an inflection point using the sequence of points;
and estimating an area where the horizontal distance between the first element and the second element that are the same in vertical direction and farther from the inflection point is less than a threshold as an area where the reliability of the road-dividing line detected from the image is low;
deriving each angle between adjacent points in the sequence, and determining the inflection point where the transition of the angle for each distance crosses zero, or the inflection point where the transition of the result of quadratically differentiating the transition of the angle with respect to the distance crosses zero;
Control method. On the computer,
a process of detecting a first element corresponding to a road dividing line and a second element that is an element outside the road dividing line from an image of the periphery of the vehicle, and projecting a sequence of points corresponding to the first element onto a two-dimensional space at a predetermined distance interval according to the actual distance;
A process of identifying an inflection point using the sequence of points;
and estimating an area where the horizontal distance between the first element and the second element that are the same vertical distance away from the inflection point is less than a threshold as an area where the reliability of the road-dividing line detected from the image is low,
A program for executing a process of deriving each angle between adjacent sequences of points and finding the inflection point where the change in angle over distance crosses zero, or the inflection point where the change in angle obtained by quadratically differentiating the change in angle with respect to distance crosses zero .