JP7650964B2 - Intelligent drive control method and device, vehicle, electronic device, and storage medium - Google Patents

Description

The present disclosure relates to the field of computer technology, and in particular to an intelligent drive control method and device, a vehicle, an electronic device, and a storage medium.

If a driver is in a dangerous driving state while driving a vehicle, it poses a risk to safe driving and is likely to lead to a traffic accident. Therefore, how to accurately detect a driver's dangerous driving state and take timely measures is very important for safe driving.

This disclosure provides technical means for intelligent drive control.

According to one aspect of the present disclosure,
An intelligent drive control method is provided, which includes acquiring driving state information of a vehicle, acquiring image information of a driving area of the vehicle, determining a driving state of a driver of the vehicle based on the driving state information of the vehicle and the image information of the driving area, and performing intelligent drive control in response to detection that the driver of the vehicle is in a predetermined dangerous driving state.

In one possible embodiment, the determining of the driving state of the driver of the vehicle based on the driving state information of the vehicle and the image information of the driving area includes:
detecting a behavior of a driver of the vehicle based on image information of the driving area;
Determining a driving state of the driver based on information about the vehicle's running state and a result of detecting the driver's behavior.

In one possible embodiment, the performing of behavior detection for a driver of the vehicle includes:
determining a danger level corresponding to the driver's behavior detection result;
The above-mentioned determining the driving state of the driver based on the vehicle running state information and the driver behavior detection result,
In response to the determination that the vehicle is in an abnormal driving state based on the driving state information of the vehicle, a danger level of the driver's behavior detection result is increased to a first danger level;
In response to determining that the first danger level has reached a predetermined alarm level, determining that the driver's driving state is in a predetermined dangerous driving state.

In one possible embodiment, the determining of the driving state of the driver based on the vehicle running state information and the driver's behavior detection result includes:
This includes determining that the driver is in a predetermined dangerous driving state in response to a determination that both the vehicle's driving state and the driver's behavior state are abnormal based on the vehicle's driving state information and the driver's behavior detection result.

In one possible embodiment, the determining of the driving state of the driver of the vehicle based on the driving state information of the vehicle and the image information of the driving area includes:
In response to detection that the vehicle is in an abnormal driving state based on the driving state information of the vehicle, detecting a behavior of the driver based on image information of the driving area.
The method includes determining that the driver is in a predetermined dangerous driving state in response to the driver's behavior detection result being determined to be inattentive driving or fatigued driving.

In one possible embodiment, the abnormal operating condition of the vehicle is
The detection signal includes at least one of the following: the number of times the vehicle steps on a line within a first set time period reaches a predetermined threshold number of times the vehicle steps on a line; the width of the vehicle's left and right sway reaches a predetermined width threshold; the vehicle's passage does not follow the instructions of road signs or traffic signals; and the vehicle's speed exceeds a predetermined speed threshold.

In one possible embodiment, performing the intelligent drive control includes:
information on the running state of the vehicle;
A behavior detection result of the driver of the vehicle based on image information of the driving area;
and a driving state of the driver.

In one possible embodiment, performing the intelligent drive control includes:
The method includes controlling the alarm device to continue outputting alarm information until both the vehicle running state and the driver's driving state return to normal.

In one possible embodiment, performing the intelligent drive control includes:
In response to detecting that the driver is in a predetermined dangerous driving state for a duration exceeding a predetermined time, control is performed to activate an assisted driving/automated driving function and/or control the vehicle to decelerate and stop.

In one possible embodiment,
The vehicle further includes acquiring environmental information about a surrounding of the vehicle;
The above-mentioned determining a driving state of a driver of the vehicle based on the driving state information of the vehicle and the image information of the driving area includes:
The method includes determining a driving state of a driver of the vehicle based on environmental information around the vehicle, driving state information of the vehicle, and image information of the driving area.

In one possible embodiment, the environmental information comprises road traffic information,
The acquiring of environmental information about the surroundings of the vehicle includes acquiring road traffic information about a road on which the vehicle is traveling,
The above-mentioned determining a driving state of a driver of the vehicle based on environmental information around the vehicle, driving state information of the vehicle, and image information of the driving area includes:
The method includes determining that the driver is in a predetermined dangerous driving state when the vehicle's driving state information and the road traffic information do not satisfy a predetermined correspondence relationship and when a behavior detection result of the driver of the vehicle based on image information of the driving area indicates that the driver's behavior state is abnormal.

In one possible embodiment, the obtaining of the vehicle driving state information includes:
Acquiring driving sensing data including at least one of video information collected by an advanced driving assistance system of the vehicle, video information collected by a drive recorder of the vehicle, and speed information detected by a speed sensor of the vehicle;
Determining driving state information of the vehicle based on the driving sensing data.

In one possible embodiment, the determining of the driving state of the driver of the vehicle based on the driving state information of the vehicle and the image information of the driving area includes:
The method includes inputting the acquired driving state information of the vehicle and image information of the driving area into a trained neural network, and determining the driving state of the driver of the vehicle through the neural network.

According to one aspect of the present disclosure,
A first acquisition module for acquiring vehicle running state information;
a second acquisition module for acquiring image information of a driving area of the vehicle;
A driving state determination module for determining a driving state of a driver of the vehicle based on driving state information of the vehicle and image information of the driving area;
and a control module for performing intelligent drive control in response to detection that a driver of the vehicle is in a predetermined dangerous driving state.

In one possible embodiment, the driving state determination module includes a first driving state determination sub-module and a second driving state determination sub-module;
The first driving state determination submodule detects a behavior of a driver of the vehicle based on image information of the driving area;
The second driving state determination submodule determines the driving state of the driver based on the vehicle running state information and the driver behavior detection result.

In one possible embodiment, the first driving state determination submodule determines a danger level corresponding to the driver's behavior detection result;
The second driving state determination submodule:
raising a danger level of the driver's behavior detection result to a first danger level in response to the vehicle being determined to be in an abnormal driving state based on the vehicle driving state information;
In response to a determination that the first danger level has reached a preset alarm level, it is determined that the driving state of the driver is in a preset dangerous driving state.

In one possible embodiment, the second driving state determination submodule determines that the driver is in a pre-defined dangerous driving state in response to the determination that both the vehicle's driving state and the driver's behavior state are abnormal based on the vehicle's driving state information and the driver's behavior detection result.

In one possible embodiment, the driving state determination module includes a third driving state determination sub-module and a fourth driving state determination sub-module;
The third driving state determination submodule performs behavior detection of the driver based on image information of the driving area in response to detection that the vehicle is in an abnormal driving state based on driving state information of the vehicle;
The fourth driving state determination submodule determines that the driver is in a predetermined dangerous driving state in response to the driver's behavior detection result being determined to be inattentive driving or fatigued driving.

In one possible embodiment, the abnormal operating condition of the vehicle is
The detection signal includes at least one of the following: the number of times the vehicle steps on a line within a first set time period reaches a predetermined threshold number of times the vehicle steps on a line; the width of the vehicle's left and right sway reaches a predetermined width threshold; the vehicle's passage does not follow the instructions of road signs or traffic signals; and the vehicle's speed exceeds a predetermined speed threshold.

In one possible embodiment, the control module comprises:
information on the running state of the vehicle;
A behavior detection result of the driver of the vehicle based on image information of the driving area;
and a driving state of the driver.

In one possible embodiment, the control module controls the alarm device to continue outputting alarm information until both the vehicle driving state and the driver's driving state return to normal.

In one possible embodiment, the control module controls an assisted/automated driving function to be activated and/or controls the vehicle to decelerate and stop in response to detecting that the driver is in a pre-defined unsafe driving state for a duration exceeding a pre-defined time.

In one possible embodiment, the device comprises:
Further comprising a third acquisition module for acquiring environmental information around the vehicle;
The driving state determination module determines a driving state of a driver of the vehicle based on environmental information about the surroundings of the vehicle, driving state information about the vehicle, and image information about the driving area.

In one possible embodiment, the environmental information comprises road traffic information,
The third acquisition module acquires road traffic information of a road on which the vehicle is traveling,
The driving state determination module determines that the driver is in a predetermined dangerous driving state when the vehicle's driving state information and the road traffic information do not satisfy a predetermined correspondence relationship and the behavior detection result of the driver of the vehicle based on the image information of the driving area indicates that the driver's behavior state is abnormal.

In one possible embodiment, the first acquisition module includes a sensing data acquisition submodule and a driving state information determination submodule;
The sensing data acquisition submodule acquires driving sensing data including at least one of video information collected by an advanced driving assistance system of the vehicle, video information collected by a drive recorder of the vehicle, and speed information detected by a speed sensor of the vehicle;
The driving state information determination submodule determines driving state information of the vehicle based on the driving sensing data.

In one possible embodiment, the driving state determination module inputs the acquired driving state information of the vehicle and image information of the driving area into a trained neural network and determines the driving state of the driver of the vehicle through the neural network.

a first sensor for collecting image information of a driving area of the vehicle;
A controller for acquiring vehicle driving state information, judging a driving state of a driver of the vehicle based on the vehicle driving state information and image information of the driving area, and performing intelligent drive control in response to detection that the driver of the vehicle is in a predetermined dangerous driving state.

In one possible embodiment, the vehicle further includes a second sensor for collecting driving sensing data;
The controller acquires information about the driving state of the vehicle based on the driving sensing data.

According to one aspect of the present disclosure, there is provided an electronic device including a processor and a memory for storing instructions executable by the processor, the processor being configured to execute the above-described intelligent drive control method by calling the instructions stored in the memory.

According to one aspect of the present disclosure, there is provided a computer-readable storage medium having stored therein computer program instructions, the computer-readable storage medium realizing the above-described intelligent drive control method when the computer program instructions are executed by a processor.

According to one aspect of the present disclosure, there is provided a computer program that includes computer-readable code, and when the computer-readable code is executed in an electronic device, causes a processor of the electronic device to execute instructions for implementing the above-described intelligent drive control method.

In an embodiment of the present disclosure, vehicle driving state information and image information of the vehicle's driving area are acquired, the driving state of the vehicle driver is determined based on the vehicle driving state information and the image information of the driving area, and intelligent drive control is performed in response to detection that the vehicle driver is in a pre-set dangerous driving state, thereby making it possible to determine the driving state of the vehicle driver by combining the vehicle driving state and the image information of the driving area, thereby increasing the accuracy of the determined driving state of the driver. In addition, by performing intelligent drive control in response to detection that the vehicle driver is in a pre-set dangerous driving state, the safety of vehicle driving is further improved.

It should be understood that the above general description and the following detailed description are merely illustrative and interpretive and are not intended to limit the present disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments taken in conjunction with the drawings.

The drawings herein are incorporated as part of the specification, show embodiments consistent with the present disclosure, and are used together with the specification to explain the technical means of the present disclosure.
2 shows a flowchart of an intelligent drive control method according to an embodiment of the present disclosure. FIG. 1 shows a block diagram of an intelligent drive controller according to an embodiment of the present disclosure. 8 shows a block diagram of an electronic device 800 according to an embodiment of the present disclosure. 19 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure.

Various exemplary embodiments, features, and aspects of the present disclosure are described in detail below with reference to the drawings. In the drawings, like numbers represent functionally identical or similar elements. While various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

The term "exemplary" as used herein means "serving as an example, example, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or superior over other embodiments.

The term "and/or" in this specification simply describes a relational relationship with a related object and indicates that three relations can exist; for example, A and/or B may indicate three cases: only A exists, both A and B exist, and only B exists. Also, the term "at least one" in this specification indicates any one of a plurality, or any combination of at least two of a plurality; for example, including at least one of A, B, and C may indicate including any one or more elements selected from the set consisting of A, B, and C.

In addition, in order to more effectively describe the present disclosure, various specific details are provided in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can be similarly implemented without any specific details. In some embodiments, detailed descriptions of methods, means, elements and circuits familiar to those skilled in the art are not provided in order to emphasize the gist of the present disclosure.

The present disclosure provides an intelligent drive control method, and the execution subject of the method may be an intelligent drive control device installed in a vehicle. In a possible embodiment, the method may be executed by a terminal device, a server, or other processing device. Here, the terminal device may be an in-vehicle device, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a wearable device, etc. The in-vehicle device may be an in-vehicle infotainment or domain controller in the vehicle cabin, or may be a device host for executing an intelligent drive control method in an ADAS (Advanced Driving Assistance System), an OMS (Occupant Monitoring System), or a DMS (Driver Monitoring System), etc. In some possible embodiments, the intelligent drive control method may be realized by a processor invoking computer-readable instructions stored in a memory.

Figure 1 shows a flowchart of an intelligent drive control method according to an embodiment of the present disclosure. As shown in Figure 1, the intelligent drive control method includes steps S11 to S14.

In step S11, vehicle driving condition information is acquired.

The vehicle here may be at least one of a private car, car sharing, online car-hailing, taxi, truck, etc., and this disclosure does not limit the specific type of vehicle.

The driving state information represents the driving state of the vehicle. In one possible embodiment, the driving state can be divided into a normal driving state and an abnormal driving state. The normal driving state may be that the vehicle drives according to a predetermined speed, direction, route, and rules such as road traffic laws. The abnormal driving state may be that the vehicle drives without following a predetermined speed, direction, route, and rules such as road traffic laws, corresponding to the normal driving state.

The driving condition information may include at least one of information indicating the normal or abnormal driving condition of the vehicle, such as the vehicle's speed, direction, acceleration, information on the road on which the vehicle is located, lane stepping information, vehicle body sway information, lane change information, speed change information, braking information, braking information, route information, speeding situation, and information on the degree of agreement between the driving condition and road signs.

In one possible embodiment, the abnormal driving condition of the vehicle includes at least one of the following: the number of times the vehicle crosses a line within a first set time period reaches a predetermined crossing threshold; the width of the vehicle's left-right sway reaches a predetermined width threshold; the vehicle's traffic does not follow the instructions of road signs or traffic signals; and the vehicle's speed exceeds a predetermined speed threshold.

Here, the fact that the number of times the vehicle steps on the line within the first set time has reached a predetermined line step count threshold indicates that the vehicle steps on the line continuously within a short time. In such a case, it can be determined that the vehicle is in an abnormal driving state. The first set time can be set according to the actual situation, and may be, for example, one minute or 15 seconds. The predetermined line step count threshold can also be set according to the actual situation, and may be, for example, two times, and is not particularly limited in this disclosure.

When the width of the vehicle's lateral sway reaches a predetermined width threshold, it indicates that the width of the vehicle body's lateral sway is relatively large. In such a case, it can be determined that the vehicle is in an abnormal driving state. The predetermined width threshold can be set according to the actual situation, such as the type of vehicle, and is not particularly limited in this disclosure.

Furthermore, if the vehicle does not follow the directions of road signs or traffic signals and if the vehicle's speed exceeds a predetermined speed threshold, this also indicates that the vehicle is in an abnormal driving state, and a detailed explanation will be omitted here.

The vehicle's driving condition information may be determined based on information such as the vehicle's driving speed, driving direction, position, and the environment around the vehicle, and this information can be obtained using a sensor for detecting environmental information outside the vehicle cabin and/or a sensor for detecting the vehicle's operating condition.

In one optional embodiment, acquiring the vehicle driving state information includes acquiring sensing data including at least one of video information collected by the vehicle's advanced driver assistance system (ADAS), video information collected by the vehicle's drive recorder, and speed information detected by the vehicle's speed sensor, and determining the vehicle's driving state information based on the driving sensing data.

For example, an image collection device, millimeter wave/laser radar, sonar sensor, etc. in the vehicle body acquires information for monitoring vehicle conditions such as images, temperature, pressure, etc. outside the vehicle cabin. These sensors can be placed in the vehicle's front and rear bumpers, side mirrors, inside the steering column, or on the windshield to collect sensing data outside the vehicle cabin. This sensing data can detect the environment around the vehicle, including information on pedestrians, vehicles, and other objects around the vehicle, as well as lanes, traffic signals, road sign lights, etc. In addition, information such as vehicle speed, acceleration, position, and attitude can be identified using speed/acceleration sensors, attitude sensors, positioning devices, etc.

In step S12, image information of the vehicle's driving area is acquired.

The driving area here may be the area in the vehicle cabin where the driver is located, or any area including the area where the driver is located. Typically, the area is the driver's seat area.

Therefore, the video information of the driving area may be video information of the area inside the vehicle where the driver is located. The video information may be collected by an on-board video collection device arranged inside or outside the vehicle, and the on-board video collection device may be an on-board camera or a video collection device with a camera arranged therein. The camera may be a camera for collecting video information inside the vehicle, or a camera for collecting video information outside the vehicle.

For example, the camera may include a camera in the DMS and/or a camera in the OMS, and these cameras can collect video information inside the vehicle. The camera may include a camera in the ADAS, and these cameras can collect video information outside the vehicle. Of course, the in-vehicle video collection device may be a camera in another system or a separately located camera, and the embodiments of the present disclosure do not specifically limit the in-vehicle video collection device.

The carrier of the image information here may be a two-dimensional image or video. For example, the image information may specifically be a visible light image/video or an infrared light image/video, or may be a three-dimensional image formed by a point cloud obtained by radar scanning, and may be determined according to the actual application scene, and the present disclosure is not particularly limited.

The collected video information can be obtained through a communication connection established with the vehicle-mounted video collection device. In one example, the vehicle-mounted video collection device can transmit the collected video information in real time to an in-vehicle controller or a remote server via a bus or a wireless communication channel, and the in-vehicle controller or the remote server can receive the real-time video information via the bus or the wireless communication channel.

In step S13, the driving state of the driver of the vehicle is determined based on the driving state information of the vehicle and the image information of the driving area.

The driving state of a driver may be the state when the driver drives a vehicle. The driving state may be divided into a normal driving state and a dangerous driving state. A dangerous driving state may be a state in which the driver is performing a pre-defined non-normative driving behavior, such as making a phone call, looking at a mobile phone, taking his/her hands off the steering wheel, or driving while fatigued.

In the embodiment of the present disclosure, when a driver exhibits non-normative driving behavior, this is often accompanied by a change in the vehicle driving state. For example, if the driver is in a dangerous driving state, such as looking away from the road or driving while fatigued, the vehicle's driving state may also exhibit abnormalities, such as an unstable vehicle speed or abnormal driving state, such as continuous line-stepping. Therefore, in the embodiment of the present disclosure, the driver's driving state can be determined based on image information of the driving area and vehicle driving state information in order to increase the reliability of the determined driver's driving state.

As described above, the video information of the driving area is video information of the area inside the vehicle where the driver is located. Therefore, by analyzing and processing the video information of the driving area using image processing technology, it is possible to detect the driver's characteristics, such as physical characteristics and facial characteristics, and to analyze the driver's behavior based on the detected driver's characteristics to determine whether the driver will make a phone call, drink water, take their hands off the steering wheel, or close their eyes.

When the vehicle's driving state is already abnormal, there is a relatively high probability that the driver is in a pre-defined dangerous driving state. Therefore, the vehicle's driving state information and the video information of the driving area can be combined to determine the driver's driving state. For example, if the reliability of determining that the driver is in an overfatigued driving state is 0.5 by analyzing the video information of the driving area and the vehicle is in a continuous stepping-on-the-line driving state at this time, the reliability of the driver being in an overfatigued driving state can be increased to 0.8. Also, for example, if the video information of the driving area is analyzed to determine that the driver is in a mild overfatigued driving state and the vehicle is in a continuous stepping-on-the-line driving state at this time, the mild overfatigued driving state can be increased to a moderate overfatigued driving state or a severe overfatigued driving state.

In step S14, intelligent drive control is performed in response to detection that the driver of the vehicle is in a pre-defined dangerous driving state.

The pre-set dangerous driving state is a pre-set driving state that poses a risk to safe driving, and may include one or more pre-defined driver states, such as a moderate overworked driving state, a severe fatigue state, a severe distracted driving state, etc. The pre-set dangerous driving state may be a state defined by combining a pre-set driver behavior with a pre-set vehicle state, for example, a state in which the driver makes a phone call and the vehicle steps on the line two or more times consecutively can be defined as a pre-set dangerous driving state. Intelligent drive control can be performed for the pre-set dangerous driving state. The intelligent drive control here may involve intervention in driving depending on the road conditions on which the vehicle is traveling by a vehicle control center or a remote control terminal. A specific intelligent drive control method may be, for example, measures such as sending alarm information, controlling the activation of assisted driving/automatic driving functions, and controlling the deceleration of the vehicle. The method will be described in detail in combination with possible embodiments of the present disclosure below, and detailed description will be omitted here.

In some alternative embodiments, the danger level can be determined based on the driver's dangerous driving state and the vehicle's running state, and a corresponding intelligent drive control method can be selected according to the danger level. For example, when the driver is in a dangerous driving state, such as making a call, looking at a mobile phone, or taking his hands off the steering wheel, and the vehicle's continuous stepping on the line does not exceed a preset time (e.g., 5 seconds), intelligent drive control is performed by sending alarm information, and when the driver is in a dangerous driving state, such as making a call, looking at a mobile phone, or taking his hands off the steering wheel, and the vehicle is in a state where the vehicle is overspeeding by 50%, the vehicle is in a state where the continuous stepping on the line is more dangerous than a state where the vehicle's continuous stepping on the line does not exceed a preset time (e.g., 5 seconds), intelligent drive control can be performed by controlling the vehicle to decelerate, thereby realizing flexible safe driving control.

In an embodiment of the present disclosure, vehicle driving state information and image information of the vehicle's driving area are acquired, the driving state of the vehicle driver is determined based on the vehicle driving state information and the image information of the driving area, and intelligent drive control is performed in response to detection that the vehicle driver is in a pre-set dangerous driving state. This allows the driving state of the vehicle driver to be determined by combining the vehicle driving state and the image information of the driving area, increasing the accuracy of the determined driving state of the driver. In addition, by performing intelligent drive control in response to detection that the vehicle driver is in a pre-set dangerous driving state, the safety of vehicle driving is further improved.

In some possible embodiments, the video information of the driving area may be video information of the interior of the vehicle cabin, and the vehicle driving state information may be information determined based on sensing data outside the vehicle cabin. In other words, in the present disclosure, the driving state of the driver can be determined by combining information from inside and outside the vehicle cabin, thereby increasing the accuracy of the determined driving state of the driver. In addition, the safety of vehicle driving is improved by determining the driving state of the driver by combining information from inside and outside the vehicle cabin and performing intelligent drive control for the vehicle.

In one possible embodiment, determining the driving state of the driver of the vehicle based on the driving state information of the vehicle and the image information of the driving area includes performing behavior detection of the driver of the vehicle based on the image information of the driving area, and determining the driving state of the driver based on the driving state information of the vehicle and the driver's behavior detection result.

In this embodiment, the behavior of the vehicle driver is detected based on the video information of the driving area to obtain the behavior detection result, and then the driving state of the driver is judged by combining the vehicle's driving state information with the behavior detection result, thereby increasing the accuracy of the obtained driving state of the driver.

The image information of the driving area is analyzed by image processing technology to perform behavior detection for the driver, and a behavior detection result can be obtained. In the embodiment of the present disclosure, the preset dangerous driving state may be a state in which the driver performs a preset non-normative driving behavior. Therefore, the behavior detection for the driver of the vehicle based on the image information of the driving area may be detection of the non-normative driving behavior of the driver. For example, when it is detected that the mobile phone is held in the driver's hand and near the ear, the behavior detection result can be determined to be a behavior in which the driver makes a call while driving. When it is detected that the time during which the driver's hands are not on the steering wheel has reached a first time, the behavior detection result can be determined to be a behavior in which the driver takes his/her hands off the steering wheel. When it is detected that the time during which the driver's eyes are closed has reached a second time, the behavior detection result can be determined to be a behavior in which the driver performs mild overfatigue driving. When it is detected that the time during which the driver's eyes are closed has reached a third time, the behavior detection result can be determined to be a behavior in which the driver performs moderate overfatigue driving.

In some optional embodiments, if the detection result is that no unsafe driving behavior by the driver is detected, and the vehicle driving state information indicates that the vehicle driving state is normal, it can be determined that the driver's driving state is normal.

In some optional embodiments, if the detection result indicates that an unsafe driving behavior by the driver has been detected, and the vehicle driving state information indicates that the vehicle driving state is normal, the driver's driving state can be determined based on the detected non-normative driving behavior. For example, if the behavior detection result indicates that the frequency with which the driver closes his/her eyes or the number of consecutive times that the driver closes his/her eyes is within an interval corresponding to mild overfatigue driving, the driver's driving state is still determined to be mild overfatigue driving even if the vehicle driving state information is normal.

In some optional embodiments, when the detection result indicates that an unsafe driving behavior by the driver has been detected, if the vehicle driving state information indicates that the vehicle driving state is abnormal, the driver's behavior detection result can be further modified based on the vehicle driving state information to obtain the driver's driving state in order to improve the reliability of the obtained behavior detection result. For example, since the abnormal driving state of the vehicle is often caused by non-normative driving behavior by the driver, the reliability of the driver's behavior being unsafe in the behavior detection result can be increased to improve the reliability of the obtained dangerous driving state. Also, since the vehicle is already in an abnormal driving state, the danger level of the detection result can be increased to take a higher level of response measures.

In one possible embodiment, performing behavior detection of the driver of the vehicle includes determining a danger level corresponding to the driver's behavior detection result, and determining the driving state of the driver based on the vehicle driving state information and the driver's behavior detection result includes increasing the danger level of the driver's behavior detection result to a first danger level in response to determining that the vehicle is in an abnormal driving state based on the vehicle driving state information, and determining that the driver's driving state is in a preset dangerous driving state in response to determining that the first danger level has reached a preset alarm level.

The driver's behavior detection result may correspond to a danger level, and the danger level represents the degree of danger of the driver's dangerous driving state. For example, if it is detected that the driver has his/her eyes closed for a second time, the behavior detection result can be determined to indicate that the driver is exhibiting a behavior of mild fatigue driving. If it is detected that the driver has his/her eyes closed for a third time, the behavior detection result can be determined to indicate that the driver is exhibiting a behavior of moderate fatigue driving. Here, the third time is longer than the second time. It is clear that the longer the driver has his/her eyes closed, the higher the degree of danger and the higher the probability of a vehicle accident occurring.

Because the vehicle is already in an abnormal driving state, the danger level of the detection result can be raised and a higher level of response measures can be taken. Here, for convenience of explanation, the raised danger level is referred to as the first danger level. For the first danger level, it is determined whether the first danger level has reached a preset alarm level, and if it is determined that it has reached the level, an alarm can be issued or other smart control operations can be performed.

For example, mildly fatigued driving behavior may not require an alarm, but severely fatigued driving behavior will be alarmed because it corresponds to a pre-defined dangerous driving state.

In this embodiment, by determining the danger level corresponding to the driver's behavior detection result, if the vehicle is in an abnormal driving state, the danger level of the driver's behavior detection result is raised to a first danger level, and if the first danger level reaches a preset alarm level, the driver's driving state is determined to be in a preset dangerous driving state. As a result, if the vehicle is already in an abnormal driving state, the danger level of the detection result is raised and countermeasures are taken in a timely and accurate manner, thereby improving the safety of vehicle driving.

In one possible embodiment, the determining of the driving state of the driver based on the driving state information of the vehicle and the behavior detection result of the driver includes determining that the driver is in a pre-set dangerous driving state in response to the determination that both the driving state of the vehicle and the behavior state of the driver are abnormal based on the driving state information of the vehicle and the behavior detection result of the driver.

In this embodiment, if the vehicle's driving condition is abnormal, it may cause a traffic accident. If the driver's behavior is also abnormal at this time, it is considered that the driver is not driving safely and there is a high safety risk. Therefore, if the vehicle's driving condition and the driver's behavior are both abnormal, it can be determined that the driver is in a preset dangerous driving condition. In this way, intelligent drive control is performed in response to detection that the driver of the vehicle is in a preset dangerous driving condition, thereby improving the safety of vehicle driving.

In one possible embodiment, determining the driving state of the driver of the vehicle based on the driving state information of the vehicle and the image information of the driving area includes performing behavior detection of the driver based on the image information of the driving area in response to detection that the vehicle is in an abnormal driving state based on the driving state information of the vehicle, and determining that the driver is in a pre-set dangerous driving state in response to the driver's behavior detection result being determined to be inattentive driving or fatigued driving.

In this case, distracted driving by the driver may be behavior that diverts attention from driving, such as taking the driver's hands off the steering wheel, making a phone call while driving, drinking water while driving, or using a mobile phone while driving, while fatigued driving may be behavior such as the driver closing their eyes for a long period of time or yawning.

In this embodiment, if the vehicle's driving condition is abnormal, it may cause a traffic accident. If the driver's behavior detection result at this time is that the driver is distracted or fatigued, it can be determined that the vehicle is in an unsafe state due to the driver's distracted or fatigued driving. Therefore, if the vehicle's driving condition is abnormal and the driver's behavior detection result is that the driver is distracted or fatigued, it can be determined that the driver is in a pre-set dangerous driving state. This increases safety by timely intervention of intelligent drive control.

In one possible embodiment, performing the intelligent drive control includes generating alarm information based on at least one of the vehicle's driving state information, the behavior detection result of the driver of the vehicle based on the video information of the driving area, and the driving state of the driver.

In this embodiment, in order to achieve a better alarm effect, at least one of the vehicle driving state information, the driver's behavior detection result, and the driver's driving state can be notified to the driver, so that the driver can recognize the severity of the current dangerous driving state. For example, if the vehicle driving state information is "the vehicle has stepped on the line three times in a row" and the vehicle driver's behavior detection result based on the video information of the driving area is "the driver is making a phone call", the generated alarm information may be "the vehicle has stepped on the line three times in a row, please do not make a phone call while driving". The specific form of issuing the alarm information may be audio or video, but the present disclosure is not particularly limited thereto.

In this embodiment, alarm information is generated based on at least one of the vehicle driving state information, the driver's behavior detection result, and the driver's driving state, so that the driver can know the degree of the vehicle's unsafe state and the severity of the current dangerous driving state, thereby achieving a better alarm effect.

In one possible embodiment, performing the intelligent drive control includes controlling an alarm device to continue outputting alarm information until both the vehicle driving state and the driver's driving state return to normal. By continuing to output alarm information to continuously alert the driver, the alarm effect is further improved, continuous monitoring and alarming of the driver's state is realized, and this is advantageous for further improving driving safety. When both the vehicle driving state and the driver's driving state return to normal, the alarm is stopped, reducing interference with the driver caused by unnecessary alarms.

In one possible embodiment, performing the intelligent drive control includes controlling an assisted driving/automated driving function to be activated and/or controlling the vehicle to decelerate and stop in response to detecting that the driver is in a pre-defined unsafe driving state for a duration exceeding a pre-defined time.

When the driver is in a pre-set dangerous driving state for a duration exceeding a pre-set time, it indicates that the driver is unable to operate the vehicle normally for an extended period of time. This has a significant impact on the vehicle's driving safety and poses a significant safety risk. Therefore, in such a case, the assisted driving/automated driving function can be controlled to be activated or the vehicle can be controlled to decelerate and stop. Controlling the vehicle to decelerate and stop may be achieved by the assisted driving/automated driving function.

Assisted driving/autonomous driving functions can adapt the vehicle to changing traffic conditions by controlling acceleration, braking and direction based on driving sensing data collected by the vehicle's sensors, and can realize driving functions such as adaptive cruise control, keeping the vehicle in its lane, and controlling the following distance between the vehicle and the vehicle ahead.

In this embodiment, when the driver is in a pre-set dangerous driving state for a duration exceeding a pre-set time, the assisted driving/automated driving function is controlled to be activated and/or the vehicle is controlled to decelerate and stop, thereby controlling the vehicle's driving state when the driver is unable to operate the vehicle normally for a long period of time, thereby improving the driving safety of the vehicle.

In one possible embodiment, the method further comprises obtaining environmental information about the vehicle;
As described above, determining the driving state of the driver of the vehicle based on the driving state information of the vehicle and the image information of the driving area includes determining the driving state of the driver of the vehicle based on environmental information around the vehicle, the driving state information of the vehicle, and the image information of the driving area.

In an actual driving scene, the driving state of a vehicle may be affected by the environment around the vehicle. For example, the vehicle's speed may increase or decrease due to traffic congestion, or the vehicle may sway from side to side when driving on a curved road. Therefore, in order to further improve the accuracy of the determined driving state of the vehicle driver, the driving state of the vehicle can be determined based on environmental information around the vehicle, driving state information of the vehicle, and image information of the driving area.

The environmental information around the vehicle here may include road traffic information of the road on which the vehicle is traveling, and the road traffic information may represent the condition of the road on which the vehicle is traveling, and may include at least one of information indicating the degree of congestion of the road, the shape of the road, the type of the road, and the speed limit of the road. The condition of the road can be divided into a number of levels, such as very congested, congested, and smooth, based on the degree of congestion of the road, the shape of the road can be divided into shapes such as curves and straight lines, and the type of the road can be divided into types such as urban roads, rural roads, and expressways.

In one possible embodiment, acquiring environmental information about the vehicle includes acquiring road traffic information about the road on which the vehicle is traveling, and determining the driving state of the driver of the vehicle based on the environmental information about the vehicle, the vehicle's driving state information, and the image information of the driving area includes determining that the driver is in a predetermined dangerous driving state when the vehicle's driving state information and the road traffic information do not satisfy a predetermined correspondence relationship and the behavior detection result of the driver of the vehicle based on the image information of the driving area indicates that the driver's behavior state is abnormal.

Road traffic information can be obtained by third party platforms, such as navigation service platforms, based on the vehicle's current geographic location.

Based on the environmental information around the vehicle and the vehicle's driving state information, it can be determined whether the vehicle's driving state information and road traffic information satisfy a preset correspondence relationship, and if the vehicle's driving state information and road traffic information do not satisfy the preset correspondence relationship, it indicates that the vehicle is in an abnormal driving state.

The preset correspondence relationship is a correspondence relationship between road traffic information and driving condition information of a vehicle that is driving safely in the road conditions indicated by the road traffic information. For example, it may include at least one of the following: a correspondence relationship between "smooth" road conditions and a vehicle traveling at a constant speed, a correspondence relationship between straight road conditions and a vehicle's left-right sway width not exceeding a predetermined width threshold, and a correspondence relationship between road speed limit information and traveling within the speed limit range.

Accordingly, the fact that the vehicle driving state information and the road traffic information do not satisfy the preset correspondence relationship may include, for example, the vehicle's left/right sway width reaches a predetermined width threshold on a straight road section, the vehicle is traveling too fast or too slow on a road section with smooth traffic flow, the vehicle is speeding, etc. All of these cases indicate that the vehicle is in an abnormal driving state.

In this way, when the vehicle is in an abnormal driving state and the driver's behavior state is also abnormal, it can be determined that the driver is detected to be in a pre-defined dangerous driving state. This increases the accuracy of the determined driver's driving state, and further improves the safety of vehicle driving.

In one optional embodiment, judging the driving state of the driver of the vehicle based on the driving state information of the vehicle and the image information of the driving area includes inputting the acquired driving state information of the vehicle and the image information of the driving area into a trained neural network and judging the driving state of the driver of the vehicle through the neural network. In this embodiment, the accuracy and speed of judging the driving state can be improved by judging the driving state of the vehicle driver using a neural network. In addition, since the neural network can be trained and deployed in advance, it can quickly realize detection of images or video streams with large data sizes, and can be adapted to intelligent drive control of real-time driving scenes.

Below, one application scenario according to an embodiment of the present disclosure is described. In this application scenario, video information of the driving area is collected by a DMS camera, the processor acquires the video information of the driving area from the DMS camera, and the behavior of the driver of the vehicle is detected based on the video information of the driving area to obtain a behavior detection result. Also, video information of the outside of the vehicle cabin is collected by an ADAS camera, the processor acquires the video information of the outside of the vehicle cabin from the ADAS camera, and the vehicle driving state information can be determined. If the determined vehicle driving state information is "the vehicle has stepped on the line three times in a row" and the behavior detection result of the vehicle driver is "the driver is making a phone call", audio alarm information is generated and broadcast, stating "the vehicle has stepped on the line three times in a row, please do not make a phone call while driving".

It should be understood that the above method embodiments mentioned in this disclosure can be combined with each other to form embodiments as long as the principles and logic are not violated. Due to space limitations, detailed descriptions are omitted in this disclosure. Those skilled in the art should understand that in the above method according to the specific embodiment, the execution order of each step is specifically determined by its function and internal logic.

The present disclosure further provides an intelligent drive control device, an electronic device, a computer-readable storage medium, and a program. Any of these can be used to realize any one of the intelligent drive control methods of the present disclosure. For the corresponding technical means and explanations, refer to the corresponding description of the method, and detailed explanations will be omitted.

2 shows a block diagram of an intelligent drive control device according to an embodiment of the present disclosure. As shown in FIG. 2, the device includes:
a first acquisition module 21 for acquiring vehicle running state information;
a second acquisition module 22 for acquiring image information of the driving area of the vehicle;
A driving state determination module 23 for determining a driving state of a driver of the vehicle based on driving state information of the vehicle and image information of the driving area;
and a control module 24 for performing intelligent drive control in response to detection of a driver of the vehicle being in a pre-defined unsafe driving condition.

In one possible embodiment, the driving state determination module 23 includes a first driving state determination sub-module and a second driving state determination sub-module;
The first driving state determination submodule detects a behavior of a driver of the vehicle based on image information of the driving area;
The second driving state determination submodule determines the driving state of the driver based on the vehicle running state information and the driver behavior detection result.

In one possible embodiment, the first driving state determination submodule determines a danger level corresponding to the driver's behavior detection result;
The second driving state determination submodule:
raising a danger level of the driver's behavior detection result to a first danger level in response to the vehicle being determined to be in an abnormal driving state based on the vehicle driving state information;
In response to a determination that the first danger level has reached a preset alarm level, it is determined that the driving state of the driver is in a preset dangerous driving state.

In one possible embodiment, the second driving state determination submodule determines that the driver is in a pre-defined dangerous driving state when it is determined that both the vehicle's driving state and the driver's behavior state are abnormal based on the vehicle's driving state information and the driver's behavior detection result.

In one possible embodiment, the driving state determination module 23 includes a third driving state determination sub-module and a fourth driving state determination sub-module;
The third driving state determination submodule performs behavior detection of the driver based on image information of the driving area in response to detection that the vehicle is in an abnormal driving state based on driving state information of the vehicle;
The fourth driving state determination submodule determines that the driver is in a predetermined dangerous driving state in response to the driver's behavior detection result being determined to be inattentive driving or fatigue driving.

In one possible embodiment, the abnormal operating condition of the vehicle is
The detection signal includes at least one of the following: the number of times the vehicle steps on a line within a first set time period reaches a predetermined threshold number of times the vehicle steps on a line; the width of the vehicle's left and right sway reaches a predetermined width threshold; the vehicle's passage does not follow the instructions of road signs or traffic signals; and the vehicle's speed exceeds a predetermined speed threshold.

In one possible embodiment, the control module 24 comprises:
information on the running state of the vehicle;
A behavior detection result of the driver of the vehicle based on image information of the driving area;
and a driving state of the driver.

In one possible embodiment, the control module 24 controls the alarm device to continue outputting alarm information until both the vehicle driving state and the driver's driving state return to normal.

In one possible embodiment, the control module 24 controls an assisted/automated driving function to be activated and/or controls the vehicle to decelerate and stop in response to detecting that the driver is in a pre-defined unsafe driving condition for a duration exceeding a pre-defined time.

In one possible embodiment, the device comprises:
Further comprising a third acquisition module for acquiring environmental information around the vehicle;
The driving state determination module 23 determines the driving state of the driver of the vehicle based on environmental information around the vehicle, driving state information of the vehicle, and image information of the driving area.

In one possible embodiment, the environmental information comprises road traffic information;
The third acquisition module acquires road traffic information of a road on which the vehicle is traveling,
The driving state determination module determines that the driver is in a predetermined dangerous driving state when the vehicle's driving state information and the road traffic information do not satisfy a predetermined correspondence relationship and the behavior detection result of the driver of the vehicle based on the image information of the driving area indicates that the driver's behavior state is abnormal.

In one embodiment, the first acquisition module 21 includes a sensing data acquisition submodule and a driving state information determination submodule;
The sensing data acquisition submodule acquires driving sensing data including at least one of video information collected by an advanced driving assistance system of the vehicle, video information collected by a drive recorder of the vehicle, and speed information detected by a speed sensor of the vehicle;
The driving state information determination submodule determines driving state information of the vehicle based on the driving sensing data.

In one possible embodiment, the driving state determination module 23 inputs the acquired driving state information of the vehicle and image information of the driving area into a trained neural network and determines the driving state of the driver of the vehicle through the neural network.

In some embodiments, the functions or modules of the apparatus according to the embodiments of the present disclosure can be utilized to perform the methods described in the above-mentioned method embodiments, and for specific implementations thereof, reference may be made to the description of the above-mentioned method embodiments, and detailed description thereof will be omitted here for the sake of simplicity.

In the embodiment of the present disclosure,
a first sensor for collecting image information of a driving area of the vehicle;
A vehicle is further provided that includes a controller for acquiring vehicle driving state information, determining a driving state of a driver of the vehicle based on the vehicle driving state information and image information of the driving area, and performing intelligent drive control in response to detection that the driver of the vehicle is in a predetermined dangerous driving state.

The vehicle of this embodiment judges the driving state of the vehicle driver by combining the vehicle's running state and image information of the driving area, thereby improving the accuracy of the judged driving state of the driver.

In some possible embodiments, the vehicle further includes a second sensor for collecting driving sensing data, and the controller obtains driving state information of the vehicle based on the driving sensing data. This allows accurate detection of the driving state by the sensor for detecting different data of the vehicle.

An embodiment of the present disclosure further provides a computer-readable storage medium having computer program instructions stored thereon, the computer-readable storage medium implementing the above-described intelligent drive control method when executed by a processor. The computer-readable storage medium may be a volatile computer-readable storage medium or a non-volatile computer-readable storage medium.

An embodiment of the present disclosure further provides an electronic device including a processor and a memory for storing instructions executable by the processor, the processor configured to execute the above-described intelligent drive control method by calling the instructions stored in the memory.

An embodiment of the present disclosure further provides a computer program product including computer readable code that, when executed in an apparatus, causes a processor of the apparatus to execute instructions for implementing an intelligent drive control method according to any of the embodiments described above.

An embodiment of the present disclosure further provides another computer program product having computer-readable instructions stored therein that, when executed, cause a computer to perform operations of the intelligent drive control method according to any of the embodiments described above.

The electronic device may be provided as a terminal, a server or other form of device.

FIG. 3 shows a block diagram of an electronic device 800 according to an embodiment of the present disclosure. For example, the electronic device 800 may be a terminal such as a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, or a personal digital assistant.

Referring to FIG. 3, the electronic device 800 may include one or more of a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, a sensor component 814, and a communication component 816.

The processing component 802 typically controls the overall operation of the electronic device 800, such as operations related to display, phone calls, data communication, camera operation, and recording operation. The processing component 802 may include one or more processors 820 that execute instructions to perform all or some of the steps of the above methods. The processing component 802 may also include one or more modules for interaction with other components. For example, the processing component 802 may include a multimedia module for interaction with the multimedia component 808.

Memory 804 is configured to store various types of data to support operation on electronic device 800. These data include, by way of example, instructions for any application programs or methods operating on electronic device 800, contact data, phone book data, messages, pictures, videos, etc. Memory 804 can be implemented by various types of volatile or non-volatile storage devices, such as, for example, static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks, or combinations thereof.

The power component 806 provides power to each component of the electronic device 800. The power component 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the electronic device 800.

The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and a user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). When the screen includes a touch panel, it may be implemented as a touch screen that receives input signals from a user. The touch panel may include one or more touch sensors to detect touches, slides, and gestures on the touch panel. The touch sensors may detect not only the boundaries of a touch or slide motion, but also the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and/or a rear camera. When the electronic device 800 is in an operational mode, such as a photography mode or a capture mode, the front camera and/or the rear camera may receive external multimedia data. Each front camera and rear camera may have a fixed optical lens system or a focal length and optical zoom capability.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a microphone (MIC) configured to receive external audio signals when the electronic device 800 is in an operational mode, such as a call mode, a record mode, and a voice recognition mode. The received audio signals may be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting the audio signals.

The I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include, but are not limited to, a home button, volume buttons, a start button, and a lock button.

The sensor component 814 includes one or more sensors for evaluating the state of each aspect of the electronic device 800. For example, the sensor component 814 can detect the on/off state of the electronic device 800, the relative positioning of components such as the display and keypad of the electronic device 800, and the sensor component 814 can further detect changes in the position of the electronic device 800 or certain components of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration or deceleration of the electronic device 800, and temperature changes of the electronic device 800. The sensor component 814 may include a proximity sensor configured to detect the presence of a nearby object in the absence of any physical contact. The sensor component 814 may further include an optical sensor for use in imaging applications, such as a complementary metal oxide semiconductor (CMOS) or charge-coupled device (CCD) image sensor. In some embodiments, the sensor component 814 may further include an acceleration sensor, a gyro sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 816 is configured to realize wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (WiFi), a second generation mobile communication technology (2G), or a third generation mobile communication technology (3G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be realized by radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the electronic device 800 may be implemented with one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processors (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic elements and may be used to perform the above method.

In an exemplary embodiment, a non-volatile computer readable storage medium, such as a memory 804, is further provided that includes computer program instructions that, when executed by the processor 820 of the electronic device 800, can cause the method to be performed.

FIG. 4 illustrates a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. Referring to FIG. 4, the electronic device 1900 includes a processing component 1922 including one or more processors, and memory resources, represented by memory 1932, for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. The processing component 1922 is also configured to perform the above method by executing the instructions.

The electronic device 1900 may further include a power component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input/output (I/O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft server operating system (Windows Server), a graphical user interface operating system from Apple (Mac OS X), a multi-user, multi-process computer operating system (Unix), a free and open source Unix-like operating system (Linux), an open source code Unix-like operating system (FreeBSD), or similar.

In an exemplary embodiment, a non-volatile computer readable storage medium, such as memory 1932, is further provided that includes computer program instructions that, when executed by processing component 1922 of electronic device 1900, can cause the method described above to be performed.

The present disclosure may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions for causing a processor to implement aspects of the present disclosure.

A computer readable storage medium may be a tangible device capable of storing and storing instructions for use in an instruction execution device. A computer readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. Further specific examples (non-exhaustive list) of computer readable storage media include portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or slot-in protrusion structures on which instructions are stored, and any suitable combination of the above. As used herein, a computer readable storage medium is not to be construed as a momentary signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated via a waveguide or other transmission medium (e.g., pulsed light passing through a fiber optic cable), or electrical signals transmitted via electrical wires.

The computer readable program instructions described herein may be downloaded from a computer readable storage medium into each computing/processing device, or may be downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface within each computing/processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage in a computer readable storage medium within each computing/processing device.

Computer program instructions for carrying out the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine language instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or target code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, and general procedural programming languages such as "C" or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. When a remote computer is involved, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, the state information of the computer-readable program instructions may be used to personalize electronic circuitry, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), and the electronic circuitry may execute the computer-readable program instructions to achieve aspects of the present disclosure.

Although aspects of the present disclosure have been described herein with reference to flowcharts and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure, it should be understood that each block of the flowcharts and/or block diagrams, and combinations of blocks in the flowcharts and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing device, such that when these instructions are executed by the processor of the computer or other programmable data processing device, an apparatus is produced that performs the functions/operations specified in one or more blocks of the flowcharts and/or block diagrams. These computer-readable program instructions may be stored on a computer-readable storage medium to cause a computer, programmable data processing device, and/or other device to operate in a particular manner. Thus, a computer-readable storage medium having instructions stored thereon includes an article of manufacture having instructions that perform each aspect of the functions/operations specified in one or more blocks of the flowcharts and/or block diagrams.

The computer-readable program instructions may be loaded into a computer, other programmable data processing device, or other device, and cause the computer, other programmable data processing device, or other device to perform a series of operational steps to produce a computer-implemented process. In this manner, the instructions executed on the computer, other programmable data processing device, or other device implement the functions/operations specified in one or more blocks of the flowcharts and/or block diagrams.

The flowcharts and block diagrams in the drawings illustrate possible system architectures, functions, and operations of systems, methods, and computer program products according to several embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, program segment, or part of instructions, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions depicted in the blocks may be implemented in a different order than that depicted in the drawings. For example, two consecutive blocks may be executed substantially in parallel, or may be executed in a reverse order depending on the function. It should also be noted that each block in the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts, may be implemented by a dedicated system based on hardware that executes the specified function or operation, or may be implemented by a combination of dedicated hardware and computer instructions.

The computer program product may be embodied in hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is embodied as a computer storage medium. In another alternative embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK).

Although the embodiments of the present disclosure have been described above, the above description is merely illustrative and not exhaustive, and is not limited to the embodiments disclosed. Various modifications and alterations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms chosen in this specification are intended to favorably interpret the principles, practical applications, or improvements to existing technology of the embodiments, or to allow other skilled in the art to understand the embodiments disclosed herein.

Claims (13)

Acquiring vehicle driving state information;
acquiring image information of a driving area of the vehicle;
Determining a driving state of a driver of the vehicle based on driving state information of the vehicle and image information of the driving area;
In response to detection of a driver of the vehicle being in a predetermined dangerous driving state, performing intelligent drive control;
The above-mentioned determining the driving state of the driver of the vehicle based on the driving state information of the vehicle and the image information of the driving area includes:
detecting a behavior of a driver of the vehicle based on image information of the driving area;
determining a driving state of the driver based on driving state information of the vehicle and a behavior detection result of the driver,
The above-mentioned detecting the behavior of the driver of the vehicle includes:
determining a first danger level corresponding to the driver's behavior detection result;
The above-mentioned determining the driving state of the driver based on the vehicle running state information and the driver behavior detection result,
In response to the vehicle being determined to be in an abnormal driving state based on the vehicle driving state information rather than the behavior detection result , a first danger level of the driver's behavior detection result is increased to a second danger level having a higher danger level;
and determining that the driver's driving state is in a predetermined dangerous driving state in response to determining that the second danger level has reached a predetermined alarm level. The above-mentioned determining the driving state of the driver based on the vehicle running state information and the driver behavior detection result,
determining that the driver is in a dangerous driving state set in advance in response to the determination that both the vehicle driving state and the driver behavior state are abnormal based on the vehicle driving state information and the driver behavior detection result;
2. The intelligent drive control method according to claim 1, further comprising at least one of: detecting a behavior of the driver based on the image information of the driving area in response to detection that the vehicle is in an abnormal driving state based on the driving state information of the vehicle; and determining that the driver is in a predetermined dangerous driving state in response to the driver's behavior detection result being determined to be inattentive driving or fatigue driving. The abnormal driving condition of the vehicle is
The intelligent drive control method according to claim 1 or 2, characterized in that the method includes at least one of the following: the number of times the vehicle steps on the line within a first set time reaches a predetermined number of times threshold; the width of the vehicle's left and right sway reaches a predetermined width threshold; the vehicle's traffic does not follow the instructions of road signs or traffic signals; and the vehicle's speed exceeds a predetermined speed threshold. The above-mentioned intelligent drive control is performed
information on the running state of the vehicle;
A behavior detection result of the driver of the vehicle based on image information of the driving area;
and a driving state of the driver.
The above-mentioned intelligent drive control is performed
Controlling the alarm device so as to continue outputting alarm information until both the vehicle running state and the driver's driving state return to normal;
The intelligent drive control method according to any one of claims 1 to 3, characterized in that it includes at least one of controlling an assisted driving/automated driving function to be activated and/or controlling the vehicle to decelerate and stop in response to detection that the driver is in a predetermined dangerous driving state for a duration exceeding a predetermined time. The method further includes acquiring environmental information about the vehicle;
The above-mentioned determining a driving state of a driver of the vehicle based on the driving state information of the vehicle and the image information of the driving area includes:
The intelligent drive control method according to any one of claims 1 to 4, further comprising: determining the driving state of the driver of the vehicle based on the environmental information around the vehicle, the driving state information of the vehicle, and the image information of the driving area. The environmental information includes road traffic information,
The acquiring of environmental information about the surroundings of the vehicle includes acquiring road traffic information about a road on which the vehicle is traveling,
The above-mentioned determining a driving state of a driver of the vehicle based on environmental information around the vehicle, driving state information of the vehicle, and image information of the driving area includes:
The intelligent drive control method according to claim 5, further comprising: determining that the driver is in a predetermined dangerous driving state according to the fact that the vehicle driving state information and the road traffic information do not satisfy a predetermined correspondence relationship and the behavior detection result of the driver of the vehicle based on the image information of the driving area indicates that the behavior state of the driver is abnormal. The above-mentioned determining the driving state of the driver of the vehicle based on the driving state information of the vehicle and the image information of the driving area includes:
The intelligent drive control method according to any one of claims 1 to 6, further comprising: inputting the acquired vehicle driving state information and driving area image information into a trained neural network, and judging the driving state of the driver of the vehicle through the neural network. A first acquisition module for acquiring vehicle running state information;
a second acquisition module for acquiring image information of a driving area of the vehicle;
A driving state determination module for determining a driving state of a driver of the vehicle based on driving state information of the vehicle and image information of the driving area;
A control module for performing intelligent drive control in response to detection of a driver of the vehicle being in a predetermined dangerous driving state,
The driving state determination module includes a first driving state determination sub-module and a second driving state determination sub-module;
The first driving state determination submodule detects a behavior of a driver of the vehicle based on image information of the driving area;
The second driving state determination submodule determines a driving state of the driver based on the vehicle running state information and the driver's behavior detection result,
The first driving state determination submodule determines a first danger level corresponding to a driver's behavior detection result;
The second driving state determination submodule:
In response to a determination that the vehicle is in an abnormal driving state based on the vehicle driving state information rather than the behavior detection result , a first danger level of the driver's behavior detection result is increased to a second danger level which is a higher danger level;
An intelligent drive control device characterized in that, in response to determining that the second danger level has reached a predetermined alarm level, it is determined that the driving state of the driver is in a predetermined dangerous driving state. a first sensor for collecting image information of a driving area of the vehicle;
A controller for acquiring vehicle driving state information, judging a driving state of a driver of the vehicle based on the vehicle driving state information and image information of the driving area, and performing intelligent drive control in response to detection that the driver of the vehicle is in a predetermined dangerous driving state,
The above-mentioned determining the driving state of the driver of the vehicle based on the driving state information of the vehicle and the image information of the driving area includes:
detecting a behavior of a driver of the vehicle based on image information of the driving area;
determining a driving state of the driver based on driving state information of the vehicle and a behavior detection result of the driver,
The above-mentioned detecting the behavior of the driver of the vehicle includes:
determining a first danger level corresponding to the driver's behavior detection result;
The above-mentioned determining the driving state of the driver based on the vehicle running state information and the driver behavior detection result,
In response to the vehicle being determined to be in an abnormal driving state based on the vehicle driving state information rather than the behavior detection result , a first danger level of the driver's behavior detection result is increased to a second danger level having a higher danger level;
determining that the driving state of the driver is in a predetermined dangerous driving state in response to determining that the second danger level has reached a predetermined alarm level. a second sensor for collecting driving sensing data;
The vehicle according to claim 9 , wherein the controller acquires information about a driving state of the vehicle based on the driving sensing data. A processor;
a memory for storing instructions executable by a processor;
8. An electronic device, characterized in that the processor is configured to execute the intelligent drive control method according to any one of claims 1 to 7 by calling instructions stored in the memory. A computer-readable storage medium having computer program instructions stored thereon, comprising:
A computer-readable storage medium, comprising the computer program instructions, when executed by a processor, for implementing the intelligent drive control method according to any one of claims 1 to 7. A computer program including computer-readable code that, when run in an electronic device, causes a processor of the electronic device to execute instructions for implementing the intelligent drive control method described in any one of claims 1 to 7.

Images