JP7735565B2 - Method, device and vehicle for detecting when a vehicle occupant's body part is placed out of a window - Google Patents

Description

This application claims priority to a Chinese patent application, application number 202111273194.9, entitled "Method, device and vehicle for detecting when a body part of a vehicle occupant has been pushed out of a window," filed with the China Patent Office on October 29, 2021, the entire disclosure of which is incorporated herein by reference.

This disclosure relates to the field of computers, and more particularly to a method, device, and vehicle for detecting when a body part of a vehicle occupant has been placed out a window.

When a driver or passenger sticks their limbs out of a window while driving a vehicle, this is an extremely dangerous act, threatening driving safety and making it likely to result in a traffic accident. Therefore, how to accurately detect when a driver or passenger sticks their limbs out of a window and respond in a timely manner is of great importance to the safety of the driver and passengers inside the vehicle.

This disclosure proposes a technical solution for detecting when a vehicle occupant's body part is sticking out of a window.

According to one aspect of the present disclosure, detecting whether a body part of a vehicle occupant has protruded from a window based on at least two types of sensing data including image data collected by an on-board camera and radar sensing data collected by a radar sensor for detecting an obstacle in a window area, and obtaining at least two systems of detection results;
determining a determination result based on the at least two detection results;
The method for detecting that a body part of a vehicle occupant has stuck out a window includes generating attention-calling information when the determination result indicates that the body part of the occupant has stuck out the window.

In one possible implementation, the method comprises:
The method further includes acquiring vehicle body signal data including door data indicating an open/close state of a door and window data indicating an open/close state of a window;
The above-described detection of whether or not a body part of a vehicle occupant has stuck out a window based on each of at least two types of sensing data, and obtaining at least two types of detection results,
When the vehicle body signal data indicates that the door is closed and the window corresponding to the seating position of the passenger is open, the method detects whether or not a body part of the passenger has protruded from the window based on each of at least two types of sensing data of the vehicle, and obtains the detection results of the at least two systems.

In one possible implementation, when the vehicle body signal data indicates that the door is closed and the window corresponding to the seating position of the vehicle occupant is open, detecting whether or not a body part of the vehicle occupant has protruded out of the window based on each of at least two types of sensing data, and obtaining at least two systems of detection results,
Transmitting a trigger signal to the radar sensor to collect sensing data, and acquiring the radar sensing data collected by the radar sensor;
acquiring image data collected by the on-board camera;
Detecting whether or not a body part of the occupant has protruded from a window corresponding to the seating position of the occupant based on the radar sensing data and the image data, respectively, and obtaining a detection result based on the radar sensor and a detection result based on the image data.

In one possible implementation, the method comprises:
The method further includes determining, as the determination result, that no body part of the occupant is sticking out of a window when the vehicle body signal data indicates that a door is open or that all windows corresponding to the occupant's seating position are closed.

In one possible implementation mode, detecting whether a body part of a vehicle occupant has stuck out a window based on each of the at least two types of sensing data and obtaining at least two systems of detection results includes:
Recognizing a relative state between a body part of a passenger and a window in the image data to obtain a first detection result;
obtaining a second detection result based on the radar sensing data;
The step of determining the determination result based on the at least two detection results includes:
The method includes determining a determination result based on the first detection result and the second detection result.

The step of determining a determination result based on the first detection result and the second detection result includes:
When both the first detection result and the second detection result indicate that a body part of the occupant has been stuck out of a window, specifying as the determination result that the body part of the occupant has been stuck out of a window;
When the first detection result indicates that a body part of the passenger has been put out of a window and the second detection result indicates that the body part has not been put out of a window, specifying that the body part of the passenger has not been put out of a window as the determination result;
When the first detection result indicates that a body part has not been put out through the window and the second detection result indicates that a body part has been put out through the window, the determination result identifies that a body part of the occupant has been put out through the window.

In one possible implementation mode, detecting whether a body part of a vehicle occupant has stuck out a window based on each of the at least two types of sensing data and obtaining at least two systems of detection results includes:
Recognizing a relative state between a body part of a passenger and a window in the image data to obtain a first detection result;
When the first detection result indicates that a body part has been stuck out of a window, acquiring seat information of the target passenger who has stuck out a body part in the image data through a window based on the first detection result;
Obtaining the radar sensing data of a target window corresponding to seat information of the target passenger;
and identifying a second detection result based on the radar sensing data of the target window.

In one possible implementation form, the acquiring of the radar sensing data of the target window corresponding to the seat information of the target occupant includes:
acquiring the radar sensing data of a target window corresponding to the seat information of the target occupant, and vehicle body signal data related to the seat information of the target occupant, including first signal data indicating an open/close state of a target door that is a door corresponding to the seat information of the target occupant, and second signal data indicating an open/close state of the target window;
Identifying the second detection result based on the radar sensing data includes:
The method includes identifying the second detection result based on the radar sensing data and the vehicle body signal data.

In one possible implementation mode, the step of determining a determination result based on the at least two detection results includes:
When the second detection result indicates that a body part of the occupant has been stuck out of the window, the determination result includes identifying that a body part of the occupant has been stuck out of the window.

In one possible implementation mode, the step of recognizing a relative state between a body part of a passenger and a window in the image data and obtaining a first detection result includes:
The method includes recognizing a relative state between a body part of a vehicle occupant and a window based on the image data, and obtaining a degree of confidence that the body part of the vehicle occupant has been pushed out of the window.

In one possible implementation mode, the step of determining a determination result based on the at least two detection results includes:
If the reliability is greater than a predetermined threshold, and if the second detection result indicates that a body part of the vehicle occupant has stuck out a window, identifying the body part of the vehicle occupant as having stuck out a window as the determination result;
If the reliability is smaller than a predetermined threshold, and if the second detection result indicates that a body part of the vehicle occupant has stuck out a window, identifying the body part of the vehicle occupant as having stuck out a window as the determination result;
The method includes at least one of: when the reliability is smaller than a predetermined threshold value; when the second detection result indicates that the body part of the vehicle occupant is not sticking out of the window; and when the second detection result indicates that the body part of the vehicle occupant is not sticking out of the window, identifying the body part of the vehicle occupant as not sticking out of the window as the judgment result.

In one possible implementation, the vehicle-mounted camera includes a camera installed inside the vehicle cabin and/or a camera installed outside the vehicle cabin.

In one possible implementation, the radar sensor is located on the upper and lower edges of the window.

In one possible implementation, the sensing data further includes infrared sensing data and/or millimeter wave sensing data.

According to one aspect of the present disclosure, there is provided a detection module for detecting whether a body part of a vehicle occupant has protruded from a window based on at least two types of sensing data, including image data collected by an on-board camera and radar sensing data collected by a radar sensor for detecting an obstacle in a window area, and for obtaining at least two types of detection results;
an identification module for identifying a determination result based on the at least two detection results;
and a warning module for generating warning information when the determination result indicates that the body part of the vehicle occupant has protruded through the window.

In one possible implementation, the device comprises:
further comprising a vehicle body signal data acquisition module for acquiring vehicle body signal data including door data indicating an open/closed state of a door and window data indicating an open/closed state of a window;
When the vehicle body signal data indicates that the door is closed and the window corresponding to the passenger's seating position is open, the detection module is used to detect whether or not the passenger's body part has protruded out the window based on each of at least two types of sensing data from the vehicle, and to obtain the at least two systems of detection results.

In one possible implementation, if the vehicle body signal data indicates that a door is closed and a window corresponding to a seating position of a passenger is open, the detection module:
Transmitting a trigger signal to the radar sensor to collect sensing data, and acquiring the radar sensing data collected by the radar sensor;
acquiring image data collected by the on-board camera;
It is used to detect whether or not a body part of the occupant has protruded from a window corresponding to the occupant's seating position based on the radar sensing data and the image data, respectively, and to obtain detection results based on the radar sensor and detection results based on the image data.

In one possible implementation, the device is further used to determine, as the determination result, that the occupant's body part is not sticking out of the window when the vehicle body signal data indicates that a door is open or that all windows corresponding to the occupant's seating position are closed.

In one possible implementation, the detection module comprises:
Recognizing a relative state between a body part of a passenger and a window in the image data to obtain a first detection result;
and obtaining a second detection result based on the radar sensing data;
The specific module is
The determination result is determined based on the first detection result and the second detection result.

In one possible implementation, the specific module:
When both the first detection result and the second detection result indicate that a body part of the occupant has been stuck out of a window, specifying as the determination result that the body part of the occupant has been stuck out of a window;
When the first detection result indicates that a body part of the passenger has been put out of a window and the second detection result indicates that the body part has not been put out of a window, specifying that the body part of the passenger has not been put out of a window as the determination result;
When the first detection result indicates that a body part has not been put out through the window and the second detection result indicates that a body part has been put out through the window, the determination result is used to at least one of identify that a body part of the occupant has been put out through the window.

In one possible implementation, the detection module comprises:
Recognizing a relative state between a body part of a passenger and a window in the image data to obtain a first detection result;
When the first detection result indicates that a body part has been stuck out of a window, acquiring seat information of the target passenger who has stuck out a body part in the image data through a window based on the first detection result;
Obtaining the radar sensing data of a target window corresponding to seat information of the target passenger;
and determining a second detection result based on the radar sensing data of the target window.

In one possible implementation form, the acquiring of the radar sensing data of the target window corresponding to the seat information of the target occupant includes:
acquiring the radar sensing data of a target window corresponding to the seat information of the target occupant, and vehicle body signal data related to the seat information of the target occupant, including first signal data indicating an open/close state of a target door that is a door corresponding to the seat information of the target occupant, and second signal data indicating an open/close state of the target window;
The step of identifying the second detection result based on the radar sensing data of the target window includes:
The method includes identifying the second detection result based on the radar sensing data and the vehicle body signal data.

In one possible implementation, when the second detection result indicates that the occupant's body part has been projected through the window, the identification module is used to identify, as the determination result, that the occupant's body part has been projected through the window.

In one possible implementation mode, the step of recognizing a relative state between a body part of a passenger and a window in the image data and obtaining a first detection result includes:
The method includes recognizing a relative state between a body part of a vehicle occupant and a window based on the image data, and obtaining a degree of confidence that the body part of the vehicle occupant has been pushed out of the window.

In one possible implementation, the specific module:
If the reliability is greater than a predetermined threshold, and if the second detection result indicates that a body part of the vehicle occupant has stuck out a window, identifying the body part of the vehicle occupant as having stuck out a window as the determination result;
If the reliability is smaller than a predetermined threshold, and if the second detection result indicates that a body part of the vehicle occupant has stuck out a window, identifying the body part of the vehicle occupant as having stuck out a window as the determination result;
The reliability is used for at least one of the following purposes: when the reliability is smaller than a predetermined threshold value; when the second detection result indicates that the body part of the vehicle occupant is not sticking out of the window; and when the second detection result indicates that the body part of the vehicle occupant is not sticking out of the window, the judgment result is used for at least one of the following purposes: when the reliability is smaller than a predetermined threshold value; and when the second detection result indicates that the body part of the vehicle occupant is not sticking out of the

In one possible implementation, the vehicle-mounted camera includes a camera installed inside the vehicle cabin and/or a camera installed outside the vehicle cabin.

In one possible implementation, the radar sensor is located on the upper and lower edges of the window.

In one possible implementation, the sensing data further includes infrared sensing data and/or millimeter wave sensing data.

According to one aspect of the present disclosure, there is provided a vehicle monitoring system including: at least two types of sensors for collecting at least two types of sensing data, including image data collected by an on-board camera and radar sensing data collected by a radar sensor for detecting an obstacle in a window area;
A vehicle is provided that includes a controller that acquires the at least two types of sensing data from the sensor, detects whether or not a body part of an occupant of the vehicle has been exposed through a window based on the at least two types of sensing data, and generates warning information when a body part of the occupant has been exposed through the window.

One aspect of the present disclosure provides an electronic device including a processor and a memory for storing instructions executable by the processor, the processor configured to call the instructions stored in the memory to execute the above-described method for detecting when a body part of a vehicle occupant has been placed out a window.

One aspect of the present disclosure provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the above-described method for detecting when a body part of a vehicle occupant has been placed out a window.

One aspect of the present disclosure provides a computer program product that includes computer-readable code or a computer-readable storage medium having computer-readable code stored thereon, and that, when the computer-readable code is executed in a processor of an electronic device, causes the processor of the electronic device to execute the method for detecting when a body part of a vehicle occupant has been placed out a window, as described above.

In an embodiment of the present disclosure, whether a body part of a vehicle occupant has protruded from a window is detected based on at least two types of sensing data, including image data collected by an on-board camera and radar sensing data collected by a radar sensor for detecting obstacles in the window area, and at least two types of detection results are obtained. A determination result is identified based on the at least two types of detection results, and if the determination result indicates that a body part of the occupant has protruded from a window, warning information is generated. According to an embodiment of the present disclosure, by combining image data and radar sensing data to identify the determination result and generate warning information, it is advantageous to improve the accuracy of the determination result and improve riding safety.

It should be understood that the foregoing general description and the following detailed description are merely exemplary and explanatory and are not intended to limit the present disclosure. Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments taken in conjunction with the accompanying drawings.

The drawings included as part of the specification illustrate embodiments of the present disclosure and, together with the specification, explain the technical means of the present disclosure.
1 is a flowchart illustrating a method for detecting a body part of a vehicle occupant sticking out of a window according to an embodiment of the present disclosure. 1 is a schematic diagram illustrating a method for detecting a body part of a vehicle occupant sticking out of a window according to an embodiment of the present disclosure. 1 is a block diagram illustrating an apparatus for detecting when a body part of a vehicle occupant is sticking out of a window according to an embodiment of the present disclosure. FIG. 19 is a block diagram illustrating an electronic device 1900 according to an embodiment of the present disclosure.

Various exemplary embodiments, features, and aspects of the present disclosure are described in detail below with reference to the drawings. In the drawings, like reference numerals represent functionally identical or similar elements. While various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

The word "exemplary" as used herein means "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or superior over other embodiments.

In this specification, the term "and/or" merely describes the relationship between related objects and indicates that three relationships can exist. For example, A and/or B can indicate three cases: A exists only, A and B exist simultaneously, and B exists only. In addition, in this specification, the term "at least one" indicates any one of a plurality of elements or any combination of at least two of a plurality of elements. For example, including at least one of A, B, and C can indicate including any one or more elements selected from the set consisting of A, B, and C.

Furthermore, in order to better explain the present disclosure, many specific details are set forth in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can be similarly practiced without some specific details. In some examples, detailed descriptions of methods, means, elements, and circuits that are well known to those skilled in the art are omitted in order to emphasize the gist of the present disclosure.

While the vehicle is moving, the driver or other passengers may make dangerous maneuvers such as sticking their hands, head, or other body parts out the window, which could result in a serious accident.

An embodiment of the present disclosure provides a technical solution for detecting whether a body part of a vehicle occupant has stuck out a window, and detects whether a body part of a vehicle occupant has stuck out a window based on each of at least two types of sensing data, obtains at least two systems of detection results, and identifies a judgment result based on the at least two systems of detection results. If the judgment result indicates that a body part of the occupant has stuck out a window, warning information is generated. According to an embodiment of the present disclosure, by combining image data and radar sensing data, it is possible to accurately identify whether a body part of a vehicle occupant has stuck out a window, and if the judgment result indicates that a body part of a vehicle occupant has stuck out a window, warning information is generated, which is advantageous in improving the riding safety of occupants.

1 is a flowchart illustrating a method for detecting a body part of a vehicle occupant sticking out a window according to an embodiment of the present disclosure. In one possible implementation, the method for detecting a body part of a vehicle occupant sticking out a window may be performed by an electronic device such as an in-vehicle device, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a wearable device, or a terminal device or server. Here, the in-vehicle device may be an in-vehicle controller, a domain controller, or in-vehicle infotainment, or may be a device host for performing data processing operations such as image processing in an Occupant Monitoring System (OMS) or a Driver Monitoring System (DMS), etc. In some possible implementations, the method for detecting when a body part of a vehicle occupant has been placed out of a window may be implemented by a processor invoking computer-readable instructions stored in a memory. As shown in FIG. 1, the method for detecting when a body part of a vehicle occupant has been placed out of a window includes steps S11 to S13.

In step S11, based on at least two types of sensing data, including image data collected by an on-board camera and radar sensing data collected by a radar sensor for detecting obstacles in the window area, it is detected whether or not a body part of a vehicle occupant has protruded from the window, and at least two detection results are obtained.

In step S12, the judgment result is determined based on the detection results of at least two systems.

In step S13, if the determination result indicates that the passenger's body part has protruded through the window, warning information is generated.

Embodiments of the present disclosure may be applied to any type of vehicle, such as a passenger car, taxi, online ride-hailing service, shared car, or bus. The vehicle type may be, for example, a compact car or a sport utility vehicle (SUV), but is not limited to this in embodiments of the present disclosure. The vehicle occupant may include any person riding in the vehicle. For example, the vehicle occupant may include the driver and/or a non-driver. For example, in a private car application scenario, the non-driver may be the owner's family or friends, etc. In an online ride-hailing application scenario, the non-driver may be a passenger, etc. The body part of the occupant is a part of the occupant's body and may include at least one of the hands, arms, head, feet, and legs.

In one possible implementation, if the vehicle has two types of sensors for detecting whether a body part of a vehicle occupant has protruded out the window, two types of sensing data can be acquired in step S11. For example, this includes image data collected by an on-board camera and radar sensing data collected by a radar sensor for detecting obstacles in the window area.

With regard to image data collected by an onboard camera, the onboard camera may include a camera installed inside the vehicle cabin and/or a camera installed outside the vehicle cabin. The onboard camera may be any camera installed in the vehicle. The number of onboard cameras may be one or more. The type of onboard camera may be a DMS camera, an OMS camera, an exterior camera, etc., but this disclosure is not limited to these. The image data collected by the onboard camera may be image data of the vehicle cabin environment captured by a camera installed inside or outside the vehicle cabin, such as a DMS camera, an OMS camera, or an exterior camera. This image data must include image information of at least the seating areas and window areas of the vehicle interior, i.e., at least some of the seating areas of people and at least some of the window areas within the field of view of the camera.

As OMS and DMS systems become more widespread in vehicles, it should be understood that if the onboard OMS and DMS cameras can be used directly, it will be possible to detect whether or not a passenger's body part has been placed out the window without the need for a separate camera, which will be advantageous in terms of reducing costs.

Regarding the radar sensing data collected by the radar sensor, the radar sensor may be provided at the upper/lower edge of the window. For example, a radar sensor may be provided at the upper/lower edge of each window of the vehicle. The number of radar sensors may be one or more. The type of radar sensor may be ultrasonic radar, electromagnetic radar, laser radar, etc., but this disclosure is not limited to these. The radar sensing data collected by the radar sensor may be radar sensing data collected by a radar sensor such as ultrasonic radar, electromagnetic radar, or laser radar provided at the upper/lower edge of each window, and is intended to detect whether or not an obstacle is present in each window area.

After acquiring image data and radar sensing data, it is possible to detect whether a vehicle occupant's body part has protruded from the window based on each of these two types of data, thereby obtaining a first detection result based on the image data and a second detection result based on the radar sensing data.

In one possible implementation, the relative state of the occupant's body parts and the window in the image data is recognized to obtain a first detection result.

Based on the image data, the relative state between the vehicle occupant and the window in the image data can be recognized to detect whether a body part of the vehicle occupant has protruded through the window, thereby obtaining a first detection result. The relative state indicates the relative position or overlap between the occupant's body and the window, and may include whether the occupant's body part is on the outside or inside of the window. As an alternative, the image data may be processed using a neural network model to recognize the relative state between the vehicle occupant and the window in the image data. For example, a large amount (1,000 images) of image data related to the vehicle interior environment may be used to pre-train a neural network model for detecting whether a body part of the vehicle occupant has protruded through the window, and the image data may be input into the trained neural network model to obtain a first detection result. Here, the neural network model for detecting whether a vehicle occupant's body part is sticking out the window may be a neural network model such as a convolutional neural network (CNN), a recurrent neural network (RNN), or a generative adversarial network (GAN), but this disclosure is not limited to this. As an alternative, the relative state of the vehicle occupant and the window may be identified by detecting the window and the person's body part in the image data, determining whether their positions overlap based on the detection results, and determining whether at least a portion of the person's body part is outside the vehicle body.

In one possible implementation, a second detection result is obtained based on the radar sensing data.

Based on radar sensing data, it may be possible to detect whether a body part of a vehicle occupant has protruded from a window and obtain a second detection result. The radar sensor is for detecting obstacles and may be located, for example, outside the window, below the lower edge of the window, or at least one of the upper and lower edges of the window. The radar sensing data may be used to determine whether an obstacle exists between the upper and lower edges of the window, or the radar sensing data may be used to obtain the distance to the detected obstacle (e.g., the distance between the obstacle and the radar sensor) and determine whether the distance to the obstacle is shorter than a preset distance (e.g., the distance to the obstacle detected by a radar sensor located on the lower edge of the window is shorter than the distance between the upper edge of the window and the radar sensor). Here, multiple radar sensors may be installed on the upper and lower edges of the same window. The second detection result may be obtained by comprehensively analyzing multiple radar sensing data collected by multiple radar sensors on the window. However, this disclosure does not limit the specific method for determining the second detection result based on radar sensing data.

In step S11, a first detection result based on image data and a second detection result based on radar sensing data are obtained. In step S12, a judgment result can be determined based on the first detection result and the second detection result. Here, the judgment result can be determined using various judgment methods, but this disclosure does not limit the specific judgment method.

For example, assume that the values of the first and second detection results corresponding to the same window can be represented by "0" and "1." Here, "0" indicates that no part of the vehicle occupant's body is sticking out of the window, and "1" indicates that the vehicle occupant's body is sticking out of the window.

In one possible implementation, an OR operation is performed on the first detection result and the second detection result, and the result of the OR operation of the first detection result and the second detection result can be identified as the determination result. In this way, if either the first detection result or the second detection result indicates that a body part of a vehicle occupant has been sticking out of the window, a determination result that the body part of a vehicle occupant has been sticking out of the window can be obtained, which is advantageous in reducing the rate at which a determination is missed that a body part of a vehicle occupant has been sticking out of the window.

Alternatively, in another possible implementation, an AND operation can be performed on the first detection result and the second detection result, and the result of the AND operation between the first detection result and the second detection result can be identified as the determination result. In this way, the occupant's body part is identified as having been stuck out of the window only if both the first detection result and the second detection result indicate that a body part has been stuck out of the window, which is advantageous in reducing the rate of misidentification of a vehicle occupant's body part having been stuck out of the window.

Alternatively, in another possible implementation, preset weights may be set for the first and second detection results, and the magnitude of the influence of the first and second detection results on the judgment result may be determined based on the magnitude of the preset weights, thereby obtaining the final judgment result.

In another possible implementation, radar sensing data may be used as the basis. If the second detection result based on the radar sensing data indicates that the body part has been stuck out the window, the determination result is determined to be that the body part of the occupant has been stuck out the window. In one possible implementation, determining the determination result based on the first detection result and the second detection result includes at least one of: if the first detection result and the second detection result both indicate that the body part has been stuck out the window, determining that the body part of the occupant has been stuck out the window; if the first detection result indicates that the body part has been stuck out the window and the second detection result indicates that the body part has not been stuck out the window, determining that the body part of the occupant has not been stuck out the window; and if the first detection result indicates that the body part has not been stuck out the window and the second detection result indicates that the body part has been stuck out the window, determining that the body part of the occupant has been stuck out the window.

In this disclosure, the magnitude of the preset weights is not limited. By setting the preset weights for the first and second detection results, an appropriate determination method can be selected for a specific scene, which is advantageous in improving the accuracy of the determination results.

In step S12, the determination result is identified. In step S13, if the determination result indicates that a body part of the occupant has protruded through the window, warning information can be generated.

For example, if the determination result indicates that the occupant has stuck a body part out of the window, a warning message stating "Do not stick your body out of the window" can be displayed via the in-vehicle infotainment and/or center console, etc. Here, the warning message may be in the form of text warning message, audio warning message, on-screen warning message, etc., but this disclosure is not limited to these.

In one possible implementation, the vehicle may have more than two types of sensors for detecting whether a vehicle occupant has placed a body part through the window, i.e., in addition to an on-board camera and radar sensor, the vehicle may also be equipped with other sensors for detecting obstacles in the window area, such as infrared sensors or millimeter-wave sensors.

After identifying the first and second detection results based on the image data and radar sensing data, it is possible to detect whether a body part of a vehicle occupant has protruded from a window based on sensing data collected by other types of sensors (e.g., infrared sensing data and/or millimeter-wave sensing data), and identify the corresponding detection results of other systems. A determination result may be identified from the detection results of multiple systems, and if the determination result indicates that a body part of the occupant has protruded from the window, warning information may be generated. Here, the more types of sensing data acquired, the more detection results can be obtained from each type of sensor, and the more accurate the determination results based on the detection results of the more systems. For specific processes, please refer to the above description of the two types of sensors, and a detailed description will be omitted here.

Accordingly, in an embodiment of the present disclosure, it is detected whether a body part of a vehicle occupant has stuck out a window based on each of at least two types of sensing data, at least two types of detection results are obtained, and a determination result is identified based on the at least two types of detection results. If the determination result indicates that a body part of the occupant has stuck out a window, warning information is generated. According to an embodiment of the present disclosure, it is possible to accurately identify whether a body part of a occupant has stuck out a window, and if the determination result indicates that a body part of the occupant has stuck out a window, warning information can be generated, which is advantageous in improving the safety of occupants in the vehicle cabin.

The following describes a method for detecting when a vehicle occupant's body part is sticking out a window, according to an embodiment of the present disclosure.

Figure 2 is a schematic diagram showing a method for detecting whether a body part of a vehicle occupant has stuck out a window according to an embodiment of the present disclosure. As shown in Figure 2, data for detecting whether a body part of a vehicle occupant has stuck out a window may include radar sensing data collected by a radar sensor, image data collected by an onboard camera, and vehicle body signal data. By comprehensively analyzing and processing the radar sensing data, image data, and vehicle body signal data by a processor, it is possible to determine whether a body part of the occupant has stuck out a window, and to generate warning information when a body part of the occupant has stuck out a window.

It should be understood that the embodiment shown in FIG. 2 is merely exemplary, and that the sensing data according to the embodiments of the present disclosure may further include radar sensing data collected by a radar sensor for detecting obstacles in the window area, millimeter wave sensing data collected by a millimeter wave sensor for detecting obstacles in the window area, etc., but that the present disclosure is not limited thereto.

In one possible implementation, the method further includes acquiring vehicle body signal data including door data indicating whether a door is open or closed, and window data indicating whether a window is open or closed.

For example, vehicle body signal data can be acquired from a vehicle body bus such as a Local Interconnect Network (LIN) bus or a Controller Area Network (CAN) bus, and the open/closed status of each door and each window can be acquired based on the vehicle body signal data.

Here, in addition to acquiring vehicle body signal data from the vehicle body bus, vehicle body signal data indicating the open/closed state of doors and windows can be acquired by installing sensors in the door or window areas, but this disclosure does not limit the specific method for acquiring vehicle body signal data.

In addition, step S11 may include, when the vehicle body signal data indicates that the door is closed and the window corresponding to the passenger's seating position is open, detecting whether or not the passenger's body part has protruded out the window based on each of at least two types of sensing data from the vehicle, and obtaining detection results from the at least two systems.

In one possible implementation, vehicle body signal data and at least two types of sensing data (e.g., including image data and radar sensing data) can be acquired in real time. If the vehicle body signal data indicates that a door is closed and a window corresponding to a passenger's seating position is open, it is possible to detect, based on the image data, whether a body part of the passenger corresponding to the open window in the image data has protruded through the window, thereby obtaining a first detection result, and to detect, based on the radar sensing data, whether the body part of the passenger has protruded through the window, thereby obtaining a second detection result.

For example, assume that the vehicle body signal data indicates that the door is closed and the window corresponding to the driver's seating position is open. In this case, based on the image data, a region of interest, i.e., a region including the driver and the driver's window, can be extracted from the image data, and it can be detected whether any part of the driver's body is sticking out of the window in that region, resulting in a first detection result. Based on the radar sensing data corresponding to the driver's window, it can be detected whether any part of the driver's body is sticking out of the window, resulting in a second detection result.

In this way, vehicle body signal data and at least two types of sensing data are acquired in real time, and if the vehicle body signal data indicates that a door is closed and a window corresponding to the passenger's seating position is open, detection is performed on each of the at least two types of sensing data, resulting in at least two detection results. This method is simple and easy to implement, and is advantageous in improving the efficiency of on-site implementation.

In one possible implementation, if the vehicle body signal data indicates that the door is closed and the window corresponding to the passenger's seating position is open, step S11 includes sending a trigger signal to the radar sensor to collect sensing data, acquiring radar sensing data collected by the radar sensor, acquiring image data collected by the onboard camera, and detecting whether a body part of the passenger has protruded from the window corresponding to the passenger's seating position based on the radar sensing data and the image data, respectively, and obtaining a detection result based on the radar sensor and a detection result based on the image data.

For example, if vehicle body signal data is first acquired and the vehicle body signal data indicates that a door is closed and a window corresponding to the passenger's seating position is open, a radar sensor is triggered to collect radar sensing data and image data collected by an onboard camera is acquired. For example, if the vehicle body signal data indicates that a door is closed and a window corresponding to the passenger's seating position is open, a trigger signal (e.g., one or more high pulse signals) can be sent to the radar sensor, activating the radar sensor to detect obstacles in the window area and collecting radar sensing data. It should be understood that the onboard camera is in an operational state because it can continuously acquire image data. If the vehicle body signal data indicates that a door is closed and a window corresponding to the passenger's seating position is open, image data collected by the onboard camera can be obtained without sending a signal to trigger its operation.

Based on the image data, it may be possible to detect whether a body part of the occupant corresponding to an open window in the image data has protruded through the window, thereby obtaining a first detection result, and based on the radar sensing data, it may be possible to detect whether the body part of the occupant has protruded through the window, thereby obtaining a second detection result.

In this way, when the trigger conditions are met, the radar sensor is triggered to collect radar sensing data, which is advantageous for reducing energy consumption and extending the useful life of the radar sensor.

In one possible implementation, if the vehicle body signal data indicates that a door is open or that all windows corresponding to the passenger's seating position are closed, the determination result is that the passenger's body part is not sticking out of the window.

For example, when a vehicle door is open, the vehicle may not be in motion, e.g., passengers getting in or out. There is a significant hidden danger behind the act of a vehicle passenger sticking their limbs out the window while the vehicle is in motion. Conversely, for a vehicle that is not in motion, there is little point in detecting whether a passenger's body parts are sticking out the window. Therefore, if the vehicle body signal data indicates that a door is open (meaning the vehicle may not be in motion) or that the windows corresponding to the passenger's seating positions are both closed, the determination result can directly determine that the passenger's body parts are not sticking out the window, without taking image data and radar sensing data into consideration.

In this way, there is no need to detect image data or radar sensing data, and it is possible to directly determine that the passenger's body parts are not sticking out of the window, thereby improving determination efficiency.

In another possible scenario, image data may be acquired first, and radar sensing data (or radar sensing data and vehicle body signal data) corresponding to a target window in the image data may be acquired in accordance with the image data. The target window is a target window corresponding to a target occupant with a body part sticking out of the window in the image data.

In one possible implementation, step S11 comprises:
Recognizing a relative state between a body part of a passenger and a window in the image data to obtain a first detection result;
When the first detection result indicates that a body part has been stuck out of a window, acquiring seat information of the target passenger who has stuck out a body part in the image data through a window based on the first detection result;
Obtaining the radar sensing data of a target window corresponding to seat information of the target passenger;
and identifying a second detection result based on the radar sensing data of the target window.

For example, the image data may contain multiple passengers and multiple windows, and the relative state of each passenger's body part in relation to the window in the image data can be recognized, and a first detection result can be obtained by detecting whether each passenger's body part is sticking out of the window. For example, suppose that certain image data includes passengers A to C, and that passenger B's body part is sticking out of the window. This image data can be input into a trained neural network to obtain a first detection result for this image data, namely, that the body parts of passengers A and C are not sticking out of the window, and that passenger B's body part is sticking out of the window.

If the first detection result indicates that a body part of an occupant has stuck out the window, occupant B, who has stuck out the window according to the first detection result, is designated as the target occupant, and seat information relating to the seat in which this occupant is sitting (for example, the passenger seat position) is identified. The window closest to the seat information of target occupant B is designated as the target window, for example, the passenger seat window corresponding to the passenger seat position. Radar sensing data of the passenger seat window is acquired, and the second detection result is determined based on this radar sensing data.

In this way, radar sensing data of the target window is obtained according to the image data, and the second detection result is determined based on the radar sensing data of the target window, which is advantageous in improving the accuracy of the second detection result.

In one possible implementation, acquiring the radar sensing data of the target window corresponding to the seat information of the target occupant includes acquiring the radar sensing data of the target window corresponding to the seat information of the target occupant, and vehicle body signal data related to the seat information of the target occupant, including first signal data indicating the open/closed state of the target door, which is the door corresponding to the seat information of the target occupant, and second signal data indicating the open/closed state of the target window, and identifying the second detection result based on the radar sensing data includes identifying the second detection result based on the radar sensing data and the vehicle body signal data.

For example, the first detection result shows that neither Passenger A nor Passenger C has their body parts sticking out the window, but Passenger B, who is sitting in the passenger seat, has their body parts sticking out the window. Here, Passenger B, who has their body parts sticking out the window, is the target passenger. From the first detection result, it is possible to identify that the seat information of target Passenger B is the passenger seat position. Radar sensing data for the passenger window (the window adjacent to the passenger seat) and vehicle body signal data related to the passenger seat position are acquired. Here, the open/closed state of the passenger door and passenger window can be determined based on the vehicle body signal data related to the passenger seat position, and the vehicle body signal data related to the passenger seat position may include first signal data indicating the open/closed state of the door at the passenger seat position and second signal data indicating the open/closed state of the window at the passenger seat position.

The second detection result can be determined based on the radar sensing data of the passenger window, the first signal data, and the second signal data. Here, if the first signal data indicates that the passenger door is open, or the second signal data indicates that the passenger window is closed, the second detection result can be determined to be that the body part of the occupant in the passenger seat is not sticking out of the passenger window.

It should be understood that the above process is merely exemplary, and that in an embodiment of the present disclosure, radar sensing data and associated vehicle body signal data corresponding to the seat information of each target occupant in the image data can be obtained, and a second detection result corresponding to each target occupant can be identified based on the radar sensing data and associated vehicle body signal data corresponding to the seat information of each target occupant, respectively.

In this way, radar sensing data corresponding to the target occupant's seat information and vehicle body signal data related to the target occupant's seat information can be obtained based on the image data, and the second detection result can be determined based on the radar sensing data of the target vehicle window and its associated vehicle body signal data, which is advantageous in further improving the accuracy of the second detection result.

In the above-described example of step S11, if the vehicle body signal data indicates that the door is closed and the window corresponding to the passenger's seating position is open, the corresponding radar sensing data, or radar sensing data and vehicle body signal data, are acquired and the second detection result is identified only if the first detection result indicates that a body part of the passenger has been projected out of the window. Therefore, in the specific example of step S12, if the second detection result indicates that a body part of the passenger has been projected out of the window, the determination result is that the body part of the passenger has been projected out of the window.

For example, assuming that the target occupant with a body part sticking out the window in the image data is a passenger in the front passenger seat, if the second detection result indicates that the passenger's body part is sticking out the window, the determination result is that the passenger's body part is identified as sticking out the window. The seat of the target occupant may be any seat, but it should be understood that this disclosure is not limited to this.

This is beneficial in improving the accuracy of the judgment results.

In one possible implementation, recognizing the relative state of the vehicle occupant's body part and the window in the image data and obtaining a first detection result includes recognizing the relative state of the vehicle occupant's body part and the window based on the image data and obtaining a confidence level that the vehicle occupant's body part has been projected out the window.

For example, recognizing the relative state of the vehicle occupant's body part and the window based on the image data may involve inputting the image data into a trained neural network model, recognizing the relative state of the vehicle occupant's body part and the window using the neural network model, and obtaining a first detection result. The first detection result may include a confidence level that the occupant's body part has been pushed out the window.

Here, the larger the reliability value, the higher the probability that a body part has been removed from the window, and the smaller the reliability value, the lower the probability that a body part has been removed from the window.

In this way, it is possible to obtain a degree of confidence that a vehicle occupant's body part has been exposed through the window, which is advantageous in improving the accuracy of the determination result indicating whether or not a vehicle occupant's body part has been exposed through the window.

In step S11, the reliability of the first detection result is determined. In step S12, the judgment result can be determined based on the reliability of the first detection result and the second detection result.

In one possible implementation, in step S12:
If the reliability is greater than a predetermined threshold, and if the second detection result indicates that a body part of the vehicle occupant has stuck out a window, identifying the body part of the vehicle occupant as having stuck out a window as the determination result;
If the reliability is greater than a predetermined threshold, and if the second detection result indicates that a body part of the vehicle occupant is not sticking out of a window, identifying that a body part of the vehicle occupant is not sticking out of a window as the determination result;
If the reliability is smaller than a predetermined threshold, and if the second detection result indicates that a body part of the vehicle occupant has stuck out a window, identifying the body part of the vehicle occupant as having stuck out a window as the determination result;
The method includes at least one of: when the reliability is smaller than a predetermined threshold value; when the second detection result indicates that the body part of the vehicle occupant is not sticking out of the window; and when the second detection result indicates that the body part of the vehicle occupant is not sticking out of the window, identifying the body part of the vehicle occupant as not sticking out of the window as the judgment result.

For example, if the reliability is greater than a preset threshold (e.g., 0.5), it is determined that there is a high reliability that the occupant's body part has been stuck out the window, based on the image data. In this case, if the second detection result indicates that the vehicle occupant's body part has been stuck out the window, it is determined as the determination result that the vehicle occupant's body part has been stuck out the window.

If the reliability is less than a preset threshold (e.g., 0.5), it is determined that the reliability of the occupant's body part sticking out the window is low based on the image data. In this case, the judgment result can be determined based on the second detection result. Specifically, if the second detection result indicates that the vehicle occupant's body part has stuck out the window, it is determined as the judgment result that the vehicle occupant's body part has stuck out the window, and if the second detection result indicates that the vehicle occupant's body part has not stuck out the window, it is determined as the judgment result that the vehicle occupant's body part has not stuck out the window.

In this way, reliability and the second detection result can be taken into consideration comprehensively, which is advantageous in improving the accuracy of the judgment results.

The above describes several methods for determining a judgment result based on vehicle body signal data and at least two types of sensing data, but it should be understood that the method for determining a judgment result can be flexibly selected depending on the actual situation.

The determination result is determined in steps S11 and S12. In step S13, if the determination result indicates that the passenger's body part has protruded through the window, warning information can be generated.

Accordingly, in an embodiment of the present disclosure, it is detected whether a body part of a vehicle occupant has stuck out a window based on each of at least two types of sensing data, at least two types of detection results are obtained, and a determination result is identified based on the at least two types of detection results. If the determination result indicates that a body part of the occupant has stuck out a window, warning information is generated. According to an embodiment of the present disclosure, by combining image data and radar sensing data, it is possible to accurately identify whether a body part of the occupant has stuck out a window, and if the determination result indicates that a body part of the occupant has stuck out a window, warning information can be generated, which is advantageous in improving the safety of occupants in the vehicle cabin.

It should be understood that the above method embodiments mentioned in this disclosure can be combined with each other to form embodiments without violating the principles and logic. Due to space limitations, detailed descriptions will not be provided in this disclosure. Those skilled in the art will understand that the execution order of each step in the above methods according to specific embodiments depends on their functions and possible underlying logic.

The present disclosure also provides a device, electronic device, computer-readable storage medium, and program for detecting when a body part of a vehicle occupant has been placed out a window, all of which can be used to implement any of the methods for detecting when a body part of a vehicle occupant has been placed out a window according to the present disclosure. For such technical solutions and explanations, please refer to the relevant descriptions in the explanation of the method, and detailed explanations will be omitted here.

3 is a block diagram illustrating an apparatus for detecting a body part of a vehicle occupant sticking out of a window according to an embodiment of the present disclosure. As shown in FIG. 3, the apparatus includes:
a detection module 71 for detecting whether a body part of a vehicle occupant has stuck out a window based on at least two types of sensing data, including image data collected by an on-board camera and radar sensing data collected by a radar sensor for detecting obstacles in the window area, and for obtaining at least two types of detection results;
an identification module 72 for identifying a determination result based on the at least two detection results;
and an attention module 73 for generating attention information when the determination result indicates that the body part of the occupant has protruded out of the window.

In one possible implementation, the device comprises:
further comprising an acquisition module for acquiring vehicle body signal data including door data indicating an open/closed state of a door and window data indicating an open/closed state of a window;
When the vehicle body signal data indicates that the door is closed and the window corresponding to the passenger's seating position is open, the detection module 71 is used to detect whether or not the passenger's body part has protruded out of the window based on each of at least two types of sensing data from the vehicle, and to obtain the at least two systems of detection results.

In one possible implementation, if the vehicle body signal data indicates that a door is closed and a window corresponding to a seating position of a passenger is open, the detection module 71
Transmitting a trigger signal to the radar sensor to collect sensing data, and acquiring the radar sensing data collected by the radar sensor;
acquiring image data collected by the on-board camera;
It is used to detect whether or not a body part of the occupant has protruded from a window corresponding to the occupant's seating position based on the radar sensing data and the image data, respectively, and to obtain detection results based on the radar sensor and detection results based on the image data.

In one possible implementation, the device is further used to determine, as the determination result, that the occupant's body part is not sticking out of the window when the vehicle body signal data indicates that a door is open or that all windows corresponding to the occupant's seating position are closed.

In one possible implementation, the detection module 71 recognizes a relative state between a body part of a passenger and a window in the image data to obtain a first detection result;
and obtaining a second detection result based on the radar sensing data;
The determination module 72 is used to determine a judgment result based on the first detection result and the second detection result.

In one possible implementation, the identification module 72 includes:
When both the first detection result and the second detection result indicate that a body part of the occupant has been stuck out of a window, specifying as the determination result that the body part of the occupant has been stuck out of a window;
When the first detection result indicates that a body part of the passenger has been put out of a window and the second detection result indicates that the body part has not been put out of a window, specifying that the body part of the passenger has not been put out of a window as the determination result;
When the first detection result indicates that a body part has not been put out through the window and the second detection result indicates that a body part has been put out through the window, the determination result is used to at least one of identify that a body part of the occupant has been put out through the window.

In one possible implementation, the detection module 71 comprises:
Recognizing a relative state between a body part of a passenger and a window in the image data to obtain a first detection result;
When the first detection result indicates that a body part has been stuck out of a window, acquiring seat information of the target passenger who has stuck out a body part in the image data through a window based on the first detection result;
Obtaining the radar sensing data of a target window corresponding to seat information of the target passenger;
and determining a second detection result based on the radar sensing data of the target window.

In one possible implementation form, the acquiring of the radar sensing data of the target window corresponding to the seat information of the target occupant includes:
acquiring the radar sensing data of a target window corresponding to the seat information of the target occupant, and vehicle body signal data related to the seat information of the target occupant, including first signal data indicating an open/close state of a target door that is a door corresponding to the seat information of the target occupant, and second signal data indicating an open/close state of the target window;
The step of identifying the second detection result based on the radar sensing data of the target window includes:
The method includes identifying the second detection result based on the radar sensing data and the vehicle body signal data.

In one possible implementation, when the second detection result indicates that the occupant's body part has been projected out of the window, the identification module 72 is used to identify, as the determination result, that the occupant's body part has been projected out of the window.

In one possible implementation mode, the step of recognizing a relative state between a body part of a passenger and a window in the image data and obtaining a first detection result includes:
The method includes recognizing a relative state between a body part of a vehicle occupant and a window based on the image data, and obtaining a degree of confidence that the body part of the vehicle occupant has been pushed out of the window.

In one possible implementation form, when the reliability is greater than a preset threshold, if the second detection result indicates that the body part of the vehicle occupant has been stuck out of a window, the identification module 72 identifies the body part of the vehicle occupant as having been stuck out of a window as the determination result;
If the reliability is smaller than a predetermined threshold, and if the second detection result indicates that a body part of the vehicle occupant has stuck out a window, identifying the body part of the vehicle occupant as having stuck out a window as the determination result;
The reliability is used for at least one of the following purposes: when the reliability is smaller than a predetermined threshold value; when the second detection result indicates that the body part of the vehicle occupant is not sticking out of the window; and when the second detection result indicates that the body part of the vehicle occupant is not sticking out of the window, the judgment result is used for at least one of the following purposes: when the reliability is smaller than a predetermined threshold value; and when the second detection result indicates that the body part of the vehicle occupant is not sticking out of the

In one possible implementation, the vehicle-mounted camera includes a camera installed inside the vehicle cabin and/or a camera installed outside the vehicle cabin.

In one possible implementation, the radar sensor is located on the upper and lower edges of the window.

In one possible implementation, the sensing data further includes infrared sensing data and/or millimeter wave sensing data.

In some embodiments, the functions or modules of the apparatus according to the embodiments of the present disclosure are used to perform the methods described in the above method embodiments. Specific implementations will be apparent from the description of the above method embodiments, and detailed descriptions will be omitted for the sake of brevity.

The embodiment of the present disclosure further includes at least two types of sensors for collecting at least two types of sensing data, including image data collected by an on-board camera and radar sensing data collected by a radar sensor for detecting an obstacle in a window area;
A vehicle is provided that includes a controller that acquires the at least two types of sensing data from the sensor, detects whether or not a body part of an occupant of the vehicle has been exposed through a window based on the at least two types of sensing data, and generates warning information when a body part of the occupant has been exposed through the window.

Here, the controller may be used to implement the method for detecting when a body part of a vehicle occupant has been pushed out of a window, as described in the above embodiment. The sensing data may include other types of sensing data, such as infrared sensing data and/or millimeter-wave sensing data. For exemplary implementations of the controller and sensor, please refer to the above content, and detailed description will be omitted here.

An embodiment of the present disclosure further provides a computer-readable storage medium having computer program instructions stored thereon, the computer-readable storage medium, when executed by a processor, implementing the method for detecting when a body part of a vehicle occupant has been placed out a window, as described above. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

An embodiment of the present disclosure further provides an electronic device including a processor and a memory for storing instructions executable by the processor, the processor configured to invoke the instructions stored in the memory to execute the above-described method for detecting when a body part of a vehicle occupant has been placed out a window.

An embodiment of the present disclosure further provides a computer program product that includes computer-readable code or a computer-readable storage medium having computer-readable code stored thereon, which, when executed in a processor of an electronic device, causes the processor of the electronic device to execute the above-described method for detecting when a body part of a vehicle occupant has been placed out a window.

The electronic device may be provided as a terminal, a server, or other form of device.

FIG. 4 is a block diagram illustrating an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. Referring to FIG. 4, the electronic device 1900 further includes a processing component 1922 including one or more processors, and memory resources, represented by memory 1932, for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. The processing component 1922 is also configured to perform the above-described method by executing the instructions.

The electronic device 1900 may further include a power component 1926 configured to perform power management for the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input/output (I/O) interface 1958. The electronic device 1900 may operate based on an operating system stored in memory 1932, such as the Microsoft Windows Server operating system (Windows Server™), the Apple graphical user interface operating system Mac OS X™, a multitasking, multiuser operating system for computers (Unix™), a free and open source Unix-like operating system (Linux™), an open source Unix-like operating system (FreeBSD™), or the like.

In an exemplary embodiment, a non-volatile computer-readable storage medium, such as memory 1932, is further provided that includes computer program instructions that, when executed by processing component 1922 of electronic device 1900, can cause the above-described method to be performed.

The present disclosure may be a system, method, and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions for causing a processor to implement aspects of the present disclosure.

A computer-readable storage medium may be a tangible device capable of storing and storing instructions for use in an instruction execution device. A computer-readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. Further specific examples (non-exhaustive list) of computer-readable storage media include portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or slot-in protrusion structures on which instructions are stored, and any suitable combination of the above. As used herein, computer-readable storage medium is not to be construed as a transitory signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated via a waveguide or other transmission medium (e.g., pulsed light passing through a fiber optic cable), or electrical signals transmitted via electrical wires.

The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium into each computing/processing device, or may be downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface within each computing/processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within each computing/processing device.

Computer program instructions for carrying out the operations of the present disclosure may be assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or target code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, and general procedural programming languages such as "C" or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. When remote computers are involved, the remote computers may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, the state information of the computer-readable program instructions may be used to personalize electronic circuitry, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), such that the computer-readable program instructions are executed by the electronic circuitry to achieve aspects of the present disclosure.

Although various aspects of the present disclosure have been described herein with reference to flowcharts and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure, it should be understood that each block of the flowcharts and/or block diagrams, and combinations of blocks in the flowcharts and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, and when executed by the processor of the computer or other programmable data processing apparatus, the instructions may produce an apparatus that performs the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams. These computer-readable program instructions may be stored on a computer-readable storage medium to cause a computer, programmable data processing apparatus, and/or other device to operate in a determined manner. Thus, a computer-readable storage medium having instructions stored thereon includes an article of manufacture having instructions that implement each aspect of the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The computer-readable program instructions may be loaded into a computer, other programmable data processing device, or other device and cause the computer, other programmable data processing device, or other device to perform a series of operational steps, thereby creating a computer-implementable process. In this manner, the instructions executed on the computer, other programmable data processing device, or other device implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The flowcharts and block diagrams in the drawings illustrate possible system architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of instructions, which may include one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions depicted in the blocks may be implemented in a different order than that depicted in the drawings. For example, two consecutive blocks may be executed substantially in parallel, or may be executed in a reverse order depending on the functionality. It should also be noted that each block in the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or by a combination of dedicated hardware and computer instructions.

The computer program product may be specifically realized by hardware, software, or a combination thereof. In one possible embodiment, the computer program product may be specifically a computer storage medium. In another possible embodiment, the computer program product may be specifically a software product, such as a software development kit (SDK).

Although the embodiments of the present disclosure have been described above, the above descriptions are merely illustrative and are not exhaustive, and are not limited to the disclosed embodiments. Various modifications and alterations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms chosen in this specification are intended to appropriately interpret the principles, practical applications, or technical improvements of the disclosed embodiments, or to allow others skilled in the art to understand the disclosed embodiments.

Claims (16)

Acquiring vehicle body signal data including door data indicating the open/closed state of a door and window data indicating the open/closed state of a window;
When the vehicle body signal data indicates that the door is closed and the window corresponding to the seating position of the vehicle occupant is open, detecting whether or not a body part of the vehicle occupant has protruded from the window based on at least two types of sensing data including image data collected by an on-board camera and radar sensing data collected by a radar sensor for detecting obstacles in the window area, and obtaining at least two systems of detection results;
determining a determination result based on the at least two detection results;
generating warning information when the determination result indicates that the body part of the occupant has been stuck out of the window. When the vehicle body signal data indicates that the door is closed and the window corresponding to the seating position of the vehicle occupant is open, detecting whether or not a body part of the vehicle occupant has protruded from the window based on each of at least two types of sensing data, and obtaining at least two systems of detection results,
Transmitting a trigger signal to the radar sensor to collect sensing data, and acquiring the radar sensing data collected by the radar sensor;
acquiring image data collected by the on-board camera;
2. The method for detecting whether a body part of a vehicle occupant has protruded out of a window as described in claim 1 , comprising: detecting whether a body part of the occupant has protruded out of a window corresponding to a seating position of the occupant based on the radar sensing data and the image data, respectively, and obtaining a detection result based on the radar sensor and a detection result based on the image data. 2. The method for detecting whether a body part of a vehicle occupant has been exposed through a window as described in claim 1 , further comprising: when the vehicle body signal data indicates that a door is open or that all windows corresponding to the occupant's seating position are closed, determining as the determination result that the occupant's body part has not been exposed through a window. The above-described detection of whether or not a body part of a vehicle occupant has stuck out a window based on each of at least two types of sensing data, and obtaining at least two types of detection results,
Recognizing a relative state between a body part of a passenger and a window in the image data to obtain a first detection result;
obtaining a second detection result based on the radar sensing data;
The step of determining the determination result based on the at least two detection results includes:
determining a determination result based on the first detection result and the second detection result;
2. The method for detecting when a body part of a vehicle occupant has been placed out of a window according to claim 1. The step of determining a determination result based on the first detection result and the second detection result includes:
When both the first detection result and the second detection result indicate that a body part of the occupant has been stuck out of a window, specifying as the determination result that the body part of the occupant has been stuck out of a window;
When the first detection result indicates that a body part of the passenger has been put out of a window and the second detection result indicates that the body part has not been put out of a window, specifying that the body part of the passenger has not been put out of a window as the determination result;
and if the first detection result indicates that the body part has not been stuck out the window and the second detection result indicates that the body part has been stuck out the window, identifying the body part of the occupant as having been stuck out the window as the determination result. The above-described detection of whether or not a body part of a vehicle occupant has stuck out a window based on each of at least two types of sensing data, and obtaining at least two types of detection results,
Recognizing a relative state between a body part of a passenger and a window in the image data to obtain a first detection result;
When the first detection result indicates that a body part has been stuck out of a window, acquiring seat information of the target passenger who has stuck out a body part in the image data through a window based on the first detection result;
Obtaining the radar sensing data of a target window corresponding to seat information of the target passenger;
The method for detecting when a body part of a vehicle occupant has gone out of a window according to claim 1 , further comprising: identifying a second detection result based on radar sensing data of the target window. The acquiring of the radar sensing data of the target window corresponding to the seat information of the target passenger includes:
acquiring the radar sensing data of a target window corresponding to the seat information of the target occupant, and vehicle body signal data related to the seat information of the target occupant, including first signal data indicating an open/close state of a target door that is a door corresponding to the seat information of the target occupant, and second signal data indicating an open/close state of the target window;
The step of identifying the second detection result based on the radar sensing data of the target window includes:
The method for detecting whether a body part of a vehicle occupant has stuck out a window according to claim 6 , further comprising: determining the second detection result based on the radar sensing data and the vehicle body signal data. The step of determining the determination result based on the at least two detection results includes:
7. The method for detecting that a body part of a vehicle occupant has been stuck out of a window, as described in claim 6 , further comprising, when the second detection result indicates that the body part of the occupant has been stuck out of the window, identifying the body part of the occupant as having been stuck out of the window as the determination result. The above-mentioned recognizing a relative state between a body part of a passenger and a window in the image data and obtaining a first detection result includes:
5. The method for detecting whether a body part of a vehicle occupant has gone out of a window as described in claim 4 , further comprising recognizing a relative state between the body part of the vehicle occupant and the window based on the image data, and obtaining a confidence level that the body part of the vehicle occupant has gone out of the window. The step of determining the determination result based on the at least two detection results includes:
If the reliability is greater than a predetermined threshold, and if the second detection result indicates that a body part of the vehicle occupant has stuck out a window, identifying the body part of the vehicle occupant as having stuck out a window as the determination result;
If the reliability is smaller than a predetermined threshold, and if the second detection result indicates that a body part of the vehicle occupant has stuck out a window, identifying the body part of the vehicle occupant as having stuck out a window as the determination result;
10. The method for detecting whether a body part of a vehicle occupant has been stuck out a window as described in claim 9 , comprising at least one of: when the reliability is smaller than a predetermined threshold value; and when the second detection result indicates that the body part of the vehicle occupant has not been stuck out a window, identifying as the determination result that the body part of the vehicle occupant has not been stuck out a window. The vehicle-mounted camera includes a camera installed inside the vehicle cabin and/or a camera installed outside the vehicle cabin, and/or
the radar sensor is located at the top and/or bottom edge of the window; and/or
The method for detecting whether a body part of a vehicle occupant has gone out of a window according to claim 1 , wherein the sensing data further includes infrared sensing data and/or millimeter wave sensing data. Acquiring vehicle body signal data including door data indicating the open/closed state of a door and window data indicating the open/closed state of a window;
a detection module for detecting whether a body part of a vehicle occupant has protruded from a window based on at least two types of sensing data, including image data collected by an on-board camera and radar sensing data collected by a radar sensor for detecting an obstacle in the window area, when the vehicle body signal data indicates that a door is closed and a window corresponding to a seating position of the occupant is open, and for obtaining at least two detection results;
an identification module for identifying a determination result based on the at least two detection results;
and an alert module for generating alert information when the determination result indicates that the occupant's body part has been stuck out of the window. Acquiring vehicle body signal data including door data indicating the open/closed state of a door and window data indicating the open/closed state of a window;
at least two types of sensors for collecting at least two types of sensing data, including image data collected by an on-board camera and radar sensing data collected by a radar sensor for detecting an obstacle in a window area, when the vehicle body signal data indicates that a door is closed and a window corresponding to a seating position of a passenger is open ;
a controller that acquires the at least two types of sensing data from the sensor, detects whether a body part of a vehicle occupant has stuck out a window based on the at least two types of sensing data, and generates warning information when a body part of the occupant has stuck out the window. a processor;
a memory for storing instructions executable by the processor;
The processor is configured to call instructions stored in the memory to execute the method for detecting when a body part of a vehicle occupant has been placed out of a window as described in any one of claims 1 to 11 . A computer-readable storage medium having stored thereon computer program instructions, the computer-readable storage medium causing, when executed by a processor, the computer-readable storage medium to implement the method for detecting when a body part of a vehicle occupant has been placed out of a window as set forth in any one of claims 1 to 11 . A computer program comprising computer readable code, the computer readable code running on a processor of an electronic device causing the processor of the electronic device to execute the method for detecting when a body part of a vehicle occupant has been put out of a window according to any one of claims 1 to 11 .