CN116543369A - Method and device for identifying drivable area, electronic equipment and storage medium - Google Patents

Abstract

The present application relates to the field of intelligent driving technologies, and in particular, to a method, an apparatus, an electronic device, and a storage medium for identifying a drivable region, where a current target road image to be identified is obtained in response to a drivable region identification request triggered by a driving object; respectively extracting features of the target road image according to the preset scales of each candidate image to obtain a corresponding road feature map; for each road feature map, the following operations are performed: based on the respective image position information of each pixel point contained in one road feature map, carrying out attention weighting processing on the road feature map by combining the image channel information contained in one road feature map to obtain a corresponding attention feature map; and adjusting the image scale of each obtained attention feature image based on the image scale of the target road image to obtain at least one target feature image, and obtaining the drivable area of the target vehicle based on the at least one target feature image. In this way, the accuracy of the identification of the drivable area is improved.

Description

A drivable area identification method, device, electronic equipment and storage medium

technical field

The present application relates to the technical field of intelligent driving, and in particular to a drivable area identification method, device, electronic equipment and storage medium.

Background technique

At present, the detection of scene information during vehicle driving is the key to assisted driving systems and automatic driving systems. The detection of scene information includes lane lines, traffic signs, road signs, and drivable areas. The system and the automatic driving system play an important role. During the driving process of the vehicle, it provides the vehicle with a drivable route and avoids the undrivable area. The undrivable area mainly includes static obstacles on the road, such as cones, guardrails, etc. area, as well as the area where dynamic obstacles, such as vehicles and pedestrians, are located.

In related technologies, when identifying a drivable area, a complex encoder is usually used to extract features from the collected pictures to obtain a quadruple downsampled feature map. For example, the high-resolution network (High-Resolution Net, HRNet), and then enlarge the four times downsampled feature map, so that the output image can be restored to the size of the input image, and then the drivable area can be obtained.

However, when the above method is adopted, the computational efficiency of the graphics processing unit (GPU) is low due to the high memory access cost of the complex encoder, which limits the recognition speed.

In addition, in order to improve the recognition speed, the size of the image is directly restored by zooming in. Since the pixels of the quadruple downsampling feature map are small, the object cannot observe the pixels clearly. Therefore, if the quadruple downsampling feature map is directly enlarged , resulting in inaccurate recognition.

For example, when the quadruple downsampling feature map has several pixels in the drivable area that are not recognized as drivable areas, after the zoom-in process, the pixels will become larger, and the object can observe that there is a small area that is not recognized as a drivable area. Identified as a drivable area, resulting in a hole problem.

Therefore, the result accuracy and recognition efficiency of drivable area recognition in the related art need to be improved.

Contents of the invention

Embodiments of the present application provide a drivable area identification method, device, electronic device, and storage medium, so as to improve the result accuracy and identification efficiency of drivable area identification.

The specific technical scheme that the embodiment of the present application provides is as follows:

In the first aspect, a method for identifying a drivable area is provided, including:

Responding to a drivable area identification request triggered by a driving object, acquiring a current target road image to be identified, where the target road image has a target image scale;

According to the preset scales of each candidate image, feature extraction is performed on the target road image to obtain the corresponding road feature map;

For each road feature map, perform the following operations: Based on the image position information of each pixel contained in a road feature map, combined with the image channel information contained in a road feature map, perform attention weighting processing on the road feature map to obtain the corresponding The attention feature map of

Based on the target image scale, image scale adjustment is performed on each of the obtained attention feature maps to obtain at least one target feature map, and based on the at least one target feature map, a drivable area of the target vehicle is obtained.

In a second aspect, a drivable area identification device is provided, including:

An acquisition module, configured to acquire a current target road image to be identified in response to a drivable area identification request triggered by a driving object, where the target road image has a target image scale;

The extraction module is used to extract the features of the target road image according to the preset scales of each candidate image, and obtain the corresponding road feature map;

The first processing module is configured to perform the following operations on each road feature map: based on the image position information of each pixel contained in a road feature map, combined with the image channel information contained in a road feature map, the road feature map is processed Attention weighting processing to obtain the corresponding attention feature map;

The second processing module is configured to perform image scale adjustment on each of the obtained attention feature maps based on the target image scale, obtain at least one target feature map, and obtain the drivable area of the target vehicle based on the at least one target feature map.

Optionally, according to the preset scales of each candidate image, feature extraction is performed on the target road image respectively, and when the corresponding road feature map is obtained, the extraction module is used for:

For each preset candidate image scale, perform the following operations:

Based on a candidate image scale, performing first feature extraction on the target road image to obtain a first feature map, the first feature map has a first number of image channels;

According to the preset receptive field of each candidate image, the second feature extraction is performed on the first feature map to obtain the corresponding second feature map, and the obtained second feature maps are superimposed to obtain the superimposed feature map; wherein, each A representation of the receptive field of a candidate image: each pixel contained in the corresponding second feature map is mapped to the area size on the target road image;

Based on the number of first image channels, image channel adjustment is performed on the superposition feature map to obtain a road feature map corresponding to a candidate image scale.

Optionally, based on the respective image position information of each pixel contained in a road feature map, combined with the image channel information contained in a road feature map, the road feature map is subjected to attention weighting processing, and when the corresponding attention feature map is obtained, The first processing module is used for:

Based on the respective image position information of each pixel contained in a road feature map, determine the respective position attention weights of each pixel contained in a road feature map, where the position attention weight represents the corresponding pixel points when identifying the drivable area The importance of the image position;

Based on the image channel information contained in a road feature map, determine the respective channel attention weights of each image channel contained in a road feature map, wherein the channel attention weight represents the importance of the corresponding image channel when identifying the drivable area;

Based on the obtained attention weights of each position and channel attention weights, a road feature map is weighted to obtain an attention feature map corresponding to a road feature map.

Optionally, based on the target image scale, image scale adjustment is performed on each of the obtained attention feature maps to obtain at least one target feature map, and the second processing module is used for:

Based on the respective image scales of the obtained attention feature maps, fusion and update are performed on each attention feature map to obtain at least one fusion feature map;

Based on the preset number of target image channels, image channel adjustment is performed on at least one fusion feature map to obtain a corresponding intermediate feature map;

Based on the target image scale, image scale adjustment is performed on at least one obtained intermediate feature map to obtain a corresponding target feature map.

Optionally, based on the respective image scales of the obtained attention feature maps, the attention feature maps are fused and updated, and when at least one fusion feature map is obtained, the second processing module is also used for:

According to the respective image scales of the obtained attention feature maps, sort the attention feature maps to obtain the sorting results;

Based on the sorting results, each two adjacent attention feature maps are read sequentially for fusion until the reading is completed. A fusion process includes:

Read two adjacent attention feature maps, and fuse the two attention feature maps to obtain a fusion feature map;

Save the fusion feature map, and use the fusion feature map as a new attention feature map to replace the two attention feature maps and add it to the sorting result.

Optionally, when obtaining the drivable area of the target vehicle based on at least one target feature map, the second processing module is also used for:

If at least one target feature map includes a target feature map, then using a target feature map as a result feature map;

If at least one target feature map includes multiple target feature maps, performing image fusion on the multiple target feature maps to obtain a resultant feature map;

Comparing the value of each pixel contained in the result feature map with the preset drivable area label value, and determining each target pixel point belonging to the drivable area in the result feature map;

In the target road image, each pixel point corresponding to the respective image position of each target pixel point is marked to obtain an image including a drivable area.

Optionally, the drivable area is obtained by inputting the target road image into the target recognition model, and the device also includes a training module, which is used for:

Perform iterative training on the recognition model to be trained based on the training sample set to obtain the target recognition model; each training sample includes: image data of the sample road, wherein each iteration process performs the following operations:

According to the preset image scales of each sample, feature extraction is performed on the selected training samples to obtain corresponding sample feature maps, and the training samples have sample image scales;

For each sample feature map, perform the following operations respectively: Based on the respective sample image position information of each pixel contained in a sample feature map, combined with the sample image channel information contained in a sample feature map, perform attention weighting processing on the sample feature map, Obtain the corresponding sample attention feature map;

Based on the sample image scale, image scale adjustment is performed on the obtained attention feature maps of each sample to obtain at least one target sample feature map;

Based on at least one target sample feature map, the drivable area prediction result of the training sample is obtained, and the parameters are adjusted based on the loss value corresponding to the drivable area prediction result.

In a third aspect, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and operable on the processor, and the processor implements any one of the above-mentioned first aspects when executing the program. method steps.

In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any one of the methods described in the above-mentioned first aspect are implemented.

In the embodiment of the present application, when the driving object triggers a drivable area identification request, the server responds to the drivable area identification request triggered by the driving object, obtains the current target road image to be identified, and then according to the preset candidate image scales, respectively Feature extraction is performed on the target road image to obtain the corresponding road feature map. Then, the attention weighting process is performed on each road feature map in the spatial position dimension and the channel dimension to obtain each attention feature map. Finally, based on the target road image The image scale is to adjust the image scale of each obtained attention feature map to obtain at least one target feature map, and obtain the drivable area of the target vehicle based on the at least one target feature map.

In this way, obtaining multi-scale road feature maps, weighting each road feature map in the spatial position dimension and channel dimension, and performing fusion operations on the weighted road feature maps can better balance multi-scale road features. The spatial information and semantic information of the map avoid the hole problem, improve the accuracy and recognition efficiency of the drivable area recognition result, and the image scale of the target feature map and the target road image are consistent, which can obtain the drivable area of the target vehicle more accurately. driving area.

Description of drawings

FIG. 1 is a schematic diagram of a possible application scenario in the embodiment of the present application;

FIG. 2 is a schematic flow diagram of one round of training of the target recognition model in the embodiment of the present application;

FIG. 3 is a schematic diagram of an encoder in an embodiment of the present application;

Fig. 4 is the schematic diagram of high-performance GPU module layer in the embodiment of the present application;

5 is a schematic diagram of a spatial attention module and a channel attention module in an embodiment of the present application;

FIG. 6 is a schematic flow chart of a drivable area identification method in an embodiment of the present application;

FIG. 7 is a schematic flow diagram of obtaining a road feature map in an embodiment of the present application;

FIG. 8 is a schematic diagram of obtaining a superposition feature map in an embodiment of the present application;

FIG. 9 is a schematic flow diagram of obtaining an attention feature map corresponding to a road feature map in an embodiment of the present application;

FIG. 10 is a schematic diagram of obtaining an attention feature map in an embodiment of the present application;

FIG. 11 is a schematic flow diagram of obtaining at least one target feature map according to an embodiment of the present application;

FIG. 12 is a schematic flow diagram of obtaining at least one fusion feature map according to an embodiment of the present application;

Fig. 13 is a schematic diagram of obtaining at least one fusion feature map in the embodiment of the present application;

FIG. 14 is a schematic diagram of obtaining at least one target feature map in an embodiment of the present application;

FIG. 15 is a schematic flow diagram of obtaining the drivable area of a target vehicle according to an embodiment of the present application;

Fig. 16 is a schematic diagram of an image including a drivable area in the embodiment of the present application;

Fig. 17 is a schematic structural diagram of a drivable area recognition device in an embodiment of the present application;

FIG. 18 is a schematic structural diagram of an electronic device in an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

In order to facilitate the understanding of the technical solutions provided by the embodiments of the present application, some key terms used in the embodiments of the present application are first explained here:

Drivable area: Regardless of traffic rules, all road areas that allow vehicles to drive normally except for all obstacles (including guardrails, vehicles, pedestrians and other obstacles) on the road where the vehicle is driving.

Loss function: A function used in machine learning to measure and optimize the distance between the predicted value of the model and the labeled value of the real sample.

The design idea of the embodiment of the present application is briefly introduced below:

At present, the detection of scene information during vehicle driving is the key to assisted driving systems and automatic driving systems. The detection of scene information includes lane lines, traffic signs, road signs, and drivable areas. The system and the automatic driving system play an important role. During the driving process of the vehicle, it provides the vehicle with a drivable route and avoids the undrivable area. The undrivable area mainly includes static obstacles on the road, such as cones, guardrails, etc. area, as well as the area where dynamic obstacles, such as vehicles and pedestrians, are located.

Currently commonly used drivable area identification methods include:

Method 1: Artificially design features to obtain the drivable area, such as: traditional threshold method, clustering method, support vector machine, decision tree and other methods.

When using method 1, due to the need for artificial design features, limited by professional theories and prior knowledge, it is difficult to represent the variability of complex traffic environments and the diversity of road structures, and it can only be applied in limited specific environments, which has limitations. And it is inefficient.

Method 2: For the collected pictures, use a complex encoder to extract features and obtain a quadruple downsampled feature map. For example, the high-resolution network (High-Resolution Net, HRNet), and then enlarge the four times downsampled feature map, so that the output image can be restored to the size of the input image, and then the drivable area can be obtained.

When the second method is adopted, since the memory access cost of the complex encoder is high, the calculation efficiency of the graphics processing unit (GPU) is low, which limits the recognition speed.

In addition, in order to improve the recognition speed, the size of the image is directly restored by zooming in. Since the pixels of the quadruple downsampling feature map are small, the object cannot observe the pixels clearly. Therefore, if the quadruple downsampling feature map is directly enlarged , resulting in inaccurate recognition.

For example, when the quadruple downsampling feature map has several pixels in the drivable area that are not recognized as drivable areas, after the zoom-in process, the pixels will become larger, and the object can observe that there is a small area that is not recognized as a drivable area. Identified as a drivable area, resulting in a hole problem.

In view of this, in the embodiment of the present application, a drivable area recognition method, device, electronic device and storage medium are proposed. When the driving object triggers a drivable area recognition request, the current target road image to be recognized can be obtained, and then according to the preset candidate image scales, feature extraction is performed on the target road image to obtain the corresponding road feature map, and then for each The road feature map performs the following operations respectively: Based on the image position information of each pixel contained in a road feature map, combined with the image channel information contained in a road feature map, the road feature map is weighted by attention to obtain the corresponding attention The force feature map, and finally based on the target image scale, image scale adjustment is performed on each of the obtained attention feature maps to obtain at least one target feature map, and based on the at least one target feature map, the drivable area of the target vehicle is obtained. In this way, the weighted processing and fusion operations are performed on road feature maps of different scales to keep the size of the target feature map and the target road image consistent, which can better balance the spatial information and semantic information of the multi-scale road feature map, and avoid the hole problem. The result accuracy and recognition efficiency of drivable area recognition are improved.

The preferred embodiments of the application will be described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described here are only used to illustrate and explain the application, and are not used to limit the application, and in the absence of conflict, the application The embodiments and the features in the embodiments can be combined with each other.

Referring to FIG. 1 , it is a schematic diagram of a possible application scenario in the embodiment of the present application. In the application scenario diagram, a server 110 and terminal devices 120 (including terminal device 1201, terminal device 1202...terminal device 120n) are included.

The server 110 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, Cloud servers for basic cloud computing services such as middleware services, domain name services, security services, content delivery network (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The terminal device 120 and the server 110 may be connected directly or indirectly through wired or wireless communication, which is not limited in this application.

The terminal device 120 may be a mobile phone, a portable computer, etc. carried by the driving object during the driving of the target vehicle, and the terminal device may also be a computer device with certain computing capabilities such as a vehicle-mounted terminal installed in the target vehicle.

In the embodiment of the present application, any one or all of the server 110 and the terminal device 120 can be configured with a trained target recognition model, so that the trained target recognition model can be used to identify the drivable area of the target road image to be recognized .

It should be noted that, in the embodiment of the present application, the recognition model adopted by the device may be obtained by its own training, or may be obtained by training of other devices.

Taking the server 110 using the trained recognition model to perform drivable area recognition operation on the target road image to be recognized as an example, the recognition model used by the server 110 can be obtained by its own training, or it can be obtained after other devices complete the training of the recognition model. , sent directly to the server 110.

In the following description, the training of the object recognition model implemented by the server will be taken as an example to describe the related training process in detail.

In addition, in the embodiment of the present application, according to actual processing needs, the server training to obtain the target recognition model may be a periodic process, and the training samples may be periodically regenerated to train the target recognition model.

Based on the training sample set, the recognition model to be trained is iteratively trained to obtain the target recognition model; each training sample includes: the image data of the sample road, as shown in Figure 2, which is a round of training of the target recognition model in the embodiment of the present application Schematic diagram of the process, including:

Step 20: Perform feature extraction on the selected training samples according to the preset image scales of each sample, and obtain corresponding sample feature maps.

where the training samples have a sample image scale.

In the embodiment of the present application, an encoder is used to perform feature extraction on training samples, and according to preset image scales of each sample, feature extraction is performed on selected training samples to obtain corresponding sample feature maps.

Wherein, the preset scales of each sample image may be: 1/2, 1/4, 1/8, 1/16, which is not limited in this embodiment of the present application.

For example, refer to Figure 3, which is a schematic diagram of the encoder in the embodiment of the present application. The encoder is mainly composed of a network stem layer and four stage layers. The network stem layer consists of a 3*3 convolutional layer and a batch normalization layer. Composed with FReLU layer, the first sample feature map with image size of 1/2 is obtained. The stage layer is composed of high-performance GPU module (HG block) layer and downsampling (LDS Layer) layer. The downsampling layer adopts a step size of 2's 2*2 maximum pooling layer to obtain the first sample feature maps with image sizes of 1/4, 1/8 and 1/16 respectively, based on each first sample feature map, using the HG block layer to obtain Sample feature maps with image sizes of 1/2, 1/4, 1/8, 1/16, respectively.

Specifically, in the HG Block layer, for the first sample feature map corresponding to each sample image scale, the following operations are respectively performed: according to the preset receptive field of each sample image, the first sample corresponding to a sample image scale The second feature extraction is performed on the feature map to obtain the corresponding second sample feature map, and the obtained second sample feature maps are superimposed to obtain the superimposed sample feature map. Based on the number of image channels of the first sample feature map, the superimposed The feature map performs image channel adjustment to obtain the sample feature map corresponding to the sample image scale.

Among them, the representation of the receptive field of each sample image: each pixel contained in the corresponding second sample feature map is mapped to the size of the area on the training sample, and the receptive field of each sample image can be: 1x1, 3x 3, 5x 5, 7x 7, 9x 9. 11 x 11, which is not limited in the embodiment of the present application.

For example, refer to FIG. 4, which is a schematic diagram of the high-performance GPU module layer in the embodiment of the present application. The first sample feature map first passes through a series of 3*3 convolutional layers to obtain the second sample feature map of different image receptive fields. : The second sample feature map with a receptive field of 3x3, the second sample feature map with a receptive field of 5x5, the second sample feature map with a receptive field of 7x7, the second sample feature map with a receptive field of 9x9, and the receptive The field is the second sample feature map of 11x11, and superimpose the obtained second sample feature maps to obtain the superimposed sample feature map, and use 1*1 convolution to adjust the image channel of the superimposed sample feature map to obtain the sample feature map , so that the number of image channels of the sample feature map is the same as that of the first sample feature map.

In this way, since the impact of memory access cost on computational efficiency is much greater than the amount of computation and parameters of the model, especially for convolutional layers that are heavily used in the model, only when the number of input and output image channels is equal, the memory access cost is the smallest. Therefore, adopting a high-performance GPU module layer and adopting the design with the same number of input and output image channels can minimize the cost of memory access, improve computing efficiency, and improve the multi-feature extraction capability of the network through layered convolution overlap.

Optionally, add residual connection and squeeze excitation to the high-performance GPU module layer included in the last stage layer, as shown in Figure 4, the dotted line part is the residual connection, in this way, since the residual connection speeds up the model convergence, Excellent performance in avoiding gradient disappearance, etc., but the element-wise addition operation brought by the residual connection is not conducive to GPU device calculation and will affect the speed of the model. Squeeze excitation is a channel attention mechanism, which can effectively improve the accuracy of the model, but it will also bring a large delay. To balance the accuracy and speed of the model, the encoder only uses residual connections and squeeze excitations in the last stage layer.

Step 21: For each sample feature map, perform the following operations: Based on the sample image position information of each pixel contained in a sample feature map, combined with the sample image channel information contained in a sample feature map, pay attention to the sample feature map Weighted processing to obtain the corresponding sample attention feature map.

In the embodiment of the present application, for each sample feature map, the following operations are respectively performed: using a spatial attention module, based on the respective sample image position information of each pixel contained in a sample feature map, determine each pixel contained in the sample feature map Attention weights of respective sample positions, using the channel attention module, based on the sample image channel information contained in the sample feature map, determine the respective sample channel attention weights of each image channel contained in the sample feature map, based on the obtained sample position Attention weight and sample channel attention weight weight the sample feature map to obtain the sample attention feature map corresponding to the sample feature map.

Among them, the sample position attention weight represents the importance of the sample image position of the corresponding pixel when identifying the drivable area, and the sample channel attention weight represents the importance of the corresponding sample image channel when identifying the drivable area.

Specifically, refer to FIG. 5, which is a schematic diagram of the spatial attention module and the channel attention module in the embodiment of the present application. The spatial attention module performs mean pooling (Mean Pooling) and maximum The pooling (Max Pooling) operation obtains the spatial average pooling sample feature map and the spatial maximum pooling sample feature map, and then stitches the spatial average pooling sample feature map and the spatial maximum pooling sample feature map to obtain the spatial pooling splicing The sample feature map is finally activated by the convolutional layer and the sigmoid function to obtain the respective sample position attention weights of each sample pixel contained in the sample feature map; the channel attention module performs Mean Pooling and Max on the sample feature map in the channel dimension The Pooling operation obtains the channel average pooling sample feature map and the channel maximum pooling sample feature map, and then stitches the channel average pooling sample feature map and the channel maximum pooling sample feature map to obtain the channel pooling splicing sample feature map, and finally Through the activation of the convolutional layer and the sigmoid function, the respective sample channel attention weights of the sample image channels included in the sample feature map are obtained.

In this way, using the spatial attention module and the channel attention module to obtain the attention feature map of each sample can make the recognition model pay more attention to the important information in the sample feature map, suppress useless information, and improve the accuracy of the recognition model.

Step 22: Based on the sample image scale, perform image scale adjustment on the obtained attention feature maps of each sample to obtain at least one target sample feature map.

In the embodiment of the present application, based on the respective image scales of the acquired attention feature maps of each sample, the attention feature maps of each sample are fused and updated to obtain at least one sample fusion feature map. Based on the preset number of sample image channels, respectively Image channel adjustment is performed on at least one sample fusion feature map to obtain a corresponding sample intermediate feature map, and based on the sample image scale, image scale adjustment is performed on the obtained at least one sample intermediate feature map to obtain a corresponding sample target feature map.

For example, using a convolutional network, based on the preset number of sample image channels, adjust the image channel of at least one sample fusion feature map to obtain the corresponding sample intermediate feature map, using bilinear interpolation, based on the sample image scale, to obtain Image scaling is performed on at least one sample intermediate feature map to obtain the corresponding sample target feature map.

In this way, at least one sample target feature map with the same sample image channel number and sample image scale is obtained, and the differences in channel and scale features of different sample attention feature maps are integrated to improve the accuracy of the target recognition model.

Specifically, there are many ways to fuse and update the attention feature maps of each sample.

The first fusion update method: sort the attention feature maps of each sample according to the respective image scales of the obtained attention feature maps of each sample, obtain the sample sorting results, and read every two phases in sequence based on the sample sorting results. Neighboring sample attention feature maps are fused until the reading is completed. A fusion process includes: reading two adjacent sample attention feature maps, and fusing the two sample attention feature maps to obtain Sample fusion feature map, save the sample fusion feature map, and use the sample fusion feature map as a new sample attention feature map, replace the two sample attention feature maps, and add it to the sample sorting result.

For example, suppose each sample attention feature map includes sample attention feature map A1, sample attention feature map A2, sample attention feature map A3, sample attention feature map A4, and the image scale of sample attention feature map A1 is 1 /2. The image scale of the sample attention feature map A2 is 1/4, the image scale of the sample attention feature map A3 is 1/8, and the image scale of the sample attention feature map A4 is 1/16, then the The attention feature maps of each sample are sorted according to the image scale from small to large, and the obtained sample sorting results are A1, A2, A3, A4, read A1 and A2, and fuse A1 and A2 to obtain the sample fusion feature map B1, Save B1, and use B1 as a new sample attention feature map, replace A1 and A2, and add it to the sample sorting result. At this time, the sample sorting result is B1, A3, A4, read B1 and A3, and perform B1 and A3 Fusion, obtain the sample fusion feature map B2, save B2, and use B2 as a new sample attention feature map, replace B1 and A3, and add it to the sample sorting result. At this time, the sample sorting result is B2, A4, read B2 and A4 , and fuse B2 and A4 to obtain a sample fusion feature map B3, save B3, and finally obtain at least one sample fusion feature map including B1, B2 and B3.

Further, before the fusion update between the sample attention feature maps corresponding to different sample image scales, the sample feature map corresponding to the small sample image scale is up-sampled to the same sample image scale as the sample feature map corresponding to the large sample image scale, Then through the spatial attention module and the channel attention module, the sample attention feature map corresponding to the small sample image scale is obtained.

The second fusion update method: sort the attention feature maps of each sample according to the respective image scales of the obtained attention feature maps of each sample, obtain the sample sorting results, and read every two phases in sequence based on the sample sorting results. Neighboring sample attention feature maps are fused until the reading is completed. A fusion process includes: reading two adjacent sample attention feature maps, and fusing the two sample attention feature maps to obtain Sample fusion feature map, save the sample fusion feature map.

For example, suppose each sample attention feature map includes sample attention feature map A1, sample attention feature map A2, sample attention feature map A3, sample attention feature map A4, and the image scale of sample attention feature map A1 is 1 /2. The image scale of the sample attention feature map A2 is 1/4, the image scale of the sample attention feature map A3 is 1/8, and the image scale of the sample attention feature map A4 is 1/16, then the The attention feature map of each sample is sorted according to the image scale from small to large, and the obtained sample sorting results are A1, A2, A3, A4, read A1 and A2, and fuse A1 and A2 to obtain the sample fusion feature map B4, Save B4, read A2 and A3, and fuse A2 and A3, obtain the sample fusion feature map B5, save B5, read A3 and A4, and fuse A3 and A4, obtain the sample fusion feature map B6, save B6 , the finally obtained at least one sample fusion feature map includes B4, B5 and B6.

The third fusion update method: fuse the attention feature maps of each sample to obtain a sample fusion feature map.

For example, assuming that each sample attention feature map includes sample attention feature map A1, sample attention feature map A2, sample attention feature map A3, and sample attention feature map A4, then A1, A2, A3, and A4 are fused, Obtain a sample fusion feature map B7.

Step 23: Obtain the drivable area prediction result of the training sample based on at least one target sample feature map, and perform parameter adjustment based on the loss value corresponding to the drivable area prediction result.

In the embodiment of the present application, the sample result feature map is obtained based on at least one target sample feature map, and the value of each sample pixel contained in the sample result feature map is compared with the preset sample drivable area label value to determine the sample result Each sample target pixel point belonging to the drivable area in the feature map is marked with each sample pixel point corresponding to the respective image position of each sample target pixel point in the training sample to obtain the drivable area prediction result of the training sample, and based on The prediction result of the drivable area, the real label corresponding to the training sample and the weight of the difficult sample, calculate the loss value, and adjust the network parameters of the recognition model based on the loss value.

In the embodiment of this application, the sample result feature map is obtained, including but not limited to the following two situations:

Case 1: At least one target sample feature map includes only one target sample feature map, and one target sample feature map is used as the sample result feature map.

Case 2: At least one target sample feature map includes multiple target sample feature maps, image fusion is performed on the multiple target sample feature maps to obtain a sample result feature map.

Among them, the preset sample drivable area label value can be 1, which is not limited in the embodiment of the present application. The objective function of model training is loss=0.9*L _FocalLoss +0.1*L _LovaszLoss , and L _FocalLoss is based on the identified The difficulty level applies the corresponding weight loss, that is, adding a smaller weight to the easy-to-recognize samples and adding a larger weight to the difficult-to-recognize samples. L _LovaszLoss is a convex Lovasz extension based on the sub-module loss, which is uniform for semantic segmentation tasks. And optimize (meanintersection over union, Mean IoU).

For example, use the sigmoid function to activate the sample result feature map, compare the value of each sample pixel contained in the sample result feature map with the preset sample drivable area label value 1, and determine that the sample result feature map belongs to the drivable area. For each sample target pixel point in the area, each sample pixel point corresponding to the image position of each sample target pixel point in the training sample is marked, and the drivable area prediction result of the training sample is obtained.

The following describes the application process based on the trained target recognition model with reference to the accompanying drawings:

Refer to Figure 6, which is a schematic flow chart of the drivable area identification method in the embodiment of the present application. The specific operations will be described in detail below in conjunction with Figure 6:

Step 60: Responding to the drivable area identification request triggered by the driving object, acquire the current target road image to be identified.

Wherein, the target road image has a target image scale.

In this embodiment of the application, after the driving object performs certain operations on the terminal device, the terminal device can be triggered to send a drivable area identification request to the server, and the server calls the shooting device in response to the drivable area identification request triggered by the driving object. Obtain the current target road image to be recognized.

Step 61: According to the preset scales of each candidate image, feature extraction is performed on the target road image to obtain a corresponding road feature map.

In the embodiment of the present application, after the target road image is obtained, feature extraction is performed on the target road image according to the preset candidate image scales, and road feature maps corresponding to each candidate image scale are obtained.

The preset candidate image scales may be: 1/2, 1/4, 1/8, 1/16, which is not limited in this embodiment of the present application.

Specifically, when obtaining a road feature map corresponding to a candidate image scale, the server specifically performs the following operations. Referring to Fig. 7, it is a schematic flow chart of obtaining a road feature map in the embodiment of the present application. In conjunction with Fig. 7, the specific operations are described in detail below:

Step 610: Based on a candidate image scale, perform first feature extraction on the target road image to obtain a first feature map.

Wherein, the first feature map has a first number of image channels.

In the embodiment of the present application, based on a candidate image scale, a corresponding downsampling operation is used to perform first feature extraction on the target road image to obtain a first feature map, and the image scale of the first feature map is the candidate image scale.

For example, assuming that the scale of a candidate image is 1/2, the first feature extraction is performed on the target road image to obtain the first feature map with an image scale of 1/2.

Step 611: Perform second feature extraction on the first feature map according to the preset receptive field of each candidate image to obtain a corresponding second feature map, and superimpose the obtained second feature maps to obtain a superimposed feature map.

Among them, the receptive field of each candidate image is characterized by: the size of the area where each pixel contained in the corresponding second feature map is mapped to the target road image, and the receptive field of each candidate image can be: 1x1, 3x 3, 5x 5, 7x 7, 9x 9. 11 x 11, which is not limited in the embodiment of the present application.

In the embodiment of the present application, after the first feature map is obtained, the following operations are respectively performed for each preset candidate image receptive field: Based on a candidate image receptive field, the second feature extraction is performed on the first feature map to obtain the corresponding first Two feature maps. and superimposing the obtained second feature maps to obtain superimposed feature maps.

For example, refer to Figure 8, which is a schematic diagram of obtaining a superimposed feature map in the embodiment of the present application, assuming that the receptive fields of each candidate image are: 1x1, 3x 3, 5x 5, 7x 7, 9x 9, 11x 11, using a series of 3 *3 convolutional layers, to obtain the second feature map of different image receptive fields, including: the second feature map with a receptive field of 3x 3, the second feature map with a receptive field of 5x 5, and the second feature map with a receptive field of 7x 7 , a second feature map with a receptive field of 9x9, and a second feature map with a receptive field of 11x11, and superimpose the obtained second sample feature maps to obtain a superimposed feature map.

Step 612: Based on the number of first image channels, perform image channel adjustment on the superimposed feature map to obtain a road feature map corresponding to a candidate image scale.

In the embodiment of the present application, after obtaining the superimposed feature map, based on the first image channel number, 1*1 convolution is used to adjust the image channel of the superimposed feature map to obtain the road feature map, and the image channel number of the road feature map is the first image number of channels.

In this way, since the impact of memory access cost on computational efficiency is much greater than the amount of computation and parameters of the model, especially for convolution operations, only when the number of input and output image channels is equal, the memory access cost is the smallest. Therefore, the road feature The design with the same number of image channels in the first feature map and the first feature map can minimize the memory access cost, and while improving the computational efficiency, multiple features of the target road image can be obtained through layered convolution and superposition.

Optionally, based on the number of first image channels, image channel adjustment is performed on the superimposed feature map corresponding to the first feature map with the smallest image scale, and when obtaining the road feature map corresponding to the smallest candidate image scale, add residual connection and squeeze excitation , as shown in Figure 8, the dotted line part is the residual connection, in this way, because the residual connection is excellent in avoiding gradient disappearance, etc., but the element-wise addition operation brought by the residual connection is not conducive to GPU equipment calculation, which affects the speed of drivable area recognition. Squeeze incentive is a channel attention mechanism, which can effectively improve the accuracy of drivable area recognition, but it will also bring a large delay. In order to balance the accuracy and speed of drivable area recognition, residual connection and squeeze excitation are only used when obtaining the road feature map corresponding to the smallest candidate image scale.

In addition, it is worth noting that in the embodiment of the present application, the first feature map with an image scale of 1/2 can be obtained through the 3*3 convolution layer, batch normalization layer and FReLU layer, and the image scale is 1/2 of the road feature map corresponding to the first feature map, and then use a 2*2 maximum pooling layer with a step size of 2 to perform feature extraction on the road feature map corresponding to the first feature map with an image scale of 1/2, Obtain the first feature map with an image scale of 1/4, and obtain the road feature map corresponding to the first feature map with an image scale of 1/4, and then use a 2*2 maximum pooling layer with a step size of 2 to Perform feature extraction for the road feature map corresponding to the first feature map of 1/4, obtain the first feature map with an image scale of 1/8, and obtain the road feature map corresponding to the first feature map with an image scale of 1/8, Then use a 2*2 maximum pooling layer with a step size of 2 to perform feature extraction on the road feature map corresponding to the first feature map with an image scale of 1/8 to obtain the first feature map with an image scale of 1/16, and The road feature map corresponding to the first feature map whose image scale is 1/16 is obtained.

Step 62: For each road feature map, perform the following operations respectively: based on the respective image position information of each pixel contained in a road feature map, combined with the image channel information contained in a road feature map, perform attention weighting processing on the road feature map , to obtain the corresponding attention feature map.

Specifically, when executing step 62, the server specifically performs the following operations. Referring to Figure 9, it is a schematic flow chart of obtaining an attention feature map corresponding to a road feature map in the embodiment of the present application. The specific operations performed will be described in detail below in conjunction with Figure 9:

Step 620: Based on the respective image position information of each pixel contained in a road feature map, determine the respective location attention weights of each pixel contained in a road feature map.

Among them, the position attention weight represents the importance of the image position of the corresponding pixel when identifying the drivable area.

In the embodiment of the present application, based on the respective image position information of each pixel contained in a road feature map, average pooling and maximum pooling operations are performed on the road feature map in the spatial position dimension to obtain the spatial average pooling feature map and the spatial maximum Pooling the feature map, and then splicing the spatial average pooling feature map and the spatial maximum pooling feature map to obtain the spatial pooling splicing feature map, and finally through the convolution layer and sigmoid function activation to obtain each pixel contained in the road feature map Point the respective positional attention weights.

Step 621: Based on the image channel information contained in a road feature map, determine the respective channel attention weights of each image channel contained in a road feature map.

Among them, the channel attention weight represents the importance of the corresponding image channel when identifying the drivable area.

In the embodiment of the present application, based on the image channel information contained in a road feature map, the average pooling and maximum pooling operations are performed on the road feature map in the channel dimension to obtain the channel average pooling feature map and the channel maximum pooling feature map, and then The channel average pooling feature map and the channel maximum pooling feature map are spliced to obtain the channel pooling stitching feature map, and finally the convolution layer and the sigmoid function are activated to obtain the channel attention of each image channel contained in the road feature map Weights.

Step 622: Based on the obtained attention weights of each position and channel attention weight, weight a road feature map to obtain an attention feature map corresponding to a road feature map.

In the embodiment of the present application, after obtaining the position attention weights and channel attention weights, in the spatial position dimension, a road feature map is multiplied by its position attention weights to obtain the position attention feature map. In the channel dimension, Multiply the road feature map with the attention weights of each channel to obtain the channel attention feature map, and finally add the position attention feature map and the channel attention feature map to obtain the attention feature map corresponding to the road feature map.

For example, referring to Figure 10, it is a schematic diagram of obtaining the attention feature map in the embodiment of the present application. Assuming that the position attention weight matrix of the road feature map 1 is A, and the channel attention weight matrix is B, then the road feature map 1 Multiply with the position attention weight matrix A to obtain the position attention feature map 1, multiply the road feature map 1 with the channel attention weight matrix B to obtain the channel attention feature map 1, and finally the position attention feature map 1 is added to the channel attention feature map 1 to obtain the attention feature map 1 corresponding to the road feature map 1.

In this way, based on the image position information of each pixel contained in a road feature map, combined with the image channel information contained in a road feature map, the attention weighting process is performed on the road feature map to obtain the corresponding attention feature map, which can pay attention to the road Important information in the feature map suppresses useless information and improves the accuracy of drivable area recognition.

Step 63: Based on the target image scale, perform image scale adjustment on the obtained attention feature maps to obtain at least one target feature map, and obtain the drivable area of the target vehicle based on the at least one target feature map.

In the embodiment of the present application, when performing image scale adjustment on each of the obtained attention feature maps based on the target image scale to obtain at least one target feature map, the server specifically performs the following operations. Referring to FIG. 11 , it is a schematic flow diagram of obtaining at least one target feature map according to an embodiment of the present application. The specific operations performed will be described in detail below in conjunction with FIG. 11 :

Step 630: Based on the obtained image scales of each attention feature map, perform fusion and update on each attention feature map to obtain at least one fusion feature map.

Specifically, there are many ways to fuse and update each attention feature map.

The first method: refer to FIG. 12 , which is a schematic flow diagram of obtaining at least one fusion feature map according to the embodiment of the present application. The specific operations performed will be described in detail below in conjunction with FIG. 12 :

Step 6300: According to the obtained image scales of each attention feature map, sort each attention feature map to obtain a sorting result.

In the embodiment of the present application, according to the respective image scales of the obtained attention feature maps, the attention feature maps are sorted in ascending order of image scale, and the sorting result is obtained.

For example, referring to FIG. 13 , it is a schematic diagram of obtaining at least one fusion feature map in the embodiment of the present application. It is assumed that each attention feature map includes attention feature map P1, attention feature map P2, attention feature map P3, attention feature map Feature map P4, the image scale of the attention feature map P1 is 1/2, the image scale of the attention feature map P2 is 1/4, the image scale of the attention feature map P3 is 1/8, and the attention feature map The image scale of P4 is 1/16, and the attention feature maps are sorted according to the image scale from small to large, and the obtained sorting results are P1, P2, P3, and P4.

Step 6301: Based on the sorting result, sequentially read every two adjacent attention feature maps for fusion until the reading is complete.

Wherein, one fusion process includes: reading two adjacent attention feature maps, and fusing the two attention feature maps to obtain the fusion feature map. Save the fusion feature map, and use the fusion feature map as a new attention feature map to replace the two attention feature maps and add it to the sorting result.

For example, as shown in FIG. 13 , the sample sorting results are P1, P2, P3, and P4. Read P1 and P2, and fuse P1 and P2 to obtain the fusion feature map R1, save R1, and use R1 as a new attention feature map to replace P1 and P2, and add it to the sorting result, and the sorting result is now R1 , P3, P4, read R1 and P3, and fuse R1 and P3 to obtain the fusion feature map R2, save R2, and use R2 as a new attention feature map instead of R1 and P3, and add it to the sorting result. When the sorting results are R2 and P4, read R2 and P4, and fuse R2 and P4 to obtain the fusion feature map R3, save R3, and finally obtain at least one fusion feature map including R1, R2 and R3.

Further, before the fusion update between the attention feature maps corresponding to different image scales, the road feature map corresponding to the small image scale is up-sampled to the same image scale as the road feature map corresponding to the large image scale, and then based on the up-sampling The image position information of each pixel contained in the post-sampled road feature map, combined with the image channel information contained in the up-sampled road feature map, performs attention weighting processing on the up-sampled road feature map, and obtains the corresponding small image scale. Attention feature map.

The second method: according to the respective image scales of the obtained attention feature maps, sort the attention feature maps to obtain the sorting results, and based on the sorting results, read every two adjacent attention feature maps in turn Fusion is performed until the read is complete.

Wherein, a fusion process includes: reading two adjacent sample attention feature maps, and fusing the two sample attention feature maps to obtain the sample fusion feature map, and saving the sample fusion feature map.

For example, assuming that each attention feature map includes attention feature map P1, attention feature map P2, attention feature map P3, and attention feature map P4, the attention feature maps are sorted according to the image scale from small to large, and the obtained The sorting results are P1, P2, P3, P4, read P1 and P2, and fuse P1 and P2 to obtain the fusion feature map R4, save R4, read P2 and P3, and fuse P2 and P3 to obtain the fusion Feature map R5, save R5, read P3 and P4, and fuse P3 and P4 to obtain a fusion feature map R6, save R6, and finally obtain at least one fusion feature map including R4, R5 and R6.

The third method: fuse each attention feature map to obtain a fusion feature map.

For example, assuming that each attention feature map includes attention feature map P1, attention feature map P2, attention feature map P3, and attention feature map P4, then P1, P2, P3, and P4 are fused to obtain a fusion feature map R7.

Step 631: Based on the preset number of target image channels, perform image channel adjustment on at least one fused feature map to obtain corresponding intermediate feature maps.

In the embodiment of the present application, after at least one fused feature map is obtained, the following operations are respectively performed for at least one fused feature map: using a convolutional layer, based on the preset number of target image channels, image channel adjustment is performed on a fused feature map, Obtain the intermediate feature map corresponding to the fused feature map.

Wherein, the preset target image channel number may be the channel number of the target road image, which is not limited in this embodiment of the present application.

For example, referring to Figure 14, which is a schematic diagram of obtaining at least one target feature map in the embodiment of the present application, assuming that at least one fusion feature map includes fusion feature map R1, fusion feature map R2 and fusion feature map R3, the preset target image The number of channels is 3, and the convolutional layer is used to adjust the image channels of the fusion feature map R1, fusion feature map R2 and fusion feature map R3 respectively, and obtain the intermediate feature map Z1 with 3 image channels and the intermediate feature map with 3 image channels. Feature map Z2, intermediate feature map Z3 with 3 image channels.

Step 632: Based on the target image scale, perform image scale adjustment on at least one obtained intermediate feature map to obtain a corresponding target feature map.

In the embodiment of the present application, after at least one intermediate feature map is obtained, the following operations are performed on at least one intermediate feature map: using bilinear interpolation, based on the target image scale, an image scale is adjusted on an intermediate feature map to obtain the intermediate The target feature map corresponding to the feature map.

For example, as shown in Figure 14, at least one intermediate feature map includes intermediate feature map Z1, intermediate feature map Z2, and intermediate feature map Z3. Image scale adjustment is performed on image Z2 and intermediate feature map Z3 to obtain a target feature map M1 with an image scale of 640x480, a target feature map M2 with an image scale of 640x480, and a target feature map M3 with an image scale of 640x480.

In this way, at least one sample target feature map with the same number of image channels and the same image scale is obtained, and the differences in channel and scale features of different attention feature maps are integrated to improve the accuracy of drivable area recognition.

In the embodiment of the present application, when obtaining the drivable area of the target vehicle based on at least one target feature map, the server specifically performs the following operations. Referring to Figure 15, which is a schematic flow chart of obtaining the drivable area of the target vehicle in the embodiment of the present application, the specific operations performed will be described in detail below in conjunction with Figure 15:

Step 633: Based on at least one target feature map, obtain a result feature map.

In the embodiment of this application, the sample result feature map is obtained, including but not limited to the following two situations:

Case 1: If at least one target feature map includes one target feature map, then use one target feature map as the result feature map.

For example, assuming that at least one target feature map only includes the target feature map M7, then the target feature map M7 is used as the result feature map.

Case 2: If at least one target feature map includes multiple target feature maps, image fusion is performed on the multiple target feature maps to obtain a resultant feature map.

For example, assuming that at least one target feature map includes a target feature map M1, a target feature map M2, and a target feature map M3, M1, M2, and M3 are fused to obtain a resultant feature map.

Step 634: Compare the value of each pixel contained in the result feature map with the preset drivable area label value, and determine each target pixel point belonging to the drivable area in the result feature map.

In the embodiment of the present application, the following operations are performed for each pixel contained in the result feature map: compare the value of a pixel with the preset drivable area label value, If the label values are the same, the pixel is determined to be the target pixel.

Wherein, the preset drivable area tag value may be 1, which is not limited in this embodiment of the present application.

For example, assuming that the label value of the drivable area is 1, compare the value of each pixel contained in the result feature map with the label value 1, if the pixel points in the result feature map that are the same as the label value 1 are: pixel 1 , pixel 2, and pixel 3, then it is determined that each target pixel in the result feature map that belongs to the drivable area is pixel 1, pixel 2, and pixel 3.

Step 635: Mark each pixel point corresponding to the respective image position of each target pixel point in the target road image, and obtain an image including a drivable area.

For example, referring to Figure 16, which is a schematic diagram of an image including a drivable area in the embodiment of the present application, after obtaining each target pixel point, each pixel point corresponding to the respective image position of each target pixel point in the target road image Mark and segment the drivable area to obtain an image containing the drivable area.

Based on the same inventive concept, the embodiment of the present application also provides a drivable area recognition device, as shown in Figure 17, which is a schematic structural diagram of the drivable area recognition device in the embodiment of the present application, specifically including:

An acquisition module 1701, configured to acquire a current target road image to be identified in response to a drivable area identification request triggered by a driving object, where the target road image has a target image scale;

The extraction module 1702 is used to perform feature extraction on the target road image according to the preset scales of each candidate image to obtain a corresponding road feature map;

The first processing module 1703 is configured to perform the following operations on each road feature map: based on the image position information of each pixel contained in a road feature map, combined with the image channel information contained in a road feature map, the road feature map Perform attention weighting processing to obtain the corresponding attention feature map;

The second processing module 1704 is configured to adjust the image scale of each attention feature map obtained based on the target image scale, obtain at least one target feature map, and obtain the drivable area of the target vehicle based on the at least one target feature map.

Optionally, according to the preset scales of each candidate image, feature extraction is performed on the target road image respectively, and when the corresponding road feature map is obtained, the extraction module 1702 is used for:

For each preset candidate image scale, perform the following operations:

Based on a candidate image scale, performing first feature extraction on the target road image to obtain a first feature map, the first feature map has a first number of image channels;

According to the preset receptive field of each candidate image, the second feature extraction is performed on the first feature map to obtain the corresponding second feature map, and the obtained second feature maps are superimposed to obtain the superimposed feature map; wherein, each A representation of the receptive field of a candidate image: each pixel contained in the corresponding second feature map is mapped to the area size on the target road image;

Based on the number of first image channels, image channel adjustment is performed on the superposition feature map to obtain a road feature map corresponding to a candidate image scale.

Optionally, based on the respective image position information of each pixel contained in a road feature map, combined with the image channel information contained in a road feature map, the road feature map is subjected to attention weighting processing, and when the corresponding attention feature map is obtained, The first processing module 1703 is used for:

Based on the respective image position information of each pixel contained in a road feature map, determine the respective position attention weights of each pixel contained in a road feature map, where the position attention weight represents the corresponding pixel points when identifying the drivable area The importance of the image position;

Based on the image channel information contained in a road feature map, determine the respective channel attention weights of each image channel contained in a road feature map, wherein the channel attention weight represents the importance of the corresponding image channel when identifying the drivable area;

Based on the obtained attention weights of each position and channel attention weights, a road feature map is weighted to obtain an attention feature map corresponding to a road feature map.

Optionally, based on the target image scale, perform image scale adjustment on each of the obtained attention feature maps to obtain at least one target feature map, and the second processing module 1704 is used to:

Based on the respective image scales of the obtained attention feature maps, fusion and update are performed on each attention feature map to obtain at least one fusion feature map;

Based on the preset number of target image channels, image channel adjustment is performed on at least one fusion feature map to obtain a corresponding intermediate feature map;

Based on the target image scale, image scale adjustment is performed on at least one obtained intermediate feature map to obtain a corresponding target feature map.

Optionally, based on the respective image scales of the obtained attention feature maps, fusion and update are performed on each attention feature map, and when at least one fusion feature map is obtained, the second processing module 1704 is also used to:

According to the respective image scales of the obtained attention feature maps, sort the attention feature maps to obtain the sorting results;

Based on the sorting results, each two adjacent attention feature maps are read sequentially for fusion until the reading is completed. A fusion process includes:

Read two adjacent attention feature maps, and fuse the two attention feature maps to obtain a fusion feature map;

Save the fusion feature map, and use the fusion feature map as a new attention feature map to replace the two attention feature maps and add it to the sorting result.

Optionally, when obtaining the drivable area of the target vehicle based on at least one target feature map, the second processing module 1704 is also used to:

If at least one target feature map includes a target feature map, then using a target feature map as a result feature map;

If at least one target feature map includes multiple target feature maps, performing image fusion on the multiple target feature maps to obtain a resultant feature map;

Comparing the value of each pixel contained in the result feature map with the preset drivable area label value, and determining each target pixel point belonging to the drivable area in the result feature map;

In the target road image, each pixel point corresponding to the respective image position of each target pixel point is marked to obtain an image including a drivable area.

Optionally, the drivable area is obtained by inputting the target road image into the target recognition model, and the device further includes a training module 1705, which is used for:

Perform iterative training on the recognition model to be trained based on the training sample set to obtain the target recognition model; each training sample includes: image data of the sample road, wherein each iteration process performs the following operations:

According to the preset image scales of each sample, feature extraction is performed on the selected training samples to obtain corresponding sample feature maps, and the training samples have sample image scales;

For each sample feature map, perform the following operations respectively: Based on the respective sample image position information of each pixel contained in a sample feature map, combined with the sample image channel information contained in a sample feature map, perform attention weighting processing on the sample feature map, Obtain the corresponding sample attention feature map;

Based on the sample image scale, image scale adjustment is performed on the obtained attention feature maps of each sample to obtain at least one target sample feature map;

Based on at least one target sample feature map, the drivable area prediction result of the training sample is obtained, and the parameters are adjusted based on the loss value corresponding to the drivable area prediction result.

Based on the above embodiments, refer to FIG. 18 , which is a schematic structural diagram of an electronic device in an embodiment of the present application.

An embodiment of the present application provides an electronic device, which may include a processor 1810 (Center Processing Unit, CPU), a memory 1820, an input device 1830, an output device 1840, etc., and the input device 1830 may include a keyboard, a mouse, a touch screen, etc., The output device 1840 may include a display device, such as a liquid crystal display (Liquid Crystal Display, LCD), a cathode ray tube (Cathode Ray Tube, CRT), and the like.

The memory 1820 may include Read Only Memory (ROM) and Random Access Memory (RAM), and provides program instructions and data stored in the memory 1820 to the processor 1810 . In the embodiment of the present application, the memory 1820 may be used to store programs of any drivable area identification method in the embodiment of the present application.

The processor 1810 invokes the program instructions stored in the memory 1820, and the processor 1810 is configured to execute any drivable area identification method in the embodiments of the present application according to the obtained program instructions.

Based on the above-mentioned embodiments, in the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the drivable area identification method in any of the above-mentioned method embodiments is implemented .

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the present application. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

Obviously, those skilled in the art can make various changes and modifications to the application without departing from the spirit and scope of the application. In this way, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.

Claims (10)

1. A drivable region recognition method, applied to a target vehicle, comprising:

responding to a driving object triggered request for identifying a drivable area, and acquiring a current target road image to be identified, wherein the target road image has a target image scale;

respectively extracting features of the target road image according to preset candidate image scales to obtain a corresponding road feature map;

For each road feature map, the following operations are performed: based on respective image position information of each pixel point contained in one road feature map, carrying out attention weighting processing on the road feature map by combining image channel information contained in the one road feature map to obtain a corresponding attention feature map;

and adjusting the image scale of each obtained attention characteristic image based on the target image scale to obtain at least one target characteristic image, and obtaining the drivable area of the target vehicle based on the at least one target characteristic image.

2. The method of claim 1, wherein the extracting features of the target road image according to the preset candidate image scales to obtain the corresponding road feature map includes:

for each preset candidate image scale, the following operations are respectively executed:

based on a candidate image scale, carrying out first feature extraction on the target road image to obtain a first feature map, wherein the first feature map has a first image channel number;

respectively extracting second features of the first feature images according to preset candidate image receptive fields to obtain corresponding second feature images, and superposing the obtained second feature images to obtain a superposed feature image; wherein each candidate image receptive field is characterized: mapping each pixel point contained in the corresponding second feature map to the size of the region on the target road image;

And based on the first image channel number, carrying out image channel adjustment on the overlapped feature map to obtain a road feature map corresponding to the candidate image scale.

3. The method according to claim 1, wherein the performing attention weighting processing on the road feature map based on the respective image position information of each pixel point included in the road feature map and in combination with the image channel information included in the road feature map to obtain a corresponding attention feature map includes:

determining the respective position attention weights of the pixels contained in one road feature map based on the respective image position information of the pixels contained in the one road feature map, wherein the position attention weights represent the importance degrees of the image positions of the corresponding pixels when the drivable region is identified;

determining the respective channel attention weights of the image channels contained in the road feature map based on the image channel information contained in the road feature map, wherein the channel attention weights represent the importance degrees of the corresponding image channels when the drivable area is identified;

and weighting the one road feature map based on the obtained attention weights of the positions and the channel attention weights to obtain an attention feature map corresponding to the one road feature map.

4. A method according to any of claims 1-3, wherein said image scaling the obtained attention profile based on the target image scale to obtain at least one target profile comprises:

based on the respective image scale of each obtained attention feature map, carrying out fusion updating on each attention feature map to obtain at least one fusion feature map;

based on the preset target image channel number, respectively carrying out image channel adjustment on the at least one fusion feature map to obtain a corresponding intermediate feature map;

and based on the target image scale, performing image scale adjustment on the obtained at least one intermediate feature image to obtain a corresponding target feature image.

5. The method of claim 4, wherein the performing a fusion update on each of the obtained attention profile based on the respective image scale sizes of the attention profile to obtain at least one fusion profile comprises:

sequencing all the attention feature images according to the respective image scale of the obtained attention feature images to obtain sequencing results;

based on the sorting result, sequentially reading every two adjacent attention feature graphs to fuse until the reading is finished, wherein one fusing process comprises the following steps:

Reading two adjacent attention feature images, and fusing the two attention feature images to obtain a fused feature image;

and storing the fusion feature map, taking the fusion feature map as a new attention feature map, replacing the two attention feature maps, and adding the fusion feature map into the sequencing result.

6. A method according to any one of claims 1-3, wherein said obtaining a drivable region of the target vehicle based on the at least one target feature map comprises:

if the at least one target feature map comprises one target feature map, taking the one target feature map as a result feature map;

if the at least one target feature map comprises a plurality of target feature maps, performing image fusion on the plurality of target feature maps to obtain a result feature map;

comparing the value of each pixel point contained in the result feature map with a preset label value of a drivable area to determine each target pixel point belonging to the drivable area in the result feature map;

and marking each pixel point corresponding to each image position of each target pixel point in the target road image to obtain an image containing a travelable area.

7. A method according to any one of claims 1-3, wherein the travelable region is obtained by inputting the target road image into a target recognition model, wherein the target recognition model is trained in the following manner:

performing iterative training on the recognition model to be trained based on the training sample set to obtain a target recognition model; each training sample includes: image data of a sample road, wherein each iterative process performs the following operations:

respectively extracting features of a selected training sample according to preset sample image scales to obtain a corresponding sample feature map, wherein the training sample has the sample image scales;

for each sample feature map, the following operations are respectively executed: based on the sample image position information of each pixel point contained in one sample feature map, carrying out attention weighting processing on the sample feature map by combining the sample image channel information contained in the one sample feature map to obtain a corresponding sample attention feature map;

based on the sample image scale, respectively carrying out image scale adjustment on each obtained sample attention feature image to obtain at least one target sample feature image;

And obtaining a prediction result of the drivable region of the training sample based on the at least one target sample feature map, and performing parameter adjustment based on a loss value corresponding to the prediction result of the drivable region.

8. A device for identifying a travelable region, comprising

The driving object identification module is used for acquiring a driving object-triggered driving area identification request, and acquiring a driving object-triggered driving area identification request;

the extraction module is used for respectively carrying out feature extraction on the target road image according to the preset candidate image scales to obtain a corresponding road feature map;

the first processing module is used for respectively executing the following operations for each road feature map: based on respective image position information of each pixel point contained in one road feature map, carrying out attention weighting processing on the road feature map by combining image channel information contained in the one road feature map to obtain a corresponding attention feature map;

and the second processing module is used for carrying out image scale adjustment on each obtained attention characteristic image based on the target image scale to obtain at least one target characteristic image, and obtaining a drivable area of the target vehicle based on the at least one target characteristic image.

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method of any of claims 1-7 when the program is executed by the processor.

10. A computer-readable storage medium having stored thereon a computer program, characterized by: the computer program implementing the steps of the method of any of claims 1-7 when executed by a processor.