CN114757301B

CN114757301B - Vehicle-mounted visual perception method and device, readable storage medium, and electronic device

Info

Publication number: CN114757301B
Application number: CN202210514756.2A
Authority: CN
Inventors: 朱红梅; 孟文明; 王梦圆; 李翔宇; 任伟强; 张骞
Original assignee: Beijing Horizon Robotics Technology Research and Development Co Ltd
Current assignee: Beijing Horizon Robotics Technology Research and Development Co Ltd
Priority date: 2022-05-12
Filing date: 2022-05-12
Publication date: 2025-02-14
Anticipated expiration: 2042-05-12
Also published as: CN114757301A

Abstract

The embodiments of the present disclosure disclose a vehicle-mounted visual perception method and device, a readable storage medium, and an electronic device, wherein the method comprises: acquiring images at multiple consecutive moments by multiple vehicle-mounted cameras arranged at preset positions on a vehicle to obtain multiple image frame sets; wherein each of the image frame sets includes multiple image frames acquired based on the same vehicle-mounted camera, and each vehicle-mounted camera corresponds to one image frame at each moment; performing feature extraction on the image frames included in the multiple image frame sets by a coding branch network in a first network model to obtain multiple first feature maps; performing spatial fusion and temporal fusion on the multiple first feature maps by a fusion branch network in the first network model to obtain a second feature map in a bird's-eye view image coordinate system; and identifying the second feature map based on a network model corresponding to a preset perception task to determine a perception result corresponding to the preset perception task.

Description

Vehicle-mounted visual perception method and device, readable storage medium and electronic equipment

Technical Field

The disclosure relates to the technical field of automatic driving, in particular to a vehicle-mounted visual perception method and device, a readable storage medium and electronic equipment.

Background

In the automatic driving field, the sensing results of a plurality of vehicle-mounted cameras in an external sensing system under respective coordinate systems cannot be directly used for a subsequent prediction and regulation system, dynamic and static sensing results of different visual angles are required to be fused in a certain mode and are uniformly expressed under an own vehicle coordinate system, and in the prior art, the fusion of different visual angles is generally carried out independently through post-processing.

Disclosure of Invention

The present disclosure has been made in order to solve the above technical problems. The embodiment of the disclosure provides a vehicle-mounted visual perception method and device, a readable storage medium and electronic equipment.

According to an aspect of the embodiments of the present disclosure, there is provided a vehicle-mounted visual perception method, including:

acquiring images at a plurality of continuous moments through a plurality of vehicle-mounted cameras arranged at preset positions on a vehicle to obtain a plurality of image frame sets, wherein each image frame set comprises a plurality of frame image frames acquired based on the same vehicle-mounted camera, and each vehicle-mounted camera corresponds to one frame image frame at each moment;

Extracting features of the image frames included in the plurality of image frame sets through a coding branch network in the first network model to obtain a plurality of first feature images;

performing space fusion and time sequence fusion on the plurality of first feature images through a fusion branch network in the first network model to obtain a second feature image under a bird's-eye view image coordinate system;

And identifying the second feature map based on a network model corresponding to a preset sensing task, and determining a sensing result corresponding to the preset sensing task.

According to another aspect of the embodiments of the present disclosure, there is provided an in-vehicle visual perception apparatus, including:

The system comprises an image acquisition module, a display module and a display module, wherein the image acquisition module is used for acquiring images at a plurality of continuous moments through a plurality of vehicle-mounted cameras arranged at preset positions on a vehicle to obtain a plurality of image frame sets, wherein each image frame set comprises a plurality of frame image frames acquired based on the same vehicle-mounted camera, and each frame image frame corresponds to one moment respectively;

the coding module is used for extracting the characteristics of the image frames included in the plurality of image frame sets obtained by the image acquisition module through a coding branch network in the first network model to obtain a plurality of first characteristic images;

The fusion module is used for performing space fusion and time sequence fusion on the plurality of first feature images determined by the encoding module through a fusion branch network in the first network model to obtain a second feature image under a bird's-eye view image coordinate system;

And the sensing module is used for identifying the second feature map determined by the fusion module based on a network model corresponding to a preset sensing task to obtain a sensing result corresponding to the preset sensing task.

According to yet another aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium storing a computer program for executing the vehicle-mounted visual perception method according to any one of the above embodiments.

According to still another aspect of the embodiments of the present disclosure, there is provided an electronic device including:

A processor;

a memory for storing the processor-executable instructions;

The processor is configured to read the executable instructions from the memory and execute the instructions to implement the vehicle-mounted visual perception method according to any one of the foregoing embodiments.

According to the vehicle-mounted visual perception method and device, the readable storage medium and the electronic equipment, the space fusion and the time sequence fusion are realized in the first neural network, so that the end-to-end learning of the neural network is realized, and the complexity of the space fusion and the time sequence fusion of the image during the post-processing can be effectively avoided because the post-processing fusion is not needed, and the situation that the same target is mistakenly identified as a plurality of targets in the post-processing can be avoided.

The technical scheme of the present disclosure is described in further detail below through the accompanying drawings and examples.

Drawings

The above and other objects, features and advantages of the present disclosure will become more apparent by describing embodiments thereof in more detail with reference to the accompanying drawings. The accompanying drawings are included to provide a further understanding of embodiments of the disclosure, and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain the disclosure, without limitation to the disclosure. In the drawings, like reference numerals generally refer to like parts or steps.

Fig. 1 is a schematic diagram of a first network model according to an exemplary embodiment of the present disclosure.

Fig. 2 is a flow chart of a vehicle-mounted visual perception method according to an exemplary embodiment of the present disclosure.

Fig. 3 is a flow chart illustrating step 203 in the embodiment shown in fig. 2 of the present disclosure.

Fig. 4 is a schematic flow chart of step 2031 in the embodiment of fig. 3 of the disclosure.

Fig. 5a is a schematic flow chart of step 401 in the embodiment shown in fig. 4 of the present disclosure.

Fig. 5b is a schematic diagram of a conversion relationship between a bird's eye view image plane and a preset plane of a vehicle coordinate system in an exemplary embodiment of the present disclosure.

Fig. 6 is a flow chart illustrating step 2032 in the embodiment of fig. 3 of the disclosure.

Fig. 7 is a flow chart illustrating step 204 in the embodiment shown in fig. 2 of the present disclosure.

Fig. 8 is a schematic structural diagram of an in-vehicle visual perception device according to an exemplary embodiment of the present disclosure.

Fig. 9 is a schematic structural view of an in-vehicle visual perception device provided in another exemplary embodiment of the present disclosure.

Fig. 10 is a block diagram of an electronic device provided in an exemplary embodiment of the present disclosure.

Detailed Description

Hereinafter, example embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. It should be apparent that the described embodiments are only some of the embodiments of the present disclosure and not all of the embodiments of the present disclosure, and that the present disclosure is not limited by the example embodiments described herein.

It should be noted that the relative arrangement of the components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless it is specifically stated otherwise.

It will be appreciated by those of skill in the art that the terms "first," "second," etc. in embodiments of the present disclosure are used merely to distinguish between different steps, devices or modules, etc., and do not represent any particular technical meaning nor necessarily logical order between them.

It should also be understood that in embodiments of the present disclosure, "plurality" may refer to two or more, and "at least one" may refer to one, two or more.

It should also be appreciated that any component, data, or structure referred to in the presently disclosed embodiments may be generally understood as one or more without explicit limitation or the contrary in the context.

In addition, the term "and/or" in this disclosure is merely an association relation describing the association object, and indicates that three kinds of relations may exist, for example, a and/or B may indicate that a exists alone, and a and B exist together, and B exists alone. In addition, the character "/" in the present disclosure generally indicates that the front and rear association objects are an or relationship.

It should also be understood that the description of the various embodiments of the present disclosure emphasizes the differences between the various embodiments, and that the same or similar features may be referred to each other, and for brevity, will not be described in detail.

Meanwhile, it should be understood that the sizes of the respective parts shown in the drawings are not drawn in actual scale for convenience of description.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses.

Techniques, methods, and apparatus known to one of ordinary skill in the relevant art may not be discussed in detail, but are intended to be part of the specification where appropriate.

It should be noted that like reference numerals and letters refer to like items in the following figures, and thus once an item is defined in one figure, no further discussion thereof is necessary in subsequent figures.

Embodiments of the present disclosure may be applicable to electronic devices such as terminal devices, computer systems, servers, etc., which may operate with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known terminal devices, computing systems, environments, and/or configurations that may be suitable for use with electronic devices such as terminal devices, computer systems, servers, etc., include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputers systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems, and the like.

Electronic devices such as terminal devices, computer systems, servers, etc. may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc., that perform particular tasks or implement particular abstract data types. The computer system/server may be implemented in a distributed cloud computing environment in which tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computing system storage media including memory storage devices.

Summary of the application

In the process of realizing the present disclosure, the inventor finds that the vehicle-mounted visual perception method provided in the prior art generally adopts a post-processing mode, and fuses the perception results of different cameras through some rules after the neural network model outputs the perception results; the method has at least the following problems that when the perception results of the overlapping areas of the adjacent visual angles are fused in a post-processing mode, as the overlapping exists among the visual angles, the detection result can be obtained for the same target in the images of different visual angles, and the same target appears in different positions in different images, so that complex identification and detection are required during post-processing fusion, and the same target is easily identified as a plurality of targets to cause ambiguity.

Exemplary network architecture

Fig. 1 is a schematic diagram of a first network model according to an exemplary embodiment of the present disclosure. As shown in fig. 1, the first network model in the present embodiment includes an encoding branch network 101, a merging branch network 102, and a decoding branch network 103.

In this embodiment, feature extraction is performed on image frames acquired by a plurality of vehicle-mounted cameras through the coding branch network 101 to obtain a plurality of first feature graphs, where the plurality of vehicle-mounted cameras are vehicle-mounted cameras disposed at different preset positions on a vehicle, each vehicle-mounted camera corresponds to a view angle, and a shooting view range corresponding to the plurality of vehicle-mounted cameras may cover a surrounding environment of the vehicle, so as to avoid a view angle blind area, and the shooting view ranges between the plurality of vehicle-mounted cameras may overlap. In the running process of the vehicle, each vehicle-mounted camera continuously collects a plurality of images at a plurality of moments based on one visual angle at a set position to obtain image frame sets respectively corresponding to each vehicle-mounted camera. Alternatively, the coding branch network 101 may be any network that can implement feature extraction, for example, a convolutional neural network, and the embodiment is not limited to a network structure specifically adopted.

And performing space fusion and time sequence fusion on the plurality of first feature images through a fusion branch network 102 to obtain a second feature image under a bird's-eye view image coordinate system, wherein the space fusion fuses multiple frame image frames acquired at the same moment by each vehicle-mounted camera in the plurality of vehicle-mounted cameras to obtain a third feature image under the bird's-eye view image coordinate system, and the time sequence fusion fuses the third feature images corresponding to each moment in the plurality of moments to obtain the time sequence fused second feature image.

The second feature map obtained by merging the branch network 102 is decoded by the decoding branch network 103 to obtain a decoded second feature map, alternatively, the decoded second feature map in this embodiment may be a feature map conforming to a structure of a subsequent sensing task, alternatively, the decoding branch network 103 may be any network capable of implementing feature extraction, for example, a convolutional neural network, and the embodiment is not limited to a specific network structure.

In order to implement the visual perception task, any visual perception task may be connected after the first network model, where the visual perception task may include, but is not limited to, a segmentation task, a detection task, a classification task, and the like.

According to the method, the spatial fusion and the time fusion are realized inside the first network model, so that disturbance of internal and external parameters of a camera caused by the change of the posture of the camera in the driving process of a vehicle is not directly applied to a sensing result, the influence of the disturbance is overcome through training and learning of the first network model, the output sensing result is more stable and is not influenced by the change of the posture of the camera in the driving process of the vehicle, the spatial fusion and the time fusion are not required in the first network model, the spatial fusion and the time fusion in the post-processing process are avoided, the post-processing complexity is reduced, the first network model can realize end-to-end learning, the potential of joint learning with candidate visual sensing tasks is provided, and the requirement of the sensing fusion method through the first network model on the computational complexity of a chip is lower.

Exemplary method

Fig. 2 is a flow chart of a vehicle-mounted visual perception method according to an exemplary embodiment of the present disclosure. The embodiment can be applied to an electronic device, as shown in fig. 2, and includes the following steps:

In step 201, image acquisition is performed at a plurality of continuous moments by a plurality of vehicle-mounted cameras arranged at preset positions on a vehicle, so as to obtain a plurality of image frame sets.

Each image frame set comprises a plurality of image frames acquired based on the same vehicle-mounted camera, and each vehicle-mounted camera corresponds to one image frame at each moment.

Optionally, each vehicle-mounted camera corresponds to one image frame set, so that the plurality of vehicle-mounted cameras obtain a plurality of image frame sets, each vehicle-mounted camera can correspond to different directions, and the image acquisition ranges corresponding to the plurality of vehicle-mounted cameras can be overlapped or not overlapped.

Step 202, extracting features of image frames included in a plurality of image frame sets through a coding branch network in a first network model to obtain a plurality of first feature images.

Optionally, feature extraction is sequentially performed on the image frames included in the plurality of image frame sets through the coding branch network in the first network model to obtain a plurality of first feature graphs, where the coding branch network in this embodiment may be understood by referring to the coding branch network 101 provided in fig. 1, so as to implement feature extraction on the image frames.

And 203, performing space fusion and time sequence fusion on the plurality of first feature images through a fusion branch network in the first network model to obtain a second feature image under the aerial view image coordinate system.

Alternatively, the coding branch network in this embodiment may be understood with reference to the merging branch network 102 provided in fig. 1, so as to implement merging of the first feature maps.

And 204, identifying the second feature map based on a network model corresponding to the preset sensing task, and determining a sensing result corresponding to the preset sensing task.

The preset sensing task in this embodiment may be any visual sensing task, for example, a segmentation task, a detection task, a classification task, and the like, and the recognition operation of this step is implemented through a network model corresponding to the visual sensing task.

According to the vehicle-mounted visual perception method provided by the embodiment of the disclosure, through the implementation of space fusion and time sequence fusion in the first neural network, the end-to-end learning of the neural network is realized, and post-processing fusion is not needed, so that the complexity degree of the space fusion and time sequence fusion of images during post-processing can be effectively avoided, and the situation that the same target is mistakenly identified as a plurality of targets in post-processing can be avoided.

As shown in fig. 3, step 203 may include the following steps, based on the embodiment shown in fig. 2, described above:

in step 2031, for each of the plurality of time points, spatial fusion is performed on the plurality of first feature maps corresponding to the time point, and a plurality of third feature maps in the aerial view image coordinate system are obtained.

Wherein each third feature map corresponds to a time instant.

In this embodiment, the image frame acquired by each vehicle-mounted camera is an image frame under a camera coordinate system corresponding to the vehicle-mounted camera, so that before fusion, a plurality of first feature images are projected under a bird's-eye view image coordinate system, and then fusion is performed on a plurality of feature images under the bird's-eye view image coordinate system, and fusion of a plurality of feature images under the same coordinate system can be achieved by a feature image fusion method in related technologies, for example, element-by-element addition, feature channel dimension stitching, element-by-element maximum value taking, neural network fusion, and the like.

Step 2032, performing timing fusion on the plurality of third feature maps to obtain a second feature map.

In this embodiment, each third feature map corresponds to a moment, and the second feature map is obtained by respectively reconstructing the third feature map corresponding to each moment into the third feature map corresponding to a certain moment (for example, the latest moment corresponding to the last image frame is acquired), and fusing the reconstructed multiple third feature maps, where the fusing in this step can also be implemented by a feature map fusing method in the related art, for example, element-by-element addition, feature channel dimension stitching, element-by-element maximum value taking, neural network fusing, and the like; in the embodiment, the space fusion is firstly executed and then the time sequence fusion is executed, so that the time sequence fusion is easier to realize, and the fusion speed of the feature map is increased.

As shown in fig. 4, on the basis of the embodiment shown in fig. 3, step 2031 may include the following steps:

Step 401, executing homography transformation on each first feature map in the plurality of first feature maps to obtain a plurality of transformed feature maps under the aerial view image coordinate system.

Wherein the homography (or projective transformation) has 8 degrees of freedom for describing the mapping of points on two planes. And converting each first feature image under the image coordinate system into the aerial view image coordinate system through homography transformation, so that a plurality of transformation feature images can be obtained.

And step 402, fusing the plurality of transformation feature images point by point to obtain a third feature image under the aerial view image coordinate system.

In the embodiment, a plurality of transformation feature maps under the same coordinate system are obtained through homography transformation, and the transformation feature maps are identical in size, so that fusion among the transformation feature maps can be realized in a point-by-point fusion mode, the point-by-point fusion method can comprise, but is not limited to, element-by-element addition, feature channel dimension stitching, element-by-element maximum value taking, neural network fusion and the like, the embodiment maps a plurality of first feature maps respectively corresponding to different coordinate systems under the same coordinate system through homography transformation, converts the feature map of each view angle from a camera view angle to a bird's eye view angle, and realizes fusion under the same coordinate system, a fused third feature map can have image features corresponding to all angles around a vehicle, and when a visual perception task is executed based on the third feature map, fusion of perception results of different vehicle-mounted cameras can be realized without post-processing, fusion efficiency of perception results is improved, and fusion difficulty is reduced.

As shown in fig. 5a, step 401 may comprise the following steps, based on the embodiment shown in fig. 4, described above:

step 4011, determining a first transformation matrix from the vehicle coordinate system corresponding to the first feature map to the camera coordinate system based on the internal reference matrix and the external reference matrix of the vehicle-mounted camera corresponding to each first feature map.

In this embodiment, the internal reference matrix corresponding to each vehicle-mounted camera is a known 3*3 matrix K, the external reference matrix corresponding to the vehicle-mounted camera can be calculated by the preset position of the vehicle-mounted camera on the vehicle, and based on the internal reference matrix and the external reference matrix, a first transformation matrix T _vcs2cam between the vehicle coordinate system and the camera coordinate system can be determined, where the first transformation matrix includes rotation parameters and translation parameters, for example, in an alternative example, the first transformation matrix can be expressed as: Where r ₁₁、r₁₂、r₁₃、r₂₁、r₂₂、r₂₃、r₃₁、r₃₂、r₃₃ denotes a rotation parameter, and t ₁、t₂、t₃ denotes a translation parameter.

Step 4012, determining a second transformation matrix from the aerial view image coordinate system to the preset plane of the vehicle coordinate system based on the perceived range on the preset plane of the vehicle coordinate system and the scaling and translation distance of the aerial view image coordinate system.

Alternatively, as shown in fig. 5b, a sensing range of the vehicle coordinate system on a preset plane (xy plane, where O _vcs represents a center point of the xy plane in the vehicle coordinate system) is denoted by d ₁,d₂,d₃,d₄, based on which a translation distance (e.g., d ₁,d₃ in fig. 5 b) of the vehicle coordinate system with respect to the aerial view image coordinate system can be determined, and assuming that the aerial view image plane (the center point of the image plane may be O _BEV in fig. 5 b) coincides with (or is parallel to) the xy plane of the vehicle coordinate system, and r represents a scale from the xy plane of the vehicle coordinate system to the aerial view image plane, the second transformation matrix may be denoted as:

step 4013, determining a third transformation matrix from the aerial view image coordinate system to the image coordinate system corresponding to the vehicle-mounted camera based on the first transformation matrix, the second transformation matrix and the internal reference matrix of the vehicle-mounted camera.

Optionally, in this embodiment, a transformation matrix from the aerial view image coordinate system to the camera coordinate system may be derived through the first transformation matrix and the second transformation matrix, and then, by combining with the internal reference matrix, a third transformation matrix T _bev2img between the aerial view image coordinate system and an image plane (corresponding to the image coordinate system) corresponding to the vehicle-mounted camera may be determined.

Step 4014, transforming the first feature map based on the third transformation matrix to obtain a transformed feature map.

In this embodiment, the third transformation matrix is determined by determining the transformation relation between the image coordinate system and the aerial image coordinate system, and by this transformation relation, the feature of the aerial image (u, v) position reverse index image position (u ₁,v₁) can be obtained, for example, by the following formula (1):

The method comprises the steps of obtaining a conversion characteristic diagram F _n after view angle conversion, similarly, executing the conversion operation on a first characteristic diagram corresponding to each view angle to obtain N conversion characteristic diagrams { F _n } after view angle conversion, wherein n=1, 2, N, and finally performing characteristic fusion on the conversion characteristic diagrams to obtain a third characteristic diagram F _t at t time (one of multiple time) after spatial fusion.

Optionally, on the basis of the above embodiment, step 3013 may further include:

a1, determining a fourth transformation matrix from a preset plane of the vehicle coordinate system to the camera coordinate system based on the first transformation matrix.

Alternatively, assuming that the imaging plane of the aerial view angle coincides with (or is parallel to) the xy plane of the vehicle coordinate system, that is, the plane in which the z value of the three-dimensional point coordinate in the vehicle coordinate system is equal to 0 (or is equal to any same value), when the imaging plane coincides with (or is parallel to) the column (3 rd column) of the corresponding z axis in the first transformation matrix T _vcs2cam, when the imaging plane of the aerial view angle coincides with (is parallel to) the column (3 rd column) may be simply eliminated, the fourth transformation matrix T _{vcs_xy2cam} between the imaging plane of the aerial image coordinate system and the camera coordinate system is obtained, for example, in an alternative example (the imaging plane of the aerial image coordinate system coincides with the xy plane of the vehicle coordinate system), the fourth transformation matrix may be expressed as:

a2, determining a fifth transformation matrix from the aerial image coordinate system to the camera coordinate system based on the fourth transformation matrix and the second transformation matrix.

Optionally, after determining the fourth transformation matrix, in combination with the second transformation matrix of the aerial image coordinate system to the vehicle coordinate system, matrix multiplication may be performed by the fourth transformation matrix and the second transformation matrix, and a fifth transformation matrix between the aerial image coordinate system to the camera coordinate system may be determined, for example, the fifth transformation matrix T _bev2cam may be determined based on the following formula (2):

t _bev2cam＝T_vcs-xy2cam×T_bev2vcs-xy formula (2)

A3, determining a third transformation matrix based on the fifth transformation matrix and an internal reference matrix of the vehicle-mounted camera.

In this embodiment, it is determined that the number of rows and columns between the fifth transformation matrix and the reference matrix do not correspond, in order to perform matrix multiplication (the reference matrix is a matrix of 3*3), a clipping operation may be performed on the fifth transformation matrix, and the first three rows of the fifth transformation matrix are clipped to obtain a clipping transformation matrix, for example, the first three rows of the fifth transformation matrix T _bev2cam are clipped to obtain a clipping transformation matrix T' _bev2cam, at this time, a matrix multiplication is performed on the clipping transformation matrix and the reference matrix, and a third transformation matrix is determined, for example, as shown in the following formula (3):

t _bev2img＝K×T′_bev2cam formula (3)

The third transformation matrix can be obtained through the processing, the first feature image of the corresponding image coordinate system can be converted into the imaging plane under the aerial view image coordinate system based on the third transformation matrix, the third transformation matrix for converting the aerial view image coordinate system into the image coordinate system is determined through coordinate system relation conversion, the feature in the image coordinate system can be directly converted into the aerial view image coordinate system based on the third transformation matrix, rapid space fusion is achieved in the fusion branch model of the first network model, specific parameters in coordinate system conversion are determined through learning of the first network model, and conversion accuracy and speed are improved.

As shown in fig. 6, on the basis of the embodiment shown in fig. 3, step 2032 may include the following steps:

In step 601, a third feature map corresponding to the latest time among the plurality of times is used as a reference feature map.

Optionally, when the plurality of time instants includes time instants t, t-1..once. The third feature map corresponding to the t-th time (the latest time) may be used as the reference feature map, and, of course, in practical application, the third feature map corresponding to any one moment may be used as a reference feature map, and the operation and the result of the time sequence fusion are not affected; when the latest moment is selected as the parameter feature map, the result of time domain fusion corresponds to the latest moment, namely the output second feature corresponds to the latest moment (for example, the current moment), so that the instantaneity of the sensing result determined based on the second feature is improved, namely the instantaneity of the vehicle-mounted visual angle sensing method is improved.

Step 602, reconstructing each third feature map in the at least one third feature map to obtain at least one fourth feature map.

In this embodiment, in order to achieve the purpose of fusing the third feature maps corresponding to the multiple moments, each third feature map is mapped into a feature space corresponding to the reference feature map, so as to perform feature fusion.

And 603, fusing the reference feature map and at least one fourth feature map point by point to obtain a second feature map.

The method for point-by-point fusion can comprise, but is not limited to, element-by-element addition, feature channel dimension splicing, element-by-element maximum value taking, neural network fusion and the like, and according to the embodiment, through time sequence fusion of the third feature images corresponding to each moment in a plurality of moments obtained after space fusion, space fusion and time sequence fusion are sequentially realized in a fusion branch network of a first network model, internal branch realization in the first network model is achieved, multi-frame image frames acquired from a plurality of visual angles at a plurality of moments are fused into a second feature image under a bird's-eye view image coordinate system, so that the first network model can realize space and time sequence fusion of the feature images by end-to-end learning without performing space and time sequence fusion after external post-processing after the network model, the complexity of post-processing is reduced, and ambiguity problems caused by fusion in post-processing are reduced.

Optionally, based on the above embodiment, step 602 may further include:

b1, determining at least one homography transformation matrix between a first feature map corresponding to the reference feature map and a first feature map corresponding to the at least one third feature map based on an inter-frame motion transformation matrix corresponding to the vehicle-mounted camera at the latest moment and at least one moment corresponding to the at least one third feature map.

Optionally, since the vehicle-mounted camera may change the position of the vehicle when the vehicle-mounted camera collects images at different times, the camera position changes accordingly, and the homography transformation matrix H between the two frames of images before and after any view angle is calculated based on the fact that the vehicle moves between frames (each time corresponds to one frame of image frame) and the distance d between the vehicle-mounted camera (for example, the vehicle-mounted front view camera) at the vehicle-mounted set position from the xy plane of the vehicle coordinate system is known (belongs to the camera external parameters, can be determined through calculation by the algorithm in the prior art or can be determined through the information collected by the sensor, etc.). Assuming that the rotation from time t to time t-1 is R and the translation is m, the homography transformation matrix H can be determined based on the following equation (4):

Wherein n= [0, 1] ^T is the normal direction of the xy plane of the vehicle coordinate system, K represents the internal reference matrix of the vehicle-mounted camera, and d represents the height of the vehicle-mounted camera from the xy plane of the vehicle coordinate system.

B2, determining at least one conversion matrix between the reference feature map and the at least one third feature map based on the at least one homography transformation matrix and the third transformation matrix.

In this embodiment, a transformation matrix is determined for each third feature map by combining the third transformation matrix of the spatial transformation and the homography transformation matrix corresponding to each third feature map, and optionally, the transformation matrix T _temp may be determined based on the following formula (5):

And b3, reconstructing each third feature map in the at least one third feature map based on each conversion matrix in the at least one conversion matrix respectively to obtain at least one fourth feature map.

In this embodiment, taking the third feature map at the time T-1 as an example, features at the (U _t-1,V_t-1) position after spatial fusion at the time T-1 may be mapped to the (U _t,V_t) of the spatial fusion feature at the time T through the transformation matrix T _temp, so as to obtain the reconstructed T-time feature F' _t, that is, the reconstruction of the third feature map at the time T-1 is realized through the transformation matrix, so as to obtain the fourth feature map, and so on, the third feature map at each time is respectively reconstructed to obtain at least one fourth feature map, and by the reconstruction of the feature maps, the feature maps corresponding to different times can be fused, and by executing the spatial fusion, the time sequence fusion is performed, so that the number of the feature maps processed by the time sequence fusion is reduced, the difficulty of the time sequence fusion is reduced, and the time sequence fusion is realized by combining the third transformation matrix in the spatial fusion, so that the repeated utilization rate of parameters is improved, and the fusion efficiency is improved.

As shown in fig. 7, step 204 may include the following steps, based on the embodiment shown in fig. 2, described above:

and step 2041, performing decoding processing on the second feature map through a decoding branch network in the first network model to obtain a decoded second feature map.

Alternatively, the decoding branch network in the present embodiment may be understood with reference to the decoding branch network 103 in the embodiment provided in fig. 1.

Step 2042, identifying the decoded second feature map based on the second network model, and determining a sensing result corresponding to the preset sensing task.

In this embodiment, the second feature map may be mapped to the feature space required by the preset sensing task through the decoding branch network, so that the network model corresponding to the subsequent preset sensing task may directly process the mapped feature map, thereby reducing other processes of intermediate processing, realizing that the network model of the preset sensing task is directly connected with the first network model, and realizing joint learning of the network model, and improving accuracy of the sensing result of the preset sensing task through joint learning.

Any of the vehicle-mounted visual perception methods provided by the embodiments of the present disclosure may be performed by any suitable device having data processing capabilities, including, but not limited to, terminal devices, servers, and the like. Or any of the vehicle-mounted visual perception methods provided by the embodiments of the present disclosure may be executed by a processor, such as the processor executing any of the vehicle-mounted visual perception methods mentioned by the embodiments of the present disclosure by invoking corresponding instructions stored in a memory. And will not be described in detail below.

Exemplary apparatus

Fig. 8 is a schematic structural diagram of an in-vehicle visual perception device according to an exemplary embodiment of the present disclosure. As shown in fig. 8, the apparatus provided in this embodiment includes:

The image acquisition module 81 is configured to acquire images at a plurality of continuous moments by using a plurality of vehicle-mounted cameras disposed at preset positions on a vehicle, so as to obtain a plurality of image frame sets.

The encoding module 82 is configured to perform feature extraction on image frames included in the plurality of image frame sets obtained by the image acquisition module 81 through an encoding branch network in the first network model, so as to obtain a plurality of first feature maps.

And the fusion module 83 is configured to perform spatial fusion and temporal fusion on the plurality of first feature maps determined by the encoding module 82 through a fusion branch network in the first network model, so as to obtain a second feature map under the aerial view image coordinate system.

The sensing module 84 is configured to identify the second feature map determined by the fusion module 83 based on a network model corresponding to the preset sensing task, so as to obtain a sensing result corresponding to the preset sensing task.

According to the vehicle-mounted visual perception device provided by the embodiment of the disclosure, through the spatial fusion and the time sequence fusion inside the first neural network, the end-to-end learning of the neural network is realized, and the post-processing fusion is not needed, so that the complexity degree of the spatial fusion and the time sequence fusion of images during the post-processing can be effectively avoided, and the situation that the same target is mistakenly identified as a plurality of targets in the post-processing can be avoided.

Fig. 9 is a schematic structural view of an in-vehicle visual perception device provided in another exemplary embodiment of the present disclosure. As shown in fig. 9, in the apparatus provided in this embodiment, the fusion module 83 includes:

And the space fusion unit 831 is configured to perform space fusion on the plurality of first feature maps corresponding to the time for each of the plurality of time points, so as to obtain a plurality of third feature maps in the aerial view image coordinate system.

Wherein each third feature map corresponds to a time instant.

The timing fusion unit 832 is configured to perform timing fusion on the plurality of third feature maps to obtain a second feature map.

Optionally, the spatial fusion unit 831 is specifically configured to perform homography transformation on each of the plurality of first feature maps to obtain a plurality of transformed feature maps under the aerial view image coordinate system, and perform point-by-point fusion on the plurality of transformed feature maps to obtain a third feature map under the aerial view image coordinate system.

Optionally, when performing homography on each first feature map in the plurality of first feature maps to obtain a plurality of transformation feature maps under the aerial view image coordinate system, the spatial fusion unit 831 is configured to determine a first transformation matrix from the vehicle coordinate system corresponding to the first feature map to the camera coordinate system based on an internal reference matrix and an external reference matrix of the vehicle camera corresponding to each first feature map, determine a second transformation matrix from the aerial view image coordinate system to the preset plane of the vehicle coordinate system based on a sensing range on a preset plane of the vehicle coordinate system and a scaling and translation distance of the aerial view image coordinate system, determine a third transformation matrix from the aerial view image coordinate system to the image coordinate system corresponding to the vehicle camera based on the first transformation matrix, the second transformation matrix and the internal reference matrix of the vehicle camera, and transform the first feature map based on the third transformation matrix to obtain the transformation feature map.

Optionally, the spatial fusion unit 831 is configured to determine, based on the first transformation matrix, the second transformation matrix, and an internal reference matrix of the vehicle-mounted camera, a fourth transformation matrix from a preset plane of the vehicle coordinate system to the camera coordinate system based on the first transformation matrix when determining a third transformation matrix from the aerial view image coordinate system to the image coordinate system corresponding to the vehicle-mounted camera, determine, based on the fourth transformation matrix and the second transformation matrix, a fifth transformation matrix from the aerial view image coordinate system to the camera coordinate system, and determine, based on the fifth transformation matrix and the internal reference matrix of the vehicle-mounted camera.

Optionally, the time sequence fusion unit 832 is specifically configured to use a third feature map corresponding to a latest time among multiple times as a reference feature map, reconstruct each third feature map in at least one third feature map to obtain at least one fourth feature map, and perform point-by-point fusion on the reference feature map and the at least one fourth feature map to obtain a second feature map.

Optionally, the timing fusion unit 832 is configured to determine at least one homography transformation matrix between a first feature map corresponding to a reference feature map and a first feature map corresponding to at least one third feature map based on an inter-frame motion transformation matrix corresponding to a latest time and at least one time corresponding to the at least one third feature map of the vehicle camera when reconstructing each third feature map of the at least one third feature map to obtain at least one fourth feature map, determine at least one transformation matrix between the reference feature map and the at least one third feature map based on the at least one homography transformation matrix and the third transformation matrix, and reconstruct each third feature map of the at least one third feature map based on each transformation matrix of the at least one transformation matrix to obtain at least one fourth feature map.

In some alternative embodiments, the perception module 84 includes:

The decoding unit 841 is configured to perform decoding processing on the second feature map through a decoding branch network in the first network model, so as to obtain a decoded second feature map.

The feature recognition unit 842 is configured to identify the decoded second feature map based on the second network model, and determine a sensing result corresponding to the preset sensing task.

Exemplary electronic device

Next, an electronic device according to an embodiment of the present disclosure is described with reference to fig. 10. The electronic device may be either or both of the first device 100 and the second device 200, or a stand-alone device independent thereof, which may communicate with the first device and the second device to receive the acquired input signals therefrom.

Fig. 10 illustrates a block diagram of an electronic device according to an embodiment of the disclosure.

As shown in fig. 10, the electronic device 10 includes one or more processors 11 and a memory 12.

The processor 11 may be a Central Processing Unit (CPU) or other form of processing unit having data processing and/or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

Memory 12 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random Access Memory (RAM) and/or cache memory (cache), and the like. The non-volatile memory may include, for example, read Only Memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on the computer readable storage medium that can be executed by the processor 11 to implement the in-vehicle visual perception methods and/or other desired functions of the various embodiments of the present disclosure described above. Various contents such as an input signal, a signal component, a noise component, and the like may also be stored in the computer-readable storage medium.

In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected by a bus system and/or other form of connection mechanism (not shown).

For example, when the electronic device is the first device 100 or the second device 200, the input means 13 may be a microphone or a microphone array as described above for capturing an input signal of a sound source. When the electronic device is a stand-alone device, the input means 13 may be a communication network connector for receiving the acquired input signals from the first device 100 and the second device 200.

In addition, the input device 13 may also include, for example, a keyboard, a mouse, and the like.

The output device 14 may output various information to the outside, including the determined distance information, direction information, and the like. The output means 14 may include, for example, a display, speakers, a printer, and a communication network and remote output devices connected thereto, etc.

Of course, only some of the components of the electronic device 10 that are relevant to the present disclosure are shown in fig. 10, with components such as buses, input/output interfaces, etc. omitted for simplicity. In addition, the electronic device 10 may include any other suitable components depending on the particular application.

Exemplary computer program product and computer readable storage Medium

In addition to the methods and apparatus described above, embodiments of the present disclosure may also be a computer program product comprising computer program instructions which, when executed by a processor, cause the processor to perform the steps in the in-vehicle visual perception method according to the various embodiments of the present disclosure described in the "exemplary methods" section of this specification.

The computer program product may write program code for performing the operations of embodiments of the present disclosure in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server.

Furthermore, embodiments of the present disclosure may also be a computer-readable storage medium, having stored thereon computer program instructions, which when executed by a processor, cause the processor to perform the steps in the in-vehicle visual perception method according to various embodiments of the present disclosure described in the above "exemplary methods" section of the present disclosure.

The computer readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of a readable storage medium include an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The basic principles of the present disclosure have been described above in connection with specific embodiments, but it should be noted that the advantages, benefits, effects, etc. mentioned in the present disclosure are merely examples and not limiting, and these advantages, benefits, effects, etc. are not to be considered as necessarily possessed by the various embodiments of the present disclosure. Furthermore, the specific details disclosed herein are for purposes of illustration and understanding only, and are not intended to be limiting, since the disclosure is not necessarily limited to practice with the specific details described.

In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different manner from other embodiments, so that the same or similar parts between the embodiments are mutually referred to. For system embodiments, the description is relatively simple as it essentially corresponds to method embodiments, and reference should be made to the description of method embodiments for relevant points.

The block diagrams of the devices, apparatuses, devices, systems referred to in this disclosure are merely illustrative examples and are not intended to require or imply that the connections, arrangements, configurations must be made in the manner shown in the block diagrams. As will be appreciated by one of skill in the art, the devices, apparatuses, devices, systems may be connected, arranged, configured in any manner. Words such as "including," "comprising," "having," and the like are words of openness and mean "including but not limited to," and are used interchangeably therewith. The terms "or" and "as used herein refer to and are used interchangeably with the term" and/or "unless the context clearly indicates otherwise. The term "such as" as used herein refers to, and is used interchangeably with, the phrase "such as, but not limited to.

The methods and apparatus of the present disclosure may be implemented in a number of ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, firmware. The above-described sequence of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the sequence specifically described above unless specifically stated otherwise. Furthermore, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, the programs including machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

It is also noted that in the apparatus, devices and methods of the present disclosure, components or steps may be disassembled and/or assembled. Such decomposition and/or recombination should be considered equivalent to the present disclosure.

The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the disclosure to the form disclosed herein. Although a number of example aspects and embodiments have been discussed above, a person of ordinary skill in the art will recognize certain variations, modifications, alterations, additions, and subcombinations thereof.

Claims

1. A vehicle-mounted visual perception method, comprising:

Identifying the second feature map based on a network model corresponding to a preset sensing task, and determining a sensing result corresponding to the preset sensing task;

performing spatial fusion and time sequence fusion on the plurality of first feature images through a fusion branch network in the first network model to obtain a second feature image under a bird's-eye view image coordinate system, wherein the method comprises the following steps:

Performing space fusion on a plurality of first feature images corresponding to each moment in the plurality of moments to obtain a plurality of third feature images under a bird's eye view image coordinate system, wherein each third feature image corresponds to one moment;

taking a third characteristic diagram corresponding to the latest moment in the plurality of moments as a reference characteristic diagram;

Reconstructing each third characteristic diagram in at least one third characteristic diagram to obtain at least one fourth characteristic diagram;

and carrying out point-by-point fusion on the reference feature map and the at least one fourth feature map to obtain the second feature map.

2. The method of claim 1, wherein the performing spatial fusion on the plurality of first feature maps corresponding to the time instants to obtain a plurality of third feature maps in a bird's-eye view image coordinate system includes:

respectively executing homography transformation on each first feature map in the plurality of first feature maps to obtain a plurality of transformation feature maps under a bird's-eye view image coordinate system;

And carrying out point-by-point fusion on the plurality of transformation feature images to obtain the third feature image under the aerial view image coordinate system.

3. The method of claim 2, wherein the performing homography on each of the plurality of first feature maps to obtain a plurality of transformed feature maps in a bird's-eye image coordinate system includes:

determining a first transformation matrix from a vehicle coordinate system corresponding to the first feature map to a camera coordinate system based on an internal reference matrix and an external reference matrix of the vehicle-mounted camera corresponding to each first feature map;

determining a second transformation matrix from the aerial view image coordinate system to the preset plane of the self-vehicle coordinate system based on the perception range on the preset plane of the self-vehicle coordinate system and the scaling and translation distance of the aerial view image coordinate system;

determining a third transformation matrix from the aerial view image coordinate system to an image coordinate system corresponding to the vehicle-mounted camera based on the first transformation matrix, the second transformation matrix and an internal reference matrix of the vehicle-mounted camera;

And transforming the first feature map based on the third transformation matrix to obtain the transformed feature map.

4. The method of claim 3, wherein the determining a third transformation matrix of the aerial image coordinate system to the image coordinate system corresponding to the vehicle-mounted camera based on the first transformation matrix, the second transformation matrix, and the internal reference matrix of the vehicle-mounted camera comprises:

Determining a fourth transformation matrix from a preset plane of the vehicle coordinate system to a camera coordinate system based on the first transformation matrix;

Determining a fifth transformation matrix of the aerial image coordinate system to the camera coordinate system based on the fourth transformation matrix and the second transformation matrix;

and determining the third transformation matrix based on the fifth transformation matrix and the internal reference matrix of the vehicle-mounted camera.

5. The method of claim 4, wherein reconstructing each of the at least one third feature map to obtain at least one fourth feature map includes:

determining at least one homography transformation matrix between a first feature map corresponding to the reference feature map and a first feature map corresponding to the at least one third feature map based on an inter-frame motion transformation matrix corresponding to the vehicle-mounted camera at least one moment corresponding to the at least one third feature map at the latest moment;

Determining at least one transformation matrix between the reference feature map and the at least one third feature map based on the at least one homography transformation matrix and the third transformation matrix;

And reconstructing each third feature map in the at least one third feature map based on each conversion matrix in the at least one conversion matrix respectively to obtain the at least one fourth feature map.

6. The method according to any one of claims 1-5, wherein the identifying the second feature map based on the second network model corresponding to the preset sensing task, and determining the sensing result corresponding to the preset sensing task, includes:

Decoding the second feature map through a decoding branch network in the first network model to obtain a decoded second feature map;

And identifying the decoded second feature map based on the second network model, and determining a perception result corresponding to the preset perception task.

7. An in-vehicle visual perception apparatus comprising:

The system comprises an image acquisition module, a display module and a display module, wherein the image acquisition module is used for acquiring images at a plurality of continuous moments through a plurality of vehicle-mounted cameras arranged at preset positions on a vehicle to obtain a plurality of image frame sets, wherein each image frame set comprises a plurality of frame image frames acquired based on the same vehicle-mounted camera, and each vehicle-mounted camera corresponds to one frame image frame at each moment;

the sensing module is used for identifying the second feature map determined by the fusion module based on a network model corresponding to a preset sensing task to obtain a sensing result corresponding to the preset sensing task;

The fusion module comprises a space fusion unit, a fusion unit and a fusion unit, wherein the space fusion unit is used for executing space fusion on a plurality of first feature images corresponding to each moment in the plurality of moments to obtain a plurality of third feature images under a bird's-eye view image coordinate system, and each third feature image corresponds to one moment;

the time sequence fusion unit is used for taking a third characteristic image corresponding to the latest moment in the plurality of moments as a reference characteristic image, reconstructing each third characteristic image in at least one third characteristic image to obtain at least one fourth characteristic image, and carrying out point-by-point fusion on the reference characteristic image and the at least one fourth characteristic image to obtain the second characteristic image.

8. A computer readable storage medium storing a computer program for executing the in-vehicle visual perception method of any one of the preceding claims 1-6.

9. An electronic device, the electronic device comprising:

A processor;

a memory for storing the processor-executable instructions;

The processor is configured to read the executable instructions from the memory and execute the instructions to implement the vehicle-mounted visual perception method according to any one of the preceding claims 1-6.