CN118870191A

CN118870191A - Video anti-shake processing method, device, computer equipment and readable storage medium

Info

Publication number: CN118870191A
Application number: CN202310481483.0A
Authority: CN
Inventors: 刘宸希; 袁文亮; 桑标; 谢亮
Original assignee: Insta360 Innovation Technology Co Ltd
Current assignee: Insta360 Innovation Technology Co Ltd
Priority date: 2023-04-27
Filing date: 2023-04-27
Publication date: 2024-10-29

Abstract

The application relates to a video anti-shake processing method, a video anti-shake processing device, computer equipment and a readable storage medium. The method comprises the following steps: acquiring three-dimensional rotation information of an initial video, acquiring global rotation motion information of the initial video according to the three-dimensional rotation information, and performing motion smoothing on the global rotation motion information to generate an anti-shake video of the initial video. By adopting the method, the three-dimensional rotation information of the initial video can be acquired, and then the three-dimensional rotation information of the initial video is processed to realize the three-dimensional anti-shake processing of the initial video, so that the accuracy of the anti-shake processing and the anti-shake effect of the video can be improved, and the stability and quality of pictures in the initial video can be effectively improved.

Description

Video anti-shake processing method and device, computer equipment and readable storage medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to a video anti-shake processing method, a video anti-shake processing device, a computer device, and a readable storage medium.

Background

With the development of image technology, capturing images or videos by an image capturing device such as a camera on an electronic device has been widely used.

Taking the shot video as an example, if the shooting equipment shakes in the process of shooting the video, the shot picture can shake, so that the shot video picture quality is poor. Based on this, the related art often guarantees the quality of a photographed video picture by performing anti-shake processing on the video.

However, the related art has a problem in that the video anti-shake effect is poor.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a video anti-shake processing method, apparatus, computer device, and readable storage medium, which can improve the anti-shake effect of an initial video.

In a first aspect, an embodiment of the present application provides a video anti-shake processing method, where the method includes:

Acquiring three-dimensional rotation information of an initial video;

Acquiring global rotation motion information of an initial video according to the three-dimensional rotation information;

and performing motion smoothing processing on the global rotation motion information to generate an anti-shake video of the initial video.

In one embodiment, acquiring three-dimensional rotation information of an initial video includes:

acquiring a matching point pair set of every two adjacent video frames in an initial video;

for any two adjacent video frames, determining a target rotation matrix between the two adjacent video frames according to the matching point pair sets of the two adjacent video frames;

and determining the three-dimensional rotation information of the initial video according to the target rotation matrix between every two adjacent video frames.

In one embodiment, obtaining a set of matching point pairs for each two adjacent video frames in an initial video includes:

For any two adjacent video frames, extracting feature points in a previous video frame in the two adjacent video frames to obtain a plurality of original feature points;

performing optical flow tracking on a next video frame in two adjacent video frames according to each original feature point to obtain a plurality of tracking feature points in the next video frame;

Carrying out reflection flow tracking on the previous video frame according to each tracking characteristic point to obtain a plurality of anti-tracking characteristic points in the previous video frame;

and determining a matching point pair set of two adjacent video frames according to each anti-tracking characteristic point.

In one embodiment, determining the matching point pair set of two adjacent video frames according to each anti-tracking feature point includes:

Acquiring the interval distance between each anti-tracking feature point and the corresponding original feature point;

determining the anti-tracking feature points with the interval distance smaller than a preset distance threshold as the effective anti-tracking feature points for optical flow tracking;

and determining a matching point pair set of two adjacent video frames according to all the anti-tracking feature points for which optical flow tracking is effective.

In one embodiment, determining a target rotation matrix between two adjacent video frames from a set of matching point pairs of the two adjacent video frames includes:

acquiring optical flow tracking results of two adjacent video frames according to the matching point pair sets of the two adjacent video frames;

If the optical flow tracking result of the two adjacent video frames is that the optical flow tracking is invalid, determining the identity matrix as a target rotation matrix between the two adjacent video frames;

if the optical flow tracking result of the two adjacent video frames is that the optical flow tracking is effective, determining a target rotation matrix between the two adjacent video frames according to the shooting states corresponding to the two adjacent video frames.

In one embodiment, determining the target rotation matrix between two adjacent video frames according to the shooting states corresponding to the two adjacent video frames includes:

If the shooting state corresponding to the two adjacent video frames is a non-stationary shooting state, acquiring an essential matrix of a matching point pair set of the two adjacent video frames, and acquiring a target rotation matrix between the two adjacent video frames based on the essential matrix of the matching point pair set;

And if the shooting state corresponding to the two adjacent video frames is the static shooting state, determining the identity matrix as a target rotation matrix between the two adjacent video frames.

In one embodiment, obtaining a target rotation matrix between two adjacent video frames based on an essential matrix of a matching point pair set includes:

Singular value decomposition is carried out on the essential matrix of the matching point pair set, and an initial rotation matrix, three-dimensional translation information and a depth information set between two adjacent video frames are obtained;

and acquiring a target rotation matrix between two adjacent video frames according to the initial rotation matrix, the three-dimensional translation information and the depth information set.

In one embodiment, obtaining a target rotation matrix between two adjacent video frames according to the initial rotation matrix, the three-dimensional translation information and the depth information set includes:

Screening a plurality of target matching point pairs from all candidate matching point pairs in the matching point pair set according to the depth information set;

and determining a target rotation matrix between two adjacent video frames according to the initial rotation matrix, the three-dimensional translation information and each target matching point pair.

In one embodiment, determining the target rotation matrix between two adjacent video frames according to the initial rotation matrix, the three-dimensional translation information and each target matching point pair comprises:

Mapping each target matching point pair to a shooting equipment coordinate system through shooting equipment internal parameters to obtain a mapping coordinate;

constructing a nonlinear least square model through the initial rotation matrix, the three-dimensional translation information and the mapping coordinates;

and solving a nonlinear least square model to obtain a target rotation matrix between two adjacent video frames.

In one embodiment, the method further comprises:

Determining candidate matching point pairs which meet preset rest point conditions in the matching point pair sets as rest point pairs;

if the total number of all the stationary point pairs in the matching point pair set is larger than a preset number threshold, determining that the shooting states corresponding to the two adjacent video frames are stationary shooting states, otherwise, determining that the shooting states corresponding to the two adjacent video frames are non-stationary shooting states.

In one embodiment, acquiring global rotational motion information of an initial video according to three-dimensional rotational information includes:

Acquiring a global rotation motion matrix of each video frame in the initial video according to the three-dimensional rotation information of the initial video;

and determining global rotation motion information of the initial video according to the global rotation motion matrix of each video frame.

In one embodiment, performing motion smoothing on global rotational motion information to generate an anti-shake video of an initial video includes:

performing motion smoothing processing on the global rotation motion information to obtain motion smoothing data of the initial video;

determining target motion smoothing data of the initial video according to motion smoothing weights of video frames in the initial video, global rotation motion information of the initial video and the motion smoothing data;

and generating an anti-shake video according to the target motion smooth data and the initial video.

In one embodiment, generating an anti-shake video from target motion smoothing data and an initial video includes:

determining the rotation compensation amount of the initial video according to the target motion smoothing data and the global rotation motion information;

And mapping the pixel points in each video frame according to the rotation compensation amount of the initial video to generate an anti-shake video.

In a second aspect, an embodiment of the present application provides a video anti-shake processing apparatus, including:

the rotation information acquisition module is used for acquiring three-dimensional rotation information of the initial video;

The global information acquisition module is used for acquiring global rotation motion information of the initial video according to the three-dimensional rotation information;

And the smoothing processing module is used for carrying out motion smoothing processing on the global rotation motion information to generate an anti-shake video of the initial video.

In a third aspect, an embodiment of the present application further provides a computer device, where the computer device includes a memory and a processor, the memory storing a computer program, and the processor when executing the computer program implements the steps of:

Acquiring three-dimensional rotation information of an initial video;

In a fourth aspect, embodiments of the present application also provide a computer readable storage medium having a computer program stored thereon, the computer program when executed by a processor performing the steps of:

Acquiring three-dimensional rotation information of an initial video;

In a fifth aspect, embodiments of the present application also provide a computer program product comprising a computer program which, when executed by a processor, performs the steps of:

Acquiring three-dimensional rotation information of an initial video;

The video anti-shake processing method, the device, the computer equipment and the readable storage medium provided by the embodiment of the application comprise the following steps: acquiring three-dimensional rotation information of an initial video, acquiring global rotation motion information of the initial video according to the three-dimensional rotation information, and performing motion smoothing on the global rotation motion information to generate an anti-shake video of the initial video; by adopting the method, the three-dimensional rotation information of the initial video can be acquired, and then the three-dimensional rotation information of the initial video is processed to realize the three-dimensional anti-shake processing of the initial video, so that the accuracy of the anti-shake processing and the anti-shake effect of the video can be improved.

Drawings

FIG. 1 is a block diagram of a video anti-shake processing system according to an embodiment;

FIG. 2 is a flow chart of an embodiment of a video anti-shake processing method;

FIG. 3 is a flowchart illustrating a video anti-shake processing method according to another embodiment;

FIG. 4 is a flowchart of a video anti-shake processing method according to another embodiment;

FIG. 5 is a flowchart of a video anti-shake processing method according to another embodiment;

FIG. 6 is a flowchart of a video anti-shake processing method according to another embodiment;

FIG. 7 is a flowchart of a video anti-shake processing method according to another embodiment;

FIG. 8 is a flowchart of a video anti-shake processing method according to another embodiment;

FIG. 9 is a flowchart of a video anti-shake processing method according to another embodiment;

FIG. 10 is a flowchart of a video anti-shake processing method according to another embodiment;

FIG. 11 is a flowchart of a video anti-shake processing method according to another embodiment;

FIG. 12 is a flowchart of a video anti-shake processing method according to another embodiment;

FIG. 13 is a flowchart of a video anti-shake processing method according to another embodiment;

FIG. 14 is a block diagram showing a configuration of an apparatus for anti-shake processing of video according to an embodiment;

fig. 15 is an internal structural view of a computer device in one embodiment.

Detailed Description

The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.

In the new media field, shooting images or videos through the self-contained video function of intelligent equipment gradually becomes a new way for people to record life, spread information and entertain and relax. Taking the shot video as an example, if the shooting equipment shakes in the process of shooting the video, the shot picture can shake, so that the shot video picture quality is poor. Based on this, the related art often guarantees the quality of a photographed video picture by performing anti-shake processing on the video. However, the related art has a problem in that the video anti-shake effect is poor. Based on the above, the embodiment of the application provides a video anti-shake processing method, which can improve the anti-shake effect of an initial video.

The video anti-shake processing method provided by the embodiment of the application can be applied to the video anti-shake processing system shown in fig. 1, wherein the video anti-shake processing system comprises shooting equipment and computer equipment, the shooting equipment and the computer equipment are in communication connection, and the communication mode can be Bluetooth, wi-Fi, mobile network connection and the like. Alternatively, the photographing apparatus may be a cellular phone or a camera capable of photographing any type of video (e.g., panoramic video, normal video, etc.); the computer device may be implemented by an independent server or a server cluster formed by a plurality of servers, or may be, but not limited to, various personal computers, notebook computers, electronic mobile phones, and tablet computers, and the specific forms of the photographing device and the computer device are not limited in this embodiment; in fig. 1, a computer device is used as a server, and a shooting device is used as a mobile phone.

The shooting device can send the collected initial video to the computer device connected with the shooting device, and correspondingly, after the computer device receives the initial video, the computer device carries out anti-shake processing on the initial video to generate an anti-shake video. In the following embodiments, specific procedures of the video anti-shake processing method will be specifically described, and specific procedures of the video anti-shake processing method will be described with an execution subject as a computer device.

As shown in fig. 2, a flow chart of a video anti-shake processing method according to an embodiment of the present application may include the following steps:

s100, acquiring three-dimensional rotation information of the initial video.

Specifically, the computer device may acquire a three-dimensional rotation matrix between each two adjacent video frames in the initial video, and then determine three-dimensional rotation information of the initial video according to the three-dimensional rotation matrix between each two adjacent video frames in the initial video. Wherein the three-dimensional rotation matrix between two adjacent video frames can be represented by a 3*3 square matrix.

In the embodiment of the present application, the initial video may be a video shot by the shooting device in a default shooting mode, or may be a video shot by the shooting device in a specific mirror mode. Correspondingly, the types of the initial video may be fixed delay video, mobile delay video, panoramic delay video, etc., which is not limited in the embodiment of the present application. Alternatively, the above-mentioned mirror mode may be a forward lens, a horizontal lens, a surrounding marker, or the like.

Here, it should be noted that, the initial video may be a video directly shot by the shooting device, where in this case, the total duration of the initial video is equal to the shooting duration corresponding to the shooting device, and in addition, the initial video may also be a video obtained by extracting frames from a video directly shot by the shooting device and fusing all the extracted video frames, where in this case, the total duration of the initial video is less than the shooting duration corresponding to the shooting device.

And S200, acquiring global rotation motion information of the initial video according to the three-dimensional rotation information.

The computer equipment can perform arithmetic operation processing on the three-dimensional rotation information of the initial video to obtain global rotation motion information of the initial video. Alternatively, the arithmetic operation may be at least one of an addition operation, a subtraction operation, a multiplication operation, a division operation, a logarithmic operation, and an exponential operation.

In an embodiment of the present application, the global rotational motion information of the initial video may include a global rotational motion matrix of each video frame in the initial video.

And S300, performing motion smoothing processing on the global rotation motion information to generate an anti-shake video of the initial video.

In practical application, the computer device may perform motion smoothing processing on the global rotational motion information by using a motion smoothing algorithm to directly generate an anti-shake video of the initial video, or may perform motion smoothing processing on the global rotational motion information by using a motion smoothing algorithm to obtain motion smoothing data, and then process the motion smoothing data to generate an anti-shake video of the initial video.

Alternatively, the motion smoothing algorithm may be a moving window least squares polynomial smoothing method, a Kernel smoothing method, an additive smoothing method, a depth smoothing method, or the like.

It should be noted that, in the video anti-shake processing method, after the anti-shake processing is performed on the initial video with larger parallax and larger scene depth, the effect of generating the anti-shake video is more obvious than that of the initial video. Alternatively, parallax may be understood as that motion floating is larger when the photographing apparatus photographs two adjacent video frames in the process of photographing video, or may be understood as that lens pulling speed is faster when the photographing apparatus photographs two adjacent video frames, and correspondingly, the difference between the two adjacent video frames is larger; scene depth can be understood as the distance between different objects and the lens of the photographing device when photographing different objects, i.e. the depth of field is different, and correspondingly, the difference between two adjacent video frames is also larger.

According to the technical scheme, three-dimensional rotation information of an initial video is obtained, global rotation motion information of the initial video is obtained according to the three-dimensional rotation information, motion smoothing processing is carried out on the global rotation motion information, and anti-shake video of the initial video is generated; by adopting the method, the three-dimensional rotation information of the initial video can be acquired, and then the three-dimensional rotation information of the initial video is processed to realize the three-dimensional anti-shake processing of the initial video, so that the accuracy of the anti-shake processing and the anti-shake effect of the video can be improved, the stability and the quality of pictures in the initial video can be effectively improved, the problems of causing the distortion and the distortion of the pictures of the video can be avoided, and the appearance of a user can be improved; meanwhile, the method does not need to be manually participated, can also save the labor cost required by anti-shake processing, can reduce the manual participatory error, and can further improve the accuracy of the video anti-shake result, the speed and the efficiency of video anti-shake; in addition, the method can perform anti-shake processing on various types of initial videos, and has a wide application range, namely, the robustness of the anti-shake processing is high; furthermore, the method can consider the actual motion state of each video frame in the space in the initial video through the three-dimensional anti-shake processing, so that the anti-shake processing is more real, and the anti-shake processing effect on the initial video with larger parallax and scene depth is better.

The procedure of acquiring the three-dimensional rotation information of the initial video described above will be described below. In an embodiment, as shown in fig. 3, the step in S100 may be implemented by:

s101, acquiring a matching point pair set of every two adjacent video frames in the initial video.

Specifically, the computer device may acquire each video frame in the initial video, then, for any two adjacent video frames, perform feature point extraction processing on the two adjacent video frames by using a feature point extraction algorithm to obtain feature points in the two adjacent video frames, then perform feature point matching processing on the feature points in the two adjacent video frames to obtain matching point pairs of the two adjacent video frames, and then obtain a matching point pair set of the two adjacent video frames according to all the matching point pairs of the two adjacent video frames.

Alternatively, the feature point extraction algorithm may be a gray-based corner detection algorithm, a fast corner detection algorithm, or a deep learning-based corner detection algorithm, or the like. It should be noted that the total number of the matching point pairs obtained by the feature point matching process may be less than or equal to the total number of feature points on any one of the two adjacent video frames.

S102, for any two adjacent video frames, determining a target rotation matrix between the two adjacent video frames according to the matching point pair sets of the two adjacent video frames.

In practical application, for any two adjacent video frames, the computer device may pre-train an algorithm model, and then input pixel coordinates of each matching point in the matching point pair set of the two adjacent video frames, where each matching point is on a corresponding video frame, into the algorithm model, where the algorithm model outputs a target rotation matrix between the two adjacent video frames.

In addition, the computer equipment can also perform rotation transformation processing on pixel coordinates of each matching point in the set on the corresponding video frame according to the matching points of the two adjacent video frames to obtain a target rotation matrix between the two adjacent video frames. Alternatively, the target rotation matrix may represent a rotation relationship between two adjacent video frames in a three-dimensional space.

S103, determining three-dimensional rotation information of the initial video according to the target rotation matrix between every two adjacent video frames.

In the embodiment of the application, the target rotation matrix of the first video frame in the initial video and the target rotation matrix between every two adjacent video frames can be determined as the three-dimensional rotation information of the initial video. Wherein, since the first video frame in the initial video is stationary with respect to the initial video before shooting, the target rotation matrix of the first video frame may be an identity matrix.

According to the technical scheme, a matching point pair set of each two adjacent video frames in an initial video is obtained, for any two adjacent video frames, a target rotation matrix between the two adjacent video frames is determined according to the matching point pair set of the two adjacent video frames, and three-dimensional rotation information of the initial video is determined according to the target rotation matrix between the two adjacent video frames; according to the method, the three-dimensional rotation information of the initial video can be determined through the matching point pair sets of every two adjacent video frames in the initial video, the processing process is simple, the three-dimensional rotation information can be determined without complex algorithms, the determination time of the three-dimensional rotation information can be shortened, and the determination speed of the three-dimensional rotation information can be improved.

The process of acquiring the matching point pair set of each two adjacent video frames in the initial video is described below. In an embodiment, as shown in fig. 4, the step in S101 may include:

s111, for any two adjacent video frames, extracting feature points in a previous video frame in the two adjacent video frames to obtain a plurality of original feature points.

In practical application, for any two adjacent video frames in the initial video, the computer device may extract feature points in a previous video frame in the two adjacent video frames by using a feature point extraction algorithm to obtain a plurality of feature points, i.e. a plurality of original feature points, of the previous video frame. In the embodiment of the present application, the feature point extraction algorithm may be a Shi-Tomas corner detection algorithm.

S121, optical flow tracking is carried out on the next video frame in the two adjacent video frames according to each original feature point, and a plurality of tracking feature points in the next video frame are obtained.

The computer device may perform optical flow tracking on a next video frame of the two adjacent video frames according to each original feature point by using an optical flow tracking method, so as to obtain a plurality of feature points in the next video frame, i.e. a plurality of tracking feature points.

Alternatively, the optical flow tracking method may be a sparse optical flow tracking method or a dense optical flow tracking method, which is not limited to the embodiment of the present application. In addition, the total number of the tracking feature points may be less than or equal to the total number of the original feature points.

S131, carrying out reflection flow tracking on the previous video frame according to each tracking feature point to obtain a plurality of anti-tracking feature points in the previous video frame.

Further, based on the plurality of tracking feature points in the subsequent video frame obtained in the previous step, the optical flow tracking method may be used to track the reflection flow of the previous video frame, so as to obtain a plurality of feature points in the previous video frame, that is, a plurality of anti-tracking feature points. Alternatively, the total number of the above-described anti-tracking feature points may be smaller than or equal to the total number of the tracking feature points.

S141, determining a matching point pair set of two adjacent video frames according to each anti-tracking feature point.

Specifically, the computer device may combine each anti-tracking feature point in the previous video frame with a corresponding tracking feature point to obtain a matching point pair, and determine all the matching point pairs as a matching point pair set of two adjacent video frames.

In addition, the computer device may further screen feature points satisfying a preset screening condition from the anti-tracking feature points in the previous video frame, then combine the feature points satisfying the preset condition with the corresponding tracking feature points to obtain matching point pairs, and determine all the matching point pairs as a matching point pair set of two adjacent video frames.

In one embodiment, as shown in fig. 5, the step of determining the matching point pair set of two adjacent video frames according to each anti-tracking feature point in S141 may be implemented as follows:

s1411, obtaining the interval distance between each anti-tracking feature point and the corresponding original feature point.

In practical application, each of the anti-tracking feature points on the previous video frame has a corresponding original feature point, but any of the anti-tracking feature points may or may not overlap with the corresponding original feature point. Naturally, for any of the anti-tracking feature points on the previous video frame, the computer device may employ a distance calculation method to calculate the separation distance between the anti-tracking feature point and the corresponding original feature point.

Alternatively, the distance calculation method may be a euclidean distance method, a manhattan distance method, a mahalanobis distance method, a chebyshev distance method, a minkowski distance method, or a barbus distance method, which is not limited to the embodiment of the present application.

S1412, determining the anti-tracking feature points with the interval distance smaller than the preset distance threshold as the anti-tracking feature points with the effective optical flow tracking.

Further, it may be determined whether the separation distance between the anti-tracking feature point and the corresponding original feature point is smaller than a preset distance threshold, if so, the anti-tracking feature point is determined to be a feature point with effective optical flow tracking, and if not, the anti-tracking feature point is determined to be a feature point with ineffective optical flow tracking.

S1413, determining a matching point pair set of two adjacent video frames according to all the anti-tracking feature points effective in optical flow tracking.

Based on all the anti-tracking feature points with effective optical flow tracking on the previous video frame obtained in the previous step, each anti-tracking feature point with effective optical flow tracking can be respectively combined with the corresponding tracking feature point on the next video frame to obtain a plurality of matching point pairs with effective optical flow tracking, and all the matching point pairs with effective optical flow tracking are combined together to obtain a matching point pair set of two adjacent video frames.

In the embodiment of the application, the computer equipment can calculate the boundary distance between each anti-tracking feature point which is effective in optical flow tracking on the previous video frame and the long side and the wide side of the previous video frame by adopting a distance calculation method, if the boundary distance between the anti-tracking feature point and the long side of the previous video frame is larger than a preset long side distance threshold value and the boundary distance between the anti-tracking feature point and the wide side of the previous video frame is larger than a preset wide side distance threshold value, the anti-tracking feature point is determined as a candidate anti-tracking feature point, and then each obtained candidate anti-tracking feature point is respectively combined with the corresponding tracking feature point on the next video frame to obtain a plurality of candidate matching point pairs, and all the candidate matching point pairs are combined together to obtain a matching point pair set of two adjacent video frames.

Optionally, the preset long-edge distance threshold may be equal to a preset wide-edge distance threshold, or may not be equal to a preset wide-edge distance threshold. The preset long-side distance threshold and the preset wide-side distance threshold can be user-defined and can be determined according to historical experience values. For example, the preset long-side distance threshold and the preset wide-side distance threshold may be equal to 5, 4, 6, and so on, and the distance in the embodiment of the present application may be understood as the number of pixels.

For example, if two adjacent video frames are M _i and M _i+1, respectively, the obtained matching point pair sets P of M _i and M _i+1 have M candidate matching point pairs, and correspondingly, the matching point pair sets P may be expressed as

Wherein, Represents the j-th candidate matching point pair (j is more than or equal to 1 and less than or equal to m) in P,Representing the pixel coordinates of the pixel point at M _i,Representing the pixel coordinates of the pixel point on M _i+1.

According to the technical scheme, for any two adjacent video frames, feature points in a previous video frame in the two adjacent video frames are extracted to obtain a plurality of original feature points, optical flow tracking is conducted on a next video frame in the two adjacent video frames according to each original feature point to obtain a plurality of tracking feature points in the next video frame, reflection flow tracking is conducted on the previous video frame according to each tracking feature point to obtain a plurality of anti-tracking feature points in the previous video frame, and a matching point pair set of the two adjacent video frames is determined according to each anti-tracking feature point; the method can adopt feature point extraction and optical flow tracking processing to determine the matching point pair set of each two adjacent video frames in the initial video, the process is simple, the implementation can be realized without complex algorithm, and the acquisition speed of the matching point pair set can be improved; in addition, the method has no complex algorithm, and can acquire the matching point pair sets of two adjacent video frames without the need of a user to learn too much expertise.

In some scenes, even if there are matching point pairs on two adjacent video frames, since the total number of matching point pairs is small for the two adjacent video frames, the two adjacent video frames cannot be determined as video frames for which optical flow tracking is effective, and based on this, a procedure of determining an optical flow tracking result of the two adjacent video frames and a procedure of determining an optimal rotation matrix between the two adjacent video frames from the optical flow tracking result of the two adjacent video frames will be described below. In an embodiment, as shown in fig. 6, the step of determining the target rotation matrix between two adjacent video frames according to the matching point pair set of two adjacent video frames in S102 may be implemented as follows:

S112, acquiring optical flow tracking results of two adjacent video frames according to the matching point pair sets of the two adjacent video frames.

Specifically, the computer device may obtain the total number of all the matching point pairs in the set of matching point pairs of the two adjacent video frames, determine whether the total number of all the matching point pairs is smaller than a preset first number threshold, if so, determine that the optical flow tracking result of the two adjacent video frames is invalid for optical flow tracking, if not, determine that the optical flow tracking result of the two adjacent video frames is valid for optical flow tracking, and if not, determine that the optical flow tracking result of the two adjacent video frames is valid for optical flow tracking.

In addition, the computer device may further obtain the total number of anti-tracking feature points on a previous video frame of the two adjacent video frames, if the total number of anti-tracking feature points on the previous video frame is smaller than a preset second number threshold, if the total number of anti-tracking feature points on the previous video frame is smaller than the preset second number threshold, the optical flow tracking result of the two adjacent video frames is determined to be optical flow tracking invalid, if the total number of anti-tracking feature points on the two adjacent video frames is not smaller than the preset second number threshold, the optical flow tracking result of the two adjacent video frames is determined to be optical flow tracking valid.

The preset first number threshold value and the preset second number threshold value may be equal, and may be user-defined, or may be determined according to a historical experience value.

S122, if the optical flow tracking result of two adjacent video frames is that the optical flow tracking is invalid, determining the identity matrix as a target rotation matrix between the two adjacent video frames; and if the optical flow tracking result of the two adjacent video frames is that the optical flow tracking is effective, determining a target rotation matrix between the two adjacent video frames according to the shooting states corresponding to the two adjacent video frames.

In the embodiment of the present application, if it is determined that the optical flow tracking result of two adjacent video frames is invalid, the identity matrix may be determined as the target rotation matrix between the two adjacent video frames. Here, the target rotation matrix is an optimal rotation matrix between two adjacent video frames.

Wherein, the identity matrix is also 3*3 square matrix, and for two adjacent video frames M _i and M _i+1, the target rotation matrix between two adjacent video framesEqual to I _3×3.

Meanwhile, if the optical flow tracking result of the two adjacent video frames is determined to be effective in optical flow tracking, the pixel coordinates of each matching point pair in the matching point pair set of the two adjacent video frames can be processed according to the shooting states corresponding to the two adjacent video frames shot by the shooting equipment, so that a target rotation matrix between the two adjacent video frames is obtained.

The computer device may train an algorithm model in advance, and then input the shooting states corresponding to shooting two adjacent video frames by the shooting device and pixel coordinates of each matching point pair in the matching point pair set of the two adjacent video frames into the algorithm model, where the algorithm model outputs a target rotation matrix between the two adjacent video frames.

According to the technical scheme, optical flow tracking results of two adjacent video frames are obtained according to a matching point pair set of the two adjacent video frames, if the optical flow tracking results of the two adjacent video frames are optical flow tracking invalidity, a unit matrix is determined to be a target rotation matrix between the two adjacent video frames, and if the optical flow tracking results of the two adjacent video frames are optical flow tracking invalidity, a target rotation matrix between the two adjacent video frames is determined according to shooting states corresponding to the two adjacent video frames; according to the method, optical flow tracking results of two adjacent video frames can be obtained according to the matching point pair sets of the two adjacent video frames, and the target rotation matrix between the two adjacent video frames is determined in a corresponding mode according to the optical flow tracking results of the two adjacent video frames, so that the accuracy of the finally obtained target rotation matrix is high.

The process of determining the target rotation matrix between two adjacent video frames according to the shooting states corresponding to the two adjacent video frames will be described below. In an embodiment, the step in S132 may include: if the shooting state corresponding to the two adjacent video frames is a non-stationary shooting state, acquiring an essential matrix of a matching point pair set of the two adjacent video frames, and acquiring a target rotation matrix between the two adjacent video frames based on the essential matrix of the matching point pair set; and if the shooting state corresponding to the two adjacent video frames is the static shooting state, determining the identity matrix as a target rotation matrix between the two adjacent video frames.

Specifically, the computer device may determine whether the shooting states corresponding to the two adjacent video frames are still shooting states, and if it is determined that the shooting states corresponding to the two adjacent video frames are still shooting states, determine the identity matrix as a target rotation matrix between the two adjacent video frames; if the shooting state corresponding to the two adjacent video frames is determined to be the non-stationary shooting state, the essential matrix of the matching point pair set of the two adjacent video frames can be obtained, and then arithmetic operation processing is carried out on the essential matrix of the matching point pair set to obtain the target rotation matrix between the two adjacent video frames.

The method for judging whether the shooting states corresponding to the two adjacent video frames are still shooting states may be to compare the pixel values of each pixel point in the previous video frame with the pixel values of the corresponding pixel points in the next video frame, and if the pixel values of all the pixel points in the previous video frame and the next video frame are equal, determining that the shooting states corresponding to the two adjacent video frames are still shooting states; otherwise, determining the shooting states corresponding to the two adjacent video frames as non-stationary shooting states.

In addition, due to some calculation errors or shooting environment errors, error amounts with equal pixel values of some pixel points are required to be reserved to determine shooting states corresponding to two adjacent video frames, and based on the error amounts, whether the shooting states corresponding to the two adjacent video frames are still shooting states or not can be determined by comparing the pixel values of each pixel point in a previous video frame with the pixel values of corresponding pixel points in a subsequent video frame, and if the pixel values of the preset number of pixel points in the previous video frame and the subsequent video frame are equal, determining the shooting states corresponding to the two adjacent video frames to be still shooting states; otherwise, determining the shooting states corresponding to the two adjacent video frames as non-stationary shooting states.

Alternatively, the preset number may be smaller than the total number of all pixels in the previous video frame or the subsequent video frame, but the preset number cannot be too small. For example, the preset number may be equal to 90%, 93%, 95% or 98% of the total number of all pixels in the previous video frame or the subsequent video frame, etc.

Meanwhile, the mode of acquiring the essential matrix of each candidate matching point pair in the matching point pair sets of two adjacent video frames can be to calculate the essential matrix of the two adjacent video frames by utilizing the epipolar constraint principle according to the pixel coordinates of each candidate matching point pair in the matching point pair sets of the two adjacent video frames.

In practical application, the epipolar constraint principle is utilized to select any candidate matching point pair in the matching point pair sets of two adjacent video frames M _i and M _i+1 There is an epipolar constraint equation in equation (1), namely:

Wherein K represents an internal parameter matrix of the shooting equipment, and E represents an essential matrix of two adjacent video frames. Further, m epipolar constraint equations formed by m candidate matching point pairs in the matching point pair set are combined, and solved by adopting an iteration method, so that an optimal essential matrix E of two adjacent video frames is obtained. Alternatively, the iterative method may be a steepest descent method, a conjugate iterative method, a variable-scale iterative method, a least square method, or the like, but in the embodiment of the present application, the iterative method is a random sampling consistency method.

In practical application, due to measurement noise and calculation errors, small errors exist in the rotation matrix and translation information calculated by using the candidate matching point pairs, and when the rotation matrix and the translation information are finally fed back to a video, a picture which is supposed to be stationary may shake slightly. In order to improve the accuracy of video frame motion estimation in a still state and the quality of a video slice, it is necessary to determine the still shooting state of a video frame. Based on this, in one embodiment, before the step of determining that the photographing states corresponding to the adjacent two video frames are the non-still photographing state or the still photographing state, as shown in fig. 7, the above method may include the steps of:

S142, determining candidate matching point pairs which meet preset rest point conditions in the matching point pair sets as rest point pairs.

Specifically, for any candidate matching point pair in the matching point pair set, the computer device may determine, according to the pixel coordinates of the candidate matching point pair, whether the candidate matching point pair meets a preset rest point condition, if so, determine the candidate matching point pair as a rest point pair, and otherwise, determine the candidate matching point pair as a non-rest point pair.

In the embodiment of the present application, for any candidate matching point pair j in the matching point pair sets of two adjacent video frames M _i and M _i+1, a preset rest point condition expressed in the following formula (2) needs to be satisfied, so that the candidate matching point pair j can be determined as a rest point pair;

Wherein, ii represents the modulo length, and th _static represents a preset threshold.

And S152, if the total number of all the stationary point pairs in the matching point pair set is larger than a preset number threshold, determining that the shooting states corresponding to the two adjacent video frames are stationary shooting states, otherwise, determining that the shooting states corresponding to the two adjacent video frames are non-stationary shooting states.

In an ideal state, all candidate matching point pairs in the matching point pair set are stationary point pairs, and the shooting states corresponding to two adjacent video frames can be directly determined to be stationary shooting states.

However, in practical application, since some calculation errors exist, some error amounts of the rest point pairs may be reserved to determine shooting states corresponding to two adjacent video frames. Correspondingly, based on the total number of all the rest point pairs in the set of the matching point pairs obtained in the step, whether the total number of all the rest point pairs in the set of the matching point pairs is larger than a preset number threshold value can be judged, if yes, the shooting states corresponding to the two adjacent video frames are determined to be static shooting states, otherwise, the shooting states corresponding to the two adjacent video frames are determined to be non-static shooting states. In this case, the preset number threshold may be equal to 87%, 92%, 95% or 97% of the total number of all candidate matching point pairs in the matching point pair set, and so on.

According to the technical scheme, when the shooting state corresponding to two adjacent video frames is a non-static shooting state, the essential matrix of each candidate matching point pair in the matching point pair set of the two adjacent video frames is obtained, the target rotation matrix between the two adjacent video frames is obtained based on the essential matrix of each candidate matching point pair, and when the shooting state corresponding to the two adjacent video frames is a static shooting state, the unit matrix is determined to be the target rotation matrix between the two adjacent video frames; according to the method, the target rotation matrix between two adjacent video frames can be accurately determined in a corresponding mode according to the shooting states corresponding to the two adjacent video frames, so that accurate basic information is provided for subsequent further processing, and further the video anti-shake accuracy can be improved.

In an embodiment, as shown in fig. 8, the step of obtaining the target rotation matrix between two adjacent video frames based on the essential matrix of the matching point pair set may be implemented by the following ways:

s132a, performing singular value decomposition on the essential matrix of the matching point pair set to obtain an initial rotation matrix, three-dimensional translation information and a depth information set between two adjacent video frames.

Specifically, the computer device may perform singular value decomposition on the essential matrix of the matching point pair set to obtain a first rotation matrix, a second rotation matrix, a first translation vector and a second translation vector, combine the singular value decomposition results to obtain four sets of data (respectively, the first rotation matrix and the first translation vector, the second rotation matrix and the second translation vector, the first rotation matrix and the second translation vector), and then triangulate all candidate matching point pairs in the matching point pair set according to each set of data to obtain a depth information set, and determine an initial rotation matrix, three-dimensional translation information and a depth information set between two adjacent video frames according to the depth information set corresponding to each set of data.

In practical application, the depth information set includes m depth information (m is equal to the total number of all candidate matching point pairs in the matching point pair set), based on the four obtained depth information sets, the total number of positive values of the depth information in the four depth information sets can be obtained, then the corresponding depth information set with the largest total number of positive values of the depth information is determined as the depth information sets of two adjacent video frames, and a set of rotation matrix and translation vector corresponding to the depth information set with the largest total number of positive values of the depth information is respectively determined as the initial rotation matrix and three-dimensional translation information between the two adjacent video frames. Here, each depth information in the depth information set corresponds to one candidate matching point pair.

Wherein, when the optical flow tracking of the adjacent two video frames M _i and M _i+1 is invalid, the three-dimensional translation information of the adjacent two video frames M _i and M _i+1 Zero vector equal to 0 _3×1, i.e. equal to 3*1.

And S132b, acquiring a target rotation matrix between two adjacent video frames according to the initial rotation matrix, the three-dimensional translation information and the depth information set.

The computer device may pre-train an algorithm model, and then input the initial rotation matrix, the three-dimensional translation information, and the depth information set into the algorithm model, where the algorithm model outputs a target rotation matrix between two adjacent video frames.

In addition, the computer device may perform screening, optimizing, and/or comparing processes according to the initial rotation matrix, the three-dimensional translation information, and the depth information set, to obtain a target rotation matrix between two adjacent video frames.

In one embodiment, as shown in fig. 9, the step of obtaining the target rotation matrix between two adjacent video frames according to the initial rotation matrix, the three-dimensional translation information, and the depth information set in S132b may include:

s1321, screening a plurality of target matching point pairs from all candidate matching point pairs in the matching point pair set according to the depth information set.

Specifically, the computer device may acquire depth information in which depth information of two adjacent video frames is concentrated to a positive value, and then determine a candidate matching point pair corresponding to the positive value of the depth information as the target matching point pair. Alternatively, the total number of all target matching point pairs may be less than or equal to the total number of all candidate matching point pairs within the set of matching point pairs.

S1322, determining a target rotation matrix between two adjacent video frames according to the initial rotation matrix, the three-dimensional translation information and each target matching point pair.

In practical applications, the computer device may process the initial rotation matrix, the three-dimensional translation information, and each target matching point pair to obtain a target rotation matrix between two adjacent video frames.

In one embodiment, as shown in fig. 10, the step of determining the target rotation matrix between two adjacent video frames according to the initial rotation matrix, the three-dimensional translation information, and each target matching point pair in S1322 may include:

and S1322a, mapping each target matching point pair to the shooting equipment coordinate system through shooting equipment internal parameters to obtain a mapping coordinate.

In the embodiment of the present application, the shooting device coordinate system is a three-dimensional space coordinate system. The computer device can map each target matching point pair to the shooting device coordinate system through shooting device internal parameters to obtain corresponding mapping coordinates.

In one implementation manner, mapping each target matching point pair to the photographing device coordinate system through the photographing device internal parameter respectively may be that a conversion relation, that is, a conversion matrix, between the pixel coordinate system and the photographing device coordinate system is calculated through the photographing device internal parameter, and then the conversion matrix between the pixel coordinate system and the photographing device coordinate system is multiplied by each target matching point respectively, so as to complete mapping each target matching point pair to the photographing device coordinate system. It should be noted that the pixel coordinate system is a two-dimensional plane coordinate system, that is, a video frame coordinate system.

In another implementation manner, mapping each target matching point pair to the photographing apparatus coordinate system through the photographing apparatus internal parameter may be performing coordinate conversion processing on each target matching point pair through the photographing apparatus internal parameter so as to map each target matching point pair to the photographing apparatus coordinate system.

S1322b, constructing a nonlinear least square model through the initial rotation matrix, the three-dimensional translation information and the mapping coordinates.

In the embodiment of the application, the computer equipment can acquire a constructed network model, then input the initial rotation matrix and three-dimensional translation information between two adjacent video frames and the mapping coordinates corresponding to each target matching point pair into the constructed network model, and the constructed network model outputs a nonlinear least square model.

Optionally, the network model may be constructed by at least one of a convolutional neural network model, a fully-connected neural network model, a cyclic recurrent neural network model, a deep belief network model, a deep automatic encoder, and a generating countermeasure network model, which is not limited in this embodiment of the present application.

S1322c, solving a nonlinear least square model to obtain a target rotation matrix between two adjacent video frames.

Based on the nonlinear least square model constructed by the steps, the computer equipment can adopt a least square problem solving method to solve the nonlinear least square model to obtain a target rotation matrix between two adjacent video frames. Alternatively, the above least squares solution may be at least one of newton's method, gradient descent method, gaussian newton's method, and levenberg-marquardt method.

According to the technical scheme, singular value decomposition is carried out on an essential matrix of a matching point pair set to obtain an initial rotation matrix, three-dimensional translation information and a depth information set between two adjacent video frames, and a target rotation matrix between the two adjacent video frames is obtained according to the initial rotation matrix, the three-dimensional translation information and the depth information set; the method can determine the target rotation matrix between two adjacent video frames by constructing a nonlinear least square problem, and the processing process is simpler and can be realized without complex algorithm, so that the acquisition speed of the target rotation matrix can be improved; in addition, the method has no complex algorithm, and can acquire the matching point pair sets of two adjacent video frames without the need of a user to learn too much expertise.

In practical application, in order to improve accuracy of a video anti-shake processing result, motion of each video frame in an initial video in a whole video shooting process needs to be combined together to achieve the anti-shake processing result, based on the motion, global rotation motion information of each video frame in the initial video can be obtained first, and further accurate anti-shake processing is achieved by taking the global rotation motion information as reference information, and a process of the global rotation motion information of each video frame in the initial video is described below. In an embodiment, as shown in fig. 11, the step of obtaining global rotational motion information of the initial video according to the three-dimensional rotational information in S200 may include:

S201, acquiring a global rotation motion matrix of each video frame in the initial video according to the three-dimensional rotation information of the initial video.

Specifically, the computer device can calculate the global rotational motion matrix of each video frame in the initial video by using a cumulative way through the three-dimensional rotational information of the initial video.

For example, if the initial video includes 4 video frames, the three-dimensional rotation information of the initial video is [ S1S 2S 3S 4], the global rotation motion matrix of the first video frame in the initial video may be equal to S1, the global rotation motion matrix of the second video frame in the initial video may be equal to S1×s2, the global rotation motion matrix of the third video frame in the initial video may be equal to S1×s2×s3, and the global rotation motion matrix of the fourth video frame in the initial video may be equal to S1×s2×s3×s4.

S202, determining global rotation motion information of an initial video according to a global rotation motion matrix of each video frame.

The global rotational motion matrix of each video frame in the initial video can be subjected to unit quaternion format conversion to obtain global rotational motion information corresponding to each video frame in the initial video, and the global rotational motion information corresponding to each video frame in the initial video is combined to obtain global rotational motion information of the initial video. Alternatively, the global rotational motion information of the initial video may be understood as a motion estimation result of the initial video.

According to the technical scheme, the global rotation motion matrix of each video frame in the initial video is obtained according to the three-dimensional rotation information of the initial video, and the global rotation motion information of the initial video is determined according to the global rotation motion matrix of each video frame; according to the method, global rotation motion information of the initial video can be calculated through the three-dimensional rotation information of the initial video, which is equivalent to taking the motion state of each video frame in the whole video shooting process into consideration, so that the motion estimation precision of each video frame in the initial video can be improved, and video anti-shake processing is realized by further combining the global rotation motion information of the initial video, so that the anti-shake processing process is more real, the accuracy of the video anti-shake processing can be improved, and the video anti-shake effect can be improved.

The process of generating the anti-shake video of the initial video by performing the motion smoothing process on the global rotational motion information will be described below. In an embodiment, as shown in fig. 12, the step in S300 may be implemented as follows:

And S301, performing motion smoothing processing on the global rotation motion information to obtain motion smoothing data of the initial video.

Specifically, the computer device may perform motion smoothing processing on global rotational motion information of the initial video by using a motion smoothing algorithm, so as to obtain motion smoothing data of the initial video. In the embodiment of the present application, the motion smoothing algorithm may be a spherical linear interpolation smoothing algorithm.

S302, determining target motion smoothing data of the initial video according to motion smoothing weights of video frames in the initial video, global rotation motion information of the initial video and the motion smoothing data.

Because there may be a case that two adjacent video frames are invalid in optical flow tracking in the initial video, a three-dimensional rotation matrix corresponding to the two adjacent video frames cannot be calculated, and the identity matrix is directly determined as the three-dimensional rotation matrix between the two adjacent video frames, a final target motion smoothing result needs to consider motion smoothing weights of all video frames, so that accuracy of the determined target motion smoothing data of the initial video is improved.

In practical application, the computer device may first obtain the motion smoothing weight of each video frame in the initial video, and then process global rotational motion information and/or motion smoothing data of the initial video based on the motion smoothing weight of each video frame in the initial video, to obtain target motion smoothing data of the initial video. It should be noted that, the object motion smoothing data of the initial video includes an object motion smoothing result of each video frame in the initial video.

By way of example, if the initial video includes p video frames, the motion smoothing weight set of the initial video is set to phi (w),

φ(W)＝{w₁,w₂,...,w_n,...,w_p} (3)；

Wherein w _n represents the motion smoothing weight of the nth video frame in the initial video, w _n can be set to 1 if optical flow tracking of the nth video frame is invalid, and at the same time, w _n is taken as the center, radius r is set, and r motion smoothing weights which are continuous in front and behind w _n in phi (w) are taken to form a set

Alternatively according toThe distance between each motion smoothing weight and w _n can be set to be corresponding to different weights, and the weight value is in the range of [0,1 ]; and r is smaller than (p-1)/2, if w _n-r,...,w_n-1 or w _n+1,...,w_n+r is not in the range of w ₁,w₂,...,w_n,...,w_p for any video frame, deleting the motion smoothing weight which is not in the range of w ₁,w₂,...,w_n,...,w_p, and only updating the motion smoothing weight in the range. It should be noted that, if the optical flow tracking of the nth video frame is effective, the motion smoothing weight w _n may be 0 correspondingly.

In the embodiment of the present application, if the motion smoothing weight w _n of the nth video frame is 1, determining global rotational motion information of the video frame in global rotational motion information of the initial video as a target motion smoothing result of the video frame; if the motion smoothing weight w _n of the nth video frame is 0, determining the motion smoothing result of the video frame in the motion smoothing information of the initial video as a target motion smoothing result of the video frame; if the motion smoothing weight w _n of the nth video frame is in the range of (0, 1), performing spherical linear interpolation on the global rotation motion information of the video frame and the motion smoothing result of the video frame in the global rotation motion information to obtain the target motion smoothing result of the video frame.

S303, generating an anti-shake video according to the target motion smooth data and the initial video.

The computer equipment can perform data conversion processing, superposition processing and/or data enhancement processing on each video frame in the initial video according to the target motion smoothing data of the initial video to obtain each processed video frame, and then fusion is performed on each processed video frame according to the shooting sequence of each video frame in the initial video to obtain the anti-shake video.

In one embodiment, as shown in fig. 13, the step of generating the anti-shake video according to the target motion smoothing data and the initial video in S303 may include:

s313, determining the rotation compensation amount of the initial video according to the target motion smoothing data and the global rotation motion information.

Alternatively, the computer device may perform arithmetic operation processing on the target motion smoothing data of the initial video and the global rotational motion information of the initial video to obtain the rotational compensation amount of the initial video.

If the initial video includes p video frames, the target motion smoothing data of the initial video is Φ (q ") = { q" ₁,q″₂,...,q″_i,...,q″_p},q″_i indicating the target motion smoothing result of the i-th video frame in the initial video, and the global rotation motion information of the initial video is Φ (q) = { q ₁,q₂,...,q_i,...,q_p }, the rotation compensation amount Φ (Δq) of the initial video can be expressed as follows by the formula (5):

Wherein the symbols are Representing a quaternion multiplication, Δq _i represents the amount of rotation compensation for the i-th video frame in the initial video.

For example, if the pixel point M in two adjacent video frames rotates 90 degrees around the x-axis in the spatial coordinate system, and the corresponding motion smoothing data is 80 degrees, 10 (i.e. 90-80) may be the rotation compensation amount by which the pixel point M rotates around the x-axis; meanwhile, if the pixel point M in two adjacent video frames rotates 90 degrees around the y-axis in the spatial coordinate system, and the corresponding motion smoothing data is 85 degrees, 5 (i.e., 90-85) can be the rotation compensation amount of the pixel point M rotating around the y-axis; and if the pixel point M in two adjacent video frames rotates by 95 degrees around the z-axis in the spatial coordinate system, and the corresponding motion smoothing data is 80 degrees, 15 (i.e. 95-80) can be the rotation compensation amount of the pixel point M rotating around the z-axis.

Correspondingly, the rotation compensation amount of the pixel point M in the adjacent two video frames in space may include a rotation compensation amount by which the pixel point M rotates around the x-axis, a rotation compensation amount by which the pixel point M rotates around the y-axis, and a rotation compensation amount by which the pixel point M rotates around the z-axis. Here, the rotation compensation amount of the initial video may include a rotation compensation amount of each pixel point on each video frame in the initial video, and naturally, the rotation compensation amount of each video frame in the initial video may be represented in a matrix form.

S323, performing mapping processing on pixel points in each video frame according to the rotation compensation amount of the initial video, and generating an anti-shake video.

In the embodiment of the application, the computer equipment can map the pixel points in each video frame according to the rotation compensation amount of each video frame in the initial video so as to map the pixel points on each video frame in the initial video to the video frame after the anti-shake, obtain the pixel coordinates of the pixel points on each video frame on the video frame after the anti-shake, then generate the corresponding video frame after the anti-shake according to the pixel coordinates of the pixel points on each video frame on the video frame after the anti-shake, and then fuse the video frames after the anti-shake according to the shooting sequence of the video frames in the initial video to obtain the anti-shake video.

In practical application, the pixel points after the mapping process of each pixel point on each video frame in the initial video may fall outside the video frame after the anti-shake, so after the mapping coordinates corresponding to each pixel point on each video frame are obtained, the anti-shake video frame is generated according to the mapping coordinates corresponding to each pixel point, then the anti-shake video frame corresponding to each video frame is cut into a uniform size, and then the cut video frames are fused to obtain the anti-shake video.

In one implementation, the minimum size video frame of all the anti-shake video frames is taken as a reference, other anti-shake video frames are cut into a uniform size with the minimum size video frame, and then the cut video frames are fused to obtain the anti-shake video.

According to the technical scheme, motion smoothing processing is carried out on global rotation motion information to obtain motion smoothing data of an initial video, target motion smoothing data of the initial video are determined according to motion smoothing weights of video frames in the initial video, global rotation motion information of the initial video and the motion smoothing data, and anti-shake video is generated according to the target motion smoothing data and the initial video; the method can carry out motion smoothing processing on the global rotation motion information of each video frame in the initial video, namely the three-dimensional motion estimation result, thereby improving the accuracy of the determined motion smoothing data of the initial video, and simultaneously, on the basis that the motion estimation can not be realized by considering the video frames with invalid optical flow tracking in the initial video, the motion smoothing weight of each video frame in the initial video can be considered, the optimal motion smoothing data of the initial video, namely the target motion smoothing data, can be determined, and further, the anti-shake video with good anti-shake effect is generated according to the target motion smoothing data and the initial video, so that the stability of video frame display in the initial video is improved, and the picture quality of the initial video is improved.

In an embodiment, the embodiment of the application also provides a video anti-shake processing method, which comprises the following steps:

(1) And for any two adjacent video frames, extracting the feature points in the previous video frame in the two adjacent video frames to obtain a plurality of original feature points.

(2) And carrying out optical flow tracking on the next video frame in the two adjacent video frames according to each original feature point to obtain a plurality of tracking feature points in the next video frame.

(3) And carrying out reflective flow tracking on the previous video frame according to each tracking characteristic point to obtain a plurality of anti-tracking characteristic points in the previous video frame.

(4) And obtaining the interval distance between each anti-tracking feature point and the corresponding original feature point.

(5) And determining the anti-tracking feature points with the interval distance smaller than a preset distance threshold as the anti-tracking feature points with effective optical flow tracking.

(6) And determining a matching point pair set of two adjacent video frames according to all the anti-tracking feature points for which optical flow tracking is effective.

(7) And for any two adjacent video frames, acquiring optical flow tracking results of the two adjacent video frames according to the matching point pair sets of the two adjacent video frames.

(8) If the optical flow tracking result of the two adjacent video frames is that the optical flow tracking is invalid, determining the identity matrix as a target rotation matrix between the two adjacent video frames.

(9) If the optical flow tracking result of the two adjacent video frames is that the optical flow tracking is effective, determining a target rotation matrix between the two adjacent video frames according to the shooting states corresponding to the two adjacent video frames.

The step (9) of determining the target rotation matrix between two adjacent video frames according to the shooting states corresponding to the two adjacent video frames includes:

(91) Determining candidate matching point pairs which meet preset rest point conditions in the matching point pair sets as rest point pairs;

(92) If the total number of all the stationary point pairs in the matching point pair set is larger than a preset number threshold, determining that the shooting states corresponding to the two adjacent video frames are stationary shooting states, otherwise, determining that the shooting states corresponding to the two adjacent video frames are non-stationary shooting states.

(93) If the shooting state corresponding to two adjacent video frames is a non-stationary shooting state, performing singular value decomposition on an essential matrix of a matching point pair set to obtain an initial rotation matrix, three-dimensional translation information and a depth information set between the two adjacent video frames, and screening a plurality of target matching point pairs from all candidate matching point pairs in the matching point pair set according to the depth information set;

(94) Mapping each target matching point pair to a shooting equipment coordinate system through shooting equipment internal parameters to obtain a mapping coordinate;

(95) Constructing a nonlinear least square model through the initial rotation matrix, the three-dimensional translation information and the mapping coordinates;

(96) Solving a nonlinear least square model to obtain a target rotation matrix between two adjacent video frames;

meanwhile, in the step (9), the step of determining the target rotation matrix between two adjacent video frames according to the shooting states corresponding to the two adjacent video frames further includes:

(97) And if the shooting state corresponding to the two adjacent video frames is the static shooting state, determining the identity matrix as a target rotation matrix between the two adjacent video frames.

(10) And determining the three-dimensional rotation information of the initial video according to the target rotation matrix between every two adjacent video frames.

(11) And acquiring a global rotation motion matrix of each video frame in the initial video according to the three-dimensional rotation information of the initial video.

(12) And determining global rotation motion information of the initial video according to the global rotation motion matrix of each video frame.

(13) And performing motion smoothing processing on the global rotation motion information to obtain motion smoothing data of the initial video.

(14) And determining target motion smoothing data of the initial video according to the motion smoothing weight of each video frame in the initial video, global rotation motion information of the initial video and the motion smoothing data.

(15) And determining the rotation compensation amount of the initial video according to the target motion smoothing data and the global rotation motion information.

(16) And mapping the pixel points in each video frame according to the rotation compensation amount of the initial video to generate an anti-shake video.

The implementation process of the above (1) to (16) may be specifically referred to the description of the above embodiment, and its implementation principle and technical effects are similar, and will not be described herein again.

It should be understood that, although the steps in the flowcharts related to the above embodiments are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiment of the application also provides a video anti-shake processing device for realizing the video anti-shake processing method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation in the embodiments of the video anti-shake processing device or devices provided below may refer to the limitation of the video anti-shake processing method described above, and will not be repeated here.

In an embodiment, fig. 14 is a schematic structural diagram of a video anti-shake processing apparatus according to an embodiment of the present application, where the video anti-shake processing apparatus according to the embodiment of the present application may be applied to a computer device. As shown in fig. 14, the video anti-shake processing apparatus according to the embodiment of the present application may include: a rotation information acquisition module 11, a global information acquisition module 12, and a smoothing processing module 13, wherein:

a rotation information acquisition module 11 for acquiring three-dimensional rotation information of an initial video;

The global information acquisition module 12 is configured to acquire global rotational motion information of the initial video according to the three-dimensional rotational information;

And the smoothing processing module 13 is used for performing motion smoothing processing on the global rotation motion information to generate an anti-shake video of the initial video.

The video anti-shake processing device provided by the embodiment of the application can be used for executing the technical scheme in the embodiment of the video anti-shake processing method, and the implementation principle and the technical effect are similar, and are not repeated here.

In one embodiment, the rotation information acquisition module 11 includes: the device comprises a matching point pair set acquisition unit, a target rotation matrix determination unit and a three-dimensional rotation information determination unit, wherein:

The matching point pair set acquisition unit is used for acquiring matching point pair sets of every two adjacent video frames in the initial video;

The target rotation matrix determining unit is used for determining a target rotation matrix between two adjacent video frames according to the matching point pair sets of the two adjacent video frames for any two adjacent video frames;

and the three-dimensional rotation information determining unit is used for determining the three-dimensional rotation information of the initial video according to the target rotation matrix between every two adjacent video frames.

In one embodiment, the matching point pair set acquisition unit includes: the device comprises a feature point extraction subunit, an optical flow tracking subunit, a reflection flow tracking subunit and a matching point pair set determination subunit, wherein:

The characteristic point extraction subunit is used for extracting the characteristic points in the previous video frame in any two adjacent video frames to obtain a plurality of original characteristic points;

the optical flow tracking subunit is used for carrying out optical flow tracking on a next video frame in two adjacent video frames according to each original characteristic point to obtain a plurality of tracking characteristic points in the next video frame;

the reflection flow tracking subunit is used for carrying out reflection flow tracking on the previous video frame according to each tracking characteristic point to obtain a plurality of anti-tracking characteristic points in the previous video frame;

And the matching point pair set determining subunit is used for determining the matching point pair sets of two adjacent video frames according to the anti-tracking feature points.

In one embodiment, the matching point pair set determining subunit is specifically configured to:

In one embodiment, the target rotation matrix determining unit includes: a tracking result acquisition subunit, a first determination subunit, and a second determination subunit, wherein:

The tracking result acquisition subunit is used for acquiring optical flow tracking results of two adjacent video frames according to the matching point pair sets of the two adjacent video frames;

a first determination subunit, configured to determine, when the optical flow tracking result of two adjacent video frames is that optical flow tracking is invalid, an identity matrix as a target rotation matrix between the two adjacent video frames;

And the second determination subunit is used for determining a target rotation matrix between the two adjacent video frames according to the shooting states corresponding to the two adjacent video frames when the optical flow tracking result of the two adjacent video frames is effective in optical flow tracking.

In one embodiment, the second determining subunit comprises: a first acquisition subunit and a third determination subunit, wherein:

The first acquisition subunit is used for acquiring the essential matrix of the matching point pair set of the two adjacent video frames when the shooting state corresponding to the two adjacent video frames is a non-static shooting state, and acquiring the target rotation matrix between the two adjacent video frames based on the essential matrix of the matching point pair set;

And the third determination subunit is used for determining the identity matrix as a target rotation matrix between two adjacent video frames when the shooting states corresponding to the two adjacent video frames are static shooting states.

In one embodiment, the first acquisition subunit comprises: a singular value decomposition subunit and a second acquisition subunit, wherein:

The singular value decomposition subunit is used for performing singular value decomposition on the essential matrix of the matching point pair set to obtain an initial rotation matrix, three-dimensional translation information and a depth information set between two adjacent video frames;

And the second acquisition subunit is used for acquiring the target rotation matrix between two adjacent video frames according to the initial rotation matrix, the three-dimensional translation information and the depth information set.

In one embodiment, the second acquisition subunit is specifically configured to:

determining a target rotation matrix between two adjacent video frames according to the initial rotation matrix, the three-dimensional translation information and each target matching point pair;

the method for determining the target rotation matrix between two adjacent video frames according to the initial rotation matrix, the three-dimensional translation information and each target matching point pair comprises the following steps:

In one embodiment, the second determining subunit further comprises: a rest point pair determining subunit, a shooting state determining subunit, wherein:

A rest point pair determining subunit, configured to determine, as a rest point pair, a candidate match point pair in the set of match point pairs that satisfies a preset rest point condition;

And the shooting state determining subunit is used for determining that the shooting states corresponding to the two adjacent video frames are static shooting states when the total number of all the static point pairs in the matching point pair set is larger than a preset number threshold, otherwise, determining that the shooting states corresponding to the two adjacent video frames are non-static shooting states.

In one embodiment, the global information acquisition module 12 includes: a global motion matrix acquisition unit and a global information determination unit, wherein:

The global motion matrix acquisition unit is used for acquiring a global rotation motion matrix of each video frame in the initial video according to the three-dimensional rotation information of the initial video;

and the global information determining unit is used for determining the global rotation motion information of the initial video according to the global rotation motion matrix of each video frame.

In one embodiment, the smoothing module 13 includes: a motion smoothing processing unit, a target smoothing data determining unit, and an anti-shake video generating unit, wherein:

the motion smoothing processing unit is used for performing motion smoothing processing on the global rotation motion information to obtain motion smoothing data of the initial video;

the target smooth data determining unit is used for determining target smooth data of the initial video according to the motion smooth weight of each video frame in the initial video, global rotation motion information of the initial video and the motion smooth data;

and the anti-shake video generation unit is used for generating an anti-shake video according to the target motion smooth data and the initial video.

In one embodiment, an anti-shake video generation unit includes: a compensation amount determination subunit and a mapping processing subunit, wherein:

The compensation amount determining subunit is used for determining the rotation compensation amount of the initial video according to the target motion smooth data and the global rotation motion information;

and the mapping processing subunit is used for carrying out mapping processing on the pixel points in each video frame according to the rotation compensation quantity of the initial video to generate the anti-shake video.

For specific limitations of the video anti-shake processing apparatus, reference may be made to the above limitations of the video anti-shake processing method, and no further description is given here. The modules in the video anti-shake processing device may be implemented in whole or in part by software, hardware, or a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

In one embodiment, there is also provided a video anti-shake processing system, with continued reference to fig. 1, comprising:

A photographing device;

and the computer equipment is connected with the shooting equipment and is used for executing the steps of any embodiment of the video anti-shake processing method.

The video anti-shake processing system provided by the embodiment of the application can be used for executing the technical scheme in the embodiment of the video anti-shake processing method, and the implementation principle and the technical effect are similar and are not repeated here.

In one embodiment, a computer device is also provided, the internal structure of which may be as shown in fig. 15. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide processing power. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used to store the initial video and the anti-shake video. The network interface of the computer device is for communicating with an external endpoint via a network connection. The computer program when executed by a processor implements a video anti-shake processing method.

It will be appreciated by those skilled in the art that the structure shown in fig. 15 is merely a block diagram of a portion of the structure associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements are applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, there is also provided a computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:

Acquiring three-dimensional rotation information of an initial video;

In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor, performs the steps of:

Acquiring three-dimensional rotation information of an initial video;

In one embodiment, a computer program product is provided comprising a computer program which, when executed by a processor, performs the steps of:

Acquiring three-dimensional rotation information of an initial video;

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, or the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory. By way of illustration, and not limitation, RAM can be in various forms such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The above examples illustrate only a few embodiments of the application, which are described in detail and are not to be construed as limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.

Claims

1.A method for anti-shake processing of video, the method comprising:

Acquiring three-dimensional rotation information of an initial video;

acquiring global rotation motion information of the initial video according to the three-dimensional rotation information;

2. The method of claim 1, wherein the acquiring three-dimensional rotation information of the initial video comprises:

acquiring a matching point pair set of every two adjacent video frames in the initial video;

3. The method of claim 2, wherein the obtaining the set of matching point pairs for each adjacent two video frames in the initial video comprises:

performing optical flow tracking on a next video frame in two adjacent video frames according to the original feature points to obtain a plurality of tracking feature points in the next video frame;

4. A method according to claim 3, wherein determining the set of matching pairs of adjacent two video frames based on each anti-tracking feature point comprises:

determining the anti-tracking feature points with the interval distance smaller than a preset distance threshold as the anti-tracking feature points with effective optical flow tracking;

and determining the matching point pair sets of two adjacent video frames according to all the anti-tracking feature points for which optical flow tracking is effective.

5. The method according to any one of claims 2-4, wherein determining a target rotation matrix between two adjacent video frames from the set of matching point pairs of the two adjacent video frames comprises:

If the optical flow tracking result of the two adjacent video frames is that the optical flow tracking is invalid, determining an identity matrix as the target rotation matrix between the two adjacent video frames;

if the optical flow tracking result of the two adjacent video frames is that the optical flow tracking is effective, determining the target rotation matrix between the two adjacent video frames according to the shooting states corresponding to the two adjacent video frames.

6. The method according to claim 5, wherein determining the target rotation matrix between two adjacent video frames according to the shooting status corresponding to the two adjacent video frames comprises:

7. The method of claim 6, wherein the obtaining a target rotation matrix between two adjacent video frames based on the essence matrix of the matching point pair set comprises:

8. The method of claim 7, wherein the obtaining a target rotation matrix between two adjacent video frames from the initial rotation matrix, the three-dimensional translation information, and the depth information set comprises:

9. The method of claim 8, wherein determining the target rotation matrix between two adjacent video frames based on the initial rotation matrix, the three-dimensional translation information, and each target matching point pair comprises:

mapping each target matching point pair to a shooting equipment coordinate system through shooting equipment internal parameters to obtain mapping coordinates;

and solving the nonlinear least square model to obtain a target rotation matrix between two adjacent video frames.

10. The method of claim 6, wherein the method further comprises:

And if the total number of all the stationary point pairs in the matching point pair set is larger than a preset number threshold, determining that the shooting state corresponding to the two adjacent video frames is the stationary shooting state, otherwise, determining that the shooting state corresponding to the two adjacent video frames is the non-stationary shooting state.

11. The method according to any one of claims 1-4, wherein the obtaining global rotational motion information of the initial video from the three-dimensional rotational information comprises:

12. The method according to any one of claims 1-4, wherein the performing motion smoothing on the global rotational motion information to generate an anti-shake video of the initial video includes:

Determining target motion smoothing data of the initial video according to the motion smoothing weight of each video frame in the initial video, global rotation motion information of the initial video and the motion smoothing data;

and generating the anti-shake video according to the target motion smooth data and the initial video.

13. The method of claim 12, wherein generating the anti-shake video from the target motion smoothing data and the initial video comprises:

Determining a rotation compensation amount of the initial video according to the target motion smoothing data and the global rotation motion information;

and mapping pixel points in each video frame according to the rotation compensation amount of the initial video to generate the anti-shake video.

14. A video anti-shake processing apparatus, the apparatus comprising:

15. A video anti-shake processing system, the system comprising:

A photographing device;

Computer device connected to the photographing device for performing the steps of the method of any of the preceding claims 1-13.

16. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1-13 when the computer program is executed.

17. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1-13.