CN115720263B - Panoramic video storage optimization method, system, terminal and storage medium - Google Patents
Panoramic video storage optimization method, system, terminal and storage mediumInfo
- Publication number
- CN115720263B CN115720263B CN202210886773.9A CN202210886773A CN115720263B CN 115720263 B CN115720263 B CN 115720263B CN 202210886773 A CN202210886773 A CN 202210886773A CN 115720263 B CN115720263 B CN 115720263B
- Authority
- CN
- China
- Prior art keywords
- panoramic video
- panoramic
- projection
- video block
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Studio Devices (AREA)
Abstract
本发明公开了一种全景视频的存储优化方法、系统、终端及存储介质，所述方法包括：获取全景视频中第一全景图像，识别所述第一全景图像中的对象，计算所述对象的运动范围，并将所述全景视频分割成全景视频块；基于所述全景视频块获取第二全景图像，对所述第二全景图像进行分割得到语义分割结果，并对所述语义分割结果进行投影得到投影平面；对所述投影平面进行强化学习处理，并将所述全景视频块投影成平面格式后进行压缩储存。本发明通过将全景视频在时间轴上分割为多段全景视频块，依据每段全景视频块的内容，对投影坐标系进行多自由度空间旋转后，对全景视频进行投影为平面格式并压缩视频，使得压缩后的全景视频所需的存储空间减少。
The present invention discloses a storage optimization method, system, terminal, and storage medium for panoramic videos. The method comprises: obtaining a first panoramic image in a panoramic video, identifying an object in the first panoramic image, calculating the motion range of the object, and segmenting the panoramic video into panoramic video blocks; obtaining a second panoramic image based on the panoramic video blocks, segmenting the second panoramic image to obtain a semantic segmentation result, and projecting the semantic segmentation result to obtain a projection plane; performing reinforcement learning processing on the projection plane, and projecting the panoramic video blocks into a plane format before compressing and storing them. The present invention divides the panoramic video into multiple panoramic video blocks on the time axis, performs multi-degree-of-freedom spatial rotation on the projection coordinate system based on the content of each panoramic video block, and then projects the panoramic video into a plane format and compresses the video, thereby reducing the storage space required for the compressed panoramic video.
Description
Technical Field
The invention relates to the field of panoramic video on demand or live broadcast, in particular to a storage optimization method, a system, a terminal and a storage medium for panoramic video.
Background
Panoramic video is a video which is shot by a 3D camera in 360 degrees in all directions, and a user can randomly adjust the video to watch up, down, left and right when watching the video. Panoramic video can provide an omnidirectional view, allows a user to freely select any direction to watch, and provides an immersive viewing experience.
In the prior art, when panoramic video is stored, due to the fact that the high-quality panoramic video code rate is large, the current bandwidth is insufficient to support real-time transmission of the high-quality panoramic video, the panoramic video is generally divided into a plurality of panoramic video blocks with the same time length on a time axis, then, each frame of panoramic image in the panoramic video block is projected into a polyhedron format by using a plurality of projection coordinate systems, then, each projection surface of the polyhedron is spliced into an image with a single plane format, and then, a traditional plane video compression technology is used for compressing continuous frames, such as an H256 video coding standard, but in the actual projection process, foreground objects such as vehicles in the single frame panoramic image are located in different projection surfaces, the two projection surfaces in the middle of the second row cut off the vehicles, and the spatial discontinuity of the foreground objects is not beneficial to the compression process of the plane video compression technology, so that the compression rate is reduced, and more storage space is occupied.
Furthermore, there may be multiple foreground objects moving simultaneously within each panoramic video block, and the longer the panoramic video block, the larger the range of motion of the foreground objects, and more likely the large number of foreground objects to span more projection surfaces, conversely, if the shorter the time length, the temporal continuity may be disturbed, which is also detrimental to the compression process of planar video compression techniques in these cases.
Accordingly, the prior art is still in need of improvement and development.
Disclosure of Invention
The invention mainly aims to provide a panoramic video storage optimization method, a system, a terminal and a storage medium, and aims to solve the problems that in the prior art, the spatial discontinuity of a foreground object and the too short time of a panoramic video block are unfavorable for the compression process of a planar video compression technology, so that the compression rate is reduced and more storage space is occupied.
In order to achieve the above object, the present invention provides a storage optimization method of panoramic video, the storage optimization method of panoramic video comprising the steps of:
acquiring a first panoramic image in a panoramic video, identifying an object in the first panoramic image, calculating the motion range of the object, and dividing the panoramic video into panoramic video blocks;
acquiring a second panoramic image based on the panoramic video block, dividing the second panoramic image to obtain a semantic division result, and projecting the semantic division result to obtain a projection plane;
And performing reinforcement learning processing on the projection plane, projecting the panoramic video block into a plane format, and then compressing and storing the panoramic video block.
Optionally, the method for optimizing storage of panoramic video, wherein the acquiring a first panoramic image in the panoramic video, identifying an object in the first panoramic image, calculating a motion range of the object, and dividing the panoramic video into panoramic video blocks specifically includes:
Uniformly sampling the panoramic video to obtain panoramic images of a plurality of frames, identifying objects in each frame of panoramic image by adopting an object identification algorithm on the panoramic images, and marking the same objects in continuous frames;
When the center point of an object is used as the center of gravity of the object, setting a time length, and calculating to obtain the center of gravity moving distance between a first frame of each object and a frame after the time length;
calculating an average value of the gravity center moving distance, taking the average value as an average moving range of all objects in the time length, and setting a threshold value of the average range;
And judging the average motion range and the threshold value, and if the average motion range is equal to the threshold value, dividing the panoramic video into a plurality of sections of panoramic video blocks on a time axis by using the time length.
Optionally, in the panoramic video storage optimization method, the threshold value is set according to a projection format.
Optionally, the method for optimizing the storage of panoramic video, wherein the determining the average motion range and the threshold value further includes:
If the average motion range is larger than the threshold value, shortening the time length, and recalculating the average motion range of all objects in the shortened time length;
if the average motion range is smaller than the threshold value, the time length is increased, and the average motion range of all objects in the increased time length is recalculated.
Optionally, the method for optimizing storage of panoramic video, wherein the obtaining a second panoramic image based on the panoramic video block, dividing the second panoramic image to obtain a semantic division result, and projecting the semantic division result to obtain a projection plane specifically includes:
Extracting a second panoramic image of a plurality of frames from the panoramic video block, and inputting the second panoramic image into a semantic segmentation algorithm for segmentation to obtain a semantic segmentation result;
And selecting a projection coordinate system corresponding to the projection format based on the projection format of the semantic segmentation result, and projecting the semantic segmentation result by using the projection coordinate system to obtain a plurality of projection planes.
Optionally, the method for optimizing the storage of the panoramic video, wherein the projection format comprises an equidistant projection format and a cubic projection format.
Optionally, the method for optimizing storage of panoramic video, wherein the performing reinforcement learning on the projection plane, and performing compression storage after projecting the panoramic video block into a plane format, specifically includes:
Inputting the projection planes into a reinforcement learning method according to a time axis sequence for processing, and outputting a multi-degree-of-freedom rotation angle after the processing is finished;
finishing the rotation of the projection coordinate system based on the multi-degree-of-freedom rotation angle, and projecting the panoramic video block by using the rotated projection coordinate system to obtain a first panoramic video block in a planar format;
And compressing and storing the first panoramic video block, and storing the multi-free rotation angle of the first panoramic video block in a record file.
Optionally, the method for optimizing storage of panoramic video, wherein the rotating the projection coordinate system based on the multiple degrees of freedom rotation angle is completed, further includes:
If a plurality of proper multi-free rotation angles exist, carrying out repeated iterative rotation on the projection coordinate system, and projecting the panoramic video block by the projection coordinate system after each rotation to obtain a second panoramic video block with a plurality of plane formats;
compressing all the second panoramic video blocks, and recording the compression rate of each second panoramic video block;
And selecting a second panoramic video block with the highest compression ratio for storage, and recording the multi-degree-of-freedom rotation angle of the second panoramic video block in a file.
Optionally, the method for optimizing storage of panoramic video, wherein the rotating the projection coordinate system based on the multiple degrees of freedom rotation angle is completed, and further includes:
according to the periodic change of the rotation angle, a change range of the rotation angle in one period is obtained;
limiting the rotation angle output by the reinforcement learning method to be in the variation range of the period based on the variation range;
and if the limited rotation angle is 0, ending the processing of the current panoramic video block.
Optionally, the method for optimizing the storage of panoramic video, wherein the compressing and storing the first panoramic video block, and storing the multiple free rotation angles of the first panoramic video block in a record file, further includes:
Receiving an access request of a user, and transmitting a panoramic video block to a user client based on the access request;
when the panoramic video block is transmitted, extracting a multi-free rotation angle corresponding to the panoramic video block from the record file, and transmitting the multi-free rotation angle to the user client;
And after the user client finishes receiving, rotating a projection coordinate system based on the multiple free rotation angles, and projecting the panoramic video block by the rotated projection coordinate system to finish the rendering process.
In addition, in order to achieve the above object, the present invention further provides a storage optimization system for panoramic video, wherein the storage optimization system for panoramic video comprises:
The video segmentation module is used for acquiring a first panoramic image in a panoramic video, identifying an object in the first panoramic image, calculating the motion range of the object, and segmenting the panoramic video into panoramic video blocks;
The video projection module is used for acquiring a second panoramic image based on the panoramic video block, dividing the second panoramic image to obtain a semantic division result, and projecting the semantic division result to obtain a projection plane;
and the compression storage module is used for performing reinforcement learning processing on the projection plane, projecting the panoramic video block into a plane format and then performing compression storage.
In addition, in order to achieve the aim, the invention also provides a terminal, wherein the terminal comprises a memory, a processor and a storage optimization program of the panoramic video, wherein the storage optimization program of the panoramic video is stored in the memory and can run on the processor, and the storage optimization program of the panoramic video realizes the steps of the storage optimization method of the panoramic video when being executed by the processor.
In addition, in order to achieve the above object, the present invention also provides a computer-readable storage medium storing a storage optimization program of panoramic video, which when executed by a processor, implements the steps of the storage optimization method of panoramic video as described above.
The method comprises the steps of obtaining a first panoramic image in a panoramic video, identifying an object in the first panoramic image, calculating the movement range of the object, dividing the panoramic video into panoramic video blocks, obtaining a second panoramic image based on the panoramic video blocks, dividing the second panoramic image to obtain a semantic division result, projecting the semantic division result to obtain a projection plane, performing reinforcement learning on the projection plane, projecting the panoramic video blocks into a plane format, and then compressing and storing the projection plane. According to the invention, the panoramic video is divided into a plurality of sections of panoramic video blocks on a time axis, the projection coordinate system is subjected to multi-degree-of-freedom spatial rotation according to the content of each section of panoramic video block, and then the panoramic video is projected into a plane format by using the rotated projection coordinate system and compressed, so that the storage space required by the compressed panoramic video is reduced.
Drawings
FIG. 1 is a flow chart of a preferred embodiment of a method for optimizing the storage of panoramic video in accordance with the present invention;
FIG. 2 is a flowchart of step S10 in a preferred embodiment of the panoramic video storage optimization method of the present invention;
FIG. 3 is a schematic view of an isoplanar projection format panoramic image according to a preferred embodiment of the present invention;
FIG. 4 is a schematic diagram of recognition results of an object recognition algorithm of the panoramic video storage optimization method of the present invention;
FIG. 5 is a flow chart of the time length of determining panoramic video blocks for the storage optimization method of panoramic video of the present invention;
FIG. 6 is a flowchart of step S20 in a preferred embodiment of the panoramic video storage optimization method of the present invention;
FIG. 7 is a schematic diagram of semantic segmentation results of an equidistant projection format of a panoramic video storage optimization method according to the present invention;
FIG. 8 is a schematic view of a stereoscopic projection format panoramic image in accordance with a preferred embodiment of the invention;
FIG. 9 is a schematic diagram of semantic segmentation results in a cube projection format in accordance with a preferred embodiment of the present invention;
FIG. 10 is a schematic view of a panoramic video block projection process of the storage optimization method of panoramic video of the present invention;
FIG. 11 is a flowchart of step S30 in a preferred embodiment of the panoramic video storage optimization method of the present invention;
FIG. 12 is a schematic view of a panoramic image of a stereoscopic projection coordinate system rotated 45 ° horizontally according to a preferred embodiment of the present invention;
FIG. 13 is an overall schematic of a preferred embodiment of a panoramic video storage optimization system of the present invention;
FIG. 14 is a schematic diagram of a preferred embodiment of a panoramic video storage optimization system of the present invention;
FIG. 15 is a schematic view of the operating environment of a preferred embodiment of the terminal of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more clear and clear, the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
It should be noted that, if directional indications (such as up, down, left, right, front, and rear are referred to in the embodiments of the present invention), the directional indications are merely used to explain the relative positional relationship, movement conditions, and the like between the components in a specific posture (as shown in the drawings), and if the specific posture is changed, the directional indications are correspondingly changed.
In addition, if there is a description of "first", "second", etc. in the embodiments of the present invention, the description of "first", "second", etc. is for descriptive purposes only and is not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions of the embodiments may be combined with each other, but it is necessary to base that the technical solutions can be realized by those skilled in the art, and when the technical solutions are contradictory or cannot be realized, the combination of the technical solutions should be considered to be absent and not within the scope of protection claimed in the present invention.
The method for optimizing the storage of the panoramic video according to the preferred embodiment of the present invention, as shown in fig. 1, comprises the following steps:
Step S10, a first panoramic image in a panoramic video is obtained, an object in the first panoramic image is identified, the movement range of the object is calculated, and the panoramic video is segmented into panoramic video blocks.
Fig. 2 is a flowchart of step S10 in the panoramic video storage optimization method according to the present invention.
As shown in fig. 2, the step S10 includes:
step S11, uniformly sampling a panoramic video to obtain panoramic images of a plurality of frames, identifying objects in each frame of panoramic image by adopting an object identification algorithm on the panoramic images, and marking the same objects in continuous frames;
Step S12, when the center point of the object is used as the gravity center of the object, setting a time length, and calculating a gravity center moving distance from a first frame of each object to a frame after the time length;
step S13, calculating an average value of the gravity center moving distance, taking the average value as an average moving range of all objects in the time length, and setting a threshold value of the average range;
and S14, judging the average motion range and the threshold value, and if the average motion range is equal to the threshold value, dividing the panoramic video into a plurality of sections of panoramic video blocks on a time axis by using the time length.
Specifically, a plurality of frames of panoramic images are obtained by uniformly sampling from panoramic video, an object recognition algorithm is used for recognizing objects in each frame of panoramic images, the same objects in continuous frames are marked, for example, in the embodiment of the invention, an image 3 (the panoramic images in the equidistant projection format are shown in fig. 3) is used as input, the objects in the panoramic images are output and are marked by using the object recognition algorithm, the marked result is shown in fig. 4, wherein the object recognition algorithm can recognize and mark the positions of the objects in the images and the types of the objects according to the content of the images, the object recognition algorithm in the embodiment of the invention can use but is not limited to Yolo-v4 algorithm, the center point of each object is assumed to be used as the center of gravity of the panoramic images, then a time length (for example, 1 s) is preset, the average value of the center of gravity movement distance (great circle distance) between the first frame of each object and one frame after 1s is calculated, the average value of the center of gravity movement distance is calculated, the average value of the center of gravity movement distance is used as the average movement range of all objects in the time length is set, the threshold value of the average movement range is set, the threshold value of the average range is dependent on the projection format, (for example, the average value of the average value is 90 DEG is not being equal to the average value is calculated, the average value of the average value is not is 60 DEG is calculated, and the average value of the average movement range is not is calculated when the average value is 60 DEG is equal to the average value is 60 DEG is shorter than the average value is 60 DEG, the average value is 60 DEG when the average value is not equal to the average value, and if the time length is close to or equal to the threshold value, dividing the panoramic video into a plurality of sections of panoramic video blocks on a time axis by using the time length.
Further, as shown in the time length flowchart of determining panoramic video blocks in fig. 5, step S101 is to uniformly sample a plurality of frames of panoramic images from the panoramic video, step S102 is to identify objects in each frame of panoramic images by using an object identification algorithm and mark the same objects in consecutive frames, step S103 is to calculate an average motion range of all objects in a period of time, step S104 is to judge the size of the average motion range, if the size exceeds a threshold, step S103 is repeated, if the size exceeds the threshold, the time length is increased, step S103 is repeated, if the size is less than the threshold, step S105 is continued, and step S105 is to divide the panoramic video into a plurality of pieces of panoramic video blocks on a time axis by using the time length.
And step S20, acquiring a second panoramic image based on the panoramic video block, dividing the second panoramic image to obtain a semantic division result, and projecting the semantic division result to obtain a projection plane.
Fig. 6 is a flowchart of step S20 in the panoramic video storage optimization method according to the present invention.
As shown in fig. 6, the step S20 includes:
S21, extracting a second panoramic image of a plurality of frames from the panoramic video block, and inputting the second panoramic image into a semantic segmentation algorithm for segmentation to obtain a semantic segmentation result;
step S22, based on the projection format of the semantic segmentation result, selecting a projection coordinate system corresponding to the projection format, and projecting the semantic segmentation result by using the projection coordinate system to obtain a plurality of projection planes.
Specifically, sampling a plurality of frames of panoramic images from a panoramic video block, taking the extracted panoramic images as input, and outputting semantic segmentation results by using a semantic segmentation algorithm, wherein the semantic segmentation algorithm can identify pixels of a foreground object from the images, in the embodiment of the invention, the foreground pixels are defined as common objects of interest such as people, automobiles, etc. (for example, using fig. 3 as input, using the semantic segmentation algorithm to output rectangular projection format semantic segmentation results such as shown in fig. 7, wherein white parts of the rectangular projection format semantic segmentation results are foreground pixels, and in the embodiment of the invention, the foreground pixels are defined as common objects of interest such as people, automobiles, etc.), it is worth noting that the semantic segmentation algorithm can be used but is not limited to ugscnn; selecting a corresponding projection coordinate system according to a projection format, wherein the projection format comprises an equidistant projection format and a cubic projection format, fig. 3 is an equal rectangular projection format panoramic image, fig. 8 is a cubic projection format panoramic image, inputting an extracted panoramic image semantic segmentation result, projecting the semantic segmentation result by using the projection coordinate system to obtain a plurality of projection planes, taking the cubic projection format as an example, projecting to obtain a cubic projection format semantic segmentation result, and the cubic projection format semantic segmentation result is shown as fig. 9, wherein white parts in the figure are foreground pixels, and the foreground pixels are defined as common objects of interest such as people, automobiles and the like in the embodiment of the invention.
Further, fig. 10 is a view of a panoramic video projection overall process, sampling a plurality of frames of panoramic images from a panoramic video block, taking the extracted panoramic images as input, dividing the panoramic images by using a semantic division algorithm, outputting semantic division results, selecting a corresponding projection coordinate system according to a projection format, inputting the extracted semantic division results of the panoramic images, projecting the semantic division results by using the projection coordinate system to obtain a plurality of projection planes, sequentially inputting the projection planes according to a time axis, outputting a multi-degree-of-freedom rotation angle by using a reinforcement learning method, rotating the projection coordinate system by using the multi-degree-of-freedom rotation angle, projecting the panoramic video block into a plane format by using the rotated projection coordinate system, compressing the panoramic video block, and inputting the compressed panoramic video block into the reinforcement learning method in a rewarding feedback mode.
And step S30, performing reinforcement learning processing on the projection plane, projecting the panoramic video block into a plane format, and then compressing and storing the panoramic video block.
Fig. 11 is a flowchart of step S30 in the panoramic video storage optimization method according to the present invention.
As shown in fig. 11, the step S30 includes:
S31, inputting the projection planes into a reinforcement learning method according to a time axis sequence for processing, and outputting a multi-degree-of-freedom rotation angle after the processing is finished;
Step S32, completing rotation of the projection coordinate system based on the multi-degree-of-freedom rotation angle, and projecting the panoramic video block by using the rotated projection coordinate system to obtain a first panoramic video block in a planar format;
and step S33, compressing and storing the first panoramic video block, and storing the multiple free rotation angles of the first panoramic video block in a record file.
The reinforcement learning method in the embodiment of the invention can be used, but is not limited to, a continuous motion space reinforcement learning method example DDPG or a discrete motion space reinforcement learning method example DQN, the projection coordinate system is rotated by using the outputted multi-degree-of-freedom rotation angle (for example, a panoramic image with a cube projection format can be rotated in a horizontal direction and a vertical direction to a projection coordinate system), the panoramic video block is projected into a plane format by using the rotated projection coordinate system and compressed, the panoramic video block is rotated to the right by the horizontal direction by 45 degrees by taking the cube projection format as an example, the projection image 3 is obtained in fig. 12, the foreground objects such as people and automobiles in fig. 12 are well stored in the projection plane, only a small part of the foreground objects span different projection planes, and the compression rate of the panoramic video block is further improved, and the multi-degree-of-freedom rotation angle of the panoramic video block is stored in a record file for subsequent rendering.
In consideration of the fact that a plurality of proper multi-freedom rotation angles possibly exist in the process of rotating the projection coordinate system, the projection coordinate system is subjected to repeated iterative rotation, panoramic video is required to be compressed after the projection coordinate system is rotated each time, compression rates of panoramic video blocks under different projection coordinate systems are compared, panoramic video blocks with the largest compression rate are selected and stored, and multi-freedom rotation angles which cause the largest compression rate are recorded in a file. In order to prevent the infinite number of iterative rotations, the output rotation angle may be determined to be 0, and the upper iteration limit may be manually set.
Further, since the polyhedron has isomerism, the spatial rotation angle of reinforcement learning output is required to be limited, the arrangement sequence and the direction of a projection plane are only changed by an integral multiple of 90 degrees of horizontal rotation by taking a cube projection format as an example, the limiting method is specifically based on the periodical change of the rotation angle to obtain the change range of the rotation angle in one period, the horizontal rotation period is [0 degrees, 90 degrees ] by taking the cube projection format as an example, the rotation angle output by the reinforcement learning method is limited in the change range of one period, if the limited spatial rotation angle is 0, the reasoning process of the current panoramic video block is ended, and any degree of freedom is mutually independent due to the fact that the limiting method is applicable to any degree of freedom.
Further, the panoramic video block subjected to the rotation processing is rendered.
The method comprises the steps of receiving an access request of a user, transmitting a panoramic video block to a user client based on the access request, extracting multiple free rotation angles corresponding to the panoramic video block from a record file and transmitting the multiple free rotation angles to the user client when the panoramic video block is transmitted, rotating a projection coordinate system based on the multiple free rotation angles after the user client finishes receiving, and projecting the rotated projection coordinate system on the panoramic video block to finish a rendering process.
Further, the overall flow of the preferred embodiment of the panoramic video storage optimization system of the present invention is shown in fig. 13, and the specific steps are as follows:
step S501, inputting a plurality of frames of panoramic images in a panoramic video, and outputting an object identification result of each frame by using an object identification algorithm;
Step S502, judging the motion range of an object, and determining the time length of each panoramic video block;
step S503, dividing the panoramic video into a plurality of sections of panoramic video blocks on a time axis;
step S504, extracting a plurality of frames of panoramic images from the panoramic video block as input, and outputting semantic segmentation results by using a semantic segmentation algorithm;
Step S505, projecting the semantic segmentation results corresponding to all the extracted panoramic images to obtain a plurality of projection planes;
step S506, inputting projection planes sequentially according to a time axis, and outputting a multi-degree-of-freedom rotation angle by using a reinforcement learning method;
step S507, rotating the projection coordinate system by using the multi-degree-of-freedom rotation angle;
and S508, projecting the panoramic video block into a planar format by using the rotated projection coordinate system and compressing the planar format.
Further, as shown in fig. 14, based on the above-mentioned method for optimizing the storage of the panoramic video, the present invention further provides a system for optimizing the storage of the panoramic video, where the system for optimizing the storage of the panoramic video includes:
the video segmentation module 51 is configured to obtain a first panoramic image in a panoramic video, identify an object in the first panoramic image, calculate a motion range of the object, and segment the panoramic video into panoramic video blocks;
the video projection module 52 is configured to obtain a second panoramic image based on the panoramic video block, divide the second panoramic image to obtain a semantic division result, and project the semantic division result to obtain a projection plane;
and the compression storage module 53 is used for performing reinforcement learning processing on the projection plane, projecting the panoramic video block into a plane format, and then performing compression storage.
Further, as shown in fig. 15, the invention further provides a terminal based on the above-mentioned panoramic video storage optimization method and system, and the terminal includes a processor 10, a memory 20 and a display 30. Fig. 15 shows only some of the components of the terminal, but it should be understood that not all of the illustrated components are required to be implemented and that more or fewer components may be implemented instead.
The memory 20 may in some embodiments be an internal storage unit of the terminal, such as a hard disk or a memory of the terminal. The memory 20 may in other embodiments also be an external storage device of the terminal, such as a plug-in hard disk provided on the terminal, a smart memory card (SMART MEDIA CARD, SMC), a Secure Digital (SD) card, a flash memory card (FLASH CARD), etc. Further, the memory 20 may also include both an internal storage unit and an external storage device of the terminal. The memory 20 is used for storing application software installed in the terminal and various data, such as program codes of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, the memory 20 stores a storage optimization program 40 of the panoramic video, and the storage optimization program 40 of the panoramic video may be executed by the processor 10, so as to implement the storage optimization method of the panoramic video in the present application.
The processor 10 may in some embodiments be a central processing unit (Central Processing Unit, CPU), microprocessor or other data processing chip for running program code or processing data stored in the memory 20, for example performing a memory optimization method for the panoramic video, etc.
The display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, or the like in some embodiments. The display 30 is used for displaying information at the terminal and for displaying a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.
In one embodiment, the following steps are implemented when the processor 10 executes the storage optimization program 40 of the panoramic video in the memory 20:
acquiring a first panoramic image in a panoramic video, identifying an object in the first panoramic image, calculating the motion range of the object, and dividing the panoramic video into panoramic video blocks;
acquiring a second panoramic image based on the panoramic video block, dividing the second panoramic image to obtain a semantic division result, and projecting the semantic division result to obtain a projection plane;
And performing reinforcement learning processing on the projection plane, projecting the panoramic video block into a plane format, and then compressing and storing the panoramic video block.
The method for obtaining the panoramic video comprises the steps of obtaining a first panoramic image in the panoramic video, identifying an object in the first panoramic image, calculating the movement range of the object, and dividing the panoramic video into panoramic video blocks, and specifically comprises the following steps:
Uniformly sampling the panoramic video to obtain panoramic images of a plurality of frames, identifying objects in each frame of panoramic image by adopting an object identification algorithm on the panoramic images, and marking the same objects in continuous frames;
When the center point of an object is used as the center of gravity of the object, setting a time length, and calculating to obtain the center of gravity moving distance between a first frame of each object and a frame after the time length;
calculating an average value of the gravity center moving distance, taking the average value as an average moving range of all objects in the time length, and setting a threshold value of the average range;
And judging the average motion range and the threshold value, and if the average motion range is equal to the threshold value, dividing the panoramic video into a plurality of sections of panoramic video blocks on a time axis by using the time length.
The size of the threshold is set according to the projection format.
Wherein the determining the average motion range and the threshold value further includes:
If the average motion range is larger than the threshold value, shortening the time length, and recalculating the average motion range of all objects in the shortened time length;
if the average motion range is smaller than the threshold value, the time length is increased, and the average motion range of all objects in the increased time length is recalculated.
The method specifically includes the steps of obtaining a second panoramic image based on the panoramic video block, dividing the second panoramic image to obtain a semantic division result, and projecting the semantic division result to obtain a projection plane, wherein the method specifically comprises the following steps:
Extracting a second panoramic image of a plurality of frames from the panoramic video block, and inputting the second panoramic image into a semantic segmentation algorithm for segmentation to obtain a semantic segmentation result;
And selecting a projection coordinate system corresponding to the projection format based on the projection format of the semantic segmentation result, and projecting the semantic segmentation result by using the projection coordinate system to obtain a plurality of projection planes.
Wherein the projection formats include equidistant shape projection formats and cubic projection formats.
The reinforcement learning processing is performed on the projection plane, and the panoramic video block is projected into a plane format and then compressed and stored, which specifically comprises the following steps:
Inputting the projection planes into a reinforcement learning method according to a time axis sequence for processing, and outputting a multi-degree-of-freedom rotation angle after the processing is finished;
finishing the rotation of the projection coordinate system based on the multi-degree-of-freedom rotation angle, and projecting the panoramic video block by using the rotated projection coordinate system to obtain a first panoramic video block in a planar format;
And compressing and storing the first panoramic video block, and storing the multi-free rotation angle of the first panoramic video block in a record file.
Wherein the rotation of the projection coordinate system based on the multiple degrees of freedom rotation angle is completed, further comprising:
If a plurality of proper multi-free rotation angles exist, carrying out repeated iterative rotation on the projection coordinate system, and projecting the panoramic video block by the projection coordinate system after each rotation to obtain a second panoramic video block with a plurality of plane formats;
compressing all the second panoramic video blocks, and recording the compression rate of each second panoramic video block;
And selecting a second panoramic video block with the highest compression ratio for storage, and recording the multi-degree-of-freedom rotation angle of the second panoramic video block in a file.
Wherein the rotation of the projection coordinate system based on the multiple degrees of freedom rotation angle is completed, and then further comprises:
according to the periodic change of the rotation angle, a change range of the rotation angle in one period is obtained;
limiting the rotation angle output by the reinforcement learning method to be in the variation range of the period based on the variation range;
and if the limited rotation angle is 0, ending the processing of the current panoramic video block.
The compressing and storing the first panoramic video block, and storing the multiple free rotation angles of the first panoramic video block in a record file, and then further includes:
Receiving an access request of a user, and transmitting a panoramic video block to a user client based on the access request;
when the panoramic video block is transmitted, extracting a multi-free rotation angle corresponding to the panoramic video block from the record file, and transmitting the multi-free rotation angle to the user client;
And after the user client finishes receiving, rotating a projection coordinate system based on the multiple free rotation angles, and projecting the panoramic video block by the rotated projection coordinate system to finish the rendering process.
The present invention also provides a computer readable storage medium storing a storage optimization program of panoramic video, which when executed by a processor, implements the steps of the storage optimization method of panoramic video as described above.
In summary, the invention provides a storage optimization method of panoramic video, which comprises the steps of obtaining a first panoramic image in the panoramic video, identifying an object in the first panoramic image, calculating the movement range of the object, dividing the panoramic video into panoramic video blocks, obtaining a second panoramic image based on the panoramic video blocks, dividing the second panoramic image to obtain a semantic division result, projecting the semantic division result to obtain a projection plane, performing reinforcement learning processing on the projection plane, projecting the panoramic video blocks into a plane format, and then compressing and storing the panoramic video blocks. According to the invention, the panoramic video is divided into a plurality of sections of panoramic video blocks on a time axis, the projection coordinate system is subjected to multi-degree-of-freedom spatial rotation according to the content of each section of panoramic video block, and then the panoramic video is projected into a plane format by using the rotated projection coordinate system and compressed, so that the storage space required by the compressed panoramic video is reduced.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Of course, those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by a computer program for instructing relevant hardware (e.g., processor, controller, etc.), the program may be stored on a computer readable storage medium, and the program may include the above described methods when executed. The computer readable storage medium may be a memory, a magnetic disk, an optical disk, etc.
It is to be understood that the invention is not limited in its application to the examples described above, but is capable of modification and variation in light of the above teachings by those skilled in the art, and that all such modifications and variations are intended to be included within the scope of the appended claims.
Claims (11)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210886773.9A CN115720263B (en) | 2022-07-26 | 2022-07-26 | Panoramic video storage optimization method, system, terminal and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210886773.9A CN115720263B (en) | 2022-07-26 | 2022-07-26 | Panoramic video storage optimization method, system, terminal and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115720263A CN115720263A (en) | 2023-02-28 |
| CN115720263B true CN115720263B (en) | 2025-09-09 |
Family
ID=85253881
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210886773.9A Active CN115720263B (en) | 2022-07-26 | 2022-07-26 | Panoramic video storage optimization method, system, terminal and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115720263B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116958228A (en) * | 2023-07-06 | 2023-10-27 | 长春理工大学重庆研究院 | Panoramic image monocular depth estimation method based on double-branch efficient fusion network |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105979267A (en) * | 2015-12-03 | 2016-09-28 | 乐视致新电子科技（天津）有限公司 | Video compression and play method and device |
| CN107622468A (en) * | 2016-07-11 | 2018-01-23 | 完美幻境(北京)科技有限公司 | A kind of panoramic video storage format, the storage method of panoramic video and device |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103686095B (en) * | 2014-01-02 | 2017-05-17 | 中安消技术有限公司 | Video concentration method and system |
| CN106851398A (en) * | 2017-03-10 | 2017-06-13 | 北京小鸟看看科技有限公司 | The storage method of panoramic video file, player method, storage device and player |
| CN109801307A (en) * | 2018-12-17 | 2019-05-24 | 中国科学院深圳先进技术研究院 | A kind of panorama dividing method, device and equipment |
-
2022
- 2022-07-26 CN CN202210886773.9A patent/CN115720263B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105979267A (en) * | 2015-12-03 | 2016-09-28 | 乐视致新电子科技（天津）有限公司 | Video compression and play method and device |
| CN107622468A (en) * | 2016-07-11 | 2018-01-23 | 完美幻境(北京)科技有限公司 | A kind of panoramic video storage format, the storage method of panoramic video and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115720263A (en) | 2023-02-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11501507B2 (en) | Motion compensation of geometry information | |
| US12154280B2 (en) | Determining multiple camera positions from multiple videos | |
| CN108353208B (en) | Optimizing media fingerprint retention to improve system resource utilization | |
| CN112073748B (en) | Panoramic video processing method and device and storage medium | |
| CN104995660B (en) | Redundant Pixel Mitigation | |
| CN110111364B (en) | Motion detection method and device, electronic equipment and storage medium | |
| US20220113675A1 (en) | Delivering pre-rendered holographic content over a network | |
| KR20210096234A (en) | Point cloud coding using homography transformation | |
| CN110516598A (en) | Method and apparatus for generating image | |
| CN112825195B (en) | Method and device for determining placement position of article and computer readable storage medium | |
| TW201911240A (en) | Image processing device and method, file generating device and method, and program | |
| CN115720263B (en) | Panoramic video storage optimization method, system, terminal and storage medium | |
| US20210092345A1 (en) | Unified coding of 3d objects and scenes | |
| CN111510717A (en) | Image stitching method and device | |
| US20210150788A1 (en) | Media vectorization | |
| US20200092444A1 (en) | Playback method, playback device and computer-readable storage medium | |
| US8885952B1 (en) | Method and system for presenting similar photos based on homographies | |
| JP6842618B2 (en) | Creating a 3D map | |
| CN112601029B (en) | Video segmentation method, terminal and storage medium with known background prior information | |
| CN114565872A (en) | Video data processing method, device, equipment and computer readable storage medium | |
| CN111885417B (en) | VR video playing method, device, equipment and storage medium | |
| CN118334255B (en) | High-resolution image three-dimensional reconstruction method, system and medium based on deep learning | |
| CN108920598B (en) | Panorama browsing method and device, terminal equipment, server and storage medium | |
| CN111225293A (en) | A video data processing method, device and computer storage medium | |
| CN114257748A (en) | Video anti-shake method and device, computer readable medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |