JP6889653B2 - Video compression device and video compression method - Google Patents

Description

The present invention relates to a moving image compression device and a moving image compression method for compressing a moving image using a machine learning model.

A device that compresses a moving image by predicting the movement of a subject in the moving image is known. Patent Document 1 discloses a technique for compressing a moving image by predicting the movement of a subject in consideration of deformation.

Japanese Unexamined Patent Publication No. 2012-080151

Internationally standardized moving image compression coding methods such as MPEG (Moving Picture Experts Group) predict the movement of the subject in units of rectangular blocks that divide the frame of the moving image into predetermined units and perform motion compensation. ing. In this case, if the subject is deformed, the prediction error becomes large, and there arises a problem that the compression rate of the moving image decreases. The prediction error is the difference between the image obtained by applying the motion prediction to the reference frame image and the frame image to be compensated for the motion. By using the conventional technique, it is possible to predict the movement of the block in consideration of the deformation based on the parameters of the geometric transformation (for example, affine transformation and co-linear transformation) calculated for each block. However, since the deformation of the subject is generally irregular, motion compensation using geometric transformation is only an approximation. In addition, different subjects appearing in different frame images may be erroneously detected as corresponding subjects. As a result, the prediction error becomes large and the compression rate of the moving image decreases.

Therefore, the present invention has been made in view of these points, and an object of the present invention is to provide a moving image compression device and a moving image compression method capable of improving the moving image compression rate.

The moving image compression device according to the first aspect of the present invention includes a moving image acquisition unit that acquires a moving image generated by an imaging device, a reference frame image that is a frame image included in the moving image, and the moving image. A plurality of processes included in a machine learning model capable of outputting the type of the subject included in the moving image based on the input moving image for each of the specific frame images that are frame images other than the included reference frame image. The propagation control unit that propagates the layers, the post-stage treatment layer selected from the plurality of treatment layers, and the pre-stage treatment layer that is the treatment layer immediately before the post-stage treatment layer are commonly activated. One or more specific outputs output from the post-stage processing layer and the pre-stage processing layer based on the specific frame image, and one or more output from the post-stage processing layer and the pre-stage processing layer based on the reference frame image. An extraction unit that extracts a reference output, one or more specific feature points are detected based on the one or more specific outputs, and one or more reference feature points are detected based on the one or more reference outputs. A vector estimation unit that estimates a vector in which a subject moves between frame images by comparing the feature point detection unit with one or more specific feature points and the one or more reference feature points, and the vector estimation unit. It has a compression unit that generates a compressed moving image obtained by compressing the moving image based on the result estimated by the unit.

The extraction unit has a plurality of post-stage identification outputs output from the post-stage processing layer by propagating the specific frame image in the order of the pre-stage processing layer and the post-stage processing layer, which are a part of the plurality of processing layers. One or more post-stage identification outputs and one or more commonly activated post-stage reference outputs from a plurality of post-stage reference outputs output from the post-stage processing layer due to the propagation of the reference frame image in the order of the pre-stage processing layer and the post-stage processing layer. A post-stage extraction unit that extracts the post-stage reference output, and a plurality of pre-stage identification outputs output from the one or more post-stage identification outputs and the pre-stage processing layer that has been a factor in activating the one or more post-stage reference outputs. And even if it has one or more pre-stage identification outputs and one or more pre-stage reference outputs that are commonly activated among the plurality of pre-stage reference outputs output from the pre-stage processing layer, and a pre-stage extraction unit that extracts one or more pre-stage reference outputs. Good.

The pre-stage extraction unit outputs one or more pre-stage identification outputs and one or more pre-stage reference outputs based on the activated magnitude of the plurality of pre-stage identification outputs and the plurality of pre-stage reference outputs. It may be extracted.

The machine learning model includes a convolutional neural network, and the post-processing layer may be any one of an output layer, a fully connected layer, a regularized layer, a pooling layer, and a convolutional layer.
The pretreatment layer may be any one of a fully bonded layer, a normalized layer, a pooling layer, a convolution layer, and an input layer.

When the last layer, which is the last treatment layer, is selected as the latter treatment layer among the plurality of treatment layers, the extraction unit is used for one or more of the above-mentioned identifications that are commonly activated in the last treatment layer. When there is no output and the one or more reference outputs, the one or more specific outputs and the one or more reference outputs that are commonly activated in the treatment layer before the rearmost layer may be extracted.

From the one or more specific feature points and the one or more reference feature points specified by the feature point detection unit, the moving image compression device includes some specific feature points and a part based on mutual correspondence. The vector estimation unit further has a selection unit for selecting the reference feature points of the above, and the vector estimation unit determines a vector on which the subject moves by comparing some of the specific feature points with the part of the reference feature points. You may estimate.

In the selection unit, when the vector estimation unit estimates a vector on which each of the part of the specific feature points included in the subject moves, the one specific feature point and the one specific feature point When the moving vector with the surrounding specific feature points exceeds a predetermined range, the one specific feature point and the reference feature point corresponding to the one specific feature point are used for some of the specific features. It may be removed from the feature points and some of the reference feature points.

The vector estimation unit estimates the vector based on the relationship between the position of the part of the specific feature point in the specific frame image and the position of the part of the reference feature point in the reference frame image. You may.

The moving image compression device further includes an instruction receiving unit that receives an instruction to select a processing layer to be used as the subsequent processing layer among the plurality of processing layers, and the extraction unit receives the instruction receiving unit. The treated layer indicated by the instruction may be used as the post-processed layer.

The extraction unit selects one of the plurality of treated layers as the post-processed layer, extracts the one or more specific outputs and the one or more reference outputs, and then selects the pre-processed layer. The treated layer may be selected as the post-processed layer to extract another one or more specific outputs and one or more reference outputs.

The moving image compression method according to the second aspect of the present invention includes a step of acquiring a moving image generated by an imaging device, a reference frame image which is a frame image included in the moving image, and the moving image included in the moving image. Propagate multiple processing layers included in a machine learning model that can output the type of subject included in the moving image based on the input moving image for each of the specific frame images that are frame images other than the reference frame image. In the specific frame image, which is commonly activated in both the processing layer of the step of performing, the post-stage processing layer selected from the plurality of processing layers, and the pre-stage processing layer which is the processing layer immediately before the post-stage processing layer. Based on this, one or more specific outputs output from the post-stage processing layer and the pre-stage processing layer and one or more reference outputs output from the post-stage processing layer and the pre-stage processing layer are extracted based on the reference frame image. A step of detecting one or more specific feature points based on the one or more specific outputs, and a step of detecting one or more reference feature points based on the one or more reference outputs, and the above one or more. The result of estimation in the step of estimating the vector in which the subject moves between the frame images and the step of estimating the vector in which the subject moves by comparing the specific feature point of the above with one or more reference feature points. Based on the above, there is a step of generating a compressed moving image obtained by compressing the moving image.

In the extraction step, a plurality of post-stage identification outputs output from the post-stage processing layer due to the propagation of the specific frame image in the order of the pre-stage processing layer and the post-stage processing layer, which are a part of the plurality of processing layers. And one or more post-stage identification outputs and one or more commonly activated from the plurality of post-stage reference outputs output from the post-stage processing layer due to the propagation of the reference frame image in the order of the pre-stage processing layer and the post-stage processing layer. A pre-stage extraction step for extracting the post-stage reference output, and a plurality of pre-stage identification outputs output from the pre-stage processing layer that has been a factor in activating the one or more post-stage identification outputs and the one or more post-stage reference outputs. , And one or more pre-stage identification outputs and one or more pre-stage reference outputs that are commonly activated among the plurality of pre-stage reference outputs output from the pre-stage processing layer, and a post-stage extraction step for extracting one or more pre-stage reference outputs. May be good.

In the moving image compression method, after executing the pre-stage extraction step, the one or more pre-stage identification outputs and the one or more pre-stage reference outputs are used as the plurality of rear-stage identification outputs and the plurality of post-stage reference outputs. The latter-stage extraction step may be executed.

In the moving image compression method, the latter-stage extraction step and the first-stage extraction step may be executed for each of the plurality of processing layers.

According to the present invention, there is an effect that the compression rate of moving images can be improved.

It is a figure for demonstrating the outline of the process of compressing a moving image. It is a figure which shows an example of the structure of the machine learning model. It is a figure which shows the structure of the moving image compression apparatus. It is a figure for demonstrating the extraction process performed by an extraction part. It is a figure for demonstrating the extraction process performed by an extraction part. It is a figure for demonstrating the extraction process performed by an extraction part. It is a figure for demonstrating the extraction process performed by an extraction part. It is a figure for demonstrating the extraction process performed by an extraction part. It is a figure for demonstrating the extraction process performed by an extraction part. It is a flowchart which shows the flow of the process performed by a moving image compression apparatus. It is a flowchart which shows the flow of the process performed by the extraction part.

[Overview of video compression device 1]
FIG. 1 is a diagram for explaining an outline of a process of compressing a moving image. The moving image compression device 1 is, for example, a PC (Personal Computer). The moving image compression device 1 is a device that compresses a moving image by estimating a vector in which a subject appearing in a specific frame image moves based on a reference frame image using a machine learning model M. The reference frame image is one frame image selected from a plurality of frame images included in the moving image. The specific frame image is a frame image other than the reference frame image, and is one or more frame images before and after the reference frame image. The vector is a motion vector indicating the direction and amount of movement of the subject based on the reference frame image.

The moving image compression device 1 acquires a moving image generated by the imaging device ((1) in FIG. 1). The image pickup device is a camera, a smartphone, or the like capable of capturing a moving image. The moving image A shown in FIG. 1 is an image showing a person who is the subject T running from the left side to the right side. The frame image A1 of the first frame included in the moving image A shown in FIG. 1 is used as a reference frame image, and the frame images A2, A3, A4, and A5 from the second frame to the fifth frame are used as specific frame images.

The moving image compression device 1 inputs the acquired moving image A into the machine learning model M and propagates a plurality of processing layers included in the machine learning model M ((2) in FIG. 1). The machine learning model M is a model learned to output the type of the subject included in the moving image based on the input moving image. Specifically, the machine learning model M outputs the type of the subject shown in each frame image of the moving image for each frame image based on the input moving image.

FIG. 2 is a diagram showing an example of the configuration of the machine learning model M. The machine learning model M includes a convolutional neural network (CNN). In this case, the machine learning model M is an input layer M1, a first convolution layer M2, a second convolution layer M3, a first pooling layer M4, a regularization layer M5, a third convolution layer M6, and a second. It has a pooling layer M7, a first fully connected layer M8, a second fully connected layer M9, and an output layer M10. In the present specification, of the two adjacent processing layers, the processing layer on the upstream side when the specific frame image and the reference frame image propagate is referred to as a pre-stage processing layer, and the processing layer on the downstream side is referred to as a post-stage processing layer. Refer to.

The treatment layers that can be the subsequent treatment layers are the first convolution layer M2, the second convolution layer M3, the first pooling layer M4, the normalization layer M5, the third convolution layer M6, the second pooling layer M7, and the second. It is one of the fully bonded layer M8 of 1, the second fully bonded layer M9, and the output layer M10. The processing layers that can be the pre-stage processing layers are the input layer M1, the first convolution layer M2, the second convolution layer M3, the first pooling layer M4, the normalization layer M5, the third convolution layer M6, and the second. It is any one of the pooling layer M7, the first fully bonded layer M8, and the second fully bonded layer M9. The moving image compression device 1 inputs the acquired moving image to the machine learning model M, and forward-propagates each processing layer from the input layer M1 to the output layer M10, that is, infers the moving image, so that the moving image is reflected in the moving image. Output the type of subject T.

Returning to FIG. 1, in the moving image compression device 1, the machine learning model M outputs the type of the subject T (for example, human, male, female, etc.), the calculation result in each processing layer, that is, the abstraction by deep learning. Using a high-degree feature amount, feature points common to the specific frame image and the reference frame image are detected ((3) in FIG. 1). The moving image compression device 1 selects, among a plurality of frame images included in the moving image A, a group of frame images in which subjects of at least the same type are continuously captured as a specific frame image and a reference frame image. .. Here, the moving image compression device 1 detects common feature points in an order opposite to the order in which each frame image is forward-propagated. By doing so, the moving image compression device 1 can detect feature points based on features with a high degree of abstraction.

The moving image compression device 1 detects the common feature points (for example, the feature points included in the face, body, limbs, etc.), and thereby, the feature points and the reference frame included in the subject T appearing in the specific frame image. It is detected that there is a correspondence relationship between the feature points included in the subject T in the image. The correspondence relationship is a relationship in which the pixels included in the specific frame image indicated by the feature points and the pixels included in the reference frame image indicated by the feature points match or approximate.

The moving image compression device 1 estimates the motion vector of the subject T between the frame images by comparing the feature points of the detected corresponding identification frame image and the reference frame image ((4) in FIG. 1). ). Then, the moving image compression device 1 compresses the moving image A by performing motion compensation for the specific frame image based on the estimated motion vector of the subject T ((5) in FIG. 1). By doing so, the moving image compression device 1 can improve the compression rate of the moving image.
Hereinafter, the details of the moving image compression device 1 will be described.

[Configuration of video compression device 1]
FIG. 3 is a diagram showing the configuration of the moving image compression device 1. The moving image compression device 1 has an operation unit 11, a storage unit 12, and a control unit 13.

The operation unit 11 is an input device that accepts user operations.
The storage unit 12 is, for example, a storage medium such as a ROM (Read Only Memory), a RAM (Random Access Memory), and a hard disk. The storage unit 12 stores various programs executed by the control unit 13. The storage unit 12 stores a moving image.

The control unit 13 is, for example, a CPU (Central Processing Unit). The control unit 13 controls the function related to the moving image compression device 1 by executing the program stored in the storage unit 12. By executing the program, the control unit 13 executes a moving image acquisition unit 131, a propagation control unit 132, an extraction unit 133, an instruction reception unit 136, a feature point detection unit 137, a selection unit 138, a vector estimation unit 139, and a compression unit. Functions as 140.
The moving image acquisition unit 131 acquires a moving image generated by the imaging device stored in the storage unit 12. The moving image acquisition unit 131 inputs the acquired moving image to the propagation control unit 132.

The propagation control unit 132 propagates a plurality of processing layers included in the machine learning model M to each of the specific frame image and the reference frame image included in the moving image input from the moving image acquisition unit 131. In the example shown in FIG. 2, the propagation control unit 132 first propagates each processing layer from the input layer M1 to the output layer M10 included in the machine learning model M to each of the plurality of frame images included in the moving image in order. Let me. Then, the propagation control unit 132 selects the specific frame image and the reference frame image based on the type of the subject for each frame image output from the machine learning model M. The propagation control unit 132 selects, for example, a group of frame images in which subjects of the same type are continuously captured among a plurality of frame images included in the moving image as a specific frame image and a reference frame image. The propagation control unit 132 inputs information indicating the selected specific frame image and the reference frame image to the extraction unit 133.

The extraction unit 133 is based on a specific frame image that is commonly activated in both the processing layer of the post-stage processing layer selected from the plurality of processing layers and the processing layer of the pre-stage processing layer that is the processing layer immediately before the post-stage processing layer. One or more specific outputs output from the post-processing layer and the pre-processing layer and one or more reference outputs output from the post-processing layer and the pre-processing layer are extracted based on the reference frame image. The details of the extraction process performed by the extraction unit 133 will be described later, but the extraction unit 133 is a part of the post-stage specification output and the reference output which are a part of the specific output commonly activated in the post-stage processing layer. The post-stage extraction unit 134 that extracts the post-stage reference output, the pre-stage specific output that is a part of the specific output that is commonly activated in the pre-stage processing layer, and the pre-stage extraction that extracts the pre-stage reference output that is a part of the reference output It has a part 135.

The specific output and the reference output extracted by the extraction unit 133 are information indicating the activated unit among the plurality of units included in the processing layer. A unit is one or more pixels contained in a frame image. The definition of activation may be, for example, when the product of the output value of the unit or the output value of the unit and the weight of the connection of the unit exceeds a predetermined threshold value, or a predetermined number or a predetermined number in descending order of output. It may be included in the ratio. Further, in the processing layer other than the fully connected layer, for example, it may be included in a predetermined number or a predetermined ratio in descending order of output for each channel. The channel is the output calculated by convolution for each filter.

It is preferable that the extraction unit 133 selects the last treatment layer, which is the last treatment layer, as the subsequent treatment layer among the plurality of treatment layers. However, there are cases where there is no specific output and reference output that are commonly activated in the rearmost layer. Therefore, when the last layer is selected as the subsequent treatment layer among the plurality of treatment layers, the extraction unit 133 has one or more specific outputs and one or more reference outputs that are commonly activated in the last layer. If not, one or more specific outputs and one or more reference outputs that are commonly activated in the treatment layer before the last layer may be extracted.

For example, when the extraction unit 133 selects the output layer M10, which is the rearmost layer, as the post-processing layer, there must be one or more specific outputs and one or more reference outputs that are commonly activated in the output layer M10. To do. In this case, the extraction unit 133 repeatedly searches for one or more specific outputs and one or more reference outputs that are commonly activated for each processing layer before the output layer M10. The extraction unit 133 is, for example, when there is one or more specific outputs and one or more reference outputs that are commonly activated in the second fully connected layer M9, which is the processing layer immediately before the output layer M10. The fully bonded layer M9 is selected as the post-treatment layer. Then, the extraction unit 133 extracts one or more specific outputs and one or more reference outputs that are commonly activated in the second fully connected layer M9 selected as the post-processing layer. By doing so, the extraction unit 133 can associate the subjects to be captured in the specific frame image and the reference frame image even when there are few matching areas.

The extraction unit 133 may select the processing layer specified by the user as the subsequent processing layer. Specifically, the instruction receiving unit 136 receives an instruction to select a processing layer to be used as the subsequent processing layer from the plurality of processing layers via the operation unit 11. Then, the extraction unit 133 uses the processing layer indicated by the instruction received by the instruction reception unit 136 as the subsequent processing layer. In the example shown in FIG. 2, the extraction unit 133 uses the second fully connected layer M9 indicated by the instruction received by the instruction receiving unit 136 as the subsequent processing layer when the user selects the second fully connected layer M9. use. The extraction unit 133 inputs the extracted specific output and the reference output to the feature point detection unit 137.

The feature point detection unit 137 detects one or more specific feature points based on one or more specific outputs, and detects one or more reference feature points based on one or more reference outputs. Specifically, the feature point detection unit 137 first searches for a corresponding feature point based on one or more specific outputs and one or more reference outputs. Then, the feature point detection unit 137 detects one or more specific feature points based on one or more specific outputs and one or more reference feature points based on one or more reference outputs. The feature point detection unit 137 inputs the detected specific feature point and the reference feature point to the selection unit 138.

The selection unit 138 is composed of one or more specific feature points and one or more reference feature points specified by the feature point detection unit 137, and some specific feature points and some reference feature points based on mutual correspondence. Select. Specifically, the selection unit 138 removes the erroneously detected correspondence, and selects one or more specific feature points and one or more reference feature points based on the correspondence after the removal. False detection of the correspondence relationship is a state in which the correspondence relationship between the specific feature point and the reference feature point is inconsistent, for example, when the conversion formula of the geometric transformation cannot be calculated. The selection unit 138 removes the correspondence by, for example, narrowing down based on the RANSAC (Random Sampling Consensus) method or the least squares median (LMedS: Least Median of Square) method.

In the selection unit 138, when the vector estimation unit 139 estimates a vector in which some of the specific feature points included in the subject move, the selection unit 138 surrounds one specific feature point and the one specific feature point. When the moving vector with a certain specific feature point exceeds a predetermined range, one specific feature point and a reference feature point corresponding to one specific feature point are selected as some specific feature points and a part. It may be removed from the reference feature point of. For example, the motion vector of one identification feature point corresponding to one reference feature point located in the center corresponds to a plurality of reference feature points around the one reference feature point. It is assumed that the motion vectors of the feature points do not match or approximate each other. In this case, the selection unit 138 determines that the motion vector of one specific feature point exceeds a predetermined range, and determines one specific feature point and one reference feature point corresponding to the specific feature point. Remove. By doing so, the selection unit 138 can prevent the prediction error from becoming large due to the erroneous detection of the correspondence relationship.

The vector estimation unit 139 estimates the vector in which the subject moves between the frame images by comparing one or more specific feature points with one or more reference feature points. Specifically, the vector estimation unit 139 moves the subject by comparing some specific feature points and some reference feature points based on the correspondence after the selection unit 138 removes the false detection. Estimate the vector to do. The vector estimation unit 139 captures the coordinates of the pixels included in the subject in the specific frame image in the reference frame image based on, for example, a plurality of adjacent specific feature points and reference feature points in a corresponding relationship. The motion vector of the subject may be estimated by calculating the geometric transformation of the pixels included in the subject into the coordinates. As a method for calculating the motion vector, a known technique can be used.

The vector estimation unit 139 may estimate the vector based on the relationship between the positions of some specific feature points in the specific frame image and the positions of some reference feature points in the reference frame image. For example, suppose that there are a plurality of estimated specific feature points for which motion vectors are estimated, and unestimated specific feature points for which motion vectors are not estimated. Further, it is assumed that each of the reference feature points corresponding to the plurality of estimated specific feature points and the reference feature points corresponding to the unestimated specific feature points are adjacent to each other in the reference frame image. In this case, the vector estimation unit 139 estimates the motion vector of the unestimated specific feature point so as to be adjacent to the plurality of estimated specific special feature points. By doing so, the vector estimation unit 139 can prevent erroneous detection of the correspondence relationship and prevent the prediction error from becoming large. The vector estimation unit 139 inputs the estimated motion vector to the compression unit 140.

The compression unit 140 generates a compressed moving image in which the moving image is compressed based on the result estimated by the vector estimation unit 139. Specifically, the compression unit 140 first obtains a difference image that is a difference between a predicted image obtained by applying a motion vector input from the vector estimation unit 139 to a reference frame image and a specific frame image that is the target of the motion vector. To generate. Then, the compression unit 140 generates a compressed moving image based on the reference frame image, the motion vector, and the difference image. The compression unit 140 stores the generated compressed moving image in the storage unit 12.

[Extraction process]
Subsequently, the extraction process performed by the extraction unit 133 will be described. As described above, the extraction unit 133 has a rear-stage extraction unit 134 and a front-stage extraction unit 135. The post-stage extraction unit 134 has a plurality of post-stage identification outputs and a reference frame output from the post-stage processing layer because the specific frame image propagates in the order of the pre-stage processing layer and the post-stage processing layer which are a part of the plurality of processing layers. From the plurality of post-stage reference outputs output from the post-stage processing layer due to the image propagating in the order of the pre-stage processing layer and the post-stage processing layer, one or more post-stage identification outputs and one or more post-stage reference outputs that are commonly activated are output. Extract.

The pre-stage extraction unit 135 is output from a plurality of pre-stage identification outputs output from the pre-stage processing layer and output from the pre-stage processing layer, which are factors that activate one or more rear-stage identification outputs and one or more post-stage reference outputs. From a plurality of pre-stage reference outputs, one or more pre-stage identification outputs and one or more pre-stage reference outputs that are commonly activated are extracted.

4 to 9 are diagrams for explaining the extraction process performed by the extraction unit 133. In FIGS. 4 to 9, for convenience of explanation, it is assumed that there is one specific frame image. 4 to 9 show a state of propagation from the pre-stage processing layer to the post-stage processing layer. In FIGS. 4 to 9, the connecting line connecting the units shown by the solid line indicates that there was an output from the connecting unit, and the connecting line shown by the broken line indicates that there was no output from the connecting unit. The thickness of the line indicating the connecting line indicates the magnitude of the output from the unit to be connected.

In the case of FIG. 4, the post-stage treatment layer is a treatment layer (for example, an output layer or a fully connected layer) or a treatment layer before the last layer selected by the extraction unit 133 (such as a fully connected layer or a pooling layer). Yes, the pre-stage treatment layer is a treatment layer immediately before the post-stage treatment layer (for example, a fully bonded layer or a pooling layer). In FIG. 4, it is assumed that the post-stage processing layer is the output layer M20 and the pre-stage processing layer is the fully connected layer M19.

FIG. 4A is a state before extraction, and FIG. 4B is a state after extraction. In the specific frame image, units U5 and U8 are activated in the output layer M20, and units U2, U5, U6, U7, and U8 are activated in the fully connected layer M19. In the reference frame image, the output layer M20 has the units U3 and U5 activated, and the fully connected layer M19 has the units U2, U4, U5 and U8 activated.

In this case, the post-stage extraction unit 134 is the units U5 and U8 which are the post-stage identification outputs output from the output layer M20 which is the post-stage processing layer, and the units U3 and U5 which are the post-stage reference outputs output from the output layer M20. To compare. Then, the post-stage extraction unit 134 extracts the unit U5 of the post-stage identification output and the unit U5 of the post-stage reference output that are commonly activated.

Subsequently, the front-stage extraction unit 135 has the units U2, U5, U6, which are the front-stage identification outputs, output from the fully-coupled layer M19, which is the front-stage processing layer, which has become a factor in activating the unit U5 of the rear-stage identification output. And the units U2, U5, and U8 which are the front-stage reference outputs output from the fully connected layer M19 which became a factor for activating the unit U5 of the rear-stage reference output are compared. Then, the pre-stage extraction unit 135 extracts the units U2 and U5 of the pre-stage identification output and the units U2 and U5 of the pre-stage reference output that are commonly activated.

When the extraction unit 133 extracts the output from the output layer M20 to the fully connected layer M19, the extraction unit 133 extracts the output for the next processing layer. Specifically, the extraction unit 133 repeats the process of extracting the specific output and the reference output that are commonly activated for each processing layer in the order opposite to the order propagated by the propagation control unit 132. .. More specifically, the extraction unit 133 selects one of the plurality of processing layers as the post-stage processing layer, extracts one or more specific outputs and one or more reference outputs, and then selects the pre-stage processing layer. The processed layer is selected as the subsequent processing layer, and another one or more specific outputs and one or more reference outputs are extracted. By doing so, the extraction unit 133 can improve the accuracy of comparison with the specific frame image and the reference frame image.

FIG. 5 shows a state in which the image is propagated from the pre-stage processing layer to the post-stage processing layer based on the specific frame image. FIG. 6 shows a state in which the image is propagated from the pre-stage processing layer to the post-stage processing layer based on the reference frame image. In the case of FIGS. 5 and 6, the post-stage treatment layer is a fully-bonded layer M18, and the front-stage treatment layer is a treatment layer other than the fully-bonded layer (for example, a pooling layer or a convolution layer). In FIGS. 5 and 6, the pretreatment layer will be described as the pooling layer M17. Further, in FIGS. 5 and 6, the pretreatment layer has three channels. The first channel in the upper row includes units U11, U12, U13, U14, and U15. The second channel in the middle stage includes units U21, U22, U23, U24, and U25. The lower third channel includes units U31, U32, U33, U34, and U35.

In the specific frame image, in the pooling layer M17, the units U13 included in the first channel and the units U21 and U24 included in the second channel are activated. In the reference frame image, in the fully connected layer M18, the units U22, U24, U25 included in the second channel and the units U32, U33 included in the third channel are activated.

The front-stage extraction unit 135 is a factor that activates the front-stage identification output unit U5 output from the pooling layer M17, which is the front-stage processing layer, and the rear-stage reference output unit U5, which are factors that activate the rear-stage identification output unit U5. The previous stage reference output output from the pooling layer M17 is compared. The pre-stage extraction unit 135 checks for the presence or absence of an activated unit, units U13 included in the first channel of the activated pre-stage identification output, U21 and U24 included in the second channel, and the first stage reference output. Focus on the units U22, U24, U25 included in the second channel and the U32, U33 included in the third channel.

Then, the pre-stage extraction unit 135 is included in the second channel of the pre-stage specific output because the channel in which the unit activated in both the pre-stage specific output and the pre-stage reference output exists is the second channel. Units U21, U24, and units U22, U24, and U25 included in the second channel of the pre-stage reference output are extracted.

In the case of FIG. 7, the post-stage treatment layer is the pooling layer M16, and the front-stage treatment layer is a treatment layer other than the pooling layer (for example, a convolution layer or a normalized layer). In FIG. 7, the pre-stage processing layer will be described as the convolution layer M15. Further, in FIG. 7, the pre-stage processing layer will be described as having one channel. In the specific frame image, the pooling layer M16 has the unit U5 activated, and the convolution layer M15 has the units U3 and U5 activated. In the reference frame image, the pooling layer M16 has the unit U3 activated, and the convolution layer M15 has the units U3 and U4 activated.

Here, in the pooling layer that compresses the image, the extraction unit 133 extracts the output based on the degree of activation for each channel among the plurality of units bonded to the pooling layer from the immediately preceding processing layer. To do. Specifically, the pre-stage extraction unit 135 has one or more pre-stage identification outputs and one or more pre-stage reference outputs based on the activated size of the plurality of pre-stage identification outputs and the plurality of pre-stage reference outputs. Is extracted. The pre-stage extraction unit 135 extracts, for example, one or more pre-stage identification outputs and one or more pre-stage reference outputs that are most activated for each channel among the plurality of pre-stage identification outputs and the plurality of pre-stage reference outputs.

In this case, the post-stage extraction unit 134 selects the pooling layer M16 selected as the pre-stage processing layer in the immediately preceding extraction process, and outputs from the post-stage identification output unit U5 and the pooling layer M16 output from the pooling layer M16. The unit U3 of the latter-stage reference output is extracted. Then, the front-stage extraction unit 135 has the front-stage identification output unit U5 and the front-stage reference output that are most activated for each channel among the rear-stage identification output units U3 and U5 and the rear-stage reference output units U3 and U4. Unit U4 is extracted. By doing so, the pre-stage extraction unit 135 can specify a characteristic region in the image.

In the case of FIG. 8, the post-stage treatment layer is a convolution layer M14, and the front-stage treatment layer is another treatment layer including the convolution layer (for example, a normalized layer or a pooling layer). In FIG. 8, the pre-stage processing layer will be described as the normalization layer M13. Further, in FIG. 8, the pre-stage processing layer will be described as having one channel. In the specific frame image, the convolution layer M14 has the unit U5 activated, and the normalization layer M13 has the units U3, U5, and U6 activated. In the reference frame image, the convolution layer M14 has the unit U3 activated, and the normalized layer M13 has the units U3, U4, and U5 activated.

In this case, the post-stage extraction unit 134 selects the convolution layer M14 selected as the pre-stage processing layer in the immediately preceding extraction process, and outputs from the post-stage identification output unit U5 and the convolution layer M14 output from the convolution layer M14. The unit U3 of the latter-stage reference output is extracted.

Subsequently, the front-stage extraction unit 135 uses the front-stage identification output and the rear-stage reference output unit U4 output from the normalization layer M13, which is the front-stage processing layer, which has become a factor in activating the rear-stage identification output unit U5. The pre-stage reference output output from the normalization layer M13, which is the pre-stage processing layer that became the factor for activation, is compared. Here, when the post-stage processing layer is a convolutional layer, the pre-stage extraction unit 135 has an output for specifying the pre-stage and a pre-stage among a plurality of units of the pre-stage processing layer that are coupled to the units extracted from the post-stage processing layer by the post-stage extraction unit 134. Extract units that are relatively the same position as the reference output and have the same channel. In this case, the pre-stage extraction unit 135 has the units U5, U6, and the pre-stage reference output of the pre-stage identification output as units having relatively the same position in the pre-stage identification output and the pre-stage reference output and having a common channel. Units U3 and U4 are extracted.

In the case of FIG. 9, the post-stage treatment layer is the normalization layer M12, and the front-stage treatment layer is a treatment layer other than the normalization layer (for example, a convolution layer or a pooling layer). In FIG. 9, the pretreatment layer will be described as the pooling layer M11. Further, in FIG. 9, the pre-stage processing layer will be described as having one channel. In the specific frame image, the unit U5 is activated in the normalization layer M12. In the reference frame image, the normalized layer M12 has the unit U3 activated.

Here, in the normalization layer that preprocesses the image, the extraction unit 133 is the center of the plurality of units included in the pretreatment layer that is bound to the unit that is activated in the post-processing layer. Extract the unit. In this case, the post-stage extraction unit 134 extracts the post-stage identification output unit U5 output from the normalization layer M12 selected as the post-stage processing layer and the post-stage reference output unit U3 output from the normalization layer M12. ..

Then, the front-stage extraction unit 135 extracts the central unit U5 from the units of the pooling layer M11 coupled to the unit U5 of the rear-stage identification output output from the normalization layer M5. Similarly, the front-stage extraction unit 135 extracts the central unit U3 from the units of the pooling layer M11 coupled to the unit U3 of the rear-stage reference output output from the normalization layer M12.

It is preferable that the extraction unit 133 repeats the above-mentioned extraction process up to the input layer. However, the extraction unit 133 may end the extraction process at an intermediate process layer (for example, a pooling layer, a normalization layer, etc.) without performing the extraction process up to the first process layer. In this way, the extraction unit 133 can extract an output having a high degree of abstraction by performing the extraction process in the order opposite to the order in which the propagation control unit 132 propagates.

[Processing of video compression device 1]
Subsequently, the flow of processing performed by the moving image compression device 1 will be described. FIG. 10 is a flowchart showing the flow of processing performed by the moving image compression device 1. This flowchart starts when the moving image compression device 1 receives an operation of storing the moving image in the storage unit 12 and executing a process of compressing the moving image by the user.

The moving image acquisition unit 131 acquires the moving image stored in the storage unit 12 (S1). The moving image acquisition unit 131 inputs the acquired moving image to the propagation control unit 132. The propagation control unit 132 has input layers M1 to output layers M10 included in the machine learning model M for each of the specific frame image and the reference frame image, which are frame images included in the moving image input from the moving image acquisition unit 131. The plurality of processing layers up to are propagated in order from the input layer M1 (S2).

The extraction unit 133 performs a process of extracting one or more specific outputs and one or more reference outputs that are commonly activated in both the treatment layers of the post-stage treatment layer and the front-stage treatment layer (S3). FIG. 11 is a flowchart showing the flow of processing performed by the extraction unit 133. The extraction unit 133 determines whether or not the instruction receiving unit 136 has received an instruction to select a processing layer to be used as the subsequent processing layer from the plurality of processing layers via the operation unit 11 (S31).

When the extraction unit 133 determines that the instruction receiving unit 136 has received the instruction, the extraction unit 133 selects the processing layer indicated by the instruction received by the instruction receiving unit 136 as the subsequent processing layer (S32). When, for example, the extraction unit 133 determines that the instruction receiving unit 136 has received the instruction indicating the first fully connected layer M8, the extraction unit 133 processes the first fully connected layer M8 indicated by the instruction received by the instruction receiving unit 136 in a subsequent stage. Used as a layer. On the other hand, when the extraction unit 133 determines that the instruction reception unit 136 has not received the instruction, one or more specific outputs and one or more references that are commonly activated in the rearmost layer (for example, the output layer M10). It is determined whether or not there is an output (S33).

When the extraction unit 133 determines that there is one or more specific outputs and one or more reference outputs that are commonly activated in the output layer M10, the output layer M10, which is the rearmost layer, is used as the post-processing layer. (S34). On the other hand, when the extraction unit 133 determines that there is no one or more specific outputs and one or more reference outputs that are commonly activated in the output layer M10, it is common to each processing layer prior to the output layer M10. Iteratively searches for one or more specific outputs and one or more reference outputs that are activated in. Then, the extraction unit 133 uses a processing layer having one or more specific outputs and one or more reference outputs (for example, the second fully connected layer M9) that are commonly activated as the subsequent processing layer (S35). ). The extraction unit 133 is one or more specifics output from the post-stage processing layer and the pre-stage processing layer based on the specific frame image that is commonly activated in both the selected post-stage processing layer and the pre-stage processing layer. Based on the output and the reference frame image, one or more reference outputs output from the post-stage processing layer and the pre-stage processing layer are extracted.

Specifically, first, the post-stage extraction unit 134 is commonly activated from a plurality of post-stage identification outputs output from the selected post-stage processing layer and a plurality of post-stage reference outputs output from the selected post-stage processing layer. One or more subsequent-stage identification outputs and one or more subsequent-stage reference outputs are extracted (S36). Then, the front-stage extraction unit 135 has a plurality of front-stage identification outputs output from the front-stage processing layer that has been a factor in activating one or more rear-stage identification outputs and one or more rear-stage reference outputs extracted by the rear-stage extraction unit 134. , And one or more pre-stage identification outputs and one or more pre-stage reference outputs that are commonly activated are extracted from the plurality of pre-stage reference outputs output from the pre-stage processing layer (S37).

Subsequently, the extraction unit 133 determines whether or not there is another processing layer before the pre-stage processing layer (S38). When the extraction unit 133 determines that there is another treatment layer (for example, the third convolution layer M6) before the pre-stage treatment layer (for example, the second pooling layer M7), the extraction unit 133 puts the second pooling layer M7 in the rear stage. It is selected as the processing layer (S39), and the processing is returned to S36. On the other hand, when the extraction unit 133 determines that there is no other processing layer before the pre-stage processing layer (for example, the input layer M1), the extraction unit 133 outputs one or more extracted specific outputs and one or more reference outputs as feature point detection units. Input to 137 to end the extraction process.

Returning to FIG. 10, the feature point detection unit 137 searches for a corresponding feature point based on one or more identification outputs and one or more reference outputs, and one or more based on one or more corresponding identification outputs. And one or more reference feature points based on one or more reference outputs are detected (S4). Subsequently, the selection unit 138 determines whether or not there are inappropriate feature points in the specific feature points and the reference feature points detected by the feature point detection unit 137 (S5). The selection unit 138 narrows down based on, for example, the RANSAC method.

When the selection unit 138 determines that the specific feature points and the reference feature points have inappropriate feature points, the selection unit 138 determines the inappropriate feature points, that is, the specific feature points and the reference feature points having a corresponding erroneous detection relationship. It is removed (S6), and some specific feature points and some reference feature points based on the correspondence after removal are selected. The vector estimation unit 139 determines that there are no inappropriate feature points in the specific feature points and the reference feature points, or after removing the erroneously detected correspondence, the vector estimation unit 139 removes one or more specific feature points. By comparing with one or more reference feature points, the vector in which the subject moves between the frame images is estimated (S7).

Then, the compression unit 140 generates a compressed moving image in which the moving image is compressed based on the result estimated by the vector estimation unit 139 (S8). Specifically, the compression unit 140 generates a difference image, and generates a compressed moving image based on the reference frame image, the motion vector, and the generated difference image. The compression unit 140 stores the generated compressed moving image in the storage unit 12.

[Effect in Embodiment]
As described above, the moving image compression device 1 propagates a plurality of processing layers included in the machine learning model M to each of the specific frame image and the reference frame image included in the acquired moving image. The moving image compression device 1 outputs one or more specific outputs and one or more reference outputs that are commonly activated in both the post-stage processing layer and the pre-stage processing layer in the reverse order of propagation. Extraction is performed for each processing layer, and the corresponding specific feature points and reference feature points are detected. Then, the moving image compression device 1 estimates the motion vector of the subject by comparing one or more specific feature points from which the erroneously detected correspondence has been removed with one or more reference feature points, and the estimated motion vector. To generate a compressed motion image obtained by compressing the motion image based on.

By doing so, the moving image compression device 1 uses a machine learning model M including a convolutional neural network, and obtains a feature amount with a high degree of abstraction by deep learning to obtain a motion vector of the subject between frame images. Can be estimated. As a result, the moving image compression device 1 can improve the compression rate of the moving image.

The moving image compression device 1 can perform arbitrary geometric transformation by encoding the displacements of the corresponding feature points detected between the frame images as motion vectors. The moving image compression device 1 can perform motion compensation in a rectangular region in the same manner as MPEG by using, for example, the motion vector of the nearest feature point. Further, the moving image compression device 1 can perform motion compensation for a region having a shape other than a rectangle (for example, a triangle) by using, for example, motion vectors of three or more feature points around it. Further, the moving image compression device 1 can reduce the prediction error in response to the deformation of the subject by performing arbitrary geometric transformation using, for example, the motion vectors of eight or more feature points around it.

In the internationally standardized moving image compression coding method such as MPEG, motion compensation is limited to the rectangular region, so that the rectangular region contains a region with little motion (for example, a background) and a region with a lot of motion. If the area (for example, the foreground) is mixed, the compression rate will decrease. On the other hand, since the moving image compression device 1 can perform motion compensation for a region other than the rectangle, the region with less motion and the region with more motion are separated, and the motion compensation is performed for each region. The compression rate can be improved by performing the above.

Further, the moving image compression device 1 can track a moving object appearing in a moving image by extracting a region of a moving object having a common feature point among a plurality of frame images. Thereby, the moving image compression device 1 can track individual soccer players in, for example, a soccer game.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes can be made within the scope of the gist thereof. is there. For example, the specific embodiment of the distribution / integration of the device is not limited to the above embodiment, and all or a part thereof may be functionally or physically distributed / integrated in any unit. Can be done. Also included in the embodiments of the present invention are new embodiments resulting from any combination of the plurality of embodiments. The effect of the new embodiment produced by the combination has the effect of the original embodiment together.

1 Video compression device 11 Operation unit 12 Storage unit 13 Control unit 131 Motion image acquisition unit 132 Propagation control unit 133 Extraction unit 134 Subsequent extraction unit 135 Pre-stage extraction unit 136 Instruction reception unit 137 Feature point detection unit 138 Selection unit 139 Vector estimation unit 140 Compressor

Claims (14)

A moving image acquisition unit that acquires moving images generated by an imaging device,
Wherein each of the plurality of frames picture images included in the moving picture, a process layer that contains the type of an object included in the moving image based on moving image input to the output enable machine learning model, a plurality of A plurality of processing layers including each unit are propagated, and one reference frame image and one reference frame image before and after the reference frame image are selected from the plurality of frame images based on the type of the subject output from the machine learning model. There, a propagation control section type a subject that is reflected in the reference frame image you select and one or more specific frame image that is reflected by the same object,
On the basis of the specific frame image, the plurality of processing layers subsequent treatment layer selected from, and the subsequent processing layer the units immediately before a treatment layer Te both processing layers odor pre-processing layer are activity of activity of the specific output of the one or more indicating units in common with the units that are activated at the both processing layers based on the reference frame image, the processing layer of the both on the basis of the reference frame image An extraction unit that extracts one or more reference outputs indicating units common to the unit activated in both processing layers based on the specific frame image among the unit.
One or more specific feature points, which are feature points included in the specific frame image, are detected based on the one or more specific outputs , and are included in the reference frame image based on the one or more reference outputs. a feature point detecting unit that detects one or more reference feature points are feature points,
A vector estimation unit that estimates a vector in which the subject moves between the frame images by comparing the one or more specific feature points with the one or more reference feature points.
A compression unit that generates a compressed moving image obtained by compressing the moving image based on the result estimated by the vector estimation unit, and a compression unit.
Video compression device with. The extraction unit
The plurality of post-stage identification outputs and the reference frame image output from the post-stage processing layer are produced by propagating the specific frame image in the order of the pre-stage processing layer and the post-stage processing layer, which are a part of the plurality of processing layers. From the plurality of post-stage reference outputs output from the post-stage treatment layer by propagating in the order of the pre-stage treatment layer and the post-stage treatment layer, one or more post-stage identification outputs indicating the units that are commonly activated and commonly activated. A post-stage extraction unit that extracts one or more post-stage reference outputs indicating the unit
A plurality of pre-stage identification outputs output from the pre-stage processing layer, and a plurality of outputs from the pre-stage processing layer, which are factors that activate the one or more post-stage identification outputs and the one or more post-stage reference outputs. Among the pre-stage reference outputs, a pre-stage extraction unit that extracts one or more pre-stage identification outputs indicating the commonly activated units and one or more pre-stage reference outputs indicating the commonly activated units, and a pre-stage extraction unit.
Have,
The moving image compression device according to claim 1. The pre-stage extraction unit has one or more pre-stage identification outputs and one or more pre-stage identification outputs having the largest numerical value indicating the degree of activation among the plurality of pre-stage identification outputs and the plurality of pre-stage reference outputs. Extract the pre-stage reference output,
The moving image compression device according to claim 2. The machine learning model includes a convolutional neural network.
The post-treatment layer is any one of an output layer, a fully connected layer, a normalized layer, a pooling layer, and a convolution layer.
The moving image compression device according to claim 2 or 3. The pretreatment layer is any one of a fully bonded layer, a normalized layer, a pooling layer, a convolution layer, and an input layer.
The moving image compression device according to claim 4. From the one or more specific feature points and the one or more reference feature points specified by the feature point detection unit, some specific feature points and some reference feature points are selected based on the mutual correspondence. It also has a selection
The vector estimation unit estimates the vector in which the subject moves by comparing some of the specific feature points with the part of the reference feature points.
The moving image compression device according to any one of claims 1 to 5. In the selection unit, when the vector estimation unit estimates a vector on which each of the part of the specific feature points included in the subject moves, the one specific feature point and the one specific feature point When the moving vector with the surrounding specific feature points exceeds a predetermined range, the one specific feature point and the reference feature point corresponding to the one specific feature point are used for some of the specific features. Remove from feature points and some of the reference feature points,
The moving image compression device according to claim 6. The vector estimation unit estimates the vector based on the relationship between the position of the part of the specific feature point in the specific frame image and the position of the part of the reference feature point in the reference frame image. ,
The moving image compression device according to claim 7. Further, it has an instruction receiving unit that receives an instruction to select a processing layer to be used as the subsequent processing layer among the plurality of processing layers.
The extraction unit uses the processing layer indicated by the instruction received by the instruction receiving unit as the subsequent processing layer.
The moving image compression device according to any one of claims 1 to 8. The extraction unit selects one of the plurality of treated layers as the post-processed layer, extracts the one or more specific outputs and the one or more reference outputs, and then selects the pre-processed layer. The processing layer is selected as the post-processing layer to extract another one or more specific outputs and one or more reference outputs.
The moving image compression device according to any one of claims 1 to 9. The steps to acquire the moving image generated by the image pickup device and
Wherein each of the plurality of frames picture images included in the moving picture, a process layer that contains the type of an object included in the moving image based on moving image input to the output enable machine learning model, a plurality of Steps to propagate multiple processing layers , each containing a unit,
Based on the type of subject output from the machine learning model, one reference frame image and the subjects before and after the reference frame image appearing in the reference frame image from the plurality of frame images. Steps to select one or more specific frame images that show subjects of the same type,
On the basis of the specific frame image, the plurality of processing layers subsequent treatment layer selected from, and the subsequent processing layer the units immediately before a treatment layer Te both processing layers odor pre-processing layer are activity of activity of the specific output of the one or more indicating units in common with the units that are activated at the both processing layers based on the reference frame image, the processing layer of the both on the basis of the reference frame image A step of extracting one or more reference outputs indicating units common to the unit activated in both processing layers based on the specific frame image among the unit.
One or more specific feature points, which are feature points included in the specific frame image, are detected based on the one or more specific outputs , and are included in the reference frame image based on the one or more reference outputs. detecting one or more reference feature points are feature points,
A step of estimating a vector in which the subject moves between the frame images by comparing the one or more specific feature points with the one or more reference feature points.
A step of generating a compressed moving image obtained by compressing the moving image based on the estimation result in the step of estimating the vector in which the subject moves, and a step of generating the compressed moving image.
A moving image compression method having. The extraction step
The plurality of post-stage identification outputs and the reference frame image output from the post-stage processing layer are produced by propagating the specific frame image in the order of the pre-stage processing layer and the post-stage processing layer, which are a part of the plurality of processing layers. From the plurality of post-stage reference outputs output from the post-stage treatment layer by propagating in the order of the pre-stage treatment layer and the post-stage treatment layer, one or more post-stage identification outputs indicating the units that are commonly activated and commonly activated. A post-stage extraction step that extracts one or more post-stage reference outputs indicating the units that are
A plurality of pre-stage identification outputs output from the pre-stage processing layer, and a plurality of outputs from the pre-stage processing layer, which are factors that activate the one or more post-stage identification outputs and the one or more post-stage reference outputs. Among the pre-stage reference outputs, a pre-stage extraction step for extracting one or more pre-stage identification outputs indicating the commonly activated units and one or more pre-stage reference outputs indicating the commonly activated units, and a pre-stage extraction step.
The moving image compression method according to claim 11. After executing the pre-stage extraction step, the post-stage extraction step is executed with the one or more pre-stage identification outputs and the one or more pre-stage reference outputs as the plurality of post-stage identification outputs and the plurality of post-stage reference outputs. ,
The moving image compression method according to claim 12. The latter-stage extraction step and the first-stage extraction step are executed for each of the plurality of processing layers.
The moving image compression method according to claim 12 or 13.

Images