JP7310148B2 - Media encoding method and apparatus - Google Patents

Description

The present invention relates to a communication system having a media data analysis function, and more particularly to a media encoding method and apparatus intended to improve analysis accuracy on a receiving side that receives encoded media data.

In recent years, there have been remarkable advances in techniques for analyzing media data such as images, videos, and sounds. Media analysis technology is being applied not only to monitoring systems for crime prevention and disaster prevention, but also to various fields such as vehicle driving support and autonomous driving. Especially when media data is transmitted over a network, an encoding technique for compressing the amount of data is indispensable. At that time, in order to obtain sufficiently high media analysis accuracy on the receiving side, it is necessary to increase the media quality of the decoded media data.

As a technique for ensuring media analysis accuracy on the receiving side, a method has been proposed in which a transmitter adaptively changes encoding parameters in accordance with a notification from a receiver. For example, according to the method disclosed in Patent Document 1, the receiving side switches image recognition processing according to the running state of the vehicle, and notifies the transmitting side of encoding control information corresponding to the image recognition processing after switching. . The transmitting side changes the encoding parameters of the encoder according to the notified encoding control information. As a result, the receiving side can receive the encoded data that enables image recognition suitable for the running state, and the image recognition rate can be improved.

Also, scalable coding (hierarchical coding) is known as a recent representative example of coding technology. In today's network environment, high-speed communication paths such as optical fibers and relatively low-speed wireless communication paths such as mobile terminals are mixed. Scalable coding enables efficient coding according to the performance of the receiving side in such a network environment. An example of a video distribution system using such scalable coding is described in Patent Document 2. Patent Literature 2 discloses a method of compressing a region desired by a user with high image quality while maintaining a constant minimum image quality of the entire screen by controlling scalable coding.

JP 2013-239995 A International Publication No. WO2015/190078

The method described in Patent Document 1 mentioned above adaptively changes the coding parameters on the transmitting side according to the notification from the receiving side. In this case, deterioration of the network environment may cause a problem that the accuracy of image recognition on the receiving side is greatly reduced. For example, there is a possibility that the bit rate of the coded video stream exceeds the network bandwidth when it is coded with predetermined coding parameters according to the notification from the receiving side. In this case, a partial loss of video data occurs, and as a result, disturbance such as block noise occurs in the video for which image recognition is performed on the receiving side, resulting in a large decrease in recognition accuracy.

In addition, in the encoding communication system described in Patent Document 2, an efficient hierarchical encoding method is proposed, but measures against degradation of analysis accuracy on the receiving side due to network environment degradation have not been considered.

SUMMARY OF THE INVENTION An object of the present invention is to provide a media encoding method and apparatus for encoding media data so as to obtain the highest possible media analysis accuracy in consideration of the network environment.

According to one aspect of the present invention, a media encoding device in a communication system that transmits encoded media data over a network includes an encoder that encodes media data according to a set of encoding parameters and outputs the encoded media data. an input unit for inputting an upper limit value of the encoding bit rate of the media data and information on an analysis method of the media data; and a plurality of encoding parameters under the conditions of the analysis method and the upper limit value of the encoding bit rate. generating set candidates, calculating analysis accuracy for each of the plurality of encoding parameter set candidates, and providing the encoding parameter set candidate with the higher calculated analysis accuracy as the encoding parameter set to the encoder; and a coding parameter set determination unit for determining.
According to another aspect of the present invention, there is provided a media encoding method in a communication system for transmitting encoded media data over a network, wherein an encoder encodes media data according to a set of encoding parameters and outputs the encoded media data. Then, the encoding parameter determination unit inputs the upper limit value of the encoding bit rate of the media data and information about the analysis method of the media data, and under the conditions of the analysis method and the upper limit value of the encoding bit rate generating a plurality of coding parameter set candidates, calculating analysis accuracy for each of the plurality of coding parameter set candidates, and providing a coding parameter set candidate with a higher calculated analysis accuracy to the encoder; Decide as a coding parameter set.
According to still another aspect of the present invention, a program causing a computer to function as a media encoding device in a communication system for transmitting encoded media data over a network encodes media data according to an encoding parameter set, A function to output data, an upper limit value of the encoding bit rate of the media data, and information on an analysis method of the media data are input, and a plurality of a function of generating a coding parameter set candidate; a function of calculating analysis accuracy for each of the plurality of coding parameter set candidates; and a determining function are realized in the computer.

According to the present invention, it is possible to select an encoding parameter set with higher analysis accuracy within the analysis method to be used and the given upper limit of bit rate. Accuracy can be obtained.

FIG. 1 is a block diagram showing a schematic configuration of a media encoding device according to the first embodiment of the present invention. FIG. 2 is a flow chart showing the overall operation of the media encoding device shown in FIG. FIG. 3 is a flow chart showing the operation of the encoding parameter calculator in the media encoding device shown in FIG. FIG. 4 is a table showing an example of analysis accuracy for encoding parameter candidates generated by the encoding parameter calculation operation shown in FIG. FIG. 5 is a block diagram showing a schematic configuration of a media encoding device according to the second embodiment of the present invention. 6 is a flow chart showing the operation of the encoding parameter calculator in the media encoding device shown in FIG. FIG. 7 is a table showing an example of analysis accuracy for encoding parameter candidates generated by the encoding parameter calculation operation in FIG. FIG. 8 is a block diagram showing a schematic configuration of a media encoding device according to the third embodiment of the present invention. FIG. 9 is a schematic diagram for explaining hierarchical coding in the third embodiment. FIG. 10A is a graph showing the amount of information per frame when using the encoding parameter set selected in the second embodiment. FIG. 10B is a graph showing the amount of information per frame when using a coding parameter set for hierarchical coding. FIG. 11 is a flow chart showing the operation of the encoding parameter calculator in the media encoding device shown in FIG. FIG. 12 is a block diagram showing a schematic configuration of a media encoding device according to the fourth embodiment of the invention. 13 is a flow chart showing the overall operation of the media encoding apparatus shown in FIG. 12. FIG. FIG. 14 is a block diagram showing a schematic configuration of a media encoding device according to the fifth embodiment of the present invention. FIG. 15 is a schematic configuration diagram of a media analysis system in which the media encoding device and the media analysis device according to each of the above embodiments are network-connected.

<Overview of Embodiment>
According to an embodiment of the present invention, the coding parameter set that gives the highest analysis accuracy is selected for the analysis method used and given bit rate upper bound. As a result, even if the bit rate upper limit is low, highly accurate media analysis is possible within the limited bit rate range. BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

1. First Embodiment 1.1) Configuration As illustrated in FIG. 1, a media encoding device 10 according to the first embodiment of the present invention includes a media data input unit 101, an encoding unit 102, an output unit 103, It is composed of an encoding parameter calculation unit 104 , an analysis accuracy calculation unit 105 , an analysis method input unit 106 and a bit rate input unit 107 . In the media encoding device 10, the encoding parameter calculation section 104 and the analysis accuracy calculation section 105 constitute an encoding parameter set determination section in this embodiment, as will be described later.

A media data input unit 101 is connected to an input device such as a camera or a microphone, a storage device, or the like, and inputs media data such as video and audio.

The encoding unit 102 encodes the media data input by the media data input unit 101 using the parameters calculated by the encoding parameter calculation unit 104 . For example, when the media data is video, H.264 is used. 264, H. H.265, motion JPEG (Joint Photographic Experts Group), or other video encoding method. If the media data is audio, G. 711, AMR (Adaptive Multi-Rate), and AAC (Advanced Audio Coding).

The output unit 103 is an interface for outputting encoded media data. For example, the output unit 103 may be a communication interface of a cellular network such as a wired LAN (Local Area Network), a wireless LAN, or LTE (Long Term Evolution). Alternatively, the output unit 103 may be a slot of the media encoding device 10 or an interface such as a USB (Universal Serial Bus). That is, a storage device may be attached to the slot, or an external storage device may be detachably connected to an interface such as USB.

Coding parameter calculation section 104 generates coding parameter set candidates based on the conditions input from analysis method input section 106 and bit rate input section 107 and outputs them to analysis accuracy calculation section 105 . When the analysis accuracy for each candidate is input from analysis accuracy calculation section 105 , encoding parameter calculation section 104 determines the encoding parameter set with the highest analysis accuracy among them and instructs encoding section 102 . The encoding parameter set determination function of the encoding parameter calculation unit 104 is realized by executing a program stored in a memory (not shown) on a processor (CPU (Central Processing Unit), etc.) of the media encoding device 10. is also possible.

Analysis accuracy calculation section 105 receives encoding parameter set candidates from encoding parameter calculation section 104 , calculates analysis accuracy (or its predicted value) for each set candidate, and returns it to encoding parameter calculation section 104 . The analysis accuracy calculation unit 105 may be prepared in advance with an analysis accuracy calculation formula corresponding to the analysis method, or a predicted value of the analysis accuracy for the coding parameter set candidate may be generated by learning and held. good too. The function of the analysis accuracy calculation unit 105 can also be realized by executing a program stored in a memory (not shown) on the processor of the media encoding device 10 .

The analysis method input unit 106 inputs information for identifying the media analysis method adopted by the receiving side or the analysis engine actually used. As an input method, there is a method using an input/output device (not shown) such as a touch panel of the media encoding device 10 or an information processing device equipped with it, or if the output unit 103 is a communication interface, a message can be sent from the receiving side. can be considered. When the media data is video, the analysis method indicates, for example, face recognition, object recognition, object movement recognition, and the like. If the media data is voice, the analysis method indicates voice recognition, speaker recognition, or the like. In the case of these analysis methods, the analysis accuracy calculated by the analysis accuracy calculation unit 105 is, for example, the rate of correct recognition (recognition rate) or its predicted value.

The bit rate input unit 107 receives input of the upper limit value of the encoding bit rate of media data. The input method is the same as that of the analysis method input unit 106 . The upper limit of the encoding bit rate is calculated from the physical band of the network to which the output unit 103 is connected, the communication speed measured in the past with the receiving device (not shown), or the future value based on statistical data. may be calculated as a predicted value of the communication speed of Also, in consideration of the amount used by other applications, these values may be multiplied by a specified value of 0 or more and 1 or less, or may be a value obtained by subtracting the specified value. The bit rate upper limit value may be changed at intervals of several hundred milliseconds to several tens of seconds, or at the timing when a change in network state is detected. When outputting to an external storage device connected to the media encoding device 10, it may be calculated based on the capacity or remaining capacity of the external storage device and the data length (time) to be stored. For example, if the capacity of the external storage device is 1GB (Gigabyte) and the data length to be saved is 1 hour, the upper limit of the bit rate is 1 [GB] × 1,000 [MB (Megabyte)/GB] × 8 [bit/ byte] / 3,600 [second] = 2.222 [Mbps (Migabit per second)].

The operation of the media encoding device 10 according to the present embodiment will be described below with reference to FIGS. 2 to 4. FIG.

1.2) Operation <Media encoding>
As illustrated in FIG. 2, when media data is input from the media data input unit 101 (operation S201), the encoding parameter calculation unit 104 receives the analysis method and bit rate from the analysis method input unit 106 and the bit rate input unit 107. Each rate is read (operations S202 and S203). Subsequently, the encoding parameter calculation unit 104 determines an encoding parameter set according to a procedure described later, and instructs the encoding unit 102 (operation S204).

The encoding unit 102 encodes the media data received from the media data input unit 101 using the encoding parameter set indicated by the encoding parameter calculation unit 104 (operation S205), and passes the media data to the output unit 103. The output unit 103 outputs the encoded media data received from the encoding unit 102 (operation S206). The operations S201 to S206 are repeated (NO in operation S207) until the operation of the media encoding device 10 is stopped (YES in operation S207).

Note that if the analysis method and bit rate are not changed, the previous encoding parameter set can be used as is. Therefore, it is also possible to control the operations S202 to S204 to be executed only when the input value of the analysis method or bit rate is changed.

<Encoding parameter set determination>
Next, a more detailed operation of the encoding parameter set determination operation S204 described above will be described with reference to FIGS. 3 and 4. FIG.

As exemplified in FIG. 3, when the analysis method and bit rate are input (operations S202 and S203 in FIG. 2), the encoding parameter calculation unit 104 calculates, as will be described later, based on the input analysis method and bit rate. A candidate encoding parameter set is generated (operation S301). Subsequently, by outputting each of the generated coding parameter set candidates to analysis accuracy calculation section 105, an estimated value of analysis accuracy for each candidate is obtained (operations S302 and S303). After acquiring the analysis accuracy for all the encoding parameter sets, the encoding parameter calculation unit 104 determines the encoding parameter set with the highest analysis accuracy and outputs it to the encoding unit 102 (operation S304).

A function f for calculating the analysis accuracy may be defined in advance in the analysis accuracy calculation unit 105 . Alternatively, deep learning or the like may be used to previously learn the correspondence between coding parameter set candidates and analysis accuracy a, and output a predicted value of analysis accuracy a for an input value.

A coding parameter set determination method will be described below using a case where media data is video as an example. In this example, the upper limit of the bit rate input from the bit rate input unit 107 is 1.5 Mbps, and the changeable encoding parameters are the video bit rate and frame rate. In general, the higher the video bit rate, the higher the image quality and the analysis accuracy. Therefore, the bit rate is fixed to the input upper limit value of 1.5 Mbps. Also, the frame rate is a value between 1 and 30. All integers between 1 and 30 may be used as candidates, but especially in areas with high frame rates, even a slight difference has little effect on image quality. values can be sampled. As a result, it is possible to reduce the amount of calculation, and it becomes possible to use even a terminal with low processing capability. In the example shown in FIG. 4, it is assumed that coding parameter set candidates consist of 11 types of bit rate/frame rate pairs.

As illustrated in FIG. 4, the analysis accuracy calculation unit 105 generates (predicted values of) analysis accuracy for each of 11 types of encoding parameter set candidates. For example, when the analysis accuracy can be calculated by a calculation formula, a function f for calculating the analysis accuracy can be defined in the analysis accuracy calculation unit 105 . When the analysis method (type of analysis engine) is k, the bit rate is rb, the frame rate is rf, and the analysis accuracy is a,
a = f(k, rb, rf)
defined by Therefore, when analysis method k and coding parameter set candidates (rb, rf) are given, analysis accuracy calculation section 105 can calculate (predicted value of) analysis accuracy a using the above equation. The correspondence between the encoding parameter set candidates (rb, rf) and the analysis accuracy a may be learned in advance using deep learning or the like instead of the calculation formula, and stored as an analysis accuracy table. By searching the analysis accuracy table according to the given coding parameter set candidate (rb, rf), the corresponding analysis accuracy predicted value a can be output.

By comparing the analysis accuracy for each candidate thus obtained, the code parameter with the highest accuracy can be determined. In the example shown in FIG. 4, the analysis accuracy a=90% is the highest, so the seventh coding parameter set (1.5 Mbps/10 fps (frame per second)) is determined.

It should be noted that H. H.264 and motion JPEG can be used, the encoding method e is added to the parameter of the function f of the analysis accuracy calculation unit 105, and the function f is
a = f(k, rb, rf, e)
may be defined as In this case, the case where e is changed is also added to the coding parameter set candidates. Alternatively, as described above, the analysis accuracy a may be calculated by storing in advance the correspondence between the encoding parameter set candidate for which the encoding method e is changed and the analysis accuracy a.

1.3) Effect As described above, according to the first embodiment of the present invention, the analysis engine used generates coding parameter set candidates for each changeable coding parameter at a given bit rate upper limit. do. Then, the analysis accuracy is calculated for each encoding parameter set candidate, and the encoding parameter set exhibiting the highest analysis accuracy is determined as the parameter set to be used for encoding. Therefore, even if the bit rate upper limit is low, media analysis with the highest accuracy is possible within the limited bit rate range.

2. Second Embodiment A second embodiment of the present invention differs from the first embodiment in that a plurality of types of analysis engines are used simultaneously. Therefore, the media encoding device 10A according to this embodiment has the same basic configuration as the media encoding device 10 according to the first embodiment, but differs in the function of the encoding parameter set determination unit. In the following, blocks having the same functions are denoted by the same reference numerals to simplify the description, and mainly the configurations and functions different from those of the first embodiment will be described.

2.1) Configuration As illustrated in FIG. 5, a media encoding device 10A according to the present embodiment includes a media data input unit 101, an encoding unit 102, an output unit 103, an encoding parameter calculation unit 104A, It is composed of an analysis accuracy calculation unit 105A, an analysis method input unit 106, and a bit rate input unit 107. FIG. In the media encoding device 10A, the encoding parameter calculation section 104A and the analysis accuracy calculation section 105A constitute an encoding parameter set determination section in this embodiment, as will be described later.

In this embodiment, a plurality (n) of analysis engine types k(1) to k(n) and weights w(1) to w(n) of the respective analysis engines are input from the analysis method input unit 106. (n is an integer of 2 or more). It is assumed that the analysis accuracy calculation unit 105A is prepared in advance with a function fi for calculating the analysis accuracy corresponding to each analysis engine type k(i).

2.2) Operation The overall operation of the media encoding device 10A is as shown in FIG. The encoding parameter set determination operation (operation corresponding to S204 in FIG. 2) according to the present embodiment will be described below with reference to FIGS. 6 and 7. FIG.

As illustrated in FIG. 6, the coding parameter calculator 104A generates coding parameter set candidates as described above based on the input analysis method k(i) and bit rate rb (operation S301). Note that i is an arbitrary integer from 1 to n. Next, encoding parameter calculation section 104A outputs analysis method k(i) and each of the generated encoding parameter set candidates to analysis accuracy calculation section 105A, and estimates analysis accuracy value a(i) for each candidate. is obtained (operations S302A, S303A).

After acquiring the analysis accuracy a(i) for each encoding parameter set in each analysis method k(i), the encoding parameter calculation unit 104A adds the weight w(i) of the analysis engine to each analysis accuracy a(i). A weighted sum of the values multiplied by is calculated, and the weighted sum is defined as the analysis accuracy a. The encoding parameter calculation unit 104A determines the encoding parameter set that maximizes the analysis accuracy a, and outputs it to the encoding unit 102 (operation S304A). By relatively changing the weight w(i) corresponding to each analysis engine k(i), it is possible to set the analysis accuracy according to the application of the media data to be encoded.

The encoding parameter set determination method according to the present embodiment will be described below, taking the case where the media data is video as an example. In this example, as in the first embodiment described above, the upper limit of the bit rate input from the bit rate input unit 107 is 1.5 Mbps, and the changeable encoding parameters are the video bit rate and frame rate. shall be Also, in the example shown in FIG. 7, it is assumed that coding parameter set candidates consist of 11 types of bit rate/frame rate pairs.

To avoid complicating the explanation, as an example, two types of analysis engines, k(1) and k(2) (n=2), are weighted w(1)=0.6 and w(2). = 0.4. It is also assumed that functions f1 and f2 for calculating analysis accuracies a1 and a2 corresponding to the respective analysis engines are defined as follows.
a1 = f1(k1, rb, rf)
a2 = f2(k2, rb, rf)

When the analysis accuracy calculation unit 105A receives the value k(i) indicating the type of analysis engine, the bit rate rb, and the frame rate rf from the encoding parameter calculation unit 104A, the analysis accuracy calculation unit 105A uses the above functions f1 and f2 to perform respective analysis Calculate the accuracies a1 and a2. These analytic accuracies a1 and a2 are multiplied by the corresponding weights w(1)=0.6 and w(2)=0.4, respectively, and the results are added to obtain the analytic accuracy a:
a = 0.6a1 + 0.4a2.

When the analysis accuracy a is obtained for each of the encoding parameter sets in this way, the parameter set showing the highest analysis accuracy among these analysis accuracy a is determined as the encoding parameter set to be given to the encoder 102 . Instead of defining the above functions f1 and f2, the analysis accuracy table may be prepared and the analysis accuracy a1 and a2 may be calculated as described in the first embodiment.

In the example shown in FIG. 7, the analysis engine 1 has the characteristic that the analysis accuracy improves as the frame rate increases, but if the frame rate is increased too much, the amount of information per frame decreases and the analysis accuracy decreases. Such an analysis engine 1 is, for example, an object motion recognition engine. The analysis engine 2 has a characteristic that the analysis accuracy increases as the frame rate is low and the amount of information per frame is large. For example, the analysis engine 2 is a face recognition engine.

By comparing the analysis accuracy a for each candidate thus obtained, a code parameter set with the highest accuracy can be determined. In the example shown in FIG. 7, the analysis accuracy a=76% is the highest, so the fifth coding parameter set (1.5 Mbps/5 fps) is determined.

In this embodiment, there are two types of analysis engines (n=2), but even when three or more types of analysis engines are used, the optimum parameter set can be selected in a similar manner.

2.3) Effect As described above, according to the second embodiment of the present invention, a parameter set is selected such that the weighted sum of the analysis accuracies estimated for each of the plurality of analysis engines is maximized. As a result, it is possible to improve analysis accuracy when applying a plurality of analysis engines to single media data. In particular, by changing the weight w(i) of each analysis engine k(i), there is an advantage that a parameter set can be selected according to the application.

3. Third Embodiment A media encoding apparatus according to a third embodiment of the present invention is similar in basic configuration and operation to the above-described second embodiment, but encodes media data using hierarchical encoding technology. It differs in that it converts and determines the parameter set for hierarchical coding. Hereinafter, blocks having the same functions are denoted by the same reference numerals to simplify the description, and mainly the configurations and functions different from those of the second embodiment will be described.

3.1) Configuration As illustrated in FIG. 8, a media encoding device 10B according to the present embodiment includes a media data input unit 101, a hierarchical encoding unit 102B, an output unit 103, and an encoding parameter calculation unit 104B. , an analysis accuracy calculation unit 105 B, an analysis method input unit 106 and a bit rate input unit 107 . As described below, the layer coding unit 102B encodes the media data into a base layer and L layers (L is an integer of 1 or more and n or less) of extension layers.

As schematically shown in FIG. 9, in hierarchical encoding, input media data is encoded into basic layer 0 and extended layers 1 to L. FIG. The extended layer is difference data for improving the quality of the basic layer. As the number of extended layers used increases, the amount of data (bit rate) increases, but the media quality improves. In the case of video layered coding, H.264. H.264 extension standard. 264/SVC, H. SHVC (Scalable HEVC), which is an extended standard of H.265 (HEVC: High Efficiency Video Coding), is defined as a hierarchical video coding standard. In the case of voice, AAC-SSR (Scalable Sampling Rate), which is an extended standard of AAC, is defined as a hierarchical voice coding standard.

The encoding parameter calculation unit 104B and the analysis accuracy calculation unit 105B constitute an encoding parameter set determination unit in this embodiment. It is assumed that the analysis accuracy calculation unit 105B is prepared in advance with functions f1 to fn for analysis accuracy calculation corresponding to a plurality (n) of analysis engines k(1) to k(n), respectively. The coding parameter calculation unit 104B also receives the types of a plurality of analysis engines from the analysis method input unit 106 and the upper limit value of the bit rate from the bit rate input unit 17 .

<Comparison of encoding parameter sets>
FIG. 10A illustrates the amount of information in the encoding parameter set selected in the second embodiment described above, and FIG. This is an example of the amount of information. Both bit rates are assumed to be 1.5 Mbps.

As shown in FIG. 10A, the encoding parameter set (1.5 Mbps/5 fps; see FIG. 7) determined in the second embodiment of the present invention has an average information amount of 0.3 Mbit per frame.

On the other hand, according to this embodiment using hierarchical coding with the number of layers 2, as illustrated in FIG. Then, the amount of information in the first frame is 0.5 Mbit in total for the basic layer and the extended layer 1 . Therefore, the lower the frame rate (that is, the greater the amount of information per frame), the higher the analysis accuracy. By analyzing the leading frame with the analysis engine, it is expected that the analysis accuracy will be greatly improved. Although the amount of information other than the first frame is slightly reduced, it is considered that the analysis accuracy will not be significantly reduced with this degree of decrease, and as a result, an improvement in the weighted sum of the analysis accuracy can be expected.

3.2) Operation The overall operation of the media encoding device 10B is as shown in FIG. 2, and the encoding parameter set determination operation according to this embodiment corresponds to operation S204 in FIG. According to this embodiment, encoding parameter set candidates are generated based on predetermined conditions, a weighted sum of analysis accuracies is calculated for each candidate, and the highest parameter set is output. The encoding parameter set determination operation according to this embodiment will be described below with reference to FIG.

In FIG. 11, the hierarchy number j is sequentially set between the minimum value 1 and the maximum value L+1 (operation S401). Here, L+1 is an integer from 1 to n, where n is the number of analysis engines. Assuming that the maximum value is the number of analysis engines n, the following operations S402 to S404 are repeated while increasing the number of layers j by one from the minimum value 1 to the maximum value n.

First, bit rate combination candidates (hereinafter referred to as BR combination candidates p) in the hierarchy number j are sequentially selected (operation S402). For example, when the number of hierarchies j=2 and the upper limit of the bit rate (BR) is 1.5 Mbps, a plurality of BR combination candidates p can be generated for each predetermined BR. For example, in units of 0.25 Mbps, candidate p = (BR of basic layer 0, BR of extended layer 1)
Candidate p1 = (0.75 Mbps, 0.75 Mbps),
Candidate p2 = (1.0 Mbps, 0.5 Mbps),
Candidate p3 = (1.25 Mbps, 0.25 Mbps),
can generate Alternatively, a plurality of BR combination candidates may be generated while increasing the bit rate of basic layer 0 from 0.1 Mbps by 0.1 Mbps. One candidate is sequentially selected from the BR combination candidates thus generated.

Subsequently, under the conditions of the set number of layers j and the selected BR combination candidate p, candidates for allocation of the analysis engine to each layer (hereinafter referred to as E allocation candidates q) are sequentially selected (operation S403). For example, when the number of hierarchies is j=2 and the number of analysis engines is n=3, a plurality of E assignment candidates q can be generated depending on which analysis engine is assigned to which hierarchy. Multiple analysis engines can be assigned to one hierarchy. For example, base tier 0 may be assigned analysis engines E1 and E2, and enhanced tier 1 may be assigned analysis engine E3. All assignment candidates may be generated by assigning at least one analysis engine to each layer.

As described above, the frame rate FR (j, p, q) is determined so as to maximize the analysis accuracy under the conditions of the set number of layers j, the selected BR combination candidate p, and the E allocation candidate q. (Operation S404). For the frame rate, as described in the first embodiment, all integers within a changeable range may be used as candidates, but representative values may be sampled as candidates. When the number of layers j is 2 or more, the analysis accuracy is calculated assuming that all layers below the extended layer are used (see FIG. 9). Also, for a layer to which two or more analysis engines are assigned, the frame rate that maximizes the weighted sum is obtained as in the second embodiment.

As described above, when the frame rate with the highest analysis accuracy is determined for all E allocation candidates q (S403, S404), the next BR combination candidate p is selected (operation S402), and operations S403 and S404 are performed. Repeatedly, when all BR combination candidates p are completed, the hierarchy number j is incremented (operation S401), and the above operations S402 to S404 are repeated.

Thus, when the frame rate FR for each candidate is calculated, the analysis accuracy is calculated as described in the second embodiment, and the analysis accuracy obtained in each layer or when using a plurality of analysis engines, A parameter set (layer number j, bit rate BR and frame rate FR of each layer, analysis engine E assigned to each layer) that gives the highest weighted sum of analysis accuracy is determined (operation S405).

3.3) Effect As described above, the media encoding device 10B according to the third embodiment of the present invention performs encoding using the layered encoding technique in the same manner as in the second embodiment. A hierarchical encoding parameter set is selected such that the analysis accuracy of the analysis engine obtained in the hierarchy or the weighted sum of the analysis accuracy estimated for each of the plurality of analysis engines is maximized. As a result, it is possible to improve analysis accuracy when applying a plurality of analysis engines to hierarchically encoded media.

4. Fourth Embodiment A media encoding device according to a fourth embodiment of the present invention has the same basic configuration as the media encoding devices according to the first to third embodiments, but differs in that it has a media analysis section. The media encoding device according to the present embodiment performs media analysis and uses the result to determine the encoding parameter set. Therefore, this embodiment can be applied when the processing capability of the media encoding device is relatively high. Hereinafter, blocks having the same functions are denoted by the same reference numerals to simplify the description, and mainly the configurations and functions different from those of the first to third embodiments will be described.

4.1) Configuration As illustrated in FIG. 12, a media encoding device 10C according to the present embodiment includes a media data input unit 101, an encoding unit 102, an output unit 103, an encoding parameter calculation unit 104C, It is composed of an analysis accuracy calculation unit 105C, an analysis method input unit 106, a bit rate input unit 107, and a media analysis unit . Media analysis section 108 acquires the media data and encoded media data respectively input from media data input section 101 and encoding section 102, and outputs the feature amount of the media data to encoding parameter calculation section 104C. In the media encoding device 10C, the encoding parameter calculation unit 104C, the analysis accuracy calculation unit 105C, and the media analysis unit 108 constitute an encoding parameter set determination unit in this embodiment, as will be described later.

4.2) Operation As illustrated in FIG. 13, the media encoding device 10C has almost the same overall operation as that shown in FIG. The difference is that the coding parameter set determination operation (operation S502) is executed later.

The media analysis unit 108 receives the media data from the media data input unit 101 and the encoded media data from the encoding unit 102, and calculates the feature amount c of the media data. The feature amount c is the brightness of the screen, the magnitude of movement, and the like. The brightness of the screen can be calculated using, for example, the average brightness of each pixel of the input media data. Also, the magnitude of motion can be calculated by extracting motion vector information from encoded data based on the temporal difference of media data or information on the difference between frames. The calculated feature amount c is notified to the encoding parameter calculation unit 104C.

The encoding parameter calculation unit 104C uses the feature quantity received from the media analysis unit 108 to determine the encoding parameter set. The encoding parameter set determination operation will be described below using a case where the media data is video as an example.

As already described, the analysis accuracy calculation unit 105C defines a function f for calculating the analysis accuracy when the analysis accuracy can be calculated using a calculation formula. When the analysis method (type of analysis engine) is k, the bit rate is rb, the frame rate is rf, the video feature amount is c, and the analysis accuracy is a,
a = f(k, rb, rf, c)
defined by Therefore, when the analysis method k, the coding parameter set candidate (rb, rf), and the feature amount c are given, the analysis accuracy calculation unit 105 can calculate the analysis accuracy (predicted value) a using the above equation. can.
Using deep learning, etc., instead of formulas, learn in advance the correspondence between coding parameter set candidates (rb, rf), feature amount c, and analysis accuracy a, store it in a table, and correspond to the input value A predicted value of the analysis accuracy a may be output.

Also, H.I. H.264 and motion JPEG can be used, the encoding method e is added to the parameter of the function f of the analysis accuracy calculation unit 105, and the function f is
a = f(k, rb, rf, e, c)
may be defined as In this case, the case where e is changed is also added to the coding parameter set candidates.

<Operation example>
As an example of the operation of the media encoding device 10C according to the present embodiment, a case where the analysis engine k1 performs face recognition and the analysis engine k2 performs motion tracking of an object will be described. In this case, the analysis accuracies a1 and a2 respectively corresponding to the analysis engines k1 and k2 are calculated using the following analysis accuracy functions f1 and f2.
a1 = f1(k1, rb, rf, c)
a2 = f2(k2, rb, rf, c)

The function f1 corresponding to the analysis engine k1 (face recognition) is effective for face recognition when the feature value c indicates an image that is too bright (overexposed) or too dark (overexposed black) to recognize the face (for example, when backlighting). cannot be performed, the value of the analysis accuracy a1 is a fixed value, for example zero, even if the values of rb and rf change. On the other hand, the function f2 corresponding to the analysis engine k2 (object motion tracking) does not affect the value of the analysis accuracy a2 for the similar feature amount c. When such analysis engines k1 and k2 are used on the receiving side, the parameter set with the highest weighted sum of the analysis accuracies a1 and a2 is determined as the encoding parameter set. This makes it possible to set a coding parameter set suitable for object motion tracking by the analysis engine k2.

Conversely, object motion tracking by analysis engine k2 becomes less important for videos with small movements, and analysis engine k1 (face recognition) functions more effectively even at low frame rates. Therefore, when the feature value c indicates an image with small motion, the function f2 corresponding to the analysis engine k2 (movement tracking of the object) is a fixed value of the analysis accuracy a2 even if the values of rb and rf change. For example zero. In the case of such a video with small movements, an encoding parameter set suitable for face recognition by the analysis engine k1 is set by determining the encoding parameter set that maximizes the weighted sum of the analysis accuracies a1 and a2. can do.

Note that the function of the media analysis unit 108 can also be provided in the analysis accuracy calculation unit 105C. That is, the analysis accuracy calculation unit 105C may calculate the feature quantity using the media data itself as a parameter.

4.3) Effect According to the fourth embodiment of the present invention, it is possible to further improve the sum of weighted analysis accuracies of a plurality of analysis engines by considering the feature amount of media. As mentioned above, if an analysis engine has the characteristic that the analysis accuracy does not improve even if the bit rate is increased for dark images, the analysis accuracy can be improved by using a coding parameter set suitable for an analysis engine that functions effectively even for dark images. improves the weighted sum of Similarly, when the motion is small, the weighted sum of the analysis accuracy is improved by using an encoding parameter set suitable for an analysis engine that provides high analysis accuracy even if the frame rate is low.

5. Fifth Embodiment The media encoding devices (10, 10A, 10B or 10C) according to the first to fourth embodiments described above are realized by small computers such as Raspberry Pi (registered trademark) and information communication devices such as smartphones. sell.

As illustrated in FIG. 14 , information communication apparatus 600 implements the encoding parameter set determination unit of each of the above embodiments by processor 109 and program memory 110 . The processor 109 reads from the program memory 110 a program for controlling the overall operation shown in FIG. 2 and a program for controlling the encoding parameter set determination operation (S204) therein, and executes them. The encoding parameter set determination operation (S204) is as described in the first to fourth embodiments above. Processor 109 may be a processor that controls the operation of information communication device 600, or a processor that controls the operation of a media encoding device. The functions of the encoding units (102, 102B, 102C) can also be realized by the processor executing the program.

6. System Configuration As illustrated in FIG. 15, consider a communication system in which an information communication device 601 and a server 602 are connected via a network 603 . The information communication device 601 has a media encoding device 604 according to each embodiment described above. A server 602 is installed in a data center or the like, and has a media decoding device 605 and a media analysis device 606 . Network 603 is, for example, a combination of a mobile phone network and the Internet. The media analysis device 606 can be implemented by running a video analysis program on the processor of the server 602 .

Media encoding device 604 encodes and transmits media data in a form suitable for data analysis by media analysis device 606, as described above. In server 602, media decoding device 605 decodes the received signal into media data, which is analyzed by media analysis device 606. FIG. Therefore, even if the processing capability of the media encoding device 604 is low, media analysis can be performed without greatly lowering the accuracy.

Further, when the media encoding device 604 has a relatively high processing capability, the media analysis unit 108 is provided as in the fourth embodiment to calculate the feature amount c of the media and obtain a more suitable encoding parameter set. By determining , the analysis accuracy can be further improved.

7. Additional Notes Some or all of the embodiments and examples described above can also be described as the following additional notes, but are not limited to these.
(Appendix 1)
an encoding unit that encodes media data;
a bit rate input unit for inputting an upper limit value of the encoding bit rate of the media data;
an analysis method input unit for inputting an analysis method for the media data;
an encoding parameter calculation unit that determines an encoding parameter set to be set in the encoding unit;
an analysis accuracy calculation unit that inputs the analysis method and the encoding parameter set and calculates the analysis accuracy in the analysis method;
The encoding parameter calculation unit generates a plurality of candidates for the encoding parameter set, calculates the analysis accuracy for the candidates using the analysis accuracy calculation unit, and performs the encoding based on the calculated analysis accuracy. A media encoding device, which determines the encoding parameter set to be set in a section.
(Appendix 2)
The media encoding device according to Supplementary Note 1,
the media data is video data, the encoding parameter set includes a bit rate and a frame rate of the video data;
The coding parameter set candidates have different frame rates,
The media encoding device, wherein the encoding parameter calculation unit outputs the encoding parameter set with the highest analysis accuracy among the encoding parameter set candidates.
(Appendix 3)
The media encoding device according to Appendix 1 or 2,
The media data is video data, and the analysis accuracy calculation unit outputs the analysis accuracy in the analysis method when the analysis method and the bit rate and frame rate of the video data are input. Media Encoder.
(Appendix 4)
The media encoding device according to appendix 2 or 3,
the encoding parameter set further comprising an encoding method;
The media encoding device, wherein the encoding parameter set candidates are different in the frame rate and the encoding method.
(Appendix 5)
The media encoding device according to appendix 3 or 4,
When the analysis method, the bit rate and frame rate of the video data, and the encoding method are input, the analysis accuracy calculation unit outputs the analysis accuracy in the analysis method. Device.
(Appendix 6)
The media encoding device according to any one of Appendices 1 to 5,
a plurality of analysis methods and the weight of each analysis method are input from the analysis method input unit;
The media encoding device, wherein the encoding parameter calculation unit outputs the encoding parameter set that maximizes the weighted sum of the analysis accuracies.
(Appendix 7)
The media encoding device according to appendix 6,
The encoding unit encodes media data using a layered encoding technique;
The coding parameter set includes the number of layers and the bit rate and frame rate of each layer,
The media encoding device, wherein the encoding parameter calculation unit outputs allocation of the encoding parameter set and each analysis method to the layer, which gives the highest weighted sum of the analysis accuracies.
(Appendix 8)
The media encoding device according to Appendices 1 to 7,
a media analysis unit that calculates a feature amount of the media data input from the media data input unit or the media data encoded by the encoding unit;
The media encoding device, wherein the analysis accuracy calculation unit calculates the analysis accuracy using an analysis result of the media analysis unit.
(Appendix 9)
The media encoding device according to Supplementary Note 8,
The media data is video data, and the analysis accuracy calculation unit receives the analysis method, the bit rate and frame rate of the video data, and the feature amount output by the media analysis unit. A media encoding device, characterized by outputting analysis accuracy in the method.
(Appendix 10)
a first information communication device having the media encoding device according to any one of Appendices 1 to 9;
a second information communication device connected to the first information communication device through a network and having an analysis engine corresponding to the analysis method for analyzing media data received from the first information communication device;
A communication system consisting of
(Appendix 11)
an encoder encoding the media data;
A bit rate input unit inputs an upper limit value of the encoding bit rate of the media data;
an analysis method input unit inputs the analysis method of the media data;
an analysis accuracy calculation unit inputs the analysis method and the encoding parameter set and calculates the analysis accuracy in the analysis method;
An encoding parameter calculation unit generates a plurality of candidates for the encoding parameter set, calculates the analysis accuracy for the candidates using the analysis accuracy calculation unit, and uses the calculated analysis accuracy to generate the encoding unit. determining the encoding parameter set to set to
A media encoding method characterized by:
(Appendix 12)
A program that causes a computer to function as a media encoding device in a communication system that transmits encoded media data over a network,
an encoder encoding the media data;
A bit rate input unit inputs an upper limit value of the encoding bit rate of the media data;
an analysis method input unit inputs the analysis method of the media data;
an analysis accuracy calculation unit inputs the analysis method and the encoding parameter set and calculates the analysis accuracy in the analysis method;
An encoding parameter calculation unit generates a plurality of candidates for the encoding parameter set, calculates the analysis accuracy for the candidates using the analysis accuracy calculation unit, and uses the calculated analysis accuracy to generate the encoding unit. determining the encoding parameter set to set to
A program that implements a function on the computer.

INDUSTRIAL APPLICABILITY The present invention can be applied to storage of video/audio captured by surveillance cameras, in-vehicle cameras, etc. in a storage medium, or to media encoding in a video distribution system that distributes captured video/audio to a monitoring center.

10, 10A, 10B, 10C, 604 media encoding device 101 media data input units 102, 102B, 102C encoding unit 103 output units 104, 104A, 104B, 104C encoding parameter calculation units 105, 105A, 105B, 105C analysis accuracy calculation unit 106 analysis method input unit 107 bit rate input unit 108 media analysis unit 109 processor 110 program memory

Claims (10)

A media encoding device in a communication system for transmitting encoded media data over a network,
an encoder that encodes media data according to an encoding parameter set and outputs the encoded media data;
an input unit for inputting an upper limit value of the encoding bit rate of the media data and information on an analysis method of the media data;
generating a plurality of coding parameter set candidates under conditions of the analysis method and the upper limit value of the coding bit rate, calculating analysis accuracy for each of the plurality of coding parameter set candidates, and calculating the calculated analysis accuracy a coding parameter set determination unit that determines a higher coding parameter set candidate as the coding parameter set to be provided to the encoder;
A media encoding device having A media encoding device according to claim 1, comprising:
A media encoding device in which the encoding parameter set determination unit calculates the analysis accuracy of the analysis method using a function or an analysis accuracy table prepared in advance. A media encoding device according to claim 1 or 2,
the media data is video data, the encoding parameter set includes a bit rate and a frame rate of the video data;
A media encoding device, wherein the encoding parameter set determination unit generates the plurality of encoding parameter set candidates such that the frame rates are different. A media encoding device according to claim 3, comprising:
the encoding parameter set further comprising an encoding method;
A media encoding apparatus, wherein the encoding parameter set determination unit generates the plurality of encoding parameter set candidates such that at least one of the frame rate and the encoding method is different. A media encoding device according to any one of claims 1-4,
the input unit inputs a plurality of analysis methods and a weight for each analysis method;
The media encoding device, wherein the encoding parameter set determination unit determines, as the encoding parameter set, a coding parameter set candidate with the highest weighted sum of the analysis accuracies of the plurality of analysis methods. A media encoding device according to any one of claims 1-5,
the encoder layer-encodes the media data;
the input unit inputs a plurality of analysis methods and a weight for each analysis method;
The coding parameter set includes the number of layers of the layered coding, the bit rate of each layer, and the allocation of the analysis method to each layer,
The coding parameter set determination unit calculates analysis accuracy for each bit rate combination candidate for each layer and for each allocation candidate for each layer of the plurality of analysis methods, and the code with the higher calculated analysis accuracy A media encoding apparatus for determining a candidate encoding parameter set as the encoding parameter set to be provided to the encoder. A media encoding device according to any one of claims 1-6,
further comprising a media analysis unit that calculates a media feature amount from the media data or the encoded media data;
The media encoding device, wherein the encoding parameter set determination unit calculates analysis accuracy for each of the plurality of encoding parameter set candidates based on the media feature quantity. A media encoding device according to claim 7, comprising:
The media data is video data, and the coding parameter set determination unit calculates analysis accuracy in the analysis method based on the analysis method, the bit rate and frame rate of the video data, and the media feature amount. A media encoder that A media encoding method in a communication system for transmitting encoded media data over a network, comprising:
an encoder encoding media data according to an encoding parameter set and outputting the encoded media data;
The encoding parameter determination unit
Inputting the upper limit value of the coding bit rate of the media data and information about the analysis method of the media data, and generating a plurality of coding parameter set candidates under the condition of the analysis method and the upper limit value of the coding bit rate. death,
calculating analysis accuracy for each of the plurality of coding parameter set candidates;
determining a coding parameter set candidate with a higher calculated analysis accuracy as the coding parameter set to be provided to the encoder;
Media encoding method. A program that causes a computer to function as a media encoding device in a communication system that transmits encoded media data over a network,
encoding media data according to an encoding parameter set and outputting the encoded media data;
Inputting the upper limit value of the coding bit rate of the media data and information about the analysis method of the media data, and generating a plurality of coding parameter set candidates under the condition of the analysis method and the upper limit value of the coding bit rate. and the ability to
a function of calculating analysis accuracy for each of the plurality of coding parameter set candidates;
a function of determining a coding parameter set candidate with a higher calculated analysis accuracy as the coding parameter set;
on the computer.

Images