JP7321237B2 - Image encoding device, image encoding method, program - Google Patents

Description

The present invention relates to an image encoding device, an image encoding method, and a program.

CCD or CMOS image sensors are often used as the imaging elements of modern digital cameras, and single-chip imaging elements pass through a so-called Bayer array color filter to capture green, blue, and red pixel data (hereinafter referred to as RAW data). ) is obtained. In a digital camera, the RAW data undergoes development processing including demosaic processing, noise reduction processing, optical distortion correction, color correction processing, etc. to generate the final image data. H. Compressed image data can be recorded by an encoding method such as H.264. On the other hand, it also has a function of recording RAW data before development processing so that development processing can be executed according to the user's own preference.

A non-compressed or reversible compression format is often used as the recording format of RAW data, and the size of recorded RAW data is generally larger than that of compressed image data after development. On the other hand, as the density and number of pixels of image pickup devices increase, it is becoming important to record RAW data in a compact manner. Therefore, as a visually lossless lossy compression format, multiple color channels are generated from the RAW data, a two-dimensional discrete wavelet transform is performed on each channel, and each generated subband is individually quantized. A method of compressing and recording as encoded data has been proposed (see Patent Document 1).

Japanese Patent Publication No. 2002-516540

However, when four color channels are generated from RAW data and octave division is performed on three layers by two-dimensional discrete wavelet transform, each channel is divided into ten subbands, so four channels x ten subbands are generated. will be If we quantize these subbands separately, we need 4*10=40 quantization parameters. Furthermore, when it is assumed that finer quantization control is performed in units of predetermined blocks according to features in the screen, quantization parameters for 4×10×the number of divided blocks are required. Since all of these quantization parameters are included in the encoded data, the data amount of the quantization parameters increases with the compression of RAW data. In this case, it becomes difficult to record RAW data compactly while maintaining visually lossless image quality.

Therefore, the present invention realizes lossy compression by flexible quantization control in compressing RAW data, and suppresses the data amount of the quantization parameter included in the encoded data, thereby recording the RAW data compactly. We provide the technology to make it possible.

In order to achieve the above object, the image encoding device according to the present invention comprises:
generating means for performing a discrete wavelet transform with two or more decomposition levels to generate a plurality of subband data from raw data;
common quantization that divides the plurality of subband data into the same number of quantization units, individually sets each quantization unit, and commonly assigns the plurality of subband data included in the same quantization unit quantization for generating parameters, and generating quantization parameters for quantizing each of the plurality of subband data based on the common quantization parameter and coefficients set for each of the plurality of subband data; parameter generation means;
quantization means for quantizing each of the plurality of sub-band data based on the quantization parameter for quantizing each of the plurality of sub-band data;
encoding means for encoding the quantization result obtained by the quantization for each subband;

According to the present invention, it is possible to compactly record RAW data by suppressing the data amount of the quantization parameter to be included in encoded data while realizing lossy compression by flexible quantization control in compression of RAW data. be possible.

1 is a block diagram showing a configuration example of an image processing apparatus according to an embodiment of the invention, and a diagram showing RAW data in a Bayer array; FIG. FIG. 2 is a block diagram showing a configuration example of a RAW data encoding/decoding section 103 corresponding to an embodiment of the invention; FIG. 4 shows an example of channel transforms corresponding to an embodiment of the invention; FIG. 4 is a diagram for explaining a frequency transform (subband division) and a predictive coding method (MED prediction) corresponding to an embodiment of the invention; The figure which shows an example of the quantization control unit corresponding to embodiment of invention. FIG. 4 is a diagram showing an example of quantization control units based on image quality characteristics, corresponding to an embodiment of the invention; 4 is a flowchart corresponding to an example of quantization control processing based on image quality characteristics, according to an embodiment of the invention; FIG. 4 is a diagram showing the relationship between quantization control units related to image quality control and RAW data, corresponding to the embodiment of the invention; The figure which shows an example of the data structure of encoded data corresponding to embodiment of invention. The figure which shows an example of the syntax element of header information corresponding to embodiment of invention. FIG. 4 is a diagram showing an example of mapping common quantization parameters to a two-dimensional coordinate system, corresponding to an embodiment of the invention; FIG. 10 is a diagram showing an example of quantization control units based on image quality characteristics, corresponding to the second embodiment of the invention; FIG. 8 is a diagram showing a comparison of the amount of data included in encoded data with a conventional method, corresponding to the second embodiment of the invention;

Preferred embodiments of the present invention will be described below with reference to the drawings.

[Embodiment 1]
Embodiments of the invention will be described below. FIG. 1(a) shows an imaging device 100 corresponding to an embodiment of the invention. The imaging device 100 can be implemented as, for example, a digital camera or a digital video camera. In addition, it can also be realized as an arbitrary information processing terminal or image processing device such as a personal computer, mobile phone, smart phone, PDA, tablet terminal, portable media player, or the like. Note that FIG. 1A shows a configuration including an imaging unit 102 in consideration of a case where it functions as an imaging device such as a digital camera. However, as an embodiment of the invention, the imaging unit 102 is provided as an image encoding device/image decoding device, an image recording device, an image compression device, an image restoration device, etc. that can compress, record, and reproduce a RAW image. It may be realized with a configuration that does not

In the imaging device 100 , the control unit 101 controls each processing unit that configures the imaging device 100 . The imaging unit 102 includes a lens optical system capable of optical zoom including an optical lens, an aperture, a focus control and a lens driving unit, and a CCD image sensor or CMOS image sensor that converts optical information from the lens optical system into an electric signal. Includes an image sensor. The imaging unit 102 outputs RAW data obtained by converting an electrical signal obtained by the imaging device into a digital signal to the RAW data encoding/decoding unit 103 .

Here, the RAW data is composed of an effective imaging area, which is a pixel area that receives light, and an optical black area, which is a light-shielded pixel area, as shown in FIG. 1(b). Each pixel consists of a periodic pattern of R (red), G1 (green), G2 (green), and B (blue), and human visual sensitivity is more sensitive to luminance components than to color components. The area of green, which contains a large amount of luminance components, is allocated twice as much as that of red and blue. In the present embodiment, the RAW data is composed of four color elements in a Bayer arrangement, but the arrangement and color elements are not limited to this arrangement, and other schemes may be used.

A RAW data encoding/decoding unit 103 encodes the RAW data acquired by the imaging unit 102 and writes the generated encoded data to the memory 105 . Alternatively, unencoded RAW data may be acquired from the recording medium 107 and encoded data may be generated by executing encoding processing corresponding to the present embodiment. Also, the RAW data encoding/decoding unit 103 can decode the encoded data to restore the RAW data. A detailed configuration and operation of the RAW data encoding/decoding unit 103 will be described later.

The memory I/F unit 104 arbitrates memory access requests from each processing unit and controls reading or writing to the memory 105 . The memory 105 is a storage area configured by, for example, a volatile memory for storing various data output from each processing unit that configures the imaging apparatus 100 . The recording processing unit 106 reads the encoded data stored in the memory 105 , converts it into a predetermined recording format, and records it on the recording medium 107 . The recording medium 107 is a recording medium composed of, for example, a non-volatile memory, and is detachably attached to the imaging apparatus 100 . An imaging apparatus is configured by the above-described processing units.

Next, the detailed configuration and processing flow of the RAW data encoding/decoding unit 103 will be described with reference to the block diagrams shown in FIGS. 2(a) and 2(b). 2A shows the configuration of the RAW data encoding unit 103E in the RAW data encoding/decoding unit 103, and FIG. 2B shows the configuration of the RAW data decoding unit 103D in the RAW data encoding/decoding unit 103. Show configuration.

2A, the RAW data encoding unit 103E mainly includes a channel transform unit 201, a frequency transform unit 202, a quantization parameter generation unit 203, a quantization unit 204, an encoding unit 205, and a quantization parameter encoding unit 206. consists of The channel conversion unit 201 converts the input Bayer array RAW data as shown in FIG. 3 into a plurality of channels. For the frequency conversion unit 202 and the quantization unit 204, blocks corresponding to the number of channels obtained by conversion are prepared and processed in parallel. In this embodiment, the channel conversion unit 201 converts the RAW data into four channels for each of R, G1, G2, and B in the Bayer array, and further converts R, G1, G2, and B using the following conversion formula: 1 can be converted into four channels C0 to C3.

C0 = a + c
C1=B−G2 Formula 1
C2=R-G1
C3=ba
However, a = G2 + [C1/2], b = G1 + [C2/2], c = a + [C3/2]
Also, in Equation 1, the floor function [x] represents the maximum integer less than or equal to x.

Here, as shown in FIG. 3(a), an example of the configuration for conversion into four channels is shown, but as shown in FIGS. 3(b) and 3(c), R, G1 and G2 are combined. It may be converted into three channels for each of G and B, and the number of channels and conversion method may be other methods than the above.

Next, frequency transform section 202 performs frequency transform processing by discrete wavelet transform at a predetermined decomposition level (hereinafter simply referred to as “level” or “lev”) for each channel, and generates a plurality of subband data ( transform coefficients) to the quantization parameter generation section 203 and the quantization section 204 .

FIG. 4(a) shows a filter bank configuration for realizing the discrete wavelet transform related to subband division processing with lev=1. As a result of executing the discrete wavelet transform processing in the horizontal and vertical directions, It is first split into one low frequency sub-band (LL) and three high frequency sub-bands (HL, LH, HH) as shown in b). The transfer functions of the low-pass filter (hereinafter referred to as lpf) and the high-pass filter (hereinafter referred to as hpf) shown in FIG.
lpf(z)=(-z ⁻² +2z ⁻¹ +6+2z−z ² )/8 Equation 2
hpf(z)=(−z ⁻¹ +2−z)/2 Equation 3

Also, "↓2" represents 2:1 downsampling. Therefore, one downsampling is performed in the horizontal direction and one in the vertical direction, so that the size of the subbands is 1/4 of the original size. If lev is greater than 1, subband division is performed hierarchically for the low-frequency subband (LL), e.g., if lev=3, then 10 subbands as shown in FIG. Subband data corresponding to the subband is generated. Here, the discrete wavelet transform is composed of 5-tap lpf and 3-tap hpf as shown in the above equations 2 and 3, but a filter configuration with a different number of taps and different coefficients may be used. .

Quantization parameter generation section 203 generates a first quantization parameter common to all channels and all subbands for performing quantization processing on subband data (transform coefficients) generated by frequency transformation section 202. and output to quantization parameter coding section 206 . Also, a second quantization parameter for each channel and each subband is generated from the first quantization parameter and supplied to the quantization section 204 . The generation unit of the first quantization parameter and the second quantization parameter and the generation method thereof will be described later with reference to FIGS. 5 to 7 and the like.

The quantization unit 204 performs quantization processing on the subband data (transform coefficients) input from the frequency transform unit 202 based on the second quantization parameter supplied from the quantization parameter generation unit 203, and performs quantization. As a result, quantized subband data (transform coefficients) are output to encoding section 205 .

The coding unit 205 performs predictive differential entropy coding on the quantized subband data (transform coefficients) input from the quantization unit 204 in raster scan order for each subband. As a result, the subband data after quantization is encoded in units of subbands. Here, as shown in FIG. 4D, a predicted value pd is generated by MED (Median Edge Detector) prediction from peripheral data of the data to be encoded (transform coefficient), and the value xd of the data to be encoded x and the predicted value The difference data from pd is entropy coded by Huffman coding, Golomb coding, or the like, and the generated coded data is stored in the memory 105 . At this time, since there is no peripheral data in the leading data in the raster scan order of each subband, the value xd of the encoding target data x is encoded as it is. Then, difference data is encoded for subsequent encoding target data. Note that the predictive coding method and the entropy coding method may be other methods. In addition, the amount of code generated for each line for each subband is supplied to the quantization parameter generating section 203 . A specific data structure of the encoded data generated here will be described later with reference to FIGS. 9 and 10. FIG.

The quantization parameter encoding unit 206 is a processing unit that encodes the first quantization parameter input from the quantization parameter generation unit 203 . Quantization parameter encoding section 206 encodes the quantization parameter by the same encoding method as encoding section 205 and stores the generated encoded quantization parameter in memory 105 . FIG. 2(a) shows the case where the quantization parameter encoding unit 206 is provided independently of the encoding unit 205, but since both employ a common encoding method, the function of the quantization parameter encoding unit 206 is can be shared by the encoding unit 205 . The RAW data encoding unit 103E is configured as described above.

Next, the configuration of the RAW data decoding unit 103D will be described. RAW data decoding section 103D mainly includes decoding section 211, quantization parameter decoding section 212, quantization parameter generation section 213, inverse quantization section 214, frequency inverse transform section 215, and channel inverse transform section 216.

First, the decoding unit 211 decodes encoded data input from the memory 105 using a decoding method corresponding to the encoding method adopted by the RAW data encoding unit 103E, and converts quantized subband data (transform coefficients) into to restore. For example, if a predictive differential entropy coding scheme has been used, a corresponding predictive differential entropy decoding scheme is adopted. At this time, as shown in FIG. 4(d), in the coded data, the difference data between the prediction value pd generated from the peripheral data of the data to be coded (transform coefficient) and the value xd of the data to be coded x is the entropy It is encoded. Therefore, when the decoding unit 211 decodes the encoded data by the entropy decoding method to restore the difference data, it generates a prediction value pd from the decoded data around the decoding target data, and adds the difference data to the encoding target data. Restore x. Since the encoded data consists of data encoded in units of subbands, decoding section 211 also restores transform coefficients in units of subbands.

Quantization parameter decoding section 212 decodes the coded first quantization parameter included in the coded data input from memory 105 by a decoding method common to decoding section 211, and obtains the first quantization parameter to restore. Quantization parameter decoding section 212 also decodes predetermined coefficients (α and β in Equation 5 described later) for calculating the second quantization parameter for each channel and subband. The decoding result is output to quantization parameter generation section 213 . Quantization parameter generation section 213 generates a second quantization parameter for each channel and subband from the first quantization parameter provided from quantization parameter decoding section 212 and a predetermined coefficient, and performs inverse quantization. 214.

Inverse quantization section 214 inversely quantizes the quantized subband data in channel units output from decoding section 211 using the second quantization parameter provided from quantization parameter generation section 213, Restore band data. The frequency inverse transform unit 215 performs frequency inverse transform processing by discrete wavelet inverse transform, and restores channels from subband data (transform coefficients) of a predetermined division level. The channel inverse transform unit 216 inverse transforms the input channel data to restore the original RAW data. For example, when the channel conversion unit 201 of the RAW data encoding unit 103E converts R, G1, G2, and B in the Bayer array into four channels C0 to C3 according to Equation 1, inverse conversion processing corresponding to Equation 1 RAW data of R, G1, G2, and B are restored from four channels C0 to C3 according to . The RAW data decoding unit 103D is configured as described above.

Next, quantization parameter generation processing in quantization parameter generation section 203 will be described. In this embodiment, a first quantization parameter common to all channels and all subbands is generated, and individual second quantization parameters for each channel and each subband are generated from the first quantization parameter. Generate. Quantization parameter generation section 203 performs quantization parameter generation processing related to code amount control for each predetermined subband data unit based on a target code amount set in advance before encoding. A common first quantization parameter is generated.

In this embodiment, the case where the first quantization parameter is not directly generated but generated in two steps will be described. First, for each subband data, a quantization parameter (third quantization parameter) common to all channels and all subbands is generated for each predetermined line. Then, the predetermined line is divided into units called segments, and the first quantization parameter is generated by correcting the third quantization parameter according to the image quality characteristics of the subband data in the segment. The procedure for generating the first quantization parameter will be described in detail below.

FIG. 5 shows a unit for updating the code amount evaluation and quantization parameter related to code amount control when each channel is divided into subbands with lev=3. Level 2 subbands (2HL, 2LH, 2HH) for 1×N (N is an integer) lines in the vertical direction of level 3 subbands (3LL, 3HL, 3LH, 3HH) of four channels from C0 to C3 2×N lines in the vertical direction and 4×N lines in the vertical direction of the level 1 subbands (1HL, 1LH, 1HH) are grouped together into one processing unit (first processing unit). do. As shown in FIG. 5, the first processing unit is set as a region (band) obtained by vertically dividing each sub-band into predetermined lines. In particular, FIG. 5 shows the case of N=1.

Quantization parameter generation section 203 compares the target code amount and the generated code amount corresponding to the first processing unit, and determines the generated code amount for data of all channels and all subbands of the next processing unit as the target generated code amount. A third quantization parameter (QpBr) common to all channels and all subbands is generated by performing feedback control so as to approach the quantity. If code amount control within a screen is not performed for RAW data, QpBr common to all channels and all subbands, which is fixed for the entire screen regardless of the code amount control unit as described above, is set or should be generated. The control of the code amount at this time can be executed according to Equation 4 below.

QpBr(i)=QpBr(0)+r×Σ{S(i−1)−T(i−1)} Equation 4
QpBr(0): initial quantization parameter QpBr(i): i-th quantization parameter (i>0)
r: control sensitivity
S: Generated code amount
T: Target code size

In Equation 4, the initial quantization parameter set for the leading band of each band (first processing unit) set in the subband data is used as a reference, and the difference between the generated code amount and the target code amount is Adjust the initial quantization parameter according to the magnitude. Specifically, the initial quantization parameter is set so that the code amount difference between the total code amount generated after the first band (total generated code amount) and the corresponding total target code amount (total target code amount) becomes small. is adjusted to determine the third quantization parameter for the band being processed. A parameter determined by the quantization parameter generation unit 203 based on the compression rate or the like can be used as the initial quantization parameter set for the leading band.

After generating the third quantization parameter (QpBr) for each band in this way, the quantization parameter generation unit 203 further subdivides (segments) the band into segments, which are second processing units. A first quantization parameter (QpBs) is generated by adjusting the third quantization parameter (QpBr) according to the image quality evaluation result of the subband data. Thereby, it is possible to generate a quantization parameter related to image quality control that adjusts the intensity of quantization according to the characteristics of RAW data.

FIG. 6A shows a second processing unit (segment) for updating the evaluation and quantization parameters of subband data related to image quality control when each channel is divided into subbands at lev=3. In each segment, the level 3 subbands (3LL, 3HL, 3LH, 3HH) of all channels are horizontally and vertically (1×M)×(1×N) (M and N are integers), respectively, and level 2 (2×M)×(2×N) in the horizontal and vertical directions of subbands (2HL, 2LH, 2HH) of level 1, and (4 xM) x (4 x N).

The segment as the second processing unit shown here is obtained by subdividing the band as the first processing unit in the horizontal direction. In this embodiment, segmentation is performed so that the number of segments included in each subband is the same. When M=1 and N=1, the segment corresponds to level 1 and can be regarded as dividing the band horizontally into units of 4 pixels. The quantization parameter generation unit 203 generates the first quantization parameter (QpBs) for each segment by correcting the third quantization parameter QpBr calculated for each band according to the image quality characteristic for each segment. .

FIG. 6B shows the relationship between the first quantization parameter (QpBs) and subband data when M=1 and N=1. As shown here, one first quantization parameter (QpBs) is generated for multiple subband data of each channel included in one segment.

As for the image quality evaluation for each segment, among the subband data included in each segment, for example, the 3LL subband data is evaluated as the low frequency component, and the 1HL, 1LH, and 1HH subband data are used to evaluate the high frequency component. At this time, the code amount control is performed so that the smaller the amplitude of the low-frequency component, the finer the quantization, and the larger the amplitude of the high-frequency component, the coarser the quantization. Feedforward control may be performed so as to adjust the offset.

FIG. 7 shows a flowchart corresponding to an example of adjustment processing of the third quantization parameter (QpBr) based on image quality evaluation, executed by the quantization parameter generation unit 203 . The processing corresponding to the flowchart can be realized, for example, by one or more processors functioning as the quantization parameter generation unit 203 executing corresponding programs (stored in ROM or the like).

In S701, the quantization parameter generation unit 203 determines low frequency components. The 3LL transform coefficient is compared with the threshold Th1 set for determination, and if the 3LL transform coefficient is equal to or less than the threshold ("YES" in S701), the process proceeds to S702. In S702, the quantization parameter generation unit 203 decreases the value of the third quantization parameter QpBr set for the band to which the segment to be processed belongs. For example, QpBr is multiplied by a factor less than 1, or a predetermined value is subtracted from QpBr.

If it is determined in S701 that the conversion coefficient of 3LL is greater than the threshold Th1 ("NO" in S701), the process proceeds to S703. In S703, the quantization parameter generation unit 203 determines high frequency components. The average value of the conversion coefficients of 1HL, 1LH, and 1HH is compared with the threshold value Th2 set for determination, and if the average value is smaller than the threshold value Th2 ("NO" in S703), the process proceeds to S704. If the average value is greater than or equal to the threshold Th2 ("YES" in S703), the process proceeds to S705. At S704, the value of QpBr is maintained. At S705, the value of QpBr is increased. For example, QpBr is multiplied by a factor of 1 or more, or a predetermined value is added to QpBr.

As described above, the result obtained by adjusting the third quantization parameter QpBr for each segment is set as the first quantization parameter QpBs of the segment. When quantization control related to image quality control is not performed, QpBr may be set to QpBs as it is.

The first quantization parameter QpBs common to all channels and all subbands generated in the above manner is, for example, when M=1 and N=1, 8×8 pixels of each channel as shown in FIG. Quantization control for each block of RAW data corresponding to 16×16 pixels of the original RAW data is possible.

Next, the process of using the first quantization parameter QpBs to generate a separate second quantization parameter QpSb for each channel, each subband will be described. In the present embodiment, as described above, one first quantization parameter QpBs is generated for each segment, but the quantization of each segment of each channel does not use QpBs as it is, but separately A corresponding second quantization parameter QpSb is generated from QpBs and quantization processing is performed. The QpSb generation process will be described below.

Formula 5 is a calculation formula for QpSb generation.
QpSb[i][j]=QpBs×α[i][j]+β[i][j] Equation 5
QpSb: second quantization parameter for each channel, each subband separately
QpBs: First quantization parameter common to all channels and all subbands α: slope β: intercept
i: channel index (0-3)
j: subband index (0 to 9)

In Equation 5, the slope α and the intercept β are coefficients (variables) individually given to each channel and each subband, and can be determined in advance. The value of α increases with the relationship C0>C1=C2>C3 for channels C0 to C3, and the larger the value of α, the larger the value of QpSb. For each subband, the values of α and β are decreased in order to decrease the quantization parameter values for the subbands closer to the low frequency components, and the quantization parameter values are increased for the subbands closer to the high frequency components. Increase the values of α and β.

Also, each value of α[i][j] and β[i][j] is set in advance, but a plurality of combinations may be prepared according to the set compression rate. For example, when the compression rate is high, information is lost if the value of QpSb is too large. Therefore, the values of α and β can be adjusted to be smaller so that QpSb does not become too large.

In this way, the first quantization parameter QpBs is modified by the individual weighting factors α, β for each channel, each subband to generate the second quantization parameter QpSb, so that for each channel, each subband: It becomes possible to flexibly control the quantization. The quantization section 204 quantizes the transform coefficients obtained from the frequency transform section 202 using the second quantization parameter QpSb for each channel and each subband generated based on Equation 5.

The method of generating the second quantization parameter QpSb in RAW data decoding section 103 can also be performed according to Equation 5 above. At that time, the first quantization parameter QpSr and the values of α and β are provided from the quantization parameter decoding unit 212 .

The subband data of each channel encoded by the RAW data encoding/decoding unit 103 as described above and the first quantization parameter QpBs are multiplexed by the recording processing unit 106 based on a predetermined data format. , are recorded as encoded data. FIG. 9(a) shows an example of a data structure for recording encoded data. As shown in FIG. 9(a), the encoded data has a hierarchical structure, starting with "main_header" indicating information related to the entire encoded data, and dividing the RAW data into tiles into a plurality of pixel blocks for encoding. Data can be stored in units of tiles by 'tile_header' and 'tile_data' on the assumption that the data will be converted into a tile. In the above description, a case has been described in which RAW data is directly encoded in units of channel and subband data, but the present invention can be applied not only to tiles but also to units of predetermined sets in RAW data. That is, when the RAW data is divided into tiles and the tiles are encoded in units of channel and subband data, the same processing as described above may be performed for each tile obtained by dividing the RAW data into a plurality of pieces. . In this case, the subband data of each channel and the first quantization parameter QpBs are coded for each tile and included in the coded data. If tile division is not performed, there will be only one "tile_header" and one "tile_data".

"tile_data" contains the following elements: First, "qp_header" indicating information related to the encoded first quantization parameter QpBs and "coded_qp_data" which is the encoded first quantization parameter itself are arranged. The following "coded_raw_data" is arranged in units of channels, and the coded data for each channel is stored in order of "channel_header" indicating information related to each channel and "channel_data" which is the main data for each channel. be done. "channel_data" is composed of a set of coded data for each sub-band, and "sb_header" indicating information related to each sub-band and "sb_data" as coded data for each sub-band are sub-bands. They are arranged in band index order. The subband index is as shown in FIG. 9(b).

Next, the syntax elements of each header information will be explained based on FIG. "main_header" consists of the following elements. "coded_data_size" indicates the data size of the entire encoded data. "width" indicates the width of RAW data. "height" indicates the height of the RAW data. "depth" indicates the bit depth of RAW data. "channel" indicates the number of channels when encoding RAW data. "type" indicates the type of channel transform. "lev" indicates the subband level of each channel.

"tile_header" is composed of the following elements. "tile_index" indicates a tile index for identifying a tile division position. "tile_data_size" indicates the amount of data included in the tile. "tile_width" indicates the width of the tile. "tile_height" indicates the height of the tile.

"qp_header" is composed of the following elements. "qp_data_size" indicates the data size of the encoded first quantization parameter. “qp_width” indicates the width of the first quantization parameter, that is, the number of horizontal first quantization parameters corresponding to RAW data. "qp_height" indicates the height of the first quantization parameter, that is, the first quantization parameter number in the vertical direction corresponding to the RAW data.

"channel_header" is composed of the following elements. 'channel_index' indicates a channel index for identifying a channel. "channel_data_size" indicates the data amount of the channel. "sb_header" is composed of the following elements. 'sb_index' indicates a subband index for identifying a subband. 'sb_data_size' indicates the coded data amount of the sub-band. 'sb_qp_a' denotes the α value shown in Equation 5 for generating the quantization parameter for each subband. 'sb_qp_b' denotes the β value shown in Equation 5 for generating the quantization parameter for each subband.

According to the invention corresponding to the above embodiments, instead of recording all quantization parameters for each channel and each subband, common quantization parameters for all channels and all subbands are recorded, and the common quantization is performed. Quantization parameters for each channel and each subband can be generated from the parameters. This makes it possible to achieve the effect of reducing the data amount of the quantization parameter as described below.

Comparison of the amount of data between recording all individual quantization parameters for each channel and each subband (conventional example) and recording common quantization parameters for all channels and all subbands (this embodiment) think. Here, one quantization parameter will be recorded for each sub-band from level 1 to level 3. Therefore, the data size of the quantization parameters to be recorded for one segment is (1×4 subbands+1×3 subbands+1×3 subbands)×4 channels, which is 40 in total. On the other hand, since only one common quantization parameter is required, it is 1. By using the common quantization parameter as the quantization parameter to be recorded, the size of the quantization parameter can be reduced to 1/40. .

Furthermore, as shown in FIG. 11, the common quantization parameters to be recorded can be regarded as screen data mapped on a two-dimensional coordinate system. prediction) in the same manner can reduce the data amount of the quantization parameter. In FIG. 11(b), QpBs[p, q] indicates the common quantization parameter assigned to each segment, p takes a value from 1 to the division number P of the segment in the vertical direction, and q is It can take values from 1 to Q, the division number of horizontal segments. Each value of [p, q] corresponds to the position of each segment in the subband.

As described above, when RAW data is converted into a plurality of channels and subband division is performed for each channel, each channel and each channel is divided by a quantization parameter and a weighting function common to all channels and all subbands in segment units. A subband-specific quantization parameter can be generated. As a result, while realizing flexible quantization control, predictive coding is performed using a common quantization parameter as two-dimensional data, which greatly reduces the amount of quantization parameter data, resulting in compact RAW data with high image quality. Recording becomes possible.

[Embodiment 2]
A second embodiment of the invention will be described below. The imaging device of this embodiment can have the same configuration as that of the imaging device of the first embodiment. However, the generation unit of the first quantization parameter in the quantization parameter generation unit 203 of the RAW data encoding/decoding unit 103 and the process of generating the second quantization parameter for each channel and each subband are different. . The description of the configuration common to the first embodiment will be omitted in this embodiment, and the configuration related to the difference in the quantization parameter generation method will be mainly described.

FIG. 12(a) shows units for evaluating subband data and updating quantization parameters related to image quality control when each channel is divided into subbands at lev=3. For (3LL, 3HL, 3LH, 3HH) horizontal and vertical directions (1×M)×1 line (M is an integer), level 2 sub-bands (2HL, 2LH, 2HH) horizontal and vertical directions , and horizontal and vertical subband data of level 1 subbands (1HL, 1LH, 1HH) and horizontal and vertical subband data of (4.times.M).times.1 line respectively. Generate quantization parameters (QpBs).

Compared with the segment related to image quality control shown in FIG. 6A, the segment size is 1/4 in the vertical direction at level 1 in FIG. 12A. Therefore, the relationship between the first quantization parameter (QpBs) and the subband data when M=1 is as shown in FIG. 12(b). Here, since the segment size is 4×1 at level 1, there are four first quantization parameters (QpBs) when converted to 4×4 segments. This enables finer quantization control in the vertical direction than in the first embodiment. It should be noted that it is also possible to make finer in the horizontal direction as well.

The evaluation method of subband data related to image quality control is substantially the same as in the first embodiment, but in this embodiment, the object to be evaluated as a low-frequency component is not 3LL, but intermediately generated in the subband division process. subband data equivalent to 1LL. The flow of processing is the same as that of the flowchart shown in FIG. 7, but the determination target in S701 is not 3LL but 1LL transform coefficients.

In the present embodiment, for example, when converted to 4×4 segments with M=1, the number of quantization parameters for each line to be subjected to image quality control differs from level 1 to level 3. From level 1 to level 3, the first quantization parameter QpBs cannot be used in common. Therefore, in this embodiment, from the four QpBs at level 1, QpLv1 to QpLv3 of each level are generated based on the calculation formula shown in Equation 6 below, and using these, individual channels corresponding to each channel and each subband Generate a second quantization parameter QpSb.
QpLv1[m]=QpBs[m]
QpLv2[n]=(QpBs[n×2]+QpBs[n×2+1])>>1 Equation 6
QpLv3[n]=(QpBs[0]+QpBs[1]+QpBs[2]+QpBs[3])>>2
m : Number of lines (0 to 3) for level 1 subbands in segment units
n : Number of lines (0 to 1) for level 2 subbands in segment units
>>: right bit shift

As shown in Equation 6, the quantization parameters of level 2 and level 3 are calculated from the average value of the corresponding QpBs, but the maximum or minimum value of QpBs may be selected. Also, QpLv3 may be calculated as the average value, maximum value, or minimum value of QpLv2. The second quantization parameter (QpSb) of each channel and each subband used for final quantization can be calculated by replacing QpBs in Equation 5 with QpLv1 to QpLv3. The relationship with the target subband is as shown in FIG. 12(c).

Next, with reference to FIG. 13, the degree of reduction in the amount of recorded quantization parameter data in this embodiment will be described. In the comparative conventional example, assuming that the quantization parameter is recorded in units of one line for each level, if the quantization parameter of the subband of level 3 is 1, level 2 is twice that of level 3, and level 1 is twice as much as level 3. Four times the quantization parameter of level 3 will be recorded. Therefore, the data size of the quantization parameters to be recorded for one segment is (1×4 subbands+2×3 subbands+4×3 subbands)×4 channels, which is 88 in total. On the other hand, the common quantization parameter is 4 because it corresponds to the level 1 subband, and by using the common quantization parameter as the quantization parameter to be recorded, the size of the quantization parameter is reduced to 1/22. can do

As described above, even when finer quantization control is performed in the vertical direction as in the present embodiment, a sufficient data reduction effect can be expected as compared with the conventional example. As in the first embodiment, this embodiment also achieves flexible quantization control while performing predictive coding using a common quantization parameter as two-dimensional data, thereby significantly reducing the data amount of the quantization parameter. RAW data can be recorded compactly with high image quality.

(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

100: imaging device, 101: control unit, 102: imaging unit, 103: RAW data encoding/decoding unit, 104: memory I/F unit, 105: memory, 106: recording processing unit, 107: recording medium

Claims (19)

generating means for performing a discrete wavelet transform with two or more decomposition levels to generate a plurality of subband data from raw data;
common quantization that divides the plurality of subband data into the same number of quantization units, individually sets each quantization unit, and commonly assigns the plurality of subband data included in the same quantization unit quantization for generating parameters, and generating quantization parameters for quantizing each of the plurality of subband data based on the common quantization parameter and coefficients set for each of the plurality of subband data; parameter generation means;
quantization means for quantizing each of the plurality of sub-band data based on the quantization parameter for quantizing each of the plurality of sub-band data;
encoding means for encoding the quantization result obtained by the quantization for each subband;
An image encoding device comprising: 2. The image encoding apparatus according to claim 1, wherein said encoding means encodes said common quantization parameter. 3. The apparatus according to claim 1, further comprising recording processing means for recording said plurality of subband data encoded by said encoding means and said common quantization parameter in a predetermined data structure. Image coding device. The quantization parameter generation means generates a plurality of common quantization parameters for each quantization unit, and quantizes a plurality of subband data included in the quantization unit from the plurality of common quantization parameters. generate the quantization parameters,
2. The image coding apparatus according to claim 1, characterized by: The quantization parameter generation means is
generating a level 1 common quantization parameter for each predetermined line of level 1 of subbands of decomposition level 1 as a plurality of common quantization parameters for each quantization unit;
generating each level common quantization parameter for each predetermined line of each decomposition level based on the plurality of level 1 common quantization parameters for each quantization unit;
generating the quantization parameter for quantizing each of the plurality of subband data based on the common quantization parameter for each level and a coefficient set for each of the plurality of subbands;
5. The image coding apparatus according to claim 4, characterized by: 6. The image coding apparatus according to claim 5, wherein said predetermined line is one line. A recording processing means for recording the plurality of subband data encoded by the encoding means and the level 1 common quantization parameter, which is the common quantization parameter, in a predetermined data structure. 7. The image encoding device according to claim 5 or 6. The encoding means encodes the level 1 common quantization parameter, which is the common quantization parameter,
The recording processing means records the level 1 common quantization parameter encoded by the encoding means in a predetermined data structure together with the plurality of subband data encoded by the encoding means. 8. The image encoding device according to claim 7. The quantization parameter generation means is
a third quantization parameter determined based on a difference between a target code amount and a generated code amount as a result of encoding when the plurality of subband data are quantized and encoded in units of the predetermined line , 9. The image according to any one of claims 5 to 8, wherein the common quantization parameter is generated by increasing or decreasing the amplitude of the frequency component for each quantization unit based on the image quality characteristics. Encoding device. The quantization parameter generating means corrects the third quantization parameter according to the image quality characteristic for each of the predetermined lines of the quantization unit of level 1 subband data, thereby performing the common quantization. 10. The image encoding device according to claim 9 , wherein parameters are generated. The quantization parameter generation means is
reducing the third quantization parameter to generate the common quantization parameter when the image quality characteristic indicates that the amplitude of the low-frequency component is small;
11. The image coding according to claim 9, wherein in the case of image quality characteristics indicating that the amplitude of high frequency components is large, the third quantization parameter is increased to generate the common quantization parameter. Device. 6. The image encoding apparatus according to claim 5, wherein said coefficient is set according to a compression rate set for said RAW data. 6. The image coding apparatus according to claim 5, wherein the coefficient is commonly assigned to a common quantization parameter for a plurality of quantization units. 6. The image coding apparatus according to claim 5, wherein the coefficients set for each of the plurality of subbands are such that the higher the frequency component of the subband, the larger the quantization parameter. 9. The image encoding apparatus according to claim 8, wherein said recording processing means records said coefficients together with said plurality of subband data in said predetermined data structure. the generating means generates the plurality of subband data for each of a plurality of channels obtained by converting the RAW data;
16. The image code according to any one of claims 1 to 15 , wherein the common quantization parameter is commonly assigned to the plurality of subband data of each of the plurality of channels of the quantization unit. conversion device. the generating means generates the plurality of subband data for each of a plurality of channels obtained by converting the RAW data;
the common quantization parameter is assigned in common to each of the plurality of subband data of the plurality of channels of the quantization unit;
the coefficient is set for each of the plurality of subbands and the plurality of channels;
6. The image coding apparatus according to claim 5, characterized by: performing a discrete wavelet transform with two or more levels of decomposition to generate a plurality of subband data from the raw data;
common quantization that divides the plurality of subband data into the same number of quantization units, individually sets each quantization unit, and commonly assigns the plurality of subband data included in the same quantization unit generating parameters;
generating a quantization parameter for quantizing each of the plurality of subband data based on the common quantization parameter and a coefficient set for each of the plurality of subband data;
quantizing each of the plurality of sub-band data based on the quantization parameter for quantizing each of the plurality of sub-band data;
encoding the quantization result obtained by the quantization for each subband;
An image encoding method, comprising: A program for operating a computer as each means of the image encoding device according to any one of claims 1 to 17 .

Images