JP7762081B2 - Image encoding device, image decoding device, and their control methods and programs - Google Patents

Description

The present invention relates to image encoding technology.

Recently, imaging devices such as digital cameras and digital camcorders use CCD or CMOS sensors as their imaging elements. These sensors have a color filter array (hereafter referred to as CFA) formed on the sensor surface, and detect one color component per pixel. A typical CFA is a Bayer array, as shown in Figure 3, in which 2 x 2 pixels (one red (R), two green (G0, G1), and one blue (B)) are periodically arranged. Naturally, a Bayer array sensor produces Bayer array image data (hereafter referred to as RAW data).

Because human vision is highly sensitive to luminance components, a typical Bayer array, as shown in Figure 3, allocates twice as many pixels to the green component, which is abundant in luminance components, as compared to the red and blue components. Because RAW data only contains information for one color component per pixel, a process called demosaicing is required to generate red, blue, and green information for each pixel. Generally, the RGB signals obtained by demosaicing, or the YUV signals obtained by converting the RGB signals, are encoded and recorded as image data. However, demosaicing results in three color components per pixel, requiring three times the amount of data compared to RAW data. Therefore, a method has been proposed in which the RAW data before demosaicing is directly encoded and recorded.

For example, Patent Document 1 discloses a method of separating RAW data into four planes (R, G0, B, and G1) and then encoding the data. Patent Document 2 discloses a method of separating RAW data into four planes (R, G0, B, and G1) in the same way as Patent Document 1, and then converting and encoding the data approximately into luminance (Y) and color differences (Co, Cg, Dg).

Meanwhile, sensors capable of simultaneously capturing color and infrared images have been developed for applications such as surveillance and inspection. Unlike Figure 3, these sensors have a pixel arrangement that includes IR (infrared) pixels in addition to R, G, and B, as shown in Figure 4.

Japanese Patent Application Laid-Open No. 2003-125209 Japanese Patent Application Laid-Open No. 2006-121669

The plane conversion methods described in Patent Documents 1 and 2 are based on the Bayer array shown in Figure 3, and do not support pixel arrangements that have IR in addition to RGB, as shown in Figure 4.

This invention aims to provide technology for efficiently encoding RAW data of pixel arrays that contain components outside the visible light spectrum in addition to RGB.

In order to solve this problem, for example, an image coding device of the present invention has the following arrangement:
An image encoding device that encodes an image obtained by an image sensor having R, G, and B primary color filters in the visible light wavelength region and an infrared filter,
a generating means for generating difference data indicating a difference between image data of one of the three primary color filters determined according to an intensity distribution of a transmission wavelength of the infrared filter and image data obtained through the infrared filter;
encoding means for encoding image data for each of the three primary color filters and the differential data ;
The generating means
When the central transmission wavelength of the infrared filter is around 780 nm, the difference data is generated using image data of the R component of the three primary colors;
When the central transmission wavelength of the infrared filter is around 870 nm, the difference data is generated using image data of the B component of the three primary colors;
When the central transmission wavelength of the infrared filter is in the vicinity of 940 nm, the differential data is generated using image data of the G component of the three primary colors .

This invention makes it possible to efficiently encode RAW image data that contains not only the three primary colors but also an infrared component.

FIG. 1 is a block diagram showing an example of the configuration of an image encoding device according to a first embodiment. FIG. 1 is a block diagram showing an example of the configuration of an image decoding apparatus according to a first embodiment. FIG. FIG. 1 shows an RGB-IR array. 3A and 3B are diagrams for explaining examples of infrared rays having three different wavelength intensity distributions according to the first embodiment, and the relationships with the wavelengths of ultraviolet rays and visible light; FIG. 1 is a diagram illustrating a wavelet transform. FIG. 10 is a diagram for explaining a case in which the effective wavelength range of the IR used is intermediate between IR1 and IR2 in the second embodiment. FIG. 1 shows an RGB-NIR-FIR array. FIG. 10 is a diagram for explaining a case in which the effective wavelength region of the IR used is between IR1 and IR2 and closer to IR1 in the second embodiment. FIG. 11 is a diagram for explaining an IRH plane according to the third embodiment. FIG. 10 is a hardware configuration diagram of an information processing device according to a modified example of the first embodiment. 10 is a flowchart showing the procedure of an image encoding process. 10 is a flowchart showing the procedure of an image decoding process.

The following describes the embodiments in detail with reference to the attached drawings. Note that the following embodiments do not limit the scope of the claimed invention. Although the embodiments describe multiple features, not all of these features are necessarily essential to the invention, and multiple features may be combined in any desired manner. Furthermore, in the attached drawings, the same reference numbers are used to designate identical or similar components, and redundant explanations will be omitted.

[First embodiment]
FIG. 1 is a block diagram showing the configuration of an image encoding device according to the first embodiment.

As shown in FIG. 1, the image encoding device has an imaging unit 100, a RAW encoding unit 110, and a control unit 150 that controls these. The RAW encoding unit 110 includes a plane conversion unit 101, a frequency conversion unit 102, a quantization unit 103, and an entropy encoding unit 104. The control unit 150 is made up of a CPU, a ROM that stores programs executed by the CPU, and a RAM that the CPU uses as a work area.

In addition, this embodiment will be described using JPEG2000 as an example of an encoding method, but the type of encoding method is not particularly important.

In the above configuration, the encoding process for input images will be explained with reference to Figure 1.

The imaging unit 100 is a typical imaging optical unit composed of an optical lens, aperture mechanism, shutter mechanism, imager, etc. The imager is a type that captures images by color separation, such as a CCD or MOS type with a color filter for color separation on its surface. This imager separates the color of the formed optical image and converts it into an electrical signal corresponding to the amount of light. In this embodiment, the color separation color filter of the imager will be described as an RGB-IR (Infrared Ray) array filter as shown in Figure 4. In the illustration, R, G, and B represent three primary color filters in the visible light wavelength range, and IR represents an infrared filter.

The plane conversion unit 101 separates RAW image data with an RGB-IR pixel array into an R plane, G plane, B plane, and IR plane, as shown in Figure 4. The R plane is a plane consisting of a single R component, the B plane is a plane consisting of a single B component, the G plane is a plane consisting of a single G component, and the IR plane is a plane consisting of a single IR component. If the number of horizontal pixels in the RAW image is W and the number of vertical pixels is H, the R plane, G plane, B plane, and IR plane are the same size, with W/2 pixels horizontally and H/2 pixels vertically. The plane conversion unit 101 then outputs the R, G, and B planes obtained by the separation. For the IR plane, the plane conversion unit 101 generates and outputs an IRH plane, which indicates differential data between the IR plane and the color planes of the other color components, in accordance with instructions from the control unit 150.

Figure 5 shows an example of infrared light with three different wavelength intensity distributions, as well as its relationship with ultraviolet and visible light wavelengths. In Figure 5, the filter with intensity (high transmittance) in the B wavelength region is a B component filter, the filter with intensity in the G wavelength region is a G component filter, and the filter with intensity in the R wavelength region is an R component filter.

The IR wavelength region includes three wavelength regions: IR1, IR2, and IR3. A filter having intensity in wavelength region IR1 is referred to as a filter for the IR1 component. IR1 has a wavelength intensity distribution in a region adjacent to the R wavelength region. Wavelength region IR2 is the infrared region closest to the R wavelength region after wavelength region IR1, and a filter having intensity in wavelength region IR2 is referred to as a filter for the IR2 component. Wavelength region IR3 has a wavelength intensity distribution in the wavelength region farthest from the R component within the IR wavelength region. A filter having intensity in wavelength region IR3 is referred to as a filter for the IR3 component. In this embodiment, the central transmission wavelength of wavelength region IR1 is approximately 780 nm, the central transmission wavelength of wavelength region IR2 is approximately 870 nm, and the central transmission wavelength of wavelength region IR3 is approximately 940 nm. In this embodiment, the full width at half maximum of the wavelength intensity distribution is approximately 50 nm. Wavelength region IR1 has a high correlation with R, wavelength region IR2 with B, and wavelength region IR3 with G.

Assume that the control unit 150 acquires, via an operation unit (not shown), information indicating that the IR filter of the imaging unit 100 has intensity in the wavelength region of the wavelength intensity distribution IR1. In this case, the control unit 150 sets the plane conversion unit 101 to calculate the difference between the IR plane and the R plane. In response to this, the plane conversion unit 101 calculates the difference plane, i.e., the IRH plane, according to the following equation (1):
IRH = R - IR×ir_gain1…(1)

As explained earlier, IR1 has a high correlation with R. Therefore, the plane conversion unit 101 performs gain adjustment using gain irgain1 to match the IR level to R. The plane conversion unit 101 then calculates the IRH plane, which indicates the difference between the R plane and the adjusted IR plane "IR x ir_gain1." The values that make up the IRH plane (difference pixel values) have positive and negative signs, but many small values close to zero appear, allowing for the expectation of a high compression rate.

Also, assume that information indicating that the IR filter of the imaging unit 100 has intensity in the wavelength region of the wavelength intensity distribution IR2 is set to the control unit 150. In this case, the control unit 150 sets the plane division unit 101 to calculate the difference between the IR plane and the B plane. In response to this, the plane conversion unit 101 calculates the IRH plane according to the following equation (2).
IRH = B - IR×ir_gain2…(2)
As explained above, IR2 has a high correlation with B. Therefore, the plane conversion unit 101 performs gain adjustment using a gain irga2 to match the IR level to B. The plane conversion unit 101 then calculates the IRH plane, which represents the difference between the B plane and the adjusted IR plane "IR × ir_gain2." Although the values that make up the IRH plane (difference pixel values) have positive and negative signs, many small values close to zero appear, and a high compression rate can be expected.

Furthermore, when information indicating that the IR filter of the imaging unit 100 has intensity in the wavelength region of the wavelength intensity distribution IR3 is set in the control unit 150, the control unit 150 sets the plane conversion unit 101 to calculate the difference between the IR plane and the G plane. In response to this, the plane conversion unit 101 calculates the IRH plane according to the following equation (3).
IRH = G - IR×ir_gain3…(3)
As explained above, IR3 has a high correlation with G. Therefore, the plane conversion unit 101 performs gain adjustment using a gain irga3 to match the IR level to G. The plane conversion unit 101 then calculates the IRH plane, which represents the difference between the G plane and the adjusted IR plane "IR × ir_gain3." Although the values that make up the IRH plane (difference pixel values) have positive and negative signs, many small values close to zero appear, and a high compression rate can be expected.

As described above, the plane conversion unit 101 determines the color components for which the difference is to be calculated based on the effective wavelength range of the IR filter of the imaging unit 100, and generates an IRH plane representing the difference. The plane conversion unit 101 then supplies the R, G, B planes and the IRH plane to the frequency conversion unit 102.

The frequency transform unit 102 obtains multiple subbands by performing a frequency transform (wavelet transform in this embodiment) on each plane supplied from the plane transform unit 101. The frequency transform unit 102 then sends the transform coefficients in each subband to the quantization unit 104.

Here, wavelet transform will be explained with reference to Figure 6. Figure 6 shows an example of wavelet transform, a method also adopted in JPEG2000, which performs subband decomposition at only one level (one time). As shown in Figure 6, input plane 400 is split into low-frequency and high-frequency component data using vertical low-pass filter 401 and vertical high-pass filter 402 for the vertical component. The data extracted through vertical low-pass filter 401 and vertical high-pass filter 402 is downsampled 2:1 via downsampling circuits 403 and 404, respectively. As a result, downsampling circuits 403 and 404 output data with half the vertical resolution. The data output from downsampling circuits 403 and 404 is split into low-frequency and high-frequency component data using horizontal low-pass filter 405 and horizontal high-pass filter 406, and horizontal low-pass filter 407 and horizontal high-pass filter 408. It is further downsampled 2:1 via downsampling circuits 409, 410, 411, and 412, respectively. As a result, output image 413 is obtained. Output image 413 is composed of blocks LL, HL, LH, and HH, each obtained through the above filtering process. For simplicity, these blocks will be referred to as LL, HL, LH, and HH as needed below. L indicates the low-frequency region, and H indicates the high-frequency region. For example, HH indicates a subband with high-frequency regions in both the horizontal and vertical directions. In the LL region of output image 413, the resolution of input image 400 is reduced by half both vertically and horizontally. The HH, HL, and LH regions are high-frequency regions. The LL, HL, LH, and HH regions shown in the figure are generally referred to as subbands. Note that a wavelet transform may be performed recursively on subband LL.

The quantization unit 104 quantizes the transform coefficients included in the subbands of each plane data using the transform coefficients sent from the frequency transformation unit 102, using a preset quantization parameter for each subband, and supplies the quantized transform coefficients to the entropy coding unit 105.

The entropy coding unit 105 performs entropy coding, such as EBCOT (Embedded Block Coding with Optimized Truncation), on the wavelet coefficients and quantization parameters quantized by the quantization unit 104 for each subband. The entropy coding unit 105 then generates a header containing information necessary for decoding. The entropy coding unit 105 then outputs the encoded data following this header. Information stored in the header includes the size of the RAW image and the number of bits per component, but in this embodiment, it also includes information indicating which of the IR filters in the image capture unit 100 is an IR1 to IR3 filter (or information indicating which color component plane the IRH plane represents as a difference between the IR plane and the IR plane). The output destination of the entropy coding unit 105 is not particularly limited, and may be a storage medium such as a HDD or memory card.

Next, we will explain how to decode the encoded image data generated using the above procedure.

Figure 2 is a block diagram of an image decoding device according to an embodiment. As shown, the image decoding device includes an entropy decoding unit 200, an inverse quantization unit 201, a frequency inverse transform unit 202, a plane inverse transform unit 203, a header analysis unit 204, and a control unit 250 that controls the entire device.

The header analysis unit 204 analyzes the header (file header) of the encoded stream to be decoded. The header analysis unit 204 then extracts information indicating which of the IR filters, IR1 to IR3, the IR filter of the image sensor (image capture unit 100) used to capture the image to be decoded was, and supplies this information to the plane inverse conversion unit 203.

The entropy decoding unit 200 decodes the coded data following the header, i.e., the coded data of wavelet coefficients and quantization parameters coded using EBCOT (Embedded Block Coding with Optimized Truncation) or the like, and transfers it to the inverse quantization unit 101.

The inverse quantization unit 201 generates frequency transform coefficients by inverse quantizing the data (quantized frequency transform coefficients) supplied from the entropy decoding unit 200 using the quantization parameter. The inverse quantization unit 201 then supplies the frequency transform coefficients to the frequency inverse transform unit 202.

The frequency inverse transform unit 202 performs an inverse frequency transform (inverse wavelet transform in this embodiment) on the frequency transform coefficients supplied from the inverse quantization unit 201 to generate R, G, B, and IRH planes. The frequency inverse transform unit 202 then supplies the R, G, B, and IRH planes to the plane inverse transform unit 203.

The plane inverse conversion unit 203 performs inverse plane conversion on the R, G, B, and IRH planes supplied from the frequency inverse conversion unit 202, reconstructing and outputting RAW image data in an RGB-IR array. Details of the inverse plane conversion are explained below.

The plane inverse conversion unit 203 generates an IR plane from the IRH plane based on information supplied from the header analysis unit 204, i.e., information indicating which of the IR filters IR1 to IR3 the IR filter of the imaging element (imaging unit 100) used to capture the image to be decoded is.
When the IR1 filter is used, the plane inverse transform unit 203 reconstructs the IR plane from the R plane and the IRH plane according to the following equation (4): IR = (R - IRH) / irgain1 (4)
When the IR2 filter is used, the plane inverse transform unit 203 reconstructs the IR plane from the R plane and the IRH plane according to the following equation (5): IR = (B - IRH) / irgain2 (5)
When the IR3 filter is used, the plane inverse transform unit 203 restores the IR plane from the G plane and the IRH plane according to the following equation (6): IR = (G - IRH) / irgain3 (6)

The plane inverse conversion unit 203 restores the IR plane using one of the methods described above. The plane inverse conversion unit 203 then generates RAW image data by arranging the pixels in the R, G, B, and IR planes as shown in Figure 4.

As explained above, according to this embodiment, highly efficient encoding is possible by taking the difference between the IR of RAW image data in an RGB-IR array and highly correlated color components before encoding. Note that in this embodiment, gain adjustment is performed on the IR to take the difference, but gain adjustment can also be performed on the RGB before taking the difference.

In the above description, the image decoding device has the gains irgain1, irgain2, and orgain3 used by the image encoding device. However, when the image encoding device generates encoded data, the gains used may be stored in the header.

[Modification of the first embodiment]
An example in which the image encoding process of the first embodiment is implemented by an application program of an information processing device, such as a personal computer, will be described as a modified example of the first embodiment.

11 is a diagram showing the hardware configuration of an information processing apparatus according to a modified example, which includes a CPU 1101, a ROM 1102, a RAM 1103, a HDD 1104, an operation unit (keyboard, mouse, etc.) 1105, a display unit 1106, and an I/F 1107.

In the above configuration, when the device is powered on, the CPU 1101 executes the boot program in the ROM 1102, loads the OS (operating system) from the HDD 1104 into the RAM 1103, and transfers processing to the OS, making it possible to perform various processes via the operation unit 1105. Then, in accordance with instructions from the user via the operation unit 1105, an image encoding application or an image decoding application is loaded from the HDD 1104 into the RAM 1103 and executed under the OS, thereby allowing the device to function as an image encoding device or an image decoding device.

Here, the processing procedure of the CPU 1101 when the information processing device functions as an image encoding device will be described with reference to the flowchart in Figure 12. Note that the RAW image data file with an RGB-IR arrangement to be encoded is assumed to have been received from an external imaging device via the I/F 1107 and already stored on the HDD 1104. It is also assumed that this file has been selected as the file to be encoded.

In S1201, the CPU 1101 reads the RAW image data file to be encoded from the HDD 1104 and inputs the RAW image data. Then, in S1202, the CPU 1101 generates R, G, B, and IR planes from the input RAW image data and stores them in the RAM 103.

In S1203, the CPU 1101 determines whether the IR filter of the imaging unit of the imaging device used when capturing the RAW image data is one of the IR1 to IR3 filters. If information specifying the IR filter is stored in the file header of the RAW image data, the determination is made from that information. Alternatively, the determination may be made in response to a user selection operation of one of the IR1 to IR3 filters.

If the CPU 1101 determines that the IR1 filter is being used, it branches the process to S1204; if it determines that the IR2 filter is being used, it branches the process to S1205; and if it determines that the IR3 filter is being used, it branches the process to S1205.

If processing branches to S1204, the CPU 1101 generates the IRH plane according to equation (1) shown above. If processing branches to S1205, the CPU 1101 generates the IRH plane according to equation (2) shown above. If processing branches to S1206, the CPU 1101 generates the IRH plane according to equation (3) shown above.

At S1207, the CPU 1101 performs frequency transform (wavelet transform in this embodiment) on each of the R, G, B, and IRH planes to generate transform coefficients. Then, at S1208, the CPU 1101 performs quantization processing on the transform coefficients of each plane. At S1209, the CPU 1101 entropy codes the quantized transform coefficients to generate coded data.

Finally, in S1210, the CPU 1101 creates a file header on the HDD 1104 that includes information necessary for decoding, such as information indicating the type of IR filter, and creates an encoded image data file by storing the encoded data of each plane (it is preferable that the encoded data of IRH is placed last) on the HDD 1104 so that it follows the file header.
do.

Next, the processing procedure of the CPU 1101 when the information processing device functions as an image decoding device will be described with reference to the flowchart in Figure 13. It is assumed that the encoded image data file to be decoded in the HDD 1104 has already been selected by the user. An example will be described in which the RAW image data obtained by decoding is stored as a file on the HDD 1104. When displaying an infrared image on the display unit 1106, the IR plane obtained by decoding can be displayed as a monochrome image. When displaying an RGB color image, an image of the three color components R, G, and B per pixel can be generated from the R, G, and B planes obtained by decoding, and then displayed.

At S1301, the CPU 1101 analyzes the header of the encoded image data file to be decoded and determines which of the IR1 to IR3 filters was used to capture the image. Then, at S1302, the CPU 1101 reads the encoded data from the encoded image data file to be decoded and performs entropy decoding to obtain quantized transform coefficients for each of the R, G, B, and IRH planes. At S1303, the CPU 1101 performs inverse quantization to obtain frequency transform coefficients for each of the R, G, B, and IRH planes. Then, at S1304, the CPU 1101 performs inverse frequency transform (inverse wavelet transform in this embodiment) on the frequency transform coefficients for each of the R, G, B, and IRH planes to restore the R, G, B, and IRH planes.

In S1305, the CPU 1101 determines which of IR1 to IR3 is the IR filter for the image to be decoded, based on the analysis result of S1301. If the CPU 1101 determines that the IR1 filter is being used, it branches the process to S1306. If the CPU 1101 determines that the IR2 filter is being used, it branches the process to S1307. If the CPU 1101 determines that the IR3 filter is being used, it branches the process to S1308.

If the process branches to S1306, the CPU 1101 generates an IR plane according to the previously shown formula (4). If the process branches to S1307, the CPU 1101 generates an IR plane according to the previously shown formula (5). If the process branches to S1308, the CPU 1101 generates an IR plane according to the previously shown formula (3).

In S1309, the CPU 1101 reconstructs RAW image data in an RGB-IR array by arranging the IR plane generated in any of S1306 to S1308 and the pixels of the R, G, and B planes restored in S1304 as shown in Figure 4.

Then, in S1310, the CPU 1101 stores the reconstructed RAW image data as a file on the HDD. At this time, information indicating the type of IR filter is also stored in the file header.

As explained above, the processing equivalent to the first embodiment can be realized by a CPU executing a computer program. It should be noted that the second and third embodiments, which will be described below, can also be realized by a CPU executing a computer program.

Second Embodiment
Next, an image coding device according to a second embodiment will be described with reference to Figures 1, 4, and 7. In this second embodiment, the configuration of the image coding device is the same as that of the first embodiment, but the conversion process by the plane conversion unit 101 is different from that of the first embodiment. Note that other operations are assumed to be the same as those of the first embodiment, and a description thereof will be omitted.

In this second embodiment, we will explain an example in which the effective wavelength range of IR in the RGB-IR array is midway between IR1 and IR2 (the central wavelength of the intensity distribution of transmitted wavelengths is near 820 nm, roughly in the middle of the range from 780 nm to 870 nm), as shown in Figure 7.

The plane conversion unit 101 generates R, G, and IR planes from the RAW image data of the RGB-IR pixel array as shown in FIG. 4, and then calculates the IRH plane according to the following equation (7).
IRH=(R+B)/2-IR×irgain4…(7)
Here, irgain4 is a gain for adjusting the IR level to "(R+B)/2".

The subsequent encoding process is the same as in the first embodiment, so a detailed description will be omitted. However, the header of the generated encoded image data file will contain information indicating that the IR filter has the characteristics shown in Figure 4.

The configuration related to the decoding process may also be the same as that of the first embodiment shown in Fig. 2. That is, when decoding, if the information about IR stored in the header represents Fig. 7, the decoding device regenerates the IR plane from the restored R, B, and IRH planes according to the following equation (8).
IR={(R+B)/2-IRH}/irgain4...(8)
When the effective wavelength range of IR in RAW image data in an RGB-IR arrangement falls between IR1 and IR2, plane conversion is performed taking into consideration the ratio between R, which has a strong correlation with IR1, and B, which has a strong correlation with IR2. In this embodiment, an example has been described in which the effective wavelength range is between IR1 and IR2, but when the effective wavelength range is closer to the IR1 side as in Figure 9, plane conversion is performed by multiplying R by a stronger ratio, for example, as in the following equation (9).
IRH=(R×0.7+B×0.3)−IR×irgain5…(9)
In this case, when decoding, the IR plane can be regenerated according to the following equation (10) using the previously decoded R, B, and IRH planes.
IR={(R×0.7+B×0.3)−IRH}/irgain5…(10)
The weighting coefficients shown in equations (9) and (10) differ depending on how close the effective wavelength region is to IR1 or IR2. Therefore, if the effective wavelength region is closer to IR2, B is multiplied by a stronger weighting coefficient to perform plane conversion.

Furthermore, in this second embodiment, the case where the effective wavelength range of IR is between IR1 and IR2 has been described, but the same concept can be applied to the case where it is between IR2 and IR3, that is, when the central transmission wavelength of the infrared filter is in the range of 870 nm to 940 nm.

[Third embodiment]
An image encoding device according to the third embodiment will be described with reference to FIGS. 1, 8, and 10. In the third embodiment, the configuration of the image encoding device is the same as that of the first embodiment, but the conversion process performed by the plane conversion unit 101 differs from that of the first embodiment. Also, unlike the first and second embodiments, the image sensor uses two types of filters: a near-infrared filter (NIR: Near Infrared Ray) and a far-infrared filter (FIR: Far Infrared Ray). The use of an NIR filter improves visibility in dark places, while the use of an FIR filter also functions as a human presence sensor for detecting intrusions, which is expected to further improve surveillance performance.

Other operations are the same as in the first embodiment, so explanations will be omitted. In addition, this third embodiment shows an example in which IR1 is used as the effective wavelength range of the NIR.

Figure 8 shows the pixel arrangement of the sensor in this third embodiment. As with Figure 4, the pixel arrangement has IR (infrared) pixels in addition to R, G, and B, but differs in that near-infrared (NIR) and far-infrared (FIR) pixels are alternately arranged.

When the plane conversion unit 101 receives information from the control unit 150 indicating that this sensor is being used, it separates the RAW image data in the RGB-IR arrangement into R, G, B planes and FIR+NIR planes, and then calculates IRH, which represents the difference plane, according to the following equations (11) and (12).
When the IR pixel is located at the NIR position, IRHr=R−NIR×irgain1 (11)
When the IR pixel is located at the FIR position, IRHg = G - FIR × irgain6 (12)

The plane conversion unit 101 determines the color components to be subtracted depending on whether NIR or FIR is used as described above, forms an IRH plane with a mixture of the calculated IRHr and IRg as shown in Figure 10, and supplies R, G, B, and IRH to the frequency conversion unit 102. The subsequent processing is the same as in the first embodiment, so a description will be omitted.

During decoding, the IR component is regenerated according to the following equations (13) and (14) using the R, G, and IRH planes previously obtained by decoding.
If the decoding pixel position is NIR,
NIR={R-IRHr}/irgain1...(13)
When the decoding pixel position is FIR,
FIR={G-IRHg}/irgain6...(14)

As described above, in pixel arrays where NIR and FIR are mixed, the color component from which the difference is taken changes depending on whether the IR pixel is NIR or FIRR. Also, rather than forming a separate plane, an IRH plane containing a mixture of different calculation results is formed, and processing from plane conversion onwards is carried out. In this way, by forming an IRH plane containing a mixture of different calculation results, the number of pixels that must be processed for R, G, B, and IRH is the same. Therefore, when processing from plane conversion onwards is performed in parallel, processing can be made uniform.

(Other Examples)
The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention.

100...imaging unit, 101...plane conversion unit, 102...frequency conversion unit, 103...quantization unit, 104...entropy coding unit, 110...RAW coding unit, 150...control unit, 204...header analysis unit, 200...entropy decoding unit, 201...inverse quantization unit, 202...inverse frequency conversion unit, 203...plane inverse conversion unit, 250...control unit

Claims (10)

An image encoding device that encodes an image obtained by an image sensor having R, G, and B primary color filters in the visible light wavelength region and an infrared filter,
a generating means for generating difference data indicating a difference between image data of one of the three primary color filters determined according to an intensity distribution of a transmission wavelength of the infrared filter and image data obtained through the infrared filter;
encoding means for encoding image data for each of the three primary color filters and the differential data ;
and
The generating means
When the central transmission wavelength of the infrared filter is around 780 nm, the difference data is generated using image data of the R component of the three primary colors;
When the central transmission wavelength of the infrared filter is around 870 nm, the difference data is generated using image data of the B component of the three primary colors;
When the central transmission wavelength of the infrared filter is around 940 nm, the differential data is generated using image data of the G component of the three primary colors.
An image encoding device comprising: An image encoding device that encodes an image obtained by an image sensor having R, G, and B primary color filters in the visible light wavelength region and an infrared filter,
a generating means for generating difference data indicating a difference between image data of one of the three primary color filters determined according to an intensity distribution of a transmission wavelength of the infrared filter and image data obtained through the infrared filter;
encoding means for encoding image data for each of the three primary color filters and the differential data;
and
The generating means
When the central transmission wavelength of the infrared filter is within a range of 780 nm to 870 nm, the difference data is generated using image data of a weighted average of the image data of the R component and the image data of the B component of the three primary colors;
When the central transmission wavelength of the infrared filter is within a range of 870 nm to 940 nm, the difference data is generated using image data of a weighted average of the image data of the B component and the G component of the three primary colors.
An image encoding device comprising: an adjusting means for adjusting the image data obtained by the infrared filter with a gain corresponding to the type of filter used to generate the difference data among the three primary color filters;
2. The image encoding device according to claim 1, wherein the generating means generates the difference data using image data obtained through the infrared filter after adjustment by the adjusting means. the infrared filters in the image sensor include two types of filters: a near-infrared filter and a far-infrared filter;
4. The image encoding device according to claim 1, wherein the generating means generates the difference data by calculating a difference between pixels obtained by passing through the near-infrared filter and pixels obtained by passing through the R component filter, and by calculating a difference between pixels obtained by passing through the far-infrared filter and pixels obtained by passing through the G component filter. An image decoding device that decodes coded data of RAW image data obtained by an image sensor having R, G, and B primary color filters in the visible light wavelength region and an infrared filter,
an acquisition means for analyzing a header of the encoded data and acquiring information representing an intensity distribution of a transmission wavelength of the infrared filter;
a decoding means for decoding the coded data to generate image data for each of the three primary color filters and differential data for the infrared filter;
a restoration means for restoring image data obtained from the infrared filter from image data of any one of the three primary color filters specified by the information obtained by the acquisition means and the difference data;
an output means for outputting the image data obtained by the decoding means through each of the three primary color filters and the image data obtained by the restoration means through the infrared filter as RAW image data ;
and
The restoration means
When the acquiring means acquires information that the central transmission wavelength of the infrared filter is near 780 nm, the image data acquired from the infrared filter is restored from the image data of the R component of the three primary colors and the difference data;
When the acquiring means acquires information that the central transmission wavelength of the infrared filter is near 870 nm, the image data acquired from the infrared filter is restored from the image data of the B component of the three primary colors and the difference data;
When the information that the transmission center wavelength of the infrared filter is near 940 nm is acquired by the acquisition means, the image data obtained from the infrared filter is restored from the image data of the G component of the three primary colors and the difference data.
1. An image decoding device comprising: A control method for an image encoding device that encodes an image obtained by an image sensor having R, G, and B primary color filters in the visible light wavelength region and an infrared filter, comprising:
a generating step of generating difference data indicating a difference between image data of any one of the three primary color filters determined according to the intensity distribution of the transmission wavelength of the infrared filter and image data obtained through the infrared filter;
an encoding step of encoding image data for each of the three primary color filters and the difference data ;
and
In the generating step,
When the central transmission wavelength of the infrared filter is around 780 nm, the difference data is generated using image data of the R component of the three primary colors;
When the central transmission wavelength of the infrared filter is around 870 nm, the difference data is generated using image data of the B component of the three primary colors;
When the central transmission wavelength of the infrared filter is around 940 nm, the differential data is generated using image data of the G component of the three primary colors.
2. A control method for an image encoding device comprising: A control method for an image encoding device that encodes an image obtained by an image sensor having R, G, and B primary color filters in the visible light wavelength region and an infrared filter, comprising:
a generating step of generating difference data indicating a difference between image data of any one of the three primary color filters determined according to the intensity distribution of the transmission wavelength of the infrared filter and image data obtained through the infrared filter;
an encoding step of encoding image data for each of the three primary color filters and the difference data;
and
In the generating step,
When the central transmission wavelength of the infrared filter is within a range of 780 nm to 870 nm, the difference data is generated using image data of a weighted average of the image data of the R component and the image data of the B component of the three primary colors;
When the central transmission wavelength of the infrared filter is within a range of 870 nm to 940 nm, the difference data is generated using image data of a weighted average of the image data of the B component and the G component of the three primary colors.
2. A control method for an image encoding device comprising: A control method for an image decoding device that decodes coded data of RAW image data obtained by an imaging sensor having R, G, and B primary color filters in the visible light wavelength region and an infrared filter, comprising:
an acquiring step of analyzing a header of the encoded data and acquiring information representing an intensity distribution of a transmission wavelength of the infrared filter;
a decoding step of decoding the encoded data to generate image data for each of the three primary color filters and differential data for the infrared filter;
a restoration step of restoring image data obtained from the infrared filter from image data of any one of the three primary color filters specified by the information obtained in the acquisition step and the difference data;
an output step of outputting, as RAW image data, the image data obtained through each of the three primary color filters in the decoding step and the image data obtained through the infrared filter in the restoration step ;
and
In the restoration step,
In the acquiring step, when information indicating that the transmission center wavelength of the infrared filter is near 780 nm is acquired, image data obtained from the infrared filter is restored from image data of the R component of the three primary colors and the difference data;
In the acquiring step, when information indicating that the transmission center wavelength of the infrared filter is near 870 nm is acquired, image data obtained from the infrared filter is restored from the image data of the B component of the three primary colors and the difference data;
In the acquisition step, when information indicating that the transmission center wavelength of the infrared filter is near 940 nm is acquired, the image data obtained from the infrared filter is restored from the image data of the G component of the three primary colors and the difference data.
2. A control method for an image decoding device comprising: A computer program that, when read and executed by a computer, causes the computer to execute each step of the control method for an image encoding device according to claim 6 . A computer program that, when read and executed by a computer, causes the computer to execute each step of the control method for an image decoding device described in claim 8.