JP7809487B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing device and an image processing method, and in particular to image synthesis technology.

A technology is known that suppresses image blur by combining multiple n images captured in succession with an exposure time that is 1/n of the required exposure time (Patent Document 1).

Japanese Patent Application Laid-Open No. 2007-243775

When combining multiple images, it is necessary to align the images. However, simply aligning the images results in pixels outside the effective pixel area being combined out of position. This creates a problem when it is desired to use the values of pixels outside the effective pixel area. Patent Document 1 does not recognize this issue.

The present invention was made in consideration of these problems with the prior art, and in one aspect provides an image processing device and image processing method capable of image synthesis that can appropriately utilize pixel values outside the effective pixel area.

The above-mentioned object can be achieved by an image processing apparatus comprising: an acquisition means for acquiring image data of a plurality of frames; an alignment means for aligning the image data of the plurality of frames ; a synthesis means for synthesizing the aligned image data of the plurality of frames; and a calculation means for calculating a value indicating the variation in the image data synthesized by the synthesis means, wherein the image data includes first data which is data of pixels in an effective pixel area of an imaging element and is composed of signals corresponding to an arrangement of predetermined color components, and second data which is data of pixels outside the effective pixel area , the alignment means is capable of aligning the first data independently of the second data, the calculation means calculates a value indicating the variation in the second data of the image data synthesized by the synthesis means, and the strength of the noise reduction processing to be applied in the development processing of the image data synthesized by the synthesis means is determined based on the value indicating the variation .

The present invention provides an image processing device and image processing method capable of image synthesis that can appropriately utilize pixel values outside the effective pixel area.

FIG. 1 is a block diagram showing an example of the functional configuration of an imaging apparatus as an image processing apparatus according to an embodiment; Schematic diagram of the pixel area of the image sensor Schematic diagram of RAW data synthesis processing in an embodiment. Flowchart for operations in composite photography mode in an embodiment

The present invention will now be described in detail based on illustrative embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Furthermore, although the embodiments describe multiple features, not all of them are necessarily essential to the invention, and multiple features may be combined in any desired manner. Furthermore, in the accompanying drawings, the same reference numbers are used to designate identical or similar components, and redundant explanations will be omitted.

The following description will focus on an embodiment of the present invention in an imaging device such as a digital camera. However, imaging functionality is not essential for the present invention, and the present invention can be implemented in any electronic device capable of handling image data. Such electronic devices include computer devices (personal computers, tablet computers, media players, PDAs, etc.), smartphones, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

Figure 1 is a block diagram showing an example of the functional configuration of an imaging device 10 as an example of an image processing device according to an embodiment of the present invention.

The control unit 113 has one or more processors capable of executing programs, RAM, and ROM. The control unit 113 can load programs stored in the ROM into the RAM and execute them using the processor. By executing the programs, the control unit 113 controls the operation of the components of the imaging device 10, including the functional blocks shown in FIG. 1, and/or external devices connected in a communicative manner. The functions of the imaging device 10 can be implemented by the control unit 113 executing the programs. Note that functions that can be realized by the control unit 113 executing the programs may also be implemented using hardware (e.g., ASIC, FPGA, etc.). The ROM is, for example, electrically rewritable, and stores programs executable by the processor of the control unit 113, as well as setting values for the imaging device 10, GUI data, etc. The control unit 113 can be in the form of, for example, a system-on-chip (SoC) or a system-on-package (SiP).

The photographic lens 100 has multiple lenses and an aperture, and generates an optical image of a subject. The image sensor 102 is, for example, a CMOS image sensor, and has multiple photoelectric conversion units (pixels) arranged two-dimensionally.

Figure 2 is a diagram showing a schematic representation of the pixel arrangement of the image sensor 102. The image sensor 102 has an effective pixel area 200 where pixels are arranged, and an optical black (OB) area 201 outside the effective pixel area. The effective pixel area 202 is an area used (exposed) for capturing an image of a subject. On the other hand, the OB area 201 is an area not used (not exposed) for capturing an image of a subject. Pixels in the OB area 201 may be provided with, for example, a light-shielding film to prevent light from entering.

The pixels arranged in the OB region 201 have the same structure as the pixels arranged in the effective pixel region 200, except for the light-shielding film. Furthermore, if the pixels in the effective pixel region 200 are provided with color filters, the pixels in the OB region 201 are also provided with color filters, just like the pixels in the effective pixel region 200.

The effective pixel area 200 and the OB area 201 are areas of the image sensor 102, but for convenience in this specification, the same expressions are used to refer to images obtained by the image sensor 102. For example, the image or effective pixel area refers to the image area obtained by the pixels in the effective pixel area 200 of the image sensor 102 within one frame of image obtained by shooting.

A color filter is configured with an array of unit filters of different colors, with one unit filter provided for each pixel. For example, a primary color Bayer array color filter has a configuration in which four types of unit filters - R (red), G1 (green), G2 (green), and B (blue) - are arranged in a repeating unit of 2x2 pixels. Below, a pixel equipped with unit filter G1 will be referred to as a G1 pixel. The same applies to pixels equipped with other types of unit filters.

Pixels arranged in the OB region 201 are used, for example, to detect noise components of the photoelectric conversion elements (e.g., photodiodes) that the pixels have. For example, by subtracting the signal values of pixels in the OB region 201 from the signal values of pixels in the effective pixel region 200, it is possible to remove offsets due to noise components and correct the black level of the image.

The pixels of the image sensor 102 generate pixel signals whose values correspond to the amount of charge generated during the charge accumulation period. When capturing images by opening and closing the mechanical shutter 101, the charge accumulation period corresponds to the exposure period. When capturing images by keeping the mechanical shutter 101 open, the charge accumulation period corresponds to the period from when the accumulated charge is reset to when the exposure period has elapsed. Generally, the former corresponds to still image capture, and the latter to video capture, but the latter can also apply to still image capture.

When one charge accumulation period ends, a group of pixel signals (analog image signals) for one frame is read out from the image sensor 102. The analog image signals for one frame include pixel signals from the effective pixel area 200 and the OB area 201. The analog image signals are converted into digital image signals (digital pixel signal groups) by the A/D converter 103. Note that if the image sensor 102 is capable of outputting digital pixel signal groups, the A/D converter 103 may not be necessary.

At this stage, each pixel signal has only one color component corresponding to the color of the unit filter of the corresponding pixel. In this specification, a digital image signal composed of pixel signals corresponding to an arrangement of predetermined color components (pixel signals having only one color component corresponding to the color of the unit filter) is referred to as RAW data.

RAW data output from the A/D conversion unit 103 or the image sensor 102 is temporarily stored in the memory 114. The memory 114 is used to temporarily store RAW data, image data processed by the signal processing unit 111, and the like.

The OB integration unit 104 calculates the average pixel value for each pixel type for the pixel values in the OB region 201 of the RAW data. Here, because the image sensor 102 has color filters in a primary color Bayer array, the OB integration unit 104 calculates the average pixel values of the R pixels, G1 pixels, G2 pixels, and B pixels in the OB region 201. This average pixel value is used as the black level.

The OB clamp unit 105 applies OB clamp processing to pixel values in the effective pixel region 200, correcting the black level based on pixel values in the OB region 201. Specifically, the OB clamp processing may be processing that subtracts the average pixel value calculated by the OB integration unit 104. The OB clamp processing can suppress black floating and color shift in the image obtained from the pixel signals in the effective pixel region 200. Note that the OB clamp processing uses a black level according to the pixel type. For example, the pixel value of an R pixel in the effective pixel region 200 is subtracted by the black level obtained from the R pixel in the OB region. The OB clamp unit 105 applies OB clamp processing to each of the original images to be combined, and to the combined image generated by the combination unit 108.

The shading correction unit 106 applies shading correction to pixel values in the effective pixel area 200. Shading correction corrects for brightness reductions that occur depending on pixel position due to factors such as the optical characteristics of the imaging lens 100 and the microlenses that pixels have. Therefore, shading correction applies a gain that depends on pixel position.

The white balance (WB) processing unit 107 applies white balance adjustment processing to the image after shading correction. The white balance adjustment processing is a process of applying a gain according to the pixel type (R, G1, G2, B) to the shading-corrected pixel values.

The positional deviation detection unit 109 detects the amount of positional deviation for each of the multiple frame images to be combined. The amount of positional deviation may be an absolute amount of deviation, with the amount of deviation for the first frame captured being set to 0, or it may be a relative amount of deviation with respect to the immediately preceding frame. When detecting the absolute amount of deviation, the image for the first frame serves as the reference image.

The misalignment detection unit 109 can detect the amount of misalignment between frames as a movement vector of the image of the effective pixel area 200. The misalignment detection unit 109 can detect the amount of misalignment using any known technology, such as a method using template matching between frames or a method using the output of a gyro sensor or the like provided in the imaging device 10. The misalignment detection unit 109 outputs the detected amount of misalignment to the misalignment correction unit 110 or stores it in memory 114.

The misalignment correction unit 110 aligns the images of the frames to be combined based on the amount of misalignment detected by the misalignment detection unit 109. The misalignment correction unit 110 can perform alignment, for example, by changing the coordinate values of each pixel of the frames to be combined in accordance with the amount of misalignment.

The misalignment correction unit 110 has a mode in which it aligns only the image in the effective pixel region 200, and a mode in which it aligns the image of an area (e.g., the entire frame) that includes the effective pixel region 200 and the OB region 201. In this way, the misalignment correction unit 110 can align the data in the effective pixel region 200 (first data) independently of the data in the OB region 201 (second data). The mode of the misalignment correction unit 110 can be set by, for example, the control unit 113.

Even if a mode for aligning the entire frame is set, the control unit 113 may change the mode to one for aligning only the images in the effective pixel region 200 if the conditions for reducing the synthesis accuracy of the OB region 201 due to alignment are met. For example, if the number of frames in which the OB region 201 overlaps is equal to or less than a predetermined percentage of the total number of frames N, the control unit 113 can change the mode to one for aligning only the images in the effective pixel region 200. The number of frames in which the OB region 201 overlaps can be determined based on the amount of misalignment of each frame detected by the misalignment detection unit 109.

The compositing unit 108 combines multiple images using one of several selectable compositing modes to generate a composite image. It is not necessary to apply the compositing process to the entire frame; it is sufficient to perform the compositing process on the effective pixel area and OB area in the reference image.

The composition unit 108 stores the generated composite image in the memory 114. In this embodiment, the imaging device 10 is capable of selecting from the addition mode, average addition mode, and comparatively bright mode as the composition mode. Note that these composition modes are merely examples, and other composition modes may be selectable, or there may be only one composition mode.

The image synthesis method for each synthesis mode will be explained below. Here, it is assumed that N frames (N is an integer greater than or equal to 2) of images are synthesized. The pixels that make up the image of each frame have coordinates (x, y) in an xy Cartesian coordinate system, and the luminance value of the pixel at coordinates (x, y) is I_i(x, y) (i = 1 to N), and the luminance value of the pixel at coordinates (x, y) in the synthesized image is I(x, y). The synthesis unit 108 calculates the luminance value I(x, y) of each pixel in the synthesized image as follows, depending on the synthesis mode.

・Addition mode I(x,y)=I_1(x,y)+I_2(x,y)+...+I_N(x,y)
In the addition mode, the composition unit 108 generates a composite image by adding the luminance values of pixels at the same coordinates in each frame. The addition mode is used, for example, when generating an image with proper exposure by combining images of N frames captured with an exposure amount that is 1/N of the proper exposure amount.

Additive average mode I(x, y) = (I_1(x, y) + I_2(x, y) + ... + I_N(x, y))/N
In the averaging mode, the synthesis unit 108 generates a synthesized image in which the luminance value of each pixel is the average value of N frames by dividing the calculated luminance value by the number of frames N, as in the addition mode. The averaging mode is used, for example, to reduce noise in an image captured at high sensitivity.

・Comparative bright mode I (x, y) = max (I_1 (x, y), I_2 (x, y), ..., I_N (x, y))
Here, max() is a function that extracts the maximum value of the elements in (). A composite image is obtained that is composed of the maximum brightness value among the N pixels that have the same coordinates in each frame. The comparatively bright mode is effective when compositing images of fireworks or starry skies, for example.

Note that for coordinates where there are no pixel values to combine due to changes in coordinates caused by alignment, the combining unit 108 can use the pixel value of that coordinate in another specified frame (for example, the first frame) instead in the addition mode and averaging mode. Furthermore, in the averaging mode or comparatively bright mode, the combining unit 108 can perform combining using only existing pixel values (in the averaging mode, the divisor is changed to the number of frames used for addition (<N)). Note that these are just some possible examples, and other methods of combining may also be used.

The composition unit 108 also prevents an image outside the effective pixel area 200 from being composited into the effective pixel area of the reference image. The composition unit 108 can handle pixel values outside the effective pixel area 200 whose coordinates have been changed to a position that overlaps the effective pixel area of the reference image due to alignment, in the same way as if the pixel value does not exist.

The signal processing unit 111 applies development processing to (uncombined) RAW data and data of the composite image generated by the composition unit 108 (combined RAW data). Note that development processing is applied to the image area corresponding to the effective pixel area 200. Development processing is a general term for multiple image processing methods, such as color interpolation processing and gradation correction processing (gamma processing). Color interpolation processing is a process that interpolates the values of color components that cannot be obtained during shooting, and is also called demosaicing processing. By applying color interpolation processing, each pixel comes to have the multiple color components (e.g., RGB or YCbCr) required for a color image, and the data is no longer RAW data.

The signal processing unit 111 can apply detection processes such as detection of characteristic regions (for example, face regions or human body regions) and their movement, and person recognition processes, as well as synthesis processes, scaling processes, encoding processes, and decoding processes to the developed image data. The signal processing unit 111 can also apply various image processing processes, such as data processing processes such as header information generation processes, generation of signals and evaluation values used for autofocus detection (AF), and evaluation value calculation processes such as calculation of evaluation values used for autoexposure control (AE). Note that these are examples of image processing that the signal processing unit 111 can apply, and do not limit the image processing that the signal processing unit 111 can apply.

The recording unit 112 records RAW data, composite RAW data, developed image data, and audio data associated with these data on a recording medium such as a memory card, depending on the shooting mode and recording settings.

The operation unit 115 is a general term for input devices (switches, keys, buttons, dials, touch panels, etc.) that allow the user to give various instructions to the imaging apparatus 10 .
The display unit 116 is, for example, a touch display, and is used to display live view images, playback images, GUI, setting values and information of the imaging device 10, and the like.

The operation of the imaging device 10 in composite shooting mode will be explained using the flowchart in Figure 4. Composite shooting mode is a shooting mode in which a composite image is recorded by combining multiple frame images captured over time. In composite shooting mode, one of the multiple synthesis modes described above is set. Note that while still image shooting is explained here, it can also be executed as a process for generating one frame of video shooting, in which each frame is used as a composite image.

In S101, the control unit 113 accepts the setting of the number of frames N to be composited from the user via the operation unit 115. Note that the number of frames N may be set in advance, similar to the setting of the composite mode.

In S102, the control unit 113 detects that a shooting instruction has been input from the user via the operation unit 115. The shooting instruction may be, for example, a full press of the shutter button included in the operation unit 115. When the input of the shooting instruction is detected, the control unit 113 executes S103.

In S103, the control unit 113 executes the shooting process for one frame. The exposure conditions (shutter speed, aperture value, sensitivity) during shooting can be set to the appropriate exposure amount obtained by executing AE processing using a live view image before S102 (for example, when a half-press of the shutter button is detected). If the composition mode is the addition mode, the control unit 113 sets the exposure conditions to be 1/N of the appropriate exposure amount, based on the number of frames N set in S101.

The control unit 113 stores the RAW data obtained during shooting in memory 114. When shooting the first frame, the control unit 113 skips S104 and executes S105, and when shooting the second frame and beyond, executes S104.

In S104, the positional deviation detection unit 109 detects the amount of deviation of the image (RAW data) obtained in the most recent capture. Here, the amount of deviation is detected for all images from the second frame onwards relative to the image of the first frame (reference image). The amount of deviation is detected using template matching, which uses a part of the reference image as a template.

Here, to improve the accuracy of misalignment detection, the misalignment detection unit 109 generates multiple templates from the reference image and detects the misalignment amount for each template. The misalignment detection unit 109 then determines the final misalignment amount based on the multiple detected misalignment amounts. For example, the misalignment detection unit 109 can determine the most frequently occurring misalignment amount or the average of the misalignment amounts as the final misalignment amount, but other methods may also be used.

In S105, the OB integration unit 104 calculates the black level for each pixel type using pixel values in the OB region 201 of the image (RAW data) obtained in the most recent capture.

In S106, the OB clamp unit 105 uses the black level calculated in S105 to apply OB clamp processing to pixels in the effective pixel area of the image (RAW data) obtained in the most recent capture.

In S107, the misalignment correction unit 110 uses the amount of misalignment detected in S104 to align the image (RAW data) obtained in the most recent capture with the reference image. This alignment process will be explained using Figure 3.

Figure 3 shows a schematic configuration for aligning and synthesizing a second frame image (RAW data) 302 with a first frame image (RAW data) 301, which serves as a reference image. Image 301 has an effective pixel area 301a and an OB area 301b. Similarly, image 302 has an effective pixel area 302a and an OB area 302b.

Here, it is assumed that the misalignment correction unit 110 is set to a mode that aligns only the image in the effective pixel area. Therefore, in the image 302' after alignment, only the effective pixel area 302a' has moved, and the OB area 302b has not moved. As a result, an area 302c where no pixels exist has appeared in the image 302' after alignment. If the misalignment correction unit 110 is set to a mode that aligns the entire frame, the coordinate information has changed internally, but there is no change in the appearance of the image 302 before and after alignment.

In S108, the composition unit 108 composites the image aligned in S107 (image 302' in Figure 3) with the reference image or the composite image (composite RAW data) up to the previous frame. The composition unit 108 performs the composition process as described above depending on the set mode.

Figure 3 shows an example of compositing processing in additive average mode or comparatively bright mode, in which area 302c, which is created by alignment and has no pixels, is excluded from the compositing target. As a result, the effective pixel area 310a of composite image 310 includes area 310c, where the image of the second frame is not composited. For OB area 310 of composite image 310, OB area 301b of the first frame and OB area 302b of the second frame are composited using a method appropriate to the mode.

Note that if the pixel signal is affected by negative noise, there is a possibility that the pixel value will become less than 0 when the black level is reduced in the OB clamping process of S106. To prevent this, a predetermined positive offset may be added to the composite image before saving it in memory 114. If an offset is added to the composite image, the same amount of offset as that added to the composite image is added to the images to be combined before applying the OB clamping process of S105. After applying the OB clamping process, the offset is removed from both images before applying the combining process in S108.

In addition, while the flowchart in Figure 4 shows compositing performed sequentially from the second frame onwards on the reference image, it is also possible to generate composite images from the second to Nth frames and then composite them onto the reference image last. In this case, compositing from the second to Nth frames may only be performed on the overlapping areas of all frames. For parts of the effective pixel area of the reference image where composite images from the second to Nth frames have not been generated, it is assumed that there are no pixel values to be composited, and the method corresponding to the composition mode described above can be applied.

In S109, the control unit 113 determines whether or not shooting the set number of frames N has been completed. If it is determined that shooting has been completed, the control unit 113 proceeds to S110; if not, the control unit 113 proceeds to S103 and shoots the next frame. Note that the control unit 113 continues to shoot images as long as shooting instructions are continuously input, for example. If the input of shooting instructions ceases before shooting the set number of frames N has been completed, the control unit 113 may, for example, discard the processing results performed up to that point and return to a shooting standby state.

In S110, the OB integration unit 104 calculates the black level based on the pixel values of the OB region of the composite image stored in memory 114. The calculation of the black level is the same as S105, except that the pixel values of the OB region are pixel values after the composite process.

In S111, the OB integration unit 104 calculates a value (e.g., a variance value) that indicates the variation in pixel values in the OB region of the composite image. The variance value may be calculated for each pixel type (color), or one variance value may be calculated for all pixels. The OB integration unit 104 may execute the processes of S110 and S111 in parallel.

In S112, the OB clamp unit 105 applies OB clamp processing to pixel values within the effective pixel area of the composite image, using the black level based on the composite image calculated in S110. Note that even though OB clamping was applied to the pre-combination image in S106, the reason for applying OB clamp processing after compositing is to reduce floating black caused by compositing.

For example, when multiple frames captured at high sensitivity are combined in the relatively bright mode, each frame contains a lot of noise. Combining in the relatively bright mode selects the highest luminance value at each coordinate. Therefore, there is a high possibility that a luminance value higher than the original luminance value due to noise will be selected for each coordinate. Therefore, by applying OB clamping to the combined image using a black level based on the pixels in the OB area 201 (composite OB area) combined in the relatively bright mode in the same way as the effective pixel area 200, it is possible to suppress floating blacks caused by combining. Here, we have explained the use of the relatively bright mode for combination as a typical example of floating blacks caused by combination, but since floating blacks caused by combination also occur in other combination modes, OB clamping is applied to the combined image using a black level based on the pixels in the composite OB area.

In S113, the WB processing unit 107 applies white balance adjustment processing to the image (RAW data) of the effective pixel area of the composite image that has been subjected to OB clamp processing.

In S114, the signal processing unit 111 applies development processing to the RAW data to which the white balance adjustment processing has been applied. The signal processing unit 111 changes the parameters of the noise reduction processing to be applied during the development processing according to the variance value calculated in S111. Specifically, if the variance value is greater than a predetermined threshold, the signal processing unit 111 changes the parameters of the noise reduction processing so that stronger noise reduction processing is applied than when the variance value is not greater. This is because when the variance value is greater than the threshold, the amount of noise is greater than when the variance value is not greater. Note that if there are three or more strengths of noise reduction processing, the strength of the noise reduction processing may be adjusted using two or more thresholds. The signal processing unit 111 generates an image data file storing image data from the image data after development processing.

In S115, the recording unit 112 records the image data file generated in S114 on a recording medium such as a memory card or an external device. In addition to, or instead of, the developed image data, composite RAW data of the effective pixel area before the OB clamping process in S112 may be recorded. Information necessary for the external device to perform the processes in S112 to S114 (the black level calculated in S110 and the amount of dispersion calculated in S111) may also be recorded in addition to the composite RAW data. The original RAW data for N frames may also be recorded so that the external device can perform the processes in S104 to S114.

When recording by the recording unit 112 is completed, the control unit 113 ends the operation shown in Figure 4 and returns to, for example, a shooting standby state.

As described above, according to this embodiment, when combining multiple frames of RAW data, the alignment of the RAW data can be applied separately to the data in the effective pixel area and the data in the OB area. By making it possible to align the data in the effective pixel area independently, it is possible to prevent the data in the OB area from being combined with the data in the effective pixel area, thereby suppressing degradation in the quality of the combined image. Furthermore, because it is possible to eliminate misalignment in the OB area between the frames to be combined, it is possible to calculate a highly accurate black level from the OB area after combination, effectively suppressing floating blacks in the image that occur due to combination.

Furthermore, if the image sensor 102 has a stacked structure, in addition to the A/D conversion unit 103 and memory 114, the misalignment detection unit 109, misalignment correction unit 110, and composition unit 108 can also be implemented within the image sensor 102. This makes it possible for the image sensor 102 to perform all processes from image capture, alignment, and composition.

(Other embodiments)
In the above embodiment, a composite image is generated when an image is captured by an image capture device. However, as previously mentioned, a capture function is not essential to the present invention. Therefore, the present invention can also be implemented with an image processing device that, instead of capturing an image in S103, acquires one frame of RAW data captured and recorded over time and executes the processes in S104 and beyond. In this case, the determination process in S109 determines whether the acquisition of the set number of frames has been completed.

The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

The present invention is not limited to the contents of the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention.

10...imaging device, 102...imaging element, 104...OB integration unit, 105...OB clamp unit, 108...combining unit, 109...positional deviation detection unit, 110...positional deviation correction unit, 113...control unit

Claims (10)

acquisition means for acquiring image data of a plurality of frames;
a positioning means for positioning the image data of the plurality of frames;
a synthesizing means for synthesizing the aligned image data of the plurality of frames ;
a calculation unit that calculates a value indicating the variation of the image data combined by the combining unit ,
the image data includes first data, which is data of pixels in an effective pixel region of the image sensor and is configured with signals corresponding to an arrangement of predetermined color components, and second data, which is data of pixels outside the effective pixel region;
the alignment means is capable of aligning the first data independently of the second data;
the calculation means calculates a value indicating a variation in the second data of the image data combined by the combination means;
the strength of noise reduction processing to be applied in development processing of the image data combined by the combining means is determined based on the value indicating the variation;
1. An image processing device comprising: The image processing device described in claim 1, characterized in that the alignment means aligns the first data and does not align the second data. The image processing device according to claim 1 or 2, wherein the combining means combines the first data of the multiple frames and the second data of the multiple frames, and the method for combining the first data and the method for combining the second data are the same. The image processing device described in any one of claims 1 to 3, characterized in that the second data is data of pixels that are not exposed. An image processing device as described in claim 4, characterized in that the second data is used to adjust the black level of the first data. The image processing device described in claim 5, characterized in that the black level adjustment is performed on each of the multiple frames of image data and on the image data combined by the combining means. 7. The image processing apparatus according to claim 1, wherein the combining means prevents the aligned second data from being combined with the first data. An imaging element;
An image processing device according to any one of claims 1 to 7 ;
and
An imaging device, characterized in that the plurality of frames of image data are captured over time using the imaging element. An image processing method executed by an image processing device,
acquiring a plurality of frames of image data;
aligning the image data of the plurality of frames;
combining the aligned image data of the plurality of frames ;
calculating a value indicative of the variability of the combined image data;
the image data includes first data, which is data of pixels in an effective pixel region of the image sensor and is configured with signals corresponding to an arrangement of predetermined color components, and second data, which is data of pixels outside the effective pixel region;
the first data is independently alignable with the second data;
The calculating step calculates a value indicating a variation of the second data of the combined image data,
the strength of noise reduction processing to be applied in the development processing of the combined image data is determined based on the value indicating the variation;
An image processing method comprising: A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 1 to 7 .