JP7654241B2 - Image capture device and method for controlling image capture device - Google Patents

Description

The present invention relates to an imaging device that acquires image data using an imaging element and a control method thereof. In particular, the present invention relates to an imaging device that employs an imaging element with a stacked structure in which photoelectric conversion units are stacked, and acquires image data with a high resolution by pixel shifting, and a control method thereof.

Conventionally, imaging devices that acquire image data using an imaging element having a color filter have become widespread. In each pixel that constitutes an imaging element having a color filter, a red (R), green (G), or blue (B) color filter is disposed in front of the photodiode, which is the photoelectric conversion unit. This allows the pixel values of the wavelength components corresponding to the color filter disposed in each pixel to be acquired by the photodiode. The array of pixel values acquired at each pixel is then acquired as image data, and this image data is subjected to demosaic processing to acquire image data for recording.

There is also known an imaging device that acquires image data with increased resolution by pixel shifting. In pixel shifting, multiple pieces of image data are acquired by shifting all or part of the image sensor or the photographing lens on a plane perpendicular to the optical axis. The multiple pieces of acquired image data are then processed to be synthesized, and finally, image data with increased resolution can be acquired.

Patent document 1 discloses an electronic still camera that acquires image data by synthesizing four pieces of image data acquired by pixel shifting. It also discloses an electronic still camera that can acquire higher quality image data by synthesizing eight pieces of image data acquired by pixel shifting.

Patent document 2 discloses an imaging device that acquires image data by synthesizing eight pieces of image data acquired by pixel shifting, and that can reduce the capacity of the buffer memory by performing chromatic aberration correction before synthesizing the image data acquired by pixel shifting.

In addition, an imaging element is known that is configured by stacking multiple photodiodes, so that each pixel value corresponding to the wavelength components of the vertically color-separated subject light can be directly obtained at each pixel. Hereinafter, such an imaging element may be referred to as an imaging element with a stacked structure.

Patent document 3 discloses an imaging element with a stacked structure (Vertical color filter sensor group array), and further discloses an imaging element in which the top layer has full resolution and the layers below it have lower resolution in order to eliminate the complexity of the wiring structure of the imaging element with a stacked structure.

Japanese Patent Application Publication No. 6-225317 JP 2017-46186 A U.S. Pat. No. 7,339,216

However, the prior art disclosed in the above patent documents has the following problems:

In an image sensor with color filters, each pixel can only obtain one pixel value corresponding to the respective color filter. For this reason, to obtain image data with increased resolution by pixel shifting, particularly using an image sensor with color filters that employ a common Bayer-type array pattern, eight pieces of image data must be obtained by pixel shifting. If the number of pieces of image data to be obtained by pixel shifting increases, the time required to obtain the image data and the time required to perform correction processing, synthesis processing, etc. on the image data increases. In addition, the buffer memory capacity required to record multiple pieces of image data increases.

When using an image sensor with color filters to acquire less than eight pieces of image data by pixel shifting and then compositing the high-resolution image data, the color filter arrangement of the high-resolution image data cannot be made the same as that of normal image data, and interpolation processing or replacement processing is required to compensate for this. Furthermore, in this case, it becomes difficult to acquire high-quality high-resolution image data that is ultimately acquired by the compositing processing.

On the other hand, a stacked image sensor can directly obtain pixel values corresponding to the wavelength components of the subject light at each pixel. Therefore, when using a stacked image sensor to obtain image data with increased resolution by pixel shifting, the amount of image data to be obtained by pixel shifting can be reduced.

However, because each layer of image data acquired by a stacked imaging element has image data with roughly the same amount of information for each wavelength component of the subject light, the amount of information in the image data acquired by a stacked imaging element becomes very large. When the amount of information in the image data acquired by pixel shifting using a stacked imaging element becomes large, the burden of transferring and processing the image data increases.

In order to solve the above problems, an imaging device according to a first aspect of the present invention is an imaging device that generates image data having a higher resolution than normal image data through a synthesis process using a plurality of image data pieces acquired by pixel shifting, and is characterized in that it comprises a photographing lens that focuses subject light as a subject image on an imaging surface, an imaging element having a stacked structure in which photoelectric conversion units that convert the subject image into an image signal are stacked, a pixel shift mechanism that shifts the relative position of the subject image and the imaging element of the stacked structure on a plane perpendicular to the optical axis, and a pixel shift synthesis processing unit that generates one piece of image data from the plurality of image data pieces acquired by pixel shifting through a synthesis process, and the pixel shift synthesis processing unit comprises an image data input unit that inputs two pieces of image data acquired by pixel shifting, an image data arrangement unit that shifts and arranges image data of each layer that respectively constitutes the two pieces of image data, a resampling unit that regenerates image data of each layer by a resampling process using image data of the same layer from the image data of each layer that has been shifted and arranged, a pixel mixing unit that generates one piece of image data by pixel mixing process using the regenerated image data of each layer, and an image data output unit that outputs the one piece of image data generated by the pixel mixing process.

The imaging device according to the second invention is characterized in that the pixel shifting mechanism shifts the relative position of the subject image and the stacked imaging element in a plane perpendicular to the optical axis by moving the stacked imaging element in a direction perpendicular to the optical axis.

The imaging device according to the third invention is characterized in that the pixel shifting mechanism shifts the relative position between the subject image and the stacked imaging element on a plane perpendicular to the optical axis by moving all or part of the photographing lens in a direction perpendicular to the optical axis.

The imaging device according to the fourth aspect of the invention is characterized in that the pixel shifting mechanism shifts the relative position between the subject image and the stacked imaging element by 1/2 the pixel pitch of the stacked imaging element in the XY direction on a plane perpendicular to the optical axis.

The imaging device according to the fifth aspect of the invention is characterized in that the pixel shift synthesis processing unit separates the image data of each layer constituting the two pieces of image data acquired by pixel shifting using the stacked imaging element into a luminance layer and a chrominance layer, and by a resampling process, regenerates image data in the luminance layer with a higher resolution than normal image data, and regenerates image data in the chrominance layer with a lower resolution than normal image data.

An imaging device according to a sixth aspect of the present invention is an imaging device that generates image data having a higher resolution than normal image data by synthesis processing using a plurality of image data pieces acquired by pixel shifting, and includes a photographing lens that focuses subject light onto an imaging surface as a subject image, an imaging element having a stacked structure in which photoelectric conversion units that convert the subject image into an image signal are stacked, a pixel shifting mechanism that shifts the relative position between the subject image and the imaging element of the stacked structure on a plane perpendicular to an optical axis, and a pixel shift synthesis processing unit that generates one image data piece by synthesis processing from the plurality of image data pieces acquired by pixel shifting, and the pixel shift synthesis processing unit includes an image data input unit that inputs two image data pieces acquired by pixel shifting, an image data arrangement unit that arranges image data of each layer constituting each of the two image data pieces with a shift, and an image data shifting unit that arranges each of the shifted image data pieces. The image data output unit is provided with a resampling unit which regenerates image data for each layer from image data for the first layer by a resampling process, a pixel mixing unit which generates one image data piece by a pixel mixing process using the regenerated image data for each layer, and an image data output unit which outputs the one image data piece generated by the pixel mixing process, wherein the image data for each layer constituting the two image data pieces obtained by the pixel shifting is divided into a luminance layer and a color layer, and image data with a resolution higher than normal image data is regenerated in the luminance layer by the resampling process, and image data with a resolution lower than normal image data is regenerated in the color layer, and the image data for the layer with the highest contrast among the image data for each layer constituting the two image data pieces obtained by the pixel shifting is set as the luminance layer, and the image data for the other layers is set as the color layers.

An imaging device according to a seventh aspect of the present invention is an imaging device that generates image data having a higher resolution than normal image data by synthesis processing using a plurality of image data acquired by pixel shifting, and includes a photographing lens that focuses subject light onto an imaging surface as a subject image, an imaging element having a stacked structure in which photoelectric conversion units that convert the subject image into an image signal are stacked, a pixel shifting mechanism that shifts the relative position between the subject image and the imaging element of the stacked structure on a plane perpendicular to an optical axis, and a pixel shift synthesis processing unit that generates one image data piece by synthesis processing from the plurality of image data pieces acquired by pixel shifting, and the pixel shift synthesis processing unit includes an image data input unit that inputs two image data pieces acquired by pixel shifting, an image data arrangement unit that arranges the image data of each layer that respectively constitutes the two image data pieces in a shifted manner, and an image data arrangement unit that arranges the image data of each layer in a shifted manner. The image data output unit is provided with a resampling unit which regenerates image data for each layer from the image data for each layer by a resampling process, a pixel mixing unit which generates a single piece of image data by a pixel mixing process using the regenerated image data for each layer, and an image data output unit which outputs the single piece of image data generated by the pixel mixing process, and is characterized in that the image data for each layer constituting each of the two pieces of image data obtained by the pixel shifting is divided into a luminance layer and a color layer, and by the resampling process, image data in the luminance layer is regenerated with a higher resolution than normal image data, and image data in the color layer is regenerated with a lower resolution than normal image data, and the image data of the layer with the highest signal value among the image data for each of the layers constituting each of the two pieces of image data obtained by the pixel shifting is designated as the luminance layer, and the image data of the other layers is designated as the color layers.

In addition, a control method for an imaging device according to the eighth invention is a control method for an imaging device that generates image data with a higher resolution than normal image data by a synthesis process using a plurality of image data acquired by pixel shifting, and is characterized by having an image data acquisition step of acquiring two image data by pixel shifting using an imaging element with a stacked structure, an image data input step of inputting the two acquired image data, an image data arrangement step of shifting and arranging the image data of each layer that constitutes each of the two input image data, a resampling processing step of regenerating image data of each layer using image data of the same layer from the image data of each layer shifted and arranged by the resampling processing, a pixel mixing processing step of generating one image data piece using the image data of each layer regenerated by the pixel mixing processing, and an image data output step of outputting the generated one image data piece.

According to the present invention, in an imaging device and a control method for an imaging device that generates image data with a higher resolution than normal image data through synthesis processing using multiple pieces of image data acquired by pixel shifting, the number of pieces of image data to be acquired by pixel shifting can be reduced by using an imaging element with a stacked structure in which photoelectric conversion units that convert a subject image into an image signal are stacked.

In addition, by dividing the image data of each layer constituting two pieces of image data acquired by pixel shifting using a stacked imaging element into a luminance layer and a color layer, and regenerating image data in the luminance layer with a higher resolution than normal image data by a resampling process, and regenerating image data in the color layer with a lower resolution than normal image data, even in an imaging device and a control method for an imaging device that acquires multiple pieces of image data by pixel shifting using a stacked imaging element, the amount of information in the image data to be composited can be reduced, and an increase in the burden on image data transfer and image processing can be suppressed.

1 is a system block diagram showing an imaging apparatus according to an embodiment of the present invention. FIG. 2 is a schematic diagram showing the structure in the depth direction of an imaging element having a three-layer structure. 11A and 11B are schematic diagrams showing a method of arranging two pieces of image data acquired by pixel shifting. 10 is a schematic diagram showing a state in which image data of each layer is shifted and arranged based on a pixel pitch P. FIG. FIG. 13 is a schematic diagram illustrating a method of resampling a luminance layer. FIG. 13 is a schematic diagram illustrating a method of resampling processing of a color layer. 4 is a flowchart showing steps of a photographing operation of the imaging apparatus of the present embodiment. 4 is a flowchart showing steps of acquiring image data by pixel shifting. 11 is a flowchart showing steps of a pixel shift synthesis process. 4 is a flowchart showing steps of a pixel mixing process.

Below, an embodiment of an imaging device and a control method for an imaging device according to the present invention will be described in detail with reference to the drawings. In the description of this embodiment, the components necessary to explain the technical features of the present invention will be listed and described.

(Figure 1: Components of an imaging device)
The components of the imaging device of this embodiment will be described with reference to Fig. 1. Fig. 1 is a system block diagram showing the imaging device of this embodiment. In Fig. 1, reference numeral 100 denotes an imaging device, 110 denotes a photographing lens, 120 denotes an image sensor, 130 denotes a shift mechanism, 140 denotes a pixel shift synthesis processing unit, 150 denotes a bus, 160 denotes a DRAM, and 170 denotes an external memory.

(Figure 1: Components of an imaging device)
The photographing lens 110 is an optical system that converts subject light into a subject image on an imaging plane. The image sensor 120 converts the subject image into an image signal (analog signal) using photodiodes, which are photoelectric conversion units, arranged two-dimensionally on the imaging plane, and acquires the image. The imaging device 100 of this embodiment employs an image sensor with a stacked structure for the image sensor 120. The image sensor with a stacked structure will be described later.

The shift mechanism 130 moves the image sensor 120 in a direction perpendicular to the optical axis. In the image sensor 100 of this embodiment, the shift mechanism 130 functions as a pixel shifting mechanism that shifts the relative position between the subject image and the image sensor 120 on a plane perpendicular to the optical axis. The pixel shifting mechanism will be described later.

The array of image signals acquired by the image sensor 120 is converted to digital signals by an A/D converter (not shown) and acquired as image data. The image data is transferred to the DRAM 160 via the bus 150 and temporarily stored there.

In the imaging device 100 of this embodiment, the image sensor 120 is moved in a direction perpendicular to the optical axis by the shift mechanism 130, which is a pixel shifting mechanism, and acquires image data at each of the positions to which it is moved. The multiple pieces of image data acquired in this way are temporarily stored in the DRAM 160.

The multiple image data temporarily stored in the DRAM 160 are transferred to the pixel shift compositing processing unit 140 via the bus 150. The pixel shift compositing processing unit 140 uses the multiple image data to generate a single image data with high resolution through compositing processing. The pixel shift compositing processing unit 140 will be described later.

The final high-resolution image data is transferred to the external memory 170 via the bus 150 and stored there.

(Figure 1: Stacked structure image sensor)
Next, a stacked-structure imaging element will be described. A stacked-structure imaging element is an imaging element configured by stacking photodiodes, which are photoelectric conversion units. In particular, in this embodiment, a three-layered imaging element is adopted for the stacked-structure imaging element, in which photodiodes, which are photoelectric conversion units, are stacked in three layers in the depth direction.

(Figure 2: Three-layer image sensor)
Fig. 2 is a schematic diagram showing the cross-sectional structure of an image sensor having a three-layer structure. As shown in Fig. 2A, each pixel constituting the image sensor having a three-layer structure is configured by stacking blue (B), green (G), and red (R) photodiodes in the depth direction. This makes it possible to directly obtain pixel values corresponding to each color in each pixel.

Also, as shown in FIG. 2B, the array of pixel values corresponding to each color acquired at each pixel is acquired as image data for each layer corresponding to each color. In this embodiment, a set of image data for each layer acquired using an image sensor with a three-layer structure is referred to as one piece of image data.

Semiconductors such as silicon used as materials for photodiodes have the property that the absorption rate varies depending on the penetration depth for each wavelength component of the incident light. Due to this property of semiconductors, of the wavelength components of the incident light, the shorter the wavelength component, the shallower the absorption. And the longer the wavelength component, the deeper the absorption.

Stacked image sensors take advantage of this semiconductor property, and by stacking photodiodes made of silicon or the like, they can vertically separate the incident light into each wavelength component. This allows the stacked image sensor to directly obtain pixel values corresponding to each color at each pixel. Therefore, compared to image sensors with color filters, stacked image sensors do not require color filters to be placed above the photodiodes to separate the incident light, and do not require demosaic processing, which can cause false colors when acquiring image data.

In the imaging device 100 of this embodiment, by using an imaging element with a stacked structure for the imaging element 120, the number of image data to be acquired by pixel shifting can be reduced.

In an image sensor with color filters, a color filter is arranged for each pixel to obtain a pixel value corresponding to each color. The color filters arranged for each pixel are regularly arranged two-dimensionally on the plane of the image sensor with color filters, for example in a Bayer array.

When acquiring image data using an image sensor with color filters, first, pixel values corresponding to each color are acquired for each pixel, and the pixel values of each pixel are acquired as an array. Image data is then generated by performing demosaic processing on this array of pixel values for each pixel.

When acquiring multiple pieces of image data by pixel shifting using an image sensor with color filters, it is also necessary to construct an array of pixel values that assumes demosaic processing. However, in order to construct an array of pixel values that assumes demosaic processing from multiple pieces of image data, a large number of pieces of image data must be acquired.

In contrast, a stacked image sensor can directly obtain pixel values corresponding to each color for each pixel. When acquiring multiple images of data by pixel shifting using a stacked image sensor, unlike when using an image sensor with color filters, there is no need to configure an array of pixel values that assumes demosaic processing.

Therefore, when acquiring multiple pieces of image data by pixel shifting using a stacked imaging element, the number of pieces of image data to be acquired can be reduced by pixel shifting; by acquiring at least two pieces of image data and performing a synthesis process on them, it is possible to generate image data with a higher resolution than normal image data.

In particular, the imaging device 100 of this embodiment uses a three-layer imaging element to acquire two pieces of image data by shifting pixels, and then performs a synthesis process on these pieces to generate image data with a higher resolution than normal image data.

(Figure 1: Pixel shift mechanism)
Next, the pixel shifting mechanism will be described. In this embodiment, a shift mechanism 130 that shifts the image sensor 120 on a plane perpendicular to the optical axis is adopted as the pixel shifting mechanism. The shift mechanism 130 can shift the relative position between the subject image and the image sensor 120 on a plane perpendicular to the optical axis. The driving means of the shift mechanism 130 is, for example, a voice coil motor.

The shift mechanism 130 shifts the image sensor 120 on a plane perpendicular to the optical axis, enabling accurate position control to a target position. The target position is determined based on the pixel pitch P of the image sensor 120. The image sensor 120 is position-controlled to multiple target positions by the shift mechanism 130, and image data is acquired at each target position.

The pixel pitch P of the image sensor 120 is set by setting the spacing in the X and Y directions of each pixel that is two-dimensionally arranged at equal intervals in the image sensor 120 as PX and PY, respectively. In this embodiment, the pixel pitch P in the X and Y directions is particularly represented as (PX, PY). PX and PY may be different spacings for the pixel pitch P in the X and Y directions.

The pixel shifting mechanism may also employ a shift mechanism that shifts all or part of the photographing lens on a plane perpendicular to the optical axis. By shifting all or part of the photographing lens on a plane perpendicular to the optical axis, the relative position of the subject image and the image sensor 120 can also be shifted on the plane perpendicular to the optical axis.

(Figure 1: Pixel shift synthesis processing section)
Next, a description will be given of the pixel shift synthesis processing unit 140. In the pixel shift synthesis processing unit 140 in Fig. 1, reference numeral 141 denotes an image data input unit, 142 denotes an image data arrangement unit, 143 denotes a resampling unit, 144 denotes a pixel mixing unit, and 145 denotes an image data output unit.

The image data input unit 141 inputs multiple pieces of image data acquired by pixel shifting and temporarily stored from the DRAM 160 to the pixel shift synthesis processing unit 140. In this embodiment, two pieces of image data are input from the DRAM 160 to the pixel shift synthesis processing unit 140.

The image data arrangement unit 142 arranges the input image data by shifting the image data based on the pixel pitch P. In this embodiment, an image sensor with a three-layer structure is used for the image sensor 120, and image data of each layer is acquired for one image data. Therefore, the image data arrangement unit 142 arranges the image data of each layer for the input image data by shifting the image data of each layer. The method of arranging the input image data of the multiple images by the image data arrangement unit 142 will be described later.

The resampling unit 143 regenerates the image data of each layer, which has been made high-resolution or low-resolution, from the image data of each layer that is arranged in a shifted manner, through a resampling process. The resampling process by the resampling unit 143 will be described later.

The pixel mixing unit 144 uses the image data of each layer regenerated by the resampling process to generate image data for recording through pixel mixing. The pixel mixing process performed by the pixel mixing unit 144 will be described later.

The image data output unit 145 outputs the image data for recording generated by the pixel mixing process. The output image data for recording is recorded in the external memory 170, transferred to an image compression unit (not shown) for image compression, or transferred to an image processing unit (not shown) for other post-processing image processing.

(Figure 3: How to arrange multiple pieces of image data)
Next, a method for arranging a plurality of input image data by the image data arrangement unit 142 will be described.

The imaging device 100 of this embodiment employs an imaging element with a three-layer structure for the imaging element 120, and acquires two pieces of image data by pixel shifting. The image data arrangement unit 142 arranges the image data of each layer that constitutes each of the two pieces of image data by shifting them based on the pixel pitch P.

Figure 3 is a schematic diagram showing a method of arranging two pieces of image data obtained by pixel shifting using the image data arrangement unit 142. In Figure 3, image data A and image data B respectively represent two pieces of image data obtained by pixel shifting.

A single image data piece captured by a three-layer image sensor is composed of image data for the B layer (blue), G layer (green), and R layer (red). That is, in FIG. 3, image data A is composed of image data AB for the B layer, image data AG for the G layer, and image data AR for the R layer. Image data B is composed of image data BB for the B layer, image data BG for the G layer, and image data BR for the R layer.

As shown in FIG. 3, the image data arrangement unit 142 arranges the image data of each layer constituting each of the image data A and image data B by shifting them. At this time, the image data arrangement unit 142 arranges the image data of each layer constituting each of the image data A and image data B by shifting them based on the pixel pitch P so that the pixel positions of each pixel do not overlap each other.

Figure 4 is a schematic diagram showing how the image data of each layer constituting each of image data A and image data B is arranged with a shift based on pixel pitch P. Figure 4 shows the image data of one layer of the image data of each layer of image data A and image data B, and the image data of the other layers is similar to Figure 4. The schematic diagram of Figure 4 is represented diagrammatically by the central positions of each pixel constituting the image data of each layer, which are arranged two-dimensionally on a plane. That is, in Figure 4, the arrangement of the central positions of each pixel indicates the arrangement of each pixel constituting the image data of each layer.

As shown in FIG. 4, the image data of each layer constituting each of image data A and image data B is shifted based on pixel pitch P so that the center positions of each pixel constituting the image data of each layer do not overlap. In this embodiment, the image data is shifted by 1/2 of the pixel pitch P. That is, the image data is shifted by (1/2PX, 1/2PY), which is 1/2 of the pixel pitch P in the XY directions in each layer. The amount of shift in each layer is not limited to this embodiment, and by making it a non-integer multiple of pixel pitch P, the image data of each layer constituting each of image data A and image data B can be shifted without the pixel centers of each pixel overlapping each other.

If the image data of each layer that makes up each of image data A and image data B is shifted so that the center positions of each pixel overlap, the pixel values of each pixel that makes up the image data of each layer will simply be added together. In this case, it will not be possible to regenerate the high-resolution image data by resampling processing.

(Figure 1: Resampling process)
Next, the resampling process will be described. In this embodiment, the resampling process refers to regenerating the image data of each layer by defining new pixel positions in the image data of each layer that is shifted and calculating pixel values at the newly defined pixel positions by interpolation.

In this embodiment, a three-layered imaging element is used for the imaging element 120, and two pieces of image data are acquired by pixel shifting. Then, the image data of each layer constituting each of the acquired two pieces of image data is regenerated by resampling processing from the image data of each layer, which is arranged with the image data of each layer shifted. In this case, since the image data of each shifted layer has twice the amount of information as the image data of each layer constituting one piece of image data, theoretically it is possible to regenerate image data of each layer with a resolution up to twice as high as the image data of each shifted layer.

Here, in this embodiment, we will explain the issues that arise when a stacked image sensor (a three-layer image sensor) is used for the image sensor 120 and multiple (two) pieces of image data are acquired by pixel shifting.

In the structure of a stacked image sensor, the photoelectric conversion units that convert the subject image into an image signal are arranged in a stacked manner. Therefore, a single piece of image data acquired by an image sensor with a stacked structure is composed of image data of each layer corresponding to the photoelectric conversion units arranged in a stack. Furthermore, the resolution of the image data of each layer is equivalent to the resolution of a single piece of image data acquired by an image sensor with a color filter. Therefore, the amount of information of image data acquired by an image sensor with a stacked structure is much greater than the amount of information of image data acquired by an image sensor with a color filter.

In addition, when using a stacked imaging element and acquiring multiple pieces of image data by shifting pixels, the amount of information in the image data that needs to be handled becomes even larger.

When the amount of image data handled increases, the burden of transferring the image data and the burden of processing the image data in subsequent processes also increase. In addition, there is a problem in that the capacity of the buffer memory for temporarily storing the image data must be increased.

In this embodiment, a stacked image sensor is used, and the image data of each layer constituting the multiple image data acquired by pixel shifting is divided into a luminance layer and a color layer. Image data of different resolutions are regenerated for each layer by a resampling process, thereby preventing the amount of information in the image data from increasing.

Human visual perception is more sensitive to luminance information than to color information, so of the image signals that make up image data, luminance signals have a greater impact on human perception of resolution than color signals. In this embodiment, in the resampling process, the image data for each layer is divided into a luminance layer and a color layer, and the amount of information in the image data regenerated in the luminance layer is increased while the amount of information in the image data regenerated in the color layer is reduced, thereby suppressing the amount of information in the image data while maintaining the sense of resolution of the image data.

Next, a method for separating the image data of each layer, which is shifted in the resampling process, into a luminance layer and a color layer will be described. In this embodiment, an image sensor with a three-layer structure is used, and the image data of each layer constituting the two pieces of image data acquired by pixel shifting is shifted, and one layer of the image data of each layer is designated as a luminance layer and the other layers are designated as color layers. In particular, in this embodiment, of the image data of each layer, the B layer is designated as a luminance layer, and the G and R layers are designated as color layers. The method for separating the image data of each layer into a luminance layer and a color layer is not limited to the method of this embodiment.

In addition, instead of dividing the image data of each layer into a luminance layer and a color layer in advance as in the method of this embodiment, the division may be performed by evaluating the image data of each layer obtained. For example, among the image data of each layer, the layer from which the image data with the highest contrast is obtained may be the luminance layer, and the other layers may be the color layers. Also, among the image data of each layer, the layer from which the image data with the highest signal value is obtained may be the luminance layer, and the other layers may be the color layers.

In this embodiment, in order to reduce the amount of information in the image data, the image data of each layer is divided into a luminance layer and a color layer, and in the resampling process, image data with a higher resolution than the normal image data of each layer is regenerated in the luminance layer from which luminance information is obtained, and image data with a lower resolution than the normal image data of each layer is regenerated in the color layer from which color information is obtained.

The resolution of the image data of each layer regenerated by the resampling process in the luminance layer and chrominance layer is determined by the pixel pitch P' of the image data of each layer regenerated by the resampling process. The pixel pitch P' is determined based on the normal pixel pitch P of the image data of each layer (the pixel pitch P of the image sensor 120).

If the pixel pitch P' of the regenerated image data is smaller than the pixel pitch P of the normal image data for each layer, high-resolution image data is regenerated, and if it is larger, low-resolution image data is regenerated.

In this embodiment, the pixel pitch P' of the image data regenerated by the resampling process is 1/√2 times the pixel pitch P in the luminance layer, and 2/√2 times the pixel pitch P in the color layers. As a result, image data with twice the normal resolution of the image data of each layer is regenerated in the luminance layer, and image data with half the normal resolution of the image data of each layer is regenerated in the color layers.

In this embodiment, by making the resolution of the image data regenerated by the resampling process different for the luminance layer and the color layer for the image data of each layer arranged in a shifted manner, it is possible to reduce the amount of information in the entire image data when acquiring image data with high resolution by pixel shifting while using an imaging element with a stacked structure.

In particular, in this embodiment, the amount of information is increased by increasing the resolution of the image data reproduced in the luminance layer, while the amount of information is reduced by decreasing the resolution of the image data reproduced in the color layer, thereby reducing the amount of information. This makes it possible to reduce the amount of information in the overall image data, which is the combined image data for the luminance layer and color layer.

Next, we will explain how to regenerate image data for each layer by resampling the luminance and chrominance layers.

First, in the image data of each layer that is shifted, the pixel position of each pixel of the image data that is regenerated by the resampling process is newly defined. The pixel position of each pixel is newly defined based on the pixel pitch P of the normal image data of each layer.

Next, the surrounding pixels to be used in calculating the pixel value at the pixel position of each newly defined pixel are determined. The surrounding pixels are determined to include pixels that are a predetermined distance away from the pixel position of each newly defined pixel in the image data of each layer that is shifted.

Next, at the pixel position of each newly defined pixel, the pixel value is calculated using the determined surrounding pixels. The pixel value is calculated by an interpolation calculation using the pixel values of the determined surrounding pixels. Conventional methods such as the bicubic method and the bilinear method can be used as the interpolation calculation method.

Then, the image signal array of the pixel values calculated at all pixel positions of each newly defined pixel is regenerated as image data for each layer.

Next, we will explain how to resample the luminance and chrominance layers.

(Figure 5: Luminance layer resampling method)
A method of resampling processing of the luminance layer will be described with reference to Fig. 5. Fig. 5 is a schematic diagram showing that an image sensor with a three-layer structure is adopted, image data of the luminance layer constituting each of two pieces of image data (image data A and image data B) acquired by pixel shifting is shifted and arranged, and image data with a higher resolution than the image data of each normal layer is regenerated from the shifted image data of the luminance layer by resampling processing.

First, as shown in FIG. 5A, in the image data of the luminance layer that is shifted, the pixel positions of each pixel of the image data that is regenerated by the resampling process are newly defined.

The pixel positions newly defined in the luminance layer are determined based on the pixel pitch P of the normal image data of each layer. In order to regenerate image data in the luminance layer with a higher resolution than the normal image data of each layer, the pixel pitch P'L of the image data regenerated by the resampling process in the luminance layer is set so that P'L < P. In this embodiment, in order to regenerate image data in the luminance layer with a resolution twice as high, P'L is set so that 1/√2P.

As shown in FIG. 5A, the pixel position of each pixel is newly defined based on the pixel pitch P'L within the range of the shifted luminance layer image data. The center position of the top left pixel of the shifted image data in the luminance layer is set as the origin (0,0), and starting from this origin (0,0), the pixel positions are newly defined in the X and Y directions based on the pixel pitch P'L (1/√2PX, 1/√2PY). This newly defines the pixel position of each pixel that constitutes the luminance layer image data regenerated by the resampling process.

In the luminance layer, the number of pixels in the X and Y directions of the image data consisting of each pixel with a newly defined pixel position will be √2 times that of the normal image data of each layer. The resolution is also doubled while maintaining the aspect ratio the same as the normal image data of each layer. Note that since the number of pixels in the X and Y directions that make up the image data must actually be integers, the number of pixels in the X and Y directions must be appropriately replaced with the number of pixels that has been rounded down to an integer.

Next, as shown in FIG. 5B, in the luminance layer, neighboring pixels are determined to be used in calculating the pixel value at the pixel position of each newly defined pixel.

The surrounding pixels are determined for each newly defined pixel position to include pixels that are at a specified distance from the pixels that make up the image data of the shifted luminance layer.

In this embodiment, the neighboring pixels for each pixel position newly defined in the luminance layer are determined by the following formula: By arbitrarily setting k in the following formula and defining the range of neighboring pixels in the luminance layer to include one or more neighboring pixels, the predetermined distance from the newly defined pixel position can be adjusted. Note that the method of determining the neighboring pixels is not limited to the following formula.
|x-x'|+|y-y'|<k (k is any positive number)
(x, y): Coordinates of the newly defined pixel position (x', y'): Coordinates of the pixel positions of the surrounding pixels

Then, for each newly defined pixel position in the luminance layer, the pixel value is calculated using the surrounding pixels.

The pixel value at the pixel position of each newly defined pixel is calculated by an interpolation calculation using the pixel values of the surrounding pixels. Conventional interpolation methods such as the bicubic method and the bilinear method can be used as appropriate for the interpolation calculation method. By calculating pixel values at all pixel positions of each newly defined pixel, it is possible to regenerate image data with increased resolution in the luminance layer.

(Figure 6: Method of resampling color layers)
The method of resampling processing of color layers will be described with reference to Fig. 6. Fig. 6 is a schematic diagram showing how an imaging element with a three-layer structure is adopted, image data of color layers constituting two pieces of image data (image data A and image data B) acquired by pixel shifting are shifted and arranged, and image data with a lower resolution than the normal image data of each layer is regenerated from the shifted image data of the color layers by resampling processing.

First, as shown in FIG. 6A, in the image data of the color layers that are shifted, the pixel positions of each pixel of the image data that is regenerated by the resampling process are newly defined.

The pixel positions newly defined in the color layer are determined based on the pixel pitch P of the normal image data of each layer. In order to regenerate image data in the color layer with a lower resolution than the normal image data of each layer, the pixel pitch P'C of the image data regenerated by the resampling process in the color layer is set so that P'C > P. In this embodiment, in order to regenerate image data in the color layer with a resolution reduced by half, P'C is set so that P'C = 2/√2P.

As shown in FIG. 6A, the pixel position of each pixel is newly defined based on pixel pitch P'C within the range of the image data of the color layer that is shifted. The center position of the top left pixel of the image data that is shifted in the color layer is set as the origin (0,0), and a pixel position (1/2√2PX, 1/2√2PY) that is shifted from this origin (0,0) in the XY direction by 1/2√2 of the pixel pitch P (PX, PY) is set as the starting point, and a new pixel position is defined in the XY direction based on pixel pitch P'C (2/√2PX, 2/√2PY). This newly defines the pixel position of each pixel that constitutes the image data of the color layer that is regenerated by the resampling process.

Also, as shown in FIG. 6B, the pixel positions of each pixel newly defined in the color layer are defined one for every four pixel positions of each pixel newly defined in the luminance layer. In other words, the resolution of the image data regenerated in the color layer is 1/4 times that of the image data regenerated in the luminance layer.

In the color layer, the number of pixels in the X and Y directions of the image data consisting of each pixel with a newly defined pixel position will be 1/√2 times that of the normal image data of each layer. The resolution is also reduced to 1/2 while maintaining the aspect ratio the same as the normal image data of each layer. Note that since the number of pixels in the X and Y directions that make up the image data must actually be integers, the number of pixels in the X and Y directions must be appropriately replaced with the number of pixels that has been rounded down to an integer.

Next, as shown in FIG. 6C, the surrounding pixels to be used in calculating the pixel value at the pixel position of each newly defined pixel in the color layer are determined.

The surrounding pixels are determined for each newly defined pixel position to include pixels that are at a specified distance from the pixels that make up the image data of the shifted color layers.

In this embodiment, the pixel pitch P'C of the pixel positions of each pixel newly defined in the color layer is larger than the pixel pitch P'L of the pixel positions of each pixel newly defined in the luminance layer, so the surrounding pixels in the color layer also include pixels that are farther away from the pixel positions of each newly defined pixel.

In this embodiment, the neighboring pixels for each pixel position newly defined in the color layer are determined by the following formula: By arbitrarily setting 2k in the following formula and defining the range of neighboring pixels in the color layer to include one or more neighboring pixels, the predetermined distance from the newly defined pixel position can be adjusted. Note that the method of determining the neighboring pixels is not limited to the following formula.
|x-x'|+|y-y'|<2k (k is any positive number)
(x, y): Coordinates of the newly defined pixel position (x', y'): Coordinates of the pixel positions of the surrounding pixels

Then, for each newly defined pixel position in the color layer, the pixel value is calculated using the surrounding pixels.

The pixel value at the pixel position of each newly defined pixel is calculated by an interpolation calculation using the pixel values of the surrounding pixels. Conventional interpolation methods such as the bicubic method and the bilinear method can be used as appropriate for the interpolation calculation method. By calculating pixel values at all pixel positions of each newly defined pixel, it is possible to regenerate image data with reduced resolution in the color layer.

(Figure 1: Pixel blending process)
Next, the pixel mixing process performed by the pixel mixer 144 will be described.

In the pixel mixing process, the image data of the luminance layer regenerated by the resampling process and the image data of the chrominance layer are combined to generate image data with a higher resolution than the normal image data of each layer.

In this embodiment, pixel mixing refers to image signal processing in which image data for each layer, each of which has a different resolution and is regenerated by a resampling process, is subjected to noise reduction processing and color separation processing for the image data for each layer, and the image data is then combined with a matching resolution to ultimately generate high-resolution image data.

The control method of the imaging device 100 of this embodiment will be described using Figures 7 to 10. Below, the shooting operation in the control method of the imaging device 100 of this embodiment will be described. Furthermore, among the various shooting operations, the explanation will be limited to the shooting operation that generates image data that is ultimately made high-resolution by pixel shifting.

(Figure 7: Each step of the shooting operation)
Each step of the shooting operation will be described with reference to FIG.

First, in S001, the control unit of the imaging device 100 detects whether the release switch has been turned ON. If the release switch has not been turned ON, the process returns to S001, and again detects whether the release switch has been turned ON. If the release switch has been turned ON, the focusing operation is then performed.

Next, in S002, the control unit of the imaging device 100 performs a focusing operation. The control unit of the imaging device 100 drives the focus lens group included in the photographing lens 110 to focus on the subject based on a focus evaluation value detected by a contrast detection method, a phase difference detection method, or the like.

Next, in S003, the control unit of the imaging device 100 acquires two pieces of image data by pixel shifting. The pixel shifting is performed by the control unit of the imaging device 100 shifting the imaging element 120 (an imaging element with a three-layer structure) in a plane perpendicular to the optical axis using the shift mechanism 130. The imaging element 120 acquires one piece of image data at each of two positions, a first position and a second position, shifted by pixel shifting. In this embodiment, the first position is the origin position (0,0), and the second position is a position (1/2PX, 1/2PY) shifted from the origin position (0,0) by 1/2 the pixel pitch P (PX, PY) in the XY direction. Each step of acquiring image data by pixel shifting in S003 will be described later.

Next, in S004, the pixel shift compositing processing unit 140 performs pixel shift compositing processing using two pieces of image data acquired at the first position and the second position by pixel shifting. Here, the two pieces of image data acquired are acquired by an imaging element with a three-layer structure, and each is composed of image data of each layer (B layer, G layer, and R layer). The pixel shift compositing processing is performed by regenerating image data with a higher or lower resolution than the normal image data of each layer from the image data of each layer that has been shifted and arranged by the resampling processing, and finally generating high-resolution image data using the image data of each layer regenerated by the pixel mixing processing. Each step of the pixel shift compositing processing in S004 will be described later.

Next, in S005, the finally generated high-resolution image data is recorded. The finally generated high-resolution image data is recorded in the external memory 170 for storage, or is recorded in temporary memory inside the image sensor for other image processing.

(FIG. 8: Acquiring image data by pixel shifting S003)
Next, each step of acquiring image data by pixel shifting (FIG. 7: S003) will be described with reference to FIG.

First, in S301, the control unit of the imaging device 100 moves the imaging element 120 to the origin position (0,0), which is the first position, by pixel shifting. Next, in S302, the imaging element 120 is exposed at the first position to obtain first image data, and in S303, the obtained first image data is temporarily stored in the DRAM 160.

Next, in S304, the control unit of the imaging device 100 uses pixel shifting to move the imaging element 120 from the second position, which is the origin position (0,0), to a position (1/2PX, 1/2PY) shifted by 1/2 the pixel pitch P (PX, PY) in the XY direction. Next, in S305, the imaging element 120 is exposed at the second position to obtain second image data, and in S306 the obtained second image data is temporarily stored in the DRAM 160. At this point, the DRAM 160 has recorded therein the two pieces of image data obtained by pixel shifting.

In this embodiment, when acquiring image data by pixel shifting, the image sensor 120 is moved to a position shifted by 1/2 the pixel pitch P (PX, PY) relative to the other positions so that the pixels constituting the image data acquired at each position do not overlap each other.

(FIG. 9: Pixel shift synthesis process S004)
Next, each step of the pixel shift synthesis process (FIG. 7: S004) will be described with reference to FIG.

First, in S401, the pixel shift synthesis processing unit 140 inputs two pieces of image data recorded in the DRAM 160 via the image data input unit 141.

Next, in S402, the pixel shift synthesis processing unit 140 separates the image data of each layer constituting the two pieces of image data into a luminance layer and a color layer. In this embodiment, the image data of the B layer is set as the luminance layer in advance, and the image data of the G and R layers are set as the color layers.

Next, in S403, the pixel shift synthesis processing unit 140 shifts and arranges the image data of the luminance layer and the color layer that respectively constitute the two pieces of image data by the image data arrangement unit 142. At this time, the image data arrangement unit 142 arranges the image data of each layer by shifting it relatively by 1/2 the pixel pitch P (PX, PY) in the same way as pixel shifting.

Next, in S404, the pixel shift synthesis processing unit 140 regenerates image data for each layer with a different resolution from the normal image data for each layer from the image data for each layer that is shifted in the luminance layer and the color layer by the resampling unit 143. At this time, image data with a higher resolution than the normal image data for each layer is regenerated in the luminance layer, and image data with a lower resolution than the normal image data is regenerated in the color layer.

Next, in S405, the pixel shift synthesis processing unit 140 uses the regenerated luminance layer or chrominance layer image data to generate final high-resolution image data by the pixel mixing unit 144. Each step of the pixel mixing process in S405 will be described later.

(FIG. 10: pixel mixing process S405)
Next, each step of the pixel mixture process (FIG. 9: S405) will be described with reference to FIG.

In the following explanation, a single image data consisting of image data of a luminance layer and a chrominance layer will be expressed using the resolution ratio of the image data of each layer. Furthermore, the resolution ratio of the image data of each layer will be expressed in the order of R layer, G layer, and B layer.

In this embodiment, the pixel shift synthesis processing unit 140 divides the image data for each layer that constitutes one image data piece into the B layer as the luminance layer, and the G and R layers as the color layers in advance. The resolution ratio of the image data for the luminance layer and the color layer regenerated by the resampling process is 4:1.

Therefore, for example, a piece of image data composed of image data of the luminance layer and chrominance layer regenerated by the resampling process can be represented as 114 image data using the ratio of the resolution of the image data of each layer. Also, for example, a piece of image data obtained by deleting the image data of the chrominance layers R and G from the 114 image data can be represented as 004 image data.

First, in S501, the pixel shift synthesis processing unit 140 generates 111 image data and 004 image data from the 114 image data regenerated by the pixel mixing unit 144 through resampling processing.

Here, the 111 image data is image data that is composed of image data of the R and G layers of the 114 image data, and image data of the B layer that is the sum of pixel values of a set of four adjacent pixels of the 114 image data. Also, the 004 image data is image data that is composed only of the image data of the B layer of the 114 image data, with the image data of the R and G layers deleted from the 114 image data.

Next, in S502, 111 image data and 001 image data are further generated from the 111 image data generated in S501.

Here, the 001 image data is image data that is composed of only the image data of the B layer of the 111 image data, with the image data of the R and G layers deleted.

Next, in S503, noise reduction processing and color separation processing are performed on the 111 image data generated in S502, and further, in S504, upsampling processing is performed on the processed 111 image data to generate 444 image data.

The image data for each layer that makes up the 444 image data generated in S504 corresponds to image data that has been made twice as high resolution as the normal image data for each layer.

On the other hand, in S505, noise reduction processing is performed on the 004 image data generated in S501. Also, in S506, upsampling processing is performed on the 001 image data generated in S502 to generate 004 image data.

Then, in S507, the difference between the 004 image data processed in S505 and the 004 image data generated in S506 is calculated to generate differential image data.

Finally, in S508, the differential image data generated in S507 is applied to the 444 image data generated in S504, and finally, 444 image data is generated that is composed of image data for each layer that is twice as high resolution as the normal image data for each layer.

In the pixel mixing process of this embodiment, 111 image data and 004 image data are generated from 114 image data, which is composed of image data of the luminance layer and chrominance layer regenerated by the resampling process. Then, 444 image data is generated from the 111 image data by performing noise reduction processing and upsampling processing, and 004 image data is generated from the 004 image data by performing noise reduction processing but without upsampling processing. Furthermore, by reflecting the 004 image data, which is the difference image data, in the 444 image data, 444 image data is generated, which is composed of image data of each layer with a resolution twice as high as that of normal image data of each layer. This makes it possible to prevent the problem of coloring of image data that occurs when performing upsampling processing after performing noise reduction processing, etc.

As explained above, according to the present invention, in an imaging device and a control method for an imaging device that acquires image data with increased resolution by pixel shifting, by employing an imaging element with a stacked structure, it is possible to reduce the number of multiple pieces of image data that need to be acquired by pixel shifting. This also reduces the burden of data transfer and image processing of multiple pieces of image data, and reduces the capacity of the buffer memory that temporarily stores multiple pieces of image data.

In addition, by adopting a stacked imaging element and resampling the image data of each layer that constitutes the multiple image data acquired by pixel shifting, the image data of each layer is divided into a luminance layer and a color layer, and the resolution of the image data regenerated in the luminance layer and the color layer is made different, thereby reducing the amount of information in the overall image data.

REFERENCE SIGNS LIST 100 Imaging device 110 Taking lens 120 Imaging element 130 Shift mechanism 140 Pixel shift synthesis processing unit 141 Image data input unit 142 Image data arrangement unit 143 Resampling unit 144 Pixel mixing unit 145 Image data output unit 150 Bus 160 DRAM
170 External Memory

Claims (8)

An imaging device that generates image data having a higher resolution than normal image data by a synthesis process using a plurality of image data acquired by pixel shifting,
The imaging device includes a photographing lens that forms a subject image on an imaging plane from subject light, an imaging element having a laminated structure in which photoelectric conversion units that convert the subject image into an image signal are laminated, a pixel shifting mechanism that shifts the relative position of the subject image and the imaging element having the laminated structure on a plane perpendicular to an optical axis, and a pixel shift synthesis processing unit that generates one image data by synthesis processing from a plurality of image data acquired by pixel shifting,
an image pickup device characterized in that the pixel shifting synthesis processing unit includes an image data input unit that inputs two pieces of image data acquired by pixel shifting; an image data arrangement unit that arranges the image data of each layer that constitutes each of the two pieces of image data with a shift; a resampling unit that regenerates image data of each layer from the image data of each layer arranged with a shift by a resampling process using image data of the same layer; a pixel mixing unit that generates one piece of image data by a pixel mixing process using the regenerated image data of each layer; and an image data output unit that outputs the one piece of image data generated by the pixel mixing process. The imaging device according to claim 1, characterized in that the pixel shifting mechanism shifts the relative position between the subject image and the stacked imaging element in a plane perpendicular to the optical axis by moving the stacked imaging element in a direction perpendicular to the optical axis. The imaging device according to claim 1, characterized in that the pixel shifting mechanism shifts the relative position between the subject image and the stacked imaging element on a plane perpendicular to the optical axis by moving all or part of the imaging lens in a direction perpendicular to the optical axis. The imaging device according to any one of claims 1 to 3, characterized in that the pixel shifting mechanism shifts the relative position between the subject image and the stacked imaging element by 1/2 the pixel pitch of the stacked imaging element in the XY direction on a plane perpendicular to the optical axis. the pixel shift synthesis processing unit divides image data of each layer constituting each of the two pieces of image data acquired by pixel shifting using the stacked image sensor into a luminance layer and a color layer;
An imaging device as described in any one of claims 1 to 4, characterized in that, by a resampling process, image data having a higher resolution than normal image data is regenerated in the luminance layer, and image data having a lower resolution than normal image data is regenerated in the chrominance layer. An imaging device that generates image data having a higher resolution than normal image data by a synthesis process using a plurality of image data acquired by pixel shifting,
The imaging device includes a photographing lens that forms a subject image on an imaging plane from subject light, an imaging element having a laminated structure in which photoelectric conversion units that convert the subject image into an image signal are laminated, a pixel shifting mechanism that shifts the relative position of the subject image and the imaging element having the laminated structure on a plane perpendicular to an optical axis, and a pixel shift synthesis processing unit that generates one image data by synthesis processing from a plurality of image data acquired by pixel shifting,
the pixel shift synthesis processing unit includes an image data input unit that inputs two pieces of image data acquired by pixel shifting, an image data arrangement unit that arranges image data of each layer constituting each of the two pieces of image data with a shift, a resampling unit that regenerates image data of each layer from the image data of each layer arranged with a shift by a resampling process, a pixel mixing unit that generates one piece of image data by a pixel mixing process using the regenerated image data of each layer, and an image data output unit that outputs the one piece of image data generated by the pixel mixing process,
Dividing the image data of each layer constituting each of the two pieces of image data acquired by the pixel shifting into a luminance layer and a chrominance layer;
By the resampling process, image data having a higher resolution than normal image data is regenerated in the luminance layer, and image data having a lower resolution than normal image data is regenerated in the chrominance layer;
An imaging device characterized in that , among the image data of each layer that constitutes each of the two image data acquired by the pixel shifting, the image data of the layer with the highest contrast is designated as a luminance layer, and the image data of the other layers is designated as color layers. An imaging device that generates image data having a higher resolution than normal image data by a synthesis process using a plurality of image data acquired by pixel shifting,
The imaging device includes a photographing lens that forms a subject image on an imaging plane from subject light, an imaging element having a laminated structure in which photoelectric conversion units that convert the subject image into an image signal are laminated, a pixel shifting mechanism that shifts the relative position of the subject image and the imaging element having the laminated structure on a plane perpendicular to an optical axis, and a pixel shift synthesis processing unit that generates one image data by synthesis processing from a plurality of image data acquired by pixel shifting,
the pixel shift synthesis processing unit includes an image data input unit that inputs two pieces of image data acquired by pixel shifting, an image data arrangement unit that arranges image data of each layer constituting each of the two pieces of image data with a shift, a resampling unit that regenerates image data of each layer from the image data of each layer arranged with a shift by a resampling process, a pixel mixing unit that generates one piece of image data by a pixel mixing process using the regenerated image data of each layer, and an image data output unit that outputs the one piece of image data generated by the pixel mixing process,
Dividing the image data of each layer constituting each of the two pieces of image data acquired by the pixel shifting into a luminance layer and a chrominance layer;
By the resampling process, image data having a higher resolution than normal image data is regenerated in the luminance layer, and image data having a lower resolution than normal image data is regenerated in the chrominance layer;
An imaging device characterized in that , among the image data of each layer that constitutes each of the two image data acquired by the pixel shifting, the image data of the layer with the highest signal value is designated as a luminance layer, and the image data of the other layers is designated as color layers. A method for controlling an imaging device that generates image data having a higher resolution than normal image data by a synthesis process using multiple pieces of image data acquired by pixel shifting, comprising the steps of:
A method for controlling an imaging device, comprising: an image data acquisition step for acquiring two pieces of image data by pixel shifting using an imaging element with a stacked structure; an image data input step for inputting the two pieces of acquired image data; an image data arrangement step for arranging the image data of each layer that constitutes each of the two input pieces of image data with a shift; a resampling processing step for regenerating image data of each layer using image data of the same layer from the image data of each layer that has been shifted and arranged by the resampling processing; a pixel mixing processing step for generating one piece of image data using the image data of each layer regenerated by the pixel mixing processing; and an image data output step for outputting the generated piece of image data.

Images