JP6186573B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method.

  2. Description of the Related Art Conventionally, downsizing of a device for photographing has been desired in order to enable photographing of an image with a small device such as a mobile phone. It is also desired to acquire a wide range of images with high resolution by a small device such as a mobile phone. A technique is known in which miniaturized lenses and sensor elements are arranged in an array, and images acquired by the sensor elements are combined to acquire a high-resolution and wide-range image (see, for example, Patent Document 1). .

International Publication No. 2009/151903

A technique for constructing a high-resolution image from images photographed for the same scene from a plurality of slightly different photographing positions as in the prior art is known as a super-resolution technique. However, in order to increase the resolution of the output image, even if a large amount of information for the output image is prepared, such as by increasing the number of shootings for the scene, noise and point spread function estimation, Due to the influence of alignment accuracy, it is difficult to increase the resolution without limitation.
This restriction can be found in “Fundamental Limits of Reconstruction-Based Superresolution Algorithms under Local Translation”, [online], [searched July 18, 2013], Internet <URL: http://research.microsoft.com/pubs/69073/ As described in 2004-TPAMI-SR.pdf>, the resolution of each sensor element is 1.6 times the theoretical and practical limit.

  Moreover, in the said prior art, since the camera which consists of a lens and a sensor element is used many, the yield in a manufacturing process falls.

  Therefore, an object of the present invention is to provide an image processing apparatus and an image processing method for acquiring a high-resolution image by arranging a plurality of cameras including lenses and image sensors while preventing a decrease in yield.

  An image processing apparatus according to an aspect of the present invention outputs a composite image by combining a camera array including a plurality of camera modules including a lens and an image sensor and a plurality of images output from each camera module of the camera array. The camera array includes a plurality of module blocks each including a plurality of camera modules, and the lenses of the plurality of camera modules included in one module block have optical axes in substantially the same direction, and the camera of each module block The optical axis directions of the module lenses are different from each other.

  An image processing method according to an aspect of the present invention is an image processing apparatus including a camera array including a plurality of camera modules including a lens and an image sensor, and the camera array includes a plurality of module blocks including a plurality of camera modules. An image processing method in an image processing apparatus, wherein each of the camera modules acquires an image, and a plurality of images output from each camera module included in the module block are synthesized by super-resolution processing, A super-resolution image having a resolution higher than the resolution of the image output from the camera module is generated for each module block, and a composite image in which a plurality of super-resolution images corresponding to the plurality of module blocks are arranged two-dimensionally An output step for outputting, and a plurality of steps included in one module block. The camera module lens substantially has an optical axis in the same direction, the optical axis direction of the lens of the camera module of each module blocks are different from each other.

  According to the above aspect, since the plurality of camera modules included in one module block have optical axes in substantially the same direction, so-called super-resolution images are generated based on the plurality of images output from the camera modules. Can be generated. Further, since the optical axis directions of the camera modules of the module blocks are different from each other, a wide range of images can be output by so-called stitching in which super-resolution images generated for each module block are two-dimensionally arranged and synthesized. That is, by combining a plurality of images output from the camera module of the camera array, a wide range and high resolution image can be acquired. In addition, since a plurality of camera modules having optical axes in substantially the same direction are provided, it is possible to measure the distance to the imaging target. In addition, since imaging is performed with a camera array in which a plurality of camera modules are arranged, it is possible to obtain a larger amount of light compared to imaging with a single camera module.

  In the image processing apparatus according to another aspect, the module block includes a red camera module capable of acquiring red information, a green camera module capable of acquiring green information, a blue camera module capable of acquiring blue information, and a Bayer array color filter. At least one Bayer-type camera module.

  According to this aspect, since one module block is composed of a camera module that can acquire color information of red, green, and blue and a camera module that has a Bayer array color filter, it is synthesized from these camera modules. In each pixel of the super-resolution image, sufficient R, G, B color information can be obtained.

  In the image processing device according to another aspect, the output unit combines a plurality of images output from each camera module included in the module block by super-resolution processing, and uses the resolution of the image output by the camera module. A super-resolution image having a high resolution is generated for each module block, and a composite image in which a plurality of super-resolution images corresponding to the plurality of module blocks are two-dimensionally arranged is output.

  According to this aspect, a super-resolution image is generated based on a plurality of images output from a plurality of camera modules included in one module block, and a composite image in which the generated super-resolution images are arranged is generated. Therefore, a wide range and high-resolution image can be output based on images from a plurality of camera modules of the camera array.

  In the image processing apparatus according to another aspect, an acquisition unit that acquires, for each module block, defect information indicating that the camera module in each module block has a defect, and a storage unit that stores the defect information acquired by the acquisition unit; , And the output unit performs a complementary process for complementing the image from the camera module having the defect with the image output from the camera module other than the camera module having the defect according to the defect information stored in the storage unit. Do.

  In the image processing method according to another aspect, according to the defect information acquisition step for acquiring, for each module block, defect information indicating that the camera module in each module block has a defect, and the defect information acquired in the acquisition step, And a complementing step of performing a complementing process for complementing an image from a camera module having a defect with an image output from a camera module other than the camera module having a defect.

  According to this aspect, according to the defect information of the camera module, a super-resolution image can be appropriately generated while performing a complementing process with an image output from a camera module other than the camera module having the defect. Even if there is a defect, the device is not necessarily defective. Therefore, the yield of the device is improved. In particular, when each camera module included in one module block is composed of a Bayer-type camera module and red, green, and blue camera modules, other cameras other than the defective camera module according to the defect information Since the complement is appropriately performed based on the image from the module, the yield can be improved while the deterioration of the image quality of the generated image is minimized.

  In the image processing apparatus according to another aspect, the camera array includes a camera module having an asymmetric lens for changing the optical axis direction of the lens of the camera module between the module blocks, and includes a plurality of cameras included in the camera array. The camera module is configured on the same plane.

  According to this aspect, the camera array is configured to include a camera module having an asymmetric lens in order to change the direction of the optical axis of the lens of the camera module. Thereby, the several camera module of a camera array can be comprised on the same plane. Therefore, the image processing apparatus can be thinned.

  In the image processing apparatus according to another aspect, the module block includes four camera modules that are two-dimensionally arranged in two rows in a first direction and in a second direction substantially orthogonal to the first direction. The camera array is composed of four module blocks that are two-dimensionally arranged in two rows in the first direction and the second direction.

  According to this aspect, in each module block, super-resolution images are generated based on images output from the four camera modules arranged two-dimensionally in two rows in the first and second directions, and each module block A composite image in which four corresponding super-resolution images are two-dimensionally arranged in two columns in the first and second directions is output.

  ADVANTAGE OF THE INVENTION According to this invention, the image processing apparatus and image processing method which acquire a high-resolution image by arranging two or more cameras which consist of a lens and an image sensor can be provided, preventing the fall of a yield.

It is a block diagram which shows the functional calibration of an image processing apparatus. It is a figure which shows typically the structure of a module block and a camera array. It is a figure which shows typically a mode that a part of camera array 2 was seen from the direction substantially orthogonal to the optical axis direction of the lens of a camera module. It is a figure which shows typically the example of the super-resolution process implemented by a super-resolution process part. It is a figure which shows typically the example of the stitching process implemented by a stitching process part. It is a figure which shows typically the example of the production | generation of the super-resolution image corresponding to one module block in which the camera module which has a defect is not included. It is a figure which shows typically the example of the production | generation of the super-resolution image corresponding to one module block in which a red camera module has a defect. Table showing the probability of occurrence of the defect pattern, the image quality when the defect pattern is generated, and the expected value of the image quality when the defect pattern is generated for each defect pattern of the lens of the camera module that can occur in one module block It is. 3 is a flowchart illustrating an operation of the image processing apparatus according to the embodiment.

  The knowledge of the present invention can be easily understood by considering the following detailed description with reference to the accompanying drawings shown as examples. Subsequently, embodiments of the image processing apparatus and the image processing method of the present invention will be described with reference to the accompanying drawings. Where possible, the same parts are denoted by the same reference numerals.

  FIG. 1 is a block diagram showing an image processing apparatus according to this embodiment. As shown in FIG. 1, the image processing apparatus 1 includes a camera array 2 and an output unit 3.

  The camera array 2 includes four module blocks 21 including a plurality of camera modules. In the example illustrated in FIG. 1, the camera array 2 includes four module blocks 21, but the number of module blocks included in the camera array 2 is not limited to four. One module block 21 includes four camera modules 22 as described later with reference to FIG. The camera module 22 includes a lens and an image sensor, and acquires an image to be captured.

  The image processing apparatus 1 includes a white balance processing unit WB. The white balance processing unit WB appropriately performs white balance processing on the image information output from the camera module of each module block 21 of the camera array 2. In addition, the white balance processing unit WB can appropriately perform gamma correction processing on the image information output from each module block 21. The white balance processing unit WB appropriately performs processing on the image output from the camera module of each module block 21 and performs super-resolution processing corresponding to each module block 21 in the output unit 3 on the processed image information. Sent to the section SR.

  The output unit 3 is a part that combines a plurality of images output from each camera module of the camera array and outputs a combined image, and includes a super-resolution processing unit SR and a stitching processing unit ST.

  The super-resolution processor SR performs super-resolution processing based on the image information from the plurality of camera modules included in the module block 21 and has a resolution higher than the resolution of the image output by the camera module. This is a part for generating a resolution image. In the present embodiment, since one module block includes four camera modules, the super-resolution processing unit SR performs super-resolution processing based on the four pieces of image information captured by each camera module, and performs one super-resolution processing. A resolution image is generated. In addition, the super-resolution processing unit SR appropriately performs demosaic processing.

  The stitching processing unit ST is a part that performs so-called stitching processing in which the super-resolution images generated by the super-resolution processing unit SR for each module block 21 are two-dimensionally arranged and combined. In addition, since it is necessary to align each image to be synthesized in the stitching process, the stitching processing unit ST performs the stitching process while appropriately performing geometric correction for performing alignment for each super-resolution image. Do.

  The output unit 3 outputs the composite image obtained by the super-resolution process and the stitching process to the memory M. The output unit 3 may output the composite image to display means for displaying an image. The image processing apparatus 1 further includes a defect information acquisition unit 4 and a defect information storage unit 5.

FIG. 2 is a diagram schematically showing the configuration of the module block and the camera array. FIG. 2A is a diagram showing the configuration of the camera module 22 included in one module block 21. As shown in FIG. As shown in FIG. 2A, the module blocks 21 are two-dimensionally arranged in two rows in the first direction (the vertical direction on the paper surface) and in the second direction (the horizontal direction on the paper surface) substantially orthogonal to the first direction. The four camera modules 22 are arranged. Specifically, the module block 21 includes a red camera module 22 _R , a green camera module 22 _G , a blue camera module 22 _B, and a Bayer type camera module 22 _BY .

The red camera module _22R is configured to acquire only a red component by a color filter. The green camera module _22G is configured to acquire only a green component by a color filter. Blue camera module 22 _B is configured to acquire only the blue component by the color filter.

The Bayer camera module 22 _BY has a Bayer array color filter. That is, of the pixel information acquired by the Bayer-type camera module 22 _BY , 50% is green pixel information, 25% is red pixel information, and 25% is blue pixel information.

The lenses of the four camera modules 22 _R , 22 _G , 22 _B , 22 _BY included in one module block 21 have optical axes in substantially the same direction. This makes it possible to generate a so-called super-resolution image based on a plurality of images output from those camera modules. In addition, since the plurality of camera modules 22 _R , 22 _G , 22 _B , and 22 _BY have optical axes in substantially the same direction, these camera modules have substantially the same field of view. Based on this, the distance to the imaging target can be measured.

FIG. 2B is a diagram showing the configuration of the module block 21 included in the camera array 2. As shown in FIG. 2B, the camera array 2 is two-dimensionally arranged in two rows in the first direction (the vertical direction on the paper surface) and in the second direction (the horizontal direction on the paper surface) substantially orthogonal to the first direction. It has four module blocks 21 ₁ , 21 ₂ , 21 ₃ , and 21 ₄ arranged. The lens of the camera module 22 included in each of the module blocks 21 ₁ , 21 ₂ , 21 ₃ , and 21 ₄ has optical axis directions X ₁ , X ₂ , X ₃ , and X ₄ that are different from each other. That is, since the module blocks _{211 to 4} have different fields of view, images according to the respective optical axis directions and fields of view are acquired. Therefore, the stitching processing unit ST of the output unit 3 performs a stitching process on the four super-resolution images generated based on the images from the module blocks ₂₁₁ to ₄ , so that a wide field of view can be obtained. Can be generated.

  FIG. 3 is a diagram schematically illustrating a state in which a part of the camera array 2 is viewed from a direction substantially orthogonal to the optical axis direction of the lens of the camera module 22. The camera module 22 of the camera array 2 includes a lens r and an image sensor s. In the camera array 2 of the present embodiment, each module block 21 has a different optical axis direction X in order to acquire images with different fields of view for each module block 21. In the present embodiment, the camera array 2 includes a camera module 22 having an asymmetric lens in order to make the optical axis direction X of the lens of each module block 21 different. As described above, by making the optical axis directions X different from each other by the asymmetric type lens, the plurality of camera modules 22 included in the camera array 2 can be configured on the same plane P. Therefore, the image processing apparatus 1 configured to include such a camera array 2 can be thinned.

FIG. 4 is a diagram schematically illustrating an example of super-resolution processing performed by the super-resolution processing unit SR. In FIG. 4, large black dots represent pixel information output from the red camera module 22 _R of the module block 21, and white circles surrounding characters R, G, B represent pixel information output from the Bayer type camera module 22 _BY. A small black dot represents a pixel generated by the super-resolution processor SR. In FIG. 4, among the pixels output from the camera module 22, the pixels output from the green and blue camera modules are omitted for convenience of illustration. The super-resolution processor SR includes pixel information P _R1 , P _R2 , P _R3 , _{PR 4} from the red camera module 22 _R and pixel information P _BY1 , P _BY2 , P _BY3 , P _BY4 from the Bayer-type camera module 22 _BY. , P _BY5 , P _BY6 are appropriately weighted and combined to calculate pixel information P ₀ of the super-resolution image.

FIG. 5 is a diagram schematically illustrating an example of stitching processing performed by the stitching processing unit ST. In FIG. 5, images G _{1 to 4} indicate super-resolution images generated based on images output from the module blocks 21 _{1 to 4} , respectively. The stitching processing unit ST appropriately aligns the images G1 to _G4 and then arranges them in a two-dimensional manner and outputs a composite image _GOUT .

  Next, complementary processing when a defect occurs in the manufacturing process of the camera module included in the module block 21 will be described. As described above, the image processing apparatus 1 includes the defect information acquisition unit 4 and the defect information storage unit 5 for the complementing process.

  The defect information acquisition unit 4 is a part that acquires defect information indicating that the camera module 22 in each module block 21 is defective for each module block. Specifically, the presence or absence of defects of the lens or image sensor of the camera module 22 in each module block 21 is inspected by a method capable of detecting the presence or absence of defects of the camera module 22, and the defect information acquisition unit 4 If there is a defect, defect information such as which camera module 22 has the defect is acquired for each module block 21. The defect information storage unit 5 stores the defect information acquired by the defect information acquisition unit 4.

  The output unit 3 is output from a camera module other than the camera module having a defect in accordance with the defect information stored in the defect information storage unit 5 when generating a super-resolution image corresponding to one module block 21. Complement processing for complementing an image from a camera module having a defect is performed using the image.

FIG. 6 is a diagram schematically illustrating an example of generation of a super-resolution image corresponding to one module block 21 in which a camera module 22 having a defect is not included. As shown in FIG. 6, when the first module block 21 does not include a camera module 22 having the defect, green image information G from the red image data G _R, green camera module 22 _G from the red camera module 22 _{R G,} blue image information _{G B} from the blue camera module 22 _B, and Bayer type camera module ₂₂ Bayer image information _{G bY} from _bY are obtained. In such a case, the super-resolution processing section SR of the output section 3, the one red pixel information P _R01 super resolution image, the pixel information P _R11 of a corresponding pixel in the red image information G _R, and Bayer image information G _{BY Are} generated by combining the pixel information P _R21 and P _R31 in the vicinity of the corresponding pixel with appropriate weighting.

On the other hand, FIG. 7 is a diagram schematically illustrating an example of generation of a super-resolution image corresponding to one module block 21 in which the red camera module _22R has a defect. As shown in FIG. 7, if it has a defect in the red camera module 22 _R of one module block 21, the image information from the red camera module 22 _R is not acquired, green image information from the green camera module 22 _G G _G, blue image information _{G B} from the blue camera module 22 _B, and Bayer type camera module ₂₂ Bayer image information _{G bY} from _bY are obtained. In such a case, the super-resolution processing section SR of the output section 3, the one red pixel information P _R02 super resolution image, the pixel information P _G32 and the pixel information of the corresponding pixels near the corresponding pixel in the Bayer image information G _{BY P R12,} P _R22, the green image data pixel information of the pixel information _{P G62} and the corresponding pixels near the corresponding pixel in the _{G G P G72, P G82} and blue image information pixel information of the corresponding pixel neighborhood in _{G B P B42, P B52} Generate by combining with appropriate weighting. Thus, the output unit 3, even if there is no image information from the red camera module 22 _R, can be generated super-resolution image corresponding to one of the module block 21.

  FIG. 8 shows the probability of occurrence of the defect pattern, the image quality when the defect pattern is generated, and the image quality when the defect pattern is generated, for each defect pattern of the lens of the camera module 22 that can occur in one module block 21. It is a table which shows the expected value. In this table, it is assumed that the yield of one lens of the camera module 22 is 98%. According to the table of FIG. 8, for example, the probability that no defect occurs in the camera module 22 in one module block 21 is 92.24%, and the image quality in this case is 100%. The expected value of the image quality at this time is calculated as 92.24% as the product of the occurrence probability and the image quality (see the first column from the left).

Further, for example, the probability that defects are generated only red camera module 22 _R in one of the module block 21 is 1.88 percent, from quality is 75% (left image obtained by complementing process in this case (See column 4). The expected value of the image quality at this time is calculated as 1.41%.

  The total yield of one module block 21 in consideration of all the defect patterns shown in FIG. 8 is calculated as 99.53%, and the expected value of the total image quality is calculated as 97.94%. .

As described above, even when the camera module 22 having a defect is included in one module block 21, the camera module 22 other than the camera module 22 having the defect is output according to the defect information of the camera module 22. It is possible to appropriately generate a super-resolution image while performing a complementing process using the image. As a result, even if the camera module 22 is defective, the device itself including the module block 21 is not necessarily a defective product, and it is possible to obtain a certain yield or better image quality. In particular, each camera module 22 included in one module block is composed of a Bayer-type camera module 22 _{BY and} red, green, and blue camera modules 22 _R , 22 _G , 22 _B. Complementation is appropriately performed based on images from other camera modules other than the defective camera module, and the yield can be improved while minimizing the degradation of the image quality of the generated image.

  Next, the image processing method of the present embodiment will be described with reference to FIG. FIG. 9 is a flowchart illustrating the operation of the image processing apparatus 1 according to the embodiment. First, each camera module 22 included in the camera array 2 acquires image information of an imaging target (S1).

  Next, the defect information acquisition unit 4 acquires defect information indicating that the camera module 22 in each module block 21 has a defect for each module block 21 and stores the defect information in the defect information storage unit 5. The defect information is information indicating a result of inspection by a method capable of detecting the presence / absence of a defect in the camera module 22, and the presence / absence of a defect in the lens or image sensor of the camera module 22 in each module block 21 In some cases, information including which camera module 22 has a defect is included. Then, the output unit 3 refers to the defect information storage unit 5 to determine the presence / absence of a camera module 22 having a defect for each module block 21 (S2). If it is determined that there is a camera module 22 having a defect, the processing procedure proceeds to step S3. On the other hand, if it is not determined that there is a camera module 22 having a defect, the processing procedure proceeds to step S4.

  In step S3, the output unit 3 uses the image output from the camera module other than the camera module having a defect for each module block 21 in accordance with the defect information stored in the defect information storage unit 5. Complement processing for complementing the image from the module is performed (S3).

  Subsequently, in step S4, the super-resolution processing unit SR of the output unit 3 performs super-resolution processing based on each image information from the plurality of camera modules included in the module block 21 to obtain a super-resolution image. Generate (S4). Further, the stitching processing unit ST of the output unit 3 performs a stitching process for two-dimensionally arranging and synthesizing the super-resolution images generated for each module block 21 to generate one synthesized image ( S5). Then, the output unit 3 outputs the generated composite image (S6).

  In the image processing apparatus 1 of the present embodiment, when a plurality of images from the camera modules 22 are combined, a process for correcting the distortion of the captured image that occurs according to the distortion of the lens is necessary. The camera module 22 has a symmetric lens or an asymmetric lens according to the arrangement position in the camera array 2. When a symmetric lens is used, correction can be made by an already established method, and a distortion coefficient is estimated according to a lens distortion formula, and an image obtained from the camera module 22 is estimated. The distortion of the input image is corrected using a distortion formula to which the applied coefficient is applied.

  On the other hand, when an asymmetric lens is used, distortion occurs irregularly, so that the distortion formula used for correcting the image of the symmetric lens cannot be used. Therefore, in this case, a calibration image obtained by photographing a calibration pattern of, for example, a checkered pattern is acquired using the camera module 22 that is symmetric with respect to the calibration. The correction data of each pixel that is located in the same manner as each intersection point is calculated and stored. Note that correction data of pixels corresponding to portions other than each intersection of the checkered pattern is calculated by interpolation by interpolation of correction data of pixels corresponding to intersections around the pixel. And at the time of image acquisition by normal imaging, distortion correction is performed by applying correction data stored in advance to the captured image. This distortion correction can be performed, for example, in the super-resolution processing unit SR in FIG. 1, and is performed during the super-resolution processing (step S4 in the flowchart of FIG. 9) or immediately before that.

  According to the image processing apparatus 1 and the image processing method of the present embodiment described above, the plurality of camera modules 22 included in one module block 21 have optical axes in substantially the same direction. A so-called super-resolution image can be generated based on a plurality of output images. Further, since the optical axis directions of the module blocks 21 are different from each other, a wide range of images can be output by so-called stitching in which super-resolution images generated for each module block 21 are two-dimensionally arranged and synthesized. That is, a wide range and high-resolution image can be acquired by combining a plurality of images output from the camera module 22 of the camera array 2. Further, since a plurality of camera modules 22 having optical axes in substantially the same direction are provided, it is possible to measure the distance to the imaging target. In addition, since imaging is performed by the camera array 2 in which a plurality of camera modules 22 are arranged, it is possible to obtain a larger amount of light as compared with imaging by one camera module 22.

  The present invention has been described in detail based on the embodiments. However, the present invention is not limited to the above embodiment. The present invention can be variously modified without departing from the gist thereof.

1 ... image processing apparatus, 2 ... camera array, 3 ... output unit, 4 ... defect information acquisition unit, 5 ... defect information storage _unit, 21 _1, 21 2, ₂₁ 3, 21 4 _... module block, 22 ... camera Module, 22 _B ... Blue camera module, 22 _BY ... Bayer type camera module, 22 _G ... Green camera module, 22 _R ... Red camera module, M ... Memory, r ... Lens, s ... Image sensor, SR ... Super-resolution processing Part, ST ... Stitching processing part, X ₁ , X ₂ , X ₃ , X ₄ ... Optical axis direction.

Claims (8)

A camera array including a plurality of camera modules including a lens and an image sensor;
An output unit that combines a plurality of images output from each camera module of the camera array and outputs a combined image;
With
The camera array includes a plurality of module blocks including a plurality of camera modules,
The lenses of the plurality of camera modules included in one module block have optical axes in substantially the same direction,
The optical axis directions of the lens of the camera module of each module block are different from each other.
Image processing device.   The module block includes at least one of a red camera module capable of acquiring red information, a green camera module capable of acquiring green information, a blue camera module capable of acquiring blue information, and a Bayer camera module having a Bayer array color filter. The image processing apparatus according to claim 1, wherein the image processing apparatuses are included one by one. The output unit is
A plurality of images output from each camera module included in the module block are synthesized by super-resolution processing, and a super-resolution image having a resolution higher than the resolution of the image output by the camera module is obtained for each module block. Generate
Outputting a composite image in which a plurality of super-resolution images corresponding to a plurality of module blocks are two-dimensionally arranged;
The image processing apparatus according to claim 1. An acquisition unit that acquires, for each module block, defect information indicating that the camera module in each module block has a defect,
A storage unit that stores defect information acquired by the acquisition unit;
The output unit performs a complementing process for complementing an image from a camera module having a defect with an image output from a camera module other than the camera module having a defect, according to the defect information stored in the storage unit.
The image processing apparatus according to claim 1. The camera array includes a camera module having an asymmetric lens for changing the optical axis direction of the lens of the camera module between module blocks.
The image processing apparatus according to claim 1, wherein the plurality of camera modules included in the camera array are configured on the same plane. The module block is composed of four camera modules arranged two-dimensionally in two rows in a first direction and in a second direction substantially orthogonal to the first direction,
The camera array is composed of four module blocks arranged two-dimensionally in two columns in the first direction and the second direction.
The image processing apparatus according to claim 1. An image processing apparatus including a camera array including a plurality of camera modules including a lens and an image sensor, wherein the camera array includes a plurality of module blocks including a plurality of camera modules. ,
An acquisition step in which each of the camera modules acquires an image;
A plurality of images output from each camera module included in the module block are synthesized by super-resolution processing, and a super-resolution image having a resolution higher than the resolution of the image output by the camera module is obtained for each module block. Generation step;
Outputting a composite image in which a plurality of super-resolution images corresponding to a plurality of module blocks are two-dimensionally arranged, and
The lenses of the plurality of camera modules included in one module block have optical axes in substantially the same direction,
The optical axis directions of the lens of the camera module of each module block are different from each other.
Image processing method. Defect information acquisition step for acquiring defect information indicating that there is a defect in the camera module in each module block for each module block;
In accordance with the defect information acquired in the acquisition step, a complementing step for performing a complementing process for complementing an image from a camera module having a defect with an image output from a camera module other than the camera module having a defect;
The image processing method according to claim 7, further comprising: