JP6704803B2 - Imaging device and control method thereof - Google Patents

Description

The present invention relates to a multi-view imaging device and a control method thereof.

2. Description of the Related Art In recent years, image pickup apparatuses having a plurality of optical systems have been proposed in accordance with higher performance and more multifunctional image pickup apparatuses, and many image processing methods for synthesizing signals from a plurality of cameras (image pickup elements) have been proposed. Has been made. For example, Patent Document 1 discloses a technique in which a compound-eye camera (so-called multi-eye camera) is provided with a plurality of sensors having different numbers of pixels, and the signals of the sensors having a large number of pixels are added and then processed. Patent Document 2 discloses a technique for combining images of a plurality of cameras having different focal lengths by combining the low frequency component of the spatial frequency of the images from the cameras with a narrow angle of view and the phase of the signal of the camera with a wide angle of view. Has been done.

JP, 2010-157863, A JP, 2010-009417, A

However, when the technique of Patent Document 1 is used, it cannot be applied as it is to an optical system having a different focal length in which the pixel pitch is strongly restricted. In addition, when the technique of Patent Document 2 is used, the calculation load is increased because the calculation in the frequency space is required.

It is an object of the present invention to provide an image pickup apparatus capable of obtaining a high quality output image by a simple calculation.

In order to solve the above-mentioned problems, an image pickup apparatus of the present invention includes an optical system having a plurality of different focal lengths, a plurality of image pickup elements provided corresponding to each of the plurality of optical systems, and a plurality of different focus points. An acquisition unit that acquires a plurality of images captured by a distance optical system, and a combining unit that combines the plurality of images to generate a combined image, each combination of the optical system and the image sensor Is (focal length)/(length of one side of pixel) corresponding to the first optical system is N times (N) ((focal length)/(length of one side of pixel) corresponding to the second optical system. the Ri der relationship integer greater than 1), said synthesizing means, wherein corresponding to the first optical system (focal length) / (the side of the pixel length) corresponding to said second optical system ( The pixels of the image corresponding to the first optical system are added and combined so as to be equal to (focal length)/(length of one side of pixel) .

According to the present invention, it is possible to provide an image pickup apparatus capable of obtaining a high-quality output image by simple calculation.

It is a block diagram which shows the multi-lens imaging device and structure of 1st Embodiment. It is a schematic diagram of an image and a signal acquired by the multi-view imaging device. It is a figure explaining the evaluation value of a phase difference detection part. It is a figure explaining the physical meaning of this invention, and the flow of operation|movement. It is a block diagram which shows the multi-lens imaging device and structure of 2nd Embodiment.

(First embodiment)
FIG. 1 is an external view showing the external appearance of an image pickup apparatus according to this embodiment and a block diagram showing the electrical configuration of the image pickup apparatus. FIG. 1A is an external view of an image pickup apparatus 100 which is a multifocal image pickup apparatus. A plurality of optical systems 101a, 101b, 101c, 101d are arranged on the front surface of the image pickup apparatus 100. The switch 106a is one of the operation units 106, and is an automatic reset type push button switch. The switch 106a is operated with a finger or the like, and the switch S1 is turned on by lightly pushing it in, and the switch S2 is turned on by pushing it in deeply. When the push-in finger is released, the state before the push-in is restored, and the switches S1 and S2 are turned off. When the switch S1 is turned on, the focus and brightness are adjusted, and when the switch S2 is turned on, the still image capturing and the moving image capturing are started and ended.

FIG. 1B is a block diagram showing the system configuration of the image pickup apparatus 100. The image pickup apparatus 100 includes optical systems 101a to 101d, image pickup elements 102a to 102d, a lens driving unit 103, an image processing unit 104, an image display unit 105, a memory unit 108, and a control unit 107 that controls them. Each of the optical systems 101a, 101b, and 101c is an imaging optical system (focus lens) and constitutes an imaging optical system group. The focal lengths of the optical systems 101a, 10b, 101c are different from each other. The image pickup devices 102a, 102b, and 102c are image pickup devices having a plurality of photoelectric conversion units (pixels) that photoelectrically convert light rays guided to the optical systems 101a, 101b, and 101c, respectively. FIG. 1A illustrates four optical systems 101a, 101b, 101c, and 101d, and FIG. 1B illustrates the configurations up to three optical systems 101a, 101b, and 101c. A dot below 101c in FIG. 1B indicates that the description of 101d is omitted.

The lens driving unit 103 drives the optical systems 101a to 101d (focus lens) to perform lens driving such as focus driving for focusing on a subject. The image processing unit 104 is an image processing unit that processes signals output from a plurality of image pickup devices to generate an image. The image processing unit 104 includes an addition unit 104a and a phase difference detection unit 104b. The image processing unit 104 can perform image processing of synthesizing a plurality of images acquired so that the exposure times at least overlap. The image after image processing is called a composite image. The image processing referred to here is a process of presenting an image such as a zoom lens using images of the optical systems 101a, 101b, and 101c having different focal lengths, and noise is reduced by using signals of a plurality of sensors. Refers to processing, etc. The process of reducing noise here is a technique of reducing noise by smoothing by adding and averaging signals from a plurality of image pickup devices. The image display unit 105 displays an image. The operation unit 106 has a plurality of operation means for receiving a user's instruction, and includes the above-described switch 106a and the like. The control unit 107 is a camera system control unit that controls the entire camera system. The control unit 107 sends an instruction to the lens driving unit 103 to control the focal length. Further, distance measurement of the subject is performed using the image sent from the image processing unit 104. The distance measurement is performed on the subject within the defined area of the image. The memory unit 108 stores the acquired image.

FIG. 2 is a diagram schematically showing an image acquired by the image pickup apparatus 100 and explaining the operation of the phase difference detection unit 104b. 2A and 2B show images acquired by optical systems having different focal lengths. An image 111a shows an image acquired by an optical system having a short focal length (wide angle of view), and an image 111b shows an image acquired by an optical system having a long focal length (narrow angle of view). A frame 111b' shown in the image 111a of FIG. 2A schematically shows that the image 111b shown in FIG. 2B captures a part of the image 111a. The detection area 112 shown in the image 111a and the image 111b has shown the part in the image used for phase difference detection. Reference numeral 121 indicates a subject.

As is clear from FIGS. 2A and 2B, an image of a wider range is formed in the optical system having a short focal length (wide angle of view). This corresponds to the fact that the object image is small on the image plane, and the image forming magnification (used herein as so-called lateral magnification) differs depending on the focal length. Therefore, if sensors with the same pixel pitch are used, the size of the object corresponding to one pixel differs depending on the imaging magnification determined by the distance of the subject and the focal length.

2A and 2B, the base line length indicates the longitudinal direction of the screen, which corresponds to the horizontal direction in a general image pickup device. The baseline length here is the distance that the optical system that performs phase difference detection is separated, and the direction perpendicular to the baseline length indicates the direction orthogonal to the direction that the optical system that performs phase difference detection is separated. There is. Considering the correspondence with FIG. 1A, the images in FIGS. 2A and 2B correspond to the set of the optical system 101a and the optical system 101b or the set of the optical system 101c and the optical system 101d.

In a multi-focus imaging device, images obtained from optical systems with different focal lengths are subjected to image processing to generate an output image, but for that purpose, it is necessary to determine regions corresponding to the same subject. In the present embodiment, the phase difference detection unit 104b determines a region corresponding to the same subject from image information. 2C to 2E schematically show the operation of the phase difference detection unit 104b. 2C to 2E, the vertical axis represents the dimensionless brightness, and the horizontal axis represents the position in the screen in the base line length direction. The dimensionless brightness is dimensionless by dividing the signal of the acquired pixel by the average brightness of the screen in order to adjust the f-number and the difference in storage time for each optical system. .. That is, it can be said that the dimensionless brightness indicates the relative brightness in the screen.

FIG. 2C shows a signal obtained by extracting the signal of the detection area 112 in the image used for the phase difference detection shown in FIGS. 2A and 2B and then adding the direction perpendicular to the baseline length. Shows. The detection area 112 in the image, which is an area corresponding to the same subject and used for phase difference detection, has a rectangular shape whose longitudinal direction is the direction of the baseline. In FIG. 2C, a luminance signal 131a indicates a signal from the image 111a, and a luminance signal 131b indicates a signal from the image 111b. Data 140 to 144 indicate data on 131b, and data 150 and data 151 indicate data on 131a. In FIG. 2C, the fact that the number of data points of the luminance signal 131b is larger than that of the luminance signal 131a corresponds to that the spatial sampling is dense. This corresponds to enlarging and capturing a partial image as shown in FIG. 2B, and also reflects the influence of the pixel pitch.

In the example shown in FIG. 2C, the sampling interval is adjusted to be an integral multiple. That is, a plurality of optical systems are configured to satisfy the condition of the following Expression 1.

(Focal length of optical system 1)/(length of one side of pixel of optical system 1)
= N* (focal length of optical system 2)/(length of one side of pixel of optical system 2) (Equation 1)

In Formula 1, the optical system 1 represents an optical system having a relatively long focal length, and N represents a natural number (an integer larger than 1). The length of one side of a pixel indicates the length of one side of a pixel (generally a square) including a light receiving element and wiring. Assuming that the light receiving element is arranged at a fixed position in each pixel, the length of one side of the pixel is equivalent to the distance between the centers of the light receiving elements (pixel pitch), and Equation 1 may be defined as such. .. When the definition of Expression 1 in the example of FIG. 1B corresponds to the figure, the optical system 1 corresponds to 101c, the pixels of the optical system 1 correspond to the pixels of the image sensor 102c, and the optical system 2 corresponds to 101a to the optical system 2. The pixels of 1 correspond to the pixels of the image sensor 102a. That is, Expression 1 is an expression that defines the focal length and the length of one side of the pixel on the image sensor in the optical system having different focal lengths.
When N=1, it means an optical system having the same sampling density on the object side. In the example shown in FIG. 2C, N=4, and the luminance signal 131b is sampled at four times the density of the luminance signal 131a.

Further, as is clear from the expression 1, regarding the pixel, it is sufficient if the length of one side of the pixel of the optical system 1 is known, and it is not necessary to set the length of one side of the pixel to a predetermined value. The degree of freedom in setting is high compared to the case where the length of one side is set to a predetermined value. Therefore, if you want to use a sensor with different characteristics, for example, if you want to reduce the pixel pitch of the optical system with characteristics due to space restrictions to make the sensor smaller, or if you want to use an existing optical system, This is useful when the length of one side is limited. That is, Expression 1 can be satisfied by adjusting the other according to the constraint of one, and the reduction of the amount of calculation can be realized as described later. For example, when the existing sensor is used and the ratio of the length of one side of the pixel cannot be freely selected, the focal length may be designed to satisfy the formula 1. Although FIG. 2 shows the case where there are two optical systems, the present invention is not limited to this, and each of a plurality of optical systems may satisfy the relationship of Expression 1.

In order to combine the images with different focal lengths acquired by the multi-focus imaging device, it is necessary to align the images. For that purpose, it is necessary to process the signals of FIG. 2C to align them. In the image pickup apparatus 100 according to the present embodiment, since the relationship of Expression 1 is established, it is possible to easily perform processing by averaging N pixels of the optical system 1. The reason for averaging here is to normalize the difference in brightness. That is, considering an optical system having a large (focal length)/(length of one side of pixel), the addition unit 104a adds N pieces to (focal length)/((length of one side of pixel)*(number of additions) N)) can be made equal. If the relationship of the equation 1 shown in the present embodiment does not hold, it is necessary to perform processing in the frequency space and perform resampling, which increases the amount of calculation.

FIG. 2D shows a signal obtained by adding N adjacent signals (N=4 as described above in the example of FIG. 2) of the luminance signal 131b in FIG. 2C. For example, the data obtained by adding the adjacent data 140, 141, 142, and 143 in FIG. 2C is the data 145 in FIG. 2D. An alternate long and short dash line 146 penetrating from FIG. 2C to FIG. 2D is a line showing the position of the center of gravity of the added signals, and shows that they correspond spatially. That is, the data 145 is located between the data 141 and 142. The signal generated by adding N adjacent signals in this way is the luminance signal 131c.

式２においてｉは相対的に位置をずらした画素数を示しており、ｊは２つの画像の位相差検出に用いる画像の大きさに対応する。つまり差分の絶対値を加算（積分）したものである。 In FIG. 2D, the pitches of the luminance signal 131a and the luminance signal 131c, which are signals of the optical system 1 and the optical system 2, are made equal by a simple addition. Therefore, the phase difference between the two brightness signals 131a and 131c is detected by performing a phase detection operation known as a correlation operation. As is well known, the minimum of the absolute difference value (hereinafter referred to as SAD) may be detected to set the position where the phases match. The position where the phases match is, for example, as shown in FIG. SAD is defined as in Equation 2.

In Expression 2, i indicates the number of pixels whose positions are relatively displaced, and j corresponds to the size of the image used for detecting the phase difference between the two images. That is, the absolute value of the difference is added (integrated).

Further, the phase can be detected in the luminance signal 131a and the luminance signal 131c with an accuracy of one data shift or less by locally polynomial approximating the change in the absolute difference. This is useful in the process of minimizing pixel shift described later. This state is schematically shown in FIG. In FIG. 3, the vertical axis represents the SAD of two signals and the horizontal axis represents the laterally displaced amount (unit is pixel), and one vertical line corresponds to one pixel. More specifically, the SAD is calculated while changing the relative position of the partial signals in the image used for detecting the phase difference between the two images as shown in FIGS. 2D and 2E. The relative position is on the horizontal axis (i in Expression 2), and the SAD is on the vertical axis (SAD(i) in Expression 2). The numbers 1, 2, 3, and 4 in FIG. 3 indicate that the SAD value is smaller in this order. 3A shows an example in which the pixel shift is small, FIG. 3B shows an example in which the pixel shift is large, and FIG. 3C shows the data in FIGS. 3A and 3B. It shows how they are stacked. Although the value of SAD can be obtained only with an integer multiple pixel shift (one pixel unit), the minimum value can be approximated as shown in FIG. That is, the position where the SAD is the smallest (number 1), the position where the SAD is the second smallest (number 2), and the position where the SAD is the third smallest (number 3) are obtained. If the third smallest SAD (number 3) is on the opposite side of the second smallest SAD (number 1) from the second smallest SAD (number 2), the numbers 1 and 3 are straight lines Tie in. This straight line is shown as a solid line. Extend this line. On the other hand, a straight line having a slope opposite to that of the straight line connecting the numbers 1 and 3 is made to pass through the place where the SAD is the second smallest (number 2). This straight line is indicated by a broken line. The intersection of the solid line and the broken line is the position of the minimum value.

When the minimum position of the SAD is obtained from the data of FIGS. 3A and 3B by the same processing, there is a minimum value in the vicinity of the smallest SAD (numeral 1) in FIG. 3A. In FIG. 3B, it can be seen that there is a minimum value at a position close to the middle of the place where the SAD is the smallest (number 1) and the place where the SAD is the second smallest (the number 2). In the case of FIG. 3A, the pixel shift is already small, so image processing is performed to generate an output image in a state where adjacent data 140, 141, 142, and 143 are added. On the other hand, in the case of FIG. 3B, the pixel before addition has a shift of one pixel or more. Therefore, after the addition start phase is determined so as to minimize the pixel shift, pixel addition is performed again. Give. This process will be described later with reference to FIG.

Further, focusing on the vertical axes in FIGS. 3A and 3B, it is assumed that each data point is on one smooth function. In FIG. 3C, the data of FIG. 3A and the data of FIG. 3B are overlapped, and the numbers 1 to 4 of FIG. 3A are numbered 1 a to 4 a and the numbers 1 to 1 of FIG. 4 are indicated by numerals 1b to 4b, respectively. The data is placed on the curved line shown by the broken line in FIG. 3C, and the number 1a, which is a data point closer to the minimum value, has the smallest value. This indicates that the degree of coincidence between images having different focal lengths is better when the addition is performed in the state where the pixel shift is small.

Next, the physical correspondence with the position of the minimum value of SAD and the processing method of this embodiment will be described with reference to FIGS. 4(A) to 4(C). 4A and 4B show a state in which pixels of the image sensor are projected on the plane 180. 4A corresponds to FIG. 3A, and FIG. 4B corresponds to FIG. 3B. Further, FIG. 4C is a flowchart showing the processing of this embodiment.

In FIGS. 4A and 4B, data 150 and data 151 are data acquired by an optical system having a short focal length, and correspond to the signals with the same numbers in FIG. The data 140, 141, 142, 143, 144 are data acquired by the optical system having a long focal length, and correspond to the signals with the same numbers in FIG. The addition data 145 indicates a signal obtained by adding the data 140, 141, 142, and 143. Addition data 165 represents a signal obtained by determining the addition start phase so that the pixel shift becomes the smallest and then adding again.

The signals in FIG. 4A and FIG. 4B are compared. Considering the data 150 and 151 acquired by the optical system having a small (focal length)/(length of one side of the pixel) as a reference, the optical system having a large (focal length)/(length of one side of the pixel) is acquired. The positions (phase) of the generated data 140, 141, 142, 143, 144 are different. This position is determined by the distance (baseline length) between the two optical systems shown in FIG. 4A, the focal length of the two optical systems, and the object distance. Since the variable includes the object distance, it varies depending on the subject and cannot be obtained in advance. For this reason, in the present embodiment, once addition is performed at an appropriate position to detect the phase difference, the addition is performed again so that the pixel shift is minimized.

In FIG. 4A, the positions of the added data 145 and the data 151 are almost aligned. This is a state in which the pixel shift is small. At this time, the SAD obtained from the signal from the optical system having a small (focal length)/(length of one side of the pixel) and the addition signal is such that the minimum value is a pixel shift of an integral multiple, as shown in FIG. It is close to the quantity. In this case, image processing is performed with the added phase unchanged. As a result, the addition data 145 obtained by adding the data 140, 141, 142, 143 can be used as it is as the addition data 165 with the phase adjusted.

On the other hand, in FIG. 4B, the positions of the added data 145 and the data 151 are shifted by one data from the data before the addition. As shown in FIG. 3B, the SAD obtained from the signal from the optical system having a small (focal length)/(length of one side of the pixel) and the addition signal has a minimum value near the pixel shift amount of an integral multiple. Not between the data but the data. When viewed with a pixel having a long focal length, a shift of one pixel or more occurs. In the example shown in FIGS. 2 and 4, since N=4 in the equation 1, when the phase between the signal from the optical system having a short focal length and the added signal is shifted by 0.25 pixel, a long focal length is obtained. In, the position of the data for one pixel is displaced. As described above, the minimum value of SAD can be obtained in units of one pixel or less, so the addition start phase is adjusted so that this deviation is minimized. More specifically, in the present embodiment, when there is a deviation of 0.125 pixels from 0.375 pixels, a process of shifting by 0.25 pixels (=shifting by 1 pixel for a pixel having a long focal length) is performed. As a result, the data 141, 142, 143, 144 are added to obtain added data 165. Physically, as shown in FIG. 4B, the state in which addition is performed in pixel units that are not directly facing each other (state of addition data 145) is the state in which the pixel shift is the smallest (addition data). 165 state). When the addition data 145 and the addition data 165 are compared, as shown in FIG. 3C, the addition data 165 has a higher degree of image coincidence. That is, in the next image processing, precise alignment is easy.

Next, the processing of this embodiment will be described step by step with reference to FIG. Step S100 indicates the start of processing. For example, the process is started by the operation of pressing the push switch 106a of the imaging device 100. In step S110, a pixel signal (image) is acquired. The image processing unit 104 acquires each signal output from the plurality of image pickup devices corresponding to each optical system. In the present embodiment, the image of each optical system is acquired at the same exposure time at the same time in synchronization with the signal instructing the imaging. In step S120, the addition unit 104a adds the pixels of the image so that (focal length)/(length of one side of pixel) becomes equal. Specifically, an integer number of images having a large (focal length)/(length of one side of pixel) are added. The added phase at this time is a value that is initially determined, and is not necessarily optimized for the subject distance.

In step S130, the phase difference detection unit 104b detects the phase difference between the signal of the optical system having a small (focal length)/(length of one side of pixel) and the signal added in step S120. Although the specific operation is shown in FIGS. 2 and 3, for example, SAD may be calculated and the minimum value thereof may be detected. In step S140, the addition unit 104a performs addition again so that the pixel shift is minimized and (focal length)/(length of one side of pixel) is aligned. As the denominator (length of one side of pixel), the size after addition is used. The specific operation has been described with reference to FIGS. This is the operation of generating the addition data 165 in FIG.

In step S150, the image processing unit 104 performs image processing to generate a composite image for recording. The multi-lens imaging device of the present embodiment synthesizes and outputs images having different focal lengths. As a combining method, for example, signals of a plurality of image pickup elements are added to reduce noise, or an image with a large (focal length)/(length of one side of pixel) is inserted in an image near the center to increase the noise. There is a method to increase the resolution. In either process, it is necessary to precisely align images from different optical systems in order to determine the addition position and the insertion position. According to the present embodiment, the amount of calculation is reduced by keeping the condition of Expression 1. Further, by minimizing the pixel shift, the degree of coincidence of images is increased and a signal convenient for alignment is generated. Step S160 is a step of recording the recording image generated in S150 in the memory unit 108. The display of the image display unit 105 on the monitor is also performed as necessary. In step S170, the operation ends.

As described above, according to the present embodiment, calculation between images having different focal lengths is performed by simple calculation by setting (focal length)/(length of one side of pixel) to an integer. be able to. Therefore, it is possible to provide a multifocal imaging device capable of obtaining a high-quality output image with few errors.

(Second embodiment)
The multifocal imaging device according to the second embodiment will be described below with reference to FIG. In FIG. 5, parts having the same functions as in FIG. 1 are given the same numbers. The image pickup apparatus 100 of FIG. 5 is different from that of FIG. 1 of the first embodiment in that the imaging optical system group includes an optical system 201a having a variable focal length. The image pickup apparatus 100 includes a so-called zoomable optical system 201a and fixed focal length optical systems 201b and 201c that do not zoom. Other configurations are the same as those in the first embodiment. In this embodiment, the variable focal length optical system 201a operates as a so-called step zoom. That is, it is possible to stop only at the position where the condition of Expression 1 shown in the first embodiment is observed. For example, assuming that the pixel pitch is the same, when the focal lengths of the fixed focal length optical systems 201b and 201c are 25 mm, the variable focal length optical system 201a stops only at the focal length positions of 50, 75, and 100 mm. It is possible to capture images.

As described above, according to the present embodiment, the operation between images having different focal lengths is performed by (focal length)/(length of one side of pixel) by operating the optical system having a variable focal length as a so-called step zoom. Can be an integer and can be combined by a simple operation. Therefore, it is possible to provide a multifocal imaging device capable of obtaining a high-quality output image with few errors.

(Other embodiments)
In the first embodiment and the second embodiment, an example in which image processing is performed in the imaging device 100 has been shown. However, the present invention is not limited to this, and for example, even if a plurality of image pickup devices each including an optical system that is a lens and an image pickup device corresponding to the lens are connected to an image processing device that performs image processing, and the image pickup system is realized. Good. In this case, the optical system of each imaging device has a different focal length.
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the gist thereof.

100 image pickup apparatus 101a optical system 102a image pickup element 104 image processing unit

Claims (13)

Optical systems with different focal lengths,
A plurality of imaging elements provided corresponding to each of the plurality of optical systems,
Acquisition means for acquiring a plurality of images captured by the optical systems having a plurality of different focal lengths,
A synthesizing unit that synthesizes the plurality of images to generate a synthetic image,
In each combination of the optical system and the image pickup device, (focal length)/(length of one side of pixel) corresponding to the first optical system corresponds to (focal length)/(pixel) of the second optical system. der relationship N times the length of one side) (N is an integer greater than 1) is,
In the synthesizing means, (focal length)/(length of one side of pixel) corresponding to the first optical system corresponds to (focal length)/(length of one side of pixel) corresponding to the second optical system. An image pickup apparatus characterized in that pixels of an image corresponding to the first optical system are added and combined so as to be equal to . Optical systems with different focal lengths,
A plurality of imaging elements provided corresponding to each of the plurality of optical systems,
Acquisition means for acquiring a plurality of images captured by the optical systems having a plurality of different focal lengths,
A synthesizing unit that synthesizes the plurality of images to generate a synthetic image,
In each combination of the optical system and the image pickup device, (focal length)/(length of one side of pixel) corresponding to the first optical system corresponds to (focal length)/(pixel) of the second optical system. der relationship N times the length of one side) (N is an integer greater than 1) is,
The image pickup apparatus , wherein the synthesizing unit adds N pixels of the image corresponding to the first optical system and synthesizes them . Optical systems with different focal lengths,
A plurality of image pickup elements having different pixel pitches provided corresponding to each of the plurality of optical systems;
Acquisition means for acquiring a plurality of images captured by the optical systems having a plurality of different focal lengths,
A synthesizing unit that synthesizes the plurality of images to generate a synthetic image,
In each combination of the optical system and the image pickup device, (focal length)/(length of one side of pixel) corresponding to the first optical system corresponds to (focal length)/(pixel) of the second optical system. The length of one side is N times (N is an integer greater than 1). The image capturing apparatus according to claim 1, wherein the acquisition unit acquires images captured in synchronization. Further comprising a phase difference detection means for detecting the phase difference of the plurality of images,
Based on the phase difference, the synthesizing unit determines a phase at which addition is started so that a pixel shift is minimized, and adds pixels of an image having a large (focal length)/(length of one side of pixel). The image pickup device according to claim 1, wherein the image pickup device and the image pickup device are combined. In the phase difference detection means, the combining means corresponds to the first optical system (focal length)/(length of one side of the pixel) corresponds to the second optical system (focal length)/(pixel 6. The image pickup according to claim 5, wherein the phase difference is detected based on an image obtained by adding pixels of an image corresponding to the first optical system so as to be equal to (length of one side). apparatus. The plurality of optical systems having different focal lengths include an optical system having a variable focal length and an optical system having a fixed focal length,
The focal length of the variable focal length optical system is such that (focal length)/(length of one side of pixel) is an integer multiple of (focal length)/(length of one side of pixel) of the optical system having the fixed focal length. The image pickup apparatus according to claim 1, wherein the image pickup apparatus is set to a position where A plurality of image pickup devices each having an optical system and an image pickup device provided corresponding to the optical system, an acquisition unit that acquires a plurality of images picked up by the optical system, and the plurality of images are combined. And an image processing device having a synthesizing unit for generating a synthetic image,
The optics have different focal lengths,
In each combination of the optical system and the image pickup device, (focal length)/(length of one side of pixel) corresponding to the first optical system corresponds to (focal length)/(pixel) of the second optical system. der relationship N times the length of one side) (N is an integer greater than 1) is,
In the synthesizing means, (focal length)/(length of one side of pixel) corresponding to the first optical system corresponds to (focal length)/(length of one side of pixel) corresponding to the second optical system. An image pickup system , wherein pixels of an image corresponding to the first optical system are added and combined so as to be equal to . A plurality of image pickup devices each having an optical system and an image pickup device provided corresponding to the optical system, an acquisition unit that acquires a plurality of images picked up by the optical system, and the plurality of images are combined. And an image processing device having a synthesizing unit for generating a synthetic image,
The optics have different focal lengths,
In each combination of the optical system and the image pickup device, (focal length)/(length of one side of pixel) corresponding to the first optical system corresponds to (focal length)/(pixel) of the second optical system. The length of one side is N times (N is an integer greater than 1),
The synthesizing unit adds N pixels of the image corresponding to the first optical system and synthesizes them.
An imaging system characterized by the above. A plurality of image pickup devices each having an optical system and an image pickup device provided corresponding to the optical system, an acquisition unit that acquires a plurality of images picked up by the optical system, and the plurality of images are combined. And an image processing device having a synthesizing unit for generating a synthetic image,
The optics have different focal lengths,
The image sensor has different pixel pitches,
In each combination of the optical system and the image pickup device, (focal length)/(length of one side of pixel) corresponding to the first optical system corresponds to (focal length)/(pixel) of the second optical system. (Length of one side) N times (N is an integer greater than 1)
An imaging system characterized by the above. A control method for an image pickup apparatus, comprising: an optical system having a plurality of different focal lengths; and a plurality of image pickup elements provided corresponding to each of the plurality of optical systems,
An acquisition step of acquiring a plurality of images captured by the optical systems having a plurality of different focal lengths;
A combining step of combining the plurality of images to generate a combined image,
In each combination of the optical system and the image pickup device, (focal length)/(length of one side of pixel) corresponding to the first optical system corresponds to (focal length)/(pixel) of the second optical system. The length of one side is N times (N is an integer greater than 1),
In the combining step, (focal length)/(length of one side of pixel) corresponding to the first optical system corresponds to (focal length)/(length of one side of pixel) corresponding to the second optical system. The control method is characterized in that the pixels of the image corresponding to the first optical system are added so as to be equal to. A control method for an image pickup apparatus, comprising: an optical system having a plurality of different focal lengths; and a plurality of image pickup elements provided corresponding to each of the plurality of optical systems,
An acquisition step of acquiring a plurality of images captured by the optical systems having a plurality of different focal lengths;
A combining step of combining the plurality of images to generate a combined image,
In each combination of the optical system and the image pickup device, (focal length)/(length of one side of pixel) corresponding to the first optical system corresponds to (focal length)/(pixel) of the second optical system. The length of one side is N times (N is an integer greater than 1),
The control method characterized in that, in the combining step, N pixels of the image corresponding to the first optical system are added and combined. A method of controlling an image pickup apparatus having a plurality of optical systems having different focal lengths, and a plurality of image pickup elements having different pixel pitches provided corresponding to the respective plurality of optical systems,
An acquisition step of acquiring a plurality of images captured by the optical systems having a plurality of different focal lengths;
A combining step of combining the plurality of images to generate a combined image,
In each combination of the optical system and the image pickup device, (focal length)/(length of one side of pixel) corresponding to the first optical system corresponds to (focal length)/(pixel) of the second optical system. The length of one side is N times (N is an integer greater than 1),
In the combining step, (focal length)/(length of one side of pixel) corresponding to the first optical system corresponds to (focal length)/(length of one side of pixel) corresponding to the second optical system. The control method is characterized in that the pixels of the image corresponding to the first optical system are added so as to be equal to.

Images