JP4606989B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus including two or more imaging units each including an imaging optical system and an imaging element.

  Among the photographing devices, there is a photographing device called a compound eye camera including two or more imaging units each including a photographing optical system and an image sensor. When photographing is performed using such a photographing apparatus, a panoramic image can be obtained by combining images photographed by two or more imaging units.

  When shooting is performed by an imaging device including two or more imaging units, an angle of view captured by one imaging optical system and an angle of view of the imaging optical system having an angle of view adjacent to the angle of view of the imaging optical system are A process of continuously connecting the images captured by both the photographing optical systems is performed by detecting the overlapping region by the image composition unit with a slight overlap.

For example, according to the technique of Patent Document 1 for performing image composition by the image composition unit, the subtraction process is performed while shifting one image signal of the image signals obtained from each of the two or more image pickup units. A process is performed in which the area where is minimized is used as an overlapping area. However, if the subtraction process is performed while shifting the image signal, there is a problem that the subtraction process takes time.
Japanese Patent Laid-Open No. 06-125497

  In view of the above circumstances, an object of the present invention is to provide a photographing apparatus that realizes high-speed image composition processing by an image composition unit.

The imaging device of the present invention that achieves the above object is an imaging device including two or more imaging units each including an imaging optical system and an imaging element.
The angle of view captured by each of the two or more photographic optical systems has an overlapping region in which both angles of view overlap each other in a portion adjacent to each other,
An image synthesizing unit that receives each of the image signals generated by each of the two or more imaging elements and generates a combined image signal representing an image obtained by combining the images represented by the received image signals;
An intermediate unit provided between the image sensor and the image composition unit for the number of the image sensor, and converting an analog image signal generated by the image sensor into a digital image signal and outputting the digital image signal to the image composition unit A processing unit,
Each of the intermediate processing units includes a storage unit in which coordinate data of each overlapping region of the angle of view captured by each of the two or more photographing optical systems is stored.
With reference to the storage unit, each image signal for each pixel included in the imaging device, which is generated by the imaging device, is an image signal of a pixel in the overlapping region or an image of a pixel that is not in the overlapping region A predetermined process is performed on the image signal so that the signal is identified by the image composition unit, and the image signal is output to the image composition unit.

  According to the photographing apparatus of the present invention, the image is processed after predetermined processing is performed so that the image processing unit can easily identify whether the intermediate signal is an image signal of the pixel in the overlapping region. The signal is output to the image composition unit.

  Then, the image synthesizing unit immediately identifies whether or not the image signal is the overlapping region without performing the process of searching for the overlapping region while shifting the pixel signal obtained from the intermediate processing unit as before. Will be able to. That is, the speed of the image synthesizing process in the image synthesizing unit is achieved as long as the process of shifting one image signal is not necessary.

  As described above, it is possible to realize a photographing apparatus that realizes high-speed image composition processing by the image composition unit.

  In order for the intermediate processing unit to perform a predetermined process on the image signal so as to be identified by the image composition unit, it is only necessary to have the following functions.

  For example, the intermediate processing unit adds a flag indicating whether the pixel is a pixel in the overlapping area to the image signal for each pixel based on the coordinate data stored in the storage unit, and then adds the flag. Even when the image is output to the image composition unit, the intermediate processing unit refers to the storage unit when the image signal for each pixel included in each of the imaging elements is in the overlapping region. Even if the image signal on either one of the adjacent image sensors is set to 0 level and output to the image composition unit, the intermediate processing unit further supplies a reset pulse to the image sensor and If it is possible to forcibly output a signal representing the 0 level from the image sensor, the intermediate processing unit refers to the storage unit and outputs an image signal for each pixel included in each of the image sensors. The Is determined to be in the overlapping region, the reset pulse is supplied to the image sensor, and the image signal on either one of the adjacent image sensors is set to 0 level and output from the image sensor. A signal representing the zero level may be output to the image composition unit.

  Here, when the intermediate processing unit is provided with the above function, it is preferable that the analog front end that has been conventionally provided has the function.

  If it does so, the said intermediate | middle process part can be comprised easily using a conventional thing.

Further, when a signal representing the 0 level signal is output from the intermediate processing unit,
In the case where the image composition unit detects that the value of the image signal for each pixel output from any of the intermediate processing units has a change of a predetermined value or more and the signal of 0 level continues thereafter, It is preferable to perform image synthesis processing of an image signal by regarding a region where a change of more than a predetermined value or more as a boundary is followed by a region in which a 0 level signal continues as an overlapping region.

  Then, it is possible to easily construct an image composition suitable for the above functions by slightly extending the functions so far.

   As described above, it is possible to realize a photographing apparatus that realizes high-speed image composition processing by the image composition unit. Further, since the conventional configuration can be used, an effect that the cost of the photographing apparatus can be reduced is also obtained.

  Embodiments of the present invention will be described below.

  FIG. 1 is a view showing an appearance of a photographing apparatus 1 according to an embodiment of the present invention.

  The photographing apparatus 1 shown in FIG. 1 includes three photographing lenses 10A, 10B, and 10C. The angle of view captured by the photographing lenses 10A to 10C is different, and a panoramic photograph can be obtained by connecting the subjects captured by the photographing lenses 10A to 10C.

  FIG. 2 is a diagram showing the angle of view captured by each of the three photographing lenses shown in FIG.

  Of the three photographing lenses 10A to 10C, the angle of view of the photographing lens 10A is denoted by A, the angle of view of the photographing lens 10B is denoted by B, and the angle of view of the photographing lens is denoted by C. It is. Each angle of view A, B, and C is set as a different coordinate system, and the coordinates of each coordinate system are expressed as (Hsize, Vsize) A, (Hsize, Vsize) B, (Hsize, Vsize) C. .

  In FIG. 2, the overlapping area captured by both the photographing lens 10A and the photographing lens 10B and the overlapping area captured by both the photographing lens 10B and the photographing lens 10C are both hatched. The horizontal length of the overlapping region is HcA, the vertical length is VcA, the horizontal length of the overlapping region on the latter side is HcB, and the vertical length is VcB.

  If the coordinate data of the overlapping area indicated by hatching is stored in the memory in the analog front end, the memory is referred to when outputting the image signal output from the image sensor to the image composition unit. If it is an overlapping area, a process of outputting an image signal by adding a flag can be performed. As shown in FIG. 2, when the three angles of view A, B, and C are set to be offset in either the upper or lower direction, the height of each angle of view is set in the offset direction (upward direction). : Offset amount V ** or downward direction: offset amount -V **) may be stored in the memory together with the coordinate data indicating the overlapping area.

  FIG. 3 is a diagram showing an internal configuration of the photographing apparatus shown in FIG.

  As shown in FIG. 3, imaging devices (herein, CCD solid-state imaging devices are used, which will be referred to as CCDs in the following description) 11A to 11C are provided behind the photographing lenses 10A, 10B, and 10C. Analog front ends (hereinafter referred to as AFEs) 12A to 12C are respectively provided downstream of the CCDs 11A to 11C. After the noises of the analog image signals output from the CCDs 11A to 11C are reduced by the AFEs 12A to 12C, the analog image signals subjected to the noise reduction are further converted into digital signals.

  In the present embodiment, when the image signals converted into digital signals in the three AFEs 12A, 12B, and 12C are supplied to the image synthesis unit 13, the memories 120A, 120B, and 120C in the AFEs 12A, 12B, and 12C are referred to, respectively. The flag signals are added to the image signals of the pixels in the overlapping area and supplied to the image composition unit 13.

  The image synthesis unit 13 that has received the image signal to which the flag bit is added determines, for example, one of the image signals in the overlap region as the image signal determined to be an image signal in the overlap region by the flag. Are combined, and a combined image signal is generated based on the image signals output from the three AFEs 12A to 12C.

  The composite image signal thus generated is supplied to the display unit to display a panoramic image, or supplied to the recording unit and image data representing the panoramic image is recorded on the memory card.

  As can be seen from FIG. 3, in the present embodiment, an example in which three imaging units each including a photographing lens and an imaging element are provided is shown.

  FIG. 4 is a diagram illustrating an input order when the image signal output from the AFE 12A is input to the image composition unit 13.

  As shown in FIG. 4, the image signal corresponding to the pixel at the lower left position is sequentially output, and the image signal is output line by line, and the image signal corresponding to the pixel at the upper right position is output. The signal output ends.

  A portion surrounded by a dotted line shown in FIG. 4 becomes a portion of an overlapping area (HcA, VcA) indicated by hatching in FIG. 2, and when the image signal of this portion is output to the image composition unit 13, N-bit In addition to the image signal, flag bit FLG_A = “1” is added and input to the image composition unit 13, and when an image signal other than that is output, FLG_A = “0” is added and input to the image composition unit 13. The 4 shows the overlapping area of the angle of view A and the angle of view B shown in FIG. 2, but the same applies to the image signal for each pixel in the overlapping area of the angle of view B and the angle of view C shown in FIG. Is processed.

  FIG. 5 is a flowchart showing a processing procedure performed by the image composition unit 13.

  As shown in FIG. 5, when image signals are received from the three AFEs 11A to 11C in step S501, whether or not there is a pixel in which a flag indicating an overlapping region is set in each image signal in the next step S502. Check. Proceeding to the next step S503, a flag is set in the image signal for one pixel, and FLG_A (here, the A area is taken as an example, but if the B area is FLG_B, if the C area is FLG_C) Is determined to be “1”, the process proceeds to Yes, and the pixel is considered to be included in the overlapping region, and the process of this flow is terminated. If it is determined in step S503 that no flag is set in the image signal for one pixel and FLG_A (or FLG_B or FLG_C) = '0', the process proceeds to No and the pixel is regarded as not included in the overlapping region. The process of this flow is terminated.

  As described above, when the flag is added on the AFE side to the image signal in the overlap area among the image signals for each pixel, the image composition unit 13 includes the image signals in two angles of view of the three CCDs 10A to 10C. It is immediately discriminated whether the image signal is in the overlapping area (hatched portion shown in FIG. 2). In this case, for example, processing for discarding the image signal in the overlapping area is performed in the image composition unit, and composition processing is performed so that the image signals supplied from the three CCDs 10A to 10C are connected well, and then the composition is performed in the display unit or recording unit. An image signal is supplied.

  In this case, the process of searching for an overlapping area while shifting the image signal by the image composition unit 13 is omitted, so that the image composition process is facilitated and speeding up is achieved.

  In the example of FIG. 4, an example is shown in which one bit is added to the flag in the AFE. However, as shown in FIG. 6, a flag adding unit 19 is separately provided after the AFE to add the flag. May be. FIG. 6 is a diagram showing a configuration when a flag adding unit 19 is separately provided outside the AFE 12.

  7 and 8 show the second embodiment.

  FIG. 7 is a diagram for explaining the internal configuration of the AFE 12. FIG. 8 is a diagram for explaining what image signal is output when the phases of the two sampling pulses SHP and SHD supplied to the CDS 121 are changed. FIG. 8A shows a waveform diagram showing the sampling state of the image signal for each pixel in the normal state (non-overlapping area), and FIG. 8B shows the waveform for each pixel in the overlapping area. A waveform diagram showing the sampling state of the image signal is shown.

  As shown in FIG. 7, when an image signal is output from the CCD to the AFE 12, the correlated double sampling is first performed by the CDS 121 in the AFE, and the correlated double sampled image signal is amplified by the amplifier 122. Thereafter, the amplified analog image signal is converted into a digital image signal by the A / D converter 123 and output to the image composition unit 13.

  As shown in FIG. 7, the CDS 121 is supplied with a sampling pulse SHP and another sampling pulse SHD, and is sampled twice in synchronization with each sampling pulse to generate an image signal for each pixel. This is supplied to the amplifier 122. When the image signal for each pixel is generated, as shown in FIG. 8A, after the black level is clamped by one sampling pulse SHP, the image signal per pixel is obtained by another sampling pulse SHD. Is extracted.

  In the present embodiment, the above-described problem is to be achieved by using an AFE having a configuration capable of freely changing the phase of these two sampling pulses.

  In this example, when processing the image signal in the non-overlapping area, the phase relationship between the two sampling pulses SHP and SHD is set to the phase relationship shown in FIG. 8A, and the black level is first sampled with one sampling pulse SHP. The image signal is sampled by another sampling pulse SHD.

  Here, when the AFEs 12A to 12C refer to the built-in memories 120A to 120C and determine that the image signals from the CCDs 11A to 11C are image signals in the overlapping region, the phase of one sampling pulse SHD is changed. The output of the image signal for each pixel is forcibly set to 0 level so that sampling is performed at the timing shown in FIG. 8B in phase with the other sampling pulse. Here, the process of keeping the clamped at the black level is the process of outputting at the 0 level.

  In this way, the signal which is forcibly set to the 0 level is output and supplied to the image composition unit 13, and the image composition unit 13 is caused to perform processing for regarding the place where the 0 level signal continues as an overlapping region. The same effect as when a flag is added can be obtained.

  FIG. 9 is a flowchart showing the procedure of image signal output processing performed by each AFE 12A, 12B, 12C. FIG. 10 is a flowchart showing the procedure of the image composition process performed by the image composition unit 13 in response to the image signal output from the AFE 12 by the process shown in FIG.

  In step S901, image signals for each pixel output from the CCD are sequentially received. In step S902, each AFE 12A, 12B, 12C (see FIG. 3) refers to the built-in memories 120A, 120B, 120C to determine whether the signal is a pixel at the coordinates of the overlapping region. When it is determined in step S902 that the pixel is in the overlapping region, the process proceeds to YES, and in step S903, the phase of one sampling pulse SHD is changed and the both phases are matched to perform CDS processing to indicate a signal representing 0 level. Is output. When it proceeds to the next step S904 and it is determined that it is determined whether or not there is a signal for the next pixel, the process proceeds to the YES side, returns to step S901, and repeats the processing from step S902 to step S904. If it is determined that there is no pixel signal and the end position shown in FIG. 4 is reached, the processing of this flow is terminated.

  If it is determined in step S902 that the signal is not in the overlapping region, the process proceeds to step S905, and normal CDS processing is performed in step S905, and the process proceeds to step S904 to determine whether there is a signal for the next pixel. If it is determined in step S904 that there is a next signal, the process returns to step S902 to repeat the processing from step S901 to step S904, and if it is determined in step S904 that there is no signal for the next pixel, the processing of this flow is terminated. Thus, the image signal for which the processing of FIG. 9 has been completed is output from the AFE.

  In response to the output of the image signal from the AFE, the image composition unit 13 starts the processing of the steps shown in FIG.

  In step S1001, image signals for each pixel output from the CCD from each AFE are sequentially received. Proceeding to the next Step S1002, it is determined whether or not the signal level 0 continues for a predetermined number of times. If it is determined in step S1002 that the signal has continued for a predetermined number of times and the process proceeds to YES, the process from the pixel at the position where the 0 level signal is received to the pixel that is not 0 is performed as the pixel area in step S1003. The process proceeds to step S1004. If it is determined in this step that the predetermined number of times is not continued, the process proceeds to No, jumps to the process in step S1003, and proceeds to the next step S1004. If it is determined in step S1004 whether or not there is a next pixel signal and it is determined that there is a next pixel signal, the process proceeds to YES, and the processing in steps S1001 to S1004 is repeated. In step S1004, the next pixel signal is received. If it determines with there being no, it will progress to No side and will complete | finish this flow process.

  Even if it does in this way, the effect similar to 1st Embodiment is acquired.

  11 and 12 are diagrams showing a third embodiment.

  In the second embodiment, a signal representing the 0 level is generated by the CDS 121 in the AFE 12, but in the third embodiment shown in FIGS. 11 and 12, the signal is overlapped by changing how the reset pulse is supplied from the AFE 12 M to the CCD 11. When the image signal of the area is output, a signal representing the 0 level is output.

  As shown in FIG. 12 (a), when outputting the image signal of the non-overlapping area, a reset pulse is repeatedly supplied to the CCD to output the image signal for each pixel as usual. Instead of repeatedly supplying a reset pulse for each pixel, the reset pulse is continuously supplied to the CCD 11 to output a pixel signal representing 0 level.

  Even if it does in this way, the effect similar to 2nd Embodiment can be acquired.

  In the present embodiment, an example of an imaging device including three imaging units each including an imaging optical system and an imaging element has been described. However, even if the imaging device includes four or more imaging units, the imaging unit is not provided. Two imaging devices may be provided.

1 is a diagram illustrating an appearance of a photographing apparatus 1 that is an embodiment of the present invention. It is a figure which shows the angle of view which each of the three imaging lenses shown in FIG. 1 captures. It is a figure which shows the internal structure of the imaging device shown in FIG. It is a figure which shows the input order when the image signal output from AFE11A is input into the image synthetic | combination part 12. FIG. It is a flowchart which shows the process sequence which the image synthetic | combination part 12 performs. It is a figure which shows the structure at the time of providing a flag addition part separately. It is a figure explaining the internal structure of AFE. It is a figure explaining what kind of image signal is output when the phase of each of two sampling pulses supplied to CDS is changed. It is a flowchart explaining the processing procedure of AFE. It is a flowchart which shows the procedure of the image composition process which an image composition part receives the image signal output from AFE. It is a figure explaining 3rd Embodiment. It is a figure explaining the operation | movement of FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image pick-up apparatus 10A 10B 10C Photographic lens 11 11A 11B 11C CCD solid-state image sensor 12 12A 12B 12C 12M AFE
120A 120B 120C Memory 121 CDS
122 Amplifier 123 A / D Converter 13 Image Composition Unit

Claims (5)

In an imaging apparatus including two or more imaging units each including an imaging optical system and an imaging element,
The angle of view captured by each of the two or more photographic optical systems has an overlapping region in which both angles of view overlap each other in a portion adjacent to each other,
An image synthesizing unit that receives each of the image signals generated by each of the two or more imaging elements and generates a composite image signal representing an image obtained by combining the images represented by the received image signals;
The number of the image sensor is provided between the image sensor and the image composition unit, the analog image signal generated by the image sensor is converted into a digital image signal and output to the image composition unit , An intermediate processing unit capable of supplying a reset pulse to the image sensor and forcibly outputting a signal representing 0 level from the image sensor ;
Each of the intermediate processing units includes a storage unit that stores in advance the coordinate data of each overlapping region of the angle of view captured by each of the two or more photographing optical systems, and is generated by the imaging device with reference to the storage unit. In addition, the image composition unit can identify whether each image signal of each pixel included in the imaging element is an image signal of a pixel in the overlapping area or an image signal of a pixel not in the overlapping area. der outputs to the image synthesizing unit from performing a predetermined process on the image signal is,
When the intermediate processing unit refers to the storage unit and determines that the image signal for each pixel included in each of the imaging elements is in an overlapping region, the intermediate processing unit supplies the reset pulse to the imaging element. An image pickup apparatus characterized in that an image signal on any one of the adjacent image pickup devices is set to 0 level, is output from the image pickup device, and a signal representing the 0 level is output to the image composition unit. .   The intermediate processing unit adds a flag indicating whether or not the pixel is a pixel in the overlapping region to the image signal for each pixel based on the coordinate data stored in the storage unit, and then combines the image The photographing apparatus according to claim 1, wherein the photographing apparatus outputs the image to a unit.   The intermediate processing unit refers to the storage unit, and when the image signal for each pixel included in each of the imaging elements is in the overlapping region, the image signal on the side of one of the adjacent imaging elements The photographing apparatus according to claim 1, wherein the level is set to 0 and output to the image composition unit. The photographing apparatus according to claim 2, wherein the intermediate processing unit is an analog front end. In the case where the image composition unit detects that the value of the image signal for each pixel output from any of the intermediate processing units has changed by a predetermined value or more and then continues with a 0 level signal, 4. The photographing apparatus according to claim 3, wherein an area in which a 0-level signal continues with a portion having a change greater than the predetermined value as a boundary is regarded as an overlapping area.

Images