JP6537231B2 - Image processing apparatus, image processing method and program - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method for synthesizing a wide-angle image from a plurality of images, and in particular, when synthesizing a wide-angle image, it is possible to obtain good image quality in which the occurrence of overexposure and blackout is suppressed. Related to technology.

  BACKGROUND In recent years, solid-state imaging devices such as CCD sensors and CMOS sensors have been widely used as imaging means in imaging devices such as video cameras (television cameras) and still cameras. A method of capturing a plurality of images in the same scene and combining the plurality of images by calculation for the purpose of expanding the dynamic range of the solid-state imaging device, reducing noise, suppressing camera shake at the time of hand-held shooting Has been proposed.

  For example, a method for generating a wide-angle image by moving an imaging device and performing continuous shooting while shifting a shooting range, performing matching (alignment) of a plurality of obtained images, and combining so that subjects are continuous. Has been proposed (see Patent Document 1). There is also proposed a method of capturing while changing exposure alternately, and combining the obtained images to obtain an image with an expanded dynamic range (see Patent Document 2).

  Hereinafter, such a method of combining a plurality of images so as to enlarge the angle of view is referred to as “wide-angle image combining”, and an image obtained by wide-angle image combining is referred to as “wide-angle combined image”. I assume.

JP, 2013-58931, A JP 11-65004 A

  However, in the technology described in Patent Document 1, the positioning coordinates are specified for a plurality of photographed images, and the wide-angle composite image is generated by joining the images at the specified coordinate positions. Therefore, although it is possible to generate a wide-angle composite image in which a very wide angle of view can be obtained, the dynamic range of the subject becomes wide, which may result in an image including overexposure and overexposure.

  On the other hand, in Patent Document 2, a wide-angle composite image in which images captured at the same exposure value are connected is generated, and then wide-angle composite images having different exposure values are combined. In this case, a large number of memories are required for storing intermediate images such as wide-angle composite images generated from images with the same exposure value, which are created during wide-angle image composition. As a result, in an inexpensive image processing apparatus that does not have a large amount of memory, there is a possibility that the image combining speed may be slow or the image combining may not be performed.

  The present invention provides a technology that reduces the amount of memory required for temporary image storage at the time of image composition and enables the generation of a wide-angle composite image in which the occurrence of overexposure and overexposure is suppressed. The purpose is

  The image processing apparatus according to the present invention is an acquisition unit for acquiring a plurality of images photographed continuously while changing the imaging range while repeating imaging with a plurality of different exposure values, and a plurality of images acquired by the acquisition unit. A detection unit that detects an overlapping area in which objects overlap between adjacent images, and a generation unit that generates a combined area in which the overlapping area detected by the detecting unit is not exceeded a predetermined number of times; Calculation means for calculating the composition ratio of each of the plurality of images acquired by the acquisition means based on the exposure value and signal value at the time of photographing of each of the plurality of images, and the calculation means It is characterized by comprising: generation means for generating a wide-angle synthesized image by performing image synthesis in the synthesis area generated by the generation means at the synthesis ratio.

  According to the present invention, it is possible to reduce the amount of memory required to store an intermediate image when generating a wide-angle composite image, and to generate a wide-angle composite image in which the occurrence of overexposure and blackout is suppressed. it can.

FIG. 1 is a block diagram showing a schematic configuration of a digital camera according to an embodiment of the present invention. It is a figure which shows the coordinate system set with respect to the digital camera of FIG. It is a block diagram which shows the structure of the image processing part with which the digital camera of FIG. 1 is provided. It is a flowchart which shows an example of the process sequence in the case of performing wide-angle image synthetic | combination by the digital camera of FIG. It is a figure which shows typically the method of panning imaging | photography of FIG.4 S401. FIG. 7 is a view showing an example of an actual photographed image obtained by the panning photographing in step S401 in FIG. 4, an example of a corrected image obtained by correcting the actual photographed image, and an example of a wide-angle composite image generated from the corrected image. is there. It is a figure explaining the identification method of the duplication area | region of step S403 of FIG. 4 using the correction | amendment image shown in FIG.6 (b). FIG. 5 is a detailed flowchart of composite map creation processing in step S404 of FIG. 4; FIG. It is a figure which illustrates typically the preparation procedure of the synthetic | combination frequency map of FIG.8 S801. FIG. 5 is a view showing an example of a composition ratio of a low exposure image, a proper exposure image, and a high exposure image used in the process of creating a composite map in step S404 in FIG. 4;

  Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. Here, a so-called digital camera is taken as an image processing apparatus according to the present invention. However, the image processing apparatus according to the present invention is not limited to a digital camera, and may be, for example, a personal computer (for example, a personal computer) as long as it can process a captured image acquired by a camera (imaging unit). PC) etc. may be used. That is, the imaging unit and the image processing unit may be integrated as in a digital camera, or may be separate as in a combination of an imaging unit and a personal computer.

  When the imaging unit and the image processing unit are integrated, the image processing apparatus according to the present invention is not limited to a device specialized for photographing such as a digital camera or digital video camera. It may be a communication terminal, a personal computer, a game machine or the like. In addition, when the imaging means and the image processing means are separate bodies, the transmission of the photographed image from the imaging means to the image processing means is regardless of whether it is wireless communication or wired communication, and direct connection or via a network. It may or may not be connected.

<Schematic structure of digital camera>
FIG. 1 is a block diagram showing a schematic configuration of a digital camera 100 according to an embodiment of the present invention. The digital camera 100 includes a system control unit 101 (hereinafter referred to as “control unit 101”), a ROM 102, a RAM 103, an optical system 104, an imaging unit 105, an A / D conversion unit 106, an image processing unit 107, a storage medium 108, and a detection unit. 110 and a display unit 111. Communication between these blocks is performed via the bus 109.

  The control unit 101 is, for example, a CPU, reads out an operation program of each block constituting the digital camera 100 from the ROM 102, and develops the program on the RAM 103, thereby integrally controlling the operation of each block. The ROM 102 is a rewritable non-volatile memory, and stores an operation program of each block constituting the digital camera 100, parameters necessary for the operation of each block, and the like. The RAM 103 is a rewritable volatile memory, and is used as a work area of the CPU and as a temporary storage area of data output by the operation of each block constituting the digital camera 100.

  The optical system 104 takes in light from a subject, and forms an optical image (subject image) on an imaging element such as a CCD sensor or a CMOS sensor included in the imaging unit 105. The optical system 104 also includes an aperture that adjusts the light amount at the time of shooting by adjusting the aperture diameter of the optical system 104. The imaging device included in the imaging unit 105 photoelectrically converts an optical image, and outputs an analog electric signal to the A / D conversion unit 106. The A / D converter 106 A / D converts the analog electrical signal supplied from the imaging unit 105, and outputs the image data, which is a generated digital signal, to the RAM 103 for storage.

  An image processing unit 107 performs necessary image processing such as white balance processing and gamma correction processing on the image data stored in the RAM 103. The storage medium 108 is, for example, a removable memory card or the like, and is image data of an image processed by the image processing unit 107 stored in the RAM 103 or digital A / D converted by the A / D conversion unit 106. A signal (image signal) etc. is stored.

  The detection unit 110 is configured of a known gyro sensor, an acceleration sensor, or the like, and an output signal from the detection unit 110 is transmitted to the control unit 101. The control unit 101 detects the attitude or movement (angle, acceleration, movement amount, etc.) of the digital camera 100 based on the output signal acquired from the detection unit 110.

  FIG. 2 is a view showing a coordinate system set for the digital camera 100 in the present embodiment. The optical axis direction of the optical system 104 of the digital camera 100 is defined as the Z direction, and an X axis and a Y axis orthogonal to the Z axis are defined in the main body of the digital camera 100. Based on the output signal obtained from the detection unit 110, the control unit 101 calculates the angles (α, β, γ) and the positions (X, Y, Z) shown in FIG. The position and the relative value from the reference value are calculated. For example, the control unit 101 determines angles (0 °, 0 °, 0 °) and positions (0, 0, 0) using the detection result of the detection unit 110 synchronized with the first imaging as a reference value. After calculating the reference value, the control unit 101 calculates a relative value, which is a relative movement amount from the reference value, to detect whether panning imaging, which will be described later, is performed as expected.

  The display unit 111 includes a display such as a liquid crystal display (LCD) that displays an image and various information, and a display control circuit that controls image display on the display. The display image data stored in the RAM 103 is converted into an analog signal by the D / A converter included in the display control circuit and displayed on the display.

<Configuration of Image Processing Unit>
FIG. 3 is a block diagram showing the configuration of the image processing unit 107. As shown in FIG. The image processing unit 107 includes an overlapping area extraction unit 301, a synthesis map generation unit 302, and a wide-angle image synthesis unit 303. The overlapping area extracting unit 301 specifies an overlapping area where the subject overlaps between the N−1th image 312 and the Nth image 313 adjacent to each other as the N−1th overlapping area. Here, "N" is an integer of 2 or more, and this definition is always applied in the following description.

  The composite map creation unit 302 performs the N-th time based on the composite map 311 (details will be described later) when the wide-angle image composition is performed N-1 times and the N-th overlap area output from the overlap area extraction unit 301. Calculate the composite map. The wide-angle image combining unit 303 performs combining processing of the wide-angle combined image 314 obtained by performing image combining N-1 times and the N-th image 313 on the basis of the output result of the combined map creation unit 302. A composite image 315 is generated. The first composite map and the first wide-angle composite image are created based on the first overlap area (the first and second captured images).

First Embodiment
FIG. 4 is a flowchart showing an example of the processing procedure when wide-angle image synthesis is performed by the digital camera 100. In the present embodiment, processing for generating a wide-angle composite image with improved overexposure and overexposure is described.

The process illustrated in the flowchart of FIG. 4 is started by the photographing unit being instructed to the photographing unit 101 by pressing the release button provided in the operation unit (not shown) of the digital camera 100 or the like. Here, with the photographer holding the digital camera 100, panning is performed by rotating from left to right in the horizontal direction (about the Y axis in FIG. 2) when viewed from the photographer, that is, changing the imaging range performed continuously shooting Te, it shall be the carrying out the wide-angle image synthesized with the photographic images obtained. In addition, shooting is performed by repeatedly changing the three exposure values at proper exposure, low exposure, and high exposure, and wide-angle image composition is performed using the obtained captured image. Note that the direction of panning imaging may be another direction, for example, the reverse horizontal direction (from right to left) or the vertical direction (from bottom to top or from top to bottom).

  Each process shown in the flowchart of FIG. 4 is realized by the control unit 101 developing a program read from the ROM 102 in the RAM 103 and controlling the operation of each unit constituting the digital camera 100.

  In step S401, the control unit 101 performs panning imaging with three different exposure values. For example, evaluation photometry is performed to determine the proper exposure value, and the determined proper exposure value is changed by a predetermined number of steps to determine the respective exposure values for low exposure shooting and high exposure shooting. Then, the control unit 101 instructs the photographer the shooting direction, and the photographer performs shooting while panning the digital camera 100 in the direction instructed by the control unit 101. At this time, shooting is performed in the order of the proper exposure value, the low exposure value, the high exposure value, the proper exposure value, the low exposure value, and so on. Note that the panning shooting in step S401 will be described later with reference to FIGS. 5 and 6.

  At this time, the control unit 101 detects the attitude or movement of the digital camera 100 based on the output signal from the detection unit 110, and based on the detection result, whether or not panning imaging expected by the digital camera 100 is performed Determine if For example, when the detection unit 110 detects an angular velocity or an acceleration, the control unit 101 calculates the angle and the position of the digital camera 100 by integrating the detected numerical values.

  In step S402, the control unit 101 determines whether or not there remains a captured image on which the processing (wide-angle image combination) of steps S403 to S405 is performed on the captured image obtained by the panning imaging in step S401. If no photographed image remains (NO in S402), the CPU 101 ends this processing. If a photographed image remains (YES in S402), the CPU 101 advances the process to step S403.

  In step S403, the control unit 101 causes the overlapping area extraction unit 301 to overlap the N-1 th and N-th captured images with each other in the captured image obtained by the panning imaging in step S401. Identify as the second overlap area. The processing in steps S403 to S405 is performed using a corrected image obtained by correcting the signal level of the photographed image, not the photographed image acquired by the panning photographing in step S401, and the details thereof will be described with reference to FIGS. We will refer to it later.

  In step S404, the control unit 101 determines in advance each of the N-1th combination map, the N-th overlap area (the N + 1th corrected image), and the exposure value of the photographed image by the combination map generation unit 302. An N-th composite map is created based on the composite ratio table. As described above, the first combined map is created based on the first overlapping area (the first and second captured images). The details of the method of creating the N-th composite map will be described later with reference to FIGS. 8 to 10.

  In step S405, the control unit 101 causes the wide-angle image combining unit 303 to select N based on the N-1th wide-angle composite image, the N-th overlapping area (the N + 1th corrected image), and the N-th composite map. Perform the second wide-angle image composition processing. After the end of step S405, the control unit 101 returns the process to step S402. The control unit 101 generates the final wide-angle composite image by repeatedly performing the processing of steps S402 to S405 on (corrected images of) all captured images obtained by panning imaging.

  FIG. 5 is a diagram schematically illustrating panning imaging in step S401. As shown in FIG. 5A, shooting is started with the digital camera 100 directed in a predetermined direction, and a shooting start image 501 is shot. Shooting is continued at a predetermined shutter speed while rotating the digital camera 100 in the rotation direction 504 (clockwise in FIG. 5 (about Y axis in FIG. 2) about the rotation center 503 of the digital camera 100, and the shooting end image 502 is shot. When it is done, the panning shooting ends.

  As shown in FIG. 5B, the shooting start image 501 and the shooting end image 502 each have a shooting margin area. These shooting margin areas are not included in the angle of view of the finally output wide-angle composite image. Therefore, the angle of view 505 of the finally output wide-angle composite image is a range from the area excluding the imaging margin area in the imaging start image 501 to the area excluding the imaging margin area in the imaging end image 502.

  FIG. 6A is a view showing an example of an actual image obtained by the panning shooting in step S401. Here, a shooting margin area is provided on the left side of the building so that the left side of the building is photographed with a margin toward the building as a subject, so that the right side of the building is photographed with a margin The imaging margin area is provided on the right side of the building to complete the imaging.

  In the present embodiment, as shown in FIG. 6A, it is assumed that seven photographed images are actually photographed. The proper exposure images 601, 604, and 607 are images captured with the proper exposure value calculated by the AE control of the digital camera 100. The low exposure images 602 and 605 are images taken at an exposure value lower than the appropriate exposure value. The high exposure images 603 and 606 are images taken at an exposure value higher than the appropriate exposure value. As described above, in the panning shooting in step S401, the shooting is repeatedly performed by sequentially changing the three exposure values. In the following description, the proper exposure images 601, 604 and 607, the low exposure images 602 and 605, and the high exposure images 603 and 606 will be collectively referred to as photographed images 601 to 607 as appropriate.

  In photographed images having different exposure values, the signal level (the output signal level from the imaging device) differs depending on the exposure difference. Therefore, correction processing is performed on the actual captured images 601 to 607 obtained by panning imaging to adjust the signal level of the same subject region. In this correction method, a method of matching the brightness by multiplying the signal value (brightness value) of the photographed images 601 to 607 by a gain proportional to the exposure value, a method of correcting the brightness of the photographed image by gamma processing, etc. Is not particularly limited.

  FIG. 6B schematically shows corrected images 611 to 617 after correcting the brightness to the photographed images 601 to 607 shown in FIG. 6A. 6C shows an example of the wide-angle composite image 620 finally generated, and the wide-angle composite image 620 is generated using the correction images 611 to 617 shown in FIG. 6B. .

  The process of specifying an overlapping area between adjacent photographed images in step S403 will be described with reference to FIG. Here, not the actual photographed images 601 to 607 shown in FIG. 6A but the corrected images 611 to 617 shown in FIG. 6B are used to specify the overlapping area between the adjacent photographed images.

  FIG. 7 is a diagram for explaining a method of specifying overlapping regions between photographed images adjacent to each other in the photographed images 601 to 607 using the corrected images 611 to 617. In the identification process between adjacent overlapping areas, overlapping areas of objects in the correction images adjacent to each other are calculated for the correction images 611 to 617. Therefore, in FIG. 7, with the movement amount of the corrected images 612 to 617 taken on the horizontal axis with the corrected image 611 as a reference, the same part of the subject as the corrected images 612 to 617 is one before the corrected image. Aligned to overlap.

  A general image registration method can be used for the registration method of the adjacent correction image. For example, a method of dividing a corrected image into small blocks of an arbitrary size, calculating corresponding points at which SAD (Sum of Absolute Difference) of an image signal (such as luminance) becomes minimum for each small block, and calculating a motion vector It can be used.

  In the alignment processing of the correction images 611 and 612, a first overlapping area 701 in which the start coordinate is SX1 and the end coordinate is EX1 is specified. Similarly, in the alignment processing of the correction images 612 and 613, a second overlap area 702 in which the start coordinate is SX2 and the end coordinate is EX2 is specified. As described above, by sequentially calculating feature amounts such as movement amounts (coordinates) of the correction images 611 to 617 by alignment processing of two adjacent correction images, the first to sixth overlapping regions 701 to 706 are obtained. Can be identified. In step S403, for the adjacent correction image subjected to the alignment process simultaneously with the identification process of the first to sixth overlap regions 701 to 706, the correction image after the time matches the position of the previous correction image. Perform geometric transformation processing as follows.

  Next, the process of creating a composite map in step S404 will be described with reference to FIGS. 8 to 10. FIG. 8 is a flowchart of the process of creating a composite map in step S404. In this embodiment, expansion of the dynamic range is realized by combining corrected images of three photographed images of a low exposure image, a proper exposure image, and a high exposure image in the same area. Therefore, up to two times of synthesis are performed in the same area. In other words, three or more syntheses are not performed in the same region.

  In step S801, the composite map creation unit 302 creates a composite frequency map. FIG. 9 is a diagram for schematically explaining the creation procedure of the number-of-combinations map. The composition frequency map indicates how many times a certain area is composited in each process of sequentially superimposing a plurality of photographed images, and FIG. 6 (b) specified in step S403 for creating the composition frequency map The overlapping areas 701 to 706 of the corrected images 611 to 617 shown in FIG. Note that FIG. 9 shows first to third and sixth synthetic frequency maps 901 to 903 and 906, and the fourth and fifth synthetic frequency maps are omitted.

  In the creation of the combination number map, the overlapping regions of the overlapping regions 701 to 706 are counted, and the number of overlapping is set as the value of the number of combinations. For example, the first composition frequency map 901 is created based on the overlapping area 701 of the corrected images 611 and 622 because there is no past composition frequency map. In the first composition frequency map 901, the value of the composition frequency is "1" in the regions SX1 to EX1 of the composition frequency map 901, and the value of the composition frequency of the regions other than the regions SX1 to EX1 is "zero (0)" .

  If there is an area overlapping the overlapping area to be superimposed this time on the previously created combination frequency map when the current combination frequency map for the second and subsequent times is created, the value of the combination frequency of that region is incremented. For example, in the second combination frequency map 902 shown in FIG. 9B, there are two, the overlapping region 701 in the first combination frequency map and the overlapping region 702 to be combined this time. Therefore, the value of the number of synthesization of the regions SX2 to EX1 of the composition number map 902 is “2”, the value of the number of syntheses of the regions EX1 to EX2 is “1”, and the value of the number of syntheses of the regions EX2 to EX2 is “0”.

  In step S802, the composite map generation unit 302 determines whether there is an area exceeding the predetermined composite frequency in the composite frequency map. In the present embodiment, since it is necessary to combine the low exposure image, the proper exposure image, and the high exposure image in the same area, it is necessary to combine two times in the same area. An area in which the number of times is two is set as a combined area. Therefore, when there is an area where the number of times of synthesis is three or more in the number-of-combinations map (YES in S802), control unit 101 advances the process to step S803, and when there is no area of three or more times (NO in S802) The process advances to step S804.

In step S 803, the synthesis map generation unit 302 sets the predetermined number of times (= 2 times) as the synthesis limit number, and changes the synthesis number of the area where the synthesis number is 3 or more in the synthesis number map to the synthesis limit number. Further, an area excluding the area is calculated as a new combined area. For example, the third combination number map 903 shown in FIG. 9C includes three overlapping areas 701 to 703, and three of the overlapping areas 701 to 703 overlap in the areas SX3 to EX1. Therefore, in the third combination number map 903, for the regions SX3 to EX1, the portion where the value of the number of combinations is calculated as "3" is changed to the number of combination limit number "2", and the regions SX3 to EX1 are to be combined Exclude from As a result, a composition frequency map in which the composition frequency of the regions SX1 to EX2 is two of the limit composition frequency is created.

  In step S804, the composite map generation unit 302 calculates a composite ratio when combining a region (composite region) that does not exceed the composite limit frequency in the composite number map obtained in the processing up to step S803, and generates a composite map. . In this composite map generation, different composite ratios are used for the low exposure image, the proper exposure image, and the high exposure image.

  FIGS. 10 (a), (b) and (c) are diagrams showing examples of the composition ratio of the low exposure image, the proper exposure image, and the high exposure image, respectively, and these are all correction images used for wide angle image synthesis. 7 shows the relationship between the signal value (brightness signal value) of and the combining ratio.

  For example, when combining a low exposure image, the combining ratio of the image area to be combined is determined with reference to the combining ratio 1001 of FIG. 10A from the signal value of the corrected image of the low exposure image to be combined, Generate a composite map. The combining ratio 1001 is configured to lower the combining ratio in an area (pixel) where the signal value is small and to increase the combining ratio in an area (pixel) where the signal value is large. This can suppress the occurrence of blackening in the wide-angle composite image.

  On the other hand, similarly, in the case of combining the proper exposure image, the combining ratio 1002 in FIG. 10B is referred to, and in the case of combining the high exposure image, the combining ratio 1003 in FIG. The composition ratio of the image area to be composited is determined, and a composite map is generated. The combining ratio 1003 is configured to increase the combining ratio in a region (pixel) where the signal value is small and to decrease the combining ratio in a region (pixel) where the signal value is large. This can suppress whiteout in the wide-angle composite image. The composition ratio 1002 is generally set to an intermediate characteristic of the composition ratios 1001 and 1003.

  The synthesis ratios 1001 to 1003 are not limited to those shown in FIGS. 10 (a) to 10 (c). For example, with regard to the composition ratio 1002 of the proper exposure image, the composition ratio may be 0 on the low luminance side and the high luminance side, and may be adjusted to 100 in the middle. In addition, the combination ratio of each of the low exposure image and the high exposure image may be adjusted based on the luminance distribution in the proper exposure image. For example, when there are many areas where the brightness value is small in the proper exposure image, the gradient portion of the combination ratio 1001 is further shifted to the high brightness side, and the gradient portion of the combination ratio 1003 is also shifted to the high brightness side. May be

  The final combined map is obtained by adding the combined ratio to the sixth combined number map 906 in FIG. 9D. In addition, since there is a possibility that the number of times of combining of the overlapping area does not reach a predetermined number of times (twice), the area of the upper, lower, left and right ends of the final combining map is a shooting margin area in this embodiment. As a result, the angle of view of the final wide-angle composite image 620 is shown by the regions SX2 to EX5 shown in FIG.

  Specifically, in the wide-angle image combining by the wide-angle image combining unit 303 in step S405, a wide-angle combined image obtained by performing the N-1th wide-angle image combining based on the N-th combined map and an N-th wide-angle image The N + 1-th captured image, which is a captured image to be combined, is combined. The N-th wide-angle composite image is calculated by the following composition formula using the (N-1) -th wide-angle composite image, the photographed image used for the N-th wide-angle image composition, and the N-th composite map.

  Note that (x, y) indicates the position on the image coordinates. Further, since the photographed image (corrected image) obtained by the panning photographing is synthesized, an area where the area of one image does not exist is generated, and the area is output from the image where the area exists. Do the processing.

  As described above, according to the present embodiment, a wide-angle composite image in which the dynamic range is expanded is improved by combining the images captured with a plurality of different exposure values including proper exposure, thereby improving the blackout and overexposure. Can be generated. In addition, in order to generate a wide-angle composite image while sequentially combining overlapping areas in a captured image, as in the prior art, a method of combining the images after combining images with different exposure values in advance is more effective than image combining. Can reduce the amount of memory required for

Second Embodiment
In the first embodiment, although the combination limit number of times in the creation of the combination number map is two times, that is, the upper limit of the number of combining of the photographed combined image in the combining map is two, the upper limit of the number of combining is not limited thereto. The number of times may be increased (three times or more). That is, as the number of changes in exposure value is increased and the number of captured images is increased, a wider dynamic range can be realized, and a composite image in which overexposure and overexposure can be suppressed can be obtained. Therefore, wide-angle image composition may be performed by setting the number of exposure steps in multiple stages without limiting to three images (three exposure steps) of low exposure, proper exposure, and high exposure. In that case, the upper limit of the number of times of combination may be determined according to the number of times of combination (the number of images to be combined) in a predetermined area of the image.

  For example, assuming that the upper limit of the number of combining is Nlim times, the start coordinate of the overlapping area between adjacent images in the Nth combining number map is SX (N), and the end coordinate is EX (N), the number of combining reaches the upper limit The region is generated for the first time at the time of Nlim synthesis. Therefore, in the creation of the synthesis map up to Na times (1 ≦ Na ≦ Nlim), the synthesis map is updated in the overlapping regions SX (Na) to EX (Na) between the adjacent images. Then, in the creation of the composition frequency map after the Nb-th (Nb> Nlim), the composition frequency map is updated in the region of EX (Nb−Nlim) to EX (Nb) with respect to the overlapping region between adjacent images. change.

  For example, assuming that the upper limit of the number of times of synthesis is three, the region where the number of times of synthesis reaches the upper limit occurs for the first time at the time of creation of the third time combination map. Therefore, the synthesis map is sequentially updated in the regions SX (Na) to EX (Na) (1 ≦ Na ≦ 3) until the third generation of the synthesis map. In the creation of the fourth and subsequent synthetic maps (Nb> 3), the synthetic maps are sequentially updated in the regions EX (Nb-3) to EX (Nb) (Nb> 3). Thus, the image processing according to the present invention can be performed even if the upper limit of the number of times of combining the same area is changed.

Third Embodiment
In the first embodiment, the overlapping area between adjacent captured images is specified from the feature amount obtained by performing the alignment processing on the entire captured image. Here, the overlapping area is further changed according to the amount of aberration of the lens constituting the optical system 104 and the focal length at the time of photographing with respect to the overlapping area obtained by the alignment processing of the entire photographed image. May be

  For example, depending on the amount of aberration of the lens and the focal distance at the time of shooting, in the area of the image far from the center of the photographed image, even if the alignment processing is performed between adjacent images subjected to panning photography, the distortion of the image is correct. There is a possibility that the position does not match. Therefore, with respect to the overlap area calculated from the feature amount obtained by the alignment processing, a process of setting the overlap area to be narrow according to the magnitude of the distortion of the captured image may be performed. In addition, conversely, when processing is performed to improve the synthesis accuracy of the wide-angle composite image such as distortion correction or cylindrical conversion, the overlap region calculated from the feature amount obtained by the alignment processing can be used to the maximum extent. It is also possible to set in the

Fourth Embodiment
In the first embodiment, the imaging margin area not used for the finally generated wide-angle composite image is set as the imaging image at the imaging start time and the imaging end time. Not limited to this, processing for changing the shooting margin area may be performed according to the upper limit of the number of times of synthesis in the synthesis map. As the number of times of synthesis limit increases, the number of captured images to be synthesized increases until reaching the number of times of synthesis limit, so the angle of view that can be used for wide angle image synthesis becomes narrower. Therefore, when the number of times of composition limit is large, the imaging margin area is widely controlled.

Fifth Embodiment
In the first embodiment, the alignment processing is performed on the adjacent captured images, and the overlapping area between the adjacent images is specified. However, the present invention is not limited to this, and the panning position or angle of the digital camera 100 may be specified using the detection result of the detection unit 110 to obtain the overlapping area between the photographed images. That is, by calculating the angle and position of the digital camera 100 calculated from the detection signal of the detection unit 110 as a relative value from the reference value, it is possible to specify an overlapping area between adjacent photographed images. By using this method, when photographing an object having low contrast or a repetitive pattern, it is possible to specify a correct overlapping area as compared to the case of obtaining an overlapping area by image analysis such as the motion vector method. it can. Note that the process of specifying the overlap area between the photographed images by image analysis may be used in combination with the process of specifying the overlap area between the photographed images from the detection signal of the detection unit 110. For example, the overlap area between the captured images obtained by the image analysis may be corrected using the detection signal of the detection unit 110.

Sixth Embodiment
In the first embodiment, although the number of times of synthesis of the synthesis map is fixed in advance, the present invention is not limited to this, and the number of times of synthesis of the synthesis map may be changed according to the panning speed at the time of shooting. . For example, when the panning speed is fast, the number of captured images for constructing the angle of view constituting the desired wide-angle composite image decreases, so as the panning speed becomes faster, the number of times of synthesis limit of the synthesis map is reduced. Perform control processing.

  For example, a case will be considered where the exposure is changed sequentially by ± 1 step around the proper exposure image (exposure correction is 0 step), a total of 5 photographed images are taken, and a wide-angle composite image is synthesized. In this case, for example, assuming that the angle of view of one photographed image is 90 degrees, each rotational angle smaller than 22.5 degrees in order to make an overlap area exist in five photographed images. It is necessary to take one shot. Therefore, in the case of performing one shooting every angle larger than 22.5 degrees, the number of combined limits is reduced. For example, if the angle of view of one shot image is 90 degrees and shooting is performed by 30 degrees, the fourth shot image has no overlapping area with the first shot image, so a maximum of two times The number of times of composition is (3 sheets).

  In addition, it detects whether there is an overlapping area between adjacent photographed images because of the high panning speed, using a detection signal from the detection unit 110, and if there is no overlapping area, issues a warning or wide-angle image The composition processing may be configured to be discontinued.

Seventh Embodiment
In the first embodiment, a wide-angle composite image is generated only from the composite region in which the number of combinations has reached two, with the number of combination limits in creating the composite map being two. However, the present invention is not limited to this, and in order to widen the angle of view, an area of the image which has not reached the synthesis limit number (area where the synthesis number is 0 or 1) may be added to the wide angle synthesized image.

  In this case, it is preferable to adjust the composition ratio of the area (area to be added) of the image which has not reached the composition limit number in accordance with the signal value (luminance value) of the image. In addition, it is determined whether whiteout or black crushing is included in the area of the image to be added, and when whiteout or black crushing is not included, an area not included in the wide-angle composite image is added to the wide-angle composite image. It is preferred to carry out the synthesis. In the case of adding an area of an image (for example, an image captured last) the composition number is 0, an image captured with a proper exposure is used as the image.

<Other Embodiments>
Although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various embodiments within the scope of the present invention are also included in the present invention. included. Furthermore, each embodiment mentioned above shows only one embodiment of the present invention, and it is also possible to combine each embodiment suitably.

  The present invention is also realized by performing the following processing. That is, software (program) for realizing the functions of the above-described embodiment is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU or the like) of the system or apparatus reads out a program code. It is a process to execute. In this case, the program and the storage medium storing the program constitute the present invention.

100 digital camera 101 system control unit 102 ROM
DESCRIPTION OF SYMBOLS 104 Optical system 105 Imaging part 107 Image processing part 110 Detection part 901, 902 1st and 2nd composition frequency map 903, 906 3rd and 6th composition frequency map

Claims (12)

An acquisition unit that acquires a plurality of images captured by continuously changing the imaging range while repeating imaging with a plurality of different exposure values;
A detection unit configured to detect an overlapping area in which a subject overlaps between adjacent images in the plurality of images acquired by the acquisition unit;
Creating means for creating a combined area in which the overlapping area detected by the detecting means is overlapped so as not to exceed a predetermined number of times;
Calculating means for calculating the composition ratio of each of the plurality of images acquired by the acquisition means based on the exposure value and the signal value at the time of photographing each of the plurality of images;
An image processing apparatus comprising: generating means for generating a wide-angle composite image by performing image combining in the combining area created by the creating means at the combining ratio calculated by the calculating means.   The calculation means is a signal of an area not having reached the predetermined number of times when an area not included in the combination area and in which the number of times the overlapping area is overlapped has not reached the predetermined number of times is added to the wide-angle composite image. The image processing apparatus according to claim 1, wherein the combining ratio is adjusted in accordance with a value. The plurality of images include images taken respectively at low exposure and high exposure for photographing at the proper exposure,
The calculation means lowers the combining ratio of the image captured at the low exposure in a region where the signal value is small and increases in a region where the signal value is large, and the combining ratio of the image captured at the high exposure is The image processing apparatus according to claim 1, wherein the image processing apparatus is increased in a small area and lowered in a large signal value area.   The calculation means is characterized in that the synthesis ratio of each of the image photographed at the low exposure and the image photographed at the high exposure is adjusted based on a luminance distribution of the image photographed at the proper exposure. The image processing apparatus according to claim 3, wherein   The image processing apparatus according to any one of claims 1 to 4, wherein the predetermined number of times is set in accordance with the number of steps of the different exposure values used for capturing the plurality of images.   The image processing apparatus according to claim 5, wherein the predetermined number of times is set so that the composite region is created by superimposing images different in the exposure value one by one.   When the last image of the plurality of images is an image photographed with proper exposure and is an image without whiteout and blackout, the generation means may perform the wide-angle combined image with the last image. The image processing apparatus according to any one of claims 1 to 6, wherein the image processing apparatus is included.   The said detection part correct | amends the said duplication area | region based on the focal distance and the amount of aberrations of the lens of the imaging means which image | photographed the said several image, It is characterized by the above-mentioned. Image processing device.   The image according to any one of claims 1 to 8, wherein the generation unit changes the predetermined number in accordance with a speed of movement of the imaging device when capturing the plurality of images. Processing unit. An acquisition unit that acquires a plurality of images captured by continuously changing the imaging range while repeating imaging with a plurality of different exposure values;
Superposition means for superposing images of different exposure values one by one so that a plurality of images acquired by the acquisition means are overlapped between adjacent images.
And a generation unit configured to generate a wide-angle composite image by performing image combination in a region where images having different exposure values overlap each other by the superposition unit.
The image is captured with the same exposure value so that, even if there is a subject range common to the images captured with the same exposure value, the overlay unit does not overlap the common subject range twice. What is claimed is: 1. An image processing apparatus comprising: An image processing method for combining a plurality of captured images by continuously changing an imaging range while repeating imaging with a plurality of different exposure values,
An acquisition step of acquiring the plurality of images;
A detection step of sequentially detecting an overlapping area where objects overlap between adjacent images, for the plurality of images acquired in the acquiring step;
A creating step of creating a combined region in which the overlapping region detected in the detecting step is overlapped so as not to exceed a predetermined number of times;
Calculating the composition ratio of each of the plurality of images acquired in the acquisition step based on the exposure value and the signal value at the time of capturing each of the plurality of images;
An image processing method comprising: generating a wide-angle composite image by performing image composition in the composition area created in the creating step using the composition ratio calculated in the calculating step; .   A program causing a computer to execute the image processing method according to claim 11.