JP6245992B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method for obtaining composite image data by comparing and selecting pixel data corresponding to pixel positions in a plurality of image data according to values.

  Conventionally, when synthesizing a plurality of images, comparatively bright composition using bright pixels and comparative dark composition using dark pixels among pixels at the same pixel (pixel) position are known. .

  An imaging apparatus has been proposed that can perform comparatively bright composite photography and comparative dark composite photography while observing a composite image with a live view.

  For example, Japanese Patent Laid-Open No. 2013-62740 discloses that in an imaging apparatus including an imaging sensor, a storage unit, and a display unit, exposure and readout of the imaging sensor are repeatedly performed at a predetermined cycle to sequentially acquire a plurality of images. The comparatively bright combination process is performed using the interval photographing means for performing the interval photographing process, and the first image and the second image among the plurality of images sequentially acquired by the interval photographing process. The processing result is stored in the storage unit as a composite image, and the third and subsequent images are subjected to comparatively bright composite processing using the image and the composite image stored in the storage unit, and the processing result is set as a new composite image. The comparatively bright combination means for repeatedly performing the process stored in the storage unit and the combination stored in the storage unit without stopping the interval shooting process when the first operation is performed. Imaging apparatus comprising: a composite image display means for displaying an image on the display unit, a is described.

JP2013-62740A

  For example, when performing comparatively bright combination, since the brightest pixel among a plurality of pixels at the same pixel position in a plurality of images taken sequentially is adopted and used for a combined image, for example, the subject is fireworks. In this case, even if a high-luminance firework and a low-luminance firework sequentially launched at the same position are photographed and combined with comparatively bright light, the low-luminance firework does not remain in the composite image. On the other hand, when performing comparative dark combination, a subject with high luminance at the same position does not remain in the combined image.

  The present invention has been made in view of the above circumstances, and when a composite image is obtained by combining a plurality of images according to brightness, a desired subject is left in the composite image regardless of the order of brightness. An object of the present invention is to provide an image processing apparatus and an image processing method capable of performing the above.

  An image processing apparatus according to an aspect of the present invention provides a composite image data as a certain image data and a comparison image data as another image data, and the composite image data and the comparison image data. The pixel data corresponding to the pixel positions in and are compared with each other, and only one of the larger and smaller values is selected, and the comparison-selected pixel data is replaced with new pixel data at the corresponding pixel position. A comparison / synthesis processing unit that repeatedly performs cumulative comparison / synthesis processing for reconstructing the combined image data each time the comparison image data is updated, and an image data selection unit that selects exceptional image data from the image data group When the pixel data constituting the exceptional image data selects a larger value in the comparison and selection, a predetermined value is used. When the threshold level is equal to or higher than the threshold level, and the smaller value is selected in the comparison and selection, when the threshold level is equal to or lower than the predetermined threshold level, the pixel is regarded as an exception to the cumulative comparison and synthesis process, An exception processing unit that performs an exception process for causing the comparison and synthesis processing unit to reconstruct the composite image data as new pixel data of a position.

  In an image processing method according to an aspect of the present invention, in a group of image data to be combined, certain image data is used as combined image data, and other image data is used as comparison image data. The pixel data corresponding to the pixel positions in and are compared with each other, and only one of the larger and smaller values is selected, and the comparison-selected pixel data is replaced with new pixel data at the corresponding pixel position. A comparison and synthesis processing step of repeatedly performing cumulative comparison and synthesis processing for reconstructing the composite image data every time the comparison image data is updated, and an image data selection step of selecting exceptional image data from the image data group When the pixel data constituting the exceptional image data selects the larger value in the comparison and selection, When the threshold value is equal to or higher than the predetermined threshold level, and the smaller value is selected in the comparison selection, when the threshold value is equal to or lower than the predetermined threshold level, the corresponding pixel is an exception to the cumulative comparison and synthesis process. And an exception processing step for performing an exception process in which the composite image data is reconstructed by the comparison and synthesis processing step using the data as new pixel data at the corresponding pixel position.

  According to the image processing apparatus and the image processing method of the present invention, when a composite image is obtained by combining a plurality of images according to brightness, a desired subject is left in the composite image regardless of the order of brightness. Is possible.

1 is a block diagram illustrating a configuration of an imaging apparatus according to Embodiment 1 of the present invention. FIG. 3 is a diagram showing a first image picked up by the image pickup apparatus in the first embodiment. FIG. 4 is a diagram illustrating a second image captured by the imaging apparatus in the first embodiment. FIG. 6 is a diagram illustrating a third image captured by the imaging apparatus in the first embodiment. FIG. 6 is a diagram showing a fourth image picked up by the image pickup apparatus in the first embodiment. The figure which shows the 1st synthetic | combination image obtained by carrying out comparative bright combination of the 1st image and the 2nd image in the said Embodiment 1. FIG. The figure which shows the 2nd synthetic | combination image obtained by carrying out comparative bright synthesis | combination with the 1st synthetic | combination image and the 3rd image in the said Embodiment 1. FIG. The figure which shows the 3rd synthesized image obtained by carrying out comparatively bright synthesis | combination with the 2nd synthesized image and the 4th image in the said Embodiment 1. FIG. The figure which shows the 2nd synthetic | combination image obtained by carrying out the special comparative bright synthesis | combination with the 1st synthetic | combination image by making the 3rd image into an exceptional image in the said Embodiment 1. In the said Embodiment 1, the figure which shows the 3rd synthetic | combination image obtained by carrying out special comparative bright synthesis | combination with the 4th image and the 2nd synthetic | combination image, making the 3rd image into the exceptional image. 5 is a flowchart showing a captured image composition process in the first embodiment. 12 is a flowchart showing a process of comparatively bright combination in step S5 of FIG. 11 of the first embodiment. 12 is a flowchart showing special comparative bright combination processing in step S9 of FIG. 11 of the first embodiment. 4 is a diagram illustrating an example of a histogram of data read from an OB area of the image sensor in the first embodiment. FIG. 4 is a timing chart illustrating an example in which an exceptional image selection operation is performed while observing an image displayed on a display unit in the first embodiment. 5 is a timing chart showing an example of performing an exceptional image selection operation while observing an actual subject in the first embodiment. The flowchart which shows the process of the special comparative bright combination in Embodiment 2 of this invention. In the said Embodiment 2, the figure which shows the example of a display of the image in the display part before an exceptional image is selected. In the said Embodiment 2, the figure which shows the example of a display of the image in a display part after an exceptional image is selected. FIG. 6 is a diagram illustrating an example of a gradation conversion curve used in comparatively bright combination according to the second embodiment. FIG. 6 is a diagram illustrating an example of a gradation conversion curve used in comparative dark combining according to the second embodiment. In the said Embodiment 2, the figure which shows an example when setting a threshold level manually with a threshold level setting bar, referring a brightness | luminance histogram. FIG. 23 is a diagram showing an example of a luminance histogram when gradation conversion is performed on the exceptional image data of the luminance histogram as shown in FIG. 22 using the gradation conversion curve as shown in FIG. 20 in the second embodiment. In Embodiment 3 of this invention, the figure which shows the example of a display of a display part when utilizing a portable information device as a display part at the time of performing an image synthesis with an imaging device, and an operation part. 10 is a flowchart showing recorded image composition processing in the third embodiment. In the said Embodiment 3, the flowchart which shows the process of the image composition in step S56 of FIG.

Embodiments of the present invention will be described below with reference to the drawings.
[Embodiment 1]

  1 to 16 show the first embodiment of the present invention, and FIG. 1 is a block diagram showing the configuration of the imaging apparatus.

  In the present embodiment, the image processing apparatus is, for example, an imaging apparatus 1 such as a digital camera (see also FIGS. 18 and 19 described later), or a portable information device 3 such as a smartphone or a tablet PC (described later). This is applied to a combination with FIG.

  As shown in FIG. 1, the imaging apparatus 1 includes a lens 11, a shutter 12, an imaging device 13, a signal processing unit 14, a comparison / synthesis processing unit 15, a memory unit 16, an image processing unit 17, and a recording. A unit 18, a display unit 19, an operation unit 20, and a system control unit 21 are provided.

  The lens 11 is an imaging optical system that forms an optical image of a subject on the imaging element 13. The focal position (or further focal length) of the lens 11 is controlled by the system control unit 21 via a lens drive system (not shown).

  The shutter 12 is a mechanical shutter such as a focal plane shutter, for example. The operation of the shutter 12 is controlled by the system control unit 21 via a shutter drive system (not shown). However, an element shutter that controls the exposure start timing and the exposure end timing of the image sensor 13 may be used, or a configuration in which a mechanical shutter and an element shutter are combined may be used. Here, when the element shutter is used, the shooting interval between images when a plurality of images are continuously shot can be made as small as possible, and if the combination processing is performed based on the shot images, the image moves at high speed. There is an advantage that the locus of the subject can be made continuous.

  The imaging element 13 is an imaging unit in which a plurality of pixels (pixels) are arranged in a two-dimensional manner, and an optical image is captured to generate image data. The image pickup device 13 is configured as, for example, a single-plate CMOS image pickup device (provided that a CCD image pickup device or another type of image pickup device may be used) provided with a color filter with an RGB Bayer array. The optical image of the formed subject is photoelectrically converted to generate and output image data. The imaging element 13 is provided with an OB (optical black) region that is shielded from light from the lens 11. And the image pick-up element 13 is controlled by the system control part 21 via the image pick-up element drive circuit which is not shown in figure. The system control unit 21 controls the image sensor 13 to sequentially generate a series of image data groups when a shooting operation is performed when an image composition mode for performing cumulative comparison and composition processing is set.

  The signal processing unit 14 performs processing such as noise reduction such as correlated double sampling, A / D conversion, and signal amplification on the image data output from the image sensor 13. Although FIG. 1 shows an example in which the signal processing unit 14 is provided outside the image sensor 13, a configuration in which similar processing is performed inside the image sensor 13 may be used.

  The comparison / synthesis processing unit 15 includes an exception processing unit 26, a comparison unit 27, and a synthesis unit 28.

  The exception processing unit 26, when the pixel data constituting the exceptional image data selected by the operation unit 20 selects a larger value in the comparison selection, when the value is higher than a predetermined threshold level, the value in the comparison selection is When the smaller one is selected, when it is below a predetermined threshold level, the pixel is regarded as an exception to the cumulative comparison and synthesis process, and the synthesized image data is compared and synthesized as a new pixel data at the pixel position. Exception processing to be reconfigured to 15 is performed.

  The comparison unit 27 compares the pixel values at the same pixel position in the two images.

  The synthesizing unit 28 performs a synthesizing process by replacing pixel values based on the determination result of the comparing unit 27.

  Then, the comparison / synthesis processing unit 15 uses image data in the image data group to be synthesized as image data to be synthesized image data and other image data as comparison image data, and the pixel positions in the synthesized image data and the comparison image data are the same. The corresponding pixel data is compared with each other by the comparison unit 27, and comparison selection is performed in which only one of the larger value and the smaller value is selected, and the comparison-selected pixel data is set as new pixel data at this pixel position. As described above, the cumulative comparison and synthesis process in which the synthesis image data is reconstructed by the synthesis unit 28 is repeatedly performed every time the comparison image data is updated.

  Here, the image composition mode includes a comparatively bright composition mode in which the larger value is selected in comparison and selection, and a comparative dark composition mode in which the smaller value is selected. In the following, the operation of the comparison / synthesis processing unit 15 in the comparatively bright combination mode will be mainly described. However, the operation of the comparison / synthesis processing unit 15 in the comparative dark combination mode can be considered in substantially the same manner.

  Note that when the image data processed by the comparison / synthesis processing unit 15 is Bayer image data, for example, there are two possible methods for comparing pixel values by the comparison unit 27. The first method is a method of comparing pixel values for each pixel while maintaining the pixel array of the Bayer image. In this case, pixel data of the same color in RGB can be compared at the same pixel position. When a subject in certain image data has a strong R color component and a subject in other image data has a strong B color component, it is possible to leave both subjects even if comparative bright combination is performed. On the other hand, the second method is a method of generating a luminance signal at a pixel position to be compared using pixel signals of other peripheral color components and comparing the generated luminance signals. In this case, it is possible to compare the brightness of the images without depending on the color, and it is possible to leave a brighter subject if the comparatively bright combination is performed.

  The memory unit 16 temporarily stores image data, and is composed of, for example, a DRAM (Dynamic Random Access Memory). The memory unit 16 includes a first memory area 16a and a second memory area 16b, and further includes a memory area used for other image processing, although not shown.

  In the present embodiment, the first memory area 16a first obtains the first image data in a series of image data groups sequentially obtained by performing a shooting operation when the image composition mode is set. Then, the image data is stored. Subsequently, when the second image data is obtained, it is combined with the first image data read from the first memory area 16a by the comparison and synthesis processing unit 15, and the result of the synthesis is again stored in the first memory area 16a. Overwritten and stored. Such processing is repeated when the third and subsequent image data are obtained, and the composite image data in the first memory area 16a is sequentially updated.

  The second memory area 16b stores exceptional image data selected by the operation unit 20 in the present embodiment. Here, when the second memory area 16b can store only one image, when the exceptional image data is selected a plurality of times, the image to be stored is updated every time it is selected. It can be considered as an example of processing. Further, as will be described later in a third embodiment, the second memory area 16b may be configured so that a plurality of images can be stored.

  The image processing unit 17 performs various correction processes such as shading correction, white balance processing, gamma correction, and defect correction, and a color in which three color components are arranged in one pixel from a single plate image in which only one color component exists in one pixel. Performs image processing such as demosaicing processing that composes the image. The image processing unit 17 can also record the RAW data of the composite image generated by the comparison and synthesis processing unit 15 in the recording unit 18.

  The recording unit 18 is a recording medium for recording image data, information related to the image data, and the like. As a specific example, a built-in flash memory of the imaging device 1 or a memory card configured to be detachable from the imaging device 1 is used. Is mentioned. Therefore, the recording unit 18 may not have a configuration unique to the imaging device 1.

  The display unit 19 displays an image represented by image data, various types of information related to the imaging device 1, and the like, for example, an EVF (electronic viewfinder) or a display provided on the back of the imaging device 1. It consists of panels. Further, as the display unit 19, for example, a display unit of a portable information device such as a smartphone or a tablet PC connected to the imaging device 1 via a wireless LAN or the like may be used. As will be described later, the display unit 19 displays a composite image represented by the composite image data being composited and a comparison represented by new image data (comparison image data) further combined with the composite image data. An image is displayed (see FIGS. 18 and 19).

  The operation unit 20 is used to perform various operation inputs related to the imaging apparatus 1. Specifically, the operation unit 20 is provided on an operation switch such as a push button or a dial or a display panel provided on the back of the imaging apparatus 1. Some examples include touch panels. Further, as the operation unit 20, a device provided outside the imaging device 1, for example, a remote control device (remote controller), a remote controller function of a portable information device such as a smartphone or a tablet PC described above, or the like may be used. I do not care. Such an operation unit 20 also functions as an image data selection unit that selects exceptional image data to be subjected to exception processing from the image data group.

  The system control unit 21 includes, for example, a CPU, and controls the entire imaging apparatus 1 in an integrated manner.

  Next, FIGS. 2 to 5 are diagrams illustrating examples of each image of the image data group captured by the imaging device 1, and FIG. 2 is a diagram illustrating a first image P1 captured by the imaging device 1. 3 is a diagram showing a second image P2 captured by the imaging device 1, FIG. 4 is a diagram showing a third image P3 captured by the imaging device 1, and FIG. 5 is a fourth image captured by the imaging device 1. It is a figure which shows the image P4.

The first to fourth images P1 to P4 shown in FIG. 2 to FIG. 5 are captured and acquired in this order, and the subjects Q1 to Q4 recorded in the first to fourth images P1 to P4 are obtained. For example, it shows fireworks launched in the night sky. Here, the brightness of the subjects Q1 to Q4 has the following magnitude relationship (brightness) Q2>Q1>Q3> Q4
There shall be.

  Next, FIGS. 6 to 8 show the state of the composite image when the first to fourth images P1 to P4 shown in FIGS. ing.

  FIG. 6 is a diagram showing a first synthesized image SP1 obtained by performing comparatively bright synthesis of the first image P1 and the second image P2. Since the subject Q2 is brighter than the subject Q1, only the subject Q2 remains in the first composite image SP1 where the subject Q1 and the subject Q2 overlap.

  FIG. 7 is a diagram showing a second composite image SP2 obtained by comparatively bright combining the first composite image SP1 and the third image P3. Since the subject Q3 is darker than the subjects Q1 and Q2, the portion of the subject Q3 that overlaps the subject Q1 or the subject Q2 does not remain in the second composite image SP2.

  FIG. 8 is a diagram illustrating a third synthesized image SP3 obtained by performing comparatively bright synthesis of the second synthesized image SP2 and the fourth image P4. Since the subject Q4 is darker than the subjects Q1 to Q3, the portion of the subject Q4 that overlaps the subject Q1, the subject Q2, or the subject Q3 does not remain in the third composite image SP3.

  As described above, in the comparatively bright combination process, the image combination is performed so that the brighter image portion is overlaid.

  On the other hand, FIG. 9 and FIG. 10 show the state of the composite image when the third image P3 is selected as an exceptional image.

  First, a first composite image obtained by performing comparatively bright combination of the first image P1 and the second image P2 is the same as the first composite image SP1 shown in FIG.

  Next, FIG. 9 is a diagram illustrating a second composite image SP4 obtained by performing special comparative bright combination with the first composite image SP1 using the third image P3 as an exceptional image. At this time, since the third image P3 is selected as an exceptional image, the subject Q3 (assuming that the subject Q3 is equal to or higher than the threshold level) overlaps the top regardless of the brightness order of the subjects. Thus, image synthesis is performed. Accordingly, the overlapping order from the top to the bottom of the subjects Q1 to Q3 is Q3, Q2, and Q1, and basically all the image portions of the subject Q3 remain in the second composite image SP4.

  FIG. 10 is a diagram showing a third composite image SP5 obtained by performing the special comparative bright combination with the fourth image P4 and the second composite image SP4, with the third image P3 as an exceptional image. . Since the third image P3 is an exceptional image, the image synthesis is performed so that the subject Q3 is superimposed on the top in the third synthesized image SP5 as well. Therefore, the overlapping order from the top to the bottom of the subjects Q1 to Q4 is Q3, Q2, Q1, and Q4, and basically all the image portions of the subject Q3 remain in the third composite image SP5.

  Next, FIG. 11 is a flowchart showing the process of combining captured images.

  In a state where the imaging apparatus 1 is set to the comparatively bright combination mode in the image combination mode, for example, when a release button of the operation unit 20 is pressed, the captured image combination process is started.

  Then, first, the memory unit 16 is initialized (step S1).

  Next, according to the set ISO sensitivity, shutter speed, shooting interval, composite number, etc., the system control unit 21 controls the image sensor 13 to perform exposure (step S2). (Step S3).

  The system control unit 21 monitors an operation input from the operation unit 20 and controls to process the selected image as an exception image when an exception image selection operation is performed. Here, there are two timings for selecting an exceptional image, as shown in FIG. 15 or FIG. 16, for example.

  First, FIG. 15 is a timing chart showing an example in which an exceptional image selection operation is performed while observing an image displayed on the display unit 19. In this case, an exception processing instruction is issued from the operation unit 20 at the timing when the third image P3 is displayed on the display unit 19.

  On the other hand, FIG. 16 is a timing chart showing an example of performing an exceptional image selection operation while observing an actual subject. In this case, an exception processing instruction is issued from the operation unit 20 at the timing when the actual subject is observed, that is, when the third image P3 is exposed by the image sensor 13.

  Which of the operation timing shown in FIG. 15 and the operation timing shown in FIG. 16 is used can be set as desired by the user via the operation unit 20. That is, the operation unit 20 that is an image data selection unit selects exceptional image data at the timing of observing a subject that is an object to be imaged by the image sensor 13, or the comparison image displayed on the display unit 19. It is possible to set whether to select exceptional image data at the timing of observing an image represented by data (image data obtained by the latest photographing).

  Further, as an operation for selecting an exceptional image, an operation button is pressed, a touch operation when the display unit 19 is equipped with a touch panel, a remote control device (remote controller), or a portable information device as described in a third embodiment described later. Any one of remote operation from 3 and operation by voice input may be used.

  Furthermore, it is not limited to selecting an exception image based on an input from the user, but an exception image is automatically selected at a timing such as when a strobe is emitted or when a loud sound is input. It doesn't matter.

  And the system control part 21 determines whether the image read from the image pick-up element 13 is an exception image (step S4).

  If it is determined that the image is not an exceptional image, the system control unit 21 causes the comparison / synthesis processing unit 15 to perform comparatively bright combination processing as will be described later with reference to FIG. 12 (step S5). The synthesized image data obtained as a result of the synthesis process is stored in the first memory area 16a (step S6).

  On the other hand, if it is determined in step S4 that the image is an exceptional image, the system control unit 21 sets the level above the noise level indicating the magnitude of dark noise in the image data to be a predetermined threshold level. The exception processing unit 26 of the comparison / synthesis processing unit 15 is automatically set (step S7).

  FIG. 14 is a diagram showing an example of a histogram of data read from the OB area of the image sensor 13.

  Since the OB area of the image sensor 13 is optically shielded, the data read from the OB area contains dark noise. This dark noise includes random noise of a readout circuit that reads out from the image sensor 13, shot noise and fixed pattern noise caused by dark current, and the like. The signal processing unit 14 calculates a luminance histogram and noise amount (such as a standard deviation value of noise) of a black image based on data read from the OB area. The frequency in the region where the luminance level is low in FIG. 14 indicates the number of pixels in which such dark noise has occurred. Therefore, the system control unit 21 sets the luminance level (the luminance level at the upper limit of the dark noise) asymptotically approaching zero as the noise level NL and outputs it to the comparison / synthesis processing unit 15. The noise level NL automatically set by the system control unit 21 is displayed as a marker on the luminance histogram of the data read from the OB area on the display unit 19 as shown in FIG. This marker display is movable along the brightness level axis, and the automatically set noise level NL can be further manually adjusted.

  The exception processing unit 26 of the comparative synthesis processing unit 15 sets a threshold level in the comparatively bright synthesis process within a range equal to or higher than the received noise level NL. Note that the threshold level set by the exception processing unit 26 is also set within a range equal to or higher than the noise level NL in the comparative dark combining process.

  Then, the exceptional image data read from the image sensor 13 is stored in the second memory area 16b (step S8).

  After performing the processing of step S6 or step S8, next, the exceptional image data stored in the second memory area 16b is used as the combined image data stored in the first memory area 16a as an exception of comparatively bright combination. The special comparative bright combination processing to be combined is performed as described later with reference to FIG. 13 (step S9). However, this process is skipped when exceptional image data is not stored in the second memory area 16b. The composite image thus obtained is updated and displayed on the display unit 19 every time the loop of steps S2 to S10 is performed. This realizes a composite live valve function that can capture a composite image of a plurality of images that have been divided and exposed while observing them in a live view.

  Thereafter, it is determined whether or not an instruction to end shooting is given by the operation unit 20 (step S10). This photographing end instruction is made by pressing an operation button, a touch operation when the display unit 19 is equipped with a touch panel, a remote control device (remote controller), or remote control from a portable information device 3 as described in a third embodiment described later. Any of operation, operation by voice input, etc. may be used. If the number of shots is set in advance, this process is automatically terminated when the set number of shots has been taken. Alternatively, when the battery of the imaging device 1 is exhausted, this process is automatically terminated.

  Here, in the case where the photographing is not yet finished, the image pickup device 13 performs the cumulative comparison / synthesis processing by the comparison / synthesis processing unit 15 and the exception processing by the exception processing unit 26 in order to return to step S2 to capture the next image. This is performed every time image data is generated. On the other hand, when the photographing is finished, this process is finished.

  Next, FIG. 12 is a flowchart showing the comparatively bright combination process in step S5 of FIG.

  Upon entering this process, the comparison image data, which is the latest image data acquired by the image sensor 13, and the composite image data stored in the first memory area 16 a are displayed for each corresponding pixel position in pixel units. Read (step S21).

  Next, it is determined whether the pixel data of the read comparison image is larger than the noise level NL described above (step S22).

  If it is determined that the noise level is higher than the noise level NL, it is further determined whether or not the pixel data of the comparison image is larger than the pixel data of the composite image (step S23).

  If larger, the pixel data of the comparison image is set as new pixel data in the composite image (step S24).

  When the process of step S24 is completed, it is determined in step S22 that the pixel data of the comparison image is less than or equal to the noise level NL, or in step S23, the pixel data of the comparison image is less than or equal to the pixel data of the composite image. If it is determined, it is determined whether or not the processing of the pixel data at all the pixel positions has been completed (step S25).

  If it is determined that the processing has not been completed yet, the process proceeds to step S21 to perform the same processing for the next pixel position, and if it is determined that the processing of the pixel data at all the pixel positions has been completed. The process returns from the comparatively bright combination process to the photographed image combination process shown in FIG.

  Next, FIG. 13 is a flowchart showing the process of special comparative bright combination in step S9 of FIG.

  Upon entering this process, the composite image data stored in the first memory area 16a and the exceptional image data stored in the second memory area 16b are read in units of pixels for each corresponding pixel position (steps). S31).

  Then, it is determined whether or not the pixel data of the read exception image is equal to or higher than the threshold level (step S32).

  If it is determined that the threshold level is exceeded, the pixel data of the exceptional image is set as new pixel data in the composite image (step S33).

  When the process of step S33 is completed or when it is determined in step S32 that the pixel data of the exceptional image is less than the threshold level, it is determined whether the process of the pixel data at all the pixel positions is completed. (Step S34).

  Here, if it is determined that the processing has not been completed yet, the process proceeds to step S31 to perform the same processing for the next pixel position, and if it is determined that the processing of the pixel data at all the pixel positions has been completed. Then, the process returns from the special comparative bright combination processing to the photographed image combination processing shown in FIG.

  As shown in FIG. 12, in the comparative bright combination, the comparison / synthesis processing unit 15 performs the cumulative comparison / synthesis process in step S24 for pixel data below the noise level indicating the magnitude of dark noise in the image data. Not performed. The same applies to the case of performing the comparative dark combining process as the comparative combining process (therefore, the target of the comparative dark combining is a pixel whose pixel value is higher than the noise level, and the target of the special comparative dark combining process. Are pixels whose pixel value is higher than the noise level and below the threshold level set for comparative dark synthesis).

  This is due to the following reason. If pixel data having a pixel value equal to or lower than the noise level is subjected to comparatively bright combination, dark noise having a higher level remains, resulting in an image with a large noise feeling. In addition, if pixel data with a pixel value below the noise level is used, if dark comparison is performed, dark noise with a lower level will remain and the dark noise will be crushed, making the dark part flat and unnatural. It becomes an image. Therefore, in both cases of comparative bright combination and comparative dark combination, the cumulative comparison combination process is not performed for pixels having a noise level or less.

  Further, when performing the special comparative bright combination, the pixel level is set to a noise level or higher as described in step S7 in FIG. 11, and the pixel data is less than the threshold level as shown in step S32 in FIG. Since no exception processing is performed for, the exception processing unit 26 inevitably does not perform exception processing for pixel data less than the noise level. In addition, when performing special comparative dark synthesis, which is an exception processing of comparative dark synthesis that selects a smaller value in comparison selection, as described above, because pixels whose pixel value is higher than the noise level are targeted. Exception processing is not performed for pixel data below the noise level.

  According to the first embodiment, exceptional image data can be selected from a group of image data to be combined, and the exceptional image data is a pixel regardless of the order of brightness with other images. Since the data is used for the composite image, a desired subject can be left in the composite image when a composite image is obtained by combining a plurality of images according to brightness.

  At this time, since the exception processing is performed when the level is equal to or higher than a predetermined threshold level in the case of comparatively bright synthesis, and when the level is lower than the predetermined threshold level in the case of comparative dark synthesis, the entire synthesized image is an exception image. Therefore, it is possible to leave only subjects that are above or below the threshold level as exception processing in the composite image.

  In addition, since pixel data below the noise level is not subjected to cumulative comparison and synthesis processing, the dark noise level is maintained at the same level as a normal single-shot image to obtain an image that looks natural. Can do.

  At this time, by automatically setting the threshold level to be equal to or higher than the noise level, it is possible to maintain the dark noise level without requiring a complicated operation.

  Even in the case of comparative dark synthesis, pixel data having a noise level or less is not subjected to exceptional processing, so that the dark noise level of the synthesized image can be maintained.

  Since the cumulative comparison and synthesis process and the exception process are performed every time image data is generated by the image sensor 13, not only can a composite image be obtained at the time of shooting, but also the display unit is updated every time the composite image is updated. By displaying on 19, the user can observe how the composite image changes.

In addition, since the user can set the timing for selecting the exceptional image data, the subject in the exceptional image can be left in the recorded image at the desired timing.
[Embodiment 2]

  FIGS. 17 to 23 show the second embodiment of the present invention, and FIG. 17 is a flowchart showing processing of special comparative bright combination.

  In the second embodiment, the same parts as those in the first embodiment are denoted by the same reference numerals and the description thereof is omitted as appropriate, and only different points will be mainly described.

  As shown in FIG. 13, the special comparative bright combination processing of the first embodiment described above replaces the corresponding pixel data of the composite image when the pixel data of the exceptional image is equal to or higher than the threshold level. On the other hand, the special comparative bright combination processing of the present embodiment performs gradation conversion that is used preferentially over the pixel data of the exceptional image over the pixel data of the combined image, and the pixel data by the comparison unit 27 Exception processing is performed while using the comparison of.

  That is, the exception processing unit 26 according to the present embodiment includes a gradation conversion processing unit, and the pixel data that is an exception of the cumulative comparison and synthesis processing is a value in the case of comparatively bright synthesis that selects a larger value in comparison selection. In the case of comparative dark synthesis that selects the smaller value in the comparison and selection, the gradation conversion processing unit performs exception processing by converting the gradation so that the value is smaller. It has become.

  Here, with reference to FIG. 18 and FIG. 19, the example of a display of the image in the display part 19 is demonstrated. FIG. 18 is a diagram showing an example of image display on the display unit 19 before the exception image is selected, and FIG. 19 is a diagram showing an example of image display on the display unit 19 after the exception image is selected.

  On the back side of the imaging apparatus 1, a display unit 19 is provided and operation switches included in the operation unit 20 are arranged.

  Before the exceptional image is selected, as shown in FIG. 18, the display unit 19 includes a composite image 19a, a comparison image 19b, a luminance histogram 19d of the comparison image 19b, and a threshold level setting bar 19e. It is displayed.

  What is displayed as the composite image 19a is a composite image stored in the first memory area 16a.

  The latest image captured by the image sensor 13 is displayed as the comparative image 19b.

  The luminance histogram 19d is a graph showing the appearance frequency for each luminance of the displayed comparison image 19b.

  The threshold level setting bar 19e is provided so as to be movable along the luminance level axis of the luminance histogram 19d, and is used for manually setting the threshold level while referring to the luminance histogram 19d. It is composed. When the threshold level is not set manually, a value automatically set by the exception processing unit 26 is used as the threshold level.

  Further, after the exceptional image is selected, as shown in FIG. 19, the exceptional image 19c is additionally displayed on the display unit 19 (so-called freeze display). The exception image 19c is displayed as an exception image stored in the second memory area 16b. In the example shown in FIG. 19, the size of the comparison image 19b is reduced compared to that shown in FIG. 18, and the exceptional image 19c is displayed below the comparison image 19b. When the exceptional image 19c is displayed, the luminance histogram 19d and the threshold level setting bar 19e are changed to functions for performing display and operation related to the exceptional image 19c, and what is displayed as the composite image 19a is a special comparative bright combination process. This is the image that has been made.

  FIG. 22 is a diagram showing an example when the threshold level is manually set by the threshold level setting bar 19e while referring to the luminance histogram 19d.

  When images are taken one after another by the image sensor 13, the distribution of the luminance histogram 19d displayed on the display unit 19 also changes for each image. For example, when the subject is fireworks, the appearance frequency of the low-intensity part indicating the night sky in the background does not change much, but the appearance frequency of the medium to high-intensity part indicating the firework part changes for each firework. It is assumed that such a portion where the appearance frequency changes is a portion surrounded by an ellipse in FIG. In this case, by operating the threshold level setting bar 19e and setting the low-luminance side end of the part surrounded by the ellipse to be the threshold level, the desired fireworks part is targeted for exception processing and the background night sky It is possible to appropriately perform selection so as not to be an exception processing target. When the threshold level is changed, it is reflected in the composite result as needed, and a composite image 19 a corresponding to the changed threshold level is displayed on the display unit 19.

  Next, with reference to FIG. 17, the process of special comparative bright combination in the present embodiment will be described.

  In step S9 of FIG. 11, when the process of special comparative bright combination shown in FIG. 17 is entered, the composite image data stored in the first memory area 16a, the exceptional image data stored in the second memory area 16b, and Are read in units of pixels for each corresponding pixel position (step S41).

  Then, it is determined whether or not the pixel data of the read exception image is larger than the noise level NL (step S42).

  Here, when it is determined that the noise level is higher than the noise level NL, it is further determined whether or not the pixel data of the exceptional image is equal to or higher than the threshold level (step S43).

  Here, when it is determined that the threshold level or higher, the pixel data of the exceptional image is subjected to gradation conversion by the gradation conversion processing unit of the exception processing unit 26 (step S44). In this way, exception processing is performed for pixel data above the threshold level in the exceptional image, and processing similar to the normal comparative bright combination processing is performed for pixel data below the threshold level.

  FIG. 20 is a diagram showing an example of a gradation conversion curve used in comparatively bright synthesis.

  In the comparatively bright combination, the gradation conversion curve in which the pixel data of the comparative image is preferentially used over the pixel data of the combined image is as shown in FIG. 20, for example. That is, when the pixel value (pixel value) before conversion is less than the threshold level TL, the pixel value is not changed (that is, it is determined that gradation conversion is not performed in step S43), and the pixel value is equal to or higher than the threshold level TL. In some cases, the gradation conversion curve increases the pixel value.

  FIG. 23 is a diagram showing an example of a luminance histogram when gradation conversion is performed on the exceptional image data of the luminance histogram as shown in FIG. 22 using the gradation conversion curve as shown in FIG.

  As shown in FIG. 23, the frequency of the portion surrounded by the ellipse in FIG. 22 decreases in FIG. 23, and the frequency of the higher luminance portion increases as shown by the dotted line in FIG.

  FIG. 21 is a diagram showing an example of a gradation conversion curve used in comparative dark synthesis.

  In the comparative dark composition, a gradation conversion curve in which the pixel data of the comparative image is used preferentially over the pixel data of the composite image is as shown in FIG. 21, for example. That is, when the pixel value (pixel value) before conversion is larger than the threshold level TL, the pixel value is not changed (that is, it is determined that gradation conversion is not performed as in step S43), and the threshold level TL A gradation conversion curve that reduces the pixel value in the following cases. However, although not shown in FIG. 21, since it is preferable not to flatten the dark noise as described above for pixel data whose pixel value is equal to or less than the noise level NL, the determination in step S42 in FIG. Similarly to the above, it is preferable not to perform gradation conversion.

  Note that the gradation conversion curve as shown in FIG. 20 or FIG. 21 may allow the user to set the curve shape as desired.

  If the processing of step S44 ends or if it is determined in step S43 that the pixel data of the exceptional image is less than the threshold level, the pixel of the exceptional image after gradation conversion is processed if it is a pixel that has undergone the processing of step S44. If the pixel data is a pixel that has not undergone the process of step S44, the comparison unit 27 determines whether the pixel data of the exceptional image is larger than the pixel data of the composite image (step S45).

  If it is determined that the pixel data is larger than the pixel data of the composite image, the pixel data of the exceptional image before gradation conversion is set as new pixel data in the composite image (step S46). Thus, the pixel data of the exceptional image subjected to gradation conversion in step S44 is used only for comparison in step S45, and the pixel data of the exceptional image before gradation conversion is used as replacement data for the composite image.

  When the process of step S46 ends, or when it is determined in step S42 that the pixel data of the exceptional image is equal to or lower than the noise level, or in step S45, it is determined that the pixel data is equal to or lower than the pixel data of the composite image, It is determined whether or not the processing of the pixel data at all pixel positions has been completed (step S47).

  Here, if it is determined that the processing has not been completed yet, the process proceeds to step S41 to perform the same processing for the next pixel position, and if it is determined that the processing of the pixel data at all the pixel positions has been completed. Then, the process returns from the special comparative bright combination processing to the photographed image combination processing shown in FIG.

  According to the second embodiment, the same effects as those of the first embodiment described above are obtained, and gradation conversion is performed on the pixel data of the exceptional image so as to be used preferentially over the pixel data of the composite image. Since the exception processing is performed in this way, it is possible to perform the special comparative light / dark combination processing while using the comparison of the pixel data by the comparison unit 27 as in the normal comparative light / dark combination as it is. . As a result, the configuration of the comparison unit 27 related to normal comparative light / dark combination can be used in special comparative light / dark combination, and there is an advantage that the configuration of the comparison / synthesis processing unit 15 is not complicated.

In addition, since the pixel value of the synthesized image and the pixel value of the exception image after gradation conversion are compared to determine which pixel data is left in the synthesized image, for example, in the case of comparatively bright synthesis, the threshold level Even an exceptional image pixel with a brightness less than that may remain in the composite image, and an exceptional image pixel with a brightness higher than the threshold level may not remain in the composite image. Therefore, it is possible to synthesize a more natural image in which the outline blends into the surroundings, rather than an unnatural image in which the brightness portion above the threshold level in the exceptional image is cut out and synthesized. .
[Embodiment 3]

  FIGS. 24 to 26 show Embodiment 3 of the present invention. FIG. 24 shows a display unit when the portable information device 3 is used as a display unit and an operation unit when the image capturing apparatus 1 performs image composition. It is a figure which shows the example of a display.

  In the third embodiment, the same parts as those in the first and second embodiments are denoted by the same reference numerals and the description thereof is omitted as appropriate, and only different points will be mainly described.

  In the first and second embodiments described above, a process of generating a composite image at the time of shooting is performed. However, in the present embodiment, an image already recorded in the recording unit 18 is selected and an image composition process is performed. Yes. Furthermore, in the present embodiment, the image composition processing is performed by the imaging apparatus 1, but the selection of an image to be composed, the selection operation of an exceptional image, the operation for starting the composition processing, and the like are performed by the portable information device 3. It has become.

  First, the portable information device 3 can communicate with a camera communication unit (not shown) built in the imaging apparatus 1 via, for example, a wireless LAN, and can exchange data and commands.

  And as shown in FIG. 24, the display part 31 of the portable information device 3 is provided with the touch panel, for example, and functions also as an operation part. A plurality of images 33 read from the recording unit 18 of the imaging apparatus 1 are arranged and reproduced and displayed on the display unit 31. When any one of the plurality of arranged images 33 is touched, the touched image 33 can be selected as an image to be synthesized. Here, the image 33a and the image 33b are selected. An image data group is shown.

  Further, the image 33b in the image data group is an image selected as an exceptional image by putting a check mark. In the example shown in FIG. 24, two images 33b are selected as exception images, given priority levels as exception images, and the assigned priority levels are displayed using numbers, for example. Therefore, the display unit 31 with a touch panel of the portable information device 3 functions as an image data selection unit that selects exception image data to be subjected to exception processing from the image data group, and sets a plurality of exception image data with priority. It can be set.

  The display unit 31 displays a composite image 34, an exception image 35, a luminance histogram 36, and a threshold level setting bar 37.

  Here, in the example shown in FIG. 24, the image 33b of the second highest priority order surrounded by a dotted frame is displayed as the exception image 35. In the present embodiment, since a plurality of exceptional image data can be set, when the exceptional image 35 is slid, other exceptional image data (for example, the image 33b with the highest priority) can be displayed. It has become.

  The brightness histogram 36 indicates information related to the exception image 35, and the threshold level setting bar 37 is for manually setting the threshold level, as in the examples shown in FIGS.

  Further, the display unit 31 includes an operation button 32 including a setting menu button 32a, a composition cancel button 32b, and a composition start button 32c configured to perform a function as an operation unit by using a touch panel. Is displayed.

  The setting menu button 32a is used to set whether to perform comparatively bright combining processing or comparative dark combining processing as combining processing, or to set priority when there are a plurality of images selected as exceptional images. Used to do

  The composition cancel button 32b is used when an operation for canceling the composition processing is performed.

  The composition start button 32c is used when an operation for starting image composition with the set contents is performed. When the compositing start button 32c is pressed, an image compositing process start command is transmitted from the portable information device 3 to the image capturing apparatus 1, and the image compositing process as shown in FIG. An image obtained as a result of this image composition processing is transmitted from the imaging device 1 to the portable information device 3 and displayed as a composite image 34 as described above.

  FIG. 25 is a flowchart showing the recorded image composition process.

  When this process is started, an image recorded in the recording unit 18 of the imaging device 1 is read out and displayed as an image 33 on the display unit 31 of the portable information device 3 (step S51).

  Thereafter, the user touches some of the plurality of images 33 and selects them as a compositing target (see images 33a and 33b) (step S52).

  Further, an exceptional image is selected by the user as necessary from the images selected as the synthesis target (see image 33b) (step S53). The exception image selected here may be plural as described above.

  The selected exception image is displayed as an exception image 35 on the display unit 31 each time it is selected, and the threshold level is set using the threshold level setting bar 37 (or the threshold level is set by automatic processing based on a luminance histogram or the like). ) Is performed for each exceptional image 35 (step S54). Regarding the noise level calculation in the automatic setting of the threshold level, for example, the noise level (such as the standard deviation value of noise) in the OB area is recorded in the tag information at the stage of image processing for generating a recorded image, and used as an exception image. When selected, the noise level read from the tag information may be set as the threshold level. Accordingly, the threshold level may be different for each exceptional image.

  Further, the priority order is set by the user for the image 33b selected as the exceptional image (step S55).

  In this way, when the image data group to be combined, the exceptional image data in the image data group, and the priority order when there are a plurality of exceptional image data are set, the combination start button 32c is pressed ( When touched), a compositing process is performed (step S56). When a composite image is generated, it is displayed on the display unit 31 as a composite image 34, and then this process ends.

  If the composition cancel button 32b is pressed at any stage before starting the process in step S56, the recorded image composition process is terminated without performing the composition process in step S56.

  Next, FIG. 26 is a flowchart showing the image composition processing in step S56 of FIG.

  In this process, first, the memory unit 16 is initialized (step S61).

  Next, the images to be combined are sequentially read out one by one from the recording unit 18 (step S62).

  Then, the system control unit 21 determines whether or not the read image is an exceptional image (step S63).

  If it is determined that the image is not an exceptional image, the system control unit 21 causes the comparative synthesis processing unit 15 to perform comparatively bright synthesis processing as shown in FIG. 12 (step S64), and the result of the synthesis processing. The obtained composite image data is stored in the first memory area 16a (step S65).

  On the other hand, when it is determined in step S63 that the image is an exception image, the system control unit 21 stores the exception image in the second memory area 16b in accordance with the priority order (step S66). Here, it is assumed that the second memory area 16b can store and store a plurality of exceptional images according to the priority order.

  After the processing of step S65 or step S66 is performed, it is determined whether or not processing has been performed for all the images to be combined (step S67), and if processing of all target images has not been completed yet. Returning to step S62, the next image is processed.

  Thus, if it is determined in step S67 that all the target images have been processed, then the exceptional images stored in the second memory area 16b are set as processing targets in order of decreasing priority ( Step S68).

  Then, the special comparative bright combination processing as shown in FIG. 13 or FIG. 17 is performed (step S69), and it is determined whether or not the processing has been completed for all the exceptional images (step S70).

  If it is determined in step S70 that all the exception images have not been processed yet, the next highest priority exception image, that is, the highest priority among the exception images that have not been processed in step S69. The process of step S69 is performed for images with a lower rank.

  By performing such processing, the exception processing unit 26 has the highest priority pixel data when there are a plurality of pieces of pixel data that are exceptions to the cumulative comparison / synthesis processing at the same pixel position in a plurality of exception image data. Is the new pixel data at this pixel position. That is, the special comparative bright combination processing as shown in FIG. 13 or FIG. 17 is performed at the end of the exceptional image with the highest priority, and the exceptional image with the highest priority is the recording image of the combined result with the highest priority. Will remain as.

  Thus, when it is determined in step S70 that all the exceptional images have been processed, the process returns to the process shown in FIG.

  In the above description, the composition processing based on the image recorded in the recording unit 18 is performed by the combination of the imaging device 1 and the portable information device 3. However, this is only an example, and may be performed by the imaging device 1 alone. Then, the image data recorded in the recording unit 18 may be transferred from the imaging apparatus 1 to a personal computer or the like, and an image processing program may be executed and synthesized by the personal computer. The data may be transferred via a network or the like and synthesized, for example, on the Internet, and is not limited to a specific configuration.

  According to the third embodiment, the above-described implementation can be performed by performing the cumulative comparison / synthesis processing by the comparison / synthesis processing unit 15 and the exception processing by the exception processing unit 26 based on the image data group read from the recording unit 18. The same effects as those of Embodiments 1 and 2 can be obtained.

  Since the exceptionally bright combination processing is performed later for the exceptional images having higher priority, it is possible to obtain a combined image in which the exceptional images that are desired to be recorded are prioritized according to the order.

  Although the image processing apparatus has been mainly described above, an image processing method for performing the same processing as the image processing apparatus may be used, and an image processing program for causing a computer to execute the image processing method, the image processing program The recording medium may be a non-temporary recording medium that can be read by a computer.

  Further, the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope in the implementation stage. In addition, various aspects of the invention can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, you may delete some components from all the components shown by embodiment. Furthermore, the constituent elements over different embodiments may be appropriately combined. Thus, it goes without saying that various modifications and applications are possible without departing from the spirit of the invention.

DESCRIPTION OF SYMBOLS 1 ... Imaging device 3 ... Portable information device 11 ... Lens 12 ... Shutter 13 ... Imaging element (imaging part)
DESCRIPTION OF SYMBOLS 14 ... Signal processing part 15 ... Comparison synthetic | combination processing part 16 ... Memory part 16a ... 1st memory area 16b ... 2nd memory area 17 ... Image processing part 18 ... Recording part 19 ... Display part 19a, 34 ... Composite image 19b ... Comparative image 19c, 35 ... Exception image 19d, 36 ... Luminance histogram 19e, 37 ... Threshold level setting bar 20 ... Operation unit (image data selection unit)
21 ... System control unit 26 ... Exception processing unit (including gradation conversion processing unit)
27: Comparison unit 28 ... Composition unit 31 ... Display unit (image data selection unit)
32 ... Operation buttons 32a ... Setting menu button 32b ... Composition cancel button 32c ... Composition start button 33, 33a, 33b ... Image

Claims (10)

Among image data groups to be combined, certain image data is combined image data, and other image data is compared image data. Pixel data corresponding to pixel positions in the combined image data and the compared image data are Comparing and comparing to select only one of the larger value and the smaller value and comparing the selected pixel data as new pixel data at the corresponding pixel position and reconstructing the composite image data A comparison and synthesis processing unit that repeats the synthesis process every time the comparison image data is updated;
An image data selection unit for selecting exceptional image data from the image data group;
When the pixel data constituting the exceptional image data is equal to or higher than a predetermined threshold level when a larger value is selected in the comparison selection, a predetermined value is selected when the smaller value is selected in the comparison selection. When the threshold level is lower than the threshold level, an exception of the cumulative comparison / synthesis process is performed for the corresponding pixel, and the comparison / synthesis process unit reconfigures the synthesized image data with the pixel data as new pixel data at the corresponding pixel position. An exception handling unit that performs
An image processing apparatus comprising: The image data selection unit can set a plurality of exceptional image data with priorities,
When there are a plurality of pieces of pixel data that are exceptions to the cumulative comparison / synthesis processing at the same pixel position in the plurality of exception image data, the exception processing unit determines that the pixel data having the highest priority is newly assigned to the corresponding pixel position. The image processing apparatus according to claim 1, wherein the image processing apparatus is pixel data.   The exception processing unit includes a gradation conversion processing unit, and the pixel data as an exception of the cumulative comparison / synthesis process is compared with the pixel data so that the pixel data becomes larger when the larger value is selected in the comparison selection. 2. The exception processing is performed by the gradation conversion processing unit performing gradation conversion so that a smaller value is selected when selecting a smaller value in selection. An image processing apparatus according to 1.   The image processing apparatus according to claim 1, wherein the comparison / synthesis processing unit does not perform the cumulative comparison / synthesis processing for pixel data having a noise level equal to or lower than a dark noise level in the image data.   The image processing apparatus according to claim 1, wherein the exception processing unit automatically sets the predetermined threshold level to be equal to or higher than a noise level indicating a magnitude of dark noise in the image data.   6. The image according to claim 5, wherein the exception processing unit does not perform the exception processing for the pixel data having the noise level or less even when the smaller value is selected in the comparison and selection. Processing equipment. An imaging unit that images the subject and generates image data;
The image data group is sequentially generated by the imaging unit,
The image processing apparatus according to claim 1, wherein the cumulative comparison and synthesis process by the comparison and synthesis processing unit and the exception process by the exception processing unit are performed every time image data is generated by the imaging unit. . A display unit for displaying an image represented by the comparison image data;
The image data selection unit selects the exceptional image data at the timing of observing the subject to be imaged by the imaging unit, or is represented by the comparison image data displayed on the display unit. The image processing apparatus according to claim 7, wherein it is possible to set whether to select the exceptional image data at a timing at which an image to be observed is observed. A recording unit for recording data;
The image data group is data recorded in the recording unit,
The image processing apparatus according to claim 1, wherein the cumulative comparison and synthesis process by the comparison and synthesis processing unit and the exception process by the exception processing unit are performed by reading the image data group from the recording unit. Among image data groups to be combined, certain image data is combined image data, and other image data is compared image data. Pixel data corresponding to pixel positions in the combined image data and the compared image data are Comparing and comparing to select only one of the larger value and the smaller value and comparing the selected pixel data as new pixel data at the corresponding pixel position and reconstructing the composite image data A comparison and synthesis process step in which the synthesis process is repeated each time the comparison image data is updated;
An image data selection step for selecting exceptional image data from the image data group;
When the pixel data constituting the exceptional image data is equal to or higher than a predetermined threshold level when a larger value is selected in the comparison selection, a predetermined value is selected when the smaller value is selected in the comparison selection. When the threshold level is lower than the threshold level, an exception of the cumulative comparison / synthesis process is performed for the corresponding pixel, and the synthesized image data is reconfigured in the comparison / synthesis process step with the pixel data as new pixel data at the corresponding pixel position. An exception handling step that performs
An image processing method comprising:

Images