JP7446715B2 - Image processing device, imaging device, image processing method and program - Google Patents

Description

The present invention relates to an image processing device that performs composition of a plurality of images having different focus positions.

When capturing images of multiple subjects at greatly different distances from an imaging device such as a digital camera, or when capturing a long subject in the depth direction, the depth of field may be insufficient and only a portion of the subject may be in focus. It may not be possible to match. Alternatively, when it is desired to obtain an image with extremely high resolution, the depth of field must be made shallow, and the entire subject may not be included in the depth of field. In order to solve this problem, Patent Document 1 discloses that a plurality of images with different focus positions are taken, only the in-focus area is extracted from each image and combined into a single image, and the entire imaging area is focused. A so-called depth compositing technique has been disclosed, which generates a composite image that includes the following:

On the other hand, in Patent Document 2, a first image processing section that receives an image signal from an imaging section or the like is arranged in front of a second image processing section, and the second image processing section that cannot receive a high transmission rate is placed in front of the second image processing section. An imaging device has been disclosed that achieves scalability by acting as a buffer against the above.

Japanese Patent Application Publication No. 10-290389 Unexamined Japanese Patent Publication No. 2017-153156

However, when attempting to perform depth compositing using the imaging device described in Patent Document 2, the first image processing unit encodes and compresses the image data once to reduce the transmission rate, so there is a possibility that the image quality will deteriorate. be. In addition, since the size of the image used for depth compositing may be extremely large, if the first image processing unit records it once in the storage unit and then transmits it to the second image processing unit, the image data due to data delay may be lost. Display and recording may be delayed.

The present invention was made in view of the above-mentioned problems, and reduces the processing load by changing the information transmitted between the image processing units when performing depth compositing using multiple image processing units. The purpose is to provide an image processing device that can

In order to solve the above problems, the present invention includes an acquisition means for acquiring a plurality of images, receives signals of the plurality of images from the acquisition means, divides the plurality of images, and divides the plurality of images. a first processing means that sets a plurality of divided regions in each of the plurality of divided regions, and a second processing means that receives signals of some of the divided regions among the plurality of divided regions from the first processing means and generates a depth composite image. processing means, wherein the first processing means and the second processing means are each constituted by an individual block, and the first processing means is configured to perform processing on the divided area in each of the plurality of images. and selects the divided area based on the result of the evaluation, wherein the first processing means sets an evaluation value for each divided area of at least some of the plurality of divided areas as the evaluation. The second processing means receives the signals of the divided regions selected by the first processing means in order of increasing evaluation value, and the second processing means receives signals from the first processing means from the first processing means. An image processing apparatus is provided, wherein a transmission rate at which the signal is received is lower than a transmission rate at which the first processing means receives the signal from the acquisition means.

According to the configuration of the present invention, it is possible to provide an image processing device that reduces information transmitted to a subsequent image processing unit when performing depth synthesis using a plurality of image processing units.

1 is a block diagram showing the hardware configuration of a digital camera in a first embodiment. FIG. 7 is a flowchart for explaining depth synthesis processing in the first embodiment. FIG. 3 is a diagram for explaining region division of an image in the first embodiment. FIG. 3 is a diagram for explaining image evaluation in the first embodiment. FIG. 2 is a block diagram showing the hardware configuration of a digital camera in a second embodiment. 7 is a flowchart for explaining depth synthesis processing in the second embodiment. FIG. 7 is a diagram for explaining the relationship between the focus position and image magnification in the second embodiment. FIG. 7 is a diagram for explaining image cutting in the second embodiment. FIG. 7 is a diagram for explaining image enlargement processing in the second embodiment. FIG. 3 is a block diagram showing the hardware configuration of a digital camera in a third embodiment. 12 is a flowchart for explaining depth synthesis processing in the third embodiment. FIG. 7 is a diagram for explaining region division in the third embodiment. FIG. 7 is a diagram for explaining setting of divided regions according to magnification in the third embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

(First embodiment)
FIG. 1 is a block diagram showing the hardware configuration of a digital camera in this embodiment.

The image sensor 101 converts light that has passed through an optical system (not shown) into an electrical signal and outputs it as an image signal, and is, for example, a complementary metal oxide semiconductor (CMOS) sensor.

The first signal processing unit 102 receives the image signal output from the image sensor 101 and performs signal processing. The first signal processing section 102 is composed of a plurality of processing sections. The image sensor 101 and the first signal processing unit 102 are electrically connected, and an image signal is transmitted as an electrical signal from the image sensor 101 over a predetermined period of time.

The first control unit 103 controls the first signal processing unit 102 and also controls the image sensor 101, the image signal evaluation unit 104, the area selection unit 105, and the like.

The image signal evaluation unit 104 is one of the units forming the first signal processing unit 102 and calculates an evaluation result for the image signal received from the image sensor 101.

The area selection unit 105 is one of the units constituting the first signal processing unit 102, and selects an image signal stored in a frame memory, which will be described later, based on the evaluation result obtained from the image signal evaluation unit 104. Outputs an area selection signal for selecting an area to read.

The frame memory 106 stores image signals read out from the image sensor 101, and the image signals are read and written via a memory control unit 107, which will be described later.

The memory control unit 107 is one of the units constituting the first signal processing unit 102 and performs control for storing the image signal read from the image sensor 101 in the frame memory 106.

The second signal processing unit 108 receives the image signal output from the first signal processing unit 102 and performs signal processing. The first signal processing section 102 and the second signal processing section 108 are electrically connected, and an image signal is transmitted as an electrical signal from the first signal processing section 102 over a predetermined period of time.

The second control unit 109 controls the entire second signal processing unit 108, controls a depth synthesis unit 110 and a storage unit 111, which will be described later, and performs transmission and reception with the first control unit 103.

The depth synthesis section 110 performs depth synthesis processing on the image signal output from the first signal processing section 102. Depth composition processing involves capturing multiple image signals at different focus positions, cutting out the image signals of at least a portion of the in-focus area from each image, and combining them to create a single image with a greater depth of field. This is a process that generates deep image signals. The image signal generated by the depth synthesis section 110 is stored in the storage section 111, which will be described later.

The storage unit 111 is, for example, a memory that includes a recording process such as a DRAM (Dynamic Random Access Memory).

In the configuration shown in FIG. 1, the image sensor 101, the first signal processing section 102, and the second signal processing section 108 are electrically connected to each other. The data transmission rate from the image sensor 101 to the first signal processing unit 102 is designed to be higher than the data transmission rate from the first signal processing unit 102 to the second signal processing unit 108. . In such a setting, if the image sensor 101 is desired to be changed, the image sensor 101 can be changed by newly incorporating a connection circuit or the like into the first signal processing unit 102. Another advantage is that the entire configuration can be easily reconstructed while maintaining the connection to the second signal processing section 108.

FIG. 2 is a flowchart for explaining depth synthesis processing in this embodiment. The flow of this embodiment will be described with reference to FIG. 2.

In step S201, the first control unit 103 determines the number of images to be captured based on a predetermined number or an instruction from the user.

In step S202, imaging is started based on instructions from the user.

In step S203, the first control unit 103 drives the focus lens. By changing the relative position of the focus lens, the focus position in the optical axis direction can be changed.

In step S204, the image sensor 101 outputs a subject image formed through an optical system (not shown) as an electrical signal.

In step S205, the image sensor 101 transmits the electrical signal of the subject image to the first signal processing unit 102.

In step S206, the image signal evaluation unit 104 divides the image obtained from the image sensor 101 into a plurality of regions, and calculates a contrast value for each region. The contrast value is a numerical representation of the amplitude of fluctuation of pixel values constituting an image signal, and is calculated using, for example, a difference value between adjacent pixels.

FIG. 3 is a diagram for explaining region division of an image in this embodiment. An image 301 is an image obtained from the image sensor 101, and a region 302 is one divided region. Objects 30 to 305 are shown in image 301. In Figure 3, the dotted lines show how one image is divided into four parts in the horizontal direction (horizontal direction) and into three parts in the vertical direction (vertical direction), resulting in a total of 12 parts. It shows. The image signal evaluation unit 104 transmits the average contrast value for each of the 12 divided areas to the area selection unit 105 as an evaluation result. Note that in this embodiment, a contrast value is used as an evaluation index of an image signal, but there are other parameters that can be used as an evaluation index. For example, even if you perform frequency analysis for each divided area and calculate the evaluation result using the ratio of high frequency components included or the amplitude value of the high frequency component as an index from the analysis results, it will not show the degree of focus. It is effective as an indicator.

In step S207, the memory control unit 107 temporarily stores the image signal output from the image sensor 101 by performing write access to the frame memory 106.

In step S208, the area selection unit 105 selects an area. The area selection unit 105 transmits the evaluation result received from the image signal evaluation unit 104 as well as area selection information for selecting a divided area of the image signal to be transmitted to the second signal processing unit 108 to the memory control unit 107.

FIG. 4 is a diagram for explaining image evaluation in this embodiment. FIG. 4 shows captured images 401, 402, and 403. Each of the images 401, 402, and 403 is divided into four parts horizontally and three parts vertically. The numbers written in the divided regions of each image in FIG. 4 represent contrast values expressed as numbers within the range of 0 to 127, with higher numbers indicating higher contrast values. In this embodiment, as an example, it is determined whether the image is in focus based on the height of the contrast value. When a certain area is in focus, the contrast value of the area is generally higher than when the area is out of focus. In a plurality of images with different focus positions, the same area is determined to be in focus when the contrast value is the highest.

Specifically, the image shown in FIG. 4 has the following meaning. The image 401 shows the contrast value when the subject 303 is in focus, and the divided regions 404 to 409 show high contrast values. Similarly, the image 402 shows the contrast value when the subject 304 is in focus, and the contrast values in the divided areas 410 and 411 are high. The image 403 schematically shows the contrast value when the subject 305 is in focus, and the contrast values are high in the divided regions 412 to 414.

The area selection unit 105 outputs area selection information to the memory control unit 107 so as to read out divided areas having a contrast value equal to or greater than a predetermined threshold from the frame memory 106. For example, when the contrast value threshold is set to 50 for the image in FIG. 4, the area selection unit 105 reads out the divided areas 404 to 414.

In step S209, the first signal processing unit 102 transmits the divided area selected by the area selection unit 105 to the second signal processing unit 108.

In step S210, it is determined whether the number of captured images has been reached. If it has been reached, the process advances to step S211; if it has not been reached, the process returns to step S203.

In step S211, the depth synthesis unit 110 performs depth synthesis processing. The depth synthesis unit 110 creates a composite image by preferentially replacing the focused area selected in step S209 with the composite image. At the same time, the depth synthesis unit 110 changes and stores the address information when storing the image signal in the storage unit 111 according to the pixel position information of the image signal. By sequentially reading out the images, it becomes possible to read them out as a single image signal.

Note that in this embodiment, for example, the evaluation values of the area indicated by the divided area 415 of the image 403 in FIG. 4 are smaller than the threshold value for all three images, so the following processing is performed. The area selection unit 105 selects, as a readout area, a divided area that is determined to have never been selected when processing the third image signal, regardless of the threshold value determination result. This makes it possible to adjust the image synthesized by the depth synthesis unit 110 so that no omissions appear.

The following effects can be obtained through the processing described above. When the data transmission rate between the image sensor 101 and the first signal processing unit 102 is higher than the data transmission rate between the first signal processing unit 102 and the second signal processing unit 108, the following processing is performed. . The first signal processing section 102 transmits the divided regions selected by the region selection section 105 to the second signal processing section 108 according to the evaluation result of the image signal evaluation section 104. Through such processing, transmission of unnecessary data from the first signal processing unit 102 to the second signal processing unit 108 can be suppressed, and a highly scalable configuration can be designed.

Note that, in this embodiment, depending on the evaluation result by the image signal evaluation unit 104, the same divided area may be selected for a plurality of image signals. In this case, when the first signal processing section 102 transmits the selected image signal to the second signal processing section 108, the evaluation result is also transmitted. Next, the depth synthesis section 110 finally uses the better of the transmitted evaluation results for depth synthesis.

Furthermore, the region selection unit 105 may select not only the divided region with the best evaluation value but also a predetermined number of regions at the top in the order of the highest evaluation value. Through such processing, the number of divided regions selected from one image signal becomes constant, and the depth compositing unit 110 further selects regions actually used for the composite image in the depth compositing in step S211.

According to the first embodiment, it is possible to suppress data that is not necessary for creating a depth composite image from among the data sent to the second signal processing unit, and perform image processing that can further reduce the processing load. Can be done.

(Second embodiment)
The second embodiment will be described below. The second embodiment takes into consideration a case where the image magnification of an image formed on an image sensor changes depending on the position of a focus lens due to the optical characteristics of an optical system, which will be described later. Description of parts similar to those in the first embodiment will be omitted.

FIG. 5 is a block diagram showing the hardware configuration of the digital camera in this embodiment.

The optical system 501 forms an image of light from a subject. The optical system 501 includes at least a focus lens and a control mechanism for the focus lens for focusing on a subject, a zoom lens and a control mechanism for the zoom lens for changing the focus distance, and an aperture for adjusting the amount of incident light. It has one or more.

The optical system driving unit 502 is one part of the first signal processing unit 102 and transmits an electric signal for driving the lens group of the optical system 501. In this embodiment, the optical system driving unit 502 generates a control signal to control a focus lens for focusing on a subject among the lens groups of the optical system 501. Further, the optical system driving unit 502 transmits the image magnification when driving the focus lens to the cutout area determining unit 503, which will be described later.

Based on the image magnification sent from the optical system drive unit 502, the cutout area determination unit 503 generates area information for a cutout unit 504 (described later) to cut out the image output from the image sensor 101.

The cropping unit 504 is one part of the first signal processing unit 102, and performs cropping of the image output from the image sensor 101 based on the image magnification sent from the optical system drive unit 502. Perform processing. The number of pixels of the image signal output by the clipping unit 504 is smaller than the number of pixels of the image signal input to the clipping unit 504.

The enlarging unit 505 receives the image signal output to the cutting unit 504 and performs processing to enlarge the image signal. The enlargement process is a process of increasing the number of horizontal pixels and the number of vertical pixels of an image signal by a predetermined magnification. The number of pixels of the image signal output by the enlarging unit 505 is greater than the number of pixels of the image signal input to the enlarging unit 505.

FIG. 6 is a flowchart for explaining depth synthesis processing in this embodiment.

In step S601, the optical system drive unit 502 drives the focus lens based on the focus lens drive control signal. Optical system drive section 502 transmits a focus lens drive control signal to optical system 501. The optical system driving unit 502 changes the focus position by changing the relative position of the focus lens.

FIG. 7 is a diagram for explaining the relationship between the focus position and image magnification in this embodiment. A curve 701 in FIG. 7 shows how the image magnification changes as the focus position changes. FIG. 7 shows how three images are captured at focus positions 702, 703, and 704, respectively.

In step S602, the cropping area determination unit 503 determines the cropping area of the image. The cropping area determination unit 503 determines a cropping area based on the image magnification when the image received from the first signal processing unit 102 is captured. For example, assume that the image magnifications 705, 706, 707 at the focus positions 702, 703, 704 shown in FIG. 7 are αn (n=0, 1, 2). In the following description, the image magnification 705 is α0, the image magnification 706 is α1, and the image magnification 707 is α2. The cropping area selection unit 105 calculates the cropping start position and the number of cropping pixels when the image magnification is αn (n=0, 1, 2).

FIG. 8 is a diagram for explaining image cutting in this embodiment. An image 801 is an image captured at an image magnification α0, an image 802 is an image captured at a multiplication factor α1, and an image 803 is an image captured at an image magnification α2. In this way, as the image field ratio increases, the area in which the subject is reflected becomes narrower, and when performing depth compositing, it is necessary to combine images based on the image with the highest image magnification (the narrowest angle of view). desirable. Therefore, some angles of view of images other than the image with the highest image magnification are not used in the composite image. The cutout area selection unit 105 cuts out only the angle of view used for the composite image. The cropping area selection unit 105 calculates the relative ratio (image magnification ratio: βn) between other images and the image signal with the highest image magnification, and determines the cropping start position and the number of cropping pixels based on the relative ratio (image magnification ratio: βn). In this embodiment, the cutout area selection unit 105 calculates based on the following (Formula 1).

The cropping start position and the number of cropping pixels are determined based on the image magnification ratio βn determined by the cropping area selection unit 105. In this embodiment, it is expressed using a zero-origin orthogonal coordinate system with the upper left pixel as the origin. Further, the horizontal direction is defined as the x coordinate, and the vertical direction is defined as the y coordinate. Then, the x-coordinate of the cutting start position can be calculated using the following (Equation 2).

The y-coordinate of the cutting start position can be calculated using (Equation 3) below.

Note that pixelH and pixelV represent the number of horizontal pixels and the number of vertical pixels in the effective area of the image signal, respectively.

Further, the cutout area selection unit 105 calculates the number of cutout horizontal pixels pixelH' and the cutout vertical pixel number pixelV' using (Equation 4) and (Equation 5) below.

The clipping regions determined by the clipping region selection unit 105 for the images 801, 802, and 803 in FIG. 8 are regions 804, 805, and 806. Note that since the image 803 has the maximum image magnification, the size of the cropping area 806 determined by the cropping area selection unit 105 is the same as that of the image 803.

Note that the relationship between the focus position and the image magnification as shown in FIG. 7 may be held in a discrete table in a storage section (not shown) inside the optical system drive section 502. Alternatively, it may be held as a discrete table in a storage unit (not shown) inside the first control unit 103 and transmitted. Furthermore, if the relationship between the focus position and the image magnification can be approximated by a polynomial, only the coefficients of the polynomial may be stored in advance in a storage unit (not shown).

In step S603, the cutting unit 504 cuts out the cutting area determined by the cutting area determining unit 503. In addition, in the enlarging process described below, if pixels outside the clipping area determined by the clipping area selection unit 105 are required, the clipping unit 504 cuts out the image signal starting from the image signal with the lowest image magnification to output the corresponding signal. By doing so, it is also possible to improve the image quality.

In step S604, the enlarging unit 505 performs enlargement processing. Enlarging section 505 receives the image magnification from first signal processing section 102 and increases the size of the image according to the predetermined enlargement factor.

FIG. 9 is a diagram for explaining image enlargement processing in this embodiment. Image 901 shows how an image taken at image magnification α0 is enlarged, image 902 shows how an image taken at image magnification α1 is enlarged, and image 903 shows how an image taken at image magnification α2 is enlarged. This is what is shown. Images 901 to 903 do not have the same number of horizontal and vertical pixels and cannot be combined as is. The enlarging unit 505 calculates an enlargement ratio γn based on the image magnification ratio βn of each image according to the following (Equation 6), and performs enlargement processing.

Note that since the image 903 has the highest image magnification, it is not necessary to perform enlargement processing.

Although we have explained that the magnification ratio γn when performing magnification is calculated from the image magnification ratio βn, there are other calculation methods, for example, based on the number of horizontal pixels or the number of vertical pixels of each image signal. It is also possible to calculate the rate γn.

In step S206, similarly to the first embodiment, the image signal evaluation unit 104 calculates a contrast value for the image output from the enlarging unit 505, and transmits it to the area selection unit 105.

Note that in the above explanation, the sizes of the objects in each image are made the same by the enlargement process in the enlarger 505, but in this embodiment, the sizes of the objects can be similarly made the same by performing the reduction process. , a similar effect can be obtained.

The following effects can be obtained through the processing described above. When the data transmission rate between the image sensor 101 and the first signal processing unit 102 is higher than the data transmission rate between the first signal processing unit 102 and the second signal processing unit 108, the following procedure is performed. Can be processed. After reducing the number of pixels of an image obtained from the image sensor 101 based on the image magnification, enlargement processing necessary for depth composition is performed to match the size of the subject image in the image. The first signal processing unit 102 selects and transmits data used for depth synthesis processing, thereby suppressing the influence of fluctuations in image magnification from the first signal processing unit 102 to the second signal processing unit 108. Data to be transmitted can be suppressed.

According to the second embodiment, it is possible to suppress data that is not necessary for creating an image for depth compositing, considering the image magnification among the data sent to the second signal processing unit, and further reduce the processing load. Image processing can be performed.

(Third embodiment)
Unlike the first and second embodiments, the third embodiment is an embodiment for reducing variations in the size of objects included in divided regions between images, variations in evaluation results, etc. .

The third embodiment will be described below. Note that descriptions of parts similar to the first embodiment and the second embodiment will be omitted.

FIG. 10 is a block diagram showing the hardware configuration of the digital camera in this embodiment. The image signal evaluation unit 1001 changes the evaluation range of the image signal obtained from the image sensor 101 based on the image magnification sent from the optical system drive unit 502.

The reduction unit 1002 performs reduction processing on the image of the divided area transmitted from the first signal processing unit 102. The reduction process is a process of reducing the number of horizontal pixels and the number of vertical pixels of an image signal by a predetermined magnification. The number of pixels of the image signal output from the reduction unit 1002 is smaller than the number of pixels of the input image signal.

FIG. 11 is a flowchart for explaining depth synthesis processing in this embodiment.

In step S1101, the image signal evaluation unit 1001 evaluates the image signal. Similar to the first embodiment, the image signal evaluation unit 1001 divides the image obtained from the image sensor 101 into a plurality of regions, and calculates a contrast value for each of the plurality of regions.

FIG. 12 is a diagram for explaining region division in this embodiment. FIG. 12 shows how divided areas 1202 and 1203 are set for an image 1201. In the first embodiment, the divided regions are explained so as not to overlap with each other, but in this embodiment, the divided regions 1202 and 1203 are shown to overlap.

If divided regions are set in the same range for each image to be combined, errors will occur in the evaluation results of each divided region due to changes in the magnification of the image signal due to image magnification fluctuations caused by the optical system 501. There is. Therefore, in this embodiment, flexibility with respect to changes in image magnification is ensured by setting divided areas to have overlapping areas according to the amount of variation in image magnification in order to reduce errors.

In step S1102, the area selection unit 105 stores area selection information for selecting a divided area of the image signal to be transmitted to the second signal processing unit 108 based on the evaluation result received from the image signal evaluation unit 1001. The information is transmitted to the control unit 107. The area selection unit 105 selects an area based on the contrast value of each divided area.

In step 1103, the reduction unit 1002 performs reduction processing. The optical system drive unit 502 sends the image magnification to the first control unit 103 , and the first control unit 103 calculates a reduction rate based on the image magnification and sends it to the second control unit 109 . The reduction unit 1002 receives the divided areas of the image output from the first signal processing unit 102, receives a reduction rate for performing reduction processing from the second control unit 109, and performs reduction processing on the image signal. I do.

The first control unit 103 calculates the reduction rate εn based on the following (Equation 7).

Note that the second control section 109 can obtain the image magnification without going through the first control section 103 by the following method. The area selection unit 105 transmits the image magnification from the image signal evaluation unit 1001 to the memory control unit 107 along with the area selection information. When transmitting the divided area data from the first signal processing unit 102 to the second signal processing unit 108, the image magnification is added and transmitted, so that the reduction unit 1002 can obtain the image magnification.

The storage unit 111 stores the divided regions after the reduction process via the depth synthesis unit 110. The subsequent processing of this embodiment is the same as that of the first and second embodiments. However, since each divided area read from the storage unit 111 includes an overlapping area, appropriate deletion processing is required when storing.

Note that in the above explanation, the sizes of the objects in each image are made the same by the reduction process in the reduction unit 1002, but in this embodiment, the sizes of the objects can be similarly made the same by the enlargement process. , a similar effect can be obtained.

The following effects can be obtained through the processing described above. When the data transmission rate between the image sensor 101 and the first signal processing unit 102 is higher than the data transmission rate between the first signal processing unit 102 and the second signal processing unit 108, the following procedure is performed. Can be processed.

The image signal evaluation unit 1001 changes the divided regions based on the image magnification obtained from the optical system driving unit 502, thereby suppressing the influence of the change in image magnification caused by driving the optical system 501. At the same time, data transmitted from the first signal processing section 102 to the second signal processing section 108 can be suppressed.

Note that in this embodiment, the image signal evaluation unit 1001 may change the size of the divided area according to the image magnification obtained from the optical system drive unit 502. FIG. 13 is a diagram for explaining the setting of divided regions according to the image magnification in this embodiment. FIG. 13 shows how divided areas are set for three images 1301, 1302, and 1303. Image 1301 is an image captured at image magnification α0, image 1302 is an image captured at image magnification α1, and image 1303 is an image captured at image magnification α2. The area separated by the dotted line shown in FIG. 13 indicates one divided area, and it is shown that the horizontal size and vertical size of one divided area are changed according to the image magnification. ing. When the divided area of the image 1303 is equally divided into 4 horizontal parts and 3 vertical parts, the horizontal size of the divided area of the image 1301 is adjusted according to the ratio of the image magnification based on the divided area of the image 1303 with the largest image magnification. Make it smaller. Even if the size of the image of the subject included in the image changes, the area being evaluated between images does not change, making it possible to improve the reliability of the evaluation results.

According to the third embodiment, it is possible to suppress data that is not necessary for creating an image for depth compositing, taking into account the variation in evaluation between images, among the data sent to the second signal processing unit, and to Image processing that can reduce processing load can be performed.

(Other embodiments)
Although the above embodiments have been described based on implementation with a digital camera, the present invention is not limited to digital cameras. For example, it may be implemented using a mobile device with a built-in image sensor, or a network camera that can capture images. Further, the captured image may be processed by a processing device other than a digital camera in the manner described above.

Note that the present invention provides a system or device with a program that implements one or more functions of the above-described embodiments via a network or a storage medium, and one or more processors in a computer of the system or device executes the program. This can also be realized by reading and activating the process. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101 Imaging device 102 First signal processing unit 103 First control unit 104 Image signal evaluation unit 105 Area selection unit 106 Frame memory 107 Memory control unit 108 Second signal processing unit 109 Second control unit 110 Depth synthesis unit 111 Storage unit 501 Optical system 502 Optical system drive unit 503 Cutout area determination unit 504 Cutout unit 505 Enlargement unit 1001 Image signal evaluation unit 1002 Reduction unit

Claims (14)

an acquisition means for acquiring a plurality of images;
a first processing means that receives signals of the plurality of images from the acquisition means, divides the plurality of images, and sets a plurality of divided regions in each of the plurality of images;
a second processing means that receives signals from some of the plurality of divided regions from the first processing means and generates a depth-synthesized image;
The first processing means and the second processing means are each composed of individual blocks,
The first processing means evaluates the divided areas in each of the plurality of images, and selects the divided area based on the result of the evaluation,
The first processing means calculates an evaluation value for each of at least some of the plurality of divided regions as the evaluation,
The second processing means receives signals of the divided regions selected by the first processing means in order of increasing evaluation value,
An image processing apparatus characterized in that a transmission rate at which the second processing means receives signals from the first processing means is lower than a transmission rate at which the first processing means receives signals from the acquisition means. The image processing apparatus according to claim 1, wherein the first processing means calculates a contrast value of the divided area and performs the evaluation based on the contrast value. The image processing apparatus according to claim 1, wherein the first processing means performs frequency analysis on the divided area and performs the evaluation based on the result of the frequency analysis. The second processing means receives at least one of the plurality of divided regions, none of which are located at the same position and selected by the first processing means. The image processing apparatus according to any one of claims 1 to 3. The image processing apparatus according to any one of claims 1 to 4, wherein the first processing means sets the divided areas so that they overlap with each other. 6. The image processing apparatus according to claim 1, wherein the first processing means performs processing of enlarging or reducing the image. 7. The image processing apparatus according to claim 6 , wherein the first processing means performs the enlargement or reduction processing based on information about an optical system in which the image is captured. 7. The image processing apparatus according to claim 6, wherein the first processing means changes the size of the divided area based on information about an optical system in which the image was captured. The image processing apparatus according to claim 7 or 8, wherein the information on the optical system includes a focus position when the image was captured. 9. The image processing apparatus according to claim 7, wherein the information on the optical system includes a position of a focus lens when the image was captured. 11. The image processing apparatus according to claim 1, wherein the first processing means performs a cutting process on the plurality of images. an imaging means for capturing a plurality of images;
a first processing means that receives signals of the plurality of images from the imaging means, divides the plurality of images, and sets a plurality of divided regions in each of the plurality of images;
a second processing means that receives signals from some of the plurality of divided regions from the first processing means and generates a depth-synthesized image;
The first processing means and the second processing means are each composed of individual blocks,
The first processing means evaluates the divided areas in each of the plurality of images, and selects the divided area based on the result of the evaluation,
The first processing means calculates an evaluation value for each of at least some of the plurality of divided regions as the evaluation,
The second processing means receives signals of the divided regions selected by the first processing means in order of increasing evaluation value,
An imaging device characterized in that a transmission rate at which the second processing means receives signals from the first processing means is lower than a transmission rate at which the first processing means receives signals from the imaging means. an acquisition step in which the acquisition means acquires a plurality of images;
A first processing step in which a first processing means receives signals of the plurality of images from the acquisition means, divides the plurality of images, and sets a plurality of divided regions in each of the plurality of images. and,
a second processing step in which a second processing means receives signals from some of the plurality of divided regions from the first processing means and generates a depth composite image;
The first processing means and the second processing means are each composed of individual blocks,
In the first processing step, the first processing means evaluates the divided area in each of the plurality of images, and selects the divided area based on the result of the evaluation,
The first processing means calculates an evaluation value for each of at least some of the plurality of divided regions as the evaluation,
In the second processing step, the second processing means receives signals of the divided regions selected by the first processing means in order of increasing evaluation value,
An image processing method characterized in that a transmission rate at which the second processing means receives signals from the first processing means is lower than a transmission rate at which the first processing means receives signals from the acquisition means. A computer program that causes a computer to operate an image processing device,
an acquisition step in which the acquisition means acquires a plurality of images;
A first processing step in which a first processing means receives signals of the plurality of images from the acquisition means, divides the plurality of images, and sets a plurality of divided regions in each of the plurality of images. and,
a second processing step in which a second processing means receives signals of some of the plurality of divided regions from the first processing means and generates a depth composite image;
The first processing means and the second processing means are each composed of individual blocks,
In the first processing step, the first processing means evaluates the divided area in each of the plurality of images, and selects the divided area based on the result of the evaluation,
The first processing means calculates an evaluation value for each of at least some of the plurality of divided regions as the evaluation,
In the second processing step, the second processing means receives signals of the divided regions selected by the first processing means in order of increasing evaluation value,
A program characterized in that a transmission rate at which the second processing means receives a signal from the first processing means is lower than a transmission rate at which the first processing means receives a signal from the acquisition means.

Images