JP7549453B2 - Image processing device and control method thereof, imaging device, program, and storage medium - Google Patents

Description

The present invention relates to a technology for detecting a subject from an image.

Technology for detecting a subject from a captured image has been known for some time.

For example, Patent Document 1 discloses a technique for selecting whether or not to crop a specified area of a captured image when extracting a subject area from a captured image, depending on the detection results of the previous captured image. By using only the cropped area, rather than the entire image, as the input image for subject detection, it is possible to improve the detection performance of small subjects.

JP 2008-172368 A

In Patent Document 1, the crop range is determined based on the size and position of at least one of the previously detected subjects. However, with this method, even if the user is not interested in subjects other than the current main subject, the crop range may be widened in an attempt to detect subjects other than the main subject. In this case, the detection rate of the main subject may be relatively sacrificed.

Conversely, even if you want to detect a subject other than the current main subject and switch to a preferred subject as the new main subject, cropping may occur in a small area, resulting in the subject other than the main subject not being detected.

The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide an image processing device capable of performing subject detection that is more in line with the user's intentions.

The image processing device of the present invention comprises a detection means for detecting a subject from an image captured by an imaging means, a first setting means for setting a candidate area which is a candidate area in the image in which the detection means detects the subject, a second setting means for setting a cut-out area for detecting a target subject in the candidate area, and a third setting means for setting a degree of transition which determines the extent to which the operation of the imaging device will continue to follow the detected current main subject or will switch to another subject , wherein the candidate area is set to include the cut-out area and the area of the current main subject, and the second setting means sets the position of the cut-out area relative to the candidate area based on the degree of transition.

The present invention makes it possible to provide an image processing device capable of performing subject detection that is more in line with the user's intentions.

1 is a block diagram showing the configuration of an image pickup apparatus according to an embodiment of the present invention; 4A to 4C are diagrams illustrating input images to a subject detection unit. 1 is a flowchart showing an overall processing procedure according to an embodiment. FIG. 4 is a schematic diagram showing a crop region according to an embodiment.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

(Configuration of the imaging device)
1 is a block diagram showing the configuration of an image capturing apparatus 100 according to an embodiment of the present invention. The configuration of the image capturing apparatus 100 will be described with reference to FIG.

The imaging device 100 may be, but is not limited to, a digital still camera or video camera that can capture an image of a subject and record video or still image data on various media such as tape, solid-state memory, optical disks, and magnetic disks. Each unit in the imaging device 100 is connected via a bus 160. Each unit is controlled by a CPU 151 (central processing unit).

The lens unit 101 is configured with a fixed first group lens 102, a zoom lens (second group lens) 111, an aperture 103, a fixed third group lens 121, and a focus lens 131. The focus lens 131 is a lens for adjusting focus, and although it is shown simply as a single lens in FIG. 1, it is usually configured with multiple lenses.

The aperture control unit 105 adjusts the aperture 103 opening diameter and adjusts the amount of light during shooting by driving the aperture 103 via the aperture motor 104 (AM) in accordance with instructions from the CPU 151. The zoom control unit 113 changes the focal length by driving the zoom lens 111 via the zoom motor 112 (ZM). The focus control unit 133 determines the drive amount for driving the focus motor 132 (FM) based on the amount of deviation (defocus amount) in the focus direction of the lens unit 101. In addition, the focus control unit 133 controls the focus adjustment state by driving the focus lens 131 via the focus motor 132 (FM). AF (autofocus) control is realized by the movement control of the focus lens 131 by the focus control unit 133 and the focus motor 132.

The subject image formed on the image sensor 141 via the lens unit 101 is converted into an electrical signal by the image sensor 141. The image sensor 141 is a photoelectric conversion element that converts the subject image (optical image) into an electrical signal. The image sensor 141 is configured with m pixels in the horizontal direction and n pixels in the vertical direction of light receiving elements. The image signal formed on the image sensor 141 and photoelectrically converted is arranged as image data by the image signal processing unit 142, and an image on the imaging surface can be obtained.

Image data output from the imaging signal processing unit 142 is sent to the imaging control unit 143 and temporarily stored in the RAM 154 (random access memory). The image data stored in the RAM 154 is compressed in the image compression/decompression unit 153 and then recorded in the image recording medium 157. In parallel with this, the image data stored in the RAM 154 is sent to the image processing unit 152. The image processing unit 152 processes the image signal, performs processing such as reducing/enlarging the image data to an optimal size and calculating the similarity between image data. By appropriately sending and displaying the image data processed to the optimal size on the monitor display 150, it is possible to display a preview image or a through image, and it is also possible to superimpose the subject detection result of the subject detection unit 162 on the image data. In addition, by using the RAM 154 as a ring buffer, it is possible to buffer multiple image data captured within a predetermined period, the detection result of the subject detection unit 162 corresponding to each image data, and the position and orientation change of the imaging device acquired by the position and orientation change acquisition unit 161.

The operation switch 156 is an input interface including a touch panel and buttons, and various operations can be performed by selecting and operating various function icons displayed on the monitor display 150.

The CPU 151 determines the accumulation time of the image sensor 141 and the gain setting value when outputting from the image sensor 141 to the image signal processing unit 142 based on instructions from the user input from the operation switch 156 or the magnitude of the pixel signal of the image data temporarily accumulated in the RAM 154. The image sensor control unit 143 receives instructions on the accumulation time and gain setting value from the CPU 151 and controls the image sensor 141.

The subject detection unit 162 uses the image signal to determine the area where a specific subject exists (main subject area). The subject detection unit 162 detects the subject using a known method such as pattern matching. The focus control unit 133 can perform AF control (automatic focus adjustment operation) for a specific subject area. The aperture control unit 105 performs exposure control (exposure adjustment operation) using the luminance value of the specific subject area. The image processing unit 152 performs gamma correction, white balance processing, etc. based on the subject area.

The monitor display (display unit) 150 displays images and subject detection results as rectangles or the like. The battery 159 is appropriately managed by the power management unit 158, and provides a stable power supply to the entire imaging device 100.

Flash memory 155 stores control programs necessary for the operation of imaging device 100 and parameters used for the operation of each part. When imaging device 100 is started by a user's operation (when the power is switched from OFF to ON), the control programs and parameters stored in flash memory 155 are loaded into a part of RAM 154. CPU 151 controls the operation of imaging device 100 according to the control programs and constants loaded into RAM 154.

The position and orientation change acquisition unit 161 is composed of a position and orientation sensor such as a gyro sensor, an acceleration sensor, an electronic compass, etc., and measures the position and orientation change of the imaging device 100 relative to the shooting scene. The acquired position and orientation change is stored in the RAM 154.

The depth calculation unit 163 calculates the distance from the imaging device to any subject in the image. The generated distance information is stored in the RAM 154 and is referenced by the image processing unit 152.

Note that the configuration described above is just one example of the configuration of an imaging device.

(Prerequisite for subject detection)
Next, the premise of this embodiment when generating an input image for the subject detection unit 162 and the problem that occurs when the method of this embodiment is not applied will be described with reference to Fig. 2. Here, it is assumed that the subject detection unit 162 can detect dogs, for example.

Figure 2(a) shows a high-resolution first image generated by the imaging signal processing unit 142. Hereinafter, this will be referred to as image a. A second image with lower resolution than image a, for example with half the number of pixels vertically and horizontally, is generated and used as input to the subject detection unit 162. Figure 2(b) shows the result of simply reducing the entire image a to generate the second image. This will be referred to as image b. The detection performance (detection rate and degree of area coincidence) of the subject detection unit 162 may depend on the size of the subject on the input image, and if image b is used as input to the subject detection unit 162, the detection performance of the subject, which has become smaller as it is reduced, will be poor.

Figure 2 (c) shows the result of generating a second image by cropping (a process of cutting out a part of the image from the full angle of view) the periphery of a main subject when it is present. This is called image c. Unlike image b, since it has not been reduced in size, the resolution of the main subject is the same as that of image a, and the degradation of detection performance associated with reduction can be suppressed. As a result, the main subject can be stably detected and continuously tracked. On the other hand, there is an issue that other subjects that are not present near the main subject cannot be detected.

In other words, there is a trade-off between including many subjects in the detection range of the subject detection unit 162 by using image b as the input to the subject detection unit 162, and being able to detect even a small main subject with high accuracy by using image c as the input to the subject detection unit 162.

(About focus candidate areas and transition degree)
In this embodiment, the user sets a focusing candidate area. Here, the focusing candidate area refers to an area in which a target subject present in the area is preferentially selected as a main subject and focused on by AF (autofocus).

The user also sets the degree of subject transition before shooting. The degree of transition is a value that indicates the degree to which, when a main subject is being tracked (tracked) and another subject is present during shooting, it is easy to continue tracking the current main subject or to select the other subject as the new main subject. The degree of transition can be set, for example, to two values, high and low, or to multiple levels. Note that a higher value for the degree of transition means that it is easier to select another subject as the new main subject (easier to transition).

Let us now assume that if the condition "there is a subject closer to the center of the focus candidate area than the current main subject" continues for a specified time, then that subject is selected as the new main subject. At this time, this specified time can be changed according to the degree of transition. For example, if the degree of transition is the greatest that can be set, then the above-mentioned specified time is shortened and the transition is performed immediately. If the degree of transition is the smallest that can be set, then the transition is not performed unless the main subject is lost. In all other cases, this specified time can be set to something proportional to the degree of transition.

Although an example of the positional relationship of the subject is given as a condition for transfer, this is not a limitation. For example, it is possible to design the system so that if there is another subject larger than the main subject, transfer will occur only if the degree of transfer is equal to or greater than a threshold value.

(Overall processing flow)
The flow of the overall process in this embodiment will be described with reference to Figures 3 and 4. In this embodiment, the degree of subject transition can be set to three levels, large, medium, and small, and it is assumed that the user sets this in advance when taking a photograph.

First, in step S200, the CPU 151 supplies an input image from the imaging control unit 143 to each unit.

Next, in step S201, the CPU 151 determines whether or not the main subject is being tracked (whether or not the main subject exists). If the main subject is being tracked, the process proceeds to step S202, and if the main subject is not being tracked, the process proceeds to step S208.

Next, in step S202, the CPU 151 refers to the degree of transition previously set by the user. If the degree of transition is small, in other words, if the setting indicates that the user wishes to continue tracking the current main subject for as long as possible, the process proceeds to step S203. If the degree of transition is medium, the process proceeds to step S204; if it is large, the process proceeds to step S205. In steps S203, S204, and S205, the center of the area to be cropped is determined.

In step S203, the CPU 151 performs cropping around the center of the main subject region, as shown in FIG. 4(a). Here, the detection result of the previous frame may be used as is for the main subject region, or simple template matching may be performed on the detection result of the previous frame to roughly estimate the main subject region in the current frame. By cropping around the center of the region thus determined, it is possible to improve the detection accuracy of the same subject region in the current frame as in the previous frame. On the other hand, it becomes impossible to detect subjects that do not exist in the vicinity. However, this can be said to be consistent with the user's settings, which place importance on a small degree of transition, i.e., on detecting the current main subject as stably as possible.

In step S204, the CPU 151 sets the midpoint between the center of the main subject area and the center of the focus candidate area as the center of the crop, as shown in FIG. 4B. This is to include the current main subject in the detection range as much as possible, while also detecting other subjects that exist in the focus candidate area, and in some cases, to make it possible to switch (transfer) the focus target to that subject.

In step S205, the CPU 151 sets the center of the focus candidate area as the center of the crop, as shown in FIG. 4(c). This is because the detection rate of the current main subject is not important, and priority is given to the detection of other subjects that exist in the focus candidate area.

In step S206, the CPU 151 determines the size of the crop region based on the size of the main subject. Details will be described later. Then, the process proceeds to step S207.

In step S207, the CPU 151 scales the crop area to match the input image size of the subject detection unit 162 and inputs it to the subject detection unit 162.

In step S208, the CPU 151 scales the entire angle of view range to match the input image size of the subject detection unit 162, and inputs it to the subject detection unit 162.

(size of crop area)
In this embodiment, the size (number of pixels vertically and horizontally) of the area to be cropped in image a (the image shown in FIG. 2A) depends on the size of the main subject, and is determined so as to be enlarged or reduced to a size that has been adjusted in advance and which the subject detection unit 162 is most comfortable with.

For example, consider a case where it is known that the desired detection rate will be satisfied when the width of the main subject in the input image to subject detection unit 162 is 20% or more of the angle of view. If the width of the main subject in image a is 20% of the angle of view, the entire image a is reduced as is as shown in (b) of Figure 2 and used as the input image to subject detection unit 162. If the width of the main subject in image a is 16% of the angle of view, an area that is 80% vertically and horizontally of image a is cropped and reduced to use as the input image to subject detection unit 162. This makes it possible to make the size of the main subject approximately 20% of the angle of view in the input image to subject detection unit 162.

If the crop size determined according to the above method falls below the input image size of the subject detection unit 162, the image will need to be enlarged, but to avoid this, a lower limit may be set for the crop size.

In the above embodiment, the size of the crop area is determined based on the size of the main subject, but it can also be determined based on the degree of transition.

Specifically, a lower limit is set for the crop size, and the higher the degree of transition, the larger the lower limit is set (so that a wider area is included in the crop area). In other words, when the degree of transition is small, a crop size that makes it easy to detect the main subject is determined, whereas when the degree of transition is large, a certain range or more is included in the crop area even if the main subject is small. This makes it easier for other subjects to be included in the target of the subject detection unit 162.

In addition, if the degree of transition is set to a medium level by default, a subject area that is a preset distance or more from the center of the focus candidate area may not be selected as a subject area to transition to, so as to avoid transitioning to a subject that is far away from the current position of the main subject.

In addition, when multiple subjects are detected, the monitor display 150 may display the area of at least one subject other than the main subject.

As described above, according to this embodiment, it is possible to determine a crop area specialized for detecting the main subject and a crop area that can also detect subjects other than the main subject based on the user's settings.

Other Embodiments
The present invention can also be realized by a process in which a program for realizing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for realizing one or more of the functions.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

100: imaging device, 101: lens unit, 133: focus control unit, 141: imaging element, 142: imaging signal processing unit, 143: imaging control unit, 151: CPU, 152: image processing unit, 162: subject detection unit

Claims (17)

A detection means for detecting a subject from an image captured by the imaging means;
a first setting means for setting a candidate area in the image, the candidate area being an area that is a candidate for detecting a subject by the detection means;
a second setting means for setting an extraction region for detecting a target subject in the candidate region;
a third setting means for setting a transition degree that determines a degree to which the operation of the imaging device should continue to follow the detected current main subject or should be switched to another subject;
an image processing device comprising: a first setting unit for setting a position of the cut-out area relative to the candidate area based on the degree of transition; and a second setting unit for setting a position of the cut-out area relative to the candidate area based on the degree of transition. The image processing device according to claim 1, characterized in that the first setting means sets the candidate area according to an instruction from a user. The image processing device according to claim 1 or 2, characterized in that the third setting means sets the degree of transition according to an instruction from a user. The image processing device according to any one of claims 1 to 3, characterized in that the cut-out area is an area for making it easier to detect the target subject in the candidate area. The image processing device according to any one of claims 1 to 4, characterized in that the operation of the imaging device is a focus adjustment operation. The image processing device according to any one of claims 1 to 5, characterized in that the operation of the imaging device is an exposure adjustment operation. The image processing device according to any one of claims 1 to 6, characterized in that the second setting means sets the position of the center of the cut-out area based on the center of the area of the current main subject and the center of the candidate area. The image processing device according to claim 7, characterized in that the second setting means sets the center position of the cutout area to a position closer to the center of the candidate area as the degree of transition increases. The image processing device according to claim 7 or 8, characterized in that the second setting means sets the center position of the cutout area to a position closer to the center of the current main subject area the smaller the degree of transition is. The image processing device according to any one of claims 1 to 9, characterized in that the second setting means sets the size of the cutout area according to the size of the current main subject. The image processing device according to claim 10, characterized in that the second setting means sets the size of the cutout area to be larger as the degree of transition increases. The image processing device according to any one of claims 1 to 11, characterized in that, when the degree of transition is a default setting, a subject that is a default distance or more from the center of the candidate area is not considered to be a subject to be transitioned. The image processing device according to any one of claims 1 to 12, further comprising a display means for displaying the image and the area of the subject, the display means displaying the area of at least one subject other than the current main subject when multiple subjects are detected. An image processing device according to any one of claims 1 to 13,
The imaging means;
A focus adjustment means for adjusting the focus on a subject;
An imaging device comprising: a detection step of detecting a subject from an image captured by an imaging means;
a first setting step of setting a candidate area in the image, the candidate area being an area that is a candidate for detecting a subject by the detection step;
a second setting step of setting an extraction region for detecting a target subject in the candidate region;
a third setting step of setting a transition degree that determines the degree to which the operation of the imaging device should continue to follow the detected current main subject or should be switched to another subject;
a control method for an image processing device, characterized in that the candidate area is set to include the cut-out area and the area of the current main subject, and in the second setting step, a position of the cut-out area relative to the candidate area is set based on the degree of transition. A program for causing a computer to execute each step of the control method described in claim 15. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method described in claim 15.

Images