JP6931369B2 - Image processing device and image processing method, and imaging device - Google Patents

Description

The present invention relates to an image processing apparatus and an image processing method, and an imaging apparatus, and particularly to a subject detection technique.

A subject detection technique that automatically detects a specific subject pattern from an image is very useful. Patent Document 1 discloses an imaging device that detects a region corresponding to a specific subject pattern such as a person's face from a captured image and optimizes the focus and exposure to the detected region.

Further, it is known that a subject in an image is learned and recognized by using a technique called deep learning (Non-Patent Document 1). Convolutional neural networks (CNNs) are a typical method of deep learning. Generally, a CNN has a multi-layer structure in which a convolutional layer that spatially integrates local features of an image, a pooling layer or subsampling layer that compresses features in the spatial direction, a fully connected layer, an output layer, and the like are combined. Has. Since CNN can acquire a complicated feature expression through stepwise feature conversion by a multi-layer structure, it is possible to perform category recognition and subject detection of a subject in an image with high accuracy based on the feature expression.

Japanese Unexamined Patent Publication No. 2005-318554

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton, "ImageNet classification with deep convolutional neural networks", NIPS'12 Proceedings of the 25th International Conference on Neural Information Processing Systems --Volume 1, PP.1097-1105

In an imaging device that focuses on a subject area detected from a captured image as in Patent Document 1, the requirements for subject detection differ before and after the determination of the main subject (the subject to be focused). Before determining the main subject, it is better to be able to detect the subject to be detected without omission regardless of the size of the blur. On the other hand, after the main subject is determined, it is not necessary to detect a subject with a large blur.

However, conventionally, in subject detection using machine learning, a method for realizing subject detection suitable before determining the main subject and subject detection suitable after determining the main subject has not been proposed. In other words, no method has been proposed for detecting a subject suitable for the amount of blur (or sharpness) of the main subject. The amount of blur and sharpness of the main subject are not limited to the degree of focusing, but also change depending on the degree of subject blur and camera shake, the magnitude of image noise, and the like.

The present invention has been made in view of such problems of the prior art. An object of the present invention is to provide an image processing device and an image processing method capable of realizing subject detection suitable for the amount of blurring and sharpness of a main subject in subject detection using machine learning, and an imaging device.

The above-mentioned purpose is to determine a main subject from a subject detection means that applies subject detection processing to image data and a subject detected by the subject detection means using parameters generated based on machine learning. It has a determination means, a storage means for storing a plurality of parameters used for subject detection processing, and a selection means for selecting a parameter to be used in the subject detection means from the parameters stored in the storage means. It is characterized by selecting different parameters suitable for each case depending on whether the focus is adjusted so that the area of the main subject is determined and the area of the main subject is in focus and the case where the main subject is not determined. This is achieved by the image processing device.

According to the present invention, in subject detection using machine learning, an image processing device and an image processing method capable of realizing subject detection suitable before determining the main subject and subject detection suitable after determining the main subject, and an image processing method. An imaging device can be provided. Further, it is possible to provide an image processing device, an image processing method, and an imaging device capable of realizing subject detection suitable for the movement state of the main subject, the movement state of the image pickup device, and the shooting conditions.

The schematic vertical sectional view of the digital single-lens reflex camera as an example of the image processing apparatus which concerns on 1st Embodiment. The block diagram which shows the functional structure example of the digital single-lens reflex camera which concerns on 1st Embodiment. The flowchart regarding the outline of the shooting operation which concerns on 1st Embodiment. The flowchart regarding the subject detection process which concerns on 1st Embodiment. The schematic diagram which shows the structural example of the CNN used by the subject detection part which concerns on 1st Embodiment. The schematic diagram which shows the structure of a part of CNN of FIG. The block diagram which shows the functional structure example of the digital single-lens reflex camera which concerns on 2nd to 4th Embodiment. The schematic diagram which shows an example of the motion vector calculation process which concerns on 2nd Embodiment. The flowchart regarding the subject detection process which concerns on 2nd Embodiment. The flowchart regarding the subject detection process which concerns on 3rd Embodiment. The flowchart regarding the subject detection process which concerns on 4th Embodiment.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Further, although a plurality of features are described in the embodiment, not all of them are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are given the same reference numbers, and duplicate explanations are omitted.

In the following embodiment, the case where the present invention is implemented by a digital single-lens reflex camera (DSLR) will be described. However, the present invention can be implemented in any electronic device capable of handling image data, and the digital single-lens reflex camera is only an example of the image processing device according to the present invention. Electronic devices to which the present invention can be implemented include, but are not limited to, for example, personal computers, smartphones, tablet terminals, game machines, robots, and the like.

<< First Embodiment >>
● (Configuration of imaging device)
FIG. 1 is a vertical cross-sectional view of a digital single-lens reflex camera (DSLR) 100 according to the present embodiment. Further, FIG. 2 is a block diagram showing a functional configuration example of the DSLR100. The same reference number refers to the same component throughout the figure.

The DSLR 100 has a main body 101 and a photographing lens 102 (interchangeable lens) that can be attached to and detached from the main body 101. A mount contact group 115 is provided on each of the attachment / detachment portion (mount) of the main body 101 and the photographing lens 102. When the photographing lens 102 is attached to the main body 101, the mount contact group 115 comes into contact with each other, and an electrical connection between the photographing lens 102 and the main body 101 is established.

The system control unit 201 has one or more programmable processors, a ROM 2011, and a RAM 2012, and controls the operations of the main body 101 and the photographing lens 102 by reading the program stored in the ROM 2011 into the RAM 2012 and executing the program. In addition to the program executed by the system control unit 201, various setting values, GUI data, and the like are stored in the ROM 2011.

The photographing lens 102 is provided with a focus lens 113 for adjusting the focusing distance and a diaphragm 114 (and a motor or an actuator for driving them) for adjusting the amount of light incident on the main body 101. The drive of the focus lens 113 and the aperture 114 is controlled by the camera body 101 through the mount contact group 115.

The main mirror 103 and the sub mirror 104 form a quick return mirror. The reflectance (transmittance) of a part of the main mirror 103 is controlled in order to separate the luminous flux incident from the photographing lens 102 into the luminous flux toward the finder optical system (upper part of the figure) and the luminous flux toward the sub mirror 104. ..

FIG. 1 shows a state when the optical viewfinder is used (when not photographed), and the main mirror 103 is located in the optical path of the luminous flux incident from the photographing lens 102. In this state, the reflected light of the main mirror 103 is incident on the finder optical system, and the luminous flux bent by the pentaprism 107 is emitted from the eyepiece 109. Therefore, the user can see the optical subject image by looking into the eyepiece 109.

Further, the transmitted light of the main mirror 103 is reflected by the sub mirror 104 and is incident on the AF sensor 105. The AF sensor 105 forms a secondary image plane of the photographing lens 102 on the line sensor, and generates a pair of image signals (focus detection signals) that can be used for focus detection by the phase difference detection method. The generated focus detection signal is transmitted to the system control unit 201. The system control unit 201 obtains the defocus amount of the focus lens 113 using the focus detection signal, and controls the drive direction and the drive amount of the focus lens 113 based on the defocus amount.

The focus plate 106 is arranged on the planned image plane of the photographing lens 102 in the finder optical system. The user looking into the eyepiece 109 observes the optical image formed on the focus plate 106. In addition to the optical image, shooting information such as shutter speed and aperture value can also be provided.

The photometric sensor 108 generates an image signal (exposure control signal) from the incident luminous flux and transmits it to the system control unit 201. The system control unit 201 performs automatic exposure control using the received exposure control signal, and controls subject detection by the subject detection unit 204, which will be described later. The photometric sensor 108 is an image sensor in which pixels including a photoelectric conversion unit are arranged two-dimensionally. The subject detection unit 204 applies the subject detection process to the image signal output by the photometric sensor 108, but the subject detection process may be applied to the image signal output by the image sensor 111.

When the image sensor 111 is exposed, the main mirror 103 and the sub mirror 104 move out of the optical path of the luminous flux incident from the photographing lens 102. Further, the focal plane shutter 110 (hereinafter, simply referred to as a shutter) opens.

Pixels including a photoelectric conversion unit are arranged two-dimensionally in the image pickup device 111, and the subject optical image formed by the photographing lens 102 is photoelectrically converted by each pixel, and the image signal is transmitted to the system control unit 201. The system control unit 201 generates image data from the received image signal, stores it in the image storage unit 202, and displays it on a display unit 112 such as an LCD. Further, the image data generated by the image sensor 111 may be supplied to the subject detection unit 204 for subject detection. The system control unit 201 may perform focus detection by a contrast method using image data. The system control unit 201 may perform focus detection by a contrast method using image data.

The operation unit 203 is a general term for a group of input devices provided with the main body 101 and the photographing lens 102 and which can be operated by the user. Specific examples of the input device included in the operation unit 203 include, but are not limited to, a release button, a power switch, a direction key, an enter button, a menu button, an operation mode selection dial, and the like. The operation of the operation unit 203 is detected by the system control unit 201. The release button has a switch SW1 that is turned on by a half-press operation and a switch SW2 that is turned on by a full-press operation.

For example, when the half-press operation of the release button (SW1 is turned on) is detected, the system control unit 201 starts the still image shooting preparation operation. The shooting preparation operation is, for example, an operation related to automatic focus detection (AF) and automatic exposure control (AE). Further, when the full press operation of the release button (SW2 is turned on) is detected, the system control unit 201 executes a still image shooting and recording operation. The system control unit 201 displays the image obtained by shooting on the display unit 112 for a certain period of time.

Further, at the time of moving image shooting (during shooting standby state or moving image recording), the system control unit 201 displays the moving image obtained by shooting on the display unit 112 in real time, so that the display unit 112 is displayed in the electronic viewfinder (EVF). To function as. A moving image and a frame image thereof displayed when the display unit 112 functions as an EVF are referred to as a live view image or a through image. Whether to shoot a still image or a moving image can be selected through the operation unit 203, and the system control unit 201 switches the control method of the camera body 101 and the shooting lens 102 between the time of shooting a still image and the time of shooting a moving image.

The subject detection unit 204 is composed of a GPU (Graphic Processing Unit). The GPU is originally a processor for image processing, but since it has a plurality of product-sum arithmetic units and is good at matrix calculation, it is often used as a processor for performing learning processing. The GPU is also generally used in the process of performing deep learning. For example, as the subject detection unit 204, NVIDIA's Jetson TX2 model can be used. An FPGA (Field-Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or the like may be used as the subject detection unit 204.

The subject detection unit 204 applies the subject detection process to the supplied image data by using one learning model selected by the system control unit 201 from the learning models stored in the learning model storage unit 205. The details of the subject detection process will be described later. The learning model storage unit 205 may be, for example, a rewritable non-volatile memory and may be a part of the ROM 2011. In the present embodiment, the learning model storage unit stores the detection learning models 206 and 207 according to the characteristics required for subject detection. The system control unit 201 determines the region of the main subject from the region of the subject detected by the subject detection unit 204, and optimizes the focus and exposure to the region of the main subject. Specifically, the exposure conditions can be determined so that the area of the main subject becomes an appropriate exposure, and the focus can be adjusted so that the area of the main subject is in focus, but the present invention is not limited to these.

(Dictionary switching in subject detection)
The DSLR 100 (system control unit 201) of the present embodiment uses the area of the subject detected by the subject detection unit 204 for determining the exposure condition and detecting the focus. More specifically, the system control unit 201 determines the main subject from the subjects detected by the subject detection unit 204, and uses the region of the main subject for determining the exposure condition and detecting the focus. Therefore, the characteristics required for subject detection differ before and after the determination of the main subject. The main subject may be determined by the system control unit 201 according to predetermined conditions (for example, the position and size of the subject area), or may be determined according to the user's selection.

The image taken before the determination of the main subject is, for example, an image whose focus is adjusted so that the center of the screen is in focus, and the subject that the user wants to be the main subject may not be in focus. Therefore, in the subject detection performed before the determination of the main subject, it is required that the subject to be detected can be detected even if the amount of blur is large. In other words, subject detection that prioritizes suppression of detection omissions rather than suppression of false positives is required.

On the other hand, once the main subject is determined, focus detection is performed so that the area of the main subject is in focus thereafter, so that the amount of blurring of the main subject in the image obtained thereafter becomes small. Therefore, after the main subject is determined, the value of detecting a subject having a large amount of blur is reduced even if the subject is a detection target. In other words, subject detection that prioritizes suppression of false positives rather than suppression of detection omissions is required.

As an attempt is made to detect a subject having a large amount of blur, the probability of erroneously detecting an area in which the target subject is not captured tends to increase. This is because as the amount of blurring of the subject increases, the variation of the image similar to the subject increases and the characteristics of the subject are lost. In this way, it is important to detect from a subject with a small amount of blur to a subject with a large amount of blur before determining the main subject, and after determining the main subject, false detection is suppressed and the subject with a small amount of blur is accurately detected. It is important to detect it.

The subject detection unit 204 of the present embodiment applies subject detection using processing parameters based on a learning model generated in advance through machine learning to an image. In the present embodiment, the learning model storage unit 205 stores the detection learning model 206 for subject detection before the main subject is determined and the detection learning model 207 for subject detection after the main subject is determined. The detection learning model 206 for subject detection before determining the main subject is generated by machine learning using an image of a large subject from a subject having a small amount of blur. Further, the detection learning model 207 for subject detection after determining the main subject is generated by machine learning that emphasizes a subject image having a small amount of blur.

Then, the system control unit 201 switches the subject detection characteristics of the subject detection unit 204 by switching the learning model used by the subject detection unit 204 before and after the determination of the main subject. The system control unit 201 uses the detection learning model 206 for subject detection before the main subject is determined before the main subject is determined, and uses the detection learning model 207 for subject detection after the main subject is determined after the main subject is determined. Switch to.

(Imaging operation)
Next, the photographing operation of the DSLR 100 of the present embodiment will be described with reference to FIGS. 3 and 4.
FIG. 3 is a flowchart relating to the outline of the shooting operation, and the processing of each step is realized by executing the program read from the ROM 2011 into the RAM 2012 by the programmable processor of the system control unit 201. Here, it is assumed that the power supply of the main body 101 is ON and the shooting standby state is set. In the shooting standby state, the system control unit 201 executes a predetermined operation such as shooting a moving image and displaying the obtained moving image on the display unit 112 to function as an EVF.

In S301, the system control unit 201 detects the states of the switches SW1 and SW2 of the release button, and if any of the switches SW1 and SW2 is on, the process proceeds to S302. On the other hand, if both the switches SW1 and SW2 are off, the system control unit 201 repeatedly executes S301.

In S302, the system control unit 201 performs an exposure process (charge accumulation) of the photometric sensor 108. The exposure process of the photometric sensor 108 is realized by accumulating charges for a predetermined time with a so-called electronic shutter. The system control unit 201 controls the operation of the photometric sensor 108, accumulates electric charges for a predetermined time, and reads out an image signal (exposure control signal) from the photometric sensor 108. Further, the system control unit 201 also performs an exposure process (charge accumulation) on the AF sensor 105 and reads out an image signal (focus detection signal).

In S303, the system control unit 201 supplies the exposure control signal to the subject detection unit 204, and acquires the subject detection result from the subject detection unit 204. Further, the system control unit 201 determines the main subject based on the subject detection result. The main subject can be determined by any method, the system control unit 201 may automatically determine the subject, the detected area of the subject is shown to the user, and the subject in the area selected by the user is set as the main subject. You may decide. When the system control unit 201 automatically determines the main subject, it can be determined based on, for example, the position and size of the area of the subject. For example, the subject closest to the center of the image or the subject having the largest area can be determined as the main subject. It should be noted that the main subject can be determined by considering other conditions such as the reliability of subject detection and past detection results, or by considering a combination of a plurality of conditions. The details of the subject detection process will be described later with reference to FIG.

In S304, the system control unit 201 selects the focus detection area closest to the position of the main subject determined in S303 from the selectable focus detection areas. Then, among the focus detection signals read in S302, the focus state (defocus amount and direction) of the selected focus detection region is detected from the focus detection signal relating to the selected focus detection region.

When the subject is not detected in S303, the system control unit 201 obtains the focus state (defocus amount and direction) for all the selectable focus detection regions based on the focus detection signal. Then, the focus detection area where the subject exists at the closest distance is selected.

In S305, the system control unit 201 adjusts the focusing distance of the photographing lens 102 by controlling the position of the focus lens 113 based on the focal state of the focus detection region selected in S304.

In S306, the system control unit 201 determines the shooting conditions (aperture value (AV value), shutter speed (TV value), ISO sensitivity (ISO value)) using the exposure control signal read in S302. There is no particular limitation on the method of determining the shooting conditions, but here, the shooting conditions corresponding to the brightness (Bv value) obtained based on the exposure control signal are determined with reference to a pre-stored program diagram. And. The shooting conditions may be determined using the brightness of the subject area detected by the subject detection process.

In S307, the system control unit 201 detects the state of the switch SW2, advances the process to S308 if the switch SW2 is on, and returns the process to S301 if the switch SW2 is off.

In S308, the system control unit 201 executes a still image shooting process. The system control unit 201 moves the main mirror 103 and the sub mirror 104 to positions that do not intersect the light flux from the photographing lens 102, and drives the shutter 110 according to the shutter speed determined in S306. As a result, the image sensor 111 is exposed by the optical image formed by the photographing lens 102. The image sensor 111 generates an image signal in which the electric charge accumulated in each pixel during the exposure period is converted into a voltage. The system control unit 201 reads an image signal from the image sensor and generates image data by applying predetermined image processing such as A / D conversion, noise reduction, white balance adjustment, and color interpolation. The system control unit 201 saves the generated image data as an image data file in the image storage unit 202, or generates a display image signal based on the image data and displays it on the display unit 112. After that, the system control unit 201 returns the process to S301.

(Flow of subject detection processing)
Next, the details of the subject detection process of S303 of FIG. 3 will be described with reference to the flowchart shown in FIG.
In S401, the system control unit 201 determines whether or not the main subject has been determined, and if it is determined that the main subject has been determined, the process proceeds to S402, and if it is determined to be determined, the process proceeds to S403.

In S402, the system control unit 201 (selection means) selects the detection learning model 206 for subject detection before the main subject is determined from the learning models stored in the learning model storage unit 205, and performs subject detection processing. It is set in the subject detection unit 204 as a parameter. The detection learning model 206 for detecting a subject before determining the main subject is a learning model for detecting a subject corresponding to a wide amount of blur, and has a wide amount of blur from a small amount of blur to a large amount of blur for the subject to be detected. This is a learning model obtained by machine learning using images. By using the detection learning model 206, it is possible to realize subject detection capable of detecting the subject to be detected even if the amount of blur is large.

In S403, the system control unit 201 (selection means) selects the detection learning model 207 for subject detection after the main subject is determined from the learning models stored in the learning model storage unit 205, and performs subject detection processing. It is set in the subject detection unit 204 as a parameter. The detection learning model 207 for detecting a subject after determining the main subject is a learning model for detecting a subject having a small amount of blur, and is obtained by machine learning for the subject to be detected by emphasizing an image with a small amount of blur. This is the obtained learning model. By using the detection learning model 207, it is possible to realize subject detection capable of accurately detecting a subject to be detected with a small amount of blur.

Further, the system control unit 201 in S402 and S403 supplies the image data generated by performing A / D conversion, noise reduction processing, or the like on the exposure control signal read in S302 to the subject detection unit 204. Then, the system control unit 201 advances the process to S404.

In S404, the subject detection unit 204 applies the subject detection process to the image data based on the exposure control signal by using the detection learning model set in S402 or S403 from the system control unit 201. The subject detection unit 204 supplies information representing the detection result to the system control unit 201. The information representing the detection result may include whether or not the subject has been detected (number of detections) and information regarding the detected subject area (for example, position and size). The subject detection unit 204 uses CNN for subject detection. Details of CNN will be described later.

In S405, if one or more subjects are detected as a result of subject detection, the system control unit 201 determines the main subject from the detected subjects. The method of determining the main subject is as described above.

Here, after the main subject is determined, the subject is detected using the detection learning model 207 obtained by machine learning that emphasizes the image of the subject with a small amount of blur. However, if the main subject is not continuously detected for a predetermined period even after the main subject is determined, the detection learning model 206, which is a learning model for detecting the subject corresponding to a wide amount of blur, should be used. It may be configured as. This is because if the main subject is not continuously detected (lost) for a predetermined period of time, the amount of blurring of the main subject may be large.

Further, even after the determination of the main subject, the detection learning model 206 may be used for the period when it is determined that the main subject is not in focus. This is because there is a possibility that the amount of blurring of the main subject is large before focusing on the main subject. Alternatively, when the amount of blurring of the main subject is equal to or greater than the threshold value, the detection learning model 206 may be used. Alternatively, the subject may be detected by using the detection learning model 206 even before the switch SW1 is turned on.

When the detection learning model 206 is used, subject detection is performed with priority given to suppressing detection omission rather than erroneous detection. Therefore, when the detection learning model 206 is used for subject detection, the threshold value of the detection reliability required when determining the main subject is stricter in S405 than when the detection learning model 207 is used for subject detection. Can be set. That is, the certainty of subject detection (detection reliability) required for the region of the main subject is higher than that of the subject region detected using the detection learning model 207 for the subject region detected using the detection learning model 206. Can be set high. As a result, it is possible to prevent the falsely detected subject from being determined as the main subject and optimizing the focus adjustment and exposure control in an inappropriate region.

(Details of subject detection unit)
Next, the subject detection unit 204 will be described. In the present embodiment, the subject detection unit 204 is composed of a neocognitron which is a kind of CNN (convolutional neural network). The basic configuration of the subject detection unit 204 will be described with reference to FIGS. 5 and 6. FIG. 5 shows a basic configuration of a CNN that detects a subject from the two-dimensional image data input in FIG. In the processing flow, the left end is used as an input, and the processing proceeds to the right. The CNN has two layers called a feature detection layer (S layer) and a feature integration layer (C layer) as one set, which are hierarchically configured. The S layer corresponds to the convolution layer described in the prior art, and the C layer corresponds to the pooling layer or the subsampling layer.

The CNN first detects the next feature in the S layer based on the feature detected in the previous layer. Further, the feature detected in the S layer is integrated in the C layer, and the detection result in that layer is transmitted to the next layer.
The S layer is composed of feature detection cell surfaces, and different features are detected for each feature detection cell surface. Further, the C layer is composed of a feature-integrated cell surface, and the detection result on the feature detection cell surface of the previous layer is pooled or subsampled. In the following, when it is not necessary to distinguish between the feature-detecting cell surface and the feature-integrated cell surface, they are collectively referred to as a feature surface. In the present embodiment, the output layer (nth layer), which is the final layer, does not use the C layer but is composed of only the S layer.

The details of the feature detection process on the feature detection cell surface and the feature integration process on the feature integration cell surface will be described with reference to FIG. One feature detection cell surface is composed of a plurality of feature detection neurons, and each feature detection neuron is connected to the C layer of the previous hierarchy in a predetermined structure. Further, one feature-integrating cell surface is composed of a plurality of feature-integrating neurons, and each feature-integrating neuron is connected to the S layer of the same hierarchy in a predetermined structure.

と表記する。また、Ｌ階層目のＣ層のＭ番目の細胞面内において、位置(ξ, ζ)の特徴統合ニューロンの出力値を

と表記する。その時、それぞれのニューロンの結合係数を

とすると、各出力値は以下のように表すことができる。 The output value of the feature detection neuron at the position (ξ, ζ) in the M-th cell plane of the S-layer of the L-layer shown in FIG.

Notated as. In addition, the output value of the feature-integrated neuron at the position (ξ, ζ) in the Mth cell plane of the C layer of the L layer is calculated.

Notated as. At that time, the connection coefficient of each neuron

Then, each output value can be expressed as follows.

[数式２]

ここで、数式１におけるｆは活性化関数であり、例えばロジスティック関数や双曲正接関数などのシグモイド関数である。また、

は、Ｌ階層目のＳ層のＭ番目の細胞面における、位置(ξ, ζ)の特徴検出ニューロンの内部状態を表す。数式２は活性化関数を用いておらず、単純な線形和で表されている。 [Formula 1]

[Formula 2]

Here, f in Equation 1 is an activation function, for example, a sigmoid function such as a logistic function or a hyperbolic tangent function. again,

Represents the internal state of the feature detection neuron at the position (ξ, ζ) on the Mth cell surface of the S layer of the L layer. Equation 2 does not use the activation function and is expressed by a simple linear sum.

と出力値

とは等しい。また、数式１の

を特徴検出ニューロンの結合先出力値と呼び、数式２の

を特徴統合ニューロンの結合先出力値と呼ぶ。 When the activation function is not used as in Equation 2, the internal state of the neuron

And output value

Is equal to. Also, in formula 1,

Is called the connection destination output value of the feature detection neuron, and is described in Equation 2.

Is called the connection destination output value of the feature integrated neuron.

が大きい場合、入力画像の画素位置(ξ, ζ)に、Ｌ階層目のＳ層のＭ番目の細胞面が検出する特徴が存在する可能性が高いことを意味する。またｎは数式１において、Ｌ−１階層目のＣ層のｎ番目の細胞面を意味しており、統合先特徴番号と呼ぶ。基本的にＬ−１階層目のＣ層に存在する全ての細胞面について積和演算を行う。（ｕ, ｖ）は、結合係数の相対位置座標であり、検出する特徴のサイズに応じて有限の範囲（ｕ, ｖ）において積和演算を行う。このような有限な（ｕ, ｖ）の範囲を受容野と呼ぶ。また受容野の大きさを、以下では受容野サイズと呼び、結合している範囲の横画素数×縦画素数で表す。 Here, ξ, ζ, u, v, and n in Equation 1 and Equation 2 will be described. The position (ξ, ζ) corresponds to the position coordinates in the input image, for example, the output value.

When is large, it means that there is a high possibility that the pixel position (ξ, ζ) of the input image has a feature detected by the M-th cell surface of the S layer of the L layer. Further, n means the nth cell surface of the C layer of the L-1 layer in the mathematical formula 1, and is called an integration destination feature number. Basically, the product-sum calculation is performed on all the cell surfaces existing in the C layer of the L-1 layer. (U, v) is the relative position coordinates of the coupling coefficient, and the product-sum operation is performed in a finite range (u, v) according to the size of the feature to be detected. Such a finite (u, v) range is called a receptive field. The size of the receptive field is hereinafter referred to as the receptive field size, and is expressed by the number of horizontal pixels x the number of vertical pixels in the combined range.

は、入力画像

である。ちなみにニューロンや画素の分布は離散的であり、結合先特徴番号も離散的なので、ξ，ζ，ｕ，ｖ，ｎは離散的な値をとる。ここでは、ξ，ζは非負整数、ｎは自然数、ｕ，ｖは整数とし、何れも有限な範囲を有する。 Further, in the formula 1, L = 1, that is, in the S layer of the first layer, the formula 1 is used.

Is the input image

Is. By the way, since the distribution of neurons and pixels is discrete and the connection destination feature numbers are also discrete, ξ, ζ, u, v, and n take discrete values. Here, ξ and ζ are non-negative integers, n is a natural number, and u and v are integers, all of which have a finite range.

は、所定の特徴を検出するための結合係数であり、結合係数を適切な値に調整することによって、所定の特徴を検出可能になる。この結合係数の調整が学習であり、ＣＮＮの構築においては、さまざまなテストパターンを用いて、

が適切な出力値になるように、結合係数を繰り返し徐々に修正していくことで結合係数を調整する。 In formula 1

Is a coupling coefficient for detecting a predetermined feature, and by adjusting the coupling coefficient to an appropriate value, the predetermined feature can be detected. This adjustment of the coupling coefficient is learning, and in the construction of CNN, various test patterns are used.

Adjust the coupling coefficient by repeating and gradually modifying the coupling coefficient so that is an appropriate output value.

は、２次元のガウシアン関数を用いており、以下の数式３のように表すことができる。
[数式３]

ここでも、（ｕ，ｖ）は有限の範囲を有し、特徴検出ニューロンの場合と同様、範囲を受容野、範囲の大きさを受容野サイズと呼ぶ。ここではＬ階層目のＳ層のＭ番目の特徴のサイズに従って、受容野サイズの値を適宜設定することができる。数式３中のσは特徴サイズ因子であり、受容野サイズに応じて適宜定めることができる定数であってよい。例えば、受容野の一番外側の値がほぼ０とみなせるような値になるように特徴サイズ因子σを設定することができる。このように、本実施形態の被写体検出部２０４は、上述した演算を各階層で行い、最終階層（ｎ階層目）のＳ層において被写体検出を行うＣＮＮによって構成される。 Next, in formula 2

Uses a two-dimensional Gaussian function and can be expressed as in the following mathematical formula 3.
[Formula 3]

Here, too, (u, v) has a finite range, and the range is called the receptive field and the size of the range is called the receptive field size, as in the case of the feature detection neuron. Here, the value of the receptive field size can be appropriately set according to the size of the Mth feature of the S layer of the L layer. Σ in Equation 3 is a feature size factor and may be a constant that can be appropriately determined according to the receptive field size. For example, the feature size factor σ can be set so that the outermost value of the receptive field is a value that can be regarded as almost 0. As described above, the subject detection unit 204 of the present embodiment is configured by the CNN that performs the above-mentioned calculation in each layer and detects the subject in the S layer of the final layer (nth layer).

の具体的な調整（学習）方法について説明する。被写体検出部２０４に直接的に学習を行わせても良いし、被写体検出部２０４と等価なＣＮＮの機能を有するクラウドや外部のエッジコンピュータ上において学習を行わせるようにしてもよい。クラウドやエッジコンピュータ上で学習を行った場合には、ＤＳＬＲ１００が生成された学習モデルを無線通信、有線通信、あるいは、着脱可能な記憶メディアを介して取得し、学習モデル記憶部２０５に記憶させるようにすればよい。学習は、ＣＮＮに特定の入力画像（テストパターン）を与えて得られるニューロンの出力値と、教師信号（そのニューロンが出力すべき出力値）との関係に基づいて、結合係数

を修正することである。本実施形態の学習では、最終階層（ｎ階層目）の特徴検出層Ｓについては最小二乗法を用いて結合係数を修正する。また、他の階層（１〜ｎ−１階層目）の特徴検出層Ｓについては、誤差逆伝搬法を用いて結合係数を修正する。最小二乗法や誤差逆伝搬法を用いた結合係数の修正手法は例えば非特許文献１に記載されるような公知技術を用いることができるため、詳細についての説明は省略する。 (Learning method for subject detection)
Coupling coefficient

The specific adjustment (learning) method of is explained. The subject detection unit 204 may be made to perform learning directly, or may be made to perform learning on a cloud or an external edge computer having a CNN function equivalent to that of the subject detection unit 204. When learning is performed on the cloud or an edge computer, the learning model generated by the DSLR 100 is acquired via wireless communication, wired communication, or a detachable storage medium, and stored in the learning model storage unit 205. It should be. Learning is based on the relationship between the output value of a neuron obtained by giving a specific input image (test pattern) to the CNN and the teacher signal (the output value that the neuron should output), and the coupling coefficient.

Is to fix. In the learning of the present embodiment, the coupling coefficient is corrected by using the least squares method for the feature detection layer S of the final layer (nth layer). Further, for the feature detection layer S of the other layers (1st to n-1th layers), the coupling coefficient is corrected by using the error back propagation method. As a method for correcting the coupling coefficient using the least squares method or the error back propagation method, for example, a known technique as described in Non-Patent Document 1 can be used, and therefore detailed description thereof will be omitted.

A large number of patterns to be detected and patterns that should not be detected are prepared as test patterns for learning. Each test pattern has image data and a corresponding teacher signal. The image data corresponding to the pattern to be detected is a teacher signal such that the output of the neuron corresponding to the region where the pattern to be detected exists is 1 on the feature detection cell surface of the final layer. On the other hand, for the image data corresponding to the pattern that should not be detected, a teacher signal is given so that the output of the neuron corresponding to the region where the pattern that should not be detected exists is -1.

In this embodiment, paying attention to the fact that the desired subject detection characteristics differ depending on the time when the subject detection is performed, the parameters used for the subject detection (learning model generated by machine learning) are set according to the time when the subject detection is performed. It was configured to switch. Therefore, a more appropriate subject detection result can be obtained as compared with the configuration using one learning model. Specifically, for the subject to be detected (for example, the face of a person), the subject before the main subject is determined by machine learning using a test pattern using image data of various amounts of blur (from a small amount of blur to a large amount of blur). A detection learning model 206 for detection is prepared. Further, for the subject to be detected, the detection learning model 207 for detecting the subject after the main subject is determined is prepared by machine learning using a test pattern that focuses on a small amount of image data.

Then, by using the detection learning model 206 before the determination of the main subject and the detection learning model 207 after the determination, the subject detection suitable before the determination of the main subject and the subject detection suitable after the determination of the main subject can be performed. realizable. Therefore, among the detected subjects, it is possible to appropriately execute the process using the area of a specific subject.

<< Second Embodiment >>
Next, a second embodiment of the present invention will be described. FIG. 7 is a block diagram showing a functional configuration example of the DSLR 100 according to the present embodiment, and the same reference numbers as those in FIG. 2 are added to the same configuration as that of the first embodiment. However, the photometric sensor 108 is different from the first embodiment in that when the image signal is generated, a gain is applied to the image signal so that the brightness value of the image signal is included in a predetermined range. The magnitude of the gain applied by the photometric sensor 108 to the image signal is notified from the photometric sensor 108 to the system control unit 201. Further, the learning model storage unit 205 is different from the first embodiment in the type of the detection learning model that is stored.

The position / attitude change acquisition unit 701 has a position / attitude sensor such as a gyro, an acceleration sensor, and an electronic compass, and outputs a signal indicating the position / attitude change of the DSLR 100. The system control unit 201 stores the signal output by the position / attitude change acquisition unit 701 in the RAM 2012.

The motion vector detection unit 702 detects the motion vector using the image data for two frames. The detected motion vector information is output to the system control unit 201. The system control unit 201 stores the motion vector information in the RAM 2012. The details of the vector detection process will be described later. Since the other parts are the same as those in the first embodiment, the description thereof will be omitted.

を始点、それらに対応するフレーム_t上の点

を終点とする、ｉ個のベクトルｖ_iを以下のように算出する。

ただし、ベクトルの第一成分を画像の横方向、第二成分を画像の縦方向とする。 (Details of motion vector calculation process)
The details of the motion vector calculation process in the motion vector detection unit 702 will be described. FIG. 8 is a diagram schematically showing images for two frames supplied to the motion vector detection unit 702. Here, the images for two frames are images taken from the same viewpoint at different timings, and a plurality of positions in the _{frame t-1 where the current frame is the frame t} and the previous frame is the frame _t-1. Is matched with the frame _t. And the point on _{frame t-1}

Starting point, the corresponding points on the _{frame t}

The i vectors v _i having the end point are calculated as follows.

However, the first component of the vector is the horizontal direction of the image, and the second component is the vertical direction of the image.

(Dictionary switching in subject detection)
When trying to detect the subject area in the current frame based on the motion vector between frames, the larger the motion vector, the higher the probability of false detection. This is because a subject having a large motion vector has a large amount of blurring of the image, and as in the case of a large amount of blurring, the variation of the image similar to the subject increases and the characteristics of the subject are lost.

The subject detection unit 204 applies the subject detection process using the processing parameters based on the learning model generated in advance through machine learning to the image. Specifically, the subject detection unit 204 includes a detection learning model 703 for a small main subject motion vector and a detection learning model 704 for a large main subject motion vector among the learning models stored in the learning model storage unit 205. refer. The detection learning model 703 for small main subject motion vector is generated by machine learning that emphasizes a subject image with a small amount of subject blur. Further, the detection learning model 704 for using the main subject motion vector is generated by machine learning that emphasizes a subject image having a large amount of subject blur.

Then, the system control unit 201 determines which of the detection learning model 703 for the main subject motion vector small and the detection learning model 704 for the main subject motion vector large is used in the subject detection unit 204 according to the magnitude of the motion vector of the main subject. Switch whether to use in. By switching the learning model used for the subject detection process according to the magnitude of the motion vector of the main subject, the system control unit 201 switches the characteristics of the subject detection process in the main subject detection unit 204. More specifically, the system control unit 201 switches to use the detection learning model 703 if it is determined that the motion vector of the main subject is large, and to use the detection learning model 704 if it is determined that the motion vector of the main subject is large. .. The threshold value for determining the magnitude of the motion vector can be experimentally determined in advance, for example.

(Flow of subject detection processing)
Next, the details of the subject detection process performed by the DSLR 100 of the present embodiment in the photographing operation will be described with reference to the flowchart shown in FIG. Since the photographing operation of the DSLR 100 of the present embodiment is the same as that of the first embodiment, the subject detection process described below is performed in S303 of FIG. Here, the main subject has already been determined, and the motion vector detection unit 702 uses, for example, a frame image of a moving image shot in the shooting standby state, and a motion vector starting from a point in the main subject area in the previous frame. It is assumed that one or more of are detected.

In S901, the system control unit 201 calculates the magnitude of the motion vector of the main subject with reference to the RAM 2012. Then, the system control unit 201 compares the calculated size with a predetermined threshold value, and determines that the motion vector is large if it is larger than the threshold value. The system control unit 201 proceeds to S903 if it is determined that the motion vector of the main subject is large, and proceeds to S902 if it is not determined.

In S902, the system control unit 201 selects a detection learning model 703 for a small motion vector from a plurality of learning models stored in the learning model storage unit 205, and sets it in the subject detection unit 204 as a parameter for subject detection processing. do. By using the detection learning model 703, it is possible to realize subject detection capable of accurately detecting a subject to be detected with a small subject blur.

In S903, the system control unit 201 selects a detection learning model 704 for a large motion vector from a plurality of learning models stored in the learning model storage unit 205, and sets it in the subject detection unit 204 as a parameter for subject detection processing. .. By using the detection learning model 704, it is possible to realize subject detection capable of detecting the subject to be detected even if the subject blur is large.

Further, the system control unit 201 in S902 and S903 supplies the image data generated by performing A / D conversion, noise reduction processing, or the like on the exposure control signal read in S302 to the subject detection unit 204. Then, the system control unit 201 advances the process to S904.

In S904, the subject detection unit 204 applies the subject detection process to the image data based on the exposure control signal by using the detection learning model set in S902 or S903 from the system control unit 201. The subject detection unit 204 supplies information representing the detection result to the system control unit 201. The information representing the detection result and the specific processing of the subject detection using the CNN may be the same as those in the first embodiment.

In S905, if one or more subjects are detected as a result of subject detection, the system control unit 201 determines the main subject from the detected subjects. The method for determining the main subject may be the same as in the first embodiment.

In S906, the motion vector detection unit 702 calculates the motion vector of the main subject determined in S905 by the above method.

In this embodiment, paying attention to the fact that the desired subject detection characteristics differ depending on the magnitude of the motion vector of the main subject, the parameter used for subject detection (learning model generated by machine learning) is set to the magnitude of the motion vector of the main subject. It was configured to switch according to the situation. Therefore, a more appropriate subject detection result can be obtained as compared with the configuration using one learning model. Specifically, for a subject to be detected (for example, the face of a person), a detection learning model 703 for a small motion vector is prepared by machine learning using a test pattern that focuses on image data with a small amount of subject blur. Further, for the subject to be detected, a detection learning model 704 for a large motion vector is prepared by machine learning using a test pattern that focuses on image data with a large amount of subject blur.

Then, the detection learning model is switched so that the subject detection process is executed using the detection learning model 704 when it is determined that the motion vector of the main subject is large and the detection learning model 703 when it is not determined to be large. As a result, it is possible to realize subject detection suitable for the magnitude of the motion vector of the main subject (that is, the magnitude of the amount of blurring of the subject image). Therefore, among the detected subjects, it is possible to appropriately execute the process using the area of a specific subject.

<< Third Embodiment >>
Next, a third embodiment of the present invention will be described. This embodiment can be implemented in the same manner as the second embodiment except for the method of switching the learning model used for subject detection. Therefore, the method of switching the learning model will be mainly described below.

(Dictionary switching in subject detection)
In the present embodiment, the learning model used for the subject detection process is switched according to the magnitude of the position / orientation change of the image pickup apparatus (DSLR100). When the position / orientation change of the image pickup apparatus is large, the camera shake becomes large, so that erroneous detection of the subject area is likely to occur. This is because when the position / orientation change of the image pickup apparatus is large, the variation of the image similar to the subject increases and the characteristics of the subject are lost, as in the case where the amount of blurring is large.

The subject detection unit 204 of the present embodiment applies subject detection using processing parameters based on a learning model generated in advance through machine learning to an image. In the present embodiment, the learning model storage unit 205 refers to the detection learning model 705 for small position / orientation change and the detection learning model 706 for large position / attitude change. The detection learning model 705 for small position / orientation change is generated by machine learning that emphasizes the subject image when the amount of camera shake is small. Further, the detection learning model 706 for large use of position / orientation change is generated by machine learning that emphasizes the subject image when the amount of camera shake is large.

Then, the system control unit 201 switches the subject detection characteristics of the subject detection unit 204 by switching the learning model used by the subject detection unit 204 according to the magnitude of the position / orientation change. More specifically, if it is not determined that the position / attitude change is large, the detection learning model 705 for small position / attitude change is detected, and if it is determined that the position / attitude change is large, the detection for large position / attitude change is detected. Switch to use the learning model 706. The threshold value for determining the magnitude of the position-posture change can be experimentally determined in advance, for example.

(Flow of subject detection processing)
Next, the details of the subject detection process performed by the DSLR 100 of the present embodiment in the photographing operation will be described with reference to the flowchart shown in FIG. Since the photographing operation of the DSLR 100 of the present embodiment is the same as that of the first embodiment, the subject detection process described below is performed in S303 of FIG.

In S1001, the system control unit 201 refers to the RAM 2012, compares the position / attitude change obtained from the position / attitude change acquisition unit 701 with a predetermined threshold value, and determines that the position / attitude change is large if it is larger than the threshold value. The system control unit 201 proceeds to S1003 if it is determined that the position / orientation change of the DSLR100 is large, and to S1002 if it is not determined.

In S1002, the system control unit 201 selects the detection learning model 705 for small position / orientation change from the learning models stored in the learning model storage unit 205, and causes the subject detection unit 204 as a parameter for the subject detection process. Set. By using the detection learning model 705, it is possible to realize subject detection capable of accurately detecting a subject to be detected with a small camera shake.
In S1003, the system control unit 201 selects the detection learning model 706 for large position / orientation change from the learning models stored in the learning model storage unit 205, and sets it in the subject detection unit 204 as a parameter for subject detection processing. do. By using the detection learning model 706, it is possible to realize subject detection capable of detecting the subject to be detected even if the camera shake is large.

Further, the system control unit 201 in S1002 and S1003 supplies the image data generated by performing A / D conversion, noise reduction processing, or the like on the exposure control signal read in S302 to the subject detection unit 204. Then, the system control unit 201 advances the process to S1004.

In S1004, the subject detection unit 204 applies the subject detection process to the image data based on the exposure control signal by using the detection learning model set in S1002 or S1003 from the system control unit 201. The subject detection unit 204 supplies information representing the detection result to the system control unit 201. The information representing the detection result and the specific processing of the subject detection using the CNN may be the same as those in the first embodiment.

In this embodiment, paying attention to the fact that the desired subject detection characteristics differ depending on the magnitude of the position / orientation change of the imaging device, the parameter (learning model generated by machine learning) used for subject detection is set to the magnitude of the position / orientation change. It was configured to switch according to the situation. Therefore, a more appropriate subject detection result can be obtained as compared with the configuration using one learning model. Specifically, for the subject to be detected (for example, the face of a person), the detection learning model 705 for small position / orientation change is performed by machine learning using a test pattern that focuses on image data when the amount of camera shake is small. prepare. Further, for the subject to be detected, a detection learning model 706 for a large change in position and orientation is prepared by machine learning based on a test pattern that focuses on image data when the amount of camera shake is large.

Then, the detection learning model is switched so that the subject detection process is executed using the detection learning model 706 when it is determined that the position / orientation change is large and the detection learning model 705 when it is not determined to be large. As a result, it is possible to realize subject detection suitable for the magnitude of the position / orientation change (that is, the magnitude of the amount of blurring of the subject image). Therefore, among the detected subjects, it is possible to appropriately execute the process using the area of a specific subject.

<< Fourth Embodiment >>
Next, a fourth embodiment of the present invention will be described. This embodiment can be implemented in the same manner as the second embodiment except for the method of switching the learning model used for subject detection. Therefore, the method of switching the learning model will be mainly described below.

(Dictionary switching in subject detection)
In the present embodiment, the learning model used for the subject detection process is switched according to the magnitude of the noise amount of the image data used for the subject detection. More specifically, the learning model used for the subject detection process depends on the magnitude of the gain (signal amplification factor) applied to the image data (image signal generated by the photometric sensor 108) supplied to the subject detection unit 204. To switch.

When the subject detection process is applied to the image data to which a large gain is applied, erroneous detection of the subject area is likely to occur. This is because by applying the gain to the image data, the noise is also amplified, the S / N ratio of the image is lowered, the variation of the image similar to the subject is increased, and the characteristics of the subject are lost.

The subject detection unit 204 of the present embodiment applies subject detection using processing parameters based on a learning model generated in advance through machine learning to an image. In the present embodiment, the learning model storage unit 205 refers to the detection learning model 707 for small gain and the detection learning model 708 for large gain. The detection learning model 707 for small gain is generated by machine learning that emphasizes the subject image when the noise is small. Further, the detection learning model 708 for large gain is generated by machine learning that emphasizes the subject image when the noise is large.

Then, the system control unit 201 switches the learning model used by the subject detection unit 204 according to the magnitude of the gain applied to the image data supplied to the subject detection unit 204, so that the subject in the main subject detection unit 204 Switch the detection characteristics. More specifically, if it is determined that the gain applied to the image data is large, the detection learning model 708 for large gain is used, and if it is not determined that the gain applied to the image is large, the gain is small. To switch to using the detection learning model 707 for. The threshold value for determining the magnitude of the gain can be experimentally determined in advance, for example. When subject detection is performed on the image data obtained by the image sensor 111, the system control unit 201 can determine, for example, the magnitude of the gain corresponding to the shooting sensitivity.

(Flow of subject detection processing)
Next, the details of the subject detection process performed by the DSLR 100 of the present embodiment in the photographing operation will be described with reference to the flowchart shown in FIG. Since the photographing operation of the DSLR 100 of the present embodiment is the same as that of the first embodiment, the subject detection process described below is performed in S303 of FIG.

In S1101, the system control unit 201 compares the magnitude of the gain applied to the image signal by the photometric sensor 108 with a predetermined threshold value, and determines that the gain is large if it is larger than the threshold value. The system control unit 201 proceeds to S1103 if it is determined that the gain is large, and proceeds to S1102 if it is not determined.

In S1102, the system control unit 201 selects the detection learning model 707 for small gain from the learning models stored in the learning model storage unit 205, and sets the detection learning model 707 in the subject detection unit 204 as a parameter for the subject detection process. .. By using the detection learning model 707, it is possible to realize subject detection capable of accurately detecting the subject to be detected when the noise is small.

In S1103, the system control unit 201 selects a detection learning model 708 for large gain from the learning models stored in the learning model storage unit 205, and sets the detection learning model 708 as a parameter for subject detection processing in the subject detection unit 204. By using the detection learning model 708, it is possible to realize subject detection capable of detecting the subject to be detected even when the noise is large.

Further, the system control unit 201 in S1102 and S1103 supplies the image data generated by performing A / D conversion, noise reduction processing, or the like on the exposure control signal read in S302 to the subject detection unit 204. Then, the system control unit 201 advances the process to S1104.

In S1104, the subject detection unit 204 applies the subject detection process to the image data based on the exposure control signal by using the detection learning model set in S1102 or S1103 from the system control unit 201. The subject detection unit 204 supplies information representing the detection result to the system control unit 201. The information representing the detection result and the specific processing of the subject detection using the CNN may be the same as those in the first embodiment.

In this embodiment, focusing on the fact that the desired subject detection characteristics differ depending on the amount of noise in the image data, the parameters used for subject detection (learning model generated by machine learning) are set to the gain applied to the image data. The configuration is such that it can be switched according to the size. Therefore, a more appropriate subject detection result can be obtained as compared with the configuration using one learning model. Specifically, for the subject to be detected (for example, the face of a person), a detection learning model 707 with a small gain is prepared by machine learning using a test pattern that focuses on image data with less noise. Further, for the subject to be detected, a detection learning model 708 with a large gain is prepared by machine learning using a test pattern that focuses on noisy image data.

Then, when it is determined that the gain applied to the image data for subject detection is large, the detection learning model 708 is used, and when it is not determined that the gain is large, the detection learning model 707 is used to execute the subject detection process. Switch the detection learning model. As a result, it is possible to realize subject detection suitable for when there is little noise and subject detection suitable for when there is a lot of noise. Therefore, among the detected subjects, it is possible to appropriately execute the process using the area of a specific subject.

(Other embodiments)
In the above-described embodiment, an example in which the present invention is applied to an imaging device that optimizes exposure and focus adjustment in a region of a main subject has been described. However, the purpose of using the main subject area is not limited to shooting. For example, it can be used for any other purpose such as applying image processing to a main subject area (or an area other than the main subject area). Therefore, in the present invention, the configuration related to photography is not essential. Further, in the above-described embodiment, the configuration in which two types of learning models are switched and used has been described, but three or more types of learning models may be switched and used.

As described above, examples of situations in which the amount of blur and sharpness of the subject to be subject to subject detection processing may differ include the degree of focusing, the magnitude of movement of the subject, the magnitude of change in the position and orientation of the image pickup device, and the noise of image data. An embodiment focusing on a large number has been described. However, it may be configured to switch the detection learning model according to other conditions.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or a computer-readable storage medium, and one or more processors of the computer of the system or device program the program. It can also be realized by executing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The above-described embodiment is merely a specific example for the purpose of assisting the understanding of the present invention, and there is no intention of limiting the present invention to the above-mentioned embodiment in any sense. All embodiments within the scope of the claims are included in the present invention.

100 ... Digital single-lens reflex camera, 101 ... Main body, 102 ... Lens, 108 ... Photometric sensor, 111 ... Image sensor, 204 ... Subject detection unit, 206 ... Detection learning model for subject detection processing before main subject determination, 207 ... Main Detection learning model for subject detection processing after subject determination

Claims (15)

A subject detection means that applies subject detection processing to image data using parameters generated based on machine learning, and
A determination means for determining the main subject from the subjects detected by the subject detection means, and
A storage means for storing a plurality of parameters used for the subject detection process, and
A selection means for selecting a parameter to be used in the subject detection means from the parameters stored in the storage means, and a selection means.
Have,
Said selection means, the main object is determined, and, in the case where the area of the main subject is being performed focusing to focus, in the case where the main subject is not determined, in each case An image processing device characterized by selecting different parameters suitable for. The selection means selects the first parameter when the main subject is determined and the focus is adjusted so that the area of the main subject is in focus, and the main subject is determined. The second parameter is selected if not present, and the second parameter is a parameter suitable for detecting a subject having a larger amount of blur than the first parameter, according to claim 1. Image processing equipment. The selection means selects the first parameter when the main subject is determined and the focus is adjusted so that the area of the main subject is in focus, and the main subject is determined. If not, select the second parameter and
The first parameter is a parameter for performing subject detection processing that prioritizes suppression of false detection of the subject rather than suppression of omission of detection of the subject.
The image processing apparatus according to claim 1, wherein the second parameter is a parameter for performing subject detection processing in which suppression of subject detection omission is prioritized over suppression of false detection of the subject. Said selection means, even after the main subject is determined, according to claim 2 or 3 wherein the main subject in the case where not detected continuously for a predetermined period and selects said second parameter The image processing apparatus according to. Claims 2 to 4 are characterized in that the selection means selects the second parameter even after the main subject has been determined, when the amount of blurring of the main subject is equal to or greater than a threshold value. The image processing apparatus according to any one of the above. The certainty of subject detection required for the area of the main subject should be set higher for the subject area detected using the second parameter than for the subject area detected using the first parameter. The image processing apparatus according to any one of claims 2 to 5, which is characterized. The image processing apparatus according to any one of claims 1 to 6, wherein the parameter is a coupling coefficient used in a convolutional neural network (CNN). The image processing apparatus according to any one of claims 1 to 7, further comprising an acquisition means for acquiring the parameter from the outside. The claim is characterized in that the acquisition means acquires the parameters from the outside via wireless communication, wired communication, or a detachable storage medium, and the storage means stores the parameters acquired by the acquisition means. Item 8. The image processing apparatus according to item 8. Image sensor and
An image pickup apparatus comprising the image processing apparatus according to any one of claims 1 to 9.
The subject detection means of the image processing device applies the subject detection process to the image obtained by the image pickup device.
An imaging device characterized by this. The focus adjusting means for performing the focus adjustment and
Determining means for determining the exposure conditions so that the area of the main subject has an appropriate exposure,
The imaging apparatus according to claim 10, characterized in that it further comprises a. The image pickup device according to claim 10 or 11, wherein the image pickup device is an image pickup device for acquiring an image for exposure control. An image processing method executed by an image processing device.
A subject detection process that applies subject detection processing to an image using parameters generated based on machine learning,
A determination step of determining the main subject from the subjects detected in the subject detection step, and
A selection step of selecting a parameter to be used in the subject detection step from a storage means for storing a plurality of parameters used in the subject detection process, and a selection step of selecting the parameter to be used in the subject detection step.
Have,
Wherein the selection step, the main object is determined, and, in the case where the area of the main subject is being performed focusing to focus, in the case where the main subject is not determined, in each case An image processing method characterized by selecting different parameters suitable for. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 9. A program for causing a computer included in an image pickup apparatus to function as an image processing apparatus included in the image pickup apparatus according to any one of claims 10 to 12.

Images