JP7799754B2 - Image processing device and method, electronic device, program and storage medium - Google Patents

Description

The present invention relates to image processing devices and methods, electronic devices, programs, and storage media, and in particular to a method for determining photometric values used for exposure control.

When controlling exposure in digital cameras, etc., a known method involves dividing the image captured for photometry into grid-like blocks, obtaining the brightness value of each block, and then controlling exposure based on an evaluation value calculated from the average of the obtained brightness values. By calculating an exposure compensation value that converges this evaluation value to the appropriate brightness and feeding it back to exposure controls such as aperture, shutter, and ISO, it is possible to maintain appropriate brightness in the image of the scene being photographed. Another known method involves detecting the subject's face or head and using the resulting detection area to calculate an exposure compensation value that will give the face appropriate brightness. However, if the detected face area includes the background or hair, these may affect the brightness and cause it to deviate from the appropriate level.

Patent document 1 proposes a mechanism that uses skin color detection to detect pixels that indicate skin color within the subject detection area, and calculates exposure compensation parameters based on the luminance values of the detected pixels. This makes it possible to calculate an exposure compensation value that provides appropriate luminance for the face area.

Japanese Patent Application Laid-Open No. 2005-148915

However, even when using skin color detection as described in Patent Document 1, there is a possibility that exposure correction may not be performed in the intended area. For example, clothing or tree trunks that are close in color to skin are likely to be misidentified. Furthermore, areas where the underlying skin is easily visible, such as the back of the head, are likely to be detected as skin color, which may result in inappropriate exposure correction.

Furthermore, when attempting to calculate an appropriate exposure value for a subject using the area obtained through subject detection, variations in the position and size of the detection frame may result in the background or hair being included in the detection frame, as shown in Figure 12(a), for example. In particular, if the frame shifts in situations where there is a luminance discrepancy, such as a backlit scene where the face is dark and the background is bright, the luminance of the dark face that you want to extract and the luminance of the bright background will be mixed and averaged, resulting in a luminance value that is higher than expected. When a luminance value is calculated in this state where the frame is shifted, there will be a discrepancy from the target luminance, making it impossible to set an appropriate exposure value. Similar discrepancies will also occur if the frame overlaps with accessories such as masks or sunglasses.

In contrast, as shown in Figure 12(b), if exposure values are calculated using skin detection and targeting only the skin, it is possible to avoid brightness discrepancies caused by the background or clothing. However, as shown in Figure 12(c), areas where the underlying skin is easily visible, such as the back of the head, are also likely to be detected as skin, raising concerns that an appropriate exposure value may not be set.

The present invention was made in consideration of the above problems, and aims to control exposure so that the subject is appropriately bright regardless of the subject's condition.

In order to achieve the above-mentioned object, the image processing device of the present invention comprises an input means for inputting an image, a first detection means for detecting a first area indicating a subject included in the image, a second detection means for detecting a second area included in the image having a predetermined first characteristic, a judgment means for judging the state of the subject, a calculation means for calculating a photometric value using the luminance value of a first overlapping area where the first area and the second area overlap when the judgment means judges that the state of the subject satisfies a predetermined condition, and a determination means for determining an exposure value based on the photometric value, and when the judgment means judges that the state does not satisfy the condition, the determination means determines the exposure value using the photometric value calculated using the luminance value of the first area .

This invention allows exposure control so that the subject is appropriately illuminated regardless of the subject's condition.

1 is a block diagram showing the functional configuration of an imaging apparatus according to a first embodiment of the present invention. 10 is a flowchart of an imaging process according to the first embodiment. 5A and 5B are diagrams illustrating divided regions and brightness values according to the first embodiment. FIG. 10 is a block diagram showing the functional configuration of an imaging apparatus according to a second embodiment. 10 is a flowchart of an imaging process according to a second embodiment. 10A and 10B are diagrams showing distribution histograms of brightness of the skin region of the side of the face and the back of the head, and brightness of the hair region in the second embodiment. 5A and 5B are diagrams showing a method for detecting a face angle in the first and second embodiments. 5A and 5B are diagrams illustrating a method for detecting an attachment in the first and second embodiments. 10 is a flowchart of an imaging process according to the third embodiment. 10 is a flowchart illustrating a method using a plurality of thresholds for determining the size of the area of a skin region according to the third embodiment. 10A and 10B are diagrams illustrating a method using a plurality of thresholds for determining the size of the area of a skin region according to the third embodiment. FIG. 1 is a diagram for explaining a problem in the related art.

The following describes the embodiments in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined by the claims. While the embodiments describe multiple features, not all of these features are necessarily essential to the invention, and multiple features may be combined in any desired manner. Furthermore, in the attached drawings, the same reference numbers are used to designate identical or similar components, and redundant explanations will be omitted.

First Embodiment
A first embodiment of the present invention will now be described.
Configuration of the Imaging Apparatus FIG. 1 is a block diagram showing the functional configuration of an imaging apparatus 100 according to the first embodiment.
1, the configurations represented as blocks can be realized by integrated circuits (ICs) such as ASICs or FPGAs, by discrete circuits, or by a combination of a memory and a processor that executes a program stored in the memory. Furthermore, one block may be realized by multiple integrated circuit packages, or multiple blocks may be realized by a single integrated circuit package. Furthermore, the same block may be implemented in different configurations depending on the operating environment, required capabilities, etc.

In the following embodiments, the present invention will be described as being implemented in an imaging device such as a digital camera, but the present invention can also be implemented in any electronic device with an imaging function. Such electronic devices include imaging devices as well as computer equipment (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, drive recorders, etc. Note that these are merely examples, and the present invention can also be implemented in other electronic devices.

The operation unit 101 is composed of various operating members such as switches and buttons that the operator of the imaging device 100 operates to input various instructions, and includes a shutter switch and a touch sensor (which can be operated by touching the display device). The shutter switch includes, for example, SW1, which turns ON during operation (e.g., halfway pressing) and instructs preparation for shooting, and SW2, which turns ON when operation is completed (e.g., fully pressed) and instructs the start of a series of imaging processing operations.

The control unit 102 includes a CPU, a nonvolatile memory, and a RAM, and can realize various functions by the CPU executing programs stored in the nonvolatile memory. For example, the control unit 102 controls the operation of each unit shown in FIG. 1 in response to instructions from the operation unit 101.
The sensor unit 103 receives light incident thereon through an optical system including a mechanical mechanism 1091 such as a lens 1081 and an aperture, converts the received light into an electric charge according to the amount of light, and outputs an analog image signal. The sensor unit 103, the lens 1081, and the mechanical mechanism 1091 constitute an imaging unit.

The A/D converter 104 performs sampling, gain adjustment, A/D conversion, etc. on the analog image signal output from the sensor unit 103, and outputs the resulting digital image signal.
The image processing unit 105 performs various image processing on the digital image signal output from the A/D conversion unit 104 and outputs the processed digital image signal (image data). For example, the image processing unit 105 converts the digital image signal output from the A/D conversion unit 104 into a YUV image signal and outputs it.
The luminance calculation unit 106 calculates the luminance using the image data output from the image processing unit 105, and calculates the difference from the appropriate luminance.

The subject detection unit 107 detects subjects within the image using image data obtained from the image processing unit 105. The subject detected here is assumed to be a person, and the area of the subject's face and head within the image is obtained. Subject detection can be performed using known methods, such as extracting an area using pattern matching from the contours of the human body, detecting important organs with distinctive features such as the eyes, nose, and mouth and detecting the head area containing them, or using an algorithm that learns a person's facial area through machine learning. For example, in machine learning techniques, learning is performed by relating concepts of each granularity, from the overall image of the object to the fine details, in a hierarchical structure. When learning about people, images of people of various races, ages, genders, facial orientations, and hair types are used. Furthermore, the detection area can be divided into categories other than the face, such as the head, torso, limbs, upper body, lower body, and entire body.

The subject state determination unit 117 determines the facial state of the subject detected by the subject detection unit 107. The facial state determined here includes, for example, the orientation of the face (up, down, left, or right), the presence or absence of hair or beard, and the presence or absence of accessories such as a mask or sunglasses.

The skin detection unit 118 detects skin within an image and acquires its area. Skin detection can be performed using known methods, such as extracting a specified color gamut defined as skin color, or using an algorithm that learns skin areas through machine learning. For example, in machine learning techniques, concepts of each granularity, from the overall image of the object to the finer details, are related in a hierarchical structure for learning. When learning skin, using images of people of various races, ages, and genders makes it possible to detect skin even if there are differences in the shade of skin color between people.

The display unit 115 is configured with a liquid crystal display or the like, and displays an image based on the digital image signal output from the A/D conversion unit 104 .
The external connection unit 114 is a connection unit for connecting an external monitor, a personal computer, etc. For example, by connecting an external monitor via the external connection unit 114, it becomes possible to display an image displayed on the display unit 115 on the external monitor.

The AF processing unit 108 determines the focus state based on the image data output from the image processing unit 105 and controls the lens 1081 to adjust the focus.

The AE processing unit 109 controls the mechanical mechanism 1091 based on the difference in brightness between the appropriate brightness and the brightness calculated by the brightness calculation unit 106 using the image data output from the image processing unit 105. This controls, for example, the aperture value and the shutter speed. The AE processing unit 109 may also be configured to control the gain used in the gain adjustment performed by the A/D conversion unit 104. If the sensor unit 103 has an electronic shutter function, the shutter speed may be controlled by controlling the charge accumulation time in the sensor unit 103.
Furthermore, it may determine whether to emit light from the flash unit 111 depending on the difference in brightness, and if it is determined that light should be emitted, it may issue a light emission instruction to the EF processing unit 110. Note that the determination of light emission and the light emission instruction may be performed by the control unit 102 in cooperation with the AE processing unit 109.
If it is determined that light should be emitted, the EF processing unit 110 causes the flash unit 111 to emit light in an amount that will provide the appropriate brightness for the subject in response to the light emission instruction.

The encoder unit 112 converts the format of the image data output from the image processing unit 105 into a format such as JPEG, and outputs the converted data to the image recording unit 113 .
The image recording unit 113 performs a process of recording the format-converted image data output from the encoder unit 112 in a memory (not shown) within the imaging device 100 or an external memory (not shown) inserted into the imaging device 100.
The memory unit 116 temporarily stores image data being processed by the control unit 102, the image processing unit 105, the encoder unit 112, and the like.
The exposure compensation value calculation unit 119 calculates an exposure compensation value based on the judgment result of the subject state judgment unit 117, using the image signal of the area where the subject area (face area or head area) detected by the subject detection unit 107 and the skin area detected by the skin detection unit 118 overlap.

The image processing unit 105, brightness calculation unit 106, subject detection unit 107, subject state determination unit 117, skin detection unit 118, and exposure compensation value calculation unit 119 may each be realized by a processor executing software, or by dedicated hardware. Although the image processing unit 105, brightness calculation unit 106, subject detection unit 107, subject state determination unit 117, skin detection unit 118, and exposure compensation value calculation unit 119 are illustrated as being separate from the control unit 102, this is not limitative. At least some of the functions of the image processing unit 105, brightness calculation unit 106, subject detection unit 107, subject state determination unit 117, skin detection unit 118, and exposure compensation value calculation unit 119 may be included in the control unit 102. In this case, the functions included in the control unit 102 are realized, for example, by the CPU executing a program stored in a non-volatile memory .

Exposure Control Next, exposure control in the first embodiment using the imaging device 100 having the above configuration will be described along the flow of the photographing operation with reference to the flowchart in FIG.

First, in step S201, when the operator of the imaging device 100 turns on the power switch included in the operation unit 101, the control unit 102 detects this and begins supplying power to each unit that makes up the imaging device 100.

When power is supplied to each component of the imaging device 100, the shutter opens in step S202, and light enters the sensor unit 103 via the lens 1081 and mechanical mechanism 1091 located in front of the camera. The sensor unit 103 reads out the charge accumulated by photoelectric conversion according to the amount of incident light, and outputs it to the A/D conversion unit 104 as an analog image signal.

The A/D conversion unit 104 performs sampling, gain adjustment, A/D conversion, etc. on the analog image signal output from the sensor unit 103, and outputs it as a digital image signal. The image processing unit 105 then performs various image processing on the digital image signal output from the A/D conversion unit 104, and outputs the processed digital image signal (hereinafter referred to as the "live image").

Next, in step S203, the brightness calculation unit 106 calculates the brightness value (image brightness value) of the entire live image output from the image processing unit 105. In this embodiment, the entire image is first divided into grid-like blocks, and the brightness values of the pixels in each block are averaged to determine the brightness value of each block. The brightness value of each block thus determined is then multiplied by a predetermined weight for each block to determine the average brightness value, which is then used as the image brightness value.

The image luminance value calculated here will be described with reference to FIG.
Fig. 3(a) shows an example of a live image output from the image processing unit 105. Fig. 3(b) shows an example of dividing the live image shown in Fig. 3(a) into blocks, and Fig. 3(c) shows an example of the luminance value of each block shown in Fig. 3(b). The image luminance value is obtained by weighting and adding the luminance of each block shown in Fig. 3(c).

Next, in step S204, the subject detection unit 107 detects a subject. Here, a person is detected in the live image, and the area of their face or head is detected as the subject.

Next, in step S205, the subject state determination unit 117 determines the state of the subject. The state of the subject determined here refers to the face area, whole body area, facial orientation, the presence and type of accessories such as a mask or sunglasses, the presence and color of hair, the presence or absence of a beard, etc. In this embodiment, the subject state determination unit 117 obtains information on important facial organs such as the eyes, nose, and mouth from the face area detected by the subject detection unit 107, and determines the orientation of the face based on the position of these organs in the face area.

Specifically, as shown in FIG. 7, the subject state determination unit 117 sets feature point coordinates for important organs within the face area.
For example, in the case of a face facing forward as shown in FIG. 7( a), as shown in FIG. 7( b), the eyes are positioned one by one relative to the center line of the detected face area, the nose is slightly below the center of the screen, and the mouth is located at the bottom of the screen relative to the center line. In this case, the subject state determination unit 117 determines that the face is facing forward (face angle is 0°). In FIG. 7( c), the positions of the eyes and nose are shifted slightly to the left overall compared to FIG. 7( b), so the subject state determination unit 117 determines the face angle to be 30°. In FIG. 7( d), the face angle is shifted further to the left, so it is determined to be 60°. In FIG. 7( e), the eye is united, and the nose and mouth are also shifted to the right edge of the frame, so it is determined to be 90°. Note that the angles shown in FIGS. 7( b) to 7( e) above are merely examples for indicating how far the face is facing sideways, and any indicator that allows the subject state determination unit 117 to grasp the magnitude relationship of the face orientation is not limited to these.

The subject state determination unit 117 also determines whether or not the subject is wearing an item such as a mask based on the presence or absence of organ information. As shown in Figure 8(a), if only the eyes are detected as important organs within the face detection frame and the nose and mouth are covered by something other than skin, it is determined that the subject is wearing a mask. As shown in Figure 8(b), if only the mouth and nose are detected as important organs within the face detection frame and the eyes are covered by something other than skin, it is determined that the subject is wearing sunglasses or glasses. With regard to hair, the subject state determination unit 117 determines whether there is an object with a color other than skin above the head region, and classifies the hair as black hair, chestnut hair, blonde hair, red hair, white hair, etc. depending on the color.

Next, in step S206, the skin detection unit 118 detects skin present in the live image. Here, a detectable skin area is defined as human skin, regardless of differences in skin color or brightness, and is capable of detecting not only facial skin areas but also skin areas on the hands and feet.

Next, in step S207, the subject state determination unit 117 determines whether the face is facing forward (frontal face) or side (profile) based on the state of the subject determined in step S205. If it is determined to be facing forward or side, the process proceeds to step S208; if it is determined not to be facing forward or side, the process proceeds to step S213.

In this embodiment, the profile condition is defined as a state in which the face is tilted from side to side within a range of approximately ±90° from the front position. This is assumed to be a level that ensures a sufficient area to obtain the brightness value of the skin area in the head region and also reduces exposure variations. If the angle exceeds 90°, it is determined that the area of the skin area is not sufficient to obtain a brightness value.

In step S208, the luminance calculation unit 106 calculates the average skin luminance SkinY for the overlapping area between the face area detected in step S204 and the skin area detected in step S206. Here, the method for calculating the average skin luminance SkinY will be described with reference to the example shown in Figure 3.

The brightness calculation unit 106 first determines in which of the divided blocks shown in FIG. 3(b) the face area detected by the subject detection unit 107 exists in the live image shown in FIG. 3(a). In this case, this corresponds to block area 301 in FIG. 3(d).

Next, the brightness calculation unit 106 determines in which of the divided blocks shown in Figure 3(b) the skin area detected by the skin detection unit 118 in the live image shown in Figure 3(a) is present. In this case, this corresponds to block area 302 in Figure 3(f).

Then, the average skin luminance SkinY is calculated by averaging the luminance values of the blocks commonly included in block area 301 and block area 302. In the example shown in Figure 3, the average value of 174, 168, 197, and 204, 185.75, is obtained as the average skin luminance SkinY.

Here, if the number of skin regions is small, there is a concern that the average skin luminance SkinY may vary. Variation in the average skin luminance SkinY will lead to variation in the photometric value calculated later in S210, which will lead to variation in the final exposure.
Therefore, the average skin luminance SkinY obtained may be a value averaged in the time direction. Specifically, the average skin luminance SkinY of the past N frames is used. In other words, if the average skin luminance of the current frame T is SkinY(T), the average skin luminance SkinY is calculated as follows: SkinY(T) + SkinY(T-1) + SkinY(T-2) + ... SkinY(T-N-1))/N.

Alternatively, the previously calculated value may be used if the difference between the previous calculated value and the current value does not exceed a predetermined value. In other words, SkinY(T) is used when SkinY(T) - SkinY(T-1) is equal to or greater than a predetermined threshold, and SkinY(T-1) is used when it is less than the threshold.
The process of suppressing the variation in the average skin luminance SkinY may be performed only when the area of the skin region is small. That is, when the area of the skin region is less than TH1, the average value of the average skin luminance SkinY for N frames is used, and when the area is TH1 or more, the value of the average skin luminance SkinY for the current frame calculated in S208 is used.
The method for suppressing the variation in the average skin luminance SkinY is not limited to the above method. By using these methods, the variation in the photometric value can be suppressed and the exposure can be stabilized.

Next, in step S209, the luminance calculation unit 106 calculates a difference ΔBvFace from a predetermined target luminance value of the face using the average skin luminance SkinY. The difference ΔBvFace can be calculated by the following equation (1), where ReferenceY is the target luminance value .
ΔBvFace=LOG ₂ (SkinY/ReferenceY)...(1)

Next, in step S210, the brightness calculation unit 106 calculates an appropriate photometric value using the difference ΔBvFace. If the photometric value calculated here is Bv, it is calculated using the following equation (2).
Bv=CtrlBv+ΔBvFace+BvCorr…(2)
Here, CtrlBv is the exposure value at which the image was acquired, and BvCorr is various correction values, including correction based on the degree of backlighting of the subject, correction based on the proportion of the sky that is a high-brightness area, night view correction, etc.

Then, in step S211, the photometric value obtained in step S210 is stored in case the face orientation is other than front or side.

The photometric value may be calculated by further using the image luminance value obtained in step S203.
When the image luminance value is further used, the photometric value Bv is calculated by the following equation (3) instead of equation (2).
Bv=(CtrlBv+ΔBvEa)×a+(CtrlBv+ΔBvFace)×b
+BvCorr…(3)
Here, ΔBvEa is the difference between the image luminance value calculated in step S203 and the target luminance value. Also, a and b are adjustment coefficients that determine the usage rate of the overall image luminance and the luminance of the skin region, and a+b=1.0.

By calculating the photometric value taking into account the image brightness value as described above, exposure can be controlled so that the brightness of the entire image, not just the face, is appropriate.

On the other hand, if step S207 determines that the face is facing away from the front or in profile, for example, as shown in Figure 12(c), and skin is detected due to the influence of the skin area underlying the hair, the photometric value will be calculated for the hair area, which will result in overexposure in the case of black hair. To avoid this, step S213 determines that the photometric value stored in the most recent step S211 will be used.

Then, in step S212, the exposure compensation value calculation unit 119 calculates an exposure value based on the photometric value obtained in step S210 or S213, and feeds this back to the AE processing unit 109, which then performs convergence control to the appropriate exposure.

In this way, the exposure compensation value calculation unit 119 calculates an exposure compensation value for the area where the detected subject area and skin detection area overlap when the detected subject state is a frontal or profile face and the skin area has been detected at a sufficient size. On the other hand, when the detected subject state is not a frontal or profile face and the skin area has not been detected at a sufficient size, the stored photometric value is used.

In step S214, the control unit 102 determines whether SW1 has been turned ON by operating the shutter switch included in the operation unit 101. If it has not been turned ON, the process returns to step S202, where a live image is acquired using the exposure value controlled in S212, and the above processing is repeated for the acquired live image.

When SW1 is turned ON in step S214, the final photometric value is acquired in step S215. At this time, the AF processing unit 108 also performs AF processing using the image information at this time point, and controls the lens 1081 to focus on the subject.

Then, in step S216, the control unit 102 determines whether SW2 has been turned ON by operating the shutter switch. If it has not been turned ON, the process returns to step S214. If it has been turned ON, the process proceeds to step S217, where the main exposure begins. The charge corresponding to the light incident on the sensor unit 103 via the lens and exposure mechanism is read out, and this is output to the A/D conversion unit 104 as an analog image signal.

The A/D conversion unit 104 performs sampling, gain adjustment, A/D conversion, etc. on the analog image signal output from the sensor unit 103, and outputs it as a digital image signal. The image processing unit 105 performs various image processing on the digital image signal, and outputs the processed digital image signal.

The digital signal output from the image processing unit 105 is converted into a format such as JPEG by the encoder unit 112 and output to the image recording unit 113. The image recording unit 113 performs processing to record the format-converted image data in a specified memory.

As described above, according to the first embodiment, the subject area, subject information, and skin area are acquired, and it is determined whether there is sufficient skin area. By controlling the exposure so that the brightness of the facial skin area is appropriate, it is possible to control the exposure without being affected by the back of the head, attached items, etc.

In the above example, the face direction was used to determine whether the subject's state allows for a sufficient skin area, but the present invention is not limited to face direction. Information on the face area, full body area, and the presence or absence of accessories such as a mask or sunglasses determined by the subject state determination unit 117 may be used comprehensively to make a determination based on, for example, the size of the face area obtained from the face area and full body area, or based on whether the subject is wearing a mask. It is also possible to make a determination using a combination of these determination methods.

Second Embodiment
Next, a second embodiment of the present invention will be described, which deals with exposure control in a case where the skin region cannot be detected sufficiently, such as in the case of a frontal face.

Configuration of the Imaging Apparatus Fig. 4 is a block diagram showing the functional configuration of the imaging apparatus 100 according to the second embodiment. Note that in the configuration shown in Fig. 4, the same components as those shown in Fig. 1 are given the same reference numerals, and descriptions thereof will be omitted.

The hair detection unit 401 detects hair and beard on the screen and acquires their area. Hair detection can be performed using known methods, such as extracting a predetermined color gamut defined as hair color, using an algorithm that learns hair areas through machine learning, or estimating the location of hair and beard from a detected face area and extracting a pattern matching area. For example, machine learning techniques involve learning by relating concepts of each granularity, from the overall image of the object to the fine details, in a hierarchical structure. When learning about hair, images of people of various races, ages, and genders are used.

Based on the determination result of the subject state determination unit 117, the exposure compensation calculation unit 402 calculates an exposure compensation value using image signals of the areas where the subject area (face area or head area) detected by the subject detection unit 107, the skin area detected by the skin detection unit 118, and the hair or beard area detected by the hair detection unit 401 overlap.

Exposure Operation Next, exposure control in the second embodiment using the imaging device 100 having the above configuration will be described along the flow of the shooting operation with reference to the flowchart in Fig. 5. However, the same steps as those in the first embodiment shown in Fig. 2 are given the same step numbers, and descriptions thereof will be omitted as appropriate.

After the calculation of the luminance value and image luminance value of each block (step S203), the detection of the subject (step S204), and the determination of the subject state (step S205) are completed based on the live image captured in step S202, the skin detection unit 118 detects the skin area present in the live image in the next step S501. Here, the detectable skin area refers to the skin of a person regardless of differences in skin color or luminance, and is capable of detecting not only the skin area of the face but also the skin areas of the hands and feet.
At the same time, the hair detection unit 401 detects hair regions present in the live image. Here, a detectable hair region is one that can detect a person's hair and beard regardless of race, age, or sex, and that can be detected without relying on color information such as white hair, black hair, or red hair.

Then, in step S502, the luminance calculation unit 106 calculates the average skin luminance SkinY for the overlapping areas of the face area detected in step S204 and the skin area detected in step S501, as shown in FIG. 3. Similarly, the luminance calculation unit 106 calculates the average hair luminance HairY for the overlapping areas of the face area detected in step S204 and the hair area detected in step S501.

Then, in step S503, the Bv value of skin, SkinBv, and the Bv value of hair, HairBv, are calculated and stored from the average skin luminance, SkinY, and the average hair luminance, HairY, calculated in step S502, and the target luminance value. When the exposure value at which the image was acquired is CtrlBv and the target luminance value is ReferenceY, the Bv value of skin, SkinBv, and the Bv value of hair, HairBv, are calculated by the following formula (4):
SkinBv=CtrlBv+LOG ₂ (SkinY/ReferenceY)
HairBv=CtrlBv+LOG ₂ (HairY/ReferenceY)
…(4)

Next, in step S504, the brightness difference ΔBvSH between the skin and hair is calculated from the thus obtained Bv value SkinBv of the skin and the Bv value HairBv of the hair, using the following equation (5).
ΔBvSH=|SkinBv−HairBv|…(5)

Next, in step S207, the subject state determination unit 117 determines whether the face is facing forward or sideways based on the state of the subject detected in step S205. If it is determined that the face is facing forward or sideways, the process proceeds to step S506; if it is determined that the face is not facing forward or sideways, the process proceeds to step S505.

In step S506, the luminance calculation unit 106 calculates the difference ΔBvFace from the target luminance value of the face using the average skin luminance SkinY calculated in step S502. ΔBvFace is calculated based on equation (1).

Then, in step S507, the brightness difference ΔBvSH between the skin and hair calculated in step S504 is stored as the previous value preΔBvSH for the next time the face is determined to be other than a frontal or profile face.

Then, in step S210, the brightness calculation unit 106 uses the difference ΔBvFace to calculate an appropriate photometric value. The photometric value Bv is calculated based on equation (2). Then, in step S211, the photometric value obtained in step S210 is stored in case a face angle other than frontal or profile is detected.

On the other hand, if step S207 determines that the face orientation is not forward or in profile, the process proceeds to step S505. For example, as shown in FIG. 12C, if the back of the head is determined and skin is detected due to the influence of the skin region underlying the hair, an inappropriate photometric value for the hair region may be calculated. In this case, the exposure for the hair region may become unstable. To avoid this, the process determines whether the difference (amount of change) between the brightness difference ΔBvSH between the skin and hair calculated in step S504 and the previous value PreΔBvSH is less than a predetermined threshold. If the difference is less than the predetermined threshold, the process proceeds to step S506, where the difference ΔBvFace is calculated using the average skin brightness SkinY calculated in step S502. If the difference is equal to or greater than the predetermined threshold, the process proceeds to step S213, where it is determined that the exposure value held in the most recent step S211 will be used. This reduces fluctuations in exposure.

Fig. 6(a) is a conceptual diagram showing an example of the subject region, skin region, and hair region detected when the face is turned sideways, and Fig. 6(b) is a conceptual diagram showing an example of the luminance distribution of the skin region and the luminance distribution of the hair region at this time. Fig. 6(c) is a conceptual diagram showing an example of the subject region, skin region 604 , and hair region 603 detected when the face is turned backward, and Fig. 6(d) is a conceptual diagram showing an example of the luminance distribution of the skin region and the luminance distribution of the hair region at this time. It can be seen that when the face is turned backward, the luminance distribution of the skin region 604 and the luminance distribution of the hair region 603 are closer than when the face is turned sideways. In other words, by comparing the previous luminance step ΔBvSH with the luminance step calculated this time, it is possible to determine whether or not the face orientation has changed.

Then, in step S212, the exposure compensation value calculation unit 119 calculates an exposure value based on the photometric value obtained in step S210 or S213, and feeds this back to the AE processing unit 109, which then performs convergence control to the appropriate exposure.

As described above, according to the second embodiment, the subject area, subject information, skin area, and hair area are acquired, and it is determined whether there is sufficient skin area and whether the external light environment is stable. By controlling the exposure to be appropriate for the face, it is possible to set appropriate exposure without being affected by the back of the head, attached items, etc.

In the first and second embodiments described above, the luminance values are calculated on a block-by-block basis, but the present invention is not limited to this, and the luminance values may also be calculated on a pixel-by-pixel basis.

Third Embodiment
Next, a third embodiment of the present invention will be described. In the third embodiment, processing will be described for a case where the subject is a front face or a profile face, but the skin area is small.
The configuration of the imaging device in the third embodiment is the same as that described in the first embodiment with reference to FIG. 1, and therefore a description thereof will be omitted.

Exposure Operation Next, exposure control in the third embodiment by the imaging device 100 will be described along the flow of the shooting operation with reference to the flowchart in Fig. 9. However, the same steps as those in the first embodiment shown in Fig. 2 are given the same step numbers, and descriptions thereof will be omitted as appropriate.

After the calculation of the brightness value and image brightness value of each block (step S203), the detection of the subject (step S204), the determination of the subject state (step S205), and the detection of the skin region (step S206) are completed based on the live image captured in step S202, in step S207, the subject state determination unit 117 determines whether the face is facing forward or sideways based on the state of the subject detected in step S205. If it is determined to be a frontal or sideways face, the process proceeds to step S901, and if it is determined not to be a frontal or sideways face, the process proceeds to step S213.

In step S901, it is determined whether the area of the skin region is large. In step S207, it is determined whether a sufficient skin area can be secured to obtain the brightness of the skin region based on the face orientation conditions. However, even if the face is a frontal or profile face, if the face is small relative to the image, or in scenes where detection using machine learning is difficult, the skin region may not be detected sufficiently. For this reason, in step S901, the area of the skin region is evaluated if the face is determined to be a frontal or profile face. The area evaluated here is specifically the number of blocks when the image is divided into blocks of a predetermined size, or the ratio of the number of skin region blocks to the face or head region of the subject detected in step S204. In step S901, it is determined whether the area is equal to or greater than a predetermined threshold value TH. If it is determined to be equal to or greater than the threshold value TH, proceed to step S208; if it is determined to be less than the threshold value TH, proceed to step S902.

The determination in step S901 may be made using multiple thresholds. Specifically, the determination is made using a flag Flg as shown in Figures 10 and 11. First, in step S1001, it is determined whether the area of the skin region is greater than or equal to a first threshold TH1. If it is less than the first threshold TH1, Flg is set to True in step S1002. Thereafter, in step S1001, if the area of the skin region is greater than or equal to the first threshold TH1, it is determined in step S1003 whether the area of the skin region is greater than or equal to a second threshold TH2. The second threshold TH2 is a predetermined threshold set to a value greater than the first threshold TH1. If the area of the skin region is less than the second threshold TH2, the previous determination result is retained; if it is greater than or equal to the second threshold TH, Flg is set to False in step S1004. In this way, a flag is generated that becomes True when the value falls below the first threshold TH1, and does not become False until the value thereafter reaches or exceeds the second threshold TH2, which is greater than the first threshold TH1.

Figure 11 shows a diagram of the area of the skin region and the flag Flg over time. The flag Flg rises when the area of the skin region falls below the first threshold TH1, and then falls when it reaches or exceeds the second threshold TH2.

When the flag Flg set in this manner is set, the judgment in step S901 is NO; when the flag Flg is not set, the judgment in step S901 is YES.

This reduces variation in the photometric value by preventing the luminance calculation method from being changed more frequently than when switching between calculating the photometric value using the luminance of the skin area or the luminance of the face area at a single threshold value in the determination in step S901 in FIG. 9 .

Furthermore, when determining the area of the skin region in step S901, there is a concern that the determination results may vary due to variations in the skin detection method using machine learning. As a result, whether the photometric value is calculated using the luminance of the skin region or the luminance of the face region may change frequently, leading to flickering of the photometric value.

Therefore, the area of the skin region can be calculated by averaging it over time. Specifically, the area of the skin region over N frames is used. In other words, if the area of the skin region in the current frame T is SkinNum(T), the area of the skin region is calculated as SkinNum = (SkinNum(T-1) + SkinNum(T-2) + ... SkinNum(T-1))/N.

Alternatively, the previously calculated value may be used if the difference between the previous calculated value and the current value does not exceed a predetermined value. In other words, SkinNum(T) is used when SkinNum(T) - SkinNum(T-1) is equal to or greater than a predetermined threshold, and SkinNum(T-1) is used when it is less than the threshold.
The method for suppressing the variation in SkinY is not limited to the above method.

This prevents frequent changes in whether the photometric value is calculated using the luminance of the skin region or the luminance of the face region in the determination in step S901 of Figure 9, thereby reducing variation in the photometric value.

If step S901 determines that the skin area is small, step S902 calculates the luminance FaceY of the face area. The face area is the face area detected in step S204. Next, step S903 determines whether photometry of the skin area of the subject to be detected has been performed in the past. Photometry of the skin area refers to the calculation of a photometric value using the average skin luminance SkinY calculated in step S208. In this embodiment, in step S208, photometric calculation using the luminance of the skin area is performed only for faces that are frontal or profile and for which the skin area has been detected to be equal to or greater than the threshold TH. If photometry of the skin area has not been performed in the past, there is no photometric value stored to be calculated in S210. In this case, proceed to step S904, and use the luminance FaceY of the face area calculated in S902 to calculate the difference ΔBvFace from a predetermined target luminance value for the face. The difference ΔBvFace is calculated based on equation (1) in which SkinY is replaced with FaceY.

Next, in step S210, the brightness calculation unit 106 uses the difference ΔBvFace to calculate an appropriate photometric value. The photometric value Bv is calculated based on equation (2).

Note that when calculating the photometric value, the luminance of the face area is used, so compared to the skin area , there is a possibility that accessories such as masks and sunglasses or hair areas may be included. For example, if the face area includes part of a white mask, the photometric value will be brighter than the luminance of the skin area, resulting in a final exposure that is darker than when the luminance of the skin area is used. On the other hand, if sunglasses are included, the photometric value will be darker than the luminance of the skin area, resulting in a final exposure that is brighter than when the luminance of the skin area is used. Similarly, when photometry is performed using a face frame for a profile shot, the inclusion of the hair area causes the calculated photometric value to be darker, resulting in a final exposure that is brighter than when the luminance of the skin area is used. In consideration of these factors, the contribution of the difference ΔBvFace to the photometric value calculation in step S210 is reduced.

Specifically, in calculating the photometric value, the following equation (6) is used instead of equation (2).
Bv=(CtrlBv+ΔBvFace)×k+BvCorr…(6)
Here, k is a coefficient ranging from 0 to 1.0. When the luminance of the face region is used, the contribution of the difference ΔBvFace to the photometric value Bv is reduced. On the other hand, when the luminance of the skin region SkinY is used, the contribution is not reduced. Therefore, k is set to k1 in step S209, and k is set to k2 in step S904. Here, k1 and k2 are predetermined values that satisfy k1 > k2, for example, k1 = 1.0 and k2 = 0.5. Through the above processing, the influence of the luminance of the face region is reduced when metering the photometry of the face region compared to when metering the photometry of the skin region, and therefore the influence of accessories such as masks and sunglasses and hair on the photometric value can be reduced.

Note that a method for suppressing the influence of the difference ΔBvFace may be applied to a value included in the exposure compensation value BvCorr. For example, the backlight compensation value AlphaCorr, which is one type of the exposure compensation value BvCorr, is calculated as follows:
AlphaCorr=(ΔBvFace−ΔBvEa)×α…(7)
The backlight correction value AlphaCorr is a correction amount that is applied to the photometric value by calculating the degree of backlighting of the subject from the difference ΔBvFace between the luminance of the face area and the target value and the difference ΔBvEa between the overall luminance and the target value. α is a coefficient that adjusts how much the degree of backlighting is reflected in the photometric value. Instead of k, which is applied directly to the difference ΔBvFace described above, the value of α may be changed depending on whether the photometric value is calculated using the luminance of the skin area or the luminance of the face area.

Then, in step S211, the photometric value obtained in step S210 is stored in case the face orientation is other than front or side, or the area of the skin region is small.

The photometric value Bv may be calculated by further using the image luminance value calculated in step S203. By calculating the photometric value taking the image luminance value into consideration, it is possible to control the exposure so that the brightness of the entire image, not just the face, is appropriate. In this case, the photometric value Bv is calculated based on the following equation (8), which is obtained by adding the coefficient k used in equation (6) to equation (3).
Bv=(CtrlBv+ΔBvEa)×a
+(CtrlBv+ΔBvFace)×b×k+BvCorr…(8)

In S207, if the face orientation is other than front or profile, or if in step S903 photometry of the skin area has been performed in the past, i.e., if a photometry value calculated in step S210 exists, the process proceeds to step S213, where it is determined that the photometry value stored in the most recent step S211 will be used.

Then, in step S212, the exposure compensation value calculation unit 119 calculates an exposure value based on the photometric value obtained in step S210 or S213, and feeds this back to the AE processing unit 109, which then performs convergence control to the appropriate exposure.

As described above, according to the third embodiment, in scenes where it is difficult to detect skin areas and there is a concern about variations in the photometric value using the luminance of the skin area, stable photometric calculations can be performed by using the luminance of the face area. Furthermore, once a sufficient number of skin areas are detected and the luminance of the skin area can be used, the photometric value of the most recent skin area can be used, making it possible to calculate appropriate exposure using the skin area.

<Other Embodiments>
The present invention may be applied to a system made up of a plurality of devices, or to an apparatus made up of a single device.

The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

<Summary>
The disclosure of this embodiment includes the following configuration.

(Item 1)
an input means for inputting an image;
a first detection means for detecting a first region representing a subject included in the image;
a second detection means for detecting a second region having a predetermined first characteristic included in the image;
a determination means for determining the state of the subject;
a calculation means for calculating a photometric value using a luminance value of a first overlapping area where the first area and the second area overlap when the determination means determines that the state of the subject satisfies a predetermined condition; and a determination means for determining an exposure value based on the photometric value.
1. An image processing device comprising:
(Item 2)
2. The image processing device according to item 1, wherein, when the determining means determines that the state does not satisfy the condition, the determining means determines the exposure value using the most recently determined photometric value.
(Item 3)
2. The image processing device according to item 1, characterized in that, when the determining means determines that the state does not satisfy the condition, the determining means determines the exposure value using a photometric value calculated using the luminance value of the first region.
(Item 4)
4. The image processing device according to item 3, wherein when calculating a photometric value using the luminance value of the first region, the calculation means reduces the contribution of the luminance value of the first region to the photometric value.
(Item 5)
Item c: The image processing device according to item 1, wherein the calculation means calculates the photometric value based on a luminance value obtained by averaging a predetermined number of luminance values obtained in the past.
(Item 6)
2. The image processing device according to item 1, wherein the calculation means calculates the photometric value based on the amount of change in the luminance value obtained most recently.
(Item 7)
a third detection means for detecting a third region having a predetermined second feature included in the image and different from the first feature;
Item 1 is an image processing device according to item 1, characterized in that, when the judgment unit judges that the state of the subject does not satisfy a predetermined condition, if the change in the difference between the luminance value of the first overlapping region and the luminance value of a second overlapping region in which the first region and the third region overlap is less than a predetermined threshold, the determination unit determines an exposure value using the photometric value, and if the change in the difference is equal to or greater than the predetermined threshold, the determination unit determines an exposure value using the photometric value most recently calculated by the calculation unit.
(Item 8)
8. The image processing device according to item 7, wherein the subject is a human face, and the region having the second feature is hair and beard.
(Item 9)
9. The image processing device according to any one of items 1 to 8, wherein the condition is a condition relating to the size of the second region included in the subject.
(Item 10)
10. The image processing device according to item 9, wherein the condition regarding the size of the second region is whether the size of the second region is larger or smaller than a predetermined first threshold value.
(Item 11)
a predetermined first threshold value for the size of the second region and a predetermined second threshold value that is greater than the first threshold value;
10. The image processing device according to item 9, wherein the determining means does not change the determination result of the condition after the size of the second region becomes less than the first threshold value until the size becomes equal to or greater than the second threshold value.
(Item 12)
10. The image processing device according to item 9, wherein the determining means determines whether the condition is met based on an average value of the sizes of a predetermined number of the second regions obtained in the past.
(Item 13)
10. The image processing device according to item 9, wherein the determining means determines whether the condition is met based on the amount of change in size of the second region most recently obtained.
(Item 14)
10. The image processing device according to item 9, wherein the size of the second region is the number of blocks when the image is divided into blocks each having a predetermined area.
(Item 15)
10. The image processing device according to item 9, wherein the size of the second region is a ratio of the second region to the first region.
(Item 16)
The image processing device described in any one of items 1 to 9, characterized in that the subject is a human face, and the state of the subject includes at least one of the orientation of the face, the presence and type of attachment, and the presence and color of beard and hair.
(Item 17)
Item 17. The image processing device according to item 16, wherein the condition is that the orientation of the face is within a predetermined angle range from the front.
(Item 18)
18. The image processing device according to any one of items 1 to 17, wherein the calculation means calculates the photometric value based on the difference between the luminance value of the first overlapping region and a predetermined target luminance value.
(Item 19)
19. The image processing device according to any one of items 1 to 18, wherein the subject is a human face, and the region having the first feature is skin.
(Item 20)
20. The image processing device according to any one of items 1 to 19, wherein the calculation means further calculates the photometric value using a luminance value of the entire image of the image.
(Item 21)
21. The image processing device according to any one of items 1 to 20,
an imaging means for capturing and outputting an image;
a control means for controlling exposure based on the exposure value determined by the determination means;
An electronic device comprising:
(Item 22)
an input step of inputting an image;
a first detection step of detecting a first region representing a subject included in the image;
a second detection step of detecting a second region having a predetermined first characteristic included in the image;
a determination step of determining a state of the subject;
a calculation step of calculating a photometric value using a luminance value of a first overlapping area where the first area and the second area overlap when the determination step determines that the state of the subject satisfies a predetermined condition; and a determination step of determining an exposure value based on the photometric value.
An image processing method comprising:
(Item 23)
a third detection step of detecting a third region having a predetermined second feature included in the image, the second feature being different from the first feature;
Item 23. The image processing method according to item 22, wherein, in the determination step, if it is determined in the determination step that the state of the subject does not satisfy a predetermined condition, and if the amount of change in the difference between the luminance value of the first overlapping region and the luminance value of a second overlapping region in which the first region and the third region overlap is less than a predetermined threshold, the exposure value is determined using the photometric value, and if the amount of change in the difference is equal to or greater than the predetermined threshold, the exposure value is determined using the photometric value most recently calculated in the calculation step.
(Item 24)
21. A program for causing a computer to function as each means of the image processing device according to any one of items 1 to 20.
(Item 25)
25. A computer-readable storage medium storing the program according to item 24.

The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention.

101: Operation unit, 102: Control unit, 103: Sensor unit, 104: A/D conversion unit, 105: Image processing unit, 106: Brightness calculation unit, 107: Subject detection unit, 108: AF processing unit, 1081: Lens, 109: AE processing unit, 1091: Mechanical mechanism, 110: EF processing unit, 111: Flash unit, 112: Encoder unit, 113: Image recording unit, 114: External connection unit, 115: Display unit, 116: Memory unit, 117: Subject condition determination unit, 118: Skin detection unit, 119: 402: Exposure compensation calculation unit, 401: Hair detection unit

Claims (33)

an input means for inputting an image;
a first detection means for detecting a first region representing a subject included in the image;
a second detection means for detecting a second region having a predetermined first characteristic included in the image;
a determination means for determining the state of the subject;
a calculation means for calculating a photometric value using a luminance value of a first overlapping area where the first area and the second area overlap when the determination means determines that the state of the subject satisfies a predetermined condition; and a determination means for determining an exposure value based on the photometric value.
and
An image processing device characterized in that, when the judgment means determines that the state does not satisfy the condition, the determination means determines an exposure value using a photometric value calculated using the luminance value of the first area . 2. The image processing device according to claim 1, wherein the calculation means applies a coefficient to the luminance value of the first region when calculating a photometric value using the luminance value of the first region to a value smaller than a coefficient to be applied to the luminance value of the first region when calculating a photometric value using the luminance value of the second region. The image processing device described in claim 1, characterized in that the condition relates to the size of the second region included in the subject. 4. The image processing apparatus according to claim 3 , wherein the condition regarding the size of the second region is whether the size of the second region is larger than a predetermined first threshold value. a predetermined first threshold value for the size of the second region and a predetermined second threshold value that is greater than the first threshold value;
4. The image processing device according to claim 3, wherein the determining means does not change the determination result of the condition until the size of the second region becomes equal to or greater than the second threshold value after the size of the second region becomes less than the first threshold value. 4. The image processing apparatus according to claim 3 , wherein the determining means determines whether the condition is met based on an average value of the sizes of a predetermined number of the second regions obtained in the past. The image processing device according to claim 3, characterized in that the determination means determines the condition based on an amount of change indicating the difference between the size of the second region most recently obtained and the size of the second region obtained one before . 4. The image processing apparatus according to claim 3 , wherein the size of the second region is the number of blocks obtained by dividing the image into blocks each having a predetermined area. 4. The image processing apparatus according to claim 3 , wherein the size of the second region is a ratio of the second region to the first region. The image processing device of claim 1, wherein the subject is a human face, and the state of the subject includes at least one of the following: facial orientation, the presence and type of clothing, and the presence and color of beard and hair. 11. The image processing device according to claim 10 , wherein the condition is that the face direction is within a predetermined angle range from the front. The image processing device described in claim 1, characterized in that the calculation means calculates the photometric value based on the difference between the luminance value of the first overlapping area and a predetermined target luminance value. 2. The image processing apparatus according to claim 1, wherein the subject is a human face, and the second region having the first feature is skin. The image processing device described in claim 1, characterized in that the calculation means further calculates the photometric value using the luminance value of the entire image of the image. an input means for inputting an image;
a first detection means for detecting a first region representing a subject included in the image;
a second detection means for detecting a second region having a predetermined first characteristic included in the image;
a third detection means for detecting a third region having a predetermined second feature included in the image, the second feature being different from the first feature;
a determination means for determining the state of the subject;
a calculation means for calculating a photometric value using a luminance value of a first overlapping area where the first area and the second area overlap when the determination means determines that the state of the subject satisfies a predetermined condition; and
a determining means for determining an exposure value based on the photometric value;
and
The image processing device is characterized in that, when the judgment means judges that the state of the subject does not satisfy a predetermined condition, the determination means compares the most recent difference between the luminance value of the first overlapping area and the luminance value of the second overlapping area where the first area and the third area overlap, and if the change in the difference between the most recent difference and the difference obtained just before the most recent difference is less than a predetermined threshold, the determination means determines the exposure value using the photometric value, and if the change in the difference is greater than or equal to the predetermined threshold, the determination means determines the exposure value using the photometric value calculated most recently by the calculation means. 16. The image processing apparatus according to claim 15, wherein the subject is a human face, and the region having the second feature is hair and a beard. 16. The image processing device according to claim 15, wherein the condition is a condition relating to the size of the second region included in the subject. 18. The image processing apparatus according to claim 17, wherein the condition regarding the size of the second region is whether the size of the second region is larger than a predetermined first threshold value. a predetermined first threshold value for the size of the second region and a predetermined second threshold value that is greater than the first threshold value;
18. The image processing device according to claim 17, wherein the determining means does not change the determination result of the condition until the size of the second region becomes equal to or greater than the second threshold value after the size of the second region becomes less than the first threshold value. 18. The image processing device according to claim 17, wherein the determining means determines whether the condition is met based on an average value of the sizes of a predetermined number of the second regions obtained in the past. The image processing device according to claim 17, characterized in that the determination means determines the condition based on an amount of change indicating the difference between the size of the second region most recently obtained and the size of the second region most recently obtained. 18. The image processing apparatus according to claim 17, wherein the size of the second region is the number of blocks obtained by dividing the image into blocks each having a predetermined area. 18. The image processing device according to claim 17, wherein the size of the second region is a ratio of the second region to the first region. 16. The image processing device according to claim 15, wherein the subject is a human face, and the state of the subject includes at least one of the orientation of the face, the presence and type of an attachment, and the presence and color of a beard and hair. 25. The image processing device according to claim 24, wherein the condition is that the orientation of the face is within a predetermined angle range from the front. 16. The image processing apparatus according to claim 15, wherein the calculation means calculates the photometric value based on the difference between the luminance value of the first overlapping area and a predetermined target luminance value. 16. The image processing apparatus according to claim 15, wherein the subject is a human face, and the second region having the first feature is skin. 16. The image processing apparatus according to claim 15, wherein the calculation means further calculates the photometric value using a luminance value of the entire image of the image. an image processing device according to any one of claims 1 to 28 ;
an imaging means for capturing and outputting an image;
a control means for controlling exposure based on the exposure value determined by the determination means;
An electronic device comprising: an input step of inputting an image;
a first detection step of detecting a first region representing a subject included in the image;
a second detection step of detecting a second region having a predetermined first characteristic included in the image;
a determination step of determining a state of the subject;
a calculation step of calculating a photometric value using a luminance value of a first overlapping area where the first area and the second area overlap when the determination step determines that the state of the subject satisfies a predetermined condition; and a determination step of determining an exposure value based on the photometric value.
and
An image processing method characterized in that, if it is determined in the judgment step that the state does not satisfy the condition, in the determination step, an exposure value is determined using a photometric value calculated using the luminance value of the first area . an input step of inputting an image;
a first detection step of detecting a first region representing a subject included in the image;
a second detection step of detecting a second region having a predetermined first characteristic included in the image;
a third detection step of detecting a third region having a predetermined second feature included in the image, the second feature being different from the first feature;
a determination step of determining a state of the subject;
a calculation step of calculating a photometric value using a luminance value of a first overlapping area where the first area and the second area overlap when it is determined in the determination step that the state of the subject satisfies a predetermined condition;
a determining step of determining an exposure value based on the photometric value;
and
In the determination step, when it is determined in the judgment step that the state of the subject does not satisfy a predetermined condition, if the change in the difference between the luminance value of the first overlapping area and the luminance value of the second overlapping area where the first area and the third area are overlapped, between the most recent difference and the difference obtained immediately before, is less than a predetermined threshold, the exposure value is determined using the photometric value, and if the change in the difference is equal to or greater than the predetermined threshold, the exposure value is determined using the photometric value calculated most recently in the calculation step. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 28 . A computer-readable storage medium storing the program according to claim 32 .