JP7634943B2 - Image processing device, image processing method, and program - Google Patents

Description

The present invention relates to image processing technology that performs gradation processing, etc. on captured images.

In recent years, high-definition technology has improved elements such as brightness range, resolution, and color gamut, making it possible to reproduce the colors and details of objects more faithfully. However, in terms of the expression of human skin in photographed images, since women and children in particular tend to prefer uniform skin with few shadows and unevenness, the image expression that accompanies such high image quality may not be desirable as an image. In particular, HDR (High Dynamic Range), a technology that expands the brightness range, has gradation characteristics with a higher peak brightness as the upper limit value than the conventional SDR (Standard Dynamic Range), and therefore the gradation on the high brightness side is significantly improved. Therefore, the shadows and unevenness that are considered undesirable for human skin as mentioned above tend to be more noticeable.

Patent Document 1 discloses a method for naturally reducing skin noise such as blemishes, freckles, and acne while preserving fine irregularities such as skin texture. Patent Document 2 discloses a method for reducing facial shine.

JP 2017-111595 A JP 2005-327009 A

However, there are also disadvantages when correcting a specific color region. For example, if the contrast of an image is lowered in order to reduce the undesirable phenomenon that occurs in an image of human skin as described above, the contrast of the entire image, including areas other than the skin, will decrease, and especially in the case of an HDR image, the high gradation will be lost. Even if the correction target is limited to the face area, the entire face may have a flat and unnatural impression. On the other hand, contrary to the case of human skin, there may be cases where you want to bring out more gradation. For example, wood grain and leaf veins often appear dark and tend to be difficult to bring out details, but if the contrast is adjusted to match these subjects, there is a problem that the contrast will be too strong in other areas. In the case of the technology disclosed in Patent Document 1, the method is limited to the specific skin noise (noise containing red components) described above, and cannot handle the shadows and unevenness of the skin. In addition, in the case of the technology disclosed in Patent Document 2, the correction process is limited to the high luminance range, so it cannot handle other luminance ranges.

The present invention aims to enable preferred expression of specific color areas while maintaining the gradation of the entire image.

The image processing device of the present invention comprises a decision means for deciding whether to perform correction, and a correction means for selecting a specific frequency band that matches a subject whose closest color is skin color as the correction target based on the decision by the decision means, and is characterized in that the correction means selects a luminance range in which the slope of the gradation characteristics of skin color is larger than the slope of the gradation characteristics of achromatic colors as the correction target.

The present invention makes it possible to achieve a desirable expression of specific color regions while maintaining the gradation of the entire image.

FIG. 1 is a diagram illustrating the configuration of a digital camera to which an image processing device according to an embodiment of the present invention is applied. 1 is a block diagram showing an example of a functional configuration of an image processing device; 4 is a flowchart of image processing according to the present embodiment. 5A to 5C are diagrams used to explain gradation characteristics and slope changes in the first embodiment. FIG. 2 is a block diagram showing a functional configuration for performing local contrast correction. 13 is a flowchart of a local gain map generation process. 1A and 1B are diagrams used to explain a mechanism of local contrast correction. 13A and 13B are diagrams used to explain gradation characteristics and slope changes in the second embodiment. FIG. 1 is a diagram showing the EOTF characteristics of PQ in SMPTE ST2084.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations shown in the drawings. The same configurations or processes will be described with the same reference symbols.
1 is a diagram showing a schematic configuration example of an imaging device (hereinafter, referred to as a digital camera or simply a camera) which is an application example of the image processing device of this embodiment. The digital camera of this embodiment is a camera with interchangeable lenses. The camera body has a mount part (not shown) which allows an interchangeable lens to be attached and detached, and the camera body and the interchangeable lens can be electrically connected via a contact part 111 provided on the mount part and a contact part 113 on the interchangeable lens side.

The control unit 101 on the camera body side is a one-chip microcomputer that incorporates an ALU (Arithmetic and Logic Unit), ROM, RAM, A/D converter, timer, and serial communication port (SPI). The control unit 101 controls the operation of the camera body and the interchangeable lens, for example, by executing a program stored in the ROM. The specific operation of the control unit 101 will be described later.

An output signal from an AF sensor 102 for focus detection and an output signal from an AE sensor 103 for exposure setting are sent to an A/D converter input terminal of the control unit 101 .
The signal processing circuit 104 controls the image sensor 105 according to instructions from the control unit 101, and applies A/D conversion and image signal processing to the image signal output by the image sensor 105 to obtain image data. The signal processing circuit 104 also performs necessary image processing such as compression and composition when recording the obtained image data. The function and operation of the signal processing circuit 104 will be described in detail later.

The memory 106 is a DRAM or the like, and is used as a work memory when the signal processing circuit 104 performs various signal processes, and as a VRAM when displaying images on a display 107 (described later).
The storage unit 108 is a non-volatile memory such as a flash memory, and receives the image data obtained by the image signal processing in the signal processing circuit 104 .

The display 107 is intended to be connected to an external display using a rear LCD display or a standard such as HDMI (registered trademark), and displays information such as camera settings, messages, and menu screens, as well as captured images. The display 107 is controlled by instructions from the control unit 101.

A motor 109, under the control of the control unit 101, moves a mirror inside the camera body up and down and charges a mechanical shutter.
The operation unit 110 is a group of input devices such as switches that are used by the user to operate the camera, including a release switch for instructing the start of preparations for shooting and the start of shooting, a shooting mode selection switch for selecting a shooting mode, direction keys, an enter key, and the like.
The contact section 111 is a contact for communicating with an interchangeable lens, and input/output signals of a serial communication port of the control section 101 are connected thereto.
The shutter driver 112 is connected to an output terminal of the controller 101 and drives the mechanical shutter.

The interchangeable lens is provided with a contact section 113 that forms a pair with the contact section 111 on the camera body side. A lens control section 114, which is a one-chip microcomputer similar to the control section 101, is connected to the contact section 113, thereby enabling communication between the lens control section 114 and the control section 101.
The lens control unit 114 executes a program stored in, for example, a ROM, and controls the operation of the interchangeable lens based on instructions from the control unit 101. The lens control unit 114 also notifies the control unit 101 of the camera body of information such as the state of the interchangeable lens.

The focus lens driving unit 115 is connected to the output terminal of the lens control unit 114 and drives the focus lens.
The zoom driver 116 drives the zoom lens under the control of the lens controller 114 to change the angle of view when photographing with the digital camera.
The aperture driver 117 adjusts the aperture opening size under the control of the lens controller 114 .

When an interchangeable lens is attached to the camera body, data communication is possible between the lens control unit 114 and the control unit 101 of the camera body via the contact unit 111 and the contact unit 113. In addition, power is also supplied from the camera body via the contact unit 111 and the contact unit 113 to drive the motor and actuator in the interchangeable lens. Lens-specific optical information and information on subject distance based on a distance encoder, which are necessary for the control unit 101 of the camera body to perform focus detection and exposure calculation, are sent from the lens control unit 114 of the interchangeable lens to the control unit 101 of the camera body by data communication. In addition, focus adjustment information and aperture information obtained as a result of the control unit 101 of the camera body performing focus detection and exposure calculation are sent from the control unit 101 of the camera body to the lens control unit 114 of the interchangeable lens by data communication. The lens control unit 114 of the interchangeable lens drives the focus lens according to the focus adjustment information, and drives the aperture according to the aperture information.

Figure 2 is a functional block diagram showing the image processing performed by the image processing device of this embodiment by function. In this embodiment, the signal processing circuit 104 in Figure 1 performs image processing corresponding to each functional unit shown in Figure 2. Note that the image processing by each functional unit may be realized by a hardware configuration such as a circuit, or may be realized by a CPU or the like executing a program related to the image processing of this embodiment. The image processing program may be prepared in advance in a ROM, or may be acquired via a removable memory (not shown) or a communication unit (not shown). In this case, the CPU performs the image processing according to this embodiment by executing the image processing program. The execution of the image processing program according to this embodiment may be performed by the control unit 101.

The raw image 201 is raw image data obtained by A/D converting the image signal of the image sensor 105, and is an image in which each pixel is composed of pixel values (intensities) corresponding to a single color, R (red), G (green), or B (blue). An image composed of pixel values corresponding to a single color is sometimes called a color plane.

The white balance correction unit (hereinafter referred to as the WB correction unit 202) performs white balance correction processing on the RAW image 201 to correct color casts caused by the light source and reproduce white. Specifically, the WB correction unit 202 first plots the RGB data of each pixel constituting the RAW image 201 in a predetermined color space, such as an xy color space, and integrates the RGB data of data plotted near the blackbody radiation locus in that color space, which is likely to be the light source color. Furthermore, the WB correction unit 202 finds white balance coefficients G/R and G/B for the R and B components from the integrated value. The WB correction unit 202 then performs white balance correction processing using the white balance coefficients generated by the above processing.

The luminance range selection unit 211 acquires the shooting conditions 210 when an image was captured, and determines what shooting mode the digital camera of this embodiment is in. Digital cameras have multiple shooting modes, and the shooting modes to be determined in this embodiment include whether the shooting mode is SDR shooting or HDR shooting, or whether the shooting mode has a function that prioritizes gradation on the high luminance side ON or OFF. The luminance range selection unit 211 then performs a luminance range setting process to set the luminance range to be corrected according to the shooting mode determined from the shooting conditions 210. The details of the luminance range setting process based on the shooting conditions 210 in the luminance range selection unit 211 will be described later.

The luminance tone correction unit 203 performs tone correction processing on the RAW image after the white balance correction processing. In the present embodiment, the luminance tone correction unit 203 performs tone correction processing such as raising the brightness of a specific subject area, such as a face, that has become dark in the image. For example, the luminance tone correction unit 203 judges the presence or absence of a subject, such as a face, to be corrected, or the presence or absence of pixels having color information similar to that of a subject, such as a face, to be corrected, for the input image, and determines whether or not to perform tone correction based on the judgment result. That is, the luminance tone correction unit 203 determines whether or not to activate tone correction processing based on the presence or absence of a subject to be corrected. When gradation correction is activated, the luminance tone correction unit 203 sets the luminance range set by the luminance range selection unit 211 as the luminance range to be corrected, and further sets a frequency band corresponding to the correction target, such as a face, as the correction target frequency band, and performs tone correction limited to those areas. In this embodiment, the luminance tone correction unit 203 performs tone correction limited to the luminance range and frequency band to be corrected, making it possible to obtain an image with favorable skin expression, etc., while maintaining the gradation of the entire image. The details of the correction target frequency band setting process in the luminance tone correction unit 203 and the tone correction process limited to the correction target frequency band and correction target luminance range will be described later.

The color interpolation unit 204 performs noise reduction and pixel interpolation on the image after the luminance tone correction process described above to generate an image in which R, G, and B color information is consistent for all pixels.

The development processing unit 205 performs development processing on an image in which R, G, and B color information is provided for all pixels by the color interpolation unit 204, to generate a color image. The development processing unit 205 performs processing such as matrix conversion and gamma conversion. In the gamma conversion processing, in the case of development processing compatible with SDR (hereinafter referred to as SDR development), the gamma characteristics of sRGB are mainly used. On the other hand, in the case of development processing compatible with HDR (hereinafter referred to as HDR development), the gamma characteristics used are, for example, the inverse characteristics of the EOTF (Electro-Optical Transfer Function) of PQ (Perceptual Quantization). Note that the gamma characteristics for HDR development may be a combination of EOTF and OOTF (Opto-Optical Transfer Function) characteristics. Figure 9 shows the EOTF characteristics of PQ standardized in SMPTE ST2084 (Society Motion Picture and Television Engineers ST2084).

The color brightness adjustment unit 206 performs color brightness adjustment processing on the color image after development processing by the development processing unit 205 to further improve the appearance of the image. For example, in the case of portrait photography, the color brightness adjustment unit 206 performs image correction processing to increase the brightness, and in the case of landscape photography, performs image correction processing to emphasize the saturation of greens and blue skies. These image correction processes are performed by applying a lookup table (LUT) for color brightness adjustment to color signal values such as RGB.

The compression unit 207 compresses the high-resolution image after the color and brightness adjustment process in the color and brightness adjustment unit 206 using a method such as HEVC (High Efficiency Video Codec).
Thereafter, a recording unit 208 generates image data (developed image 209) to be recorded on a recording medium such as a flash memory.

In this embodiment, as described above, the luminance tone correction unit 203 performs tone correction processing that limits the luminance range and frequency band to be corrected, thereby making it possible to achieve preferable skin expression, etc., while maintaining a certain degree of the gradation of the entire image.
In the functional block diagram of Fig. 2, the luminance tone correction unit 203 and the luminance range selection unit 211 are listed as separate functional units, but a determination/correction unit that combines these two functional units may be provided. In this case, the output of the WB correction unit 202 and the shooting conditions 210 are input to the determination/correction unit, which performs a determination process for determining whether correction should be performed and a gradation correction process for setting the correction target frequency and the correction target luminance range as the correction targets, respectively, based on the determination. The output of the determination/correction unit is then sent to the color interpolation unit 204.
Here, examples of the correction target luminance range and whether or not to execute (invoke) luminance correction processing for the correction target luminance range include two methods as described in the first and second embodiments below.

First Embodiment
First, the luminance range to be corrected in the first embodiment will be described.
In the first embodiment, a method for correcting a skin image that can be used in the same manner not only in the case of HDR development but also in the case of SDR development will be described. Image processing in the first embodiment will be described in detail below with reference to the functional blocks shown in FIG. 2 and the flowchart shown in FIG.

First, in step S301, the luminance tone correction unit 203 uses a face detection method to determine whether or not a human face area is included in the input image. The face detection method is a known method. For example, face detection methods using learning such as a neural network, and face detection methods that extract parts with characteristic physical shapes such as eyes and noses from a captured image by template matching processing, etc. are known. In addition, face detection methods that detect a face by detecting image features such as skin color and eye shape from an input image and analyzing them using statistical methods are known. If it is determined that a face area is included in the input image by these methods, the luminance tone correction unit 203 decides to perform the luminance tone correction process in this embodiment (i.e., decides to activate the luminance tone correction process).

Next, in step S302, the brightness range selection unit 211 acquires the shooting conditions 210 when the image was captured, and determines what shooting mode the digital camera of this embodiment is in. In this embodiment, the shooting mode to be determined is, as described above, whether the shooting mode is SDR shooting or HDR shooting, or whether the function that prioritizes the gradation on the high brightness side is ON or OFF.

Then, in the next step S303, the luminance range corresponding to the shooting mode determined in step S302 is set as the luminance range to be corrected. In this embodiment, the luminance range selection unit 211 holds information on the luminance range to be corrected that is calculated in advance for each shooting mode, and in step S303, the luminance range to be corrected corresponding to the shooting mode determined in step S302 is selected from the held luminance ranges to be corrected.

Hereinafter, a method for calculating the "luminance range to be corrected" will be described in detail with reference to FIG.
Graph 410 in FIG. 4 is a graph showing a chromatic color tone characteristic 401 and an achromatic color tone characteristic 402 in a shooting mode determined based on the shooting conditions 210 in step S302. The horizontal axis of graph 410 in FIG. 4 shows the input signal value of a RAW image, and the I value on the vertical axis is a luminance evaluation value calculated from the ICtCp color space that can correctly evaluate up to the high luminance range expressed in HDR. Graph 420 in FIG. 4 is a graph showing a change characteristic 403 of the slope of the output I value corresponding to the chromatic color tone characteristic 401 shown in graph 410, and a change characteristic 404 of the slope of the output I value corresponding to the achromatic color tone characteristic 402. The horizontal axis of graph 420 shows the input signal value of a RAW image, and the vertical axis shows the slope of the output I value. The tone characteristic indicates the correspondence relationship of the output I value to the input signal value (RAW value). The ICtCp color space is defined in ITU-R BT. 2100. I (intensity) is a luminance component that represents brightness, and Ct and Cp are color difference components that represent blue-yellow (derived from tritan=type 3 color vision) and red-green (derived from protan=type 1 color vision), respectively. In other words, in graphs 410 and 420 in Fig. 4, the gradation characteristics are characteristics that represent the output I value of the ICtCp color space relative to the input signal value.

In this embodiment, a "predetermined chromatic color" is a color with saturation determined by any saturation and hue, and an "achromatic color" is a gray color with no saturation. In this embodiment, the predetermined chromatic color is a color that is set in advance, for example, human skin color. However, human skin color can have various saturations and hues, so it is not uniquely determined. A reference value for human skin may be determined in advance, or it may be determined based on color information detected from a captured image, for example, by taking an average from a skin color area. In this embodiment, the hue and saturation of the reference skin color are determined in advance, and the gradation characteristics corresponding to them are acquired.

Here, when comparing the achromatic tone characteristics with the skin tone characteristics, it can be seen from the change characteristics 403 and 404 that the chromatic (skin tone) tone characteristics 401 have a steeper slope than the achromatic tone characteristics 402 in the luminance range 405 shown in FIG. 4. In the first embodiment, the luminance range to be corrected is the "luminance range in which the skin tone characteristics have a steeper slope than the achromatic tone characteristics" as shown in the luminance range 405. The steeper the slope of the output I value is, the more contrast there is in the tone characteristics, so that fine light and dark areas are more clearly expressed. Therefore, the luminance range in which the skin tone characteristics have a steeper slope is a luminance range in which the unevenness of the skin is more noticeable than in other colors. Therefore, by performing gradation correction only in this luminance range, it can be said that the skin can be expressed favorably without impairing the contrast of the entire image. That is, in the first embodiment, the luminance range of the input image in which the gradient of the gradation characteristics of a specific chromatic color is greater than the gradient of the gradation characteristics of an achromatic color is set as the luminance range to be corrected, and a specific luminance correction is performed on the luminance range to be corrected.

As mentioned above, the gradation characteristics of skin tones and achromatic colors are known as design values, and the luminance range selection unit 211 stores information on the luminance range corresponding to the results of comparing these in advance for each shooting mode. The luminance range selection unit 211 then selects the luminance range corresponding to the shooting mode determined in step S303 as the luminance range to be corrected.

In the above explanation, the saturation and hue for determining the skin tone characteristics are determined as reference values, but the reference saturation and hue may be single or multiple. When there are multiple reference saturations and hues, the luminance range selection unit 211 may hold information on the luminance range determined for each saturation and hue, and the luminance tone correction unit 203 may adopt the luminance range corresponding to the color (skin color) closest to the face area detected from the captured image as the luminance range to be corrected. Alternatively, the luminance range selection unit 211 may calculate a luminance range from the achromatic tone characteristics and the tone characteristics corresponding to a predetermined chromatic color obtained from the captured image, and output the luminance range to the luminance tone correction unit 203 as the luminance range to be corrected.

Furthermore, in the first embodiment, the frequency band to be subjected to gradation correction for the input image is set to an arbitrary band, and correction is performed limited to the frequency region in which unevenness and shading of the skin are of concern. Possible methods for determining the frequency band include setting a frequency band suitable for shading and unevenness of the skin, or determining (or changing) the frequency band based on the face area detected as the subject to be corrected (based on the size of the face area, for example).

In this way, in the first embodiment, the luminance range determined by comparing the gradation characteristics of a specific chromatic color with the gradation characteristics of an achromatic color is set as the luminance range to be corrected, and a frequency band to be corrected that matches the target of the gradation correction is set, and gradation correction is performed limited to that range. Gradation correction that limits the frequency band to be corrected is also called local contrast correction processing.

As described above, the tone correction in this embodiment is assumed to be local contrast correction.
Therefore, in step S304 in Fig. 3, the luminance tone correction unit 203 generates a local contrast gain map in the correction target luminance range selected in step S303 so that correction is applied only to the correction target frequency band set as described above. A gain map is a map in which gains for adjusting the contrast of an image for each pixel are two-dimensionally arranged. That is, the local contrast gain map in this embodiment is a gain map in which each gain is arranged so that correction is applied only to the correction target frequency band, and will be simply referred to as a local gain map hereinafter. Details of the local gain map generation process according to this embodiment will be described later.

Next, in step S305, the luminance tone correction unit 203 performs local contrast correction by applying a gain to the input image using the local gain map generated in step S304.
Here, if the luminance signal value at coordinates (x, y) of the input image is IN(x, y) and the value of the local gain map at coordinates (x, y) is GMlocal(x, y), the luminance signal value OUT(x, y) of the output image after gain processing is expressed as shown in the following equation (1).

OUT(x,y)=GMlocal(x,y)×IN(x,y) Equation (1)

The local gain map generation process in step S304 will be described in detail below with reference to Figs. 5 and 6. Fig. 5 is a block diagram showing an example of the configuration for the local gain map generation process, and Fig. 6 is a flowchart showing the flow of the local gain map generation process. In the case of this embodiment, the configuration shown in Fig. 5 is formed within the luminance tone correction unit 203 in Fig. 2.

The local gain map generating unit 501 is a part that generates a local gain map for adjusting the local contrast of an image. In the first embodiment, the local gain map generating unit 501 generates a low-frequency image from an input image, and generates a local gain map by applying gain processing to the low-frequency image. The local gain map generating unit 501 is configured to include a first low-frequency image generating unit 502, a second low-frequency image generating unit 503, a correction gain conversion unit 504, a gain map conversion unit 505, a gain map synthesis unit 506, and a local gain map calculation unit 507.

In step S601 of FIG. 6, the first low-frequency image generating unit 502 performs a reduction process on the input image to generate a first reduced image. The second low-frequency image generating unit 503 performs a further reduction process on the first reduced image to generate a second reduced image. In this way, the first reduced image is generated by reducing the input image, and the second reduced image is generated by further reducing the first reduced image. Furthermore, the first low-frequency image generating unit 502 generates a first low-frequency image having a different frequency level from the original image by enlarging the first reduced image to the size of the original image. Furthermore, the second low-frequency image generating unit 503 generates a second low-frequency image having a different frequency level from the original image and the first low-frequency image by enlarging the second reduced image to the size of the original image. As a method for reducing and enlarging such an image, a known method such as a process using a bilinear method can be used. Furthermore, as a method for obtaining a low-frequency image, a known method such as a low-pass filter process can be used.

In this embodiment, the input image is a luminance image, and hereinafter the input image is referred to as the input luminance image, the luminance image obtained by enlarging the first reduced image is referred to as the first layer luminance image, and the luminance image obtained by enlarging the second reduced image is referred to as the second layer luminance image. Note that in FIG. 5, the input luminance image at coordinates (x, y) is referred to as IN(x, y), the first layer luminance image is referred to as INM(x, y), and the second layer luminance image is referred to as INS(x, y).

In the next step S602, the correction gain conversion unit 504 generates a correction gain characteristic.
The generation of the correction gain characteristic will be described in detail with reference to FIG. 7. Graphs 710 and 712 in FIG. 7(a) and graphs 720 and 722 in FIG. 7(b) are diagrams in which the signal values of the image are plotted in the x direction, with the horizontal axis representing the position in the x direction and the vertical axis representing the value of the luminance signal. Graphs 711 in FIG. 7(a) and graph 721 in FIG. 7(b) are diagrams in which the signal values of the image are plotted in the x direction, with the horizontal axis representing the position in the x direction and the vertical axis representing the value of the gain signal. Graphs 710 to 712 in FIG. 7(a) are diagrams for explaining local contrast correction when the gradation characteristic is such that the gain decreases monotonically with respect to the luminance (the higher the luminance, the greater the gain). Graphs 720 to 722 in FIG. 7(b) are diagrams for explaining local contrast correction when the gradation characteristic is such that the gain increases monotonically with respect to the luminance (the higher the luminance, the greater the gain).

In the graph 710 in FIG. 7(a), the solid line indicates the input luminance signal 701, and the dotted line indicates the luminance signal 702 that has been reduced and filtered to a low frequency. The gain signal 703 (gain map) generated by local tone mapping of the signal values shown in the graph 710 in FIG. 7(a) with a gradation characteristic in which the gain decreases monotonically with respect to the luminance is represented by the dotted line in the graph 711 in FIG. 7(a). As shown in these graphs 710 and 711, the gain map (gain signal 703) is generated based on the low-frequency luminance signal 702, so it changes gradually with respect to the change in the input luminance signal 701. That is, while the input luminance signal 701 decreases sharply from left to right on the graph, the gain signal 703 increases gradually from left to right on the graph. As a result, as shown in the graph 712 in FIG. 7(a), the signal portion 705 in the luminance signal 704 after gain processing can produce a contrast improvement effect.

In FIG. 7B, where the gain has a monotonically increasing gradation characteristic with respect to luminance, graph 720 represents the input luminance signal 701 and the low-frequency luminance signal 702, just like graph 710 in FIG. 7A. Graph 721 in FIG. 7B shows the gain signal 706 generated by local tone mapping of the signal value with a gradation characteristic in which the gain increases monotonically with respect to luminance, and the gain map in this case is shown by the dotted line in the figure. As shown in these graphs 720 and 721, the gain map (gain signal 706) is generated based on the low-frequency luminance signal 702, so it changes gradually with respect to the change in the input luminance signal 701. That is, the input luminance signal 701 decreases rapidly from left to right on the graph, while the gain signal 706 decreases gradually from left to right on the graph. In this way, it can be seen that the graph 721 in FIG. 7B has the opposite characteristics to the graph 711 in FIG. 7A. As a result, as shown in graph 722 in FIG. 7(b), the signal portion 708 in the luminance signal 707 after gain processing can produce a contrast reduction effect.

The above is the mechanism by which contrast can be improved or, conversely, reduced by varying the correction gain characteristics.
Here, in this embodiment, since it is necessary to apply a correction that improves the appearance of the skin, it is necessary to set the local gain in a direction that reduces the local contrast. Therefore, in this embodiment, the correction gain characteristic is set to a monotonically increasing characteristic. Furthermore, in this embodiment, it is necessary to apply local contrast correction limited to the correction target luminance range selected in step S303. In addition, it is necessary to apply correction to the frequency band that is the target of gradation correction in an arbitrarily set frequency band. Therefore, in step S602, the correction gain characteristic is determined taking these limiting factors of the correction target into account.

Next, in S603, the gain map conversion unit 505 generates a gain map for each layer by applying the correction gain characteristics generated as described above to the input luminance image. Hereinafter, the gain map generated by applying the correction gain characteristics to an input luminance image that has not been subjected to reduction processing is referred to as a 1x layer gain map. Also, the gain map generated by applying the correction gain characteristics to the first layer luminance image INM generated in step S601 is referred to as a first layer gain map. Also, the gain map generated by applying the correction gain characteristics to the second layer luminance image INS is referred to as a second layer gain map.

In FIG. 5, the equal-magnification hierarchical gain map to which the above-mentioned correction gain characteristics have been applied to the input luminance image IN(x,y) is denoted as GML(x,y). The first-layer gain map to which the correction gain characteristics have been applied to the first-layer luminance image INM(x,y) is denoted as GMM(x,y), and the second-layer gain map to which the correction gain characteristics have been applied to the second-layer luminance image INS is denoted as GMS(x,y). The equal-magnification hierarchical gain map is then sent to the local gain map calculation unit 507, and the first-layer gain map and the second-layer gain map are sent to the gain map synthesis unit 506.

Next, in step S604, the gain map synthesis unit 506 generates a synthetic gain map by synthesizing the first hierarchical gain map and the second hierarchical gain map. The synthetic gain map is generated using a known method (see JP 2014-154108 A) that performs weighted addition according to the gain signal difference between the gain signal of the gain map with a large image size and the gain signal of the gain map with a small image size.

Next, in step S605, the local gain map calculation unit 507 generates a final local gain map to be used for local contrast correction from the equal-magnification hierarchical gain map and the composite gain map generated in step S604. If the signal of the composite gain map at coordinates (x, y) is GMC(x, y) and the final local gain map at coordinates (x, y) is GMlocal(x, y), GMlocal(x, y) is expressed by the following formula (2).

GMlocal (x, y) = GMC (x, y) - GML (x, y) Equation (2)

This concludes the description of the process of generating the local gain map in step S304 of FIG.
Then, in step S305, the gain processing unit 508 applies the final local gain map to the input luminance image IN(x,y) to generate an output luminance image OUT(x,y).

Second Embodiment
In the first embodiment, an example in which the development method is not particularly limited is given, but in the second embodiment, a skin correction method assuming HDR development (development having gradation characteristics with a peak luminance higher than that of conventional SDR development) will be described. In other words, if the gradation characteristics in the case of HDR development are gradation characteristics with a first peak luminance as the upper limit, the gradation characteristics in the case of SDR development are gradation characteristics with a second peak luminance as the upper limit, and the first peak luminance is a luminance value greater than the second peak luminance. In addition, the first peak luminance and the second peak luminance are maximum luminance values based on the output dynamic range.

The overall flow of image processing in the second embodiment is basically the same as the flow of the first embodiment described in FIG. 3, so illustration and description thereof will be omitted. The processing in the second embodiment that differs from the first embodiment is the method of calculating the "luminance range to be corrected" that is selected according to the shooting mode in step S303 in FIG. 3. The method of calculating the "luminance range to be corrected" in the second embodiment will be described below.

In the second embodiment, the skin color is determined by an arbitrary saturation and hue, as in the first embodiment, and this arbitrary saturation and hue may be determined in advance as a reference value for human skin, or may be determined by taking an average from the skin area of the captured image. In this embodiment, the hue and saturation of the reference skin color are determined in advance, and the gradation characteristics for SDR development and the gradation characteristics for HDR development are obtained for that skin color.

Graph 810 in FIG. 8 shows gradation characteristics 801 when developing in HDR and gradation characteristics 802 when developing in SDR (developing with gradation characteristics with a peak luminance lower than HDR as the upper limit) for skin color in the shooting mode determined in step S302. The horizontal axis of graph 810 indicates the input signal value of the RAW image, and the vertical axis indicates the output I value. Graph 820 in FIG. 8 is a graph showing change characteristics 803 and 804 of the slope of the output I value corresponding to gradation characteristics 801 and 802, respectively. The horizontal axis of graph 820 indicates the input signal value of the RAW image, and the vertical axis indicates the slope of the output I value.

When comparing the gradation characteristics in the case of HDR development with the gradation characteristics in the case of SDR development, it can be seen from the change characteristics 803 and 804 that in the luminance range 805, the gradation characteristics in the case of HDR development have a steeper slope than the gradation characteristics in the case of SDR development. In the second embodiment, gradation correction is performed with the luminance range to be corrected being the "luminance range in which the gradation characteristics in the case of HDR development have a steeper slope than the gradation characteristics in the case of SDR development" as shown in the luminance range 805. That is, in the second embodiment, gradation correction is performed with the luminance range to be corrected being the luminance range in which the slope of the gradation characteristics of a predetermined chromatic color in the gradation characteristics with the first peak luminance as the upper limit is steeper than the slope of the gradation characteristics of a predetermined chromatic color in the gradation characteristics with the second peak luminance as the upper limit.

As explained in the first embodiment, the greater the slope of the gradation characteristics, the greater the contrast, and the clearer the finer the light and dark areas. Therefore, the luminance range in which the gradation characteristics in HDR development have a greater slope is the luminance range in which the unevenness of the skin is more noticeable after HDR development than after SDR development. Therefore, by performing gradation correction limited to this luminance range, it can be said that skin can be expressed in a good way without compromising the contrast of the entire image. Also, as explained in the first embodiment, the gradation characteristics of skin color in SDR development and the gradation characteristics of skin color in HDR development can be obtained in advance as design values, so these can be compared in advance for each shooting mode.

Therefore, in the second embodiment, the luminance range selection unit 211 also holds information on the luminance range corresponding to the result of comparing these in advance for each shooting mode. Then, the luminance range selection unit 211 selects the luminance range corresponding to the shooting mode determined in step S303 as the luminance range to be corrected. Note that in the second embodiment, as in the case of the first embodiment described above, the saturation and hue for determining the gradation characteristics of the skin color are determined as references, but the reference saturation and hue may be single or multiple. When there are multiple reference saturations and hues, information on the luminance range determined for each saturation and hue may be held, and the luminance tone correction unit 203 may adopt the luminance range corresponding to the skin color closest to the face area detected from the captured image as the luminance range to be corrected.

In the second embodiment, the contents of the subsequent processes in steps S304 and S305 are similar to those in the first embodiment, and therefore a description thereof will be omitted.
The above is a description of the image processing in the image processing device of the second embodiment.

As described above, according to the first and second embodiments, it is possible to reduce shadows and unevenness in human skin while maintaining the gradation of the entire image. In particular, when HDR shooting or HDR development is performed, it is possible to favorably express human skin while suppressing loss of the HDR effect.
In the second embodiment, a determination and correction unit may also be provided that integrates both the luminance tone correction unit 203 and the luminance range selection unit 211 in the functional blocks of FIG.

<Other embodiments>
In the first embodiment described above, an example was described in which a luminance range in which the gradation characteristics of a predetermined chromatic color have a greater slope than the gradation characteristics of an achromatic color is determined as a luminance range to be corrected. In the second embodiment, a method was described in which a luminance range in which the gradation characteristics of a predetermined chromatic color in HDR development have a greater slope than the gradation characteristics of a predetermined chromatic color in SDR development is determined as a luminance range to be corrected.

As yet another embodiment, it is also possible to determine a luminance range in which the magnitude relationship of the gradient of the gradation characteristics is reversed as the luminance range to be corrected.
For example, in the first embodiment, when the predetermined chromatic color is green, the luminance range in which the green gradation characteristics have a smaller slope than the achromatic gradation characteristics is set as the luminance range to be corrected. That is, the luminance range in which the gradation characteristics of the predetermined chromatic color have a smaller slope than the achromatic gradation characteristics is set as the luminance range to be corrected, among the luminance ranges of the input image. Then, luminance correction processing is performed on the luminance range to be corrected. Then, by generating a local gain map in a direction that improves contrast, it is possible to improve areas where the details of the veins are difficult to see.

It is also possible to combine correction gains for color regions where contrast should be strengthened and color regions where contrast should be weakened. For example, in a captured image that includes both leaves and people, based on the flow of the flowchart in Figure 3, a correction gain is set in the direction of reducing contrast for luminance regions where the skin tone characteristics have a greater slope than the achromatic tone characteristics. Also, a correction gain is set in the direction of improving contrast for luminance regions where the green tone characteristics have a smaller slope than the achromatic tone characteristics. In this case, it is possible to express both subjects in a favorable manner.

As an example in which the magnitude relationship of the gradient of the gradation characteristics is reversed from that of the second embodiment, the luminance range in which the gradient of the gradation characteristics of a given chromatic color in HDR development is smaller than the gradient of the gradation characteristics of the given chromatic color in SDR development may be set as the luminance range to be corrected. In other words, the luminance range in which the gradient of the gradation characteristics of a given chromatic color in the gradation characteristics with the first peak luminance as the upper limit is smaller than the gradient of the gradation characteristics of a given chromatic color in the gradation characteristics with the second peak luminance as the upper limit is used as the luminance range to be corrected for gradient correction.

In the above-described embodiment, the luminance tone correction unit 203 performs local contrast correction such as performing luminance correction with a different correction amount for each frequency band, but the luminance correction may be performed with the same correction amount for all frequency bands.
Furthermore, in each of the above-described embodiments, an example was given in which the input and output in the local contrast correction process were luminance images, but the input and output images may be images in other formats, such as so-called YUV.

The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.
The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

101: Control unit, 104: Signal processing circuit, 202: WB correction unit, 203: Brightness tone correction unit, 204: Color interpolation unit, 205: Development processing unit, 206: Color brightness adjustment unit, 207: Compression unit, 208: Recording unit, 211: Brightness range selection unit

Claims (22)

A determination means for determining whether to perform a correction;
a correction means for correcting a specific frequency band that is adapted to a subject having a skin color as a closest color based on the determination by the determination means;
having
The image processing device according to claim 1, wherein the correction means corrects a luminance range in which a gradient of the gradation characteristics of flesh color is larger than a gradient of the gradation characteristics of achromatic colors. The correction means is
a band setting means for setting a specific frequency band of an input image as the frequency band to be corrected;
a luminance range setting means for setting a luminance range in which the gradient of the gradation characteristic of the skin color is larger than that of the gradation characteristic of an achromatic color, as a luminance range to be corrected, from among the luminance ranges of the input image;
having
2. The image processing apparatus according to claim 1, wherein a gradation correction is performed on the input image based on the set frequency band and the set luminance range. A determination means for determining whether to perform a correction;
a correction means for correcting a specific frequency band that is adapted to a subject having a color that is closest to green based on the determination by the determination means;
having
The image processing device according to claim 1, wherein the correction means corrects a luminance range in which a gradient of the green gradation characteristic is smaller than a gradient of the achromatic gradation characteristic. The correction means is
a band setting means for setting a specific frequency band of an input image as the frequency band to be corrected;
a luminance range setting means for setting a luminance range in which the gradient of the green gradation characteristic is smaller than the gradient of the achromatic gradation characteristic as a luminance range to be corrected, from among the luminance ranges of the input image;
having
4. The image processing apparatus according to claim 3, wherein a gradation correction is performed on the input image based on the set frequency band and the set luminance range. A determination means for determining whether to perform a correction;
a correction means for correcting a specific frequency band that is adapted to a subject having a skin color as a closest color based on the determination by the determination means;
having
The image processing device is characterized in that the correction means corrects a luminance range in which a slope of the skin color gradation characteristic in the gradation characteristic having an upper limit value of a first peak luminance is larger than a slope of the skin color gradation characteristic in the gradation characteristic having an upper limit value of a second peak luminance. The correction means is
a band setting means for setting a specific frequency band of an input image as the frequency band to be corrected;
a luminance range setting means for setting, from among the luminance ranges of the input image, a luminance range in which a gradient of a skin tone characteristic in a gradation characteristic having an upper limit value of a first peak luminance is greater than a gradient of the skin tone characteristic in a gradation characteristic having an upper limit value of a second peak luminance, as a luminance range to be corrected;
having
6. The image processing apparatus according to claim 5, wherein a gradation correction is performed on the input image based on the set frequency band and the set luminance range. A determination means for determining whether to perform a correction;
a correction means for correcting a specific frequency band that is adapted to a subject having a color that is closest to green based on the determination by the determination means;
having
The image processing device is characterized in that the correction means corrects a luminance range in which a slope of the green gradation characteristic in the gradation characteristic having an upper limit value of a first peak luminance is smaller than a slope of the green gradation characteristic in the gradation characteristic having an upper limit value of a second peak luminance. The correction means is
a band setting means for setting a specific frequency band of an input image as the frequency band to be corrected;
a luminance range setting means for setting, as a luminance range to be corrected, a luminance range in which a gradient of a green gradation characteristic in a gradation characteristic having an upper limit value of a first peak luminance is smaller than a gradient of the green gradation characteristic in a gradation characteristic having an upper limit value of a second peak luminance;
having
8. The image processing apparatus according to claim 7, wherein a gradation correction is performed on the input image based on the set frequency band and the set luminance range. The image processing device according to any one of claims 5 to 8, characterized in that the first and second peak luminances are maximum luminance values based on an output dynamic range. The image processing device according to any one of claims 5 to 9, characterized in that the first peak luminance is a luminance value greater than the second peak luminance. The image processing device according to any one of claims 1 to 4, characterized in that the correction means determines the specific frequency band based on a subject to be corrected in the input image. The image processing device according to any one of claims 1 to 11, characterized in that the decision means decides whether or not to perform the correction based on the presence or absence of a subject to be corrected in the input image, or the presence or absence of pixels having a color close to the color of the subject to be corrected. The image processing device according to any one of claims 1 to 12, characterized in that the correction means performs the correction with a different correction amount for each frequency band or with the same correction amount for all frequency bands. The image processing device according to any one of claims 1 to 13, characterized in that the gradation characteristics are characteristics that represent the output of the ICtCp color space for an input signal value. The image processing device according to any one of claims 1, 2, 5, and 6, characterized in that the skin color is determined based on a preset skin color or a skin color obtained from an input image. The image processing device according to any one of claims 1, 2, 5, 6 and 15, characterized in that the correction means sets, as the luminance range to be corrected, a luminance range selected from a plurality of luminance ranges prepared in advance according to the shooting mode when the input image was acquired by shooting, or a luminance range calculated based on the gradation characteristics according to the achromatic gradation characteristics and the skin color obtained from the input image. The image processing device according to claim 16, characterized in that, when a luminance range selected from the plurality of luminance ranges prepared in advance according to the shooting mode is set as the luminance range to be corrected, the correction means selects a luminance range corresponding to a color closest to the skin color obtained from the input image. An image processing method executed by an image processing device, comprising:
a determination step of determining whether to perform a correction;
a correction step of selecting a specific frequency band that is adapted to a subject having a skin color as a closest color based on the determination step;
having
The image processing method according to the present invention, wherein the correction step targets a luminance range in which the gradient of the gradation characteristics of flesh color is greater than the gradient of the gradation characteristics of achromatic colors. An image processing method executed by an image processing device, comprising:
a determination step of determining whether to perform a correction;
a correction step of selecting a specific frequency band that is matched with a subject having a color that is closest to green as a correction target based on the determination step;
having
The image processing method according to the present invention, wherein the correction step targets a luminance range in which the gradient of the green gradation characteristic is smaller than the gradient of the achromatic gradation characteristic. An image processing method executed by an image processing device, comprising:
a determination step of determining whether to perform a correction;
a correction step of selecting a specific frequency band that is adapted to a subject having a skin color as a closest color based on the determination step;
having
an upper limit of the first peak luminance, and a luminance range in which a slope of the gradation characteristic of skin color in the gradation characteristic having an upper limit of the second peak luminance is larger than a slope of the gradation characteristic of skin color in the gradation characteristic having an upper limit of the second peak luminance, An image processing method executed by an image processing device, comprising:
a determination step of determining whether to perform a correction;
a correction step of selecting a specific frequency band that is matched with a subject having a color that is closest to green as a correction target based on the determination step;
having
The image processing method according to the present invention, characterized in that, in the correction step, a luminance range in which a slope of the green gradation characteristic in the gradation characteristic having an upper limit value equal to a first peak luminance is smaller than a slope of the green gradation characteristic in the gradation characteristic having an upper limit value equal to a second peak luminance is targeted for correction. A program for causing a computer to function as an image processing device according to any one of claims 1 to 17.

Images