JP7634944B2 - Image processing device, image processing method, imaging device, and program - Google Patents

Description

The present invention relates to image processing that performs gradation correction based on input display luminance and brightness information.

Dynamic range expansion is a process that reproduces the gradations of an image more accurately. For example, a high dynamic range (HDR) camera system can capture images that reproduce the gradations on the high-brightness side more accurately than the human eye, making it possible to accommodate the increasing brightness of displays.

When outputting images with an HDR camera system, an OETF curve based on the OETF (Opto-Electronic Transfer Function) characteristics is used. The OETF characteristics are the inverse of the EOTF (Electro-Optical Transfer Function) characteristics described in the HDR standard ST2084, which is used by HDR monitors. By converting images taken with a camera using the OETF curve, it is possible to express gradations on the high brightness side on an HDR monitor. In other words, HDR images have a wider range of expressible brightness and a richer gradation expression than conventional SDR (Standard Dynamic Range) images.

On the other hand, if the number of blown-out areas in scenes such as ocean or nightscapes is reduced in HDR images compared to SDR images, the impression may be that the brightness is weakened. In order to take advantage of the expression of brightness on the high-brightness side achieved through HDR processing and create images that emphasize the sparkle of the water surface, the light sources in a nightscape, and the luster of metal, technology is required to control the sense of brightness.

Patent Document 1 discloses a control for imaging by expanding the gradation on the high luminance side relative to the reference gradation characteristics, and a technology for controlling the amount of enhancement of the display luminance based on the amount of expansion of the gradation on the high luminance side in the imaging step when displaying the image. Patent Document 2 discloses a technology that enables the specification of the image area and gradation range for which gradation correction processing of an image is performed, and performs gradation correction processing independently inside and outside the attention area. A process of determining gradation correction characteristics based on the central luminance value of the attention area and performing gradation correction within the attention area, and a process of determining gradation correction characteristics based on statistics calculated for pixels outside the attention area and performing gradation correction outside the attention area are performed.

JP 2016-134862 A JP 2010-256536 A

However, the conventional techniques have the following problems with respect to the control of the brightness of an input image. In the technique disclosed in Patent Document 1, exposure control and gradation correction are performed in a direction that expands the gradation on the high luminance side, so it is not suitable for correction to make a subject brighter by controlling in a direction that reduces the gradation on the high luminance side. In addition, since gradation correction is performed to brighten dark areas, there is a possibility that the S/N ratio (signal-to-noise ratio) of dark areas will decrease. In addition, in the technique disclosed in Patent Document 2, the gradation correction characteristics are not determined after the luminance of the scene is associated with the absolute luminance of the display, so it is not suitable for correction to make a subject brighter according to the display luminance.
An object of the present invention is to realize image processing capable of controlling the brightness of an image.

An apparatus according to an embodiment of the present invention is an image processing apparatus that performs gradation correction of an image, and includes a first acquisition means for acquiring information on display luminance of the image, a second acquisition means for acquiring information on brightness representing a degree of reduction in gradation of a luminance region in the image having a luminance value equal to or greater than a threshold value, and a correction means for performing gradation correction of the image using the acquired information on display luminance and information on brightness. When the display luminance value acquired by the first acquisition means is a first display luminance value, an initial value of the display luminance is a second display luminance value, and the first display luminance value is greater than the second display luminance value, the correction means performs gradation correction to bring the brightness of the luminance region in which the input luminance is equal to or greater than the threshold value closer to the first display luminance value, in accordance with the information on brightness for a gradation characteristic before gradation correction representing a relationship between an input luminance and an output luminance , by making the output luminance of the gradation characteristic in the luminance region in which the input luminance is equal to or greater than the output luminance of the gradation characteristic before gradation correction.

The present invention makes it possible to realize image processing that can control the brightness of an image.

1 is a block diagram showing a configuration of an imaging apparatus according to an embodiment; FIG. 2 is a block diagram showing a configuration of an image processing unit according to the first embodiment; FIG. 2 is a block diagram showing a configuration of a tone correction unit according to the first embodiment. 5 is a flowchart illustrating the operation of an image processing unit according to the first embodiment. FIG. 13 is a diagram showing an example of a display of an input screen for inputting display luminance and brightness. FIG. 4 is a diagram showing gradation characteristics in the first embodiment. FIG. 11 is a diagram showing gradation characteristics according to the degree of chromaticity in the first embodiment. FIG. 11 is a block diagram showing a configuration of an image processing unit according to a second embodiment. 10 is a flowchart illustrating the operation of an image processing unit according to the second embodiment. FIG. 13 is a diagram showing an example in which a plurality of high luminance areas are detected. 11A and 11B are diagrams illustrating a detection process of a high luminance area. FIG. 13 is a block diagram showing a configuration of an image processing unit according to a third embodiment. 13 is a flowchart illustrating the operation of an image processing unit according to the third embodiment. 11 is a diagram showing the relationship between display luminance and brightness level in the calculation of the brightness level. FIG. 11A and 11B are diagrams illustrating a calculation process of a brightness level using a histogram.

Preferred embodiments of the present invention are described below in detail with reference to the accompanying drawings. In each embodiment, an image processing device is an image capture device such as a smartphone or digital camera. An image processing unit performs gradation correction processing on an image captured by the image capture device of the embodiment or an image recorded after receiving it via a network, according to the display luminance and brightness.

Figure 1 is a block diagram showing an overview of an example configuration applied to an imaging device 100. The imaging unit 101 includes an imaging optical system including a lens, an imaging element (image sensor), and an A (analog)/D (digital) conversion processing unit. The imaging unit 101 captures an image of a subject and generates image data based on a control signal output from a system control unit 103 in accordance with an instruction from an operation unit 108.

The image processing unit 102 performs predetermined image processing on the image data acquired by the imaging unit 101, the image data recorded by the recording unit 106, or the data of the image file input from the network processing unit 107. For example, development processing such as white balance processing, demosaic processing, and gamma processing is performed. In addition to performing such signal processing, the image processing unit 102 also performs tone correction processing of the image according to the display brightness and brightness level based on control signals output from the system control unit 103 and the display control unit 105. The processing content of the image processing unit 102 related to the present invention will be described later.

The system control unit 103 includes, for example, a CPU (Central Processing Unit) and executes a predetermined program to control the entire imaging device. For example, the system control unit 103 performs drive control of the imaging unit 101 according to instruction signals transmitted from the network processing unit 107 and the operation unit 108.

The display unit 104 includes a display device such as a liquid crystal display or an organic EL (Electro Luminescence) display, and receives the output signal from the image processing unit 102 to display images, an interface screen, and the like. For example, the display unit 104 supports HDR display conforming to ITU-R BT.2100, and can display HDR images created by the image processing unit 102. The display control unit 105 controls the display of the display unit 104 in response to the output signal from the image processing unit 102.

The recording unit 106 performs processing to record image data, etc., on an information recording medium. The information recording medium may be a memory card equipped with a semiconductor memory, or a package containing a rotating recording medium such as a magneto-optical disk. For example, the information recording medium is detachable from the imaging device 100.

The network processing unit 107 can acquire image file data from an external input device via a network. Alternatively, the network processing unit 107 acquires an output signal from the image processing unit 102 and performs processing to send the image signal via the network to an external display device or an external device (such as a personal computer). Images created by the image processing unit 102 can not only be sent to the display unit 104, but can also be transmitted by the network processing unit 107 to an external output device. The image can be displayed on a display unit provided in the destination device.

The bus 109 is used to transmit and receive image data and the like between the imaging unit 101, image processing unit 102, system control unit 103, display unit 104, display control unit 105, recording unit 106, and network processing unit 107.

The following describes in detail the configuration of the image processing unit of the imaging device in the first to third embodiments, which is capable of providing HDR images with controlled brightness by taking advantage of the brightness expression in high-brightness areas.

[First embodiment]
The image processing unit 102 of this embodiment includes a first acquisition unit that performs input processing of the display luminance that is the basis for correction, and a second acquisition unit that performs input processing of the degree to which the gradation on the high luminance side is reduced as the brightness level. The input screens for each piece of information from the first and second acquisition units are displayed on the same screen of the display unit 104. Gradation correction is performed based on the input information of the display luminance and the input information of the brightness level.

The processing contents of the image processing unit 102 related to this embodiment will be described with reference to FIG. 2. FIG. 2 is a block diagram illustrating the display unit 104, the display control unit 105, and the image processing unit 102 that are related to this embodiment.

The image processing unit 102 includes a display luminance input unit 201, a brightness level input unit 202, and a gradation correction unit 203. Input/output processing of display luminance information is performed between the display luminance input unit 201 and the display control unit 105. Input/output processing of brightness level information is performed between the brightness level input unit 202 and the display control unit 105. The display control unit 105 controls the display of the display unit 104 based on the display luminance information and the brightness level information.

The tone correction unit 203 acquires input image data and performs tone correction processing based on the display luminance information and brightness information. The input image to the tone correction unit 203 is assumed to be an HDR image developed by the signal processing performed by the image processing unit 102. The HDR image is developed in a specified color space (Rec.2020) based on, for example, the PQ gamma curve described in ST2084, and the same is true for the output image.

Figure 3 is a block diagram showing the configuration of the gradation correction unit 203, and Figures 4 (A) and (B) are flowcharts explaining the operation of the image processing unit 102. In S401 of Figure 4 (A), processing is executed to accept input of display luminance and brightness. The display luminance input unit 201 instructs the display control unit 105 to perform display processing of a display luminance input area for inputting the display luminance that is the basis for correction. The brightness input unit 202 instructs the display control unit 105 to perform display processing of a brightness input area for inputting the degree of reduction of the gradation on the high brightness side as the brightness. The display luminance input area and the brightness input area are displayed on the same screen in the display unit 104, and processing is executed to accept input of the display luminance and brightness. A specific example will be described later with reference to Figure 5.

In S402, a process is performed to obtain the display luminance information input in S401. For example, the maximum luminance at which the input image is displayed is obtained as the display luminance, but an average luminance may also be obtained.

In S403, a process is performed to obtain the brightness information input in S401. The input brightness information is the proportion of pixels to be displayed at the display brightness obtained in S402. In this embodiment, the slope of the gradation characteristics of the high brightness area is obtained as the brightness. By changing the slope of the gradation characteristics of the high brightness area, the proportion of pixels to be displayed at the display brightness obtained in S402 changes. The slope of the gradation characteristics of the high brightness area will be described in detail together with the gradation correction process in S404 described later.

In S404, the gradation correction unit 203 performs gradation correction processing on the input image according to the display luminance acquired in S402 and the brightness acquired in S403. The gradation correction processing performed in S404 will be described with reference to FIG. 3.

As shown in FIG. 3, the gradation correction unit 203 includes a gradation characteristic calculation unit 301 and a gradation processing unit 302. The input image to the gradation correction unit 203 is an HDR image developed by the signal processing performed by the image processing unit 102. The gradation characteristic calculation unit 301 acquires information on the display luminance and the brightness level, calculates the gradation characteristics, and outputs the calculation result to the gradation processing unit 302. The gradation processing unit 302 performs gradation processing on the acquired input image according to the calculated gradation characteristics, and outputs image data that has been gradation processed. Details of the gradation correction will be described later using the flowchart in FIG. 4(B).

Figure 5 shows an example of an interface screen used to input display brightness and brightness. Figure 5(A) shows an example in which slide bars are displayed in the display brightness input area and brightness input area on the display screen 601. A tone curve area 607 showing the characteristics of the control of display brightness and brightness is displayed below the image area 606.

The display luminance input unit 201 accepts input of the display luminance by operating the slide bar shown to the right of the tone curve area 607. This slide bar is composed of a display luminance input bar 602 and a display luminance input slider 603. The display luminance input bar 602 is an axis that changes the display luminance of the input image according to the position of the display luminance input slider 603.

The brightness input unit 202 accepts input of the brightness level by operating a slide bar shown below the tone curve area 607. This slide bar is composed of a brightness level input bar 604 and a brightness level input slider 605. The brightness level input bar 604 is an axis that indicates the degree of gradation on the high brightness side, and the degree to which the gradation is reduced, and has the characteristic that the gradation on the high brightness side of the input image is changed depending on the position of the brightness level input slider 605.

In the image area 606, an image that has been tone-corrected in S404 of FIG. 4A based on the information input by the slider operation is displayed. In the example of FIG. 5A, a tone curve area 607 showing the characteristics of the control of display brightness and brightness is displayed on the display screen 601.

Figure 5 (B) shows an example in which the tone curve area 607 is not displayed. That is, an image area 609 and multiple slide bars are displayed on the display screen 608. On the right side of the image area 609, a slide bar for inputting display luminance, which is composed of a display luminance input bar 610 and a display luminance input slider 611, is displayed. In addition, below the image area 609, a slide bar for inputting brightness, which is composed of a brightness input bar 612 and a brightness input slider 613, is displayed.

The input values in the display brightness input section 201 and brightness level input section 202 do not need to be continuous values, but may be stepped values such as strong, medium, weak, etc. The user interface processing can be changed according to the specifications, and is not limited to a specific input form.

Next, the operation of the gradation correction unit 203 will be described with reference to Fig. 4(B) and Fig. 6. Fig. 6 is a diagram for explaining gradation characteristics. In S501 of Fig. 4(B), the gradation characteristic calculation unit 301 calculates the gradation characteristics using the display luminance information acquired in S402 of Fig. 4(A) and the brightness information acquired in S403.

Figure 6 (A) is a graph showing the gradation characteristics of this embodiment, with the horizontal axis representing input luminance and the vertical axis representing output luminance. Threshold value 701 on the horizontal axis is threshold value th that defines high luminance areas in the input image. Brightness value 702 on the vertical axis is the initial value of display luminance, and luminance value 703 is the display luminance value acquired in S402 of Figure 4 (A). The gradation characteristics before gradation correction are shown by dashed dotted line 704, and the gradation characteristics calculated by gradation characteristic calculation unit 301 according to the display luminance and brightness acquired by gradation characteristic calculation unit 301 are shown by solid line 705.

In this embodiment, a region where the luminance value is equal to or greater than the threshold value th is determined as a high luminance region, using a predetermined threshold value 701 as a reference. In other words, with regard to the brightness of the high luminance region, the gradation characteristics are calculated so as to approach the display luminance (luminance value 703) obtained in S402 according to the brightness level obtained in S403 of FIG. 4A. In addition, in the luminance region 706, the input luminance is mapped to the display luminance (luminance value 703) and the gradation is reduced, resulting in a gradation characteristic that increases the sense of luminance.

（式１）中のmin［Ａ，Ｂ］はＡとＢのうちの小さい方を選択する関数である。 In Fig. 6, the input luminance is represented as x, the output luminance as y, the gradient of the gradation characteristic before gradation correction as _g1 , and the threshold value defining the high luminance region as th. The gradient indicating the brightness level acquired in S403 in Fig. 4A is represented as _g2 , and the luminance value 703 of the display luminance acquired in S402 is represented as limit. The gradation characteristic is expressed by the following (Equation 1).

In equation 1, min[A, B] is a function that selects the smaller of A and B.

Figure 6 (A) shows an example of a gradation characteristic expressed by a straight line shown in (Equation 1), that is, a characteristic expressed by a linear function. On the other hand, Figure 6 (B) shows an example of a characteristic expressed by a non-linear function. Curve 709 is a curve that passes through points 707 and 708 where the slope changes in (Equation 1), and can be expressed using spline interpolation, etc.

In this embodiment, the histogram shown in FIG. 6(C) is used to calculate the threshold value th for determining high brightness areas. The horizontal axis of the histogram represents the brightness value for the evaluation value generated from the input image. This brightness value represents the brightness in the color space after the input image is converted into the color space for the evaluation value. The vertical axis of the histogram represents the number of pixels having that brightness value. As an example of the color space for the evaluation value, this embodiment employs the ICtCp color space described in ITU-R BT.2100, and the value of the brightness component I in the ICtCp color space is taken as the brightness value.

When the histogram shown in FIG. 6(C) is viewed from the high luminance side, the first minimum luminance value 710 is detected at the valley position, and luminance value 710 is determined as the threshold value th that defines the high luminance region. Luminance value 711 is the luminance value that indicates the end point (rightmost point) of the histogram of the input image, and region 712 from luminance value 710 to luminance value 711 is determined as the high luminance region. The method is not limited to calculating the threshold value using a histogram, and a method in which the threshold value is set as a fixed value may also be used.

In this embodiment, an example of calculating the gradation characteristic based only on the brightness has been described. Fig. 7 shows an example of calculating the gradation characteristic based on the degree of chromaticity of the input image. For example, the saturation (denoted as S) calculated by the following (Equation 2) from the color difference signal CtCp in the ICtCp color space can be used as an evaluation value indicating the degree of chromaticity.

Figure 7 (A) is a graph showing the relationship between saturation S and the degree of chromaticity (denoted as Sadj) on the horizontal axis and the degree of chromaticity (denoted as Sadj) on the vertical axis. Based on the saturation S determined by (Equation 2), the degree of chromaticity Sadj is determined by using a reference table corresponding to the graph of Figure 7 (A). The saturation S input to the reference table is the saturation that is the representative value of the input image, and for example, the average value or maximum value of the input image is input. In addition, an upper limit value and a lower limit value are set for the degree of chromaticity Sadj. In the example of Figure 7 (A), the upper limit value is set to 1, and the lower limit value is set to zero. Within the range from the lower limit value to the upper limit value, the higher the saturation S, the higher the degree of chromaticity Sadj. In the example of Figure 7 (A), when the saturation S is less than the first threshold value, the degree of chromaticity Sadj is the lower limit value, and when the saturation S is greater than the second threshold value, the degree of chromaticity Sadj is the upper limit value. When the saturation S is greater than or equal to the first threshold and less than or equal to the second threshold, the saturation S and the chromaticity degree Sadj are in a linear relationship.

Figure 7 (B) is a diagram for explaining the calculation of the gradation characteristics, and the horizontal and vertical axes are set in the same way as in Figure 6 (A). The gradation characteristics calculation unit 301 calculates the gradation characteristics using the degree of chromaticity. Figure 7 (B) shows a first graph line 801 and a second graph line 802 representing different gradation characteristics, and a display luminance 803 obtained in S402 of Figure 4 (A). The slope of the gradation characteristics represented by the first graph line 801 is set to the slope indicating the brightness degree obtained in S403 of Figure 4 (A). The slope of the gradation characteristics represented by the second graph line 802 shows a slope that is less prone to color saturation than the slope of the gradation characteristics represented by the first graph line 801. In other words, the slope angle of the second graph line 802 is smaller than the slope angle of the first graph line 801.

（式３）は、彩度Sが高くなるにつれて、第１のグラフ線８０１で表される階調特性が、第２のグラフ線８０２で表される階調特性（つまり高彩度部分の色飽和が減少する階調特性）に近づく事を示している。 The initial value of the gradient indicating the brightness level is denoted as _g1 , the input brightness as x, the output brightness as y, and the threshold value defining the high brightness region as th. The gradient of the gradation characteristic represented by the first graph line 801 is denoted as _g2 , and the display brightness 803 is denoted as limit. The gradation characteristic according to the chromaticity level Sadj of the input image is expressed by the following (Equation 3).

Equation 3 indicates that as the saturation S increases, the gradation characteristics represented by the first graph line 801 approach the gradation characteristics represented by the second graph line 802 (i.e., the gradation characteristics in which the color saturation of the high saturation areas decreases).

In S502 of Fig. 4B, the gradation processing unit 302 performs processing to correct the gradation based on the gradation characteristics calculated in S501. Specifically, the gradation characteristics of the above (Equation 1) are applied, with the brightness component I of the input image in the ICtCp space being x and the brightness component after gradation correction being y. In this embodiment, processing to correct the gradation is performed by adjusting the brightness of the input image based on the brightness component I of the perceptually uniform color space ICtCp. However, the present invention is not limited to this, and gradation correction processing may be performed based on the perceptually uniform color space by Lab.
According to this embodiment, by carrying out the above-described processing, it is possible to realize image processing capable of controlling the brightness of high-brightness areas of an HDR image.

[Second embodiment]
Next, a second embodiment of the present invention will be described. This embodiment differs from the first embodiment in that a high brightness area in an input image is detected and tone correction is performed only on that area. It is possible to control the sense of brightness for a specific area (high brightness area). Below, differences from the first embodiment will be mainly described, and detailed descriptions of matters similar to those in the first embodiment will be omitted by reusing the symbols and signs already used. The method of omitting such descriptions will be the same in the embodiments described later.

The processing contents of the image processing unit 102 according to this embodiment will be described with reference to FIG. 8. The difference from FIG. 2 is that a high brightness area detection unit 903 is added to the image processing unit 102. The image processing unit 102 includes a display brightness input unit 901, a brightness degree input unit 902, a high brightness area detection unit 903, and a gradation correction unit 904. The display brightness input unit 901, the brightness degree input unit 902, and the gradation correction unit 904 correspond to the display brightness input unit 201, the brightness degree input unit 202, and the gradation correction unit 203 in FIG. 2, respectively.

The image processing unit 102 performs input/output processing of display luminance information and brightness level information for the display control unit 105. The display control unit 105 performs display control of the display unit 104 based on the display luminance information and brightness level information.

The operation of the image processing unit 102 in this embodiment will be described with reference to the flowchart in FIG. 9. In S1001, the high-luminance area detection unit 903 detects high-luminance areas in the input image, which are target areas for processing in S1002 and thereafter. Details of the high-luminance area detection process will be described later.

The processes from S1002 to S1005 are similar to the processes from S401 to S404 in FIG. 4A, but these processes are not performed uniformly on the entire screen, but are performed only on the high-luminance areas detected in S1001. In S1005, tone correction processing is performed on one or more high-luminance areas detected in S1001.

The tone correction process for multiple high-luminance areas will be described in detail with reference to FIG. 10. FIG. 10(A) shows an example of a display when multiple high-luminance areas are detected in S1001 of FIG. 9. A display screen 1101 displays an image area to be corrected 1102, a display luminance input area 1103, a brightness input area 1104, and a tone curve area 1105.

In image area 1102, areas 1106 and 1107 indicate multiple high-luminance areas detected in S1001 of FIG. 9. The gradation correction unit 904 performs gradation correction on areas 1106 and 1107 in accordance with the display luminance acquired in S1003 of FIG. 9 and the gradation characteristics according to the brightness acquired in S1004. Gradation correction processing is performed on the multiple detected high-luminance areas.

Figure 10 (B) shows an example in which different tone correction processes are performed on each of the detected high-luminance areas. An image area 1109, a display luminance input area 1110, a brightness input area 1111, and a tone curve area 1112 are displayed on a display screen 1108. Of the multiple high-luminance areas detected in S1001 of Figure 9, the left side of Figure 10 (B) shows a case in which area 1113 has been selected by user operation, and the right side of Figure 10 (B) shows a case in which area 1114 has been selected by user operation.

When area 1113 is selected by a user operation, gradation correction unit 904 performs gradation correction on area 1113, which is the selected area, according to the display luminance acquired in S1003 and the brightness acquired in S1004 of FIG. 9. Also, when area 1114 is newly selected by a user operation, information on display luminance and brightness different from area 1113 is input. Then, gradation correction unit 904 performs gradation correction on area 1114, which is the selected area, according to the display luminance acquired in S1003 and the brightness acquired in S1004. In this way, detailed gradation correction can be performed for each of the multiple detected high brightness areas according to the corresponding gradation characteristics.

Next, the high-luminance area detection process executed in S1001 of FIG. 9 will be described in detail with reference to FIG. 11. FIG. 11(A) is a diagram for explaining the high-luminance area detection process by dividing the screen into blocks. Specifically, the high-luminance area detection unit 903 divides the input image 1201 into a plurality of areas 1202. The high-luminance area detection unit 903 performs an average calculation to average the pixel values within the divided regions, and compares the calculated average value with a predetermined threshold. A high-luminance area is determined from an area that includes a divided region where the average value is equal to or greater than the predetermined threshold.

Figure 11 (B) is a diagram explaining the process of detecting high-luminance areas using multiple images. For example, it shows an input image 1204 to be corrected, and images 1203 and 1205 that were acquired (or photographed) at different times than the input image 1204. For the input image 1204 at time t2, image 1203 at time t1, which is earlier in time, is used as the first reference image, and image 1205 at time t3, which is later in time, is used as the second reference image.

The high-luminance area detection unit 903 obtains the input image 1204, the first reference image 1203, and the second reference image 1205, and performs a high-luminance area detection process. Specifically, the high-luminance area detection unit 903 calculates the difference between the input image 1204 and the reference image 1203 or 1205 for each predetermined divided area, and compares it with a first threshold. The high-luminance area detection unit 903 also compares the luminance value of the input image 1204 with a second threshold. An area 1206 where the calculated difference between the images is greater than the first threshold and where the luminance value of the input image 1204 is equal to or greater than the second threshold is detected as a high-luminance area. The number of reference images is not limited to two, and may be one or three or more.

There are two types of embodiment: the high-luminance region detection unit 903 automatically detects high-luminance regions from the input image, and the image region based on the user's operation is determined as the high-luminance region. In the latter case, a desired region selected by the user using an operation input member, touch panel, or the like can be determined as the high-luminance region. Alternatively, as a semi-automatic process, the high-luminance region detection unit 903 may display candidates for high-luminance regions detected from the input image, and the user may select one of them using the operation input means to give an instruction.
According to this embodiment, image processing capable of controlling the sense of brightness for high brightness areas in an HDR image can be realized.

[Third Example]
Next, a third embodiment of the present invention will be described. This embodiment differs from the first embodiment in that the brightness level is calculated according to the input display brightness. By reflecting the brightness level calculated from the input display brightness in the input, the brightness of high brightness areas in an HDR image can be easily controlled.

The processing contents of the image processing unit 102 according to this embodiment will be described with reference to FIG. 12. The difference between FIG. 12 and FIG. 2 is that a brightness degree calculation unit 1302 is added to the image processing unit 102. The image processing unit 102 includes a display brightness input unit 1301, a brightness degree calculation unit 1302, a brightness degree input unit 1303, and a gradation correction unit 1304. The display brightness input unit 1301, the brightness degree input unit 1303, and the gradation correction unit 1304 correspond to the display brightness input unit 201, the brightness degree input unit 202, and the gradation correction unit 203 in FIG. 2, respectively.

The operation of the image processing unit 102 will be described with reference to the flowchart in FIG. 13. The processes of S1401 and S1402 are similar to the processes of S401 and S402 in FIG. 4(A), and therefore description thereof will be omitted.

In S1403, the brightness calculation unit 1302 calculates the brightness based on the display luminance acquired in S1402, and performs processing to reflect the brightness in the brightness input unit 1303. The brightness calculation processing will be described later.

In S1404, a process is performed to acquire the brightness level information reflected in the brightness level input unit 1303 in S1403. For example, the brightness level information reflected in the brightness level input unit 1303 in S1403 is acquired as is (unadjusted). Alternatively, the brightness level information reflected in the brightness level input unit 1303 in S1403 is corrected (e.g., fine-tuned) in accordance with a user operation input, and then the brightness level information is acquired. In either case, the acquired brightness level information is output from the brightness level input unit 1303 to the gradation correction unit 1304.

In this embodiment, a process is performed in which the brightness level calculated from the display brightness input by the display brightness input unit 1301 is reflected in the brightness level input unit 1303, but the process is not limited to this. The display brightness may be calculated from the brightness level input by the brightness level input unit 1303, and the calculated display brightness may be reflected in the display brightness input unit 1301. In this case, instead of the brightness level calculation unit 1302, the display brightness calculation unit calculates the display brightness from the brightness level. Information on the display brightness calculated by the display brightness calculation unit, or information obtained by correcting this information, is obtained. The process of S1405 is similar to the process of S404 in FIG. 4(A), so a detailed description will be omitted.

Next, referring to FIG. 14, the calculation process of the brightness degree performed in S1403 in FIG. 13 will be described in detail. FIG. 14 is a graph of a reference table showing the brightness degree corresponding to the display brightness. The brightness degree calculation unit 1302 calculates the brightness degree using the reference table. In FIG. 14, the horizontal axis represents the display brightness, and the vertical axis represents the brightness degree, with the display brightness increasing toward the right on the horizontal axis, and the number of gradations decreasing and the brightness degree increasing toward the top of the vertical axis. The relationship between the display brightness and the brightness degree is shown using the characteristic represented by the graph line 1501 as an example. In this example, the brightness degree is a constant value in the range of display brightness lower than the threshold value Th, and the brightness degree decreases as the display brightness increases in the range of the threshold value Th or higher. If the saturation degree increases in the range of low display brightness, the appearance may deteriorate, so the brightness degree does not exceed the upper limit value in this range.

A function representing a predetermined look-up table is denoted as TBL, and display luminance is denoted as hy. If the brightness level is denoted as sh, sh is expressed by the following (Equation 4).

In addition, in the brightness calculation process of S1403 in FIG. 13, a process of changing the reference table of (Equation 4) may be performed according to a representative value calculated from the histogram of the input image. The process of changing the reference table using a histogram will be described in detail with reference to FIG. 15. The horizontal axis of the histogram shown in FIG. 15(A) represents the pixel value indicating the brightness component I of the perceptually uniform color space ICtCp in the input image, and the vertical axis represents the number of pixels. The brightness calculation unit 1302 first calculates the representative brightness value required to change the reference table using the histogram shown in FIG. 15(A). The pixel value 1601 is the representative pixel value when the value obtained by accumulating the histogram from the high brightness side is a predetermined ratio of the number of pixels to the total number of pixels.

Next, the brightness calculation unit 1302 changes the reference table using the calculated representative brightness value. FIG. 15(B) is a graph showing a reference table that represents the relationship between display brightness and brightness, with the horizontal axis representing display brightness and the vertical axis representing brightness. Graph line 1602 represents the reference table before the change, and graph line 1603 represents the reference table after the change.

The reference table is changed using the following (Equation 5) and (Equation 6). The threshold value defining the high brightness region is denoted as th, the representative pixel value calculated using the histogram as x ^* , and the reference brightness value corresponding to the reference table before the change as xb ^* . If the change rate of the reference table data before and after the reference table is changed using the histogram is denoted as R, R is expressed by the following (Equation 5). In other words, the change rate R corresponds to the ratio of the difference between the reference brightness value xb ^* and the threshold value th to the difference between the representative pixel value x ^* and the threshold value th. If the function representing the reference table before the change is denoted as TBLb, the function TBL representing the reference table after the change is expressed by the following (Equation 6) using the change rate R.

The brightness calculation unit 1302 uses the changed reference table to calculate the brightness sh corresponding to the display luminance hy obtained in S1402 as shown in (Equation 4). For example, the gradation characteristics are calculated based on the brightness calculated using the changed reference table represented by the graph line 1603 in FIG. 15(B). An example of the gradation characteristics is shown in FIG. 15(C).

The horizontal axis of FIG. 15C represents input luminance, and the vertical axis represents output luminance. A graph line 1605 represents the gradation characteristic calculated using the reference table before the change, and a graph line 1606 represents the gradation characteristic calculated using the reference table after the change. The starting point of the input luminance with the display luminance 1604 as the output luminance is changed from the reference luminance value xb ^* corresponding to the reference table before the change to the representative pixel value x ^* calculated using the histogram. The luminance region where the gradation is reduced by being mapped to the display luminance 1604 is changed from the region 1607 to the region 1608. That is, the region 1608 is larger than the region 1607. This change enables the control of the sense of brightness that is optimal for the luminance distribution of the input image.

The configurations described in the first to third embodiments above can provide an image processing device and an image processing method that can control the brightness of high-brightness areas in an HDR image. Note that while the preferred embodiments of the present invention have been described, the present invention is not limited to the above embodiments, and various modifications and variations are possible within the scope of the gist of the present invention.

[Other Examples]
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

104 Display unit 105 Display control unit 201, 901, 1301 Display luminance input unit 202, 902, 1303 Brightness level input unit 203, 904, 1304 Gradation correction unit 903 High luminance area detection unit 1302 Brightness level calculation unit

Claims (18)

An image processing device that performs gradation correction of an image,
a first acquisition means for acquiring information on the display luminance of the image;
A second acquisition means for acquiring brightness information representing a degree of reduction in gradation of a brightness area having a brightness value equal to or greater than a threshold value in the image;
a correction means for performing gradation correction of the image using the acquired information on the display luminance and the information on the brightness level,
an image processing device characterized in that, when the display luminance value acquired by the first acquisition means is a first display luminance value, an initial value of the display luminance is a second display luminance value, and the first display luminance value is greater than the second display luminance value, the correction means performs gradation correction to bring the brightness of the luminance area where the input luminance is equal to or greater than its threshold value closer to the first display luminance value by increasing the output luminance of the gradation characteristics before gradation correction in accordance with the information on the degree of brightness for the gradation characteristics before gradation correction that represent the relationship between the input luminance and the output luminance. 2. The image processing device according to claim 1, further comprising a control means for controlling display of the image, a first input area used to acquire the display luminance information, and a second input area used to acquire the brightness information. 3. The image processing apparatus according to claim 2, further comprising a display means for displaying the first and second input areas on the same screen. 4. The image processing device according to claim 1, wherein the correction means includes a calculation means for calculating a gradation characteristic using the display luminance information and the brightness information, and performs the gradation correction in accordance with the gradation characteristic. 5. The image processing device according to claim 4, wherein the calculation means calculates a histogram representing the number of pixels corresponding to the luminance values of the image, and calculates the input luminance threshold value in the gradation characteristics from the luminance value that first becomes a minimum value when the histogram is viewed from the high luminance side. 4. The image processing device according to claim 1, wherein the correction means comprises: a calculation means for calculating the gradation characteristics using the display luminance information, the brightness information, and the chromaticity degree of the image; and a gradation processing means for performing a process of correcting the gradation based on the calculated gradation characteristics. 4. The image processing device according to claim 1, wherein the correction means comprises: a calculation means for calculating a gradation characteristic in which a brightness component of an image before gradation correction is an input luminance in a perceptually uniform color space of the image, and a brightness component of an image after gradation correction is an output luminance; and a gradation processing means for performing a process of correcting the gradation based on the calculated gradation characteristic. A detection unit is further provided for detecting a high-luminance region in the image, the high-luminance region having a luminance value equal to or greater than a threshold value.
The image processing device according to claim 1, characterized in that, when the detection means detects a plurality of high-brightness areas, the correction means performs gradation correction on the plurality of high-brightness areas using the display luminance information and the brightness information acquired for the plurality of high-brightness areas. an operation input means for receiving an operation input for selecting an image area;
The image processing apparatus according to claim 8 , wherein the detection means detects the high luminance area by using information received by the operation input means. 9. The image processing apparatus according to claim 8, wherein said detection means obtains a reference image for said image, and detects said high luminance area by comparing said image with said reference image. The image processing apparatus according to claim 10 , wherein the reference image is an image whose acquisition or photographing time is different from that of the image. When the detection means detects the first and second high luminance areas,
The correction means is
performing gradation correction based on a first gradation characteristic for the first high luminance region using the display luminance information and the brightness information acquired for the first high luminance region;
12. The image processing device according to claim 8, further comprising: performing gradation correction based on a second gradation characteristic for the second high-luminance area using the display luminance information and the brightness information acquired for the second high-luminance area. A calculation means for calculating the brightness level information from the acquired display brightness information,
The image processing device according to claim 1 , wherein the second acquisition means acquires the information on the brightness level calculated by the calculation means. A calculation means for calculating the display luminance information from the acquired brightness level information,
The image processing device according to claim 1 , wherein the first acquisition means acquires the information on the display luminance calculated by the calculation means. An imaging device comprising the image processing device according to claim 1 . 16. The imaging device according to claim 15, further comprising a display unit that displays an image that has been tone-corrected by the correction unit, or the image and information indicating characteristics of the control of the display luminance and the brightness level. An image processing method executed by an image processing device that performs gradation correction of an image, comprising:
an acquisition step of acquiring information on the display luminance of the image and information on the brightness level representing a degree of reduction in the gradation of a luminance area in the image having a luminance value equal to or greater than a threshold;
and a correction step of performing gradation correction of the image using the acquired information on the display luminance and the information on the brightness level,
the correction step is performed to make the brightness of the luminance region where the input luminance is equal to or greater than the threshold value closer to the first display luminance value, by making the output luminance of the gradation characteristics before the gradation correction, which represents the relationship between the input luminance and the output luminance, greater than the output luminance of the gradation characteristics before the gradation correction, in the luminance region where the input luminance is equal to or greater than the threshold value, in accordance with the information on the degree of brightness for the gradation characteristics before the gradation correction, which represents the relationship between the input luminance and the output luminance, closer to the first display luminance value. A program for causing a computer of an image processing apparatus to execute the steps according to claim 17.

Images