JP6675461B2 - Image processing apparatus, control method therefor, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, a control method therefor, a program, and a recording medium, and more particularly to a technique for correcting the brightness of an input image.

  Conventionally, there is known a technique called relighting, which irradiates light from a virtual light source (virtual light source) to a subject in an image photographed under a certain environmental light to change a shadow of the subject. I have. By using the relighting technique, it is possible to correct the captured image so that the shadow of the subject caused by the ambient light is in a preferable state. For example, Patent Literature 1 discloses a technique for performing relighting on a captured image using a virtual light source.

JP 2010-135996 A

  Since the shadow of the subject is determined by the three-dimensional shape of the subject and the characteristics of the light to be irradiated, when correcting the shadow by the relighting process, it is necessary to appropriately determine the characteristics of the virtual light source used for relighting. .

Patent Literature 1 discloses performing rewriting by defining, as lighting processing parameters, a range of a subject image to be subjected to lighting processing, a direction of a virtual light source, and an intensity of the lighting processing. However, Patent Document 1, no consideration is given to control the diffusion properties of the light emitted from the virtual light source. For this reason, the state of the shadow of the subject may not be properly corrected in some cases.

The present invention has been made in view of the above-described problems of the related art, and has an image processing apparatus and a control method thereof capable of appropriately correcting the state of shadow of a subject when performing a relighting process using a virtual light source, and an object of the present invention is to provide a program.

In order to solve this problem, for example, an image processing apparatus of the present invention has the following configuration. That is, acquiring means for acquiring a captured image, determining means for determining the degree of shading of a partial area of the captured image, setting means for setting parameters of a virtual light source which is a virtual light source for the captured image, and setting means Correction means for adding the effect of irradiating the virtual light from the virtual light source to the captured image based on the parameters set by the control unit, and the correction means is configured to irradiate the virtual light by combining a plurality of virtual light sources. A correction for adding an effect is possible, and the setting unit sets a combination of a plurality of virtual light source parameters based on the degree of shading detected by the determining unit, thereby setting the virtual light emitted to the captured image. It is characterized in that the diffusion characteristics are changed.

  ADVANTAGE OF THE INVENTION According to this invention, when performing relighting processing using a virtual light source, it becomes possible to correct the state of shading of a subject appropriately.

FIG. 1 is a block diagram illustrating a functional configuration example of a digital camera 100 as an example of an image processing apparatus according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating a functional configuration example of an image processing unit according to the embodiment. 5 is a flowchart illustrating a series of operations of a subject shadow state detection unit according to the first embodiment. FIG. 3 is a diagram illustrating an example of a subject according to the first embodiment. FIG. 3 is a diagram illustrating an example of a virtual light source according to the first embodiment. 5 is a flowchart illustrating a series of operations of the virtual light source characteristic control unit according to the first embodiment. FIG. 9 is a diagram illustrating an example of a captured image according to the second embodiment. 9 is a flowchart illustrating a series of operations of the virtual light source characteristic control unit according to the second embodiment.

(Embodiment 1)
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the embodiment described below, an example in which the image processing apparatus according to the present invention is applied to a digital camera will be described. Note that a digital camera refers to an electronic device having a shooting function using a photoelectric conversion element, and includes any electronic device having or usable with a camera, such as a mobile phone, a game machine, and a personal computer. Further, a photographing function is not essential to the present invention, and the image processing apparatus according to the present invention is applicable to any electronic device capable of performing image processing.

(Configuration of Digital Camera 100)
FIG. 1 is a block diagram illustrating a functional configuration example of a digital camera 100 as an example of an image processing apparatus according to the present embodiment. Note that one or more of the functional blocks illustrated in FIG. 1 may be realized by hardware such as an ASIC or a programmable logic array (PLA), or may be realized by a programmable processor such as a CPU or an MPU executing software. You may. Further, the present invention may be realized by a combination of software and hardware. Therefore, in the following description, even when different functional blocks are described as the operation subject, the same hardware can be realized as the subject.

  The lens group 101 is a zoom lens including a focus lens. The shutter 102 has an aperture function, and exposes an imaging element included in the imaging unit 103 under the control of the system control unit 50. The imaging unit 103 includes an imaging device such as a CCD / CMOS image sensor that converts an optical image obtained through the lens group 101 into an electric signal by photoelectric conversion. The A / D conversion unit 104 converts an analog signal read from the imaging unit 103 into a digital signal and outputs image data to the image processing unit 105.

  The image processing unit 105 performs various image processing such as white balance adjustment and γ correction on the image data output from the A / D conversion unit 104 or the image data output from the memory control unit 107. Further, the image processing unit 105 performs a predetermined evaluation value calculation process (not shown) using the face detection result of the face detection unit 113 and the captured image data, and based on the obtained evaluation value, the system control unit 50. Performs exposure control and distance measurement control. Thereby, TTL (through-the-lens) type AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, and the like are performed. In addition, the image processing unit 105 also performs a relighting process of changing the shadow of the subject so as to be in a state in which light from a virtual light source (virtual light source) is emitted by a method described later.

  The image memory 106 temporarily stores image data when the image processing unit 105 performs various types of image processing, displays image data read from the recording medium 112 via the I / F 111, and displays the image data on the display unit 109. For storing image data. The memory control unit 107 controls reading and writing of the image memory 106. The D / A converter 108 converts an input digital signal into an analog signal. For example, data for image display stored in the image memory 106 is converted into an analog signal and output to the display unit 109.

  The display unit 109 has a display device such as an LCD, and displays a captured image, an image read from the recording medium 112, a live view image, and the like, and also displays a user interface for performing an operation. The codec unit 110 performs compression encoding / decoding of image data. The codec unit 110 encodes or decodes image data recorded in the image memory 106 in a format conforming to a standard such as MPEG.

  The interface (I / F) 111 mechanically and electrically connects the detachable recording medium 112 such as a semiconductor memory card or a card-type hard disk to the digital camera 100. The face detection unit 113 analyzes the image data and detects an area where the face in the image is reflected.

  The system control unit 50 includes a CPU or an MPU, executes a program stored in the nonvolatile memory 121 by expanding the program in a work area of the system memory 122, and controls each function of the entire digital camera 100.

  The operation unit 120 includes a touch panel, buttons, and switches that display the above-described interface, and notifies the system control unit 50 of the operation when the operation by the user is detected.

  The nonvolatile memory 121 includes, as an auxiliary storage device, a nonvolatile semiconductor memory such as an EEPROM that stores programs, parameters, and the like. The system memory 122, as a main storage device, stores programs and the like read from the nonvolatile memory 121 and stores constants and variables for operation of the system control unit 50.

(Configuration of Image Processing Unit 105)
The configuration of the image processing unit 105 and the processing of each unit of the image processing unit 105 will be described with reference to FIG.

  The image signal generation unit 201 inputs the image data output from the A / D conversion unit 104. The image signal generation unit 201 performs a synchronization process on the input RGB image data configured in the Bayer array, and generates image signals R, G, and B having a plurality of colors per pixel. The image signal generation unit 201 outputs the generated image signal to the WB amplification unit 202.

  The WB amplifying unit 202 adjusts the white balance by applying a predetermined white balance gain to the image signals R, G, B based on the white balance gain value calculated by the system control unit 50. The WB amplification unit 202 outputs the adjusted image signals R, G, and B to the gamma processing unit 203 and the shadow state detection unit 207.

  Based on the input image signals R, G, and B, the shadow state detection unit 207 detects a state of a shadow generated in an object area due to environmental light illuminating the object at the time of shooting. Here, the ambient light includes illumination such as sunlight that is separate from the digital camera 100, a shooting auxiliary light (not shown) provided in the digital camera 100, and the like. The shadow state detection unit 207 outputs the detection result to the characteristic determination unit 208 as subject shadow information. The process of detecting the shadow state of the subject by the shadow state detection unit 207 will be described later in detail.

  The characteristic determining unit 208 determines characteristics of a virtual light source for rewriting a subject based on the input subject shadow information. Then, the characteristic of the determined virtual light source is output to the luminance correction unit 205 as diffusion characteristic information. The process of determining the diffusion characteristics of the virtual light source will be described later in detail.

  The gamma processing unit 203 performs gamma correction on the RGB image signals for recording and display. The gamma processing unit 203 outputs the image signal after the gamma correction to the degamma processing unit 204. However, in the case of shooting without performing relighting processing, the image signal after gamma correction is output to the image memory 106, and the image signal is stored.

  The degamma processing unit 204 performs degamma processing on the input image signals R, G, and B, generates image signals R, G, and B before gamma correction, and outputs the image signals to the luminance correction unit 205.

  The luminance correction unit 205 sets the characteristics of the virtual light source based on the diffusion characteristic information, and performs a relighting process of adding a virtual light source signal corresponding to the virtual light source to the input image signal. The luminance correction unit 205 outputs the image signal corrected by the relighting process to the re-gamma processing unit 206.

  The re-gamma processing unit 206 performs the gamma correction again on the input image signal after rewriting. The re-gamma processing unit 206 outputs the gamma-corrected image signal to the image memory 106.

(Series of operations related to shadow state detection processing)
Next, a series of operations of a process in which the shadow state detection unit 207 detects a shadow state of a subject caused by ambient light will be described with reference to FIG. This processing is started, for example, when image data obtained by a photographing operation of the digital camera 100 is input to the shadow state detection unit 207.

  In the present process, information representing the characteristics of the boundary of the region where the shadow occurs is used as the information indicating the state of the shadow occurring in the subject. Specifically, information indicating the degree of whether the boundary of the shadow is sharp or blurred is used.

  In step S <b> 301, the shading state detection unit 207 extracts only an image signal corresponding to a subject area to be subjected to the relighting process from the image signals input from the WB amplification unit 202. For example, a face area of the subject detected by the face detection unit 113 is set as a target of the rewriting process, and an image signal corresponding to the face area is extracted. FIG. 4 shows an example of the extracted subject area, in which a shaded area 402 is shaded with respect to the subject. Further, the face area of the subject detected by the face detection unit 113 is detected, for example, as an area indicated by a face area 401.

  In S302, the shadow state detection unit 207 calculates the brightness of the subject based on the image signal included in the extracted face area 401. Specifically, the shadow state detection unit 207 divides the face area into a plurality of blocks (for example, 8 × 8 = 64) as shown in FIG. 4 and calculates an average value of luminance for each of the divided blocks. I do.

  In S303, the shadow state detection unit 207 calculates subject shadow information based on the calculated average value of the luminance. First, the shadow state detection unit 207 detects a shadow area by extracting a block (low luminance area) in which the calculated average luminance value of each block is equal to or less than a predetermined threshold. Next, a difference value of luminance is calculated between a block located at the boundary of the detected shadow area and a block adjacent to the block and not being a shadow area. Here, the larger the calculated difference value of the brightness, the sharper the boundary of the shadow area of the subject, and conversely, the smaller the difference value of the brightness, the blurred the boundary of the shadow area of the subject. I can judge. The shadow state detection unit 207 outputs the calculated luminance difference value to the characteristic determination unit 208 as subject shadow information.

  In S304, the shadow state detection unit 207 estimates the position of the ambient light. Since the estimation of the position of the ambient light can be performed using various known methods, a detailed description thereof will be omitted. For example, the brightest area in the subject is detected, and the normal vector of the area is detected. May be calculated, and it may be determined that environmental light is located at a predetermined position in the calculated direction. In the case of a face area of a person, the position of the ambient light may be estimated by detecting a predetermined part such as a nose and a position behind the predetermined part. The shadow state detection unit 207 outputs the estimated position information of the environment light to the characteristic determination unit 208.

(Outline of control method for diffusion characteristics)
In the present embodiment, an example will be described in which the number, shape, size, and relative positional relationship between virtual light sources are controlled to control the diffusion characteristics of the virtual light sources. In particular, a method of controlling the diffusion characteristics of each of the representative virtual light sources shown in FIGS.

  FIG. 5A shows a case where only one point light source is arranged as a virtual light source. The object surface 501 and the virtual light source 502 are located at a distance D, and the light from the virtual light source 502 spreads at an angle of β. In this case, as the distance D or the angle β increases, the range in which the light from the virtual light source 502 irradiates the subject surface 501 increases. In addition, light is gradually attenuated at the boundary of the irradiation range. Therefore, the characteristic determining unit 208 can perform rewriting with light having high diffusivity by setting the distance D or the angle β to a large value. Conversely, by setting the distance D or the angle β to a small value, it becomes possible to perform rewriting with light having low diffusivity.

  FIG. 5B shows a case where a plurality of point light sources are arranged as virtual light sources. The distance L represents a distance between the plurality of virtual light sources 502. When a plurality of virtual light sources 502 are arranged at different positions from each other, a range in which light from the virtual light source 502 is applied to the subject surface 501 is widened. In addition, light is gradually attenuated at the boundary of the irradiation range. Furthermore, the light of the virtual light source 502 disposed at another position is also radiated to a region that becomes negative when only one virtual light source 502 is disposed.

  For example, the position of the point 503 on the subject surface 501 is not irradiated with light from the virtual light source 502 (V2) arranged at the center, but the light from the virtual light source 502 (V1) that is separated by a distance L to the left is Irradiated. That is, it is possible to realize relighting using light having high diffusivity such as bounce light in which illumination light is reflected by a reflection plate or the like. Therefore, the characteristic determining unit 208 sets the number N of the virtual light sources and the distance L between the virtual light sources to a large value to increase the diffusivity, and conversely, sets the number N or the distance L between the virtual light sources to a small value. By doing so, the diffusion characteristics of the virtual light source can be controlled so as to reduce the diffusion.

  FIG. 5C shows a case where a large-sized spot light is arranged as a virtual light source. S represents the size of the virtual light source 502. For example, when the shape of the virtual light source 502 is a sphere, it represents the radius of the sphere. It is assumed that spot light is emitted from each point on the surface of the virtual light source 502. As in the case of FIG. 5B, when the size S of the virtual light source 502 is increased, the range in which the light from the virtual light source 502 is irradiated is widened, and the area becomes shaded when the virtual light source 502 is a point light source. Light is also emitted. Therefore, the characteristic determining unit 208 can control the diffusion characteristic of the virtual light source such that the diffusion is high by setting the size S to a large value, and conversely, the diffusion is low by setting the size S to a small value. .

  FIG. 5D shows a case where a surface light source having a curvature R or a linear light source is arranged as a virtual light source. When the curvature R approaches 0, the light from the virtual light source 502 becomes parallel light. When the curvature R is a value other than 0, the light of the virtual light source 502 is also applied to an area that is shaded when the virtual light source 502 that emits parallel light is arranged. Therefore, the characteristic determining unit 208 can control the diffusion characteristic of the virtual light source such that the diffusion property is high by setting the curvature R to a value larger than 0, and low by setting the curvature R to a value close to 0. .

  FIG. 5E shows a case where a plurality of surface light sources or linear light sources are arranged at different angles as virtual light sources. The angle γ represents a relative angle between the virtual light sources 502. When the angle γ takes a large value, the range in which light from the virtual light source 502 is irradiated is widened. In addition, light is gradually attenuated at the boundary of the irradiation range. In other words, if the angle γ is set to a large value, light from the virtual light sources arranged at different angles is radiated even to a shadow region when the angle γ is 0. Therefore, the characteristic determining unit 208 sets the diffusion characteristic of the virtual light source such that the diffusion characteristic is high by setting the angle γ between the virtual light sources to a large value, and conversely, the diffusion characteristic is low by setting the angle γ to a small value. Can control.

  In the following description, the characteristic determining unit 208 controls the diffusion characteristics of the virtual light source by changing any of the parameters of the virtual light source described in FIGS. 5A to 5E or a combination thereof. Shall be.

(A series of operations related to the process of determining the diffusion characteristics of the virtual light source)
Further, with reference to FIG. 6, a description will be given of a series of operations of the process in which the characteristic determining unit 208 determines the diffusion characteristic of the virtual light source based on the subject shadow information.

  In step S <b> 601, the characteristic determining unit 208 acquires information representing a standard setting of the virtual light source (that is, a standard value of the setting of the virtual light source) from the system control unit 50. The information indicating the setting of the virtual light source refers to information in which parameters of the number, shape, and positional relationship of the virtual light source described in FIG. 5 are set. As the information indicating the standard setting of the virtual light source, a value determined in advance by an experiment or the like is stored in the non-volatile memory 121, and is provided to the characteristic determining unit 208 by the system control unit 50 reading this value.

  In step S <b> 602, the characteristic determination unit 208 acquires the subject shadow information output from the shadow state detection unit 207. As described above, the subject shadow information is information indicating the state of the shadow of the subject, and the difference value of the luminance at the boundary of the shadow region is set.

  In step S603, the characteristic determination unit 208 acquires target shadow information output from the system control unit 50. The target shadow information is a target shadow state at the time of the relighting process, and represents, for example, a target brightness difference value at the boundary of the shadow area (hereinafter, simply referred to as a target brightness difference). As for the target luminance difference, for example, a value determined in advance by an experiment or the like is stored in the nonvolatile memory 121, and the characteristic determining unit 208 acquires the target luminance difference via the system control unit 50.

  In step S604, the characteristic determining unit 208 determines whether the difference value of the luminance indicated by the subject shadow information acquired in step S602 is higher than the target luminance difference acquired in step S603. The characteristic determining unit 208 compares the luminance difference value with the target luminance difference. If the luminance difference value is higher than the target luminance difference, the process proceeds to S605. In this case, the shadow area on the object is sharper than the target shadow state, and the shadow of the object is dominant. On the other hand, when the luminance difference value is equal to or smaller than the target luminance difference, the characteristic determining unit 208 proceeds to S606. In this case, the boundary state of the shadow area of the subject caused by the ambient light is blurred, and the shadow of the subject is inferior.

  In S605, the characteristic determining unit 208 sets the diffusion characteristic of the virtual light source higher than the standard value acquired in S601. That is, the diffusion characteristics of the virtual light source are controlled so as to perform rewriting with a virtual light source having a high diffusion characteristic. More specifically, the characteristic determining unit 208 changes the standard value read in S601 and controls the diffusion characteristics of the virtual light source to be higher. A specific control method of the set value is as described above with reference to FIG.

  In step S606, the characteristic determination unit 208 changes the standard value read in step S601 to control the diffusion characteristics of the virtual light source to be lower.

  In step S607, the characteristic determining unit 208 outputs the changed setting value of the diffusion characteristic of the virtual light source to the luminance correction unit 205 as diffusion characteristic information. The position where the virtual light source is arranged is set by the shadow state detection unit 207 to be the same as the position and direction of the ambient light estimated in S304 of FIG. However, when the position of the ambient light cannot be accurately estimated, the position where the virtual light source is arranged is set so that the virtual light source is set to a position outside the captured image and relighting is performed.

  When the above-described processing ends, the characteristic determining unit 208 ends a series of operations according to the present processing.

(Rewriting process)
Further, rewriting processing using a virtual light source, which is performed by the luminance correction unit 205, will be described. In the present embodiment, the output values (R, G, B) of the target pixel illuminated by the virtual light source are calculated. Note that the luminance correction unit 205 calculates (R, G, B) by the following equation.

R = [Rt + sum_n [Rv × A_n × cos (α_n) / D_n ＾ 2]] / M
G = [Gt + sum_n [Gv × A_n × cos (α_n) / D_n ＾ 2]] / M
B = [Bt + sum_n [Bv × A_n × cos (α_n) / D_n ＾ 2]] / M

Here, (Rt, Gt, Bt) is a pixel value to be processed, n is a suffix for distinguishing a plurality of virtual light sources, and n = 1... N when N virtual light sources are set. Becomes sum_n represents a process for summing n. A_n is a predetermined constant representing the intensity of the n-th virtual light source, D_n is the distance between the n-th virtual light source and the subject, and (Rv, Gv, Bv) represents the light source reflection color. M is a constant for normalizing the output RGB values after the relighting process. The angle α_n is an angle formed by a straight line connecting the position where the n-th virtual light source is arranged and the target pixel, and a normal vector of the subject at the target pixel.

  When calculating the angle α, the normal vector of the subject may be calculated for each pixel of the rewriting target pixel, and the angle α may be calculated. Further, as shown in FIG. 4, the subject area may be divided into a plurality of blocks, and the angle α may be calculated based on the normal vector calculated for each block. Further, when the three-dimensional shape of the subject is known in advance, such as a person's face, the direction of the normal vector is estimated based on the position of the target pixel in the face area, and the angle α is determined. You may make it calculate. The light source reflection color is obtained by estimating the reflection color when the virtual light source reflects on the surface of the subject, based on the preset virtual light source color and the subject color.

  As described above, in the present embodiment, the characteristic of the virtual light source is set based on the result of detecting the luminance difference at the boundary of the shadow area generated in the subject and comparing the luminance difference with the target luminance difference. I made it. By doing so, it is possible to perform the relighting process so that the state of the shadow of the subject caused by the ambient light approaches the target state. In other words, by determining the diffusion characteristic of the virtual light source according to the state of the shadow of the subject due to the ambient light, the shadow of the subject can be made appropriate when performing the relighting process using the virtual light source.

(Modification according to the present embodiment)
The configuration for determining the diffusion characteristic of the virtual light source in accordance with the state of the shadow of the subject due to the ambient light and performing the relighting processing described in the present embodiment is not limited to the above-described embodiment, and is, for example, a modification described below Can also be realized.

  In the present embodiment, the method of determining the diffusion characteristic of the virtual light source based on the luminance difference at the boundary of the shadow area is used. However, any information that indicates the state of the shadow of the subject may be used. good. For example, the characteristic of the virtual light source may be determined based on a result of detecting a contrast value of a subject caused by ambient light and comparing the detected value with a target contrast value. Specifically, when the contrast value of the subject is higher (lower) than a predetermined target, the diffusion characteristic of the virtual light source may be set higher (lower). By doing so, it is possible to control the relighting process so that the contrast of the subject approaches a desirable state.

  Alternatively, the area of the shadow area in the subject area may be calculated, and the result may be used as information indicating the state of the shadow of the subject. Specifically, the ratio of the shadow area (for example, low-luminance area) in the object area to the object area is calculated, and when the value is higher (lower) than a predetermined target, the diffusion characteristic of the virtual light source is increased. (Low) should be set. That is, if there is a large area that is a shadow area in the subject area, rewriting processing is performed to reduce the area of the shadow area, and if the area that is a shadow area in the subject area is small, A relighting process can be performed to increase the area. This makes it possible to obtain an image in which the state of the shadow of the subject is appropriately corrected.

  Furthermore, a method may be used in which a luminance histogram is calculated for the subject region and the characteristics of the virtual light source are determined according to the bias of the frequency distribution (ie, the brightness distribution). Specifically, a frequency corresponding to a luminance value lower than a predetermined threshold is calculated, and if the value is higher (lower) than a predetermined target, the diffusion characteristic of the virtual light source is set higher (lower).

  Further, in the present embodiment, a description has been given assuming that a target shadow state is determined in advance, but the present invention is not limited to a method of determining a target shadow state. For example, the value representing the target shade state may be determined according to the shooting parameters at the time of shooting, such as the shooting mode and camera parameters of the digital camera. For example, if the characteristic of the gamma curve used at the time of shooting is a characteristic that strongly expresses the contrast, the target luminance difference is set high, and if the characteristic of the gamma curve is a characteristic that expresses the contrast weakly, the target is set. Set the brightness difference low.

  Further, the target shade state may be determined according to the shade state of a region other than the subject region to be subjected to the relighting process, for example, a background region located in the background of the subject. Specifically, if the boundary of the shadow on the background subject is clear, the target luminance difference is set high, and if the boundary of the shadow on the background subject is blurry, the target luminance difference is set. Be set low. By doing so, the state of the shadow of the subject to be rewritten and the surrounding subject can be brought close to each other, and the sense of discomfort that occurs when performing the rewriting process can be reduced.

  Further, in the present embodiment, the method of determining the diffusion characteristic of the virtual light source by comparison with the target shadow state has been described. However, if the method of determining the diffusion characteristic of the virtual light source based on the subject shadow information, Other methods may be used. For example, the diffusion characteristic of the virtual light source may be determined based on whether or not the detected luminance difference of the subject is within a predetermined threshold range. Specifically, when the luminance difference at the boundary of the shadow region of the subject is equal to or greater than (or less than) a predetermined threshold, the diffusion characteristic of the virtual light source is set higher (lower) than the standard. In this way, even when a specific target value is not determined, the diffusion characteristic of the virtual light source can be determined based on the subject shadow information, and the relighting process can be controlled. Further, when the difference between the information indicating the shadow state of the subject and the information indicating the target shadow state is equal to or smaller than a predetermined threshold value, control may be performed so that the relighting process is not performed.

  Further, brightness correction processing may be performed on a subject that is not a target of the relighting processing. Specifically, the contrast value of the subject to be subjected to the rewriting process is calculated after the rewriting process, and the brightness is corrected so that the contrast of the subject not to be subjected to the rewriting process approaches the value. By doing so, it is possible to reduce the sense of discomfort that occurs between the subject to be rewritten and the non-target subject.

  Further, in the present embodiment, the position of the virtual light source is set to be the same as the position of the estimated environment light. However, the present invention is not limited to the method of determining the position of the virtual light source. For example, the virtual light source may be arranged in a predetermined direction such as a 45-degree diagonal direction with respect to the front of the subject.

  Further, in the present embodiment, an example has been described in which the subject is a single person, but the present invention is not limited to the subject to be relighted. For example, the method of the present invention may be used for relighting processing when a plurality of subjects are photographed. At this time, even if the illumination is performed by the same environmental light, the state of the shadow of the subject may be different depending on conditions such as the direction of the face. For this reason, the process of detecting the shadow state described in the present embodiment and the process of determining the diffusion characteristic of the virtual light source based on the result may be individually performed for each subject. At this time, a common value may be used as the information of the shadow state as a target of the relighting process. In this way, even when a plurality of subjects are photographed, it is possible to perform the relighting process so that the shading state of each subject is in a preferable state.

  Furthermore, when a plurality of subjects are photographed, the characteristics of the virtual light source are determined based on the state of the shadow of the most main subject among the subjects, and rewriting is performed for other subjects using the virtual light source having the same characteristics. Control may be performed. By doing so, it is possible to prevent unnaturalness in which subjects in the same environment are illuminated by light sources having different characteristics.

  Further, in the present embodiment, a typical example of the number and shape of the virtual light source is illustrated to explain the control processing of the diffusion characteristics of the virtual light source. It is not limited to. For example, although a case has been described in which a plurality of point light sources are arranged on a straight line, a plurality of point light sources may be arranged on a curved curve to perform rewriting. In this case, by controlling the curvature of the curve and the distance between the virtual light sources, the diffusion characteristics of the virtual light sources can be set to the intended state. Also, in the present embodiment, a description has been given of using a curved virtual light source curved with a uniform curvature, but other shapes may be used. For example, the shape of the virtual light source may be a broken line bent at a predetermined angle.

  Furthermore, in the present embodiment, the description has been made assuming that the diffusion characteristics of the virtual light source are controlled by controlling parameters such as the shape and the number of the virtual light sources. It is not limited. For example, after calculating a virtual light source signal corresponding to a virtual light source set to a predetermined shape and number, by applying a smoothing filter to the signal, the diffusion characteristics of the virtual light source are increased in a pseudo manner. It may be controlled. That is, any method may be used as long as the method controls the diffusion characteristics of the virtual light source based on the detection result of the shadow state of the subject.

  Further, the diffusion characteristics of the virtual light source may be controlled according to the diffusion characteristics of the ambient light. For example, when estimating a range illuminated by ambient light and determining a range in which light illuminated from the virtual light source spreads, the diffusion characteristics of the virtual light source are controlled so that the ambient light and the virtual light source illuminate the same range. You may do it. By doing so, a more natural relighting result can be obtained.

(Embodiment 2)
Next, a second embodiment according to the present invention will be described. In the first embodiment, the diffusion characteristic of the virtual light source is controlled according to the state of the shadow generated on a specific subject. On the other hand, in the second embodiment, the diffusion characteristic of the virtual light source is controlled according to the state of the shadow in the entire captured image. For this reason, the present embodiment is different from the first embodiment in the shading state detection process and the control process of the diffusion characteristic of the virtual light source, but the configuration of the digital camera 100 and the configuration of the image processing unit are shown in FIGS. The same configuration as that described above can be used. For this reason, the same components are denoted by the same reference numerals, and redundant description will be omitted, and differences will be mainly described.

  The shadow state detection processing by the shadow state detection unit 207 will be described. FIG. 7 illustrates an example of a captured image in the present embodiment, and the entire captured image is used as a target area for the relighting process. The captured image is divided into a plurality of blocks and processed as described later. FIG. 7 shows a photographed image having many shaded areas, that is, many shaded areas, and shows a state in which ambient light is low in diffusivity and is only locally illuminated. For this reason, the entire image has a dark impression.

(Series of operations related to shadow state detection processing)
Next, a series of operations related to a shadow state detection process performed by the shadow state detection unit 207 on the captured image illustrated in FIG. 7 will be described with reference to FIG.

  In step S <b> 301, the shadow state detection unit 207 extracts an image signal of a region to be subjected to relighting processing from image signals input from the WB amplification unit 202. In the first embodiment, a specific subject area is extracted and relighting processing is performed. In the present embodiment, the shadow state detection unit 207 extracts a wider area (that is, the entire captured image) and performs relighting processing.

  In S302, the shadow state detection unit 207 calculates the brightness for each region of a predetermined size based on the extracted image signal. Specifically, as shown in FIG. 7, an area to be rewritten is divided into a plurality of blocks (for example, 8 × 8 = 64), and an average value of luminance is calculated for each of the divided blocks. .

  In S303, the shadow state detection unit 207 calculates subject shadow information based on the calculated average value of the luminance. In the present embodiment, the ratio of the area of the region illuminated and brightened by the ambient light to the entire image (simply referred to as the ratio of the ambient light) is calculated, and the ratio is used as the subject shadow information. In the example of FIG. 7, when the diffusion of the ambient light is low and the light is only locally illuminated, if the relighting is performed so that the diffusion characteristic of the virtual light source is increased, it is possible to correct the image to be a preferable image. . Conversely, when the ambient light has high diffusivity and the entire image area is uniformly illuminated, the contrast of the entire image is low, and the image has a flat impression. For this reason, if rewriting is performed so that the diffusion characteristic of the virtual light source is reduced, it is possible to correct the image so as to obtain a sharp and preferable image. The shadow state detection unit 207 outputs the calculated ratio of the environment light to the characteristic determination unit 208.

  The shadow state detection unit 207 estimates the position of the ambient light in subsequent S304 in the same manner as in the first embodiment, and outputs the result to the characteristic determination unit 208.

(A series of operations related to the process of determining the diffusion characteristics of the virtual light source)
Next, a process in which the characteristic determining unit 208 determines the diffusion characteristic of the virtual light source based on the subject shadow information will be described with reference to FIG.

  As in the case shown in FIG. 5, the diffusion characteristics of the virtual light source are controlled by controlling characteristics such as the shape, size, number, and positional relationship of the virtual light source. Here, an example will be described in which the value of the angle β representing the range in which the light from the virtual light source spreads out of the virtual light sources shown in FIG. 5A is controlled, but any of FIGS. 5A to 5E will be described. May be used.

  In step S <b> 801, the characteristic determination unit 208 acquires a standard value of the setting of the virtual light source output from the system control unit 50. In the present embodiment, the characteristic determining unit 208 acquires a standard setting value of the value of the angle β, for example.

  In step S802, the characteristic determination unit 208 acquires the subject shadow information output from the shadow state detection unit 207. In the present embodiment, the value of the ratio of the ambient light set as the subject shadow information is acquired.

  In step S <b> 803, the characteristic determination unit 208 obtains the state of the shadow of the target subject at the time of the relighting process, that is, the ratio of the target environmental light (also simply referred to as a target value) output from the system control unit 50. More specifically, the target value is obtained by setting, as a predetermined target value, a ratio of a region that is brightened by irradiating environmental light to a region of the captured image.

  In S804, the characteristic determining unit 208 determines whether the ratio of the ambient light is smaller than the target value. The characteristic determining unit 208 compares the ratio of the ambient light with the target value. If the ratio of the ambient light is smaller than the target value, the process proceeds to S805; otherwise, the process proceeds to S806. Here, when the ratio of the ambient light is smaller than the target value, the area to which the ambient light is illuminated is narrow, so that the shadow of the subject is dominant and the entire image has a dark impression. Therefore, rewriting is performed with a virtual light source having high diffusivity, and the process proceeds to S805 in order to reduce dark areas in the image.

  On the other hand, when the ratio of the ambient light is equal to or greater than the target value, the shadow of the subject is inferior because the ambient light hits a wide area in the captured image. For this reason, rewriting is performed with a virtual light source having a low diffusion characteristic, and the process proceeds to S806 in order to obtain a preferable image with sharp image brightness.

  In step S805, the characteristic determination unit 208 changes the standard value read in step S801, and controls the virtual light source to have a higher diffusivity. That is, the angle β is set as a value larger than the standard value by a predetermined value.

  In step S806, the characteristic determining unit 208 changes the standard value read in step S801 to control the virtual light source to have a lower diffusion characteristic. That is, the angle β is set as a value smaller than the standard value by a predetermined value.

  In step S807, the characteristic determination unit 208 performs the same processing as step S607 described in the first embodiment, that is, outputs diffusion characteristic information, and ends a series of operations related to the present processing.

  In the subsequent processing, as in the first embodiment, the luminance correction unit 205 performs processing for correcting the image signal by adding a signal corresponding to the virtual light source to the image signal.

  As described above, in the present embodiment, the diffusion characteristic of the virtual light source is set based on the comparison result between the target value and the ratio of the area of the image area that is brightly illuminated by the ambient light. Thereby, the relighting process can be performed so that the state of the shadow in the image area caused by the ambient light approaches the target state. In other words, by determining the diffusion characteristics of the virtual light source according to the state of the shadow due to the ambient light, it is possible to appropriately correct the state of the shadow in the image when performing the relighting process using the virtual light source.

(Other embodiments)
In the above description, the case where the diffusion characteristic of the virtual light source is changed by a predetermined value from the standard setting value has been described. However, the present invention provides a method of controlling the diffusion characteristic of the virtual light source. It is not limited to this. For example, the diffusion characteristic of the virtual light source may be set, subject shadow information may be further calculated in a state where the relighting process has been performed, and the value may be repeatedly compared with a target state. In this way, the state of the shadow of the subject asymptotically approaches the target, and more precise control becomes possible.

  Further, the diffusion characteristic of the virtual light source may be controlled based on the position of the subject in the image area. For example, in the case shown in FIG. 7, the diffusion characteristic of the virtual light source may be set high to widen the range in which light from the virtual light source is irradiated. At this time, if the person located at the lower left of the image shown in FIG. 7 is the main subject, the diffusion characteristics of the virtual light source are set so that the light of the virtual light source is irradiated to the area where the person is located. I do. Conversely, when the diffusion characteristics of the virtual light source are set low because the diffusion characteristics of the ambient light are high, the diffusion characteristics of the virtual light source are set so that the light of the virtual light source is not irradiated to the position of the person who is the main subject. Should be set. By doing so, it is possible to produce a lighting effect that makes the main subject in the image stand out, and it is possible to generate an image in which the shadow state of the subject is preferable.

  In the above description, when controlling the diffusion characteristics of the virtual light source, the angle at which the light from the virtual light source spreads is controlled. However, the present invention is not limited to the method for controlling the diffusion characteristics of the virtual light source. . For example, a parameter for calculating an output value of a pixel with respect to light emitted from a virtual light source may be changed. The light from the virtual light source is attenuated in inverse proportion to the square of the distance from the virtual light source. May be controlled.

  Further, the method for controlling the diffusion characteristic of the virtual light source described in the second embodiment and the method for controlling the diffusion characteristic of the virtual light source described in the first embodiment may be used in combination. For example, when the angle representing the spread of the light emitted from the virtual light source is set to a large value as in the second embodiment, the shape and number of the virtual light sources are further set by the control method described in the first embodiment. Alternatively, the diffusion characteristics of the virtual light source may be set high. By doing so, it becomes possible to set a virtual light source having a more natural diffusion characteristic and perform relighting processing.

  Furthermore, in the above description, when correcting the brightness of an image by relighting, the method of correcting using only a virtual light source has been described. However, the present invention limits the brightness correction method to the case of using only a virtual light source. It does not do. For example, when there are very few areas brightly illuminated by ambient light and the remaining areas are dark, it may be difficult to correct the brightness using only the virtual light source. In this case, it is only necessary to emit the other illumination such as a strobe or a video light to uniformly brighten the entire image, and then perform the relighting process using the virtual light source. By doing so, it becomes possible to correct a preferable image while preventing the noise from being remarkably amplified by the relighting process using the virtual light source.

  Further, in the above description, when correcting the brightness of an image by rewriting, a method of adding a change in pixel value caused by irradiation of a virtual light source to a pixel value to be processed has been described. The method is not limited to this. For example, the pixel value to be processed may be subtracted such that the state of the shadow of the subject caused by the ambient light approaches the target state. Alternatively, some processing targets may add pixel values, and some processing targets may subtract pixel values. That is, if the relighting process is performed for the purpose of setting the state of the shadow of the subject to the state of the target shadow, the image does not have to be brightened by the irradiation of the virtual light source.

  The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program and reads the program. This is the process to be performed.

Reference numeral 50: system control unit, 103: imaging unit, 105: image processing unit, 207: shadow state detection unit, 208: characteristic determination unit, 205: luminance correction unit

Claims (16)

Acquisition means for acquiring a captured image;
Determining means for determining the degree of shading of the partial region of the captured image,
Setting means for setting parameters of a virtual light source that is a virtual light source for the captured image,
Correction means for adding an effect of irradiating the virtual light from the virtual light source to the captured image based on the parameters set by the setting means,
The correction means is capable of performing a correction to add an effect of irradiating the virtual light by combining a plurality of virtual light sources,
The setting unit changes a diffusion characteristic of the virtual light emitted to the captured image by setting a combination of parameters of the plurality of virtual light sources based on a degree of shading detected by the determining unit. An image processing apparatus characterized by the above-mentioned.   The image processing apparatus according to claim 1, wherein the setting unit sets a positional relationship between the plurality of virtual light sources.   The image processing apparatus according to claim 2, wherein the setting unit sets a distance between the plurality of virtual light sources.   The image processing apparatus according to claim 1, wherein the setting unit sets the number between the plurality of virtual light sources.   The image processing apparatus according to claim 1, wherein the plurality of virtual light sources are point light sources.   3. The image processing apparatus according to claim 2, wherein the plurality of virtual light sources are each a surface light source or a linear light source.   The image processing apparatus according to claim 6, wherein the setting unit sets a relative angle between the plurality of virtual light sources. If the degree of the shadow is determined by the determining means is a first degree, the setting means the combination of the parameters of the first virtual light before Symbol plurality of virtual light sources that will be irradiated in the diffusion properties on the captured image And set
When the degree of the shadow determined by the determining means is a second degree that is stronger than the first degree, the setting means sets a second degree of diffusion larger than the first diffusion characteristic in the captured image. the image processing apparatus according to any one of claims 1 to 7 virtual light second diffusion characteristic and sets a combination of parameters of the previous SL plurality of virtual light sources that will be illuminated. The image processing apparatus further includes a determination unit configured to determine a degree of shading of a partial region of the captured image,
The setting means, based on the degree of the shadow is determined by the determining means, so that the degree of shading of the partial region is weakened, according to claim 1 to 8, characterized in that to set the parameters of the virtual light source The image processing device according to claim 1. The judgment unit may image processing apparatus according to claim 8 or 9 the degree of the shadow, and wherein the determining based on the luminance value obtained from the captured image. Acquisition means for acquiring a captured image;
Setting means for setting parameters of a virtual light source that is a virtual light source for the captured image,
Correction means for adding an effect of irradiating the virtual light from the virtual light source to the captured image based on the parameters set by the setting means,
The setting unit is capable of setting a curvature of a virtual light irradiation surface of the virtual light source, and changing a magnitude of a curvature of the virtual light to change a diffusion characteristic of the virtual light irradiated on the captured image. An image processing device characterized by changing. The acquisition unit, an image processing apparatus according to any one of claims 1 to 11, characterized in that to obtain the captured image using the imaging unit including an imaging device.   The image processing apparatus according to claim 1, further comprising a photographing auxiliary light for illuminating a subject at the time of photographing.   The image processing apparatus according to any one of claims 1 to 13, wherein the correction unit performs correction to reduce luminance of a processing target region as an effect of irradiating the virtual light. Acquiring means for acquiring a captured image,
Determining means for determining the degree of shading of a partial region of the captured image;
Setting means for setting parameters of a virtual light source that is a virtual light source for the captured image;
Correction means, based on the parameters set in the setting step, a correction step of adding an effect of irradiating the virtual light from the virtual light source to the captured image,
In the correction step, it is possible to perform correction to add the effect of irradiating the virtual light by combining a plurality of virtual light sources,
In the setting step, a diffusion characteristic of the virtual light emitted to the photographed image is changed by setting a combination of parameters of the plurality of virtual light sources based on a degree of shading detected by the determination unit. A method for controlling an image processing apparatus, comprising:   A program for causing a computer to function as each unit of the image processing apparatus according to claim 1.

Images