JP4236433B2 - System and method for simulating fill flash in photography - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates generally to photography, and more particularly to simulations using flash or strobe lights.
[0002]
[Prior art]
Photographers often use flash lights or strobe lights to give the desired effect to their photos. Many modern cameras have built-in electronic flash lamps to make the flash easy to use. A flash may be used to provide the main lighting of the scene when there is not enough light from other light sources. The light obtained from other light sources is often referred to as ambient light.
[0003]
Even if a flash is not used, sufficient ambient light can be obtained for the exposure of the photograph, but the distribution of the ambient light may lead to a disappointing result. This phenomenon is often seen in subjects that are brightly illuminated at the back, such as humans against a sunlit window in a relatively dark room. Since the camera sets the exposure according to the average brightness of the scene, the person is significantly underexposed.
[0004]
A technique for improving the quality of photographs in such a situation is to use a fill flash, that is, an auxiliary flash. Fill flash is a scene where there may be enough ambient light in a photo, but it can be further improved by using a flash, and it is usually a technology that uses a flash or strobe built in or external to the camera to compensate for the light. is there. In the above example, using a camera flash can improve human illumination, but has little effect on the brightness of objects illuminated by the sun outside the window. In this way, the relative illuminance between the human and the outdoor object is closer, thus improving the relative exposure of the various components of the scene.
[0005]
The fill flash can also be used for other purposes. The fill flash can emphasize highlights on the near-side subject. In addition, for example, an unpleasant shadow that appears when a person to be photographed is illuminated by intense sunlight from an angle close to the overhead can be reduced with a fill flash.
[0006]
Using fill flash can improve the quality of photos in some situations, but this technique also has some drawbacks. There are cases where flash photography is not allowed, such as in a museum. Or flash photography at weddings or formal dinners is considered rude.
[0007]
Using a fill flash may cause unwanted shadows. For example, when a person is photographed using a flash, an unpleasant and undesired shadow that falls on an object directly behind the person may be clearly drawn on one side of the person in the photograph.
[0008]
Also, electronic flash consumes considerable electrical energy. When the camera flash is used, the battery that supplies power to the camera is rapidly consumed, so that the time that the camera can be used is limited by one battery replacement or charging.
[0009]
Furthermore, the use of an electronic flash may produce an undesirable effect called “red-eye phenomenon” when the flash light is reflected off the retina of the subject's eye and returns to the camera. It is not desirable to use a method for reducing the “red-eye phenomenon” such as performing a preflash that preliminarily emits light before the actual light emission to contract the iris of the subject because the shutter response time of the camera is delayed.
[0010]
Finally, some cameras require users to decide whether to use a fill flash before taking a picture. In the past, flash effects could not be easily added or removed after taking a picture. While some cameras can automatically determine whether or not to use a fill flash, photos taken with an automatic camera are susceptible to all other drawbacks of the fill flash. In addition, photographers are not convinced of camera decisions and are seeking a way to correct the results.
[0011]
[Problems to be solved by the invention]
A camera that can add or adjust the effect of fill flash after taking a photo, provide the beneficial effect of fill flash, while preventing unwanted effects, and even in situations where flash shooting is not available Is required.
[0012]
[Means for Solving the Problems]
A digital camera shoots several scenes continuously without using an electronic flash. Individual photos are focused at different distances from the camera. Each photograph is recorded together with its focused distance, that is, in-focus distance. Analyze the photos using spatial contrast analysis to determine which elements of each scene are in focus. The distance from the camera to each element of the scene is determined by combining information about which element is in focus and the recorded in-focus distance. Based on the distance information, the brightness of each area of the scene is selectively adjusted to simulate the effect obtained when the flash is used.
[0013]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 shows a block diagram of a digital camera. A lens (101) collects light from a scene (not shown). The collected light changes direction (102) to form an image of the scene on the sensor (103). The sensor may be a CCD element array or a CMOS sensor. The operation of the focus adjustment mechanism including all or part of the lens (101) can be controlled by a control signal (113) from a logic unit (110) comprising a microprocessor system. A feedback signal (114) indicating the position of the focus adjustment mechanism is transmitted from the lens (101) to the logic unit (110). Similarly, the operation of the sensor is controlled by a control signal (105) from the logic unit (110). The image information signal (104) is transmitted from the sensor to the logic unit (110). A flash or strobe (106) may be used to provide additional light (107) to the scene. The strobe is operated with strobe electronics (108), which is further controlled with a logic unit (110). The camera may comprise a display (109) for displaying image data. Further, the camera may include a memory (111) for storing and retrieving image data and exchanging data with other devices (not shown). The camera user can operate various control inputs (112) that affect the operation of the camera.
[0014]
The focus adjustment mechanism of the digital camera embodying the present invention can be controlled by the camera logic unit (110). The lens (101) or a part of the lens moves in the axial direction under the control of the logic unit (110), and the focusing operation is performed so that the logic unit (110) can know the position of the focus adjustment mechanism at the time of shooting. I do. Instead of doing this, the position of the sensor (103) can be adjusted under the control of the logic unit (110) to perform the focusing operation. A feedback signal may also be provided from the focus adjustment mechanism to the logic unit (110) to directly indicate the position of the focus adjustment mechanism. By combining with the built-in information about the optical properties of the lens (101) and the design of the camera, the distance from the position of the focusing mechanism to the focused subject can be determined.
[0015]
Digital cameras essentially generate a numerical representation of each photograph taken. Each location on the photograph is called a “pixel” or “pixel” and the camera records a numerical value that indicates the brightness at that location in the scene. As a result, the scene representation is an array of numbers. The position in this array corresponds to a particular pixel, that is, the position of the scene, and the numerical value stored at each position of the array represents the brightness of that position in the scene. In addition, the camera may record information regarding the color of each pixel position in the captured scene. For example, many cameras represent the color of a pixel using three components that represent the contribution of the red, green, and blue wavelengths of light to the brightness of that pixel. The overall brightness of a pixel can be calculated as the sum of red, green and blue contributions, as a weighted sum, or as some other combination of color information. Various methods for calculating pixel brightness from color information are well known in the art. One skilled in the art will readily appreciate that the present invention applies to both cameras that record only luminance information for each pixel and cameras that also record color information.
[0016]
For the purposes of this disclosure, the photograph may be a numerical representation of the scene captured by the camera, and need not be a scene burned into the photograph.
[0017]
FIG. 2 is a schematic representation of a scene used to illustrate an embodiment of the present invention. In the scene, a human (201) is standing in the shadow of the tree (202) close to the camera. The tree (202) is farther from the camera than the human (201), but is not so far away. The background of the scene is other objects including the house (203). The background object is far from the camera compared to the human (201) or tree (202). Some objects in the scene, especially the human shirt (204) and hat (205), tree trunk (206), and house (203) appear to have the same brightness.
[0018]
This scene provides an example of a scene that can be improved by using fill flash. While the overall exposure of the scene is balanced, the foreground person (201) appears darker than the photographer wants due to insufficient ambient light at that location. Photographers often want to express the skin tone in a photo with a perceptual brightness that is roughly halfway between black and white. In this expression, the skin tone (207) of the human (201) is much darker than that and is perceptually located about 25% from black to white.
[0019]
FIG. 9 shows a flowchart of the embodiment of the present invention. The operation of each block in FIG. 9 is as follows.
901: Determine distance 902: Take a series of trial shots at various in-focus distances 903: Store trial shots and corresponding distances 904: Analyze photos 905: Spatial location of each position in each photo 906: Specify the most focused photograph for each position in the scene 907: Specify the distance from the camera as the optimum focusing distance for each position of the scene 908: Final 909: Get an improved photo according to an embodiment of the invention that selectively adjusts the area on the photo based on distance. First, use the camera to test several consecutive scenes. (902). Each photograph is focused at various distances from the camera, and the in-focus distance of each photograph is recorded (903). The camera changes the focus distance by adjusting the position of the focus adjustment mechanism for each photo. These trial shots can be performed using camera settings that minimize the camera's depth of field, as is well known in the art. The term “depth of field” refers to the range of distance from the camera that allows an object in it to be in acceptable focus. A series of trial shots is stored in the memory (111) of the camera.
[0020]
3, 4, and 5 are examples of three consecutive photographs of the scene shown in FIG. 2 taken by changing the setting of the focus adjustment mechanism. Depending on the camera design, the entire series of trial shots may be more or less than three. The example of three photos was chosen here for illustration.
[0021]
FIG. 3 shows a picture of the scene of FIG. 2 taken with the camera focusing mechanism set to focus on an extremely distant object and with the camera set to the minimum depth of field. In FIG. 3, the object far from the camera including the house (203) is clearly in focus. The tree (202) is at a medium distance from the camera and is somewhat out of focus. Since the human (201) is relatively close to the camera, the focus is significantly deviated accordingly. Along with the photograph of FIG. 3 is stored an annotation that the photograph was taken using an essentially infinite camera focus distance determined by a combination of position information of the focus adjustment mechanism and information on the optical properties of the lens (101). (903).
[0022]
FIG. 4 shows a photograph of the scene of FIG. 2 taken with the camera focus adjustment mechanism focusing on an object 2 m from the camera and setting the camera's depth of field to the minimum. In FIG. 4, objects far from the camera, including the house (203), are somewhat out of focus. The tree (202) is at a medium distance from the camera and is clearly in focus. The human (201) is relatively close to the camera and is somewhat out of focus accordingly. Along with the photograph of FIG. 4, an annotation that the photograph was taken using a camera focus distance of 2 m is stored (903).
[0023]
FIG. 5 shows a photograph of the scene of FIG. 2 taken with the camera focus adjustment mechanism focusing on an object 1 meter from the camera and setting the depth of field of the camera to a minimum. In FIG. 5, objects far from the camera including the house (203) are significantly out of focus. The tree (202) is at an intermediate distance from the camera and is somewhat out of focus accordingly. The human (201) is relatively close to the camera and is clearly in focus. Along with the photograph of FIG. 5, an annotation that the photograph was taken using a camera focus distance of 1 m is stored (903).
[0024]
When a series of trial shots is stored in the camera memory, the camera logic unit (110) analyzes (904) the series of photographs, and each area of the scene is most in focus in the series. You can determine which one is there. Alternatively, a series of photos can be transferred to a computer and analyzed by the computer. The external computer used for analysis can also be incorporated in another device such as a printer.
[0025]
For each position in the scene and each trial photo, a spatial contrast metric is calculated (905). As is well known in the art, the spatial contrast of a photograph is an effective measure of focus. FIG. 6 shows the calculation of the spatial contrast evaluation value for a specific position in the scene for each of the three trial shots. For example, using a trial photograph (601) with a focus distance of essentially infinity, a table (602) is provided that lists pixel luminance values at locations on the photograph. In this table (602), it can be seen that the central pixel of the region has a luminance value of 190, and the surrounding pixels have different luminance values.
[0026]
One way to calculate a spatial contrast estimate at a pixel is to sum the squares of the difference between the luminance value of the pixel and the luminance value of the surrounding individual pixels. When this method is used, the larger the spatial contrast evaluation value is, the clearer the focus is. Using the values listed in table (602), the spatial contrast estimate is calculated as follows.
Spatial contrast rating = (220-190) ** 2+ (35-190) ** 2+ (26-190) ** 2+ (220-190) ** 2+ (50-190) * * 2 + (28-190) ** 2+ (40-190) ** 2+ (30-190) ** 2 = 146665 (where ** is a power)
When this method is used in the same region of a test photograph (603) taken at a focus distance of 2 m, a spatial contrast evaluation value of 46990 is obtained. This indicates that this region of the scene has a lower sharpness of the photo (603) than the photo (601).
[0027]
Similarly, the evaluation value of the spatial contrast is 186 in the same region of the test photograph taken at the focal distance of 1 m (604). This indicates that this area of the photo is significantly out of focus.
[0028]
One skilled in the art will appreciate that this method of calculating spatial contrast estimates is an example, and there are many methods in the art.
[0029]
Using the spatial contrast rating at a particular location calculated for all three trial shots, this particular location is essentially the most focused photo in focus at infinity (906). I understand that. Therefore, the object at that position in the scene is far from the camera.
[0030]
If this technique is applied to all positions in the test photograph (906), a distance map can be constructed. FIG. 7 shows the distance map obtained for the example scene. In this simple example, three regions appear. There is an object of about 1 m from the camera in one area. In the other area there is an object about 2 m from the camera. In the last region, there is an object at infinity that is relatively far from the camera. The distance map shown in FIG. 7 is a convenient tool for conceptually explaining a part of the process in which the camera simulates the fill flash, but the camera does not necessarily have to compose the entire distance map at the same time.
[0031]
The camera or external computer can use this distance information to simulate the effect that the fill flash would have on the photo when using the fill flash. In order to execute the simulation (909), it is only necessary to change the numerical value stored in the memory representing the luminance value of the pixel on the photograph by software or firmware of the camera or computer.
[0032]
A flash external to the camera can be modeled as a point light source that essentially illuminates the scene being photographed, according to the well-known inverse square law. That is, the brightness of the object obtained by flash irradiation is inversely proportional to the square of the distance from the camera to the object. For example, the brightness of an object located 2 m from the camera is 1/4 of the brightness of an object located 1 m from the camera. Of course, the inverse square law applies only to the camera flash. The flash light has the effect of being added to the ambient light that shines into the scene.
[0033]
One useful feature implemented by the present invention is that a variable amount of pseudo fill flash can be applied to a photo after the camera user has taken the photo. There is no need to estimate the energy required for the flash before taking a picture. Since the camera can display a photograph on the display (109) of the camera, the user can adjust the pseudo fill flash effect by operating the control means (112) of the camera while viewing the effect on the display (109). Similar control means are provided when analysis and simulation are performed on a computer external to the camera.
[0034]
In the case of the example, the user can specify to apply a fill flash effect sufficient to render the human (201) skin tone (207) in a photograph with a perceptual luminance level between black and white. . The user selects a fill flash amount such that the human (201) skin tone (207) represented in the final photograph has an appropriate perceptual luminance level. The camera uses this fill flash volume information and the distance map of the scene so that other objects on the photo can also have the same fill flash that the illuminance achieves proper exposure with respect to human (201) skin tone (207) Express as if influenced by
[0035]
For example, the shirt (204) and hat (205) worn by a human (201) in the scene are also as bright as the human skin (201) tone (207) in the final photo, but still somewhat Since it is a dark material, it should appear somewhat darker than the skin tone (207).
[0036]
Since the scene tree (202) is far from the camera, the influence of the pseudo fill flash is less in accordance with the inverse square law. The tree (202) is also brighter, but not as good as the human (201) shirt or (204) and hat (205). Specifically, the bark (206) of the tree (206) that looked the same brightness as the shirt (204) and the hat (205) of the test photograph shown in FIG. ) And hat (205).
[0037]
Since an object far from the camera is essentially unaffected by the camera flash, the fill flash simulation does not affect its brightness level. Specifically, house (203) will appear at the same brightness level as in FIG. 2 in the final photo.
[0038]
Therefore, multiple objects that appeared to have the same brightness in the scene of FIG. 2, shirt (204) and hat (205), tree bark (206), and house (203), have different fill distances due to their different distances from the camera. The final photo, including a flash simulation, is displayed at three different brightness levels.
[0039]
When the process is complete for all locations on the photo that need to be adjusted, the camera stores the final photo reflecting the effect of the pseudo fill flash. The final photo can be created from either a trial photo that has already been taken or another photo taken with the preferred camera settings (908). For example, a final photograph can be obtained by applying a fill flash simulation after taking a picture with a camera with the maximum depth of field.
[0040]
FIG. 8 shows the final photograph obtained after applying the pseudo fill flash according to an embodiment of the present invention.
[0041]
In the example above, the fill flash simulation was realized by brightening each element of the photograph. When taking a picture of another scene, such as a brightly illuminated object from behind, it should be noted that the use of the fill flash may interact with the camera's exposure determination method. In the final photo, the foreground object is brighter than the distant object, but the photo result shows that part of the photo area appears darker when using the fill flash than when using the fill flash. Sometimes. The present invention is intended to take this situation into account.
[0042]
It will be appreciated that a fill flash simulation according to an example embodiment of the present invention has several advantages over methods that actually use fill flash. Using a pseudo-fill flash provides the desired effects of fill flash, such as correcting objects that are exposed to light from behind, but eliminates many of the disadvantages of actual fill flash. Pseudo-fill flash may be used when camera flash is not allowed or considered rude. The pseudo-fill flash can be applied after taking a picture, and the intensity of the fill flash effect can be adjusted according to user preference. The pseudo fill flash saves battery capacity because it does not use a camera flash. In fact, the pseudo fill flash does not drain new battery capacity because the camera can collect the necessary test shots during the automatic focus determination process. Pseudo-fill flash projects an object in the foreground and does not cast an unpleasant dark shadow on the background. There is no “red-eye phenomenon” in the object's eyes.
[0043]
The foregoing description of the invention has been presented for purposes of illustration and description. This description is not intended to be exhaustive or to limit the invention to the precise form disclosed, and other modifications and variations are possible in light of the above description. For example, a pseudo fill flash need not be restricted to perform according to the inverse square law as an actual flash. This embodiment has been chosen and described in order to best illustrate the principles of the invention and the practical application of the invention. Accordingly, other persons skilled in the art can make best use of the present invention in various embodiments and variations tailored to the particular application planned. Except as limited by the prior art, the claims are to be construed to include other alternative embodiments of the invention.
[Brief description of the drawings]
FIG. 1 is a simplified block diagram of a digital camera.
FIG. 2 is a diagram showing a scene to be photographed.
3 is a picture of the scene of FIG. 2 taken with a camera that focuses on a distant object.
4 is a picture of the scene of FIG. 2 taken with a camera set to a medium focus distance.
5 is a picture of the scene of FIG. 2 taken with a camera that focuses on an object at a short distance.
FIG. 6 is a diagram showing a configuration of spatial contrast evaluation values using three different photographs.
FIG. 7 is a distance map of the scene of FIG.
8 is a photograph of the scene of FIG. 2 after applying a pseudo fill flash in accordance with an embodiment of the present invention.
FIG. 9 is a diagram representing steps for implementing an embodiment of the present invention.
[Explanation of symbols]
108: Strobe electronics 109: Display 110: Logic unit 111: Memory 112: User operation means

Claims (14)

The following steps (a) to (c) are provided to simulate a fill flash without using a flash in a camera system:
(a) determine the distance from the camera to the object in the scene;
(b) take a picture of the scene;
(c) selectively adjusting the respective luminance of the plurality of areas on the photographic based on the distance to simulate the fill flash. The method according to claim 1, wherein the step (a) includes the following (a-1) to (a-3):
(a-1) Take a series of photos with a camera set to focus on objects at various distances from the camera;
(a-2) storing the photograph and the focusing distance for each photograph;
(a-3) Analyzing the series of photographs and the in-focus distance corresponding to each of the photographs.   The method of claim 2, wherein analyzing the sequence of photographs includes calculating a spatial contrast estimate. 4. The method of claim 3, wherein determining the distance to the object at each position in the scene includes the following steps (a-4) and (a-5):
(a-4) using the spatial contrast evaluation value to detect a specific one of the series of photographs in which the object at the position in the scene is in focus better than other photographs;
(a-5) The distance from the camera to the object at the position of the scene is specified as the in-focus distance stored together with the specific photograph.   The method of claim 1, wherein in the finally obtained photograph, an area containing an object closer to the camera is made brighter than an area containing an object far from the camera.   6. The method of claim 5, wherein in the final photograph, the region is modified according to the inverse square law. A camera system that provides the following means (a) to (c) to simulate a fill flash without using a flash:
(a) means for determining the distance from the camera to the object in the scene;
(b) means for taking a picture of the scene;
(c) means for selectively adjusting the brightness of each of the plurality of regions of the photograph based on the distance to simulate a fill flash ; The camera system according to claim 7, wherein the means (a) includes the following means (a-1) to (a-3):
(a-1) means for taking a series of photos with a camera set to focus on an object at various distances from the camera;
(a-2) means for storing the photograph and the focusing distance for each photograph;
(a-3) Means for determining the distance of the object by analyzing the series of photographs and the corresponding in-focus distance.   9. The computer according to claim 8, further comprising a computer separate from the camera, wherein a series of trial photographs and respective in-focus distances are transmitted to the separate computer for analysis and fill flash simulation. The camera system described.   9. The camera system of claim 8, wherein the means for analyzing the series of photographs includes calculating a spatial contrast estimate. 9. The camera system according to claim 8, wherein the means for calculating the distance to the object in each position of the scene includes the following means (d) and (e):
(d) Means for detecting a specific sheet in which the object at the position in the scene is in focus better than the other photographs from the series using the evaluation value of the spatial contrast ;
(e) means for specifying a distance from the camera to the object at the position of the scene as a focusing distance of the camera stored together with the specific photograph;   8. The camera system according to claim 7, wherein an area including an object closer to the camera is made brighter than an area including an object far from the camera in the finally obtained photograph.   13. The camera system according to claim 12, wherein the luminance of the finally obtained photograph area is corrected according to the inverse square law. A camera that simulates a fill flash without using a flash with the following means (a) to (c):
(a) means for determining the distance from the camera to the subject in the scene;
(b) means for taking pictures;
(c) means for selectively modified based the respective brightness of the plurality of regions of the finally obtained photographs to simulate the fill flash on the distance.

Images