JP7632404B2 - Light Source Estimation System - Google Patents

Description

This disclosure relates to a light source estimation system.

Patent document 1 discloses a method for accurately estimating the lighting environment using a wide-angle image captured from a single viewpoint.

JP 2021-149679 A

The applicant has discovered the following problem. When setting a light source for a three-dimensional model that models a real environment, it is sometimes desirable to set the light source so as to reproduce an actual light source. However, captured images are affected not only by the light source but also by the camera's shutter speed and gain, making it difficult to find optimal light source settings.

This disclosure was made in consideration of these problems, and realizes a light source estimation system that synchronizes shooting parameters related to the brightness of a rendered image with the camera settings and searches for appropriate light source settings for a three-dimensional model.

A light source estimation system according to an aspect of the present disclosure includes: a first acquisition unit that acquires a first captured image of an imaging area under a specified light source;
an adjustment means for setting a light source corresponding to the specified light source in a three-dimensional model that models the imaging area, and adjusting a shooting parameter related to brightness of the first rendering image so that a difference between a first rendering image generated from the three-dimensional model and the first captured image becomes small;
a second acquisition means for acquiring a second captured image of the imaging area under an unknown light source;
a search means for searching for a light source setting corresponding to the unknown light source so that a difference between a second rendering image generated from the three-dimensional model and the second captured image becomes small after adjusting the camera parameters;
Equipped with.

The present disclosure makes it possible to realize a light source estimation system that synchronizes the shooting parameters related to the brightness of a rendered image with the camera settings and searches for appropriate light source settings for a three-dimensional model.

1 is a block diagram showing a configuration of a light source estimation system according to a first embodiment. FIG. 2 is a block diagram showing a configuration of a server according to the first embodiment. 13 is a flowchart showing the flow of a material setting method.

Specific embodiments to which the present disclosure is applied will be described in detail below with reference to the drawings. However, the present disclosure is not limited to the following embodiments. In addition, the following descriptions and drawings have been simplified as appropriate for clarity of explanation.

EMBODIMENT 1
Hereinafter, a light source estimation system according to the first embodiment will be described with reference to the drawings. Fig. 1 is a block diagram showing a configuration of a light source estimation system 1000 according to the first embodiment. The light source estimation system 1000 includes a camera 100 and a server 200. The camera 100 and the server 200 are connected to each other via a network N so as to be able to communicate with each other.

The camera 100 is installed in the house 10, etc. There may be more than one camera. The camera 100 may be placed, for example, in the corner of a room. The camera 100 is also called a home sensor. The camera 100 captures an image of the imaging area 20. The camera 100 outputs the captured image to the server 200 via the network N.

The imaging area 20 captured by the camera 100 is three-dimensionally modeled, and the server 200 stores the three-dimensional model. For example, a sink and furniture (e.g., a sofa, a shelf, a desk) are arranged in the imaging area 20. Multiple pieces of furniture may be arranged in the imaging area 20. The imaging area 20 may also include walls and a floor.

The camera 100 has a function for changing settings related to the brightness of the captured image. In other words, the camera 100 has a function for adjusting the shutter speed, gain, aperture, etc. The position and orientation of the camera 100 are assumed to be known through camera calibration.

The imaging area 20 is illuminated by a light source 30. The light source 30 includes an artificial light source such as a fluorescent light, an LED (Light Emitting Diode) light source, or a lamp. The artificial light source 30 is configured to be switchable between a lit state and an off state. The artificial light source 30 may be configured to be adjustable in brightness. The three-dimensional model of the imaging area 20 may include an object (called a light source object) representing the artificial light source 30.

The light source 30 may also include light from natural light sources such as the sun or the moon. The room containing the imaging area 20 may be equipped with windows and curtains.

The server 200 searches for appropriate light source settings for a three-dimensional model that models the imaging area 20. The server 200 includes a calculation unit such as a CPU (Central Processing Unit) and a storage unit such as a RAM (Random Access Memory) or a ROM (Read Only Memory) that stores various control programs and data. In other words, the server 200 functions as a computer and performs processing based on the various control programs and the like.

The processing by the server 200 may be executed on the camera 100 side. The camera 100 is assumed to include a processor, memory, and the like. Therefore, the light source estimation system 1000 that does not include the server 200 may also be included in the first embodiment.

Next, the functions of the server 200 will be described in detail with reference to FIG. 2. The server 200 includes a memory unit 210, a first acquisition unit 220, an adjustment unit 230, a second acquisition unit 240, a search unit 250, a change unit 260, an update unit 270, and a setting unit 280.

The storage unit 210 stores a three-dimensional model 211 of the imaging area 20. The server 200 has a function of generating a rendering image from the three-dimensional model 211. The server 200 has a function of setting shooting parameters (e.g., aperture, exposure time, gain) related to the brightness of the rendering image.

The three-dimensional model 211 includes objects such as a sink, furniture, floors, and walls. The geometry of each object is assumed to be known. A material is set for each object in the three-dimensional model 211 by a setting unit 280, which will be described later. A material refers to the quality of the material that is set for an object. By setting a material for an object, the optical characteristics of the furniture, floors, walls, etc. are expressed.

The server 200 also has a function for setting a light source in the three-dimensional model 211. The light source may be, for example, a point light source, a linear light source, a surface light source, a parallel light source, a spotlight, or ambient light. The ambient light may be generated using an environmental map that controls the color and intensity of light from different directions. The server 200 has a function for setting the type, position, direction, brightness, etc. of the light source.

The three-dimensional model 211 may include a light source object corresponding to an artificial light source 30. The light source object may be set to an on or off state and may have configurable brightness.

The memory unit 210 also stores the above-mentioned control program (not shown).

The first acquisition unit 220 acquires a first captured image of the imaging area 20 under a specified light source. The first captured image is captured, for example, with only a specific artificial light source 30 (e.g., a fluorescent light installed on the ceiling) turned on. To eliminate the influence of the natural light source 30, the first captured image may be captured at night. The first captured image may also be captured with all artificial light sources 30 turned off.

The adjustment unit 230 performs light source setting corresponding to a specified light source in the three-dimensional model 211. In other words, the adjustment unit 230 performs a light source setting that is preset so as to reproduce a specified light source. The adjustment unit 230, for example, places a light source (e.g., a point light source) of a specified brightness at a specified position in the three-dimensional model 211. The adjustment unit 230 may set a light source object corresponding to a specified artificial light source 30 to a lit state. If the first captured image is captured with all artificial light sources 30 turned off, the adjustment unit 230 may set an ambient light of a specified brightness.

Then, the adjustment unit 230 adjusts the shooting parameters related to the brightness of the first rendering image so that the difference between the first rendering image generated from the three-dimensional model 211 and the first captured image is reduced.

Specifically, the shooting parameters related to brightness are aperture, exposure time, and gain. Shooting parameters are also called image shooting parameters or camera parameters.

The difference described above may be a difference in average luminance, a difference in average brightness, a difference in luminance distribution, or a difference in pixel values of each pixel. The pixel values may be RGB (Red Green Blue) values or gray values. The difference may be the sum of squares of the differences in each pixel value. The adjustment unit 230 may generate binary images by binarizing each of the rendering image and the captured image with a predetermined threshold value, and adjust the shooting parameters so that the difference between the binary images is small. The adjustment unit 230 may also adjust the camera parameters so that the similarity between the two images is increased using a machine learning model that outputs the similarity between the two images.

The second acquisition unit 240 acquires a second captured image of the imaging area 20 under an unknown light source. The second captured image is captured, for example, when a user living in the house 10 turns on any artificial light source 30 or when illuminated by a natural light source 30.

After the adjustment unit 230 adjusts the shooting parameters, the search unit 250 searches for a light source setting corresponding to an unknown light source so that the difference between the second rendering image generated from the three-dimensional model 211 and the second captured image is small. In other words, the search unit 250 sets the second captured image as a target image, sets the light source setting in the three-dimensional model 211 as parameters (e.g., the position, direction, and brightness of the light source), and searches for parameters that minimize the above difference. The search unit 250 may search an environment map that controls the color and brightness of light from different directions. Like the adjustment unit 230, the search unit 250 can search for a light source setting when the difference is small based on the average luminance, luminance distribution, the difference in pixel values of each pixel, the sum of squares of the differences in each pixel value, a binary image, similarity, etc.

A technique called a differentiable renderer is known that searches for light source settings in a three-dimensional space from a two-dimensional image. The differentiable renderer is a technique that uses geometry, materials, light sources, and shooting parameters as variables to generate an image similar to a target image from a noise image (initial values). In the differentiable renderer, the least square error between the created image and the target image may be used as a loss function. The search unit 250 may use the differentiable renderer with the environment map and light sources as parameters to search for light source settings.

The search unit 250 may search for light source settings in response to a command from a robot or other user device placed in the house 10. For example, when the robot detects a change in brightness in the house 10, the robot may instruct the server 200 to search for light source settings.

After searching for the light source setting, the light source setting may be performed in the three-dimensional model 211 based on the search result. In this case, the rendering image generated from the three-dimensional model 211 reflects the actual state of the light source 30. Therefore, by performing machine learning using the rendering image, a machine learning model with high recognition accuracy can be generated. The server 200 may add the rendering image to the training image.

A technique called domain adaptation is also known, which adapts a machine learning model to learning data from a different domain. In domain adaptation, knowledge obtained from a source domain with sufficient information is applied to a target domain with insufficient information, generating a machine learning model with high recognition accuracy in the target domain. For example, by performing domain adaptation using an image captured by camera 100 as the source domain and a rendered image as the target domain, it is possible to generate a machine learning model with high recognition accuracy under any light source.

The change unit 260 changes the brightness-related settings of the camera 100 (e.g., shutter speed, gain, aperture). For example, when the brightness of an image captured by the camera 100 has changed (e.g., when the number of whiteout pixels or blackout pixels is equal to or greater than a threshold), the change unit 260 sets the automatic exposure function (auto exposure) of the camera 100 to on. This sets an optimal shutter speed or gain for the camera 100. The aperture is assumed to be fixed. After the optimal shutter speed or gain has been set, the change unit 260 locks (stops) the automatic exposure function and fixes the above settings.

The brightness of the captured image may change when the user turns on or off the artificial light source 30, or when the brightness of the natural light source 30 changes over time. In this case, whiteout or blackout occurs in the captured image, so the change unit 260 changes the settings related to the brightness of the camera 100. The change unit 260 may change the settings related to the brightness of the camera 100, for example, when the number of pixels having a minimum brightness value (e.g., 0) is equal to or greater than a threshold value, or when the number of pixels having a maximum brightness value (e.g., 255) is equal to or greater than a threshold value.

When the change unit 260 changes the settings related to the brightness of the camera 100, the update unit 270 updates the shooting parameters (e.g., exposure time, gain) related to the brightness of the rendered image. Like the adjustment unit 230, the update unit 270 adjusts the shooting parameters related to the brightness of the rendered image so that the difference between the rendered image generated from the three-dimensional model 211 and the captured image becomes small. Like the adjustment unit 230, the update unit 270 updates the shooting parameters based on the difference in average luminance, the difference in average brightness, the difference in luminance distribution, the difference in pixel values of each pixel (e.g., RGB values, GRAY values), a binary image, similarity, etc. The shooting parameters related to the brightness of the rendered image may be fixed after updating.

The setting unit 280 sets materials in advance for each object in the three-dimensional model 211. The flow of processing performed by the setting unit 280 will be described with reference to Figure 3.

First, the setting unit 280 generates a rendering image from the three-dimensional model 211 and performs segmentation of the rendering image (step S101). It is assumed that the three-dimensional model 211 has a light source setting according to the state of the light source 30. When the imaging area 20 is illuminated with a specified light source, a light source corresponding to the specified light source is set. When the imaging area 20 is illuminated with an unknown light source, the searched light source is set.

Next, the setting unit 280 estimates the materials of segments with large areas (e.g., walls, floors, sinks) (step S102). The processes in steps S102 to S105 are performed for each segment.

Next, the setting unit 280 estimates the material of the segment using a machine learning model or the like (step S103). The estimated result is used as an initial value. The setting unit 280 may use, for example, the technique described in Abhimitra Meka, et.al., LIME: Live Intrinsic Material Estimation, CVPR, 2018. Next, the setting unit 280 sets the estimated material to the segment (step S103).

Next, the setting unit 280 searches for material settings for the segment so that the difference between the actual image of the portion corresponding to the segment and the above-mentioned segment is small (step S104). The actual image is extracted from the image captured by the camera 100. The difference is expressed as a difference in average luminance, a difference in luminance distribution, a difference in average brightness, a difference in brightness distribution, a difference in pixel values of each pixel (e.g., RGB values, GRAY values), a binary image, etc. The setting unit 280 may set the material so that the similarity output by the machine learning model is large. The setting unit 280 may also search for a material for the segment using a differentiable renderer.

Next, the setting unit 280 sets the searched material to a segment, calculates the final difference between the actual image of the portion corresponding to the segment and the above-mentioned segment, and calculates the confidence according to the difference (step S105). The confidence is calculated, for example, by the formula conf = λ * e ^{- aΔ} , where λ and a are constants and Δ represents the difference. The confidence may also be calculated using a machine learning model.

Next, the setting unit 280 determines whether processing has been completed for all segments with large areas among the segments extracted in step S101 (step S106). If materials have been set for all segments (YES in step S106), the processing ends, and if not (NO in step S106), the processing returns to step S102.

The light source estimation system according to the first embodiment uses a first captured image captured under a specified light source to synchronize the shooting parameters of a rendering image with the settings of the camera 100. This allows the light source estimation system 1000 to search for a light source in the three-dimensional model 211 with high accuracy. By reproducing the light source of the real environment in the three-dimensional model 211, it is possible to improve the accuracy of object recognition by domain adaptation and generate training images that reflect the actual light source environment.

The above-mentioned programs include instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions. The programs may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The programs may be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, transitory computer-readable media or a communication medium include electrical, optical, acoustic, or other forms of propagated signals.

Note that this disclosure is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the present disclosure.

Reference Signs List 1000 Light source estimation system 100 Camera 200 Server 210 Storage unit 211 Three-dimensional model 220 First acquisition unit 230 Adjustment unit 240 Second acquisition unit 250 Search unit 260 Change unit 270 Update unit 280 Setting unit 10 House 20 Imaging area 30 Light source N Network

Claims (1)

A first acquisition means for acquiring a first captured image of an imaging area under a specified light source;
an adjustment means for setting a light source corresponding to the specified light source in a three-dimensional model that models the imaging area, and adjusting a shooting parameter related to brightness of the first rendering image so that a difference between a first rendering image generated from the three-dimensional model and the first captured image becomes small;
a second acquisition means for acquiring a second captured image of the imaging area under an unknown light source;
a search means for searching for a light source setting corresponding to the unknown light source so that a difference between a second rendering image generated from the three-dimensional model and the second captured image becomes small after adjusting the shooting parameters; and
A light source estimation system comprising:

Images