JP7633790B2 - Texture acquisition system, texture acquisition device, and texture acquisition program - Google Patents

Description

The present invention relates to a texture acquisition system, texture acquisition device, and texture acquisition program that acquire shape, texture, and other information about a subject, which is material data used in computer graphics (CG).

With the recent spread of AR (Augmented Reality)/VR (Virtual Reality) technology, there is an increasing need to convert real-world subjects into subject information (hereinafter sometimes referred to as "3D models") that can be handled using CG technology. This is a technology generally known as volumetric capture, which captures the surface pattern and shape of the entire subject using cameras and depth measurement devices arranged to surround the subject (see Non-Patent Documents 1 and 2).

When rendering a 3D model under the desired lighting conditions using CG technology, conventional volumetric capture involves placing the subject in a flat lighting environment, lighting the subject's surface evenly, and photographing the subject without shadows. Since the reflectance of the subject cannot be measured directly, the lighting is photographed as if it were a pseudo-non-directional lighting effect, normalizing the amount of light incident on each part of the subject, and treating the brightness in the resulting photographed image as if it were a pseudo diffuse reflectance coefficient. This makes it possible to express the appearance of the 3D model as expected under the lighting conditions in the virtual space by adding shading during rendering.

O. Schreer, et al., “Capture and 3D Video Processing of Volumetric Video,” 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, 2019, pp. 4310-4314. O. Schreer, et al., “ADVANCED VOLUMETRIC CAPTURE AND PROCESSING,” [online], IBC2018, [Retrieved July 20, 2020], Internet <URL: https://www.ibc.org/download?ac=6559>

However, in conventional volumetric capture techniques, the diffuse reflection component of reflected light is calculated artificially. In addition, the specular reflection component is not handled, or the coefficient (specular reflection coefficient) is set manually either uniformly for the entire subject or for each area. When the subject is a non-rigid body, it is difficult to manually assign coefficients for each area. For this reason, it is particularly difficult to apply it to live video.
Furthermore, when observing the surface of a subject microscopically, the behavior of specularly reflected light differs depending on the presence or absence of fine irregularities. The presence or absence of irregularities on the subject's surface is a factor that greatly affects the visual impression of the subject when looking at it, such as its gloss and feel to the touch. Therefore, in order to express this texture, it is necessary to describe the behavior of surface reflection in units finer than area units, but this cannot be expressed at present.

The present invention has been made in consideration of the above points, and aims to provide a texture acquisition system, texture acquisition device, and texture acquisition program that can acquire the fine unevenness of a subject's surface in real time as a three-dimensional model in order to express the subject's texture.

In order to solve the above problems, the texture acquisition system of the present invention is configured to include a photography booth having a depth measurement device, a polarized lighting device, and a polarized photography device arranged to surround a subject, and a texture acquisition device that is communicatively connected to the depth measurement device, the polarized lighting device, and the polarized photography device, and acquires the texture represented by the uneven state of the subject's surface.
The texture acquisition device includes a storage unit, a subject shape acquisition unit, a concave-convex state calculation unit, a reflection coefficient calculation unit, a surface shape acquisition unit, and a three-dimensional model detailed information output unit.

With this configuration, the texture acquisition device acquires shape information of the subject by receiving depth information measured by the depth measurement device through the subject shape acquisition unit, and the measurement control unit controls the polarized lighting device to irradiate the subject with polarized light of multiple colors.The unevenness state calculation unit acquires information on the reflected light from the subject irradiated with polarized light of multiple colors captured by the polarized photography device, and can calculate the roughness of the subject surface from the variance of the color vectors of the polarized light generated by the unevenness of the subject surface.
This enables the texture acquisition device of the texture acquisition system to calculate the roughness of the object surface based on the variance of the color vector of the deflected light caused by the unevenness of the object surface.

In addition, the texture acquisition device separates reflected light into a diffuse reflection component and a specular reflection component using the reflection coefficient calculation unit, and calculates the diffuse reflection coefficient of the subject surface using the diffuse reflection component, the amount of light incident on the subject, and the shape information of the subject acquired by the subject shape acquisition unit and the lighting position, and also calculates the specular reflection coefficient of the subject surface using the specular reflection component, the amount of light incident on the subject, the roughness of the subject surface calculated by the unevenness calculation unit, and the camera position.
This allows the texture acquisition device of the texture acquisition system to separate the reflected light into a diffuse reflection component and a specular reflection component, and then calculate the diffuse reflection coefficient and the specular reflection coefficient of the subject surface.

In addition, the texture acquisition device can acquire shape information of the subject's surface as subject's surface shape information by irradiating the subject with polarized light of multiple colors from multiple different light source directions via the measurement control unit, and estimating normals based on the brightness of the diffuse reflection components of the subject's surface photographed by the polarization imaging device.
As a result, the texture acquisition device of the texture acquisition system can estimate the normal of the subject's surface using only the diffuse reflection components excluding the specular reflection components, thereby making it possible to more accurately acquire detailed shape information of the subject's surface.

In addition, the texture acquisition device can output information including the subject's surface roughness, diffuse reflection coefficient, specular reflection coefficient, and subject's surface shape information as three-dimensional model detailed information via the three-dimensional model detailed information output unit in accordance with the time when the polarized imaging device captured the image in order to calculate the information including the subject's surface roughness, diffuse reflection coefficient, specular reflection coefficient, and subject's surface shape information .
This allows the texture acquisition device of the texture acquisition system to output more detailed information of the three-dimensional model to, for example, a CG drawing device in real time.

In this way, the texture acquisition system of the present invention makes it possible to acquire in real time accurate diffuse reflection coefficients and specular reflection coefficients of the subject's surface, the state of the subject's surface unevenness (roughness), and detailed shape information of the subject's surface (subject's surface shape information) in order to express the subject's texture, which cannot be acquired using conventional volumetric capture techniques.

The present invention provides a texture acquisition system, texture acquisition device, and texture acquisition program that can acquire the fine unevenness of a subject's surface in real time as a three-dimensional model to express the subject's texture.

1 is a functional block diagram showing an overall configuration of a texture acquisition system including a texture acquisition device according to an embodiment of the present invention. 4 is an explanatory diagram relating to separation of a diffuse reflection component and a specular reflection component of reflected light in the present embodiment. FIG. FIG. 13 is an explanatory diagram showing an example of a photography booth in this embodiment configured as a regular hexahedron. 2 is an explanatory diagram showing an RGB Bayer array of a camera imaging plate of the polarization imaging device according to the present embodiment. FIG. 1 is an explanatory diagram showing the relationship between roughness, which indicates the uneven state of the subject surface, and the degree of blurring of the specular reflection component; 1 is an explanatory diagram showing an example in which coded illumination is irradiated onto an object in the polarized illumination device according to the present embodiment. FIG. 1 is an explanatory diagram showing the relationship between roughness, which indicates the uneven state of the subject surface, and the dispersion of RGB color vectors due to coded illumination; FIG. 10 is an explanatory diagram showing the distribution of color vectors of specularly reflected light in RGB space. 13 is a graph showing roughness σ calculated based on principal component analysis for <A> a subject with a smooth surface, <B> a subject with a surface having low roughness, and <C> a subject with a surface having high roughness. 4 is an explanatory diagram relating to separation of a diffuse reflection component and a specular reflection component of reflected light in the present embodiment. FIG. FIG. 2 is an explanatory diagram of the diffuse reflection component of a reflection model based on Lambert's cosine law. FIG. 2 is an explanatory diagram of specular reflection based on the Phong reflection model. FIG. 11 is an explanatory diagram showing an example of a dynamic pattern of coded illumination according to the present embodiment. FIG. 13 is an explanatory diagram showing another example of a dynamic pattern of coded illumination according to the embodiment. FIG. 1 is an explanatory diagram showing an example of a photography booth in this embodiment configured in a geodesic dome shape. FIG. 13 is an explanatory diagram showing an example of a photography booth in this embodiment configured as a dodecahedron. 4 is a flowchart showing a processing flow of the texture acquisition device according to the present embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, modes for carrying out the present invention (hereinafter, referred to as "embodiments") will be described with reference to the drawings.
<Configuration of the texture acquisition system>
First, with reference to FIG. 1, an overview of a texture acquisition system 1000 including a texture acquisition device 1 will be described, and then each component will be described.

As shown in FIG. 1, the texture acquisition system 1000 includes a depth measurement device 20, a polarized lighting device 30, and a polarized photography device 40 that are arranged in a photography booth BH where a subject is photographed, and a texture acquisition device 1 that is communicatively connected to the depth measurement device 20, the polarized lighting device 30, and the polarized photography device 40.

The texture acquisition device 1 according to this embodiment is useful for measuring subject information (three-dimensional model) for use in AR/VR and as a method for capturing subject information in three-dimensional televisions, and performs the following processes for acquiring the texture of a subject (hereinafter referred to as "texture acquisition process"): (1) separation process of diffuse reflection components and specular reflection components, (2) measurement process of the unevenness of the subject surface, and (3) calculation process of detailed shape information of the subject surface using photometric stereo.

(1) In the process of separating the diffuse reflection component and the specular reflection component, the texture acquisition device 1, as shown in Fig. 2, irradiates the subject S with polarized light (illumination light) by polarized illumination (e.g., red illumination, green illumination, and blue illumination) from the polarized illumination device 30. Then, the reflected light of the irradiated subject S is photographed by a polarized imaging device 40 that selectively transmits the light in the polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees (see Figs. 4 and 10 described later). The photographed image P is an image that combines the specular reflection component that is reflected as a highlight as if a light source is reflected, and the diffuse reflection component that is the incident light that penetrates the inside of the subject and diffuses it by absorbing the color components of the object. The texture acquisition device 1 utilizes the difference in the reflection behavior of the polarized light on the surface of the subject to separate the diffuse reflection component _Id and the specular reflection component _Is , thereby acquiring the diffuse reflection coefficient and the specular reflection coefficient (details will be described later).

(2) In the process of measuring the unevenness of the subject's surface, polarized lighting (e.g., three colors: RGB (red, green, and blue)) is irradiated onto the subject. In the case of specular reflection, for example, on the surface of a dielectric subject, only images are captured where the angle of incidence = angle of reflection. However, if the subject surface is uneven, the light vector of the reflected light spreads due to the unevenness, and as a result, the unevenness is observed as a mixed color. In this embodiment, the roughness σ, which indicates the unevenness of the subject, is estimated using a principal component analysis method from the variance state of the color vectors near the pixel of interest (details will be described later).

(3) In the process of calculating detailed shape information of the subject surface using photometric stereo, the lighting in the photography booth BH is dynamically changed according to the size of the subject and the scale of the unevenness of the surface to be measured. Photometric stereo is a method of measuring the shape of the subject by estimating the normal based on the luminance of the subject surface photographed from multiple different light source directions to determine the inclination of the surface.
The texture acquisition device 1 (1) uses the diffuse reflection component _Id and the diffuse reflection coefficient calculated in the process of separating the diffuse reflection component and the specular reflection component to dynamically change the lighting in the photography booth BH while photographing, thereby making it possible to acquire a more detailed shape of the subject's surface than with conventional methods.

In this way, the texture acquisition device 1 is able to acquire accurate diffuse reflection coefficients and specular reflection coefficients on the subject's surface, the uneven state of the subject's surface (roughness σ), and detailed shape information of the subject's surface based on photometric stereo, which cannot be acquired by methods using conventional volumetric capture technology.The texture acquisition device 1 then outputs detailed information of the three-dimensional model acquired in this way ("detailed three-dimensional model information" described below) to a CG drawing device (not shown) or the like.
Hereinafter, the texture acquisition system 1000 including the texture acquisition device 1 according to this embodiment will be described in detail.

<Configuration of texture acquisition device>
The texture acquisition device 1 according to this embodiment is configured by a general computer, and includes a control unit 10, an input/output unit 11, and a storage unit 12, as shown in FIG.
The input/output unit 11 includes a communication interface for transmitting and receiving information, and an input/output interface for transmitting and receiving information to and from an input device such as a keyboard and an output device such as a monitor.

The storage unit 12 is configured with a flash memory, a hard disk, a RAM (Random Access Memory), etc. The storage unit 12 stores initial setting information 200 (details of which will be described later) for the texture acquisition process by the texture acquisition device 1.
The storage unit 12 also temporarily stores a program (texture acquisition program) for executing each function of the control unit 10 and information required for processing by the control unit 10.

The control unit 10 is responsible for overall control of the texture acquisition device 1, and as shown in FIG. 1, is composed of an initial setting information acquisition unit 110, a subject shape acquisition unit 120, a measurement control unit 130, a texture calculation unit 140, and a three-dimensional model detailed information output unit 150.
The control unit 10 is realized, for example, by a CPU (Central Processing Unit) (not shown) expanding a program (texture acquisition program) stored in the storage unit 12 into a RAM and executing it.

When the texture acquisition device 1 executes the texture acquisition process, the initial setting information acquisition unit 110 acquires information including the lighting positions, light distribution characteristics, and camera postures of the devices installed in the photography booth BH in advance as initial setting information 200, and stores the information in the storage unit 12. An example of how the initial setting information acquisition unit 110 acquires the initial setting information 200 will be described with reference to FIG.
Fig. 3 shows an example in which the photography booth BH is configured as a regular hexahedron. The depth measurement device 20, the polarized illumination device 30, and the polarized photography device 40 are arranged to surround the subject. As shown in Fig. 3, for example, the depth measurement device (depth camera) 20 and the polarized photography device 40 (e.g., a polarized RGB camera) are arranged at the vertices and the center of the sides of the regular hexahedron to photograph the subject placed inside the photography booth BH. The polarized illumination device 30 is arranged on each face of the regular hexahedron. Here, it is assumed that the photography booth BH is configured with a flat panel display (surface light source) and a linear polarizing filter is arranged.
Note that a regular hexahedron is just one example and is not limited to this (see FIGS. 15 and 16 described later).

The initial setting information acquisition unit 110 acquires information on the lighting position where the polarized lighting device 30 is placed, the light distribution characteristics measuring the luminous intensity in each direction in space, and the camera positions (X, Y, Z coordinates) and camera attitudes (roll, pitch, yaw) where the depth measurement device 20 and the polarized shooting device 40 are placed from an external device, etc., as initial setting information 200, and stores it in the memory unit 12.
It is assumed that the pair of depth measurement device 20 and polarized illumination device 30 arranged at one location in the photography booth BH has the same camera position and the same camera attitude.

1 , the subject shape acquisition unit 120 transmits instruction information for causing each depth measurement device 20 arranged in the photography booth BH to perform measurement via the measurement control unit 130. Then, the subject shape acquisition unit 120 receives measurement information from each depth measurement device 20 to acquire a (rough) subject shape.
The depth measurement device 20 may be, for example, a Time of Flight (ToF) camera that emits near-infrared light, reflects off an object, and measures the time it takes for the light to return to calculate depth.

The measurement control unit 130 cooperates with the subject shape acquisition unit 120 and the texture calculation unit 140 to control each device by sending instruction information to the depth measurement device 20, polarized lighting device 30, and polarized photography device 40 arranged in the photography booth BH.
Specifically, the measurement control unit 130 transmits instruction information to the depth measurement device 20 to cause it to measure the subject shape.

In addition, when the texture calculation unit 140 executes the texture acquisition process, the measurement control unit 130 transmits instruction information to the polarized lighting device 30 to control the device to achieve a predetermined lighting pattern as shown in Figures 13 and 14 described below.
The predetermined pattern shown in Fig. 13 dynamically changes the lighting conditions according to the size of the object and the scale of the unevenness of the surface to be measured, and here, RGB (red, green, blue) lighting is divided into sections of a predetermined size and coded for illumination. Note that the hatched lines in each color section indicate the polarization direction of the linear polarizing filter, and here indicate that polarized light with polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees is irradiated.
The measurement control unit 130 transmits instruction information for controlling the dynamic pattern of this coded illumination to the polarized illumination device 30 .

In addition, the measurement control unit 130 transmits instruction information to the polarized photographing device 40 to photograph the reflected light in response to the texture calculation unit 140 executing the texture acquisition process and the polarized lighting device 30 irradiating the subject with illumination light.

The camera imaging plate of the polarization imaging device 40 (herein referred to as a "polarized RGB camera") has an RGB Bayer array, for example as shown in FIG. 4. Furthermore, each RGB pixel is composed of four divided pixels, and each pixel selectively transmits components with polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees. Note that in FIG. 4, the direction of the hatched area indicated by diagonal lines indicates the direction of the polarized light components that are transmitted.

As part of the texture acquisition process, the texture calculation unit 140 performs the above-mentioned separation process of diffuse reflection components and specular reflection components, measurement process of the unevenness of the subject's surface, and calculation process of detailed shape information of the subject's surface using photometric stereo. This texture calculation unit 140 includes a roughness calculation unit 141, a reflection coefficient calculation unit 142, and a photometric stereo shape acquisition unit (surface shape acquisition unit) 143.

The unevenness calculating section 141 calculates roughness σ, which indicates the unevenness of the object surface, from the variance of the color vector of the deflected light caused by the unevenness of the object surface.
The unevenness of a subject can be expressed by various parameters such as the fineness of the unevenness, but it can also be treated as the degree to which reflected objects such as light sources are blurred on the subject's surface.
5, when RGB illumination indicated by reference numeral 51 is irradiated onto a spherical subject 52 and the specular reflection component is photographed, the degree to which each of the RGB color components is blurred increases as the surface of the subject 52 becomes rougher, as indicated by reference numeral 53. In other words, the greater the variance of the color vectors, the greater the roughness of the subject surface.

In this embodiment, an example will be described in which polarized illumination is used as a surface light source, and the three colors RGB are used to code on the wall of the photography booth, and this is irradiated onto the subject as illumination (coded illumination γ _l ) (see FIG. 6).
6 shows an example in which the three colors of RGB are irradiated onto the subject as differently polarized light (γ _l1 , γ _l2 , γ _l3 ) at angles of 0 degrees, 45 degrees, and 90 degrees, respectively. This is to avoid cases in which the incident angle α becomes Brewster's angle and measurement becomes impossible in a single direction. Note that the polarized light is not limited to 0 degrees, 45 degrees, and 90 degrees.
Furthermore, the present invention is not limited to using polarized illumination in three colors, RGB, but may use polarized light in a plurality of colors, and may also be applicable to five-color or polychromatic light.

When focusing on a certain part of the subject's surface, if the surface is flat like a mirror, only those parts for which the angle of incidence = the angle of reflection are captured by the camera (polarized RGB camera 40a) (see reference number 71 in FIG. 7). However, if the subject's surface is uneven, the degree to which RGB is observed as a mixture of colors at the focused pixel position increases depending on the degree of roughness. In other words, the greater the degree of roughness of a part of the subject's surface (microsurface structure), the greater the variance of the color vectors (see reference numbers 72 and 73 in FIG. 7). In this embodiment, the relationship between the roughness of the subject's surface and the variance of the color vectors is used to estimate the roughness of the subject's surface from the variance of the color vectors.

Specifically, the unevenness calculation section 141 estimates the roughness σ, which indicates the unevenness of the subject, from the variance state of color vectors in the vicinity of a pixel of interest by using a principal component analysis technique.
Fig. 8 shows the distribution of color vectors of specular reflected light in RGB space. The specular reflected light around a part of a subject of interest was photographed with a polarized RGB camera 40a (see Fig. 6), and the specular reflected light was plotted in RGB space. Fig. 8(a) shows the distribution of color vectors when the subject has a smooth surface. Fig. 8(b) shows the distribution of color vectors when the subject has a rough surface.

Then, the unevenness calculation unit 141 applies principal component analysis to the specular reflected light color vector in the RGB space to calculate the first principal component contribution rate and the third principal component contribution rate. The unevenness calculation unit 141 uses the calculated first principal component contribution rate and third principal component contribution rate to calculate the roughness σ according to formula (1) shown in FIG. 8.

Here, according to formula (1), the closer the values of the first principal component contribution rate and the third principal component contribution rate are, the closer the roughness σ is to "1". In other words, the greater the variance of the color vector of the specular reflected light, the greater the variation in the color vector, the smaller the difference between the values of the first principal component contribution rate and the third principal component contribution rate, and the closer the roughness σ is to "1". In this way, the state of roughness σ can be quantified using the first principal component contribution rate and the third principal component contribution rate obtained by principal component analysis.

FIG. 9 is a graph showing the roughness σ calculated based on the principal component analysis for <A> an object with a smooth surface, <B> an object with a surface having low roughness, and <C> an object with a surface having high roughness.
In this graph, the average roughness σ is 0.55 for the subject with a smooth surface (A), 0.82 for the subject with a small roughness (B), and 0.91 for the subject with a large roughness (C). Thus, the smaller the roughness σ value, the smoother the surface is, and the larger the value, the rougher the surface is. Note that here, principal component analysis is performed on a size of ±3 pixels around the pixel of interest.

In this way, the unevenness calculation unit 141 can calculate the roughness σ of the target portion of the subject's surface using the contribution rates (the first principal component contribution rate and the third principal component contribution rate) calculated by principal component analysis.

Returning to Fig. 1, the reflection coefficient calculation unit 142 separates the reflected light into a diffuse reflection component _Id and a specular reflection _component Is by utilizing the difference in the reflection behavior of the polarized light on the surface of the subject. The reflection coefficient calculation unit 142 then calculates a diffuse reflection coefficient _Kd of the subject using the acquired diffuse reflection component _Id , the amount of light incident on the subject, shape information of the subject, and the illumination position. The reflection coefficient calculation unit 142 also calculates a specular reflection coefficient _Ks based on the acquired specular reflection _component Is, the amount of light incident on the subject, roughness σ, and the camera position. This will be described in detail below.

<Separation of diffuse reflection and specular reflection components>
First, with reference to FIG. 10, separation of the diffuse reflection component and the specular reflection component in the reflected light will be described.
Here, the polarized lighting device 30 is used as a surface light source, and the three colors RGB are used to code the light on the wall of the photography booth under the control of the measurement control unit 130, and the light is irradiated onto the subject as illumination. At this time, the subject is photographed by the polarized photography device 40 (polarized RGB camera 40a).
The camera imaging plate of this polarization RGB camera 40a has an RGB Bayer array as shown by reference numeral 101 in Fig. 10, and each RGB pixel is composed of a pixel divided into four, and each pixel selectively transmits components with polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees. Note that the pixel divided into four and the polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees are merely examples, and the present invention is not limited to these.

The observed luminance I is expressed by the following equation (2) based on the dichromatic reflection model. The dichromatic reflection model states that the spectrum of reflected light can be expressed as a linear sum of the spectra of the diffuse reflection component and the specular reflection component.

I=I _d +a(1+cos(2Θ-β))...Formula (2)
Here, _Id is the diffuse reflection component, a(1+cos(2Θ-β)) is the specular reflection component _Is , a is the amplitude, Θ is the polarization direction of the pixel, and β is the phase.

The luminance values obtained from pixels with polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees (represented by I ₀ , I ₄₅ , I ₉₀ , and I ₁₃₅ , respectively) behave as sine waves with respect to their average value I _ave , as shown in FIG. 10, but become sine waves at double the polarization angle. Therefore, using the trigonometric formula sin(θ-β) ² +cos(θ-β) ² =1 (β takes into account the phase), for example, I ₀ ² +I ₄₅ ² =a (a is the amplitude of the trigonometric function). This makes it possible to calculate the amplitude a. Furthermore, the minimum value I _min of the observed luminance I is the diffuse reflection component I _d , and 2a, which is twice the amplitude a, is the specular reflection component I _s .

Next, for the diffuse reflection component of reflected light, we will explain the process of calculating the diffuse reflection coefficient using the Lambert reflection model (reflection of a diffuse surface based on Lambert's cosine law) as an example. Also, for the specular reflection component, we will explain the process of calculating the specular reflection coefficient using the Phong reflection model as an example.

<Calculation of diffuse reflectance coefficient>
The calculation process of the diffuse reflection coefficient _Kd by the reflection coefficient calculation unit 142 will be described.
The reflection coefficient calculation unit 142 calculates the diffuse reflection coefficient _Kd based on Lambert's cosine law, which states that the intensity of reflected light is proportional to the cosine of the angle of incidence.

11, the diffuse reflection light of the incident light amount _I0 incident on the subject at an angle θ is directly proportional to the cosine of the angle θ between the incident light and the normal line N of the surface, based on Lambert's cosine law. The diffuse reflection coefficient _Kd at this time can be calculated by the following formula (3) _{if Id} calculated in the above-mentioned process of separating the diffuse reflection component and the specular reflection component, the normal line N calculated from the shape information of the subject, the light source position (illumination position), and the incident light amount _I0 are known.
I _d = K _d I ₀ cosθ...Formula (3)

<Calculation of specular reflection coefficient>
The calculation process of the specular reflection coefficient _Ks by the reflection coefficient calculation unit 142 will be described.
FIG. 12 is an explanatory diagram of specular reflection based on the Phong reflection model.
If the observation (photography) viewpoint is shifted by α from the direction in which the incident light (incident light amount _I0 ) is specularly reflected, the intensity I _s of the reflected light, which is the specular reflection component, can be calculated by the following formula (4). Note that the shift α of the observation (photography) viewpoint is calculated from the camera position.
I _s = K _s I ₀ cos ⁿ α ...Formula (4)

The exponent n of the trigonometric function cos in equation (4) is a value determined according to the roughness of the surface of the subject.
In this embodiment, the relationship between the exponent n and the roughness σ calculated as above is linked by the following methods. Here, we will explain [Method 1], which calculates the exponent n and the specular reflection coefficient _Ks from information obtained from a single viewpoint of a part of a subject of interest, and [Method 2], which calculates the exponent n and the specular reflection coefficient _Ks from information obtained from multiple viewpoints.

[Method 1]
The reflection coefficient calculation unit 142 uses the roughness σ (here, σ ₁ ) obtained from a single viewpoint of a part of the subject to observe subjects having various surface roughnesses in advance, and calculates the exponent n and the specular reflection coefficient _Ks , for example, by the following methods 1-1 to 1-3.

(Method 1-1) The reflection coefficient calculation unit 142 calculates the exponent n by using a lookup table created in advance.
α, σ ₁ , I _s , K _s and n are created in advance as a look-up table.
Then, from the roughness σ ₁ calculated by the unevenness calculation unit 141, α obtained by prior observation, and the specular reflection component I _s , a lookup table is referenced to extract the optimal exponent n and specular reflection coefficient K _s .

(Method 1-2) Parameter fitting to a three-dimensional or higher order equation.
In the above formula (4), α, I ₀ , and I _s are known from the measurement conditions and the measurement results. By using n obtained by the following formula (5) in this formula, for example, the specular reflection coefficient K _s is calculated.
n=aσ ₁ ² +bσ ₁ +c ... Formula (5)

(Method 1-3) Estimate K _s and n using a trained neural network.
A deep neural network (DNN) with inputs of α, σ ₁ , and I _s and outputs of K _s and n is used to learn weights from a training data set measured in advance. This training data set uses data acquired from various subjects, various light source positions (illumination positions), and camera measurement positions.
When the reflection coefficient calculation unit 142 actually finds K _s and n, α, σ ₁ and I _s are input to a trained DNN, which outputs K _s and n.

[Method 2]
The reflection coefficient calculation unit 142 uses roughness σ (here, σ ₁ and σ ₂ ) obtained from multiple viewpoints of a part of the subject to observe subjects with various surface roughnesses in advance, and calculates the exponent n and the specular reflection coefficient K _s , for example, by the following methods 2-1 and 2-2.

(Method 2-1) Estimate n using a trained neural network.
A nonlinear regression problem is considered, and a DNN with inputs α ₁ , α ₂ , σ ₁ , σ ₂ and output n is created, and weights are learned using a learning dataset measured in advance.
When the reflection coefficient calculation unit 142 actually calculates n, α ₁ , α ₂ , σ ₁ , and σ ₂ are input to the trained DNN, which outputs n. The output n is inserted into the above equation (4) to calculate the specular reflection coefficient K _s .

(Method 2-2) Estimate K _s and n using a trained neural network.
The inputs of a DNN are _α1 , _α2 , _σ1 , _σ2 , _Is1 , and _Is2 , and the outputs are _Ks and n. Weights are trained using a training data set measured in advance. This training data set uses data acquired from various subjects, various light source positions (illumination positions), and camera measurement positions.
When the reflection coefficient calculation unit 142 actually finds K _s ,n, α ₁ , _{α 2} , σ _{1 , σ 2} , I _s1 , and I _s2 are input to the trained DNN, which outputs K _s ,n.

In this manner, the reflection coefficient calculation section 142 can calculate the diffuse reflection coefficient K _d and the specular reflection coefficient K _s of the subject.

Dynamic lighting control
Here, the dynamic control of the polarized lighting device 30 that the texture calculation unit 140 executes via the measurement control unit 130 will be described.
In this embodiment, polarized light from the polarized lighting device 30 is used as a surface light source, and the subject is illuminated with coded light (coded light) using the three colors RGB (see Figs. 6 and 10). At this time, the measurement control unit 130 dynamically changes the lighting conditions according to the size of the subject and the scale of the unevenness of the surface to be measured. This satisfies the lighting conditions that allow the photometric stereo shape acquisition unit 143 to acquire the detailed shape of the subject surface.

With reference to FIG. 13, an example of a dynamic pattern of coded illumination is illustrated.
As shown in FIG. 13, the measurement control unit 130 changes the color, polarization direction, and scale pattern for the three colors RGB on a frame-by-frame basis to illuminate the subject.
In the example shown in Fig. 13, the scale is changed from small to medium to large every two frames out of a total of six frames. And, in the two frames, the combination of color and polarization direction changes every other frame. Note that "black" in Fig. 13 indicates a state where the illumination is not emitting light.
In this way, even when subjects of different sizes are targeted, or when the scale of the surface irregularities varies depending on the part of the subject, more accurate measurements are possible.
The dynamic pattern is not limited to changing the pattern with a predetermined regularity as shown in Fig. 13. As long as there is no anisotropic reflection, it is also possible to change the code randomly with respect to the three colors of RGB, the polarization direction, and the scale.

FIG. 14 is a diagram showing another example of a dynamic pattern of coded illumination according to the present embodiment.
In the example shown in Fig. 14, the surface light source of a regular hexahedral photography booth illuminates the three surfaces adjacent to the reference vertex with each of the RGB colors, and one sequence is made up of six frames. One set is made up of two frames, and each color moves to the opposite surface in the next frame. By changing the direction of the light source of the same color within this set, shape measurement is performed using photometric stereo, which will be described later, and detailed shape information of the subject's surface ("subject surface shape information", which will be described later) is obtained.

The photography booth according to this embodiment is not limited to a regular hexahedron, and may be, for example, a geodesic dome shape as shown in FIG. 15 or a dodecahedron as shown in FIG. 16. If the lighting is configured as a surface light source such as a flat panel display, each surface may be treated as a light source unit (see FIG. 15(a)). By adjusting the pattern presented as lighting, it is not necessary to control the emitted color on a surface-by-surface basis, and it is also possible to reflect each color arbitrarily on the surface of the subject (see FIG. 15(b)).

Returning to FIG. 1 , the photometric stereo shape acquisition unit (surface shape acquisition unit) 143 dynamically changes the coded illumination in accordance with the size of the subject and the scale of the surface irregularities, as described above, and acquires the information captured by the polarization imaging device (polarized RGB camera) 40.
In this case, the photometric stereo shape acquisition unit 143 obtains the inclination of the surface by estimating a normal based on the luminance of the object surface photographed from a plurality of different light source directions, and obtains detailed shape information of the object as object surface shape information. In this case, the photometric stereo shape acquisition unit 143 uses only the diffuse reflection component _Id , excluding the specular reflection component _Is , so that the photometric stereo shape acquisition unit 143 can obtain the inclination of the surface while more accurately reflecting the uneven state of the object surface.

The three-dimensional model detailed information output section 150 outputs information calculated by the texture calculation section 140, including the diffuse reflection coefficient K _d , the specular reflection coefficient K _s , the roughness σ indicating the uneven state of the subject's surface, and detailed shape information of the subject's surface (subject's surface shape information), as three-dimensional model detailed information to an external device such as a CG drawing device in accordance with the time of image capture by the polarization imaging device 40 to calculate each piece of information. By acquiring this three-dimensional model detailed information, the CG drawing device can represent the fine uneven state of the subject's surface as a texture in real time.

<Operation of the texture acquisition device>
Next, the operation performed by the texture acquisition device 1 according to the present embodiment will be described with reference to FIG.

First, the initial setting information acquisition unit 110 of the texture acquisition device 1 acquires information such as the camera position, camera attitude, lighting position, and light distribution characteristics of each device (depth measurement device 20, polarized lighting device 30, and polarized shooting device 40) installed in the shooting booth BH from an external device, etc., as initial setting information 200, and stores it in the memory unit 12 (step S1).

Next, the subject shape acquisition unit 120 of the texture acquisition device 1 transmits instruction information to the depth measurement device 20 arranged in the photography booth BH via the measurement control unit 130 to perform measurement. The subject shape acquisition unit 120 then receives measurement information from the depth measurement device 20 to acquire a (rough) subject shape (step S2).

Next, the texture calculation unit 140 (unevenness calculation unit 141) of the texture acquisition device 1 calculates the roughness σ that indicates the unevenness of the subject surface (step S3).
Here, the unevenness calculation unit 141 calculates the roughness σ from the variance of color vectors in the vicinity of a pixel of interest using a principal component analysis technique.
Specifically, as shown in Fig. 6, the unevenness calculation unit 141 causes the polarized lighting device 30 to irradiate RGB polarized light via the measurement control unit 130, and causes the polarized RGB camera 40a to capture the image. Then, the unevenness calculation unit 141 applies principal component analysis to the specular reflected light color vector in the RGB space, and calculates the first principal component contribution rate and the third principal component contribution rate. The unevenness calculation unit 141 calculates the roughness σ based on the formula (1) in Fig. 8 using the calculated first principal component contribution rate and third principal component contribution rate.

Next, the texture calculation unit (reflection coefficient calculation unit 142) of the texture acquisition device 1 separates the polarized light reflected from the subject surface into a diffuse reflection component _Id and a specular reflection component _Is (step S4).
Specifically, the reflection coefficient calculation unit 142 generates a sine wave from the luminance values obtained from pixels with polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees, as shown in Fig. 10. Then, the reflection coefficient calculation unit 142 determines the minimum value _Imin of the luminance I of the sine wave as the diffuse reflection component _Id , and determines 2a, which is twice the amplitude a, as the specular reflection component _Is .
At this time, the reflection coefficient calculation unit 142, via the measurement control unit 130, illuminates the subject with coded illumination from the polarized illumination device 30 by dynamically changing the color, polarization direction, and scale pattern for the three colors RGB on a frame-by-frame basis, for example, as shown in FIG. 13 .

Next, the reflection coefficient calculation unit 142 calculates the diffuse reflection coefficient K _d (step S5).
Here, the reflection coefficient calculation unit 142 inserts the observed angle θ, the amount of incident light I ₀ , and the diffuse reflection component I _d into the above-mentioned equation (3) based on Lambert's cosine law to calculate the diffuse reflection coefficient K _d .

Next, the reflection coefficient calculation unit 142 calculates the specular reflection coefficient _Ks (step S6).
The reflection coefficient calculation unit 142 calculates the relationship between the roughness σ and the exponent n in the above-mentioned equation (4) based on the Phong reflection model, using the above-mentioned [Method 1] (Methods 1-1 to 1-3) or [Method 2] (Method 2-1, Method 2-2).

The photometric stereo shape acquisition unit (surface shape acquisition unit) 143 dynamically changes the coded illumination of the polarized illumination device 30 via the measurement control unit 130, for example as shown in FIG. 14, and acquires information captured by the polarized imaging device (polarized RGB camera) 40. As a result, the photometric stereo shape acquisition unit 143 uses the photometric stereo technique to estimate the normal based on the luminance of the diffuse reflection components of the subject surface captured from multiple different light source directions, thereby determining the inclination of the surface, and acquires detailed shape information of the subject surface as subject surface shape information (step S7).

Next, the 3D model detailed information output unit 150 outputs the information generated by the texture calculation unit 140, including the diffuse reflection coefficient K _d , the specular reflection coefficient K _s , the roughness σ indicating the uneven state of the subject's surface, and detailed shape information of the subject's surface (subject's surface shape information), to an external device such as a CG drawing device as 3D model detailed information in accordance with the time when the polarization imaging device 40 captured the image in order to calculate each piece of information (step S8), and then the processing is completed.

In this way, the texture acquisition system 1000 including the texture acquisition device 1 according to this embodiment can acquire in real time the accurate diffuse reflection coefficient and specular reflection coefficient of the subject surface, the state of the unevenness of the subject surface (roughness σ), and detailed shape information of the subject surface based on photometric stereo (subject surface shape information) for the fine unevenness of the subject surface to express the texture of the subject.

REFERENCE SIGNS LIST 1 Material quality acquisition device 10 Control unit 11 Input/output unit 12 Memory unit 20 Depth measurement device 30 Polarized lighting device 40 Polarized photography device 40a Polarized RGB camera 110 Initial setting information acquisition unit 120 Subject shape acquisition unit 130 Measurement control unit 140 Material quality calculation unit 141 Roughness state calculation unit 142 Reflection coefficient calculation unit 143 Photometric stereo shape acquisition unit (surface shape acquisition unit)
150 3D model detailed information output unit 200 Initial setting information 1000 Texture acquisition system BH Photo booth

Claims (5)

a photography booth having a depth measurement device, a polarized lighting device, and a polarized photography device, the photography booth being arranged to surround a subject;
a texture acquisition device that is communicatively connected to the depth measurement device, the polarized illumination device, and the polarized imaging device, and that acquires a texture represented by a state of unevenness on the surface of the subject,
The texture acquisition device includes:
a storage unit for storing initial setting information including an illumination position of the polarized illumination device, and camera positions and camera orientations of the depth measurement device and the polarized imaging device;
a subject shape acquisition unit that acquires shape information of the subject by receiving depth information measured by the depth measurement device;
a measurement control unit that controls the polarized illumination device to irradiate the subject with polarized light of a plurality of colors;
an unevenness calculation unit that acquires information obtained by capturing, by the polarization imaging device, reflected light from the object irradiated with the polarized light of the plurality of colors, and calculates the roughness of the object surface from the variance of the color vector of the polarized light generated by the unevenness of the object surface;
a reflection coefficient calculation unit that separates the reflected light into a diffuse reflection component and a specular reflection component, calculates a diffuse reflection coefficient of the subject surface using the diffuse reflection component, the amount of light incident on the subject, the shape information of the subject acquired by the subject shape acquisition unit, and the illumination position, and calculates a specular reflection coefficient of the subject surface using the specular reflection component, the amount of light incident on the subject, the roughness of the subject surface calculated by the unevenness calculation unit, and the camera position;
a surface shape acquisition unit that irradiates the subject with the polarized light of the multiple colors from multiple different light source directions via the measurement control unit, and acquires shape information of the subject surface as subject surface shape information by estimating a normal based on the luminance of a diffuse reflection component of the subject surface photographed by the polarization imaging device;
a three-dimensional model detailed information output unit that outputs information including the roughness of the object surface, the diffuse reflection coefficient, the specular reflection coefficient, and the object surface shape information as three-dimensional model detailed information in accordance with the time when the polarization imaging device captures the image in order to calculate information including the roughness of the object surface, the diffuse reflection coefficient, the specular reflection coefficient, and the object surface shape information;
A texture acquisition system comprising: A texture acquisition device that is communicatively connected to a depth measurement device, a polarized lighting device, and a polarized photography device that are arranged to surround a subject in a photography booth, and that acquires a texture represented by a state of unevenness on the surface of the subject,
a storage unit for storing initial setting information including an illumination position of the polarized illumination device, and camera positions and camera orientations of the depth measurement device and the polarized imaging device;
a subject shape acquisition unit that acquires shape information of the subject by receiving depth information measured by the depth measurement device;
a measurement control unit that controls the polarized illumination device to irradiate the subject with polarized light of a plurality of colors;
an unevenness calculation unit that acquires information obtained by capturing, by the polarization imaging device, reflected light from the object irradiated with the polarized light of the plurality of colors, and calculates the roughness of the object surface from the variance of the color vector of the polarized light generated by the unevenness of the object surface;
a reflection coefficient calculation unit that separates the reflected light into a diffuse reflection component and a specular reflection component, calculates a diffuse reflection coefficient of the subject surface using the diffuse reflection component, the amount of light incident on the subject, the shape information of the subject acquired by the subject shape acquisition unit, and the illumination position, and calculates a specular reflection coefficient of the subject surface using the specular reflection component, the amount of light incident on the subject, the roughness of the subject surface calculated by the unevenness calculation unit, and the camera position;
a surface shape acquisition unit that irradiates the subject with the polarized light of the multiple colors from multiple different light source directions via the measurement control unit, and acquires shape information of the subject surface as subject surface shape information by estimating a normal based on the luminance of a diffuse reflection component of the subject surface photographed by the polarization imaging device;
a three-dimensional model detailed information output unit that outputs information including the roughness of the object surface, the diffuse reflection coefficient, the specular reflection coefficient, and the object surface shape information as three-dimensional model detailed information in accordance with the time when the polarization imaging device captures the image in order to calculate information including the roughness of the object surface, the diffuse reflection coefficient, the specular reflection coefficient, and the object surface shape information;
A texture acquisition device comprising: The unevenness state calculation unit
3. The texture acquisition device according to claim 2, further comprising: a principal component analysis being performed on the color vector of the polarized light; and a roughness of the surface of the object being calculated using a first principal component contribution rate and a third principal component contribution rate. The measurement control unit is
4. The texture acquisition device according to claim 2 or 3, wherein the polarized lighting device irradiates the subject with the polarized light of the plurality of colors, the polarized light being a surface light source, and the color, polarization direction, and scale of the polarized light being changed on a frame-by-frame basis. A texture acquisition program for causing a computer to function as the texture acquisition device according to any one of claims 2 to 4.

Images