JP7587191B2 - Shape estimation device, shape estimation method, and program - Google Patents

Description

The present invention relates to a shape estimation device, a shape estimation method, and a program for estimating the three-dimensional shape of a transparent refractive layer such as a water surface or air fluctuation from an image.

Technology that estimates the three-dimensional shape of objects in images is particularly important in fields such as robot vision, augmented reality, and autonomous driving.

In typical 3D shape estimation, multiple cameras are used and the 3D shape is estimated from the difference in appearance due to the difference in the camera's installation position. This conventional method assumes that the estimation target is an opaque, diffuse reflecting surface. Therefore, it is not possible to estimate the shape of a transparent refracting surface such as the surface of water.

To estimate the 3D shape of such a refractive surface, it is necessary to model the propagation of light from the camera to the refractive surface to the object behind the refractive surface. However, the 3D shapes of both the refractive surface and the object behind it are generally unknown. In addition, because the refraction model is complex, it is difficult to formulate it as a clear optimization problem.

Non-Patent Document 1 theoretically and experimentally discloses that at least two cameras are required to estimate the three-dimensional shape of a scene where one refraction occurs. Non-Patent Document 2 discloses that it is possible to estimate the three-dimensional shape of a refracting surface with one camera by considering the difference in how the background appears depending on whether or not refraction occurs. Non-Patent Document 3 discloses a method for estimating the three-dimensional shape of a transparent surface using polarization information.

[Retrieved June 4, 2021], Internet <URL: https://www.cs.toronto.edu/~kyros/pubs/08.ijcv.indirect.pdf> [Retrieved June 4, 2021], Internet <URL: http://grail.cs.washington.edu/projects/glass-recon/cvpr12_refraction.pdf> [Retrieved June 4, 2021], Internet: <URL: http://www.info.hiroshima-cu.ac.jp/~miyazaki/publication/paper/Miyazaki-PAMI2007.pdf>

However, the method of Non-Patent Document 1 requires at least two cameras, and requires alignment and time synchronization between the cameras. In addition, in Non-Patent Document 2, the model formulated as an optimization problem is complex, and shape estimation takes a long time and requires high calculation costs. In addition, in Non-Patent Document 3, it is necessary to know the rough three-dimensional shape of the estimation target in advance.

As such, conventional techniques have the drawback of requiring multiple cameras, high computational costs, and prior knowledge of the 3D shape, and there is no suitable technology for estimating 3D shape.

The present invention has been made in consideration of this problem, and aims to provide a shape estimation device, a shape estimation method, and a program capable of estimating the three-dimensional shape of a transparent refractive surface using a single camera, with a low computational effort, and without prior training data.

A shape estimation device according to one embodiment of the present invention comprises a polarization camera that captures a first image of a subject when no transparent refractive layer is present and a second image of the subject when the refractive layer is present; a feature acquisition unit that applies a feature point tracking method between the first and second images to acquire a distortion vector, which is a feature representing a change in geometric appearance due to refraction; a first estimation unit that acquires sets of brightness values corresponding to at least three different polarization angles from the second image to estimate polarization features; a second estimation unit that estimates two candidates for the normal vector of the refractive layer to be estimated using the polarization features; and a refractive surface generation unit that uses the distortion vector to select one of the candidates for the normal vector and generate three-dimensional shape information of the refractive surface representing the surface shape of the refractive layer.

In addition, a method for estimating the three-dimensional shape of a refraction surface according to one aspect of the present invention is a method for estimating the three-dimensional shape of a refraction surface performed by the above-mentioned three-dimensional shape estimation device of a refraction surface, in which a polarized camera captures a first image which is an image of the subject when a transparent refraction layer is not present and a second image which is an image of the subject when the refraction layer is present, a feature acquisition unit applies a feature point tracking method between the first image and the second image to acquire a distortion vector which is a feature representing a geometric change in appearance due to refraction, a first estimation unit acquires at least three different pairs of angles and brightness values from the second image to estimate polarization features, a second estimation unit estimates two candidates for the normal vector of the refraction layer to be estimated using the polarization features, and a refraction surface generation unit selects one of the candidates for the normal vector using the distortion vector, and generates three-dimensional shape information of the refraction surface representing the surface shape of the refraction layer.

In addition, one aspect of the present invention relates to a program for causing a computer to function as the above-mentioned refraction surface three-dimensional shape estimation device.

Our invention enables estimation of the 3D shape of transparent refractive surfaces using a single camera, low computational complexity, and without prior training data.

1 is a block diagram showing an example of a functional configuration of a shape estimation device according to an embodiment of the present invention. 2 is a diagram illustrating a schematic diagram of the relationship between the shape estimation device shown in FIG. 1, a subject, and a transparent refractive layer. 1A and 1B are diagrams showing a model of a change in polarization, in which FIG. 1A shows a normal vector and light reflected by a refractive layer, and FIG. 1B shows a relationship between a zenith angle and a degree of polarization. FIG. 2 is a diagram illustrating a relationship between an azimuth angle and image brightness. FIG. 1 is a schematic diagram showing modeling of geometric changes. 10A and 10B are diagrams illustrating examples of three-dimensional shape information of a refracting surface. 2 is a flowchart showing a processing procedure of a shape estimation method performed by the shape estimation device shown in FIG. 1 . FIG. 1 is a block diagram showing an example of the configuration of a general-purpose computer system.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The same reference symbols are used for the same parts in the drawings, and the description will not be repeated.

Figure 1 is a block diagram showing an example of the functional configuration of a shape estimation device according to an embodiment of the present invention. The shape estimation device 100 shown in Figure 1 is a device that estimates the three-dimensional shape of a transparent refractive layer that is interposed between the object and the object.

The shape estimation device 100 includes a polarization camera 10, a feature acquisition unit 20, a first estimation unit 30, a second estimation unit 40, and a refraction surface generation unit 50. Each functional component except for the polarization camera 10 can be realized by a computer including, for example, a ROM, a RAM, a CPU, etc. In this case, the processing contents are described by a program.

The polarization camera 10 is a typical polarization camera. The polarization camera 10 incorporates, for example, polarizers (polarizing filters) with four different polarization angles.

The polarized camera 10 captures a first image, which is an image of the subject when no transparent refractive layer is present, and a second image, which is an image of the subject when a refractive layer is present. Transparent refractive layers include the surface of water, a layer of fluctuating air, etc.

The feature acquisition unit 20 applies a feature point tracking method between the first and second images captured by the polarization camera 10 to acquire a distortion vector, which is a feature that represents a geometric change in appearance due to refraction. The feature point tracking method is, for example, optical flow.

The first estimation unit 30 estimates the polarization feature by acquiring a set of luminance values corresponding to at least three different polarization angles from the second image. The polarization feature is the degree of polarization. This will be described in detail later.

The second estimation unit 40 estimates two candidates for the normal vector of the refractive layer to be estimated using the polarization feature. In other words, the candidates for the normal vector of the change in polarization are narrowed down to two.

The refraction surface generation unit 50 uses the distortion vector to select one of the normal vector candidates and generates refraction surface three-dimensional shape information representing the surface shape of the refraction layer. The normal vector is a vector that is orthogonal to the tangent of the pixel of the second image. The refraction surface three-dimensional shape information representing the surface shape of the refraction layer can be generated from the normal vector of the pixel in which the refraction layer of the second image is reflected.

Figure 2 is a schematic diagram showing the relationship between the shape estimation device 100, the subject (background), and a transparent refractive layer (the refractive surface to be estimated). Figure 2 shows only the image sensor and the polarizing filter that make up the polarization camera 10.

The rectangular image sensor shown in Figure 2 is, for example, a CMOS image sensor with millions of pixels. Pixel i on the image plane receives light that has passed through a refractive layer from the background and is reflected by a polarizing filter.

A polarizing filter is placed in front of the image sensor with four different polarization angles. Four polarization angles is common.

The shape estimation device 100 estimates the azimuth angle φ and elevation angle θ of a normal vector n ^{i ,} which is a vector orthogonal to the tangent of pixel i, for each pixel i when the direction of incident light is the optical axis Z _c , to estimate the surface shape of the refractive layer. Hereinafter, each functional component of the shape estimation device 100 will be described.

When the image plane and the plane of the polarizing filter are parallel, the azimuth angle φ of the normal vector n ⁱ and the polarization angle have the same geometric meaning, so in the following description, the azimuth angle and the polarization angle will be described as the same variable φ.

(First Estimation Unit)
The first estimating unit 30 obtains sets of luminance values corresponding to at least three different polarization angles from a second image of the subject when a refractive layer is present, and estimates polarization features.

The following relationship holds between the polarization angle and the brightness value:

Here, the polarization feature amounts are I _max , I _min , and Ψ. The polarization feature amounts can be estimated from three or more different pairs of polarization angles and luminance values.

Using the polarization feature quantities I _max , I _min , and Ψ, the Stokes vector s representing the polarization state can be expressed by the following equation.

The change in the Stokes vector s can be expressed by multiplying the Stokes vector before the change by the Mueller matrix M. In other words, when the Stokes vector before the change is s _in and the Stokes vector after the change is s _out , s _out =M·s _in .

Here, Ts is the Fresnel transmission coefficient (component horizontal to the plane of incidence), and Tt is the Fresnel transmission coefficient (component perpendicular to the plane of incidence).

The degree of polarization ρ (Degree of Polarization), which represents the degree of change in polarization, can be expressed by the following equation.

In particular, when s _in is unpolarized, it can be expressed by the following equation.

Although equation (7) appears complicated at first glance, it is a monotonically increasing function and can be formulated as a convex optimization problem. Therefore, the elevation angle θ of the normal vector n ⁱ can be uniquely estimated from the observed value of the degree of polarization.

FIG. 3 is a diagram showing a model of a change in polarization, in which (a) shows a normal vector n ⁱ and light reflected by a refractive layer, and (b) shows the relationship between the zenith angle and the degree of polarization.

In FIG. 3A, _sin represents light incident on the refractive surface (surface of the refractive layer), and s _out represents light captured by the polarization camera 10.

The horizontal axis of Figure 3 (b) represents the zenith angle, and the vertical axis represents the degree of polarization. As shown in Figure 3 (b), if the elevation angle θ is known, the degree of polarization ρ can be uniquely determined.

(Second Estimation Unit)
The second estimation unit 40 estimates two candidates for the normal vector n ⁱ of the refractive layer (refractive surface (surface of the refractive layer)) to be estimated using the polarization feature amount.

When the image plane and the plane of the polarizing filter are parallel, the azimuth angle φ of the normal vector n ⁱ coincides with the polarization angle Ψ (the polarization angle at which the luminance value is maximized). Note that when the polarizing filter is rotated once, there are two angles at which the luminance value is maximized, so there remains an ambiguity of 180°.

Figure 4 is a diagram showing a schematic diagram of the relationship between the azimuth angle φ and the brightness value. The horizontal axis of Figure 4 shows the azimuth angle φ, and the vertical axis shows the brightness I(φ) of the image. As shown in Figure 4, the brightness value I(φ) has two maximum values.

Therefore, the candidates for the normal vector n ⁱ are narrowed down to the following two:

(Feature acquisition unit)
A feature point tracking technique is applied between a first image of a subject when no transparent refractive layer is present and a second image of the subject when a refractive layer is present to obtain a distortion vector, which is a feature that represents the geometric change in appearance due to refraction.

Fig. 5 is a diagram showing a model of the geometric change. _vf in Fig. 5 represents the light space on the side of the polarization camera 10. μ is the relative refractive index. Also, _vr is the direction vector of the refracted light.

The directional vector _vr of the refracted light can be expressed by the following equation.

The distortion vector _Δg, which is a feature quantity representing a geometric change in appearance due to refraction, can be expressed by the following equation.

(Bending surface generation unit)
The refraction surface generation unit 50 selects one of the candidates n ⁱ ₊ , n ⁱ ₋ for the normal vector n ⁱ using the distortion vector Δ _g , and generates three-dimensional shape information of the refraction surface that represents the surface shape of the refraction layer.

The refracting surface generating unit 50 generates 3D shape information of the refracting surface by finding the optimum solution of the following equation from candidates n ⁱ ₊ , n ⁱ ₋ of the normal vector n ⁱ narrowed down based on the constraints of polarization.

The calculation to find the optimal solution of equation (12) is performed for all pixels i.

Figure 6 is a diagram showing a schematic example of refracting surface three-dimensional shape information. Three-dimensional shape information can be generated as shown in Figure 6.

(Shape Estimation Method)
FIG. 7 is a flowchart showing the processing steps of the shape estimation method performed by the shape estimation device 100.

First, the polarized camera 10 captures a first image of the subject when no transparent refractive layer is present, and a second image of the subject when a refractive layer is present (step S1).

Next, the first estimating unit 30 obtains sets of luminance values corresponding to at least three different polarization angles from the second image, and estimates polarization feature amounts I _max , I _min , and Ψ (step S2).

Next, the second estimating unit 40 estimates two candidates (n ⁱ ₊ , n ⁱ ₋ ) for the normal vector n ⁱ of the refractive layer to be estimated using the polarization feature amount (step S3).

Next, the feature amount acquiring unit 20 acquires a distortion vector _Δg , which is a feature amount representing a geometric change in appearance due to refraction, by applying a feature point tracking method between the first image and the second image (step S4).

Next, the refraction surface generator 50 selects one of the candidates n ⁱ ₊ , n ⁱ _- for the normal vector n ⁱ using the distortion vector Δ _g , and generates three-dimensional shape information of the refraction surface that represents the surface shape of the refraction layer (step S5). The processes of steps S2 to S5 are repeated until they are completed for all pixels i (NO in step S6).

The processes in steps S2 to S5 are easily parallelized because they are performed independently for each pixel. Parallelization enables faster estimation of 3D shapes.

As described above, the shape estimation device 100 according to this embodiment includes a polarization camera 10 that captures a first image of a subject when a transparent refractive layer is not present and a second image of the subject when a refractive layer is present, a feature acquisition unit 20 that applies a feature point tracking method between the first and second images to acquire a distortion vector Δ _g, which is a feature representing a geometric change in appearance due to refraction, a first estimation unit 30 that acquires sets of luminance values corresponding to at least three different polarization angles from the second image and estimates polarization feature amounts I _max , I _min , Ψ, a second estimation unit 40 that estimates two candidates (n ⁱ ₊ , n ⁱ _- ) for the normal vector n ⁱ of the refractive layer to be estimated using the polarization feature amounts, and a refraction surface generation unit 50 that uses the distortion vector Δ _g to select one of the candidates n ⁱ ₊ , n ⁱ _- for the normal vector n ⁱ and generate three-dimensional shape information of the refraction surface representing the surface shape of the refraction layer.

Furthermore, the shape estimation method according to this embodiment is a shape estimation method performed by the shape estimation device 100, in which the polarization camera 10 captures a first image of the subject when no transparent refraction layer is present and a second image of the subject when a refraction layer is present, the feature acquisition unit 20 applies a feature point tracking technique between the first and second images to acquire a distortion vector Δ _g which is a feature representing a geometric change in appearance due to refraction, the first estimation unit 30 acquires sets of luminance values corresponding to at least three different polarization angles from the second image to estimate polarization feature amounts I _max , I _min , and Ψ, the second estimation unit 40 estimates two candidates (n ^{i + , n i − ) for the normal vector n i of the refraction layer to be estimated using the polarization feature amounts, and the refraction surface generation unit 50 selects one of the candidates n i} ₊ ^, _n ^{i −} _for ^the _normal vector n ⁱ using the distortion vector Δ _g to generate three-dimensional shape information of the refraction surface representing the surface shape of the refraction layer.

The shape estimation device 100 can be realized by a general-purpose computer system as shown in FIG. 8. For example, in a general-purpose computer system including a CPU 90, a memory 91, a storage 92, a communication unit 93, an input unit 94, and an output unit 95, the CPU 90 executes a predetermined program loaded onto the memory 91, thereby realizing each function of the shape estimation device 100. The predetermined program can be recorded on a computer-readable recording medium such as an HDD, SSD, USB memory, CD-ROM, DVD-ROM, or MO, or can be distributed via a network.

As described above, the shape estimation device 100 and shape estimation method according to this embodiment enable the estimation of the three-dimensional shape of a transparent refracting surface with a single camera, low computational complexity, and no prior knowledge of the shape (no prior learning data), which was not possible with the conventional approach. In other words, by constructing a model that takes into account not only the geometric changes in the light propagation path due to refraction, but also optical changes (polarization), the constraints necessary for the three-dimensional shape of the refracting surface shape can be obtained from information obtained with only a single camera. In addition, since it can be formulated as a convex optimization problem and the search range for the solution is narrowed, it becomes possible to estimate the three-dimensional shape with less computational complexity than with conventional technology.

The present invention includes various embodiments not described herein. Therefore, the technical scope of the present invention is determined only by the invention-specific matters related to the claims that are appropriate from the above description.

10: Polarization camera 20: Feature acquisition unit 30: First estimation unit 40: Second estimation unit 50: Refraction surface generation unit 100: Shape estimation device

Claims (3)

a polarized camera that captures a first image of a subject without a transparent refractive layer and a second image of the subject with the refractive layer;
a feature amount acquiring unit that acquires a distortion vector, which is a feature amount that represents a geometric change in appearance due to refraction, by applying a feature point tracking method between the first image and the second image;
a first estimation unit that estimates a polarization feature by acquiring sets of luminance values corresponding to at least three different polarization angles from the second image;
A second estimation unit that estimates two candidates of a normal vector of the refractive layer to be estimated using the feature amount of polarization;
a refraction surface generation unit that uses the distortion vector to select one of the candidates for the normal vector and generates three-dimensional shape information of the refraction surface that represents a surface shape of the refraction layer. A shape estimation method performed by a shape estimation device, comprising:
the polarized camera captures a first image of the object without a transparent refractive layer and a second image of the object with the refractive layer;
a feature amount acquiring unit that acquires a distortion vector, which is a feature amount representing a geometric change in appearance due to refraction, by applying a feature point tracking method between the first image and the second image;
a first estimation unit that estimates a polarization feature by acquiring sets of luminance values corresponding to at least three different polarization angles from the second image;
The second estimation unit estimates two candidates of a normal vector of the refractive layer to be estimated using the feature amount of polarization;
a refraction surface generating unit using the distortion vector to select one from the candidates for the normal vector, and generating three-dimensional shape information of the refraction surface representing a surface shape of the refraction layer. A program for causing a computer to function as the shape estimation device according to claim 1.

Images