JP7819902B2 - Information processing device and information generation method - Google Patents

Description

This disclosure relates to an information processing device and an information generation method.

Research is being conducted on technology that uses images of a target acquired from multiple positions and orientations to perform 3D reconstruction, and then generates an image of the target photographed from any position and orientation based on the reconstructed 3D data. Research is also being conducted on technology that uses a brightness field (neural network) to represent this 3D information, but as the target area becomes larger, issues arise: the memory required to store the brightness field information becomes enormous, and it becomes difficult to represent the 3D information.

B. Mildenhall, et. al., “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis”, 3 August 2020, arXiv, https://arxiv.org/abs/2003.08934v2 C. Reiser, et. al., “KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs”, 2 August 2021, arXiv, https://arxiv.org/abs/2103.13744v2 T. Takikawa, et. al., “Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes”, 26 January 2021, arXiv, https://arxiv.org/abs/2101.10994v1 A. R. Kosiorek, et. al., “NeRF-VAE: A Geometry Aware 3D Scene Generative Model”, 1 April 2021, arXiv, https://arxiv.org/abs/2104.00587v1 A. v. d. Oord, et. al., “Neural Discrete Representation Learning”, 30 May 2018, arXiv, https://arxiv.org/abs/1711.00937 A. Radford, et. al., “Improving language understanding by generative pre-training”, 2018, https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf P. Weng, et.al., “NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction”, 20 June 2021, arXiv, https://arxiv.org/abs/2106.10689v1

This disclosure provides an information processing device that improves the accuracy of three-dimensional information representation using a neural network model.

According to one embodiment, an information processing device for assigning discrete codes includes one or more memories and one or more processors. The one or more processors use an autoregressive model to generate information about multiple discrete codes that represent three-dimensional information.

According to one embodiment, an information processing device that generates an image from discrete codes generated and assigned by the above device or by a different device includes one or more memories and one or more processors. The one or more processors calculate a first feature amount at a first coordinate in three-dimensional space based on a plurality of discrete codes representing a first scene, and generate an image of the first scene based on output information obtained by inputting at least the information on the first coordinates and the first feature amount into a neural network, where the plurality of discrete codes are generated using an autoregressive model.

According to one embodiment, an information processing device that assigns the above-mentioned discrete codes or that trains an autoregressive model used in the information processing device that generates the above-mentioned image includes one or more memories and one or more processors. The one or more processors update the parameters of the autoregressive model that generates information about multiple discrete codes that represent three-dimensional information, using training data that includes information about the multiple discrete codes.

FIG. 2 illustrates an example hierarchy of voxels according to one embodiment. FIG. 2 illustrates an example of a voxel representation of a luminance field according to one embodiment. FIG. 2 is a diagram showing an example of voxels in the 0th layer according to an embodiment. FIG. 3 is a diagram showing an example of voxels in a first layer according to an embodiment. FIG. 3 is a diagram showing an example of voxels in a first layer according to an embodiment. FIG. 10 is a diagram showing an example of voxels in a second layer according to an embodiment. FIG. 10 is a diagram showing an example of voxels in a second layer according to an embodiment. FIG. 10 is a diagram for explaining an example of feature amount acquisition according to an embodiment. FIG. 10 is a diagram for explaining an example of feature amount acquisition according to an embodiment. FIG. 10 is a diagram for explaining an example of feature amount acquisition according to an embodiment. 10 is a flowchart showing an example of processing of an inference device according to an embodiment. 10 is a flowchart illustrating an example of processing by a training device according to an embodiment. FIG. 2 is a diagram illustrating an example of vertices of a voxel according to an embodiment. FIG. 2 is a diagram illustrating an example of vertices of a voxel according to an embodiment. FIG. 10 is a diagram showing an example of allocation of discrete codes according to an embodiment. FIG. 10 is a diagram showing an example of allocation of discrete codes according to an embodiment. FIG. 1 is a diagram illustrating an example of a schematic relationship of a neural network model according to an embodiment. FIG. 1 illustrates an example of a tree representation of voxels according to one embodiment. FIG. 1 illustrates an example of a tree representation of voxels according to one embodiment. FIG. 1 is a diagram showing an example of the configuration of a neural network model for voxel generation according to an embodiment. FIG. 10 is a diagram showing an example of voxel generation according to an embodiment. FIG. 10 is a diagram showing an example of voxel generation according to an embodiment. FIG. 10 shows pseudocode for generating an octree according to one embodiment. FIG. 10 is a diagram showing an example of the process of generating an octree according to an embodiment. FIG. 1 is a diagram showing an example of implementation of an inference device and a training device according to an embodiment.

Embodiments of the present invention will now be described with reference to the drawings. The drawings and description of the embodiments are provided by way of example only and are not intended to limit the scope of the present invention.

First, as an example of a function used to represent three-dimensional information in this embodiment, a radiation field (radiation field) representing a scene will be described. When coordinates p = (x, y, z), orientation d = (θ, φ), and a feature z(p) at coordinates p are input, an image of the scene captured by a camera placed at an arbitrary position and direction (viewpoint direction) is generated using a function F that returns density ρ and color c = (r, g, b) according to the following formula.

x, y, and z indicate coordinates relative to the respective axes, and coordinate p indicates a point in the luminance field representing the scene. θ and φ indicate angles representing pose d, which indicates the direction from which the point representing coordinate p is viewed. In other words, pose d indicates the direction of the ray extending from the focal position of the camera to each pixel that makes up the image in the direction in which the image is acquired, for example, the angle relative to a specified axis. z(p) is a value that represents the characteristics of coordinate p in the scene to be photographed, and may be a vector with a specified number of dimensions. Details on extracting z(p) will be described later.

F in equation (1) is expressed, for example, by including a neural network model. The training of this neural network model will be described later. As a non-limiting example, the density ρ is a value that depends on the coordinates p and z(p), and the color c is a value that depends on the coordinate p, the orientation d, and z(p). This function F is expressed, for example, by an MLP (Multi-Layer Perceptron). Furthermore, the MLP includes an appropriate activation function such as ReLU.

In this embodiment, a scene is represented using hierarchical voxels. Each voxel in each layer is assigned a discrete code. This hierarchical structure is represented, for example, using an octree.

More specifically, the feature z(p) at coordinate p is obtained from the discrete code defined for each hierarchically formed voxel. The trained model generates the color c and density ρ at coordinate p and orientation d in the scene based on coordinate p, orientation d, and the obtained feature z(p). Using this color c and density ρ, an image of the scene from the desired camera position and direction can be obtained.

First, we will explain the hierarchical structure of voxels and how to obtain feature values at coordinates from the discrete codes defined for voxels.

Figure 1 shows an example of a representation of voxels that contain a scene. For example, the area you want to capture an image of is represented by voxel _V0 , as shown in the figure above. This voxel _V0 is considered to be a voxel in the 0th layer.

Each edge of this voxel _V0 is divided into two (for example, two equal parts) to generate new edges, and each three-dimensional area cut by these edges is designated as a voxel _V1 in the first layer. In this way, by generating eight voxels for one voxel, the area of the scene is represented as a hierarchical voxel structure. Below, each voxel in the nth layer will be represented as _{Vn_ν} , etc., where ν represents the identifier of the voxel in the nth layer.

Voxels may be represented as an octree, as described in detail below. When represented as an octree, the root voxel is represented as a voxel at the 0th level, and voxels are represented by a tree formed by edges and nodes derived from this root.

In the example shown in the figure, the formation of voxels from the 0th layer to the 1st layer involves dividing one region into eight regions, but in subsequent layers, it is not necessary to form voxels in the next layer for all voxels. For example, in the formation of voxels _{V2 in the second layer from voxel V1} in the _first layer, voxel _{V2 in the next layer may be formed only for voxels V1 that} need to be divided or that would benefit from division. Conversely, voxels that do not particularly need to be divided may not need to be divided any further.

When using voxels with this hierarchical structure as a luminance field, for example, voxels at the next level may be formed for voxels with density information that is not 0. Also, voxels at the next level may be formed for voxels with color information that is not 0. In other words, in areas where no subject is present, voxels at the next level may not be formed. By not having lower-level structures for unnecessary voxels in this way, resources such as storage space and computing performance can be saved.

When expressed as an octree, it is possible to specify that a voxel is an empty voxel when generating a voxel in the layer of interest. It is also possible to specify that the next layer will not be generated for this empty voxel.

Voxels representing the surface of the object may also be placed as a luminance field. In this case, the area surrounded by the surface is likely to be inside the object and cannot be seen from the outside; in other words, it is likely to be an area that cannot be photographed by a camera. As with the above, such areas may also be configured not to form voxels in the next hierarchy.

FIG. 2 is a diagram illustrating an example of a voxel representation according to an embodiment. Specifically, the diagram illustrates an extracted voxel V _{(n-1)_0} , which is one voxel in the (n-1)th layer. The coordinate system shown in the lower left is an example of a coordinate setting, but the present invention is not limited to such a coordinate system. The origin of the coordinate system can be set at any position. For example, the origin of the coordinate system may be set at the center of voxel _V0 for ease of calculation, but the present invention is not limited to this.

In the following, for example, the voxels are ordered in order starting from the voxel having the smallest x, y, and z points in this coordinate system, but the ordering is not limited to this. In Figure 2, for example, attention is focused on the voxel V _{(n-1)_0 in the (n-1} )th layer, but other voxels can also be defined in the same way.

Each voxel V is assigned a discrete code in one-to-one correspondence. For example, a voxel V _{n_00} in the nth layer is assigned discrete codes z _{vn_0} to z _{vn_7} , as shown in the lower diagram of Figure 2. These discrete codes may be vectors selected from K different vectors, and may be assigned to each vertex that makes up the voxel. As another example, a discrete code may be assigned to any point within a voxel, such as the center point of the voxel. If voxels in different layers have a common vertex (a vertex with the same coordinates), they may be assigned different discrete codes for each layer, or the same discrete code may be assigned.

In the following, for simplicity, unless otherwise specified, a discrete code z is assigned to each vertex of each voxel V. This is a non-limiting example definition, and a discrete code may be assigned to any point within a voxel V, including its boundary.

Also, as shown in the upper diagram of Figure 2, a voxel is assigned the same discrete code at vertices that it shares with adjacent voxels. For example, voxel V _{n_00} and voxel V _{n_01} share discrete codes z _{vn_1} , z _{vn_3} , z _{vn_5} , and z _{vn_7} at the four vertices of the shared surface. For example, voxel V _{n_00} and voxel V _{n_03} share discrete codes z _{vn_3} and z _{vn_7} at the two vertices they share.

In this way, a discrete code is assigned to each vertex of a voxel V, in other words, to each grid point that is the vertex of a voxel at each layer. Of course, a discrete code does not have to be assigned in areas where no voxels are formed at the layer in question, i.e., areas where no lower voxels are formed in a voxel at the next higher layer.

The brightness (color c and density ρ) representing a point within a voxel V is determined based on the discrete code of each vertex. For example, the feature value of a point within a voxel _Vn is calculated using the discrete codes of each vertex of the voxel _Vn and weighting based on the distance from each vertex. For example, the feature value of a point of interest may be calculated simply by taking a weighted average of the discrete codes proportional to the distance from the point of interest to the vertex. Alternatively, the feature value of an arbitrary point may be calculated using a method such as tri-linear interpolation.

At points belonging to the voxel V _{n_ν} in the nth layer, the feature of the nth layer is calculated using the discrete code z _{Vn_ν} ⁽ⁱ⁾ (i ∈ [0, ... , 7]) of the vertex. For example, if coordinate p indicates a position within the voxel V _n in Figure 2, the feature z _n (p) at the point with coordinate p is calculated using the following formula:

Here, i is an index representing each vertex, Σ represents the sum over i, and α ⁽ⁱ⁾ is, for example, a weight for each vertex of coordinate p in voxel _Vn in tri-linear interpolation, and may be a value that satisfies Σα ⁽ⁱ⁾ = 1. The interpolation method is not limited to tri-linear interpolation, and other appropriate interpolation methods can be used. α can be calculated as an appropriate weighting coefficient based on various interpolation methods, for example, depending on the position of p within the voxel and the position to which the discrete code is assigned in the voxel. The discrete code _zv0 ⁽ⁱ⁾ is assigned to each vertex of the target voxel, and is _{zvn_0} , _{zvn_1} , ..., _{zvn_7} for points belonging to _{Vn_00} in Figure 2.

According to equation (2), the feature values of points belonging to a voxel are calculated using all of the discrete codes defined for the eight vertices, but the feature values may also be calculated using only some of the discrete codes for the eight vertices.

Here, if p exists on a surface that forms the boundary of a voxel, the feature value may be calculated using equation (2) with the discrete codes of the four vertices that form the surface. If p exists on an edge that forms the boundary of a voxel, the feature value may be calculated using equation (2) with the discrete codes of the two vertices defined at both ends of the line segment of that edge.

As another example, if p is on a surface that forms a boundary of a voxel, 12 discrete codes at the vertices of each of the two voxels that make up that surface may be used. If p is on an edge that forms a boundary of four voxels, 18 discrete codes at the vertices of each of the four voxels that share that edge may be used.

As another example, if p exists on the boundary of a voxel, an arbitrary voxel may be selected from among multiple voxels that share the point corresponding to p, and the feature value may be calculated using equation (2). For example, the discrete codes of each vertex of the voxel with the smallest identifier may be used.

The feature value at a vertex of a voxel may be the discrete code assigned to that vertex. As another example, the feature value of that vertex may be determined by performing a weighting operation using the discrete code assigned to that vertex and the discrete codes assigned to the eight vertices adjacent to that vertex.

As will be described later, if there is no target in a voxel for which an image is to be acquired in the scene, the voxel information may indicate that the voxel is empty. If the voxel is empty, the calculation of the feature amount may be omitted.

To reduce the amount of calculations and/or the memory space required for the calculations, it is desirable to assign discrete codes to specific vertices of voxels, but it is also possible to assign discrete codes to any point within a voxel depending on the design, etc.

After the feature z _n (p) is calculated for each layer, the feature z(p) at coordinate p as seen from the entire voxel is calculated by combining the feature z _n (p) calculated for each layer.

β _n indicates the mixing ratio allocated to each layer. For example, when acquiring features for all layers from features up to the Nth layer, β ₀ , β ₁ , ... , β _n = 1 / N may be used. As another example, β _n (p, s) may be used so as to depend on the scene, input, etc. As yet another example, the mixing ratio may be set so that it becomes smaller the deeper the layer, or so that it becomes larger the deeper the layer. Furthermore, the present invention is not limited to these examples, and any ratio may be used so as to acquire appropriate features.

To infer pixel values, the feature quantities shown in equation (3) are used to obtain the color and density of the sequence of points belonging to a path (a ray or line segment) that passes through three-dimensional space from the shooting position to the pixel position. The shooting position is the position where the image is acquired (the camera's focal point). The orientation d is obtained from the direction information from which the camera acquires the image at the shooting position. As mentioned above, the orientation d may be defined, for example, as the direction of the ray extending from the camera's focal point to each pixel in the image acquired.

The color c and density ρ at coordinate p and orientation d can be obtained, for example, by inputting the feature z(p) obtained using equation (3) and the coordinate p and orientation d information into the function F in equation (1). The trained model representing this F is a decoder that obtains color c and density ρ from the feature z(p), coordinate p, and orientation d. In other words, by substituting the feature z(p) shown in equation (3) into the decoder, information about color c and density ρ at coordinate p and orientation d can be obtained.

The path in 3D space for a pixel in an image acquired from the shooting position can be obtained from the direction of a ray extending from the camera's focus to the acquired pixel, as described above. The orientation d can be calculated from the direction of this ray and the position information of the camera's focus (and, if necessary, the camera's internal parameters, etc.), and the coordinate p can be obtained as a point on this ray.

Of course, the feature values at each point in the sequence of points along the path from the shooting position to coordinate p can also be calculated using the discrete codes of each vertex of the hierarchically formed voxels, just as with coordinate p.

Then, color and density information for the sequence of points from the camera position to coordinate p is generated, and this generated color and density information is used to obtain color information for the captured image. Note that although the above does not mention information about the surface, it is of course possible to use surface information along with color and other information. This surface information may be included in the feature z.

As described above, multiple pieces of information about coordinate p are extracted for each pixel based on the shooting position and direction. Then, the color information at coordinate p is projected onto this pixel based on the density information, determining the pixel's color value.

Finally, the color information of a pixel can be expressed as follows, for example, by the following equation, where r(t) is the camera's reference point (for example, the camera's focus) o and the orientation d, as r(t) = o + td.

Here, t is a parameter indicating the distance r from the reference point in the direction of orientation d used to obtain color c and density ρ. _tn and _tf are parameters indicating the minimum and maximum values of r to obtain color c and ρ in the direction of orientation d, respectively. τ(t) represents the probability that the ray does not hit any other particles from _tn to t, i.e., the density added up on the path of the ray traveling from _tn to t.

In other words, equation (5) is related to alpha blending at pixels. The feature z(p) at the synthesized coordinate p described above is used to calculate ρ and c in equation (4). Calculations similar to those in papers such as NeRF-VAE may be performed to calculate ρ(r) and c(r, d).

The following section provides a more detailed explanation of the method for generating the feature z and the hierarchical voxels. In the explanation, voxels such as those shown in Figures 1 and 2 are sometimes expressed in two dimensions, but these representations are equivalent to drawings (voxels) in three-dimensional space.

First, we will explain a specific example of obtaining features from the hierarchical voxel structure and the discrete codes assigned to each vertex of the voxels.

The space representing the scene to be photographed is defined as the voxel _V0 in the 0th layer as shown in Figure 1. As another example, if the scene is divided by clustering or the like, a voxel _V0 in the 0th layer may be set for each divided area.

Instead of defining voxels in a space that contains the entire object for which you want to acquire an image, you can define voxels in the 0th layer in a space that contains only a portion of the object.

Voxels may basically be cubes, but they may also be defined as rectangular parallelepipeds or other three-dimensional shapes as long as they allow for appropriate feature acquisition.

As another example, the rectangular parallelepiped region where the object to be imaged exists can be divided into multiple cubes of equal size, and these divided cubes can be used as voxels in the 0th layer. In this case, during training or inference, the overall feature can be calculated by combining the feature values at each layer from the 0th layer to the nth layer with the feature values of the supervoxel obtained from the discrete codes of the vertices of the final rectangular parallelepiped region (supervoxel) where the object to be imaged exists.

As another example, a cubic voxel that encompasses the entire rectangular parallelepiped area from which an image is to be acquired may be defined as a voxel in the 0th layer. In this way, the definition of a voxel in the 0th layer may be any shape, size, etc., as long as it allows appropriate acquisition of scene features.

This is equivalent to defining supervoxels as voxels in a layer above layer 0 voxels that are not constrained by the octree structure. Which layer is layer 0 may be determined taking into account issues such as computational cost and memory size. In this case, the output values of each layer 0 voxel may be blended appropriately using various interpolation methods when the supervoxel image is acquired.

For simplicity, the area from which the scene is captured is defined as the voxels in the 0th layer of a cube, but this can be changed as appropriate based on the above description. Also, the illustration is of a front view from the z axis direction in Figure 2. Of course, voxels, discrete codes, etc. are defined in the same way for the direction along the z axis.

For example, in Figure 3 and subsequent figures, if the dog is in the foreground of the drawing (the side with the smaller z coordinate) and the person is in the background of the drawing (the side with the larger z coordinate), then the voxels in the first layer (Figure 4) would contain, as a non-limiting example, _{V1_0} : dog, _{V1_1} : dog's tail and hind leg area, _{V1_2} : empty, _{V1_3} : empty, _{V1_4} : other side of the dog, _{V1_5} : person's legs, V1_6 _: empty, _{V1_7} : person's upper body.

FIG. 3 is a diagram showing an example of a voxel _V0 in the 0th layer. As shown in this figure, for example, voxels in the 0th layer may be set so that all objects for which images are to be acquired are included in the voxels. For each vertex of the voxel _V0 , the discrete code _zV0 ⁽⁰⁾ of the voxel is assigned (defined) to the lower left vertex, and the discrete codes _zV0 ⁽¹⁾ , _zV0 ⁽²⁾ , _zV0 ⁽³⁾ are assigned (defined) to the other vertices in a predetermined order (furthermore, _zV0 ⁽⁴⁾ , _zV0 ( ⁵⁾ , _zV0 ⁽⁶⁾ , _zV0 ⁽⁷⁾ at the back of the drawing).

In the following, indices indicating vertices, voxels, and voxel vertices are assigned in Morton order, starting with the smallest x, y, and z, but this is for the sake of explanation and is not intended to be limiting; any symbols may be assigned.

For example, in Figure 3, the discrete codes _zv0 ⁽⁰⁾ _, zv0 ^{(1), zv0(2)} _{, zv0} ⁽³⁾ , _zv0 ⁽⁴⁾ , zv0 ⁽⁵⁾ , _zv0 ( ₆ ⁾ , _zv0 ^{( 7)} ) are assigned to the vertices of the voxel _V0 , which is indicated by a dot. At the 0th layer, the feature quantity at any coordinate p within the voxel is calculated from these discrete codes _zv0 ⁽ⁱ⁾ .

When a supervoxel is defined for a voxel in the 0th layer, a common discrete code may be assigned as the discrete code of each vertex at a vertex that is common to adjacent voxels in the 0th layer. For example, when a supervoxel is defined, the discrete code of each vertex may be assigned the discrete code defined for adjacent voxels.

In the following explanation, we will assume that supervoxels are not present, but if supervoxels are defined, the discrete codes of adjacent voxels may be used to calculate the features within the voxels at the boundary of layer 0 for each layer.

4 is a diagram showing an example of a voxel _V1 in the first layer generated from a voxel _V0 . As shown by the dotted lines, the paired voxel _V0 in the 0th layer is divided at the center of each side to generate a voxel _V1 in the first layer. This generation may be performed recursively, as will be described later.

Voxel _V0 is divided into, for example, voxels _{V1_0} , ..., _{V1_7} in the first layer. For each voxel _{V1_j} in the first layer, a discrete code _{zV1_j} ⁽ⁱ⁾ is assigned to each vertex. For example, in the front view shown in Figure 4, in voxel _{V1_0} , as indicated by the arrows, discrete codes _{zV1_0} ⁽⁰⁾ , _{zV1_0} ⁽¹⁾ , _{zV1_0} ⁽²⁾ , _{zV1_0} ⁽³⁾ , ( _{zV1_0} ⁽⁴⁾ , _{zV1_0} ⁽⁵⁾ , _{zV1_0} ⁽⁶⁾ , _{zV1_0} ⁽⁷⁾ ) are assigned to specific vertices.

As shown in the figure, the same discrete code may be defined for vertices shared between adjacent voxels, for example, zv _{1_0} ⁽¹⁾ = zv _{1_1} ⁽⁰⁾ , zv _{1_0} ⁽²⁾ = zv _{1_2} ⁽⁰⁾ , zv _{1_0} ⁽³⁾ = zv _{1_1} ⁽²⁾ = zv _{1_2} ⁽¹ ) = zv _{1_3} ⁽⁰⁾ .

For example, suppose that voxel _{V1_2} has no scene from which a color value is to be acquired. A discrete code representing an empty voxel, which is preset as a fixed value, may be assigned to such an empty voxel. That is, in FIG. 4, regardless of the values of _{zV1_2} ⁽⁰⁾ , _{zV1_2} ⁽¹⁾ , _{zV1_2} ⁽²⁾ , and _{zV1_2} ⁽³⁾ , a feature representing the sky may be assigned to any point within voxel _{V1_2} .

If we set the discrete code of this empty voxel to a discrete code (z _{V1_2} ⁽⁰⁾ , z _{V1_2} ⁽¹⁾ , z _{V1_2} ⁽³⁾ ) that is common to adjacent voxels for each vertex, the result will be that the intensity of reflected light, etc., at a point that would normally be 0 will have some value that is attributed to the adjacent voxels.

To avoid such a case, before performing the calculation of equation (2), it is possible to determine whether the voxel is empty by reading the discrete code defined for the voxel. If the voxel is determined to be empty, the calculation for acquiring the feature value for the empty voxel can be omitted. In other words, when the discrete code of the empty voxel is acquired, the feature value at any point within the voxel n can be represented by the discrete code z _empty , which indicates that the voxel is empty. For example, in FIG. 4, z _empty can be assigned as zv _{1_2} ⁽²⁾ .

Figure 5 shows the voxel _{V1_0} . As shown in Figure 5, the vertices of the voxels in the first layer are assigned discrete codes, just like in Figure 3. The same is true for the other voxels in the first layer.

Whether or not an object is absent may be determined by considering whether at least a portion of the object is present in the three-dimensional voxel. While the drawings in this disclosure are shown in front view, for example, in FIG. 4, it is possible that no object is present in voxel _{V1_3} , and that an object is present only in voxel _{V1_7} . In such a case, it is of course possible to determine that voxel _{V1_3} is also an empty voxel, and not generate voxels in the hierarchy below this voxel _{V1_3} .

6 is a diagram showing an example of the formation of second-layer voxels _V2 . After the discrete codes for the first-layer voxels are defined, the second-layer voxels are generated by dividing the first-layer voxels _V1 . The second-layer voxels _V2 are generated, for example, by dividing each pair of sides of the first-layer voxel _V1 into two (or equally divided), as described above.

For example, for a voxel _{V1_0} in the first layer, voxels _{V2_00} , ..., _{V2_07} are generated in the second layer. Similarly, for a voxel _{V1_1} in the first layer, voxels _{V2_10} , ..., _{V2_17} are generated in the second layer. Note that, as described above, the numbering is shown as an example and does not have to be as shown. For example, excluding empty voxels, simple indexes such as 0, 1, 2, 3, ..., 9, 10, 11, ... may be assigned. Furthermore, the indexes do not need to be divided by layer, and may be in any form that allows discrete codes to be appropriately stored in a memory area.

For voxels _{V1_0} , _{V1_1} , _{V1_4} , _{V1_5} , and _{V1_7} in the first layer where the subject is present, voxels _V2 in the second layer are generated. On the other hand, for voxels _{V1_2} , _{V1_3} , and _{V1_6} in the first layer where the subject is not present, voxels in the second layer and layers below do not need to be generated.

Voxel _{V1_0} in the first layer is divided into voxels _{V2_00} , ..., _{V2_07} in the second layer. Similarly, voxels _{V1_1} , _{V1_4} , _{V1_5} , _{V1_7} in the first layer are divided into voxels in the second layer. In this way, voxels are divided by 8 times for each layer without taking empty voxels into consideration.

For each voxel _V2 in the second layer, a voxel _V3 in the third layer is generated in the same manner as described above. In FIG _. 6, for example, voxel _{V2_31} is an empty voxel. Such an empty voxel may be assigned a discrete code indicating an empty voxel, as described above. Furthermore, it is not necessary to generate voxels in the third layer or lower layers.

For these voxels, just like the 0th and 1st layers, discrete codes are assigned to calculate features for each vertex. As in Figure 2, discrete codes may be assigned to any position within the voxel (including the boundary) rather than to the vertices.

FIG. 7 is a diagram showing an example of the definition of discrete codes for voxels in the second layer. Voxel _{V2_03} is extracted as _{a voxel in the second layer. As shown in FIG. 7, discrete codes zV2_03 (} ⁰⁾ , _{zV2_03} ⁽¹⁾ , _{zV2_03} ⁽²⁾ , and _{zV2_03} ⁽³⁾ are assigned to the vertices of voxel V2_03. As above, for example, _{zv2_03} ⁽⁰⁾ = _{zv2_00} ⁽³⁾ = _{zv2_01} ⁽²⁾ = _{zv2_02} ⁽¹⁾ , and the same may be true for the discrete codes of other vertices. Discrete codes are defined similarly for other voxels _V2 in the second layer.

This layering of voxels and assignment of discrete codes is repeated until the required layer is reached. The required layer may be determined, for example, based on the resolution, or based on the computational capabilities of the training device or inference device (e.g., the processor's computing power or the size of the storage or memory area) used to perform the training or inference. The depth of the layer may also be determined for any other appropriate reason.

Not only for empty voxels, but also for voxels that represent the interior of an object, it is not necessary to generate lower-level voxels. For example, areas surrounded by surfaces with a transparency of 0 (which blocks light) (such as closed areas surrounded by surfaces with a transparency of 0) do not need to be included in the inference and calculation of light intensity. In such areas, generating more voxels will not improve the accuracy of the pixel values obtained.

For this reason, voxels surrounded by other surface voxels do not need to be further divided into lower-level voxels. For voxels surrounded by surface voxels in this way, a discrete code indicating that the voxel is inside an object may be assigned to the voxel itself, just as with empty voxels. Then, just like with empty voxels, calculation of feature quantities from the discrete code may be omitted.

When a processing circuit that performs training or inference reads an empty voxel or a voxel inside an object from a storage device, it may exclude that voxel from the calculation. The processing circuit may not calculate the feature value at the specified coordinates for the empty voxel.

For example, when acquiring a feature at coordinate p, the processing circuit may scan from the voxel at the deepest level to which coordinate p belongs. If information indicating that this voxel is an empty voxel or a voxel located inside an object is attached to the voxel, the processing circuit may omit the calculation of that feature. However, if the processing circuit is a multi-core or many-core processor acting as an accelerator and the calculation speed does not change, the processing circuit may calculate all voxels, and then assign the feature that is an empty voxel or a voxel located inside an object as the feature z(p) for that coordinate p.

As another example, if the processing circuit determines that the coordinates are empty voxels or voxels inside an object, the processing circuit may omit calculating color information, etc. for those coordinates. The processing circuit may indicate that the voxel is empty or inside an object by, for example, setting density information to 0. As another example, the processing circuit may predefine color and/or density information indicating that the coordinates belong to an empty voxel, and color and/or density information indicating that the coordinates belong to a voxel inside an object.

If the voxel to which coordinate p belongs is an empty voxel or a voxel inside an object, the processing circuit may perform subsequent calculations (e.g., calculations based on equations (4) and (5)) using the color information/density information defined for each case as the color information/density information for coordinate p. Furthermore, once the processing circuit has determined that coordinate p belongs to a voxel that is an empty voxel or a voxel inside an object, it may omit calculations for coordinate p from subsequent calculations.

Furthermore, when the processing circuit reaches a voxel representing the surface in the line of sight (including the light rays entering the camera), it may exclude voxels further in the line of sight (on the other side) from the calculation target. In making this judgment, if the discrete code of a voxel inside an object is obtained, the calculation may be performed using the feature values that can be obtained up to the voxel immediately before the voxel in question.

Note that the voxels to which discrete codes are assigned may be only those that contain the surface of the object (living thing, object, building, etc.) within the scene for which images are being acquired. By assigning discrete codes only to voxels that contain surfaces in this way, the required memory space can be significantly reduced. Furthermore, by acquiring information only about voxels that contain surfaces, alpha blending for other voxels can be omitted (density ρ = 0), thereby reducing computational costs.

However, density ρ may be defined, for example, to represent a foggy appearance. In such cases, based on the line of sight from the camera position, the feature value may be calculated from the discrete code for voxels to which the line of sight leads to a voxel having a discrete code indicating a surface.

Next, we will explain an example of how to obtain the features for equations (2) and (3). As mentioned above, for example, one voxel in the 0th layer may correspond to one scene. These voxels may be configured in a multi-layer structure. Each vertex of a voxel in each layer is assigned a discrete code for calculating the features within that voxel.

Figure 8 is a diagram for explaining how to obtain feature values. For example, we will explain how to obtain feature values at points indicated by ★.

First, as shown by the arrows, the feature z _V0 ⁽⁰⁾ , z V0 (1 ₎ , z V0 ⁽² ), z _V0 ( ³ ), (z _V0 ⁽⁴⁾ , z _V0 ⁽⁵⁾ , z _V0 ^{(6) ,} z _V0 ⁽⁷⁾ ) defined at the vertices of _the voxel V 0 in _the 0th layer are used to obtain the feature z ₀ in the 0th layer from equation (2), for example, using equation (6).

α is a weighting coefficient that is set, for example, based on the distance to the target coordinate. As described above, this weighting may be determined using at least one of a variety of interpolation techniques.

When generating an image from only the features of the 0th layer, the discrete code of voxel _V0 is referenced for the point corresponding to the pixel of the image to obtain the feature at coordinate p, and this feature is input into function F in equation (1) to obtain color c and density ρ, and based on this, the pixel value of each pixel can be obtained using equations (4) and (5). In this case, equation (3) can be omitted and pixel values can be obtained using the feature obtained in equation (6).

If features from layers deeper than layer 0 are required, obtain the features for each layer in order, starting from layer 1 and working your way up.

FIG. 9 is a diagram for explaining how to obtain features in the first layer. Similarly, the feature at ★ is calculated from the discrete code defined at the vertex of voxel _V1 in the first layer, as indicated by the arrow. Since ★ exists in voxel _{V1_0} , the feature at the first layer is obtained using the discrete code at the vertex of this voxel _{V1_0} . The feature _z1 in the first layer can be calculated, for example, using the following equation (7):

When generating an image using features up to the first layer, it is possible to obtain the feature at the coordinate p by inputting _z0 (p) and _z1 (p) calculated by equations (6) and (7) into equation (3). By appropriately processing these features, it is possible to obtain the pixel value for each pixel, as described above.

FIG. 10 is a diagram for explaining how to obtain features at the second layer. The feature at the same ★ is calculated from the discrete code of the vertex of voxel _V2 at the second layer, as indicated by the arrow. Since ★ exists at voxel _{V2_03} , the feature at the second layer is obtained using the discrete code of the vertex of this voxel _{V2_03} . The feature _z2 at the second layer can be calculated, for example, using the following equation (8).

When generating an image using features up to the second layer, it is possible to obtain the feature at the coordinate p by inputting _z0 (p) to _z2 (p) calculated by equations (6) to (8) into equation (3). By appropriately processing these features, it is possible to obtain the pixel value for each pixel, as described above.

Similarly, for the following layers, the feature values for each layer are obtained based on equation (2), and the feature values for coordinate p are obtained from the obtained feature values for each layer based on equation (3). The color c and density ρ are obtained from the obtained feature values, coordinate p, and orientation d based on equation (1). Then, using the necessary information obtained, the color value for each pixel in the captured image can be obtained from equations (4) and (5).

To summarize the voxel division discussed so far, according to the present disclosure, by calculating features using discrete codes, it is possible to reduce the memory area required to store the features. For example, this allows for a significant reduction in memory area compared to when continuous features are defined for each coordinate.

Furthermore, by setting either blank voxels or voxels inside an object, calculation and storage costs can be reduced. Blank voxels can also be made uniformly transparent even when there is a line of sight. Therefore, such voxels can be excluded from calculations when obtaining pixel values.

In addition, by setting voxels for each layer and defining features for each layer in a form that allows them to be calculated from discrete codes, it is possible to infer images with higher accuracy. Furthermore, depending on the resolution of the image to be output, it is possible to decide whether to prioritize high accuracy or computational cost. Features with different resolutions can be defined at each voxel layer. This makes it possible to control computational costs according to the resolution desired by the user. Furthermore, even when acquiring high-resolution images or videos, by processing the voxels in order from the highest, it is possible to first acquire a rough shape and then gradually acquire more detailed information.

In addition, by not generating lower-level voxels for voxels surrounded by surfaces, it is possible to reduce the number of voxels, as well as the computational costs and storage space required.

In addition, voxels in each layer may be generated based on various conditions, such as no object being present or being inside an object (object).

For example, objects often have flat surfaces and surfaces with complex textures. In such cases, discrete codes may be defined up to shallow levels in flat regions, and discrete codes may be defined up to deeper levels in complex texture regions. In this case, information such as an end flag may be added to voxels, and a voxel with this end flag set may indicate that no voxels at deeper levels are defined.

In the above, the hierarchical voxels were generated based on the object for which images are being acquired, but this is not limited to this. As another example, they may be generated automatically during training, as described below.

(reasoning device)
An inference device that infers pixel values includes, for example, a processing circuit (processor) and a storage device (memory, storage). The processing circuit may include, for example, a processor with appropriate computing power that operates as an accelerator such as a GPU capable of massive parallel operations, or may be connected to an accelerator. In this case, the processing circuit may include a processor such as a CPU for control separately from the accelerator. The storage device is a device that stores and holds data required for processing.

This inference device generates captured images from a camera installed so that it can capture at least a portion of a scene, based on information generated by a training device (described below). Hereinafter, the unit that makes up a captured image (including a shadow image; hereafter referred to as an image) is referred to as a pixel, and the value indicating the color of this pixel (which may include an alpha channel) is referred to as the pixel value.

Figure 11 is a flowchart showing the process of obtaining the pixel value of each pixel in a scene.

First, the processing circuit acquires the acquisition position and acquisition direction of an image containing the target (S100). This position and direction may, for example, be information about the camera that acquires the image. This information indicates the position and angle from which the image is acquired.

Based on this information, the processing circuitry may determine whether the scene (voxel) is present within the image (the area captured by a camera positioned in three-dimensional space according to specified information), and may return an error if the scene is outside the image range. In other words, the processing circuitry may determine whether at least a portion of an object present within the scene would be included in the area captured when an image is captured based on the acquired camera position and direction information, and perform processing based on the determination result.

To avoid this, for example, information about the image to be acquired may be set using the distance from the object and the direction in which the image is acquired (for example, a parameter indicating the direction from which the image is acquired relative to the center of the voxel). In this case, the processing circuitry may calculate the image acquisition position and image acquisition direction.

Next, the processing circuit performs a loop process on each pixel of the image (S102).

The processing circuit acquires, for each pixel, information about one or more coordinates for calculating the pixel value within the voxel based on the camera position and the direction (line of sight) in which the pixel value is acquired (S104). The information about one or more coordinates is, for example, information indicating the position of the coordinates at which luminance field information required for calculating the pixel value is acquired. This coordinate information is, for example, information about the coordinates of a point in an area where a line extending from the pixel deployed at the position where the image is acquired in the line of sight (or in a direction based on the line of sight, depending on the processing) crosses the voxel _V0 in the 0th layer.

The coordinates to be acquired may be, for example, information acquired at a predetermined distance from the position of a pixel on the line of sight. Alternatively, these coordinates may be information acquired at a predetermined distance from the point where the voxel intersects with the line of sight. This is not limited to these, and the processing circuit may acquire information regarding any number of coordinates. The granularity of the accumulation in equation (4) is determined by the granularity of the acquisition.

In addition, the coordinate information may be, for example, information indicated using the parameter t in equation (4). In this case, the processing circuit may obtain information indicating the coordinate position within the voxel from the parameter t at a timing before obtaining the luminance field information.

Next, the processing circuit performs a loop process for each pixel coordinate (S106).

The processing circuit then executes a loop process for each layer for the coordinates (S108).

The processing circuit acquires information about the voxel to which the coordinates belong in each layer (S110). For example, the processing circuit acquires information about the identifier assigned to the voxel from the coordinate information. In the example of Figures 8 to 10, the processing circuit acquires information about _V0 in the 0th layer, _{V1_0} in the 1st layer, and _{V2_03} in the 2nd layer.

Next, the processing circuit calculates the feature value (second feature value) of the coordinate in the acquired voxel (S112). The processing circuit acquires information about the discrete code assigned to the vertex of the voxel, and acquires the feature value of the coordinate in the layer of interest based on the information about the discrete code of the acquired vertex. The processing circuit calculates the feature value for each layer based on, for example, equation (2).

The processes of S110 and S112 are repeated for each layer until processing is completed for all layers (S108').

After acquiring the coordinate feature values for all layers, the processing circuit acquires the feature values for the entire voxel for that coordinate value (S114). The processing circuit acquires the feature values for the entire voxel (first feature values) by inputting the feature values acquired for each layer (second feature values) into, for example, equation (3).

The processing circuit then inputs the feature z(x) (first feature) for each coordinate obtained, as well as the coordinate x and orientation d information, into a neural network model such as that shown in equation (1), to obtain the color c and density ρ at the coordinate x and orientation d (S116). The position and orientation information input to the neural network may be information encoded using a transformer, for example. Hereinafter, the position and orientation information may also be appropriately position-encoded information. By appropriately position-encoding using a transformer, the discrete code can be more appropriately converted into a continuous feature and restored.

The processes from S108 to S116 are repeated for each coordinate until the color c and density ρ for all coordinates along the line of sight are obtained (S106').

After obtaining the color c and density ρ at all coordinates required for calculations along the line of sight, the processing circuit obtains the pixel value of the target pixel (S118). This process is performed using, for example, equations (4) and (5).

The processing circuit then repeats the process from S102 onwards until all pixel values have been acquired (S102'). Once pixel values have been acquired for the pixels that make up the image, the processing circuit outputs scene information, such as a captured image of the scene, or video information, based on these pixel values (S120).

In the above, there are loop processes in S102, S106, and S108. These loop processes are not limited to being executed sequentially. In S102, calculations for at least some of the pixels may be executed in parallel. In S106, calculations for at least some of the coordinates may be executed in parallel. In S108, calculations for at least some of the hierarchies may be executed in parallel. In other words, the processing circuit may execute parallel calculations at any granularity.

For example, the processing circuit may simultaneously perform calculations at multiple levels for each of multiple coordinates for each of multiple pixels (S110, S112). After performing calculations at multiple levels, the processing circuit may acquire feature values for each coordinate through parallel processing (S114) and execute processing of a neural network model based on these feature values through parallel processing (S116). The processing circuit may also execute parallel processing to acquire pixel values from output information of the luminance field for each pixel (S118). Furthermore, when performing parallel calculations, the processing circuit may, as necessary, speed up the parallel calculations by, for example, performing reduction processing.

Note that the above describes the processing after the voxel configuration has been defined, but the generation of this voxel configuration will be explained in detail later. When the voxel configuration is generated by the processing circuit, the processing circuit executes the processing to generate this voxel configuration before and after the processing of S100, and then executes the processing from S102.

(Training device)
A training device that trains a neural network model that represents a luminance field includes, for example, a processing circuit (processor) and a storage device (memory, storage), similar to an inference device. The processor may have a processor capable of massively parallel operations and have the computing power to operate as an accelerator, or may be connected to a processor having the computing power to operate as an accelerator. The storage device is a device that stores and holds data required for processing. The training device may also be at least partially the same device as the inference device.

As a non-limiting example, the training device may perform training in the same framework as VQ-VAE.

The training device uses multiple images to train a model for generating voxels corresponding to a single scene. The training device uses images of a specific scene from which voxels are to be formed, captured using multiple camera information C (multiple shooting positions and shooting directions), as learning data. As another example, the training data may include images captured by cameras with camera information having different internal parameters.

The training device, for example, uses the VAE technique to train an encoder that receives training data and outputs a luminance field (as a probability distribution), and a decoder that generates an image based on the luminance field and camera information C. As an example, the training device represents the output of this VAE encoder as a discrete code, such as VQ-VAE, as described below.

Figure 12 is a flowchart showing an example of the processing performed by the training device.

First, the training device's processing circuitry acquires the above camera information C and an image based on that camera information C (S200).

Next, the processing circuit sets voxels based on the camera information C and the images I (S202). The processing circuit may estimate where an object is located in the scene using multiple pieces of camera information C and multiple corresponding images I, and set voxels according to the estimation results as follows: For example, the processing circuit obtains k pieces of camera information C ₀ to C _k-1 and information on images I ₀ to I _k-1 corresponding to each piece of camera information.

For example, if information regarding the three-dimensional positions of objects included in the scene has been acquired in advance, the processing circuit may set voxels based on that information. Alternatively, the processing circuit may use, for example, existing three-dimensional reconstruction technology to estimate information regarding the three-dimensional positions of objects included in the scene, and set voxels based on that estimated information.

As mentioned above, voxels in the 0th layer may be set as a region that includes the entire target scene (3D space). However, this is not limited to this, and they may be a region that includes at least a portion of the target scene, or a region based on the concept of supervoxels.

Figure 13 shows an example of the settings for voxels in the 0th layer. As shown in this figure, the voxels in the 0th layer are set, as a non-limiting example, to include all objects (subjects) present in the scene. The vertices of the voxels are then set as shown by the black dots in the figure.

In the training device (and estimation device), information about the region of a voxel may be expressed as world coordinates. The world coordinates may be set arbitrarily, but as a non-limiting example, the center of a voxel in the 0th layer may be set as the origin. The position p may be expressed as a position relative to the origin of the world coordinates, and the orientation d may be expressed by a predetermined expression (e.g., an angle or direction vector of a predetermined expression) relative to the axes of the world coordinates.

Alternatively, camera information C may be set relative to the origin of the world coordinate system. In this case, external parameters may be set for each camera information C and used to estimate the positional relationship between a pixel (camera coordinate system) and a point or small region of an object (world coordinate system).

World coordinates may typically be expressed in the Cartesian coordinate system, but are not limited to this and may be expressed in other Cartesian coordinate systems or other coordinate systems in which points in space can be uniquely represented.

After setting the voxels in the 0th layer, the processing circuit generates voxels in the 1st layer and below. As described above, empty voxels or voxels representing closed regions may be set among the voxels in the 1st layer and below. Also, for example, voxels in the 1st layer and below may be generated only for regions where the surface of an object exists.

For example, the voxel _{V1_2} in Fig. 4 may be an empty voxel, and subsequent voxels (including the voxel _{V1_2} itself) may not be generated for this empty voxel. That is, as an example, for an empty voxel, processing may be performed in which no voxels of the corresponding layer are generated from higher voxels (for example, the voxel _{V1_2} is generated for the voxel _V0 ). As another example, processing may be performed in which the empty voxel _{V1_2} is generated, and then no voxels of the lower layer are generated for this voxel.

Figure 14 shows an example of setting voxels in the first layer. As shown in this figure, as a non-limiting example, voxels in the first layer are generated in areas where objects exist, without generating empty voxels. As in Figure 13, the vertices of the voxels are set as indicated by black dots.

There are no limitations on processing and data generation, as long as the target voxels are appropriately indicated, for example, as empty, a surface, or a closed region within an object.

As mentioned above, the number of layers may be increased or decreased depending on the texture of the object's surface. For example, in areas where the object's surface exists, the voxels below the first layer may be set in detail.

As mentioned above, the number of voxel levels (level depth D) may be determined in advance by the user, or it may be determined automatically when generating the hierarchical voxels.

Based on the above, the processing circuit generates voxels in the first layer from voxels in the 0th layer. Then, in the same way, voxels in the second layer, voxels in the third layer, ... are generated up to a layer depth D, i.e., voxels up to the Dth layer. Furthermore, the processing is not limited to this sequential processing; after setting the voxels in the 0th layer, voxels in the first layer and below can also be set at the same time.

Returning to Figure 12, the processing circuit then calculates the vectors of the vertices of the voxel (S204). This voxel vertex vector is, for example, a quantity required to calculate the feature quantity at a point inside the voxel. In other words, this vector is a quantity with the same dimensions as the feature quantity. The processing circuit inputs camera information C and image I to the encoder to be trained and obtains the vectors.

Vectors are obtained for each black dot shown in Figure 13 or Figure 14, i.e., each vertex of a voxel. Note that calculations may also be performed for vertices of empty voxels (e.g., white dots in Figure 14). In this case, calculations may be performed for empty voxels, but the amounts may not be used to obtain feature values.

When the encoder receives the camera information C and the image I, it outputs vectors to be set at the vertices of the voxels in each layer. That is, the encoder outputs 8 vectors corresponding to the vertices of the voxel _V0 in the 0th layer, 27 vectors corresponding to the vertices of the 8 voxels _V1 in the first layer, 125 vectors corresponding to the vertices of the 64 voxels _V2 in the second layer, and so on. As mentioned above, vertices belonging only to empty voxels may also be output.

In addition, to obtain vectors from the first layer onwards, it is also possible to use the value of the coefficient β for the second feature of each layer to obtain the first feature shown in equation (3). As another example, an encoder may be formed as a model that optimizes the value of β in parallel based on equation (3).

Since multiple images I are input to the encoder, the processing circuit may combine the vectors set for each vertex of the hierarchy in any manner. For example, the processing circuit may input each image to the encoder and use the average of multiple vectors for that vertex as the vector for that vertex.

As another example, a model can be used that combines multiple encoded vectors into a single vector. In this case, training of this combining model can be performed in conjunction with training of the encoder, etc. For example, some images may contain invisible areas, such as the back of a 3D model.

In such cases, including vector information from the image in unseen areas can result in errors or noise. To appropriately mix vectors, a model that includes techniques such as attention may be used.

Returning to Figure 12, the processing circuit then assigns discrete codes to the vertices (S206). The processing circuit obtains a discrete code based on the calculated vertex vector and assigns the discrete code to the vertex. The discrete code may have multiple latent variable vectors defined in advance, such as in VQ-VAE. For example, 256 discrete codes are provided, but this is not limited to this and the number may be 8192, etc.

This is not limited to this, and the discrete code itself may also be the training target. In this case, a model that generates the discrete code may also be arranged as a model to be trained. By calculating the discrete code for each scene, it is possible to obtain features that are more appropriate for each scene.

There may be, for example, L discrete codes, which can be represented using integer indices l ∈ {0, ..., L - 1}. A vector e _l is predefined for the index l of the discrete code. The processing circuit searches for a vector e _l , l ∈ {0, ..., L - 1} that is closest to the calculated vector of the vertex, and based on the search result, assigns the searched vector to the vertex as the discrete code e _l . The processing circuit can use any search method to search for the discrete code e _l .

For example, the processing circuit may calculate an arbitrary distance (e.g., L1 norm or L2 norm) between the vector of the vertex being obtained and vectors e _l , l ∈ {0, ..., L − 1}, and assign to the vertex a discrete code l corresponding to the vector e _l with the shortest distance.

As another example, the processing circuit may obtain a discrete code using a model that, for example, inputs the vector of vertices being obtained and outputs the closest discrete code. The training device may also train this model.

As another example, the processing circuit may obtain a discrete code using, for example, a Softmax function. When using a Softmax function, the differential coefficient in the corresponding backpropagation may be set to 1. Processing in this way allows backpropagation to be performed regardless of the argmax processing in the forward propagation. This is equivalent to backpropagating the gradient of the latent variable (feature) to the encoder without selecting a discrete code.

By appropriately setting the loss function, training may be performed to update the discrete code so that it approaches the actual feature. For example, the processing circuit may train to update this discrete code up to a predetermined number of times during repeated calculations in training, and after the predetermined number of times, stop updating the discrete code by setting the differential coefficient to 1 as described above. The discrete code may also be updated using EMA (Exponential Moving Average).

The processing circuit may also use, for example, a Gumbel-Softmax function instead of a Softmax function. When using the Gumbel-Softmax function, the gradient in backpropagation may be calculated using, for example, the reparameterization trick. As another example, the Continuous Relaxation technique may be used.

15 and 16 are diagrams showing non-limiting examples of the assignment of discrete codes to vertices in the 0th layer and the 1st layer, respectively. As shown in these figures, a discrete code can be specified by an index indicating the discrete code. For example, in FIG. 15, in a voxel _V0 in the 0th layer, an index of 168 is assigned to the bottom-left vertex, 84 to the bottom-right vertex, 55 to the top-left vertex, and 119 to the top-right vertex, and the discrete codes corresponding to these indexes are assigned as the discrete codes of each vertex. The same applies to FIG. 16.

The discrete code itself is a high-dimensional vector, but it is possible to obtain the vector from a predefined index. In this way, discrete codes are assigned to vertices by assigning them their respective discrete code indices.

By storing only the index, the storage device can store the amount of data corresponding to a vertex compressed to, for example, about 1/(vector dimension), compared to when high-dimensional vectors corresponding to pseudo-features for obtaining feature points inside a voxel are stored as data for each vertex. The processing circuit can obtain the discrete code assigned to the vertex by, for example, referencing a database or lookup table when necessary.

As mentioned above, discrete codes may also be assigned to vertices corresponding to empty voxels (top left vertices) in Figure 16. If assigned, these discrete codes may be ignored in the calculations of the training device after this step and in the calculations of the estimation device.

Returning to Figure 12, the processing circuit then calculates the feature z(p) at coordinate p (S208). This step and the next are steps for inferring the color of a pixel from the discrete code assigned to the voxel.

The processing circuit extracts arbitrary information from the input camera information C and, based on this camera information C, obtains point sequence information for estimating pixels using the luminance field. The point sequence may be expressed, for example, using r(t) used in equation (4).

The processing circuit calculates the second feature z _n (p) for each layer (nth layer) using equation (2) for each coordinate of the point sequence, and obtains the first feature z(p) from the second feature z _n (p) using equation (3).

Next, the processing circuit calculates the color c and density ρ at coordinate p (S210). When the processing circuit receives the coordinate p and the feature (latent variable) as input, it uses a decoder that outputs color c and density ρ to obtain the color c and density ρ at coordinate p.

Next, the processing circuit updates the parameters of the encoder, decoder, and other models used (S212). Also, when training a model that generates β, a model that generates a discrete code, etc., the parameters of these models are updated in parallel or sequentially.

The processing circuit generates an image (the image that results from rendering using camera information C) based on, for example, color c and density ρ. Using the correct image (image I corresponding to camera information C input to the encoder) as training data, the processing circuit updates the model parameters using an error based on the difference in pixel values between the generated image and the correct image.

The loss function may be the error function used in NeRF-VAE. That is, the processing circuit may use a loss function that adds a regularization term similar to that of VAE to the reconstruction error between the generated image and the ground truth image.

The processing circuit may perform the above-mentioned discrete code update based on the above error. In this case, as described above, it is possible to update the discrete code up to a predetermined number of times, and then not update the discrete code after the predetermined number of times.

The processing circuit repeatedly executes steps S204 to S212 as necessary. The processing circuit terminates parameter updating when an appropriate termination condition is met. The termination condition may be the same as the termination condition of a general machine learning method.

After the parameter update is complete, the processing circuit outputs the parameters (S214) and ends the process.

Note that, depending on the scene, a trained luminance field may already be available. In such cases, training can be performed using the known luminance field as training data to restore it using a model conditioned on the special information.

If a trained luminance field is obtained as an encoder and decoder, these two models can be used to train multiple models that acquire features for regions divided into grids and acquire color c and density ρ from the features. For example, similar to the method used in KiloNeRF (arXiv: 2103.13744), distillation learning can be performed by dividing a known luminance field into grids, thereby reducing the network configuration for each grid and improving the accuracy of image restoration. During this training process, it is also possible to train an encoder that acquires features in a grid directly from image information based on information from multiple cameras by using image information as input.

More specifically, as a non-limiting example, the processing circuit may divide the trained luminance field into grids and train multiple models (e.g., MLPs formed with fewer layers than the encoder that generates the luminance field) to obtain features in each grid. This grid may be a region that coincides with voxels at any hierarchical level. Furthermore, the region of the trained luminance field and the region of the 0th voxel in this embodiment may be set as the same region.

The processing circuit may perform more accurate training of the encoder by using the error between a feature at any coordinate output from multiple models formed for the grid and a first feature obtained from a discrete code that is the output of the encoder to be trained.

Furthermore, the processing circuit may use the decoder's output information for the known luminance field. In this case, backpropagation processing can be performed based on the error between the color c and density ρ information obtained from the first features acquired by the encoder and decoder being trained and the color c and density ρ information obtained from the known luminance field.

The processing circuit may also train the above-mentioned encoder and decoder that outputs a luminance field from discretely coded features using the Auto Encoder method. This Auto Encoder may use, for example, a VAE (Variational Auto Encoder), particularly the NeRF-VAE (arXiv:2104.00587) method. The processing circuit may define a loss function by formulating these VAE methods. Furthermore, the configuration and training of the encoder and decoder, and the generation of images based on features obtained from the discrete code, may be implemented in a manner equivalent to that described in the NeRF-VAE paper. For example, when using the NeRF-VAE method, the processing may be performed as shown below.

Figure 17 shows an example of the configuration of an encoder and decoder according to one embodiment. The processing circuit inputs image information I and corresponding camera information C to the encoder and acquires a first feature. As described above, the first feature is acquired based on equation (3) using the discrete code output from the encoder.

The processing circuit inputs the first feature acquired by the decoder and pixel coordinates p and orientation information d acquired from the camera information C, and acquires the color c and density ρ for each pixel. The coordinates p and orientation d may be positionally encoded. The processing circuit uses a renderer to acquire an image from the information acquired from the decoder and the camera information.

The processing circuit then calculates the loss by comparing the image obtained from the renderer with the input image and updates the parameters of each neural network model.

The inference device described above obtains a feature (first feature) z(p) based on coordinate p from the discrete code obtained for each voxel, and inputs this first feature z(p) into a decoder to obtain color c and density ρ information based on coordinate p and orientation d.

By training the same model for various scenes, a scene-independent encoder and decoder can be trained. Images associated with multiple camera information constituting an arbitrary scene can be input to the encoder optimized through this training, allowing features forming a new scene to be acquired. The processing circuitry can generate voxels for the new scene by discretely encoding these features using hierarchical voxels, making it possible to quickly acquire images at a camera position and orientation that includes at least a portion of the scene in its capture area.

The processing circuitry may perform training of the discrete code using multiple MLPs divided into a grid, as described above.

Furthermore, the method of generating voxels is not limited to the above, and they may be generated by any method. For example, the processing circuit may generate voxels using a trained VQ-VAE encoder. Furthermore, the processing circuit may estimate a 3D intensity field from a set of 3D pose-attached images, in which case a trained intensity field may exist as described above. The processing circuit may perform voxel and discrete code assignment that outputs a result similar to this trained intensity field.

For learning voxels, the following types of voxels may be used. For example, multiple voxels corresponding to each scene are prepared. One scene corresponds to one voxel among the multiple voxels. These voxels then have discrete codes assigned to the vertices for each layer. As with the above, the discrete codes assigned to each vertex may be different for each layer.

(Discrete code generation)
Next, we will explain how to generate voxels and assign discrete codes in the new scene generation stage. The processing circuitry of the estimation device can generate voxels and assign discrete codes using discrete codes obtained by inputting arbitrary camera information and corresponding images into an encoder trained by the above-mentioned method.

In the following, when the term "discrete code" is simply mentioned, it will be interpreted as either a vector used to calculate features or a discrete code ID (discrete code identification information) corresponding to the vector, depending on the context. In a neural network model, the input and output may be, for example, a discrete code ID, but this may also be expanded into an internally discretized vector. The expansion into a vector may be performed, for example, using an LUT that links the vector and ID.

The input may also be a vector. In this case, the output layer may convert it to a discrete code ID using an appropriate activation function (e.g., a Softmax function).

Figure 18 shows an example of a voxel tree structure. The voxels described above can be represented, for example, in the form of an octree. Eight child nodes C0, C1, ..., C7 are provided for the parent node P as needed. Further child nodes are generated from each child node as needed, forming a hierarchical voxel structure.

Each node stores information about whether the node exists and a discrete code assigned to the vertex of the voxel corresponding to the node. The information about whether the node exists may indicate, for example, whether the voxel is empty or inside an object. In such cases, the corresponding child node may be deleted, or the corresponding child node may be given an identifier such as empty or inside an object.

As mentioned above, for nodes that indicate the sky or the interior of an object, it is not necessary to generate subsequent child nodes. By suppressing the generation of child nodes in this way, it is possible to reduce the memory area required to store node information.

The voxels described above can be expressed as a tree by assigning, for example, information on the 0th voxel as the top-level parent node. As an example, the tree configuration in the cases of Figures 15 and 16 will be explained using the drawings. Hereinafter, unless otherwise noted, notation of voxels in the depth direction (for example, voxels corresponding to _{V1_4} to _{V1_7} in Figure 16) may be omitted, but when considered in three-dimensional space, these voxels are naturally formed and processed in the same way as the voxels being explained.

FIG. 19 is a diagram showing an example of a tree structure related to the voxel structures of FIGS. 15 and 16. Information on the voxel _V0 at the 0th layer is assigned as the parent node. The node representing _V0 may function as the root node. The node of P corresponding to this _V0 stores the discrete codes of each vertex, which is information on the voxel _V0 .

For example, starting from the information for zv ₀ , the information 168, 84, 55, 119, ... , (eos) is given in order. (eos) indicates that the previous data is the last data corresponding to the voxel. In this way, a tree may be constructed using a token indicating the end of data. However, (eos) is not a required construction, and an expression without using a token is also possible.

From this node P, a child node corresponding to the voxel in the first layer is generated. The processing circuitry of the information processing device (e.g., training device, estimation device) that forms the voxels stores as node information whether a node in the next layer will be generated from parent node P, and if so, the discrete code assigned to that vertex.

The processing circuit infers whether the first voxel _{V1_0} in the first layer is generated (exists) for the parent node P, and if it is generated (exists), the discrete code assigned to the vertex of voxel _{V1_0} becomes the node information.

In the example of Figure 16, a voxel _{V1_0} is generated, and the discrete codes corresponding to each vertex are 54, 254, 51, 95, .... Based on this, the processing circuit stores data such as "54, 254, 51, 95, ..., (eos)" as information for node C0. As shown in the figure, discrete codes for eight vertices are stored as information for node C0.

Similarly, the processing circuit generates a voxel as voxel _{V1_1} in the first layer and stores the discrete code ID "67, 43, ..., (eos)" as information for node C1. Here, the discrete codes assigned to four vertices of voxel _{V1_1} overlap with those of voxel _{V1_0} . For these overlapping nodes, the processing circuit can use already stored discrete codes during training and inference. By not storing these overlapping discrete codes, the memory area used can be reduced.

The processing circuit determines whether to generate a voxel _{V1_3} in the first layer. The voxel _{V1_2} is an empty voxel. In this case, the child node C2 may be an empty node, and no subsequent child nodes may be generated. As another example, the child node C2 itself may not be generated.

The processing circuit performs the same process on all children, and stores as node information whether or not to generate a voxel, as well as the discrete code assigned to the voxel's vertices, as appropriate.

This tree generation can also be expressed in list form as follows:
{(168, 84, 55, 119, ...)}: Stores the discrete code of node P.
{1|(168, 84, 55, 119, ...)}: Generates a child node C0 from node P. For example, 0 or 1 represents the probability of generating a voxel (0: do not generate, 1: generate).
{(54, 254, 51, 95, ...)|1, (168, 84, 55, 119, ...)}: Stores the discrete code of node C0.
{1|(54, 254, 51, 95, ...), 1, (168, 84, 55, 119, ...)}: Generates node C1.
{(67, 43, ...)|1, (67, 43, ...), 1, (168, 84, 55, 119, ...)}: Stores the discrete code of node C1.
{0|(67, 43, ...), 1, (67, 43, ...), 1, (168, 84, 55, 119, ...)}: Node C2 is not generated because the corresponding voxel is empty.
{1|0, (67, 43, ...), 1, (67, 43, ...), 1, (168, 84, 55, 119, ...)}: Generates node C3.
:

As described above, starting from node P, by iteratively generating child nodes and assigning discrete codes, it is possible to generate a tree representation of the hierarchical voxels and the discrete codes assigned to the voxel vertices. If tokens are used, the token information may be appropriately included in the tree representation above.

The above explains the construction of a tree based on the existence of voxels and the discrete codes assigned to the vertices of the voxels. Next, we will explain an example of forming a tree while inferring the discrete codes.

Figure 20 shows an example of a neural network model that forms hierarchical voxels and assigns discrete codes according to one embodiment. This neural network model is formed, for example, as an autoregressive model using a Transformer such as a General Pretrained Transformer (GPT). It is not limited to this, and may also be formed as an autoregressive model with various gates. A model similar to a general language model can also be used.

E represents a token. This token may represent, for example, 0: Do not generate a node, 1: Generate a node, or 2-257: Discrete code ID (if the total number of discrete codes is 256). In other words, the number of tokens defined may be equal to the number of discrete codes + 2. Of course, the nodes correspond to the voxels mentioned above.

As shown in the figure, when a starting token (e.g., (sos)) is input, the next token is generated based on an autoregressively trained prior distribution. In assigning discrete codes, the processing circuit may start with the (sos) token and sequentially sample tokens. In training a neural network model, the processing circuit may also input values given as training data in parallel, achieving parallel training.

During training, the processing circuit sets the tokens representing the discrete codes for the voxel vertices mentioned above and the tokens indicating whether the voxel exists or not to the respective inputs and outputs, and then trains the model. During training, masking processing, etc. may be performed on the Transformer as appropriate.

An autoregressive model like the one above is a probabilistic model. Therefore, there is a high probability that a different output will be produced each time the process is repeated. This probability is based on a prior distribution derived from training, so the output will be proportional to this prior distribution.

Therefore, the processing circuitry of the inference device may repeat the token acquisition process multiple times at the timing of generating voxels, and perform token sampling by using an appropriate activation function such as Softmax.

As another example, the autoregressive model may include a neural network model that outputs a vector as a probability distribution, and an activation function such as Softmax may be used on the probability distribution to extract a discrete code from the output vector from a discretely coded vector. When outputting a probability distribution, training may be performed using the reparameterization trick.

The condition for terminating token output may be, for example, when a predetermined number of samples have been taken or when a predetermined voxel hierarchy has been reached. As another example, the processing circuitry of the training device may train an autoregressive model to output a token corresponding to (eos) for voxel generation. In this case, the processing circuitry of the inference device inputs a start token and repeats the process until the (eos) token is output.

Figure 21 is a diagram that shows an example of generating voxels corresponding to child nodes and generating discrete codes to be assigned to the vertices of voxels corresponding to child nodes.

For example, when a voxel _V0 in the 0th layer and a voxel _{V1_0} in the 1st layer are generated and assigned a discrete code, the generation probability of the next child node, voxel _{V1_1} , is determined by the states of voxels _V0 and _{V1_0} . Similarly, the discrete code when generated is obtained based on a probability distribution set by the states of voxels _V0 and _{V1_0} .

22 is a diagram illustrating another example of generating voxels corresponding to child nodes and generating discrete codes to be assigned to the vertices of the voxels corresponding to the child nodes. For example, whether voxel _{V2_03} in the second layer is generated is determined based on a probability distribution based on the voxels _V0 , _{V1_0} , _{V2_00} , _{V2_01} , and _{V2_02} that have been generated up to that point. If voxel V2_03 is generated, the discrete code assigned to it is also obtained according to a probability distribution based on the same combination of voxels.

Here, even if voxels _{V1_2} etc. have already been generated, the probability distribution is obtained independently from these, based on the direct ancestors of the child node itself and child nodes of the same generation that have already been generated.

If no child nodes are generated, the token may be set to "0" as described above. Also, even if discrete codes are assigned only to voxels that contain surfaces, an appropriate token (e.g., token "0") may be set for voxels that do not contain surfaces.

In the above, we said that a mask is used during training, but when performing this type of inference, a mask may also be used during inference. Considering the hierarchical structure of voxels, lower-level voxels are not heavily influenced by all voxels, but rather by their direct ancestor voxels.

For this reason, when generating voxels or assigning discrete codes to voxels for the Attention used in the Transformer, masking can be performed to appropriately apply Attention to the input that is the direct ancestor of the voxel in question, and voxel generation and discrete code assignment can be performed for lower levels. The same applies during training; masking can be performed to obtain the configuration of lower voxels from information about their direct ancestors.

Note that direct ancestors refer to the voxels _{V1_0} and _V0 of the voxel _{V2_00} in Fig. 22. For example, the voxel _{V1_1} is not a direct ancestor of the voxel _{V2_00} .

Parent-child relationships can be properly obtained, for example, by indexing nodes (voxels) based on the Morton order.

Figure 23 shows the above flow in pseudocode. First, a loop is set up to scan all of the nodes that make up the octree. Each node is treated as the root (parent), and child nodes are generated and assigned discrete codes.

If the depth of r is equal to or greater than a predetermined depth D, the process terminates. In other words, it is determined that at depth D, none of the nodes are parent nodes, and the process terminates. This depth D may be determined based on factors such as computational cost, memory cost, and resolution.

If the depth meets the condition, a loop is performed on the eight edges branching from one parent to generate a child node. In this loop, a decision is first made as to whether to generate the i-th child, based on the probability distribution of the tree state formed so far.

When a child is generated, the discrete code of the i-th node is sampled according to the probability distribution conditioned by the tree up to the current point. This i-th node is then added and registered as a child node in the octree.

In this way, an octree is generated. As mentioned above, it is also possible to generate only the octree structure first, in which case obtaining the probability distribution on line 4 and making the judgment on line 5 are not necessarily required processes.

The above process can be performed by obtaining each token using the neural network model shown in Figure 20. Of course, if tokens can be obtained appropriately, the model in Figure 20 does not need to be used.

The above operation will be explained using a more specific example. The tokens output are based on the distributions in Figures 15 and 16.

Figure 24 shows an example of the input and output of the neural network model shown in Figure 20. For ease of explanation, vertices and voxels are assigned consecutive numbers. The designations in each of the aforementioned figures are explained below.

In Figure 24, for example, the numbers shown in Figure 15 or 16 are assigned as identifiers representing vertices. In this non-limiting example, identifiers are assigned for each layer in ascending order of voxel number, in the same order as the voxels (for example, as you face the drawing, bottom left in front of voxel (0) → bottom right → top left → top right → bottom left → top right → bottom right behind voxel (0) → bottom right → top left → top right → bottom right in the next voxel (1), excluding vertices that overlap with voxel (0)). Of course, the method of assigning these voxel and vertex numbers is not limited to this method, and any method that can properly identify voxels and properly link vertices to discrete code IDs will suffice.

In the first stage, a start token (sos) is input to the Transformer. In response to this token input, the discrete code of the vertex immediately before the bottom left of voxel _V0 at layer 0 is output. Next, by inputting this discrete code into the Transformer, the discrete code of the vertex immediately before the bottom right of voxel _V0 is output. The output of the discrete code ID at layer 0 continues up to vertex 7 at the top right and back of voxel _V0 .

Next, in the ninth iteration, when the discrete code of vertex 7 of voxel _V0 is input, a token indicating whether or not to generate voxel _{V1_0} in the first layer is output. In the example shown, the value is "1," so voxel _{V1_0} is generated. Next, this generation token is input, and the discrete code of the voxel in the zeroth layer is input via the transformer, and the discrete code of the vertex just before the bottom left of voxel _{V1_0} is obtained.

This same operation is repeated as many times as necessary. For example, in the 23rd iteration, based on the input discrete code and the input from the transformer, it is determined that the voxel _{V1_2} is empty, and the output is that the voxel _{V1_2} is not generated.

In this way, the discrete codes assigned to voxels and voxel vertices are performed autoregressively. As another example, an autoregressive model may be used to generate other voxels from some voxels that have already been generated. Other voxels include, for example, other vertices and other hierarchies. In this way, voxels of another scene can be generated from one scene.

In this way, information about the octree, i.e., the hierarchical structure of voxels, is obtained autoregressively. When using an autoregressive model, samples must be taken one by one, but during training, training can be performed in parallel by using the Causal CNN or Causal Transformer techniques.

Scene generation is performed according to the process described in the inference device above, based on the voxel features defined in this way.

As described above, this embodiment allows voxels to be divided and processed, making it possible to generate scenes with a large amount of information, such as scenes with large target objects, with greater accuracy. Furthermore, by using an autoregressive Transformer for hierarchical voxel generation, it is possible to condition position information in detail using feature quantities, making it possible to generate complex voxels.

By using the embodiments disclosed herein, when a camera with arbitrary coordinates and pose is given, it is possible to generate an image from the camera's viewpoint, and it is also possible to use shape information to plan navigation.

In addition, it is possible to implement generative models of rooms, buildings, or outdoor scenes. For example, by training using scan datasets of various rooms or buildings, it is possible to sample various types of rooms or buildings. This can be applied to architectural design, or to generate simulation environments for robots, AGVs (Automatic Guided Vehicles), etc. in the generated data (virtual space). In the case of autonomous driving, it is also possible to generate simulation environments for generating training data and validation data.

As another example, it is also possible to set conditions when generating a scene. For example, in the room example above, it is possible to set conditions such as whether the room has one window or two windows.

Furthermore, because it is possible to create a generative model, even if only some features can be defined in the scan, it is possible to infer the features that could not be obtained from those features and infer and restore the unobserved scene. By applying this, the form disclosed herein can be applied to restoration from partial observation.

In the above disclosure, features are defined as discrete codes at the vertices of voxels, but this is not limited to this; embedding vectors may also be defined at the center (centre of gravity) of the voxel. In this case, the embedding vectors will not match on the surface of the voxel, but just as when features are set at vertices, the discrete representation may be converted to a continuous representation using tri-linear correction or similar.

In the above disclosure, an image of a predetermined scene was generated by expressing the function shown in equation (1) using a neural network, but the function used for image generation is not limited to this. An image of a predetermined scene may be generated by calculating a feature value (a non-limiting example, a first feature value) corresponding to any coordinate based on information about multiple discrete codes corresponding to the predetermined scene (a non-limiting example, a hierarchical voxel with a discrete code assigned to each vertex), and inputting this feature value into a function (a non-limiting example, an implicit function) expressed by a neural network.

The neural network may also represent information (3D information) related to at least one of the following: the space in which the target (object, etc.) exists; the 3D shape of the target; color information about the target; or density information about the target. Images may be generated by rendering based on the output information of the neural network. For example, information about the 3D shape of the target may be generated based on a function (neural network) used in papers such as Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes (arXiv: 2101.10994v1) and NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction (arXiv: 2106.10689) and information about multiple discrete codes. Here, the neural network may output information about the distance from the coordinates of the target to the surface of the object.

Furthermore, the multiple discrete codes assigned to the vertices of voxels in the above disclosure are one non-limiting example of information regarding multiple discrete codes generated for a specified scene. The information regarding the multiple discrete codes may represent at least information (3D information) regarding one of the following: the space in which an object (such as an object) in the specified scene exists, the three-dimensional shape of the object, color information about the object, or density information about the object. The information regarding the discrete code may include at least one of the following: the discrete code, the ID of the discrete code, information about the destination to which the discrete code is assigned, and whether the discrete code is empty or generates a voxel (region). Furthermore, the discrete code may be represented as a tensor rather than a vector, and the output of the encoder and the input to the decoder may also be tensors. Furthermore, features extracted from the multiple discrete codes (first and second features, as one non-limiting example) may also be represented as tensors.

The trained model in the above-described embodiment may, for example, be a concept that includes a model that has been trained as described and then further distilled using general techniques.

Furthermore, the trained model in the above-described embodiment may be, for example, at least one image constituting a video. In other words, it may be generated to reconstruct a three-dimensional space capturing a scene at a certain moment in the video. In this case, for example, by reconstructing the scene for each frame of the video, it is possible to obtain an image I based on any camera information C at any moment.

The information processing device described above can also be summarized as follows. That is, according to one embodiment, an information processing device for assigning discrete codes includes one or more memories and one or more processors. The one or more processors use an autoregressive model to generate information about multiple discrete codes representing three-dimensional information. This information processing device can appropriately generate and assign discrete codes for determining the features of the three-dimensional structure required to generate an image by using a trained autoregressive model through a processing circuit. The three-dimensional information may also be information about at least one of the space in which an object (such as an object) exists in a specified scene, the three-dimensional shape of the object, color information about the object, or density information about the object. The information about the discrete code may also include at least the discrete code, the ID of the discrete code, information about the destination to which the discrete code is assigned, and whether the discrete code is empty or whether it generates a voxel (region).

The information about the multiple discrete codes may be organized into multiple layers. For example, the information about the discrete codes may be organized hierarchically using vectors indicating the discrete codes or indices identifying the discrete codes, corresponding to the three-dimensional structure of the D layer described above. The discrete codes may be represented by tensors.

The multiple layers may be represented by an octree. For example, if the three-dimensional structure is a voxel, eight new voxels can be generated by bisecting each edge of the voxel, and this form can be represented by an octree. Three-dimensional structures of any shape can also be processed in the same way by dividing them into eight three-dimensional structures of approximately equal size.

The multiple layers may divide the three-dimensional space into different regions for each layer, and the multiple discrete codes may be assigned to the regions of each layer.

The region may be a voxel, and the multiple discrete codes may be assigned to the vertices of the voxels. For example, the three-dimensional structure of the D hierarchy described above may be formed by voxels, and discrete codes may be assigned to the vertices of voxels in each hierarchy. When assigning discrete codes, for example, memory space can be reduced by storing discrete code indices associated with each voxel vertex.

The autoregressive model may generate information about the discrete code for each layer, starting from the highest layer and continuing down the hierarchy. As described above, the information about the discrete code may be, for example, a vector representing the discrete code, an index identifying the discrete code, or other information about the discrete code. As described above, the processing circuit can use the autoregressive model to generate voxels and assign information about the discrete code sequentially, starting from the 0th voxel to deeper layers.

The autoregressive model may generate information about the discrete code of a lower layer based on information about the discrete code generated for the higher layer. In other words, by using the autoregressive model, the processing circuit can generate information about the discrete code of a layer higher than the layer of interest based on information about the discrete code of the layer of interest, and assign this discrete code as information about the discrete code of the layer of interest.

When a higher level is divided into at least a first region and a second region, and a third region in a lower level is included in the first region, the autoregressive model may generate information about the discrete code in the third region based on information about the discrete code in the first region, rather than information about the discrete code in the second region. In other words, the processing circuit may use the autoregressive model to generate information about the discrete code for a three-dimensional structure of interest based on information about the three-dimensional structure that is the direct ancestor of the three-dimensional structure. Hierarchically formed lower three-dimensional structures are contained within the three-dimensional structure of at least one higher level. For example, voxels are generated by dividing the upper level into eight, and then these generated voxels are further divided into eight to generate voxels in the lower level. Then, information about voxels in the higher level that directly include the voxel of interest is used to generate the discrete code to be assigned to that voxel.

The autoregressive model may output at least either information regarding whether to generate a region or identification information for the discrete code. In the previous paragraph, it was stated that information regarding the discrete code assigned to the three-dimensional structure can be obtained from information at a higher level that includes the three-dimensional structure in question, but whether or not this three-dimensional structure will be generated may also be obtained from information at a higher level. More specifically, it may be determined based on an autoregressive model whether or not the three-dimensional structure of interest will be generated. Then, after it has been determined that the structure will be generated, information regarding the discrete code may be generated by the autoregressive model.

The autoregressive model may include a Transformer. Using a Transformer allows an autoregressive model to be formed without the need to model complex gates, etc. Also, as shown in some of the above, information on the three-dimensional structure of the model's direct ancestor is used, but this can also be appropriately applied to training and inference by masking it with Attention.

The multiple discrete codes may be used to calculate a feature value at a first coordinate in three-dimensional space, and the feature value may include at least color information at the first coordinate. The processing circuit can obtain color information at a certain coordinate from the feature value using the hierarchically formed discrete codes. The processing circuit can also obtain density information at that coordinate using the feature value. Alternatively, the processing circuit may obtain information about the three-dimensional shape of an object using the feature value.

According to one embodiment, the information generation method is a method of generating information about the plurality of discrete codes using any of the information processing devices described above. Specific details of this information generation method can be understood by taking into consideration the explanations of the above-mentioned embodiments.

According to one embodiment, an information processing device that generates an image from discrete codes generated and assigned by the above-described device or method, or by a different device or method, includes one or more memories and one or more processors. The one or more processors calculate a first feature amount at a first coordinate in three-dimensional space based on a plurality of discrete codes representing a first scene, and generate an image of the first scene based on output information obtained by inputting at least the information on the first coordinates and the first feature amount into a neural network, where the plurality of discrete codes are generated using an autoregressive model. Here, the image of the first scene may be generated by rendering using the output information.

The multiple discrete codes may constitute multiple layers, and the one or more processors may calculate a second feature value at the first coordinate based on the multiple discrete codes in each layer, and calculate the first feature value based on the multiple second feature values calculated for the multiple layers. This second feature value is a feature value of the first coordinate in the layer, and the feature value for each layer can be used to obtain a first feature value that is a feature value of the coordinate mixed across layers.

The one or more processors may obtain color and density information at the first coordinate by inputting the first coordinate information, posture information, and the first feature amount into a neural network.

The first coordinate information and the pose information may be generated based on camera information for the first scene. For example, pose information may be obtained based on internal parameters of the camera and the direction from the focus to the pixel for each pixel based on the internal parameters, and the first coordinate information may be obtained based on this pose information.

According to one embodiment, the information generation method is a method of generating an image of the first scene using any of the information processing devices described above. The specific content of this information generation method can be understood by taking into consideration the explanations of the above-mentioned embodiments. Furthermore, the discrete code used in this information generation method may be a discrete code assigned by the information processing device or information generation method that assigns the above-mentioned discrete code.

According to one embodiment, an information processing device or information generation method that assigns the aforementioned discrete codes, or an information processing device that trains an autoregressive model used in the aforementioned information processing device or information generation method that generates the aforementioned image, includes one or more memories and one or more processors. The one or more processors update the parameters of the autoregressive model that generates information about multiple discrete codes that represent three-dimensional information, using training data that includes information about the multiple discrete codes.

The autoregressive model may output identification information for the discrete code, and the one or more processors may update the parameters based on the identification information for the discrete code output by the autoregressive model and the training data.

The information relating to the plurality of discrete codes may form a plurality of layers, and the plurality of layers may divide a three-dimensional space into different regions for each layer. The autoregressive model may output information regarding whether or not to generate the region. The one or more processors may update the parameters based on the information regarding whether or not to generate the region output by the autoregressive model and the training data.

The autoregressive model may include a Transformer. As mentioned above, using a Transformer allows for autoregressive processing.

According to one embodiment, the information generation method is a method of generating the autoregressive model using any of the information processing devices described above. The specific details of this information generation method can be understood by referring to the explanations of the above-mentioned embodiments. By using the autoregressive model generated by this information processing device or information generation method, it is possible to assign the discrete codes described above and generate images from the discrete codes. For example, it is possible to generate discrete codes based on image information and camera information obtained by photographing an object from various angles, and to infer information about images photographed from angles that are not included in the various angles described above from the generated discrete codes.

Part or all of the devices (including inference devices and training devices; hereinafter referred to as information processing devices) in the above-described embodiments may be configured as hardware, or may be configured as software (program) information processing executed by a CPU (Central Processing Unit) or GPU (Graphics Processing Unit), etc. If configured as software information processing, software that realizes at least some of the functions of each device in the above-described embodiments may be stored on a non-transitory storage medium (non-transitory computer-readable medium) such as a CD-ROM (Compact Disc-Read Only Memory) or USB (Universal Serial Bus) memory, and the software information processing may be executed by loading it into a computer. The software may also be downloaded via a communications network. Furthermore, all or part of the software processing may be implemented in a circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), thereby allowing the software information processing to be performed by hardware.

The storage medium that stores the software may be a removable medium such as an optical disk, or a fixed medium such as a hard disk or memory. The storage medium may also be located inside the computer (such as a main memory device or auxiliary memory device) or outside the computer.

Figure 25 is a block diagram showing an example of the hardware configuration of each device (information processing device) in the above-mentioned embodiment. Each device may be realized as a computer 7 including, for example, a processor 71, a main storage device 72 (memory), an auxiliary storage device 73 (memory), a network interface 74, and a device interface 75, all of which are connected via a bus 76.

The computer 7 in Figure 25 includes one of each component, but may include multiple of the same component. Also, while Figure 25 shows one computer 7, the software may be installed on multiple computers, with each of those computers executing the same or different parts of the software. In this case, a form of distributed computing may be used, in which each computer communicates via a network interface 74 or the like to execute processing. In other words, each device (information processing device) in the above-described embodiment may be configured as a system in which one or more computers execute commands stored in one or more storage devices to achieve its functions. Furthermore, the system may be configured so that information sent from a terminal is processed by one or more computers located on the cloud, and the processing results are sent to the terminal.

The various calculations of each device (information processing device) in the above-described embodiments may be executed in parallel using one or more processors, or using multiple computers via a network. Furthermore, the various calculations may be distributed to multiple processing cores within a processor and executed in parallel. Furthermore, some or all of the processes, means, etc. disclosed herein may be implemented by at least one processor and storage device provided on a cloud that can communicate with computer 7 via a network. In this way, each device in the above-described embodiments may be implemented in the form of parallel computing using one or more computers.

Processor 71 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.) that performs at least one of computer control and calculations. Processor 71 may also be a general-purpose processor, a dedicated processing circuit designed to perform a specific calculation, or a semiconductor device that includes both a general-purpose processor and a dedicated processing circuit. Processor 71 may also include an optical circuit, or may include calculation functions based on quantum computing.

Processor 71 may perform arithmetic processing based on data and software input from each device, etc., within computer 7, and may output arithmetic results and control signals to each device, etc. Processor 71 may control each component that makes up computer 7 by executing computer 7's OS (Operating System), applications, etc.

Each device (information processing device) in the above-described embodiments may be realized by one or more processors 71. Here, processor 71 may refer to one or more electronic circuits arranged on a single chip, or to one or more electronic circuits arranged on two or more chips or two or more devices. When multiple electronic circuits are used, the electronic circuits may communicate with each other via wires or wirelessly.

The main memory device 72 may store instructions executed by the processor 71 and various data, and information stored in the main memory device 72 may be read by the processor 71. The auxiliary memory device 73 is a memory device other than the main memory device 72. Note that these memory devices refer to any electronic component capable of storing electronic information, and may be semiconductor memory. The semiconductor memory may be either volatile memory or non-volatile memory. The memory device for saving various data in each device (information processing device) in the above-described embodiments may be realized by the main memory device 72 or the auxiliary memory device 73, or may be realized by internal memory built into the processor 71. For example, the memory device in the above-described embodiments may be realized by the main memory device 72 or the auxiliary memory device 73.

When each device (information processing device) in the above-described embodiments is configured with at least one storage device (memory) and at least one processor connected (coupled) to this at least one storage device, at least one processor may be connected to one storage device. Also, at least one storage device may be connected to one processor. Also, a configuration in which at least one processor out of multiple processors is connected to at least one storage device out of multiple storage devices may be included. This configuration may also be realized by storage devices and processors included in multiple computers. Furthermore, a configuration in which a storage device is integrated with a processor (for example, a cache memory including an L1 cache and an L2 cache) may also be included.

The network interface 74 is an interface for connecting to the communication network 8 wirelessly or via a wired connection. The network interface 74 may be an appropriate interface, such as one that conforms to existing communication standards. Information may be exchanged with an external device 9A connected via the communication network 8 via the network interface 74. The communication network 8 may be a WAN (Wide Area Network), LAN (Local Area Network), PAN (Personal Area Network), or any combination thereof, as long as information is exchanged between the computer 7 and the external device 9A. An example of a WAN is the Internet, an example of a LAN is IEEE802.11 or Ethernet (registered trademark), and an example of a PAN is Bluetooth (registered trademark) or NFC (Near Field Communication), etc.

The device interface 75 is an interface such as USB that directly connects to the external device 9B.

External device 9A is a device connected to computer 7 via a network. External device 9B is a device connected directly to computer 7.

External device 9A or external device 9B may be, for example, an input device. The input device may be a device such as a camera, microphone, motion capture device, various sensors, keyboard, mouse, or touch panel, and provides acquired information to computer 7. It may also be a device equipped with an input unit, memory, and processor, such as a personal computer, tablet terminal, or smartphone.

Furthermore, external device 9A or external device 9B may be, for example, an output device. The output device may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) panel, or a speaker that outputs sound, etc. Alternatively, it may be a device equipped with an output unit, memory, and a processor, such as a personal computer, tablet terminal, or smartphone.

Furthermore, external device 9A or external device 9B may be a storage device (memory). For example, external device 9A may be network storage, and external device 9B may be storage such as an HDD.

Furthermore, external device 9A or external device 9B may be a device that has some of the functions of the components of each device (information processing device) in the above-mentioned embodiments. In other words, computer 7 may send some or all of the processing results to external device 9A or external device 9B, or may receive some or all of the processing results from external device 9A or external device 9B.

When the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used in this specification (including the claims), it includes any of a, b, c, a-b, a-c, b-c, or a-b-c. It may also include multiple instances of any element, such as a-a, a-b-b, a-a-b-b-c-c, etc. It also includes the addition of elements other than the listed elements (a, b, and c), such as having d, as in a-b-c-d.

When this specification (including claims) uses expressions such as "using/using data as input/based on/according to/in response to" (including similar expressions), unless otherwise specified, this includes cases where the data itself is used, or where data that has been processed in some way (e.g., data with noise added, normalized data, features extracted from data, intermediate representations of data, etc.). Furthermore, when it is stated that a result is obtained "using data as input/based on/according to/in response to" (including similar expressions), this includes cases where the result is obtained based solely on the data in question, or where the result is influenced by other data, factors, conditions, and/or states. Furthermore, when it is stated that "data is output" (including similar expressions), unless otherwise specified, this includes cases where the data itself is used as output, or where data that has been processed in some way (e.g., data with noise added, normalized data, features extracted from data, intermediate representations of data, etc.) is used as output.

When the terms "connected" and "coupled" are used in this specification (including the claims), they are intended as open-ended terms that encompass direct connection/coupling, indirect connection/coupling, electrically connection/coupling, communicatively connection/coupling, functionally connection/coupling, and physically connection/coupling. These terms should be interpreted appropriately according to the context in which they are used, but should be interpreted open-endedly to include any form of connection/coupling that is not intentionally or naturally excluded.

When the expression "A configured to B" is used in this specification (including the claims), it may include the physical structure of element A being configured to perform operation B, and element A's permanent or temporary setting/configuration being configured/set to actually perform operation B. For example, if element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and that the processor is configured to actually perform operation B through permanent or temporary program (instruction) settings. Furthermore, if element A is a dedicated processor or dedicated arithmetic circuit, it is sufficient that the circuit structure of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached to it.

When terms implying containing or possessing (e.g., "comprising," "including," "having," etc.) are used in this specification (including the claims), they are intended to be open-ended terms that include the inclusion or possession of things other than the object designated by the object of the term. When the object of such terms implying containing or possessing does not specify a quantity or suggests a singular number (e.g., expressions using the articles "a" or "an"), the expression should be construed as not being limited to a specific number.

In this specification (including the claims), even if expressions such as "one or more" or "at least one" are used in some places and expressions that do not specify a quantity or that imply a singular number (expressions using the articles "a" or "an") are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or that imply a singular number (expressions using the articles "a" or "an") should be interpreted as not necessarily being limited to a specific number.

When this specification states that a particular advantage/result is obtained with respect to a particular configuration of a certain embodiment, it should be understood that the same advantage/result can also be obtained with one or more other embodiments having that configuration, unless otherwise stated. However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and/or states, etc., and that the effect is not necessarily obtained with that configuration. The effect is merely obtained with the configuration described in the embodiment when various factors, conditions, and/or states, etc. are satisfied, and the effect does not necessarily occur in claimed inventions that define that configuration or a similar configuration.

When terms such as "maximize" and "maximization" are used in this specification (including the claims), they include finding a global maximum, finding an approximation of a global maximum, finding a local maximum, and finding an approximation of a local maximum, and should be interpreted appropriately according to the context in which the term is used. They also include probabilistic or heuristic approximations of these maxima. Similarly, when terms such as "minimize" and "minimization" are used, they include finding a global minimum, finding an approximation of a global minimum, finding a local minimum, and finding an approximation of a local minimum, and should be interpreted appropriately according to the context in which the term is used. They also include probabilistic or heuristic approximations of these minima. Similarly, when terms such as "optimize" and "optimization" are used, they include finding a global optimum, finding an approximation of a global optimum, finding a local optimum, and finding an approximation of a local optimum, and should be interpreted appropriately according to the context in which the term is used. It also includes finding approximations to these optimal values probabilistically or heuristically.

In this specification (including the claims), when multiple pieces of hardware perform a predetermined process, the pieces of hardware may work together to perform the predetermined process, or some of the hardware may perform all of the predetermined process. Furthermore, some of the hardware may perform part of the predetermined process, and other hardware may perform the rest of the predetermined process. In this specification (including the claims), when expressions such as "one or more pieces of hardware perform a first process, and the one or more pieces of hardware perform a second process" (including similar expressions) are used, the hardware performing the first process and the hardware performing the second process may be the same or different. In other words, it is sufficient that the hardware performing the first process and the hardware performing the second process are included in the one or more pieces of hardware. Note that hardware may also include electronic circuits or devices including electronic circuits.

In this specification (including the claims), when multiple storage devices (memories) store data, each of the multiple storage devices may store only a portion of the data, or may store the entire data. It may also include a configuration in which only some of the multiple storage devices store data.

Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, modifications, substitutions, and partial deletions are possible within the scope of the conceptual idea and spirit of the present disclosure, which is derived from the content defined in the claims and their equivalents. For example, when numerical values or formulas are used in the above-described embodiments, these are shown for illustrative purposes and do not limit the scope of the present disclosure. Furthermore, the order of each operation shown in the embodiments is also shown for illustrative purposes and does not limit the scope of the present disclosure.

Claims (15)

one or more memories;
one or more processors;
the one or more processors:
Using an autoregressive model, multiple layers are constructed, and information on multiple discrete codes that represent a three-dimensional space is generated.
calculating a feature amount in the coordinates of the three-dimensional space based on information about the plurality of discrete codes ;
generating an image of the three-dimensional space based on output information obtained by inputting at least the feature amount into a neural network ;
Information processing device. The plurality of layers are represented by an octree.
The information processing device according to claim 1 . The plurality of layers divide the three-dimensional space into different regions for each layer,
the plurality of discrete codes are assigned to regions of each layer;
3. The information processing device according to claim 1 . the region is a voxel,
the plurality of discrete codes are assigned to vertices of the voxels;
The information processing device according to claim 3 . the autoregressive model generates information about discrete codes for each layer in order from a higher layer to a lower layer;
The information processing device according to claim 1 . the autoregressive model generates information about the discrete code of a lower layer based on information about the discrete code generated for the higher layer;
The information processing device according to claim 5 . When an upper layer is divided into at least a first region and a second region and a third region of a lower layer is included in the first region, the autoregressive model generates information about the discrete code in the third region based on information about the discrete code in the first region, not information about the discrete code in the second region.
The information processing device according to claim 6 . The autoregressive model outputs at least one of information on whether to generate a region or identification information of a discrete code.
The information processing device according to any one of claims 3 to 7 . the autoregressive model comprises a Transformer;
The information processing device according to claim 1 . the one or more processors generate the image based on the output information acquired by inputting at least the coordinate information and the feature amount into the neural network .
The information processing device according to claim 1 . the one or more processors generate the image using color and density information at the coordinates acquired by inputting the coordinate information, posture information, and feature amount into the neural network.
The information processing device according to claim 10 . The coordinate information and the posture information are generated based on camera information.
The information processing device according to claim 11. the one or more processors:
calculating a second feature amount at the coordinate based on the plurality of discrete codes in each layer;
calculating the feature amount based on the second feature amounts calculated for the plurality of layers;
The information processing device according to any one of claims 1 to 12 . An image generating method for generating the image using the information processing device according to any one of claims 1 to 13 . A program for causing at least one information processing device to execute the image generating method according to claim 14.