JP7598058B2 - Learning device, trained model generation method, data generation device, data generation method, and program - Google Patents

Description

The present invention relates to a learning device, a method for generating a trained model, a data generation device, a data generation method, and a program.

Estimating three-dimensional data (such as depth information) corresponding to two-dimensional image data from that data has long been a problem that has attracted attention. One method known to solve this problem is to use paired data of two-dimensional images and three-dimensional data as training data to train a converter that can obtain three-dimensional data from two-dimensional images. However, there is a problem in that data collection costs are high because dedicated equipment is required to collect paired data, and accurate alignment between the data is also required after acquisition.

Non-patent document 1 discloses a technology that uses a generative model to learn image data from different viewpoints so that the empirical distribution of the generated image data matches the empirical distribution of the actual image.

A. Noguchi, T. Harada, “RGBD-GAN: Unsupervised 3D Representation Learning From Natural Image Datasets via RGBD Image Synthesis,” in Proc. ICLR 2020.

However, in order to realize the technology described in Non-Patent Document 1, the dataset used for learning needs to contain a sufficient amount of image data captured from a plurality of different viewpoints. Therefore, if it is difficult to collect image data related to different viewpoints, there is a possibility that appropriate learning cannot be performed.
An object of the present invention is to provide a learning device, a method for generating a trained model, a data generation device, a data generation method, and a program that are capable of generating a two-dimensional image and three-dimensional data corresponding to the two-dimensional image without supplementary data regarding the three-dimensional data.

One aspect of the present invention is a learning device that includes an image generation unit that generates first image data simulating a captured image related to a first depth of field by inputting latent variables into an image generation model that is a machine learning model; a three-dimensional data generation unit that generates three-dimensional data corresponding to the first image data by inputting the latent variables into a three-dimensional generation model that is a machine learning model; a conversion unit that generates second image data simulating a captured image related to a second depth of field based on the first image data and the three-dimensional data; a calculation unit that calculates a learning reference value that indicates the degree of closeness between the distribution of the first image data and the second image data and the empirical distribution of an actual captured image; and an update unit that updates parameters of the image generation model and the three-dimensional generation model based on the learning reference value.

One aspect of the present invention is a method for generating a trained model, comprising the steps of: generating first image data simulating a captured image relating to a first depth of field by inputting latent variables into an image generation model that is a machine learning model; generating three-dimensional data corresponding to the first image data by inputting the latent variables into a three-dimensional generation model that is a machine learning model; generating second image data simulating a captured image relating to a second depth of field based on the first image data and the three-dimensional data; calculating a learning reference value indicating the degree of closeness between the distribution of the first image data and the second image data and the empirical distribution of an actual captured image; updating parameters of the image generation model and the three-dimensional generation model based on the learning reference value; and outputting the trained image generation model and the three-dimensional generation model.

One aspect of the present invention is a data generation device comprising: an image generation unit that generates image data simulating a captured image by inputting latent variables into an image generation model output by the trained model generation method relating to the above aspect; and a three-dimensional data generation unit that generates three-dimensional data by inputting the latent variables into a three-dimensional generation model output by the trained model generation method relating to the above aspect.

One aspect of the present invention is a data generation method comprising the steps of generating image data simulating a captured image by inputting latent variables into an image generation model output by the trained model generation method relating to the above aspect, and generating three-dimensional data by inputting the latent variables into a three-dimensional generation model output by the trained model generation method relating to the above aspect.

One aspect of the present invention is a program for causing a computer to function as a learning device relating to the above aspect.

According to at least one of the above aspects, a two-dimensional image and three-dimensional data corresponding to the two-dimensional image can be generated without supplementary data regarding the three-dimensional data.

1 is a diagram illustrating a configuration of a data generation system according to a first embodiment. 1 is a schematic block diagram showing a configuration of a model learning device according to a first embodiment. 4 is a flowchart showing the operation of the model learning device according to the first embodiment. FIG. 4 is a diagram showing data transition in the learning process according to the first embodiment. 1 is a schematic block diagram showing a configuration of a data generating device according to a first embodiment. FIG. 1 is a schematic block diagram illustrating a configuration of a computer according to at least one embodiment.

First Embodiment
Configuration of data generation system 1
1 is a diagram showing a configuration of a data generation system 1 according to a first embodiment. The data generation system 1 generates image data simulating an image captured with a deep depth of field of the same subject (deep depth of field image), image data simulating an image captured with a shallow depth of field (shallow depth of field image), and depth data indicating the depth of the subject. The deep depth of field image is image data with a wide range that appears to be in focus. The shallow depth of field image is image data with a narrow range that appears to be in focus.

The data generation system 1 includes a data generation device 11 and a model learning device 13 .
The data generating device 11 generates a set of deep depth-of-field images, shallow depth-of-field images, and depth data using an image generation model and a three-dimensional generation model, which are machine learning models.
The model learning device 13 uses actual captured images as learning data to learn the image generation model and the three-dimensional generation model. Note that the captured images related to the learning data do not need to have supplementary data related to the three-dimensional data. In other words, the learning data does not need to include three-dimensional data such as depth data or pairs of captured images with different viewpoints.

Configuration of model learning device 13
2 is a schematic block diagram showing the configuration of the model learning device 13 according to the first embodiment. The model learning device 13 according to the first embodiment includes a learning data storage unit 131, a model storage unit 132, a latent variable generation unit 133, an image generation unit 134, a three-dimensional data generation unit 135, a conversion unit 136, a recognition unit 137, a calculation unit 138, and an update unit 139.

The learning data storage unit 131 stores multiple image data. Each image data is an image captured by an imaging device. The learning data storage unit 131 stores image data relating to various depths of field.

The model storage unit 132 stores an image generation model G _I , a depth generation model G _D , a discrimination model C, and a depth of field effect renderer R. The image generation model G _I , the depth generation model G _D , and the discrimination model C are all configured by neural networks (e.g., a convolutional neural network, a fully connected neural network, a recurrent neural network, etc.). The depth of field effect renderer R is configured by a mathematical model that simulates the optical system of an imaging device.

The image generation model G _I has a latent variable z as input and outputs a deep depth of field image I ^g _d , which is image data simulating a captured image with a deep depth of field. The latent variable is an arbitrary numerical value that serves as a seed for generating image data.

The depth generation model _GD takes a latent variable z as input and outputs depth data ^Dg representing the depth of a subject in _a deep depth-of-field image ^Igd . The number of elements of the depth data ^Dg is equal to the number of elements of the deep depth-of-field image ^Igd . The image generation model G _I and _{the depth generation model GD may} share some layers (e.g., an input layer and an intermediate layer).

The depth of field effect renderer R receives a pair of a deep depth of field image I ^g _d and depth data D ^g as input, and outputs a shallow depth of field image I ^g _s which is image data simulating a captured image with a shallow depth of field relating to the same subject as the deep depth of field image I ^g _d . Details of the depth of field effect renderer R will be described later.

The discrimination model C receives image data as input and outputs an evaluation value indicating the probability that the input image data is an actual captured image or the degree to which it is an actual captured image. For example, when outputting the probability that the input image data is an actual captured image, the discrimination model C outputs a value closer to 0 as the probability that the input image data is a deep depth-of-field image I ^g _d or a shallow depth-of-field image I ^g _s increases, and outputs a value closer to 1 as the probability that it is actual captured data increases.

The image generation model G _I , the depth generation model G _D , the discriminative model C, and the depth of field effect renderer R constitute GANs (Generative Adversarial Networks). The combination of the image generation model G _I , the depth generation model G _D, and the depth of field effect renderer R is a Generator. The discriminative model C is a Discriminator.

The latent variable generation unit 133 generates a latent variable z based on a random number. For example, the latent variable generation unit 133 randomly extracts the latent variable z from an arbitrary distribution such as a Gaussian distribution or a uniform distribution. The latent variable generation unit 133 may extract the latent variable z from image data stored in the learning data storage unit 131.

The image generation unit 134 generates a deep depth-of-field image I ^g _d by inputting the latent variable z generated by the latent variable generation unit 133 to the image generation model G _I stored in the model storage unit 132. That is, the image generation unit 134 calculates the deep depth-of-field image I ^g _d by the following formula (1).

The three-dimensional data generation unit 135 generates depth data ^Dg, which is three-dimensional data, by inputting the latent variable z generated by the latent variable generation unit 133 to the depth generation model _GD stored in the model storage unit 132. That is, the three-dimensional data generation unit 135 calculates the depth data ^Dg by the following formula (2).

The conversion unit 136 generates a shallow depth-of-field image I g s by inputting the deep depth-of-field image I ^g _d generated by the image generation unit 134 and the depth data D ^g generated by the three-dimensional data generation unit 135 to the depth-of-field effect renderer R stored in the model storage unit 132. That is, the conversion unit 136 converts the deep depth-of-field image I ^g _d to a shallow depth-of-field image I ^g _s . That is, the conversion unit 136 generates the shallow depth-of-field image I ^g _s by the following formula (3). In formula (3), ^{s represents the degree of mixture of the deep depth-of-field image I g d} _and the shallow depth-of-field image I ^g _s , that is, the degree of depth of field. Note that when the degree of mixture _s is 0, the calculation result is equal to the deep depth-of-field image I ^g _d .

Here, the depth of field effect renderer R will be described. The depth of field effect renderer R according to the first embodiment calculates the path of a ray in a virtual optical system, and converts the depth of field by widening the aperture area in the virtual optical system. Specifically, the depth of field effect renderer R calculates a depth map M ^{g (x, u) indicating the relationship between the position coordinate x on the image plane and the angle coordinate u on the aperture plane and the depth of the subject, as shown in formula (4), by transforming the depth data D g using a transformation function T. Next, the depth of field effect renderer R calculates an image L g (} x, u) formed by the light incident for each line of sight direction on the aperture plane from the deep depth of field image I ^g _d based on the depth map M ^g (x, u), as shown in formula (5). Then, the depth-of-field effect renderer R generates a shallow depth-of-field image I g s by integrating the image L ^g (x, u) using an indicator A(u) that simulates the aperture of the optical system as shown in equation ( ⁶ ₎ .

In formula (5), m (hat) indicates the distance to the focal position (hyperfocal distance). m (hat) may be a value acquired by learning, a random number extracted from a predetermined distribution, or a constant (e.g., zero).
Indicator A(u) is a matrix in which the values of elements corresponding to openings are positive, the values of elements other than openings are 0, and the sum of the values of all elements is 1. The shape of the opening simulated by indicator A(u) is, for example, a circle. In this way, the depth of field effect renderer R is given optical constraints that take into account the light space. This allows the depth of field effect renderer R to realize optically consistent image conversion.

The depth of field effect renderer R may be configured by a warping function (transformation function) based on the depth data D _g (x), or may be configured by a neural network model. When the depth of field effect renderer R includes a neural network model, it may be given a constraint based on the transformation result by the warping function. Specifically, the depth of field effect renderer R may be configured to input the depth data D _g transformed by the warping function to the neural network model.

The identification unit 137 inputs the deep depth-of-field image I ^g _d , the shallow depth-of-field image I ^g _s, and the captured image stored in the learning data memory unit 131 into the identification model C, and calculates an evaluation value indicating the degree to which the input image data is an actual captured image.

The calculation unit 138 calculates a learning criterion (loss function) used for training the image generation model G _I , the depth generation model G _D, and the discriminative model C. Specifically, the calculation unit 138 calculates the learning criterion based on an adversarial learning criterion.
The adversarial learning criterion L _AR-GAN is an index indicating the accuracy of determining whether image data is an actual captured image or image data simulating a captured image. The calculation unit 138 calculates the adversarial learning criterion L _AR-GAN as shown in the following formula (7).

In formula (7), s to P _s (s) represent the degree of mixing of the deep depth of field image I ^g _d and the shallow depth of field image I ^g _s , that is, the degree of depth of field. The distribution P _s (s) can be a distribution related to a range of values from 0 to 1, such as a binomial distribution or a uniform distribution. In addition, z to P _z (z) represent a process of extracting the latent variable z from the distribution P _z (z). Note that, although cross-entropy is used as the learning criterion in formula (7), this is not limited thereto, and a learning criterion based on any distance criterion such as L1 distance, L2 distance, or Wasserstein distance may be used.

The update unit 139 updates parameters of the image generation model G _I , the depth generation model G _D, and the discriminative model C based on the adversarial learning reference L _AR-GAN calculated by the calculation unit 138. Specifically, the update unit 139 updates parameters for the discriminative model C such that the adversarial learning reference L _AR-GAN increases. Furthermore, the update unit 139 updates parameters for the image generation model G _I and the depth generation model G _D such that the adversarial learning reference L _{AR-GAN decreases. Furthermore, when the depth of field effect renderer R has learnable parameters, the update unit 139 updates parameters for the depth of field effect renderer R such that the adversarial learning reference L AR- GAN} decreases.

The shallow depth-of-field image I ^g _s is generated from the depth data D ^g using a depth-of-field effect renderer R having optical constraints. Therefore, in order for the discrimination model C to erroneously determine that the shallow depth-of-field image I ^g _s is an actual captured image, it is necessary to generate appropriate depth data D ^{g under the above optical constraints. Therefore, the model learning device 13 according to the first embodiment can update the parameters of the image generation model G I and the depth generation model G D} so that a set of the deep depth-of-field image I ^g _d , the shallow depth-of-field image I ^g _s, and the depth data D ^g can be appropriately generated by updating the parameters according to the above learning criterion. In addition, when the depth-of-field effect renderer _R has parameters that can be learned, the parameters of the depth-of-field effect renderer _R can be updated.

<<Operation of model learning device 13>>
Fig. 3 is a flowchart showing the operation of the model learning device 13 according to the first embodiment. Fig. 4 is a diagram showing the transition of data in the learning process according to the first embodiment.
When the model learning device 13 starts the learning process, it repeatedly executes the processes from step S1 to step S6 shown below a predetermined number of times. First, the latent variable generation unit 133 generates a latent variable z based on a random number and a predetermined distribution (step S1). Next, the image generation unit 134 inputs the latent variable z generated in step S1 to the image generation model G _I stored in the model storage unit 132 to generate a deep depth of field image I ^g _d (step S2).

The three-dimensional data generating unit 135 generates depth data ^Dg by inputting the latent variable z generated in step S1 to the depth generation model _GD stored in the model storage unit 132 (step S3). Next, the conversion unit 136 determines a mixing degree s between 0 and 1 in accordance with a predetermined distribution (step S4). The conversion unit 136 generates a shallow depth-of-field image Igs by _inputting the deep depth-of-field image ^Igd generated in step S2 and the depth data ^Dg generated in step S3 multiplied by the mixing degree s to the depth-of-field effect renderer R stored in the model storage unit 132 (step S5). Note that when the mixing degree s is zero ^{, the generated shallow depth-of-field image Igs} _coincides ^with _the deep depth _- of-field image ^Igd . Next, the classifier 137 inputs the shallow depth-of-field image I ^g _s to the classification model C, and calculates an evaluation value indicating the degree to which the input shallow depth-of-field image I ^g _s is an actual captured image (step S6).

After calculating an evaluation value for the shallow depth-of-field image I ^g _s generated from a predetermined number of latent variables z, the model learning device 13 repeatedly executes the processes of steps S7 to S8 described below a predetermined number of times. First, the classification unit 137 reads out an arbitrary captured image from the learning data storage unit 131 (step S7). The classification unit 137 inputs the read captured image to a classification model C to calculate an evaluation value indicating the degree to which the input captured image is an actual captured image (step S8).

The calculation unit 138 calculates the adversarial learning reference L _AR-GAN based on the above-mentioned formula (7) using the evaluation value calculated in step S6 and the evaluation value calculated in step S8 (step S9). The update unit 139 updates the parameters of the image generation model G _I , the depth generation model G _D, and the discriminative model C based on the adversarial learning reference L _AR-GAN calculated in step S9 (step S10). In addition, if the depth of field effect renderer R has learnable parameters, the update unit 139 updates the parameters of the depth of field effect renderer R.

The update unit 139 determines whether the parameter update from step S1 to step S10 has been repeated a predetermined number of epochs (step S11). If the repetition does not reach the predetermined number of epochs (step S11: NO), the model learning device 13 returns to step S1 and repeats the learning process.

On the other hand, if the repetition reaches a predetermined number of epochs (step S11: YES), the model learning device 13 ends the learning process. This allows the model learning device 13 to generate the image generation model G _I and the depth generation model G _D , which are trained models. In addition, if the depth of field effect renderer R has trainable parameters, the model learning device 13 can generate the depth of field effect renderer R, which is a trained model.

Configuration of data generating device 11
FIG. 5 is a schematic block diagram showing the configuration of a data generating device 11 according to the first embodiment.
The data generating device 11 according to the first embodiment includes a model storage unit 111, a latent variable generating unit 112, an image generating unit 113, a three-dimensional data generating unit 114, a converting unit 115, and an output unit 116.

The model storage unit 111 stores the image generation model G _I and depth generation model G _D that have been trained by the model learning device 13 , and the same depth of field effect renderer R as that of the model learning device 13 .

The latent variable generation unit 112, the image generation unit 113, the three-dimensional data generation unit 114, and the conversion unit 115 execute the same processes as the latent variable generation unit 133, the image generation unit 134, the three-dimensional data generation unit 135, and the conversion unit 136 included in the model learning device 13. Note that the conversion unit 115 performs calculations assuming the mixture degree s to be 1 when generating the shallow depth-of-field image I ^g _s .

The output unit 116 outputs the deep depth-of-field image I ^g _d generated by the image generation unit 113 , the depth data D _g generated by the three-dimensional data generation unit 114 , and the shallow depth-of-field image I ^g _s generated by the conversion unit 115 .

<Action and Effects>
This allows the data generating device 11 to generate a set of a deep depth-of-field image I ^g _d and a shallow depth-of-field image I ^g _s , which are two-dimensional images, and depth data D _g , which is three-dimensional data. The image generation model G _I and the depth generation model G _D used by the data generating device 11 to generate these data sets are trained without supplementary data regarding the three-dimensional data. In other words, according to the data generating system 1 according to the first embodiment, a two-dimensional image and three-dimensional data corresponding to the two-dimensional image can be generated without supplementary data regarding the three-dimensional data.

This is because the model learning device 13 trains the model so that the empirical distribution of the deep depth _- of-field image ^{I g d} _and the empirical distribution of the shallow depth-of-field image I ^g _s generated from the deep depth-of-field image I ^g _d and the depth data D _g are both close to the empirical distribution of the actual captured image. In other words, in order for the shallow depth-of-field image I _{g s} generated from the deep depth-of-field image I ^g d and the depth data D _{g to be close to the empirical distribution of the actual captured image, appropriate depth data D g} must be obtained, and the fact that the shallow depth-of-field image I ^g _s has become close to the empirical distribution of the actual captured image indicates that the model has been trained to be able to obtain appropriate depth data D _g .

<Experimental Results>
A description will be given of an example of an experimental result of generating data pairs using the data generation system 1 according to the first embodiment. In the experiment, captured images related to flower images, bird images, and face images were used as learning data.

In the experiment, the data generation system 1 randomly extracted latent variables z based on a normal distribution N(0,1). The image generation model G _I , the depth generation model G _D and the discriminative model C were configured using CNN. The transformation function T of the subject depth effect renderer R was configured to apply CNN after warping based on D ^g (x).

In the experiment, the RGBD-GAN described in Non-Patent Document 1 was used as a comparative example.
As an evaluation method, a method was adopted in which the accuracy of a converter from image data to depth data using data pairs generated by the data generation system 1 according to the first embodiment and the method of Non-Patent Document 1 as teacher data was compared. Specifically, the degree of agreement between the calculation result by the converter using the data pairs generated by the data generation system 1 according to the first embodiment and the method of Non-Patent Document 1 as teacher data and the calculation result by the converter using a pair of an actual captured image and actual depth data (actual pair data) as teacher data was compared. Scale-Invariant Depth Error (SIDE) was used as the evaluation scale. The smaller the SIDE value, the better the performance.

As a result, it was confirmed that the data generation system 1 according to the first embodiment has a smaller SIDE value than the method of Non-Patent Document 1. That is, it was confirmed that the data generation system 1 according to the first embodiment has a higher degree of agreement with the conversion result learned using actual pair data than the method of Non-Patent Document 1. Note that the method of Non-Patent Document 1 requires a clue regarding the viewpoint to be acquired in advance, whereas the data generation system 1 according to the first embodiment does not require such information. In addition, while the method of Non-Patent Document 1 generates a pair of image data and depth data, the data generation system 1 according to the first embodiment can obtain a set of a deep depth-of-field image I ^g _d , depth data D _g, and a shallow depth-of-field image I ^g _s .

Second Embodiment
The data generation system 1 according to the first embodiment generates the shallow depth-of-field image I ^g _s using a depth-of-field effect renderer R having optical constraints based on the light space, whereas the data generation system 1 according to the second embodiment generates the shallow depth-of-field image I ^g _s using a depth-of-field effect renderer R having optical constraints based on the relationship between depth and the magnitude of the bokeh effect.

The depth of field effect renderer R of the second embodiment performs calculations using the kernel k expressed by the following equations (8) and (9).

In formula (8), [.] represents Iverson's notation, and takes the value 1 if the condition in the brackets is true, and 0 if it is false. m represents the depth of the focus position based on the position of the subject. A depth m of zero indicates that the subject is in focus. A positive value for depth m indicates that the focus position is located in front of the subject. A negative value for depth m indicates that the focus position is located behind the subject. By this definition, k(hat)(x,m) has a value of 1 only in the center within a radius of |m|, and a value of 0 elsewhere. In other words, the larger the depth |m|, the larger the size of the circular part in the center, and the greater the blur effect when this kernel is convolved with the image. Formula (9) represents the process of normalizing k(hat)(x,m) so that the sum of the values of the elements of k(x,m) is 1.

In the second embodiment, the three-dimensional data generator 135 generates a probability distribution P(x, m) of depth information as a preliminary step of calculating ^Dg . Here, P(x, m) represents the probability of existence of a depth at depth m of position coordinate x. In other words, the sum of the existence probabilities P(x, m) for each coordinate x is 1. When P(x, m) is obtained, ^Dg (x) can be obtained by calculating the maximum depth m for each coordinate x for P(x, m), for example, as shown in formula (10).

When performing the calculation of formula (10), the three-dimensional data generation unit 135 may perform smoothing or the like on P(x, m) as preprocessing. Then, the conversion unit 136 can synthesize a shallow depth-of-field image I ^g _s by multiplying the result of convolving the kernel k(x, m) and the deep depth-of-field image I ^g _d by the probability distribution P(x, m) of the depth data and taking the sum, as shown in the following formula (11).

In formula (11), the operator * represents a convolution operation. In this way, the operation using the depth of field effect renderer R according to the second embodiment is restricted to a convolution with a kernel k whose shape is determined in advance, and a weighted sum thereof. This allows the conversion unit 136 to realize an optically consistent conversion, as in the first embodiment, and to change only the degree of blurring in the conversion process without significantly impairing the content of the image.
In the data generation system 1 according to the first embodiment, the depth of field is adjusted by multiplying the depth data by the degree of mixture s in formula (3), but in the data generation system 2 according to the second embodiment, the degree of depth of field can be adjusted by multiplying m in k(x,m) by the degree of mixture s in formula (11) (i.e., k(x,s·m)). For example, when s is 0, k(x,m) has a kernel of 1 only at the center and 0 elsewhere, so that the output of formula (11) is an image with a deep depth of field.

Third embodiment
Blurring in an image occurs when the subject is in front of the focal position and when it occurs because the subject is in the back. However, it is difficult to determine from image data whether blurring occurs because the subject is in front or in the back. Therefore, in the data generation system 1 according to the third embodiment, an objective function used for updating the model is calculated based on heuristics that a subject (object) in focus is likely to be near the center of the image, and objects near the object are often background.

The calculation unit 138 according to the third embodiment calculates the deep learning criterion L _p expressed by the following formulas (12) and (13), in addition to the adversarial learning criterion L _AR-GAN expressed by formula (7).

In formula (12), r indicates the distance from the center of the image, and r _th and g are hyperparameters that determine the size and depth of the object, respectively. The pre-depth data D _p shown in formula (12) indicates that the focus position is within a radius r _th , and the farther the distance from the center is from r _th , the deeper the focus position is. In formula (13), λ _p is a hyperparameter that represents the weight of the depth learning criterion L _{p .} The closer the distance between the depth data D _g and the pre-depth data D _p is to zero, the lower the depth learning criterion L _p is. In other words, the closer the distance between at least a part of the depth data D ^g that is likely to be focused and the depth related to the predetermined focus position is to zero, the lower the depth learning criterion L p is. Note that, in formula (13), the learning criterion based on the L2 distance is used, but is not limited to this, and a learning criterion based on any distance criterion such as the L1 distance or the Wasserstein distance may be used.

In the above example, the depth at the predetermined focus position is set to zero, and the depth at other positions is set to a value that is predetermined to be deeper than the focus position, but is not limited to this. For example, when the depth at each position in a part or the entire region of the image can be predetermined based on heuristics, the depth learning standard L _p may be lower as the distance between the depth data D _g and the predetermined depth at each position approaches zero. In addition, the depth learning standard L _p shown in formula (13) obtains the distance between the depth indicated by the depth data D _g and the predetermined depth for the entire region of the image, but is not limited to this. For example, the depth learning standard L _p may be calculated based on the distance in only a part of the region of the depth data D _g , such as a region within a radius r _th of the depth data D _g or a region determined to be highly likely to be focused based on heuristics. Furthermore, for example, the depth learning standard _Lp may be such that the depth in an area of the depth data _Dg that is outside the radius _rth or an area that is determined to be unlikely to be focused based on heuristics becomes lower as the area approaches a predetermined depth related to the distant view or the near view.

The update unit 139 updates the parameters of the depth generation model G _D based on the sum of the adversarial learning reference L _AR-GAN and the depth learning reference L _p . This promotes the generation of depth data D ^g in which the focus is on the center and the surroundings are the background. The update unit 139 may use the depth learning reference L _p throughout the entire learning process, or may use it until the middle of the learning process. The update unit 139 uses the depth learning reference L _p until a predetermined number of epochs are reached, and then does not use it thereafter, thereby suppressing the negative effects that occur when there is a gap between the actual depth data and the prior depth data D _p .

<<Variation>>
The data generation system 1 according to the third embodiment calculates the preliminary depth data D p based on the formula (12) based on the heuristics that the object is likely to be present near the center of the image and the object near the object is likely to be the background, but is not limited to this. For example, in other embodiments, the _calculation may be based on the heuristics that the object near the object is likely to be the foreground, or the preliminary depth data D _p may be calculated based on the position of the face detected by the pattern matching process based on the heuristics that the human face is likely to be focused. In addition, r _th and g may be parameters that are updated by learning instead of hyperparameters.

Other Embodiments
Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes are possible. That is, in other embodiments, the order of the above-mentioned processes may be changed as appropriate. Also, some of the processes may be executed in parallel.
The data generation system 1 according to the embodiment described above includes the data generation device 11 and the model learning device 13, but may be configured by a single computer.

The data generation system 1 according to the embodiment described above updates the parameters of the image generation model G _I and the depth generation model G _D using GANs, but is not limited to this. For example, the data generation system 1 according to another embodiment may update the parameters of the image generation model G _I and the depth generation model G _D using a learning criterion in any generation model, such as a Variational Autoencoder, a Flow Model, or a Denoising Diffusion Probablistic Model.

In addition, the depth of field effect renderer R according to the above embodiment is a mathematical model that simulates the constraints of the optical system, but is not limited thereto. For example, the depth of field effect renderer R may be a trained model configured by a neural network. In addition, some parameters of the depth of field effect renderer R, such as parameters related to the optical system, may be held as training parameters. In addition, when the depth of field effect renderer R has trainable parameters, it may be trained simultaneously with the image generation model G _I and the depth generation model G _D.

Furthermore, the data generating device 11 according to the above embodiment outputs a set of a deep depth of field image I ^g _d , a shallow depth of field image I ^g _s and depth data D ^g , but is not limited thereto. For example, the data generating device 11 according to another embodiment may output a set of a deep depth of field image I ^g _d and depth data D ^g , and may not output a shallow depth of field image I ^g _s . Furthermore, the data generating device 11 according to another embodiment may output a set of a shallow depth of field image I ^g _s and depth data D ^g , and may not output a deep depth of field image I ^g _d . Furthermore, the data generating device 11 according to another embodiment may output a set of a deep depth of field image I ^g _d and a shallow depth of field image I ^g _s , and may not output depth data D ^g . Furthermore, the data generating device 11 according to another embodiment may output data that integrates at least a part of a set of the deep depth-of-field image I ^g _d , the shallow depth-of-field image I ^g _s and the depth data D ^g . For example, the data generating device 11 may output the deep depth-of-field image I ^g _d and the depth data D ^g as image data including depth information.

Computer Configuration
FIG. 6 is a schematic block diagram illustrating a computer configuration according to at least one embodiment.
The computer 20 includes a processor 21 , a main memory 23 , a storage 25 , and an interface 27 .
The above-mentioned data generating device 11 and model learning device 13 are implemented in a computer 20. The operations of the above-mentioned processing units are stored in the storage 25 in the form of a program. The processor 21 reads the program from the storage 25, expands it in the main memory 23, and executes the above-mentioned processing in accordance with the program. The processor 21 also secures storage areas in the main memory 23 corresponding to the above-mentioned storage units in accordance with the program. Examples of the processor 21 include a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and a microprocessor.

The program may be for realizing part of the functions to be performed by the computer 20. For example, the program may be for realizing the functions by combining with other programs already stored in the storage or by combining with other programs implemented in other devices. In another embodiment, the computer 20 may be provided with a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to or instead of the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, part or all of the functions realized by the processor 21 may be realized by the integrated circuit. Such an integrated circuit is also included as an example of a processor.

Examples of storage 25 include a magnetic disk, a magneto-optical disk, an optical disk, a semiconductor memory, etc. Storage 25 may be an internal medium directly connected to the bus of computer 20, or an external medium connected to computer 20 via interface 27 or a communication line. In addition, when this program is distributed to computer 20 via a communication line, computer 20 that receives the program may expand the program in main memory 23 and execute the above-mentioned processing. In at least one embodiment, storage 25 is a non-transitory tangible storage medium.

The program may be for realizing part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that realizes the above-mentioned functions in combination with another program already stored in storage 25.

1...Data generation system 11...Data generation device 111...Model storage unit 112...Latent variable generation unit 113...Image generation unit 114...Three-dimensional data generation unit 115...Conversion unit 116...Output unit 13...Model learning device 131...Learning data storage unit 132...Model storage unit 133...Latent variable generation unit 134...Image generation unit 135...Three-dimensional data generation unit 136...Conversion unit 137...Identification unit 138...Calculation unit 139...Update unit

Claims (10)

an image generation unit that generates first image data simulating a captured image related to the first depth of field by inputting the latent variables into an image generation model that is a machine learning model;
a three-dimensional data generation unit that generates three-dimensional data corresponding to the first image data by inputting the latent variables into a three-dimensional generation model that is a machine learning model;
A conversion unit that generates second image data simulating a captured image related to a second depth of field based on the first image data and the three-dimensional data;
a calculation unit that calculates a learning reference value indicating a degree of similarity between a distribution of the first image data and the second image data and an empirical distribution of an actual captured image;
and an update unit that updates parameters of the image generation model and the three-dimensional generation model based on the learning reference value. The learning device according to claim 1 , wherein the second depth of field is shallower than the first depth of field. The learning device according to claim 1 or 2, wherein the conversion unit generates the second image data by inputting the first image data and the three-dimensional data into a mathematical model that simulates constraints of an optical system. an identification unit that inputs at least one of the first image data and the second image data and the actual captured image into an identification model that is a machine learning model that receives image data as input and determines whether the image data is an actual captured image or image data simulating a captured image, and identifies the input image data;
the learning reference value indicates a degree of difficulty in discrimination by the discrimination unit,
The learning device according to claim 1 , wherein the update unit further updates parameters of the discrimination model based on the learning reference value. The learning device described in any one of claims 1 to 4, wherein the learning reference value is the sum of a first reference value indicating the degree of closeness between the distribution of image data and the empirical distribution of actual captured images, and a second reference value relating to the distance between each position relating to at least a portion of the three-dimensional data and a predetermined depth. The three-dimensional data is depth data representing a depth based on a hyperfocal distance,
The learning device according to claim 1 , wherein the conversion unit varies the second depth of field by multiplying the three-dimensional data or a parameter representing a magnitude of depth by a coefficient. A step of generating first image data simulating a captured image related to a first depth of field by inputting the latent variables into an image generation model that is a machine learning model;
generating three-dimensional data corresponding to the first image data by inputting the latent variables into a three-dimensional generation model that is a machine learning model;
generating second image data simulating a captured image relating to a second depth of field based on the first image data and the three-dimensional data;
calculating a learning reference value indicating a degree of similarity between a distribution of the first image data and the second image data and an empirical distribution of an actual captured image;
updating parameters of the image generation model and the three-dimensional generation model based on the learning reference value;
A method for generating a trained model comprising the steps of: outputting the trained image generation model and the trained three-dimensional generation model. An image generation unit that generates image data simulating a captured image by inputting a latent variable into an image generation model output by the trained model generation method according to claim 7;
A three-dimensional data generation unit that generates three-dimensional data by inputting the latent variables into a three-dimensional generation model output by the trained model generation method according to claim 7;
A data generating device comprising: A step of generating image data simulating a captured image by inputting latent variables into an image generation model output by the trained model generation method according to claim 7;
A step of generating three-dimensional data by inputting the latent variables into a three-dimensional generation model output by the trained model generation method according to claim 7;
A data generation method comprising: A program for causing a computer to function as a learning device according to any one of claims 1 to 6.

Images