JP7580441B2 - Information processing device, information processing method, and information processing program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

Conventionally, techniques have been provided for converting various types of color attribute information related to the colors of an image. For example, a technique has been provided for learning a color conversion model using a pair of an image before and after color conversion.

JP 2021-149825 A

"Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks", Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros <Internet> https://junyanz.github.io/CycleGAN/ (Retrieved December 5, 2022) “Denoising Diffusion Probabilistic Models”, Jonathan Ho, Ajay Jain, and Pieter Abbeel, <Internet> https://arxiv.org/pdf/2006.11239.pdf (Retrieved December 5, 2022) “Palette: Image-to-Image Diffusion Models”, Chitwan Saharia, William Chan, Huiwen Chang, Chris A Lee, Jonathan Ho, Tim Salimans, David J Fleet, and Mohammad Norouzi, <Internet> https://arxiv.org/ pdf/2111.05826.pdf (searched on December 5, 2020)

However, there is room for improvement in the above conventional technology. For example, if a pair of an image before color conversion and an image after color conversion is required, and a pair of an image before color conversion and an image after color conversion does not exist, it is difficult to generate an image that has undergone color conversion corresponding to that pair. Therefore, it is desirable to enable appropriate image generation.

The present application has been made in view of the above, and aims to provide an information processing device, an information processing method, and an information processing program that enable appropriate image generation.

The information processing device according to the present application is characterized by comprising: an acquisition unit that acquires a learning grayscale image obtained by grayscaling a learning image, which is an image of a predetermined object captured, and a label indicating the characteristics of the predetermined object in the learning image; and a learning unit that learns a diffusion model that converts a noise image, which is an image containing noise, so as to approximate the learning image, based on the learning grayscale image and the label acquired by the acquisition unit.

According to one aspect of the embodiment, it is possible to achieve the effect of enabling appropriate image generation.

FIG. 1 is a diagram illustrating an example of information processing according to an embodiment. FIG. 2 is a diagram illustrating another example of information processing according to the embodiment. FIG. 3 is a diagram illustrating an example of the configuration of an information processing system according to the embodiment. FIG. 4 is a diagram illustrating an example of the configuration of an information processing device according to an embodiment. FIG. 5 is a diagram illustrating an example of a model information storage unit according to the embodiment. FIG. 6 is a flowchart illustrating an example of information processing according to the embodiment. FIG. 7 is a flowchart illustrating an example of information processing according to the embodiment. FIG. 8 is a diagram illustrating an example of a hardware configuration.

Below, the information processing device, information processing method, and information processing program according to the present application will be described in detail with reference to the drawings. Note that the information processing device, information processing method, and information processing program according to the present application are not limited to these embodiments. In addition, the same parts in the following embodiments will be denoted by the same reference numerals, and duplicated descriptions will be omitted.

(Embodiment)
[1. Information Processing]
An example of information processing according to the embodiment will be described below with reference to FIG. 1. FIG. 1 is a diagram showing an example of information processing according to the embodiment. Specifically, FIG. 1 shows an example of information processing including a learning process of a model according to the embodiment and a generation process of an image using the model. In FIG. 1, a case will be described as an example in which an information processing device 100 (see FIG. 3) learns (generates) a diffusion model M1 (hereinafter also referred to as "model M1"), which is a diffusion model for generating an image from a noise image, and generates an image using the generated model M1. Note that, in the following, a case will be described as an example in which a grayscale image including an object whose characteristics such as color are converted is used, but the image including the object is not limited to a grayscale image and any image may be used, but this point will be described later.

[1-1. Learning process example]
First, an example of the learning process of the model M1 will be described with reference to Fig. 1. First, data (learning data) used in the learning process of the model M1 will be described.

In FIG. 1, the information processing device 100 acquires data (learning data) to be used in the learning process of the model M1. For example, the information processing device 100 acquires the learning data of the model M1 from the learning data storage unit 121 (see FIG. 4). Note that if the learning data is not registered in the learning data storage unit 121, the information processing device 100 may request the provision of learning data from an external device such as the information providing device 50 (see FIG. 3), and register the data received from the external device in the learning data storage unit 121 as learning data.

In FIG. 1, as an example of data used in the learning process of model M1, an image group IG1 including multiple stage images in which noise is added in stages to a specific image is shown. The information processing device 100 learns model M1 using learning data including the image group IG1 shown in FIG. 1.

Image group IG1 includes image P10, which is an image (learning image) of a specified object. Image P10 is an image of a hoodie as the specified object. Image P10 is at level #0 in terms of noise addition, i.e., an image with no noise added. Note that in Figure 1, colors are shown with hatching for ease of illustration, but the color of the hoodie included in image P10 is blue.

Image group IG1 also includes image P11, which is an image in which noise has been added to image P10. Image P11 is an image in which one level of noise has been added to image P10. Image P11 is an image in which the level of noise addition is level #1, i.e., an image in which one level of noise has been added. For example, image P11 is generated by a noise addition process (process SN1 in FIG. 1) that adds noise to image P10.

Image group IG1 also includes image P12, which is image P11 with noise added. Image P12 is image P11 with one more level of noise added. Image P12 is an image in which the level of noise added is level #2, i.e., an image with two levels of noise added. For example, image P12 is generated by a noise addition process (process SN2 in FIG. 1) that adds noise to image P11.

The image group IG1 also includes an image P13, which is an image in which noise has been added to image P12. For example, a noise addition process (process SN3 in FIG. 1) that adds noise to image P12 generates an image (image P13) at stage #3 in which three stages of noise have been added. Note that because noise is repeated, a description of the intermediate processes will be omitted.

The image group IG1 also includes image P1K-2, which is an image in which noise has been added to image P1K-3 to which noise has been added at K-3 levels. Note that K is an arbitrary number (value), for example, 30, 1000, etc. For example, a noise adding process (process SNK-2 in FIG. 1) that adds noise to image P1K-3 generates an image at level #K-2 (image P1K-2) in which noise has been added at K-2 levels.

Image group IG1 also includes image P1K-1, which is an image in which noise has been added to image P1K-2. Image P1K-1 is an image in which one more level of noise has been added to image P1K-2. Image P1K-1 is an image in which the level of noise addition is level #K-1, i.e., an image to which K-1 levels of noise have been added. For example, image P1K-1 is generated by a noise addition process (process SNK-1 in FIG. 1) that adds noise to image P1K-2.

The image group IG1 also includes image P1K, which is a noise image. For example, image P1K is an image in which one more level of noise has been added to image P1K-1. Image P1K is an image in which the level of noise addition is level #K, i.e., an image to which K levels of noise have been added. For example, image P1K is generated by a noise addition process (process SNK in FIG. 1) that adds noise to image P1K-1. Processes SN1 to SNK in FIG. 1 correspond to the forward process in the diffusion model.

For example, each image in the image group IG1 may be generated by a process (noise addition process) in which Gaussian noise is added to the data, and the data gradually transitions from the actual data to complete Gaussian noise. For example, when K is 1000, each image in the image group IG1 is generated by mixing the data and noise while changing the ratio so that the data becomes complete noise after 1000 noise addition processes. Note that when K is 1000, that is, when the number of times the noise addition process is performed is 1000, this is merely an example, and the number of times the noise addition process is performed may be less than 1000 times, such as 32 times or 1 time, or may be more than 1000 times. Also, each image in the image group IG1 may be generated by the information processing device 100, or an external device such as the information providing device 50.

In this way, the image group IG1 shown in FIG. 1 includes multiple images such as images P10, P11, P12, ... P1K-2, P1K-1, P1K, etc. Below, a case where the information processing device 100 learns the model M1 using the image group IG1 described above will be described as an example. Note that the above is merely an example, and the images included in the image group IG1 may be any images. For example, image P1K may be a noise image such as so-called white noise.

Now, the details of the learning process example will be described. In FIG. 1, the information processing device 100 executes a learning process to learn a model M1 using an image group IG1 and the like (step S1). The information processing device 100 inputs an image containing noise (noise image) and learns the model M1 to output (predict) an image obtained by performing the previous process, that is, an image in which noise has been removed by one step from the noise image. For example, the information processing device 100 learns the model M1 to input a noise image, a grayscale image obtained by grayscaling an original image (an image in a state in which noise has not been added) corresponding to the noise image, and a label indicating features such as the color of an object contained in the original image.

In this way, the information processing device 100 trains the model M1 to generate an image in which one stage of noise has been removed from a noise image at one stage. In FIG. 1, the information processing device 100 trains the model M1 using a grayscale image (also called a "training grayscale image") obtained by grayscaling the image P10, and a label (also called a "training label") indicating that the color of the hoodie in the image P10 is blue. In this way, the information processing device 100 can train the model M1 as a conditional diffusion model by learning information (image) indicating an object to be included in the image to be generated and a label indicating the characteristics of the object as inputs.

For example, the information processing device 100 executes a learning process (process SR1 in FIG. 1) in which the model M1 is trained to output image P1K-1 using image P1K, a training grayscale image (e.g., grayscale image GS1), and training labels as input. In this way, the information processing device 100 trains the model M1 to generate image P1K-1, which is a noise image at stage #K-1 in the previous process, from image P1K, which is a noise image at stage #K.

In addition, the information processing device 100 executes a learning process (process SR2 in FIG. 1) in which the model M1 is trained to output image P1K-2, which is a noise image of stage #K-2, using image P1K-1, which is a noise image of stage #K-1, a learning grayscale image, and a learning label as input. Similarly, the information processing device 100 executes a learning process (process SR3 in FIG. 1) in which the model M1 is trained to output image P1K-3, which is a noise image of stage #K-3, using image P1K-2, which is a noise image of stage #K-2, a learning grayscale image, and a learning label as input. Note that the processes between process SR3 and process SRK-2 in FIG. 1 are similar to the above-mentioned processes in that they train the model M1 to output an image one stage before the noise addition of the noise image using a noise image as input, and therefore detailed explanations are omitted.

The information processing device 100 executes a learning process (process SRK-2 in FIG. 1) to train the model M1 to output image P12, which is a noise image of stage #2, using as input image P13, which is a noise image of stage #3, a learning grayscale image, and a learning label. The information processing device 100 executes a learning process (process SRK-1 in FIG. 1) to train the model M1 to output image P11, which is a noise image of stage #1, using as input image P12, which is a noise image of stage #2, a learning grayscale image, and a learning label.

Then, the information processing device 100 executes a learning process (process SRK in FIG. 1) to train the model M1 so that it outputs image P10, which is stage #0, that is, an image with no noise added, using image P11, which is a noise image of stage #1, a learning grayscale image, and learning labels as input. Processes SR1 to SRK in FIG. 1 correspond to the generation process (reverse process) in the diffusion model. In FIG. 1, the information processing device 100 trains the model M1 using an image group IG1 including images P10, P11, P12, ... P1K-2, P1K-1, P1K, etc.

In this way, the information processing device 100 learns model M1, which is a model that generates data by inputting a noise image and gradually removing noise. In FIG. 1, the information processing device 100 learns model M1, which receives a noise image, a grayscale image, and a label as input, and gradually removes noise from the noise image to generate data (also called a "converted image") in which the characteristics indicated by the label are reflected in the object included in the grayscale image.

In FIG. 1, the information processing device 100 learns a model M1 that generates a converted image by gradually removing noise from the noise image so that the hoodie, which is an object contained in the grayscale image, is colored "blue," the color indicated by the label. In this way, the information processing device 100 learns a model M1 that converts the noise image so as to approach the learning image, based on the grayscale image and the label.

The learning process using the image group IG1 shown in FIG. 1 is a part of the learning of the model M1, and the information processing device 100 learns the model M1 using learning data including various image groups other than the image group IG1. For example, the information processing device 100 learns the model M1 using an image group including learning images of various trading objects such as shoes, bags, and hats, not limited to hoodies. For example, the information processing device 100 learns the model M1 using an image group including learning images of various colors such as red, yellow, green, and purple, not limited to blue. The above-mentioned process is merely an example, and the information processing device 100 may learn the model M1 based on a learning method related to a diffusion model disclosed in Non-Patent Documents 2 and 3, etc.

In this way, the information processing device 100 learns a diffusion model that converts a noise image so as to approximate the learning image based on a grayscale image of the learning image including a specific object and a label indicating the characteristics of the specific object. This allows the information processing device 100 to learn a model that converts the characteristics of the specific object in the image. Therefore, the information processing device 100 can enable appropriate image generation.

In the above example, the information processing device 100 inputs the color domain (red, blue, etc.) of the object such as a product contained in the original image as a label along with the grayscale image into the diffusion model to train the diffusion model. For example, if images are collected by color when collecting images, the labels are also known and training data can be easily collected, so the information processing device 100 can easily train the model by self-supervised learning.

[1-2. Example of generation process]
Next, an example of an image generation process using the model M1 will be described. In FIG. 1, for the sake of explanation, an image P10 is used as a target image for generating a converted image (also referred to as a "target image"). However, the target image is not limited to the image P10 and may be any image. For example, the information processing device 100 receives a designation from a user that the image P10 is the target image.

In addition, FIG. 1 shows a case where a label LB1 is used to indicate that the feature to be converted (also called the "target feature") is the color "green," but the target feature is not limited to green and may be any color such as red, brown, etc. For example, the information processing device 100 accepts the label LB1 from the user.

First, the information processing device 100 generates a grayscale image of the target image by performing a process of grayscaling the target image (step S11). The information processing device 100 performs grayscaling of the image by appropriately using various techniques related to grayscaling of the image. In FIG. 1, the information processing device 100 generates a grayscale image GS1 of image P10 by performing a process of grayscaling image P10.

Then, the information processing device 100 executes a generation process to generate a converted image using the noise image, grayscale image, and label as inputs (step S12). For example, the noise image NZ1 may be the same noise image as the image P1K, or may be a different noise image. In FIG. 1, the information processing device 100 generates an image P20 by repeating image generation using the model M1, using the noise image NZ1, grayscale image GS1, and label LB1 as input information IN1 to the initial model M1. Note that in FIG. 1, colors are shown with hatching for ease of illustration, but the color of the hoodie included in image P20 is green.

For example, the information processing device 100 generates an image P20 by repeating the input to the model M1 K times. For example, when K is 1000, the information processing device 100 repeats the input to the model M1 999 times, using as input the image output by the model M1 to which the noise image NZ1, the grayscale image GS1, and the label LB1 have been input.

For example, in the first (first) iteration, the information processing device 100 inputs noise image NZ1, grayscale image GS1, and label LB1 to model M1 and causes model M1 to output image #1. Then, in the second iteration, the information processing device 100 inputs image #1, grayscale image GS1, and label LB1 to model M1 and causes model M1 to output image #2. In the third iteration, the information processing device 100 inputs image #2, grayscale image GS1, and label LB1 to model M1 and causes model M1 to output image #3.

In the 4th to K-1th iterations, the information processing device 100 similarly inputs each of images #3 to #K-2, grayscale image GS1, and label LB1 to model M1, and causes model M1 to output each of images #4 to #K-1. Then, in the Kth iteration, the information processing device 100 inputs image #K-1, grayscale image GS1, and label LB1 to model M1, and causes model M1 to output image P20.

In this way, the information processing device 100 gradually removes noise from the noise image to generate an image P20 in which the characteristics indicated by the label LB1 are reflected in the object included in the grayscale image GS1. In FIG. 1, the information processing device 100 gradually removes noise from the noise image using the model M1 to generate an image P20, which is a converted image in which the hoodie, which is an object included in the grayscale image GS1, is colored "green," which is indicated by the label LB1.

[1-3. Flattening]
Although the above-mentioned process can generate an image that reflects the desired features of a desired target, there is still room for improvement. For example, when learning a diffusion model using an input image (learning image) that has only been converted to grayscale (a grayscale image), when features such as color are converted, the conversion of the features may not be performed appropriately.

For example, when there is a bias in the distribution of pixel values of each pixel of a grayscale image obtained by grayscaling a training image, it may be difficult to learn a diffusion model that appropriately generates a converted image. For example, when there is a bias in the distribution of pixel values (e.g., values from 0 to 255) corresponding to brightness (e.g., gradations from white to black) among multiple types of color attribute information of each pixel of a grayscale image, it may be difficult to learn a diffusion model that appropriately generates a converted image.

Therefore, the information processing device 100 generates an image (also called a "flattened image") in which the distribution of color attribute information of a predetermined type in the learning image is flattened, and the diffusion model is trained using the generated flattened image. An example of this point will be explained using Figure 2. Note that the same points as those in Figure 1 will be appropriately omitted from the explanation by assigning the same reference numerals, etc.

For example, the information processing device 100 generates a grayscale image in which the lightness distribution in the learning image is flattened as a flattened image, and uses the grayscale image in which the lightness distribution is flattened (also called the "flattened grayscale image") to learn the diffusion model. Note that lightness is merely one example of color attribute information to be flattened, and color attribute information to be flattened is not limited to lightness, but this point will be described later.

The histogram HG1 in FIG. 2 is a histogram showing the distribution of brightness of each pixel in the grayscale image GS1 when flattening processing is not performed. The histogram HG1 corresponds to the histogram before flattening processing. In FIG. 2, the shade of hatching in the horizontal bar from white to black indicates the high and low of the pixel value. For example, white (right end) of the histogram HG1 corresponds to the minimum value (e.g., 0) of the pixel value (e.g., brightness) that a pixel can have in the grayscale image, and black (left end) corresponds to the maximum value (e.g., 255) that a pixel can have in the grayscale image. The vertical direction of the histogram HG1 indicates the number (number) of pixels having each pixel value. For example, the low (lower end) of the histogram HG1 indicates that the number of pixels having the corresponding pixel value is 0, and the high (upper end) of the histogram HG1 indicates that the number of pixels having the corresponding pixel value is large. The waveform in the histogram HG1 indicates the number of pixels having each pixel value. In FIG. 2, the histogram HG1 corresponding to the grayscale image GS1 is a histogram in which the number (occurrence frequency) of pixels having certain pixel values (values near the middle between black and white) is high, and the distribution is biased.

Furthermore, histogram HG2 in FIG. 2 is a histogram that shows the distribution of brightness of each pixel in grayscale image GS2 when flattening processing has been performed. Histogram HG2 corresponds to the histogram after flattening processing. Note that explanations of the similarities between histogram HG2 and histogram HG1 will be omitted as appropriate. In FIG. 2, histogram HG2 corresponding to grayscale image GS2 is a histogram in which there is no bias in the distribution of the number of pixels having each pixel value from white to black, and the number of each pixel value is uniform overall.

The flattening process referred to here is a histogram flattening process, which converts the density so that the histogram of a specified pixel value becomes flat (uniform) overall. For example, the flattening process is a process of converting the density so that the histogram of a specified type of color attribute information (such as pixel values corresponding to a specified type) out of multiple types of color attribute information for each pixel in an image becomes flat overall. Various conventional techniques can be used for the histogram flattening process, so a detailed description will be omitted.

For example, when generating a grayscale image from image P10 in FIG. 1, the information processing device 100 performs a flattening process in addition to a grayscale conversion process to generate a grayscale image GS2 that is a flattened grayscale image of image P10. Note that the information processing device 100 may also generate the grayscale image GS2 that is a flattened grayscale image by performing a flattening process on the grayscale image GS1.

The information processing device 100 learns the diffusion model using the grayscale image GS2, which is a flattened grayscale image generated by the flattening process. That is, the information processing device 100 learns the diffusion model using the grayscale image GS2, which is a flattened grayscale image, as an input to the model M1.

In this case, the information processing device 100 uses the flattened grayscale image as the learning grayscale image in the learning process shown in FIG. 1 to learn the model M1. For example, the information processing device 100 uses the grayscale image GS2 as the learning grayscale image in the learning process shown in FIG. 1 to learn the model M1. Note that the learning process using the flattened grayscale image is similar to the learning process shown in FIG. 1 except that the grayscale image input to the model M1 is the flattened grayscale image, and therefore a detailed description thereof will be omitted.

In this way, the information processing device 100 generates a flattened image in which the distribution of brightness, which is a predetermined type of color attribute information, is flattened from a learning image in which multiple types of color attribute information are associated with each pixel. Then, the information processing device 100 learns a diffusion model that converts a noise image so as to approach the learning image, based on the generated flattened image and the label. This allows the information processing device 100 to appropriately learn the model using a flattened image that is not biased in the predetermined type of color attribute information. Therefore, the information processing device 100 can enable appropriate image generation.

For example, if there is a bias in the combination of histograms of grayscale images input for each color label specified when training a diffusion model, and the diffusion model is trained including this bias, when a combination of a blue image turned gray and a white label is input during color conversion, there may be a large difference from the data distribution at the time of training, and conversion may not be successful.

Therefore, when learning the model, the information processing device 100 can properly learn the model by using as an input image an image that has been subjected to grayscale conversion as well as histogram equalization. For example, the diffusion model generated by the information processing device 100 not only colors a grayscale image, but also learns to convert the flattened histogram into a histogram corresponding to a specified color. This allows the information processing device 100 to generate a converted image that is well colored even when the label of an arbitrary color different from that of the original image is replaced during coloring.

When using a flattened grayscale image, in the image generation process using the model M1, the information processing device 100 may generate a converted image using the flattened grayscale image of the target image. In this case, in step S11 of the generation process shown in FIG. 1, the information processing device 100 generates a grayscale image GS2, which is a flattened grayscale image of the image P10, by performing a flattening process in addition to the grayscale conversion process (grayscale conversion process). Then, the information processing device 100 generates a converted image using the flattened grayscale image. Note that the generation process using the flattened grayscale image is similar to the generation process shown in FIG. 1 except that the grayscale image input to the model M1 is a flattened grayscale image, so a detailed description will be omitted.

[1-4. Other (removal of restrictions, services, etc.)]
The above-described process is merely an example, and the information processing device 100 may perform the process using various information as appropriate. In this regard, an example will be described below. For example, in the above-described example, a grayscale image is used as an image including a target to be input to the model M1 (also referred to as a "target captured image"), but the target captured image is not limited to a grayscale image, and any image such as a color image may be used.

In the above example, the case where the color is the conversion target has been described, but the information processing device 100 may convert any object, such as the shape or atmosphere of an object, rather than the color of the object contained in the image. For example, the information processing device 100 converts at least one of the color, shape, and atmosphere of an object contained in the image.

For example, when the shape of an object contained in an image is the object to be converted, the information processing device 100 uses a label indicating the shape of the object (also called a "shape label") as the label to be input to the learning diffusion model. For example, during the learning process, when the shape of the object contained in the image is rectangular, the information processing device 100 uses a shape label indicating a rectangle as the label when learning using that image.

For example, during the generation process, if the information processing device 100 wants to make the shape of an object contained in a target image circular, the information processing device 100 generates a transformed image with a transformed shape by inputting a shape label indicating a circular shape along with the target image into the diffusion model. Note that the above is merely an example, and the information processing device 100 learns the diffusion model using appropriate information according to the conversion object, and generates a transformed image using the learned diffusion model.

For example, in the above example, a case has been described in which, among multiple types of color attribute information, lightness is the target of flattening, but the information processing device 100 may flatten any target related to the image, not limited to lightness. For example, the information processing device 100 may flatten hue, saturation, etc.

In addition, the information processing device 100 may flatten color attribute information according to the color expression format of the image. For example, when the color expression format of the image is RGB (red, green, blue), the information processing device 100 may flatten at least one of the values related to R (red), G (green), and B (blue) associated with each pixel of the image. For example, when the values related to G (green) are to be flattened, the information processing device 100 performs a process of density conversion so that the histogram of pixel values related to G (green) in the image becomes uniform overall.

Also, for example, when the color representation format of an image is CMYK (cyan, magenta, yellow, black (key plate)), the information processing device 100 may target at least one of the values related to C (cyan), M (magenta), Y (yellow), and K (black) associated with each pixel of the image for flattening. For example, when the information processing device 100 targets values related to K (black) for flattening, it performs a density conversion process so that the histogram of pixel values related to K (black) in the image becomes uniform overall.

In addition, the information processing device 100 may provide various services using the generated post-conversion image. In FIG. 1, the information processing device 100 may provide a service using the generated post-conversion image, image P20, to a user (also referred to as the "designating user") who designated the target image, image P10. The information processing device 100 transmits search results of a similar image search similar to image P20 to the terminal device 10 used by the designating user.

The information processing device 100 may provide a search service to the designated user. In this case, the information processing device 100 accepts the image P20 as an image that serves as a search query (also referred to as a "query image"), executes a search process to search for images similar to the query image (image P20) from among various images to be searched, and provides the search results.

For example, the information processing device 100 executes a search process for images similar to image P20 using an index of images (trading target images) that include trading targets such as products listed in an online shopping mall (EC site, etc.), and provides the search results for images similar to image P20. The information processing device 100 provides the designated user with information regarding the search results for images similar to image P20. The information processing device 100 transmits the search results for images similar to image P20 to the terminal device 10 used by the designated user.

As described above, the information processing device 100 learns a diffusion model that converts a noise image to approach a training image according to a grayscale image of a training image in which a specific object is captured and a label indicating the characteristics of the specific object. For example, the information processing device 100 generates multiple stage images in which noise is added to the training image in stages, and learns the diffusion model to generate a stage image in which noise is removed from the stage image by one stage. Also, for example, the information processing device 100 acquires an image from a user, converts the acquired image into grayscale, generates an image in which a desired object is captured using the input characteristics, and uses the generated image to perform an image search for products similar to the specific object in an EC (electronic commerce) service or the like.

The information processing device 100 also generates a flattened image (suggested image) that is a suggested image indicated by a predetermined type of color information and in which the distribution of the predetermined type of color information is flattened, from a learning image in which a predetermined object is photographed and in which multiple types of color attribute information (color information) are associated with each pixel. For example, the information processing device 100 learns a diffusion model that converts a noise image to approach the learning image according to the suggested image and a label indicating the characteristics of the predetermined object. For example, the information processing device 100 may generate a channel estimation model that has learned to flatten an image with blown out highlights and a normal image. In this case, the information processing device 100 may generate a suggested image using the generated channel estimation model.

As described above, the information processing device 100 learns to color objects contained in a grayscale image as desired. For example, the information processing device 100 prepares an original image and a noise image in which noise is added to the original image, and learns to return the noise image to the original image. For example, the information processing device 100 creates and learns various noise images in stages, and the model removes noise multiple times, thereby reproducing the original image.

In addition, when training the diffusion model, the information processing device 100 inputs information indicating the original image (various labels, the name of the subject, the type of scene, the color of the subject, etc.) to train the model. The information processing device 100 inputs various images as training images, noise images, grayscale images of the training images, and labels, and performs noise removal multiple times, thereby training a model that generates an image in which the features indicated by the labels are reflected in the grayscale image.

As described above, the information processing device 100 learns a model to restore a noise image to a learning image from a flattened image obtained by flattening one channel, which is a specific type among multiple types of color attribute information, and a label corresponding to the learning image. For example, the information processing device 100 may generate a blown-out image from a normal image and perform learning using the generated blown-out image.

The information processing device 100 may also switch processing depending on the bias of the histogram of a specific type of color attribute information among the multiple types of color attribute information. For example, if the histogram of the specific type of color attribute information has a bias equal to or greater than a specific standard, the information processing device 100 may generate a converted image using a GAN model such as StarGAN (Generative Adversarial Network). If the histogram of the specific type of color attribute information does not have a bias equal to or greater than a specific standard, the information processing device 100 may generate a converted image using a diffusion model.

In addition, the information processing device 100 may generate a converted image in which the features of only a portion of the area (range) in the image are converted. In this case, the information processing device 100 may learn a model that also takes as input range designation information that designates a portion of the area in the image, and use the learned model to generate a converted image in which only a portion of the area is the target of conversion. For example, the information processing device 100 may learn a model that takes as input mask information that masks areas other than the portion of the image, and use the learned model to generate a converted image in which only a portion of the area is the target of conversion. In addition, the above-mentioned label may be any information, not limited to text information, as long as it is information that indicates the features to be converted. For example, the label may be any image, such as an image indicating an atmosphere, a histogram, or another image.

2. Configuration of Information Processing System
Next, a configuration example of the information processing system 1 will be described with reference to Fig. 3. Fig. 3 is a diagram showing a configuration example of the information processing system according to the embodiment. As shown in Fig. 3, the information processing system 1 includes an information processing device 100, an information providing device 50, and a terminal device 10. The information processing device 100, the information providing device 50, and the terminal device 10 are connected to each other via a predetermined network N so as to be able to communicate with each other by wire or wirelessly. Note that the information processing system 1 shown in Fig. 3 may include a plurality of information processing devices 100, a plurality of information providing devices 50, and a plurality of terminal devices 10.

The information processing device 100 is a computer that learns a model such as a diffusion model. In other words, the information processing device 100 is a learning device that performs learning processing. The information processing device 100 learns a diffusion model based on a learning grayscale image in which a learning image of a specific object is grayscaled, and a label that indicates the characteristics of the specific object in the learning image. For example, the information processing device 100 learns a diffusion model that converts a noise image, which is an image that includes noise, so as to approximate the learning image.

In addition, the information processing device 100 is a generating device that generates a flattened image in which the distribution of a predetermined type of color attribute information is flattened from a learning image in which multiple types of color attribute information are associated with each pixel. For example, the information processing device 100 learns a diffusion model that converts a noise image so as to approach the learning image, based on the flattened image and a label that indicates the characteristics of a predetermined target in the learning image.

The information processing device 100 receives from the user a designation of a target image to be transformed, and generates a transformed image using a grayscale image of the target image designated by the user and a diffusion model. The information processing device 100 also receives designation information from the user that designates post-transformation features, and generates a transformed image that reflects the features designated by the user. The information processing device 100 provides the user with a service that uses the generated transformed image. The information processing device 100 transmits search results of a similar image search that is similar to the transformed image to the terminal device 10 used by the user.

The information processing device 100 is a computer that provides a search service to a user who uses the terminal device 10. In this case, the information processing device 100 has a function as a search server that provides the search service. For example, when the information processing device 100 receives a converted image as an image (query image) that serves as a search query, it executes a search process that searches for images similar to the query image (converted image) from among various images that are the subject of the search, and provides the search results.

The information processing device 100 receives from the terminal device 10 specification information that specifies the target image and features to be transformed, and uses model M1, which is a diffusion model, to generate a transformed image in which the features indicated by the specification information are reflected in the objects included in the target image. For example, the information processing device 100 uses the noise image, the target image, and specification information indicating a color specified by the user as input to model M1, and repeats a process of removing noise to generate a transformed image in which the objects included in the target image are colored in the color indicated by the specification information.

The information processing device 100 executes a search process based on the converted image and provides search results. For example, the information processing device 100 may execute a search process and provide search results using an index of an image (a trading object image) that includes a trading object such as a product that is the subject of the search. For example, the information processing device 100 executes a search process for each trading object listed in an online shopping mall (an EC site, etc.) using a trading object image registered as an image indicating the trading object.

For example, the information processing device 100 searches for trading objects whose images are similar to the query image (converted image) from among trading objects listed in the online shopping mall, and extracts information on trading objects whose converted images are similar to the query image (converted image) as search results for similar images. Note that as long as the information processing device 100 is capable of searching for similar images, it can perform image search processing using any image search technology as appropriate, and a detailed description of image search will be omitted.

The information processing device 100 provides the user with information regarding the search results for similar images. The information processing device 100 transmits the search results for similar images to the terminal device 10 used by the user. For example, the information processing device 100 provides the converted image and price of the transaction object extracted as the search result, information on the store in the online shopping mall that sells the transaction object, etc.

In this way, the information processing device 100 provides information on trading objects whose trading object images are similar to the converted image that is the query image. The information processing device 100 may generate a ranking according to the similarity between the converted image and the trading object image, and provide the generated ranking as a search result. In this case, the information processing device 100 may assign a higher ranking to the trading object images with the highest similarity, generate list information showing the ranking of trading objects arranged in descending order of the ranking, and transmit the generated list information to the terminal device 10 used by the user as a search result for similar images.

Note that the above is merely an example, and the device that provides a search service to the user is not limited to the information processing device 100, but may be another device included in the information processing system 1. For example, the information providing device 50 may provide a search service to the user, or the information processing system 1 may include a search server that provides a search service to the user.

The information providing device 50 is a server device that provides the information processing device 100 with information (data) that can be used for model learning. For example, the information providing device 50 transmits data that is permitted to be used for model learning to the information processing device 100 as learning data. For example, the information providing device 50 publishes public data on the Internet, and transmits the requested data to the external device that is the request destination in response to a request from an external device such as the information processing device 100. For example, the information providing device 50 receives request information requesting data from the information processing device 100, and transmits data corresponding to the request information to the information processing device 100 in response to the reception of the request information.

For example, the information providing device 50 transmits information as learning data to the information processing device 100, which associates a group of images in which noise has been gradually added to an image with feature information that indicates the features of the object contained in the image. For example, the information providing device 50 transmits data as learning data to the information processing device 100, which associates a plurality of staged images in which noise has been gradually added to an image of a trading object traded on an EC site or the like with feature information that indicates the features, such as the color, of the trading object.

The terminal device 10 is an information processing device used by a user. The terminal device 10 is realized, for example, by a smartphone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), etc. The terminal device 10 may be referred to as a user. In other words, the user can also be read as the terminal device 10.

The terminal device 10 accepts operations by the user. In addition, the terminal device 10 transmits various information to the information processing device 100 and the information providing device 50 in response to the user's operations. For example, the terminal device 10 transmits request information requesting various information to the information processing device 100 and the information providing device 50. In addition, the terminal device 10 receives various information from the information processing device 100 and the information providing device 50.

The terminal device 10 transmits an image to be converted (target image) specified by the user to the information processing device 100. The terminal device 10 transmits information indicating the characteristics specified by the user (specified information) to the information processing device 100.

The terminal device 10 receives the search results for similar images from the information processing device 100. For example, the terminal device 10 receives from the information processing device 100 the converted image and price of the transaction object extracted as the search result, information on the store in the online shopping mall that sells the transaction object, etc.

The terminal device 10 receives from the information processing device 100 information on transaction objects whose images are similar to the transformed image that is the query image. The terminal device 10 receives from the information processing device 100 a ranking according to the degree of similarity between the transformed image and the transaction object image. The terminal device 10 assigns a higher rank to the transaction object with the highest degree of similarity between the transformed image and the transaction object image, and receives list information from the information processing device 100 indicating the ranking of the transaction objects arranged in descending order of the rank, as the search result of similar images.

3. Configuration of information processing device
Next, the configuration of the information processing device 100 according to the embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. As shown in Fig. 4, the information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130. Note that the information processing device 100 may have an input unit (e.g., a keyboard, a mouse, etc.) that accepts various operations from an administrator of the information processing device 100, and a display unit (e.g., a liquid crystal display, etc.) that displays various information.

(Communication unit 110)
The communication unit 110 is realized by, for example, a network interface card (NIC) etc. The communication unit 110 is connected to a network via a wired or wireless connection, and transmits and receives information between the terminal device 10 and the information providing device 50.

(Memory unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 4 , the storage unit 120 according to the embodiment has a learning data storage unit 121 and a model information storage unit 122.

(Learning Data Storage Unit 121)
The learning data storage unit 121 according to the embodiment stores various information (learning data) related to data used for learning. The learning data storage unit 121 stores teacher data used for generating a model. The learning data storage unit 121 stores learning data information including information such as data used for input and correct answer information (correct answer image) corresponding to the data.

The learning data storage unit 121 stores learning data that associates a combination of noise images, grayscale images, and labels used for input with image data (correct image) that corresponds to the input.

The learning data storage unit 121 may store various information according to the purpose, not limited to the above. For example, the learning data storage unit 121 stores information indicating the type of input data in association with each input data. For example, the learning data storage unit 121 stores information indicating the type of color attribute to be flattened in association with each data.

(Model information storage unit 122)
The model information storage unit 122 according to the embodiment stores information about a model. For example, the model information storage unit 122 stores information (model data) of a trained model (model) trained (generated) by a learning process. Fig. 5 is a diagram illustrating an example of a model information storage unit according to the embodiment. In the example shown in Fig. 5, the model information storage unit 122 includes items such as "model ID", "purpose", and "model data".

"Model ID" indicates identification information for identifying a model. "Use" indicates the use of the corresponding model. "Model data" indicates the data of the model. Figure 5 shows an example in which conceptual information such as "MDT1" is stored in "model data", but in reality, various information that constitutes the model is included, such as information on the model configuration (network configuration) and information on parameters. For example, "model data" includes information including the nodes in each layer of the network, the functions employed by each node, the connection relationships between the nodes, and the connection coefficients set for the connections between the nodes.

In FIG. 5, the model (model M1) identified by the model ID "M1" has an application of "image generation," which indicates that, for example, model M1 is a model used to convert or color an object in an image of the object. Also, it indicates that the model data of model M1 is model data MDT1.

The model information storage unit 122 may store various types of information according to the purpose, not limited to the above. Models such as model M1 may be expected to be used as program modules that are part of artificial intelligence software. Models such as model M1 may be programs.

(Control unit 130)
Returning to the explanation of Fig. 4, the control unit 130 is a controller, and is realized, for example, by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) executing various programs (corresponding to examples of information processing programs such as a learning program and a generation program) stored in a storage device inside the information processing device 100 using a RAM as a working area. The control unit 130 is also a controller, and is realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

As shown in FIG. 4, the control unit 130 has an acquisition unit 131, a generation unit 132, a learning unit 133, a processing unit 134, and a provision unit 135, and realizes or executes the functions and actions of the information processing described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 4, and may be other configurations as long as they perform the information processing described below.

(Acquisition unit 131)
The acquisition unit 131 acquires various information. The acquisition unit 131 acquires various information from an external information processing device such as the terminal device 10 or the information providing device 50. The acquisition unit 131 may receive various information indicated in the learning data storage unit 121, the model information storage unit 122, or the like from the external information processing device.

The acquisition unit 131 acquires various information from the storage unit 120. The acquisition unit 131 acquires various information from the learning data storage unit 121, the model information storage unit 122, etc. The acquisition unit 131 receives various information from an external device.

The acquisition unit 131 receives various information from the terminal device 10 used by the user. For example, the acquisition unit 131 receives a grayscale image including an object to be color converted from the terminal device 10. For example, the acquisition unit 131 receives a grayscale image specified by the user from the terminal device 10. For example, the acquisition unit 131 receives feature information indicating features specified by the user using the terminal device 10 from the terminal device 10.

The acquisition unit 131 receives various information from the information providing device 50. For example, the acquisition unit 131 receives information used for learning from the information providing device 50. For example, the acquisition unit 131 receives learning data used for learning the model M1 from the information providing device 50.

The acquisition unit 131 acquires a learning grayscale image in which a learning image, which is an image of a specific object, is grayscaled. The acquisition unit 131 acquires a label indicating the characteristics of the specific object in the learning image. The acquisition unit 131 acquires a label indicating the characteristics including at least one of the color, shape, and atmosphere of the specific object in the learning image. The acquisition unit 131 acquires a label indicating the color of the specific object in the learning image.

The acquisition unit 131 acquires one noise image. The acquisition unit 131 acquires multiple stage images in which noise is added to a learning image in stages. The acquisition unit 131 acquires multiple stage images including a first stage image and a second stage image in which one stage of noise is added to the first stage image. The acquisition unit 131 acquires multiple stage images including a third stage image in which one stage of noise is added to the second stage image.

The acquisition unit 131 acquires a target image to be converted from a user. The acquisition unit 131 acquires specification information that specifies features from the user.

(Generation unit 132)
The generating unit 132 executes a generation process to estimate various pieces of information. For example, the generating unit 132 executes the generation process based on various pieces of information acquired by the acquiring unit 131. The generating unit 132 executes the generation process based on various pieces of information stored in the storage unit 120. The generating unit 132 executes the generation process based on various pieces of information received from an external information processing device. The generating unit 132 executes the generation process based on a model learned by the learning unit 133. The generating unit 132 executes the generation process based on information determined by the processing unit 134.

The generation unit 132 generates a flattened image in which the distribution of a predetermined type of color attribute information is flattened from a learning image, which is an image of a predetermined object and in which multiple types of color attribute information are associated with each pixel. The generation unit 132 generates, as the flattened image, a grayscale image in which the distribution of brightness of the multiple types of color attribute information is flattened.

(Learning Unit 133)
The learning unit 133 learns various information. The learning unit 133 generates various models through a learning process. For example, the learning unit 133 performs a learning process based on information stored in the learning data storage unit 121, etc. The learning unit 133 generates various models based on information stored in the learning data storage unit 121, etc. For example, the learning unit 133 learns a model M1 using the learning data stored in the learning data storage unit 121.

The learning unit 133 learns a diffusion model that converts a noise image, which is an image containing noise, so as to approach the learning image, based on the learning grayscale image and label acquired by the acquisition unit 131. The learning unit 133 learns a diffusion model that converts a noise image, which is an image containing noise, so as to approach the learning image, based on the flattened image generated by the generation unit 132 and a label indicating the feature of a specified target in the learning image.

The learning unit 133 learns a diffusion model that converts a noise image so that it approaches a learning image that includes a specific object of the color indicated by the label. The learning unit 133 learns a diffusion model that converts a noise image acquired by the acquisition unit 131 so that it approaches the learning image.

The learning unit 133 learns a diffusion model using the multiple stage images acquired by the acquisition unit 131. The learning unit 133 learns the diffusion model to generate a stage image that is an image in which noise has been removed by one stage from one of the multiple stage images. The learning unit 133 learns the diffusion model to generate a first stage image in which noise has been removed by one stage from a second stage image. The learning unit 133 learns the diffusion model to generate a second stage image in which noise has been removed by one stage from a third stage image.

For example, the learning unit 133 executes a learning process based on various information acquired by the acquisition unit 131. For example, the learning unit 133 learns a model based on information generated by the generation unit 132. The learning unit 133 executes a learning process based on information from an external information processing device and information stored in the storage unit 120. The learning unit 133 executes a learning process based on information stored in the model information storage unit 122. The learning unit 133 stores the model generated by learning in the model information storage unit 122.

The learning unit 133 performs a learning process. The learning unit 133 performs various types of learning. The learning unit 133 learns various types of information based on the information acquired by the acquisition unit 131. The learning unit 133 learns (generates) a model. The learning unit 133 learns various types of information such as a model. The learning unit 133 generates a model through learning. The learning unit 133 learns the model using various machine learning techniques. For example, the learning unit 133 learns the parameters of the model (network). The learning unit 133 learns the model using various machine learning techniques.

The learning unit 133 generates various learning models such as model M1. The learning unit 133 learns network parameters. For example, the learning unit 133 learns network parameters of various learning models such as model M1. The learning unit 133 generates various learning models such as model M1 by performing a learning process using the learning data stored in the model information storage unit 122. For example, the learning unit 133 generates a model used for generating an image. The learning unit 133 generates various learning models such as model M1 by learning network parameters of various learning models such as model M1.

The learning unit 133 performs a learning process based on the learning data (teacher data) stored in the model information storage unit 122. The learning unit 133 performs a learning process using the learning data stored in the model information storage unit 122 to generate various learning models such as model M1.

For example, the learning unit 133 performs a learning process using a method such as backpropagation so that the image output by the model M1 approaches correct answer information (information indicating a desired generation result) associated with the input information input to the model M1. For example, the learning unit 133 performs a learning process so that the image output by the model M1 to which input information including a noise image is input approaches an image (correct answer information) in which the noise of the input noise image has been removed by one step.

For example, the learning unit 133 adjusts the value of the weight (i.e., the connection coefficient) that is taken into account when values are transmitted between nodes by the learning process. In this way, the learning unit 133 learns the model M1 by processing such as backpropagation that corrects the parameters (connection coefficients) so as to reduce the error between the output in the model M1 and the correct answer information corresponding to the input. For example, the learning unit 133 generates the model M1 by performing processing such as backpropagation to minimize a predetermined loss function. This allows the learning unit 133 to perform a learning process that learns the parameters of the model M1.

The model learning method is not limited to the above-mentioned method, and any known technology can be applied. Each model may be generated using various conventional techniques related to machine learning as appropriate. For example, the model may be generated using a supervised learning machine learning technique such as SVM (Support Vector Machine). Also, for example, the model may be generated using an unsupervised learning machine learning technique. For example, the model may be generated using a deep learning technique. For example, the model may be generated using various deep learning techniques such as DNN (Deep Neural Network), RNN (Recurrent Neural Network), and CNN (Convolutional Neural Network). Note that the above description of the model generation is an example, and the model may be generated by a learning method appropriately selected according to the obtainable information, etc. That is, the learning unit 133 may generate the model M1 by any method as long as the model M1 can be trained to output information corresponding to the correct answer information when input information included in the learning data is input.

As described above, the learning method used by the learning unit 133 is not particularly limited, but for example, learning data that links data (input information) with its correct answer information (output information) may be prepared, and the learning data may be input to a computational model based on a multi-layer neural network for learning. Also, a method based on DNN, such as CNN or 3D-CNN, may be used. Specifically, the learning method used by the learning unit 133 may be a method based on U-net. When targeting time-series data such as voice, the learning unit 133 may use a method based on a recurrent neural network (RNN) or LSTM (Long Short-Term Memory units), which is an extension of RNN.

(Processing Unit 134)
The processing unit 134 processes various types of information. The processing unit 134 performs information processing based on the information stored in the storage unit 120. The processing unit 134 functions as an inference unit that performs inference processing. The processing unit 134 performs inference processing using the model stored in the model information storage unit 122. For example, the learning unit 133 performs inference processing based on the information generated by the generation unit 132.

The processing unit 134 performs inference using the model M1 learned by the learning unit 133. The processing unit 134 performs inference using the model M1, which is a diffusion model. The processing unit 134 performs inference processing to generate an image in which objects in the grayscale image are given colors corresponding to the labels by inputting a noise image, a grayscale image, and labels as input information to the model M1. The processing unit 134 generates a converted image using the diffusion model learned by the learning unit 133.

(Providing Unit 135)
The providing unit 135 provides various information. For example, the providing unit 135 provides various information to the terminal device 10 and the information providing device 50. The providing unit 135 transmits various information to the terminal device 10 and the information providing device 50. The providing unit 135 provides various information based on the various information acquired by the acquiring unit 131. The providing unit 135 provides various information based on the various information generated by the generating unit 132.

The providing unit 135 provides various information based on the various information learned by the learning unit 133. The providing unit 135 provides various information based on the model learned by the learning unit 133. The providing unit 135 provides various information based on the various information processed by the processing unit 134. For example, the providing unit 135 transmits information to the terminal device 10.

The providing unit 135 provides a service using the diffusion model learned by the learning unit 133. The providing unit 135 provides a service using the converted image generated by the diffusion model to the user. The providing unit 135 provides an image search service using the converted image to the user.

The providing unit 135 transmits the result of the image search using the converted image to the terminal device 10 used by the user. The providing unit 135 transmits a similar image similar to the converted image to the terminal device 10 used by the user. The providing unit 135 provides the user with a service using the converted image generated using a grayscale image of the target image and a diffusion model. The providing unit 135 provides the user with a service using the converted image generated using the specified information.

[4. Information processing flow]
Next, a flow of information processing by the information processing device 100 will be described with reference to Fig. 6 and Fig. 7. Fig. 6 and Fig. 7 are flowcharts showing an example of information processing according to the embodiment.

First, the processing example shown in FIG. 6 will be described. As shown in FIG. 6, the information processing device 100 acquires a learning grayscale image obtained by grayscaling a learning image, which is an image of a specific object captured, and a label indicating the characteristics of the specific object in the learning image (step S101).

Based on the training grayscale image and the label, the information processing device 100 learns a diffusion model that converts a noise image, which is an image that contains noise, so as to approximate the training image (step S102).

Next, a processing example shown in FIG. 7 will be described. As shown in FIG. 7, the information processing device 100 generates a flattened image in which the distribution of a predetermined type of color attribute information is flattened from a learning image, which is an image of a predetermined object and in which multiple types of color attribute information are associated with each pixel (step S201).

Based on the flattened image and a label indicating the characteristics of a specific target in the training image, the information processing device 100 learns a diffusion model that converts a noise image, which is an image containing noise, so as to approximate the training image (step S202).

5. Effects
As described above, the information processing device 100 according to the embodiment includes the acquisition unit 131 and the learning unit 133. The acquisition unit 131 acquires a learning grayscale image obtained by grayscaling a learning image, which is an image of a predetermined target, and a label indicating a feature of the predetermined target in the learning image. The learning unit 133 learns a diffusion model that converts a noise image, which is an image including noise, so as to approximate the learning image, based on the learning grayscale image and the label acquired by the acquisition unit 131.

In this way, the information processing device 100 according to the embodiment can learn a model that transforms the features of a specific object in an image by learning a diffusion model that transforms a noise image, which is an image that contains noise, to approach the training image based on a training grayscale image in which a training image of a specific object has been grayscaled and a label indicating the features of the specific object in the training image. Therefore, the information processing device 100 can enable appropriate image generation.

In addition, in the information processing device 100 according to the embodiment, the acquisition unit 131 acquires a label indicating characteristics including at least one of the color, shape, and atmosphere of a specific object in the learning image.

In this way, the information processing device 100 according to the embodiment can learn a model that converts the color, shape, atmosphere, etc. of a specific object in an image by acquiring a label that indicates a feature including at least one of the color, shape, and atmosphere of the specific object in the learning image. Therefore, the information processing device 100 can enable appropriate image generation.

In addition, in the information processing device 100 according to the embodiment, the acquisition unit 131 acquires a label indicating the color of a predetermined object in the learning image. The learning unit 133 learns a diffusion model that converts a noise image so as to approximate a learning image that includes a predetermined object of the color indicated by the label.

In this way, the information processing device 100 according to the embodiment can learn a model that converts the color of a specific object in an image by acquiring a label that indicates the color of the specific object in the learning image. Therefore, the information processing device 100 can enable appropriate image generation.

In addition, in the information processing device 100 according to the embodiment, the acquisition unit 131 acquires one noise image. The learning unit 133 learns a diffusion model that converts the one noise image acquired by the acquisition unit 131 so as to approximate the learning image.

In this way, the information processing device 100 according to the embodiment can appropriately learn the model by acquiring one noise image and learning the diffusion model using the acquired one noise image. Therefore, the information processing device 100 can enable appropriate image generation.

In addition, in the information processing device 100 according to the embodiment, the acquisition unit 131 acquires a plurality of stage images in which noise is added in stages to the learning image. The learning unit 133 uses the plurality of stage images acquired by the acquisition unit 131 to learn a diffusion model.

In this way, the information processing device 100 according to the embodiment can appropriately learn the diffusion model by using a plurality of step images in which noise is gradually added to the learning image. Therefore, the information processing device 100 can enable appropriate image generation.

In addition, in the information processing device 100 according to the embodiment, the learning unit 133 learns a diffusion model to generate a stage image that is an image in which one stage of noise is removed from one of the multiple stage images.

In this way, the information processing device 100 according to the embodiment can appropriately learn the model by learning the diffusion model so as to generate a stage image in which one stage of noise has been removed from one of a plurality of stage images. Therefore, the information processing device 100 can enable appropriate image generation.

In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires a plurality of stage images including a first stage image and a second stage image in which one stage of noise has been added to the first stage image. The learning unit 133 learns a diffusion model so as to generate a first stage image in which one stage of noise has been removed from the second stage image.

In this way, the information processing device 100 according to the embodiment can appropriately learn the model by learning the diffusion model so as to generate a first-stage image in which one level of noise has been removed from a second-stage image in which one level of noise has been added to the first-stage image. Therefore, the information processing device 100 can enable appropriate image generation.

In addition, in the information processing device 100 according to the embodiment, the acquisition unit 131 acquires a plurality of stage images including a third stage image in which one stage of noise has been added to the second stage image. The learning unit 133 learns a diffusion model so as to generate a second stage image in which one stage of noise has been removed from the third stage image.

In this way, the information processing device 100 according to the embodiment can appropriately learn the model by learning the diffusion model so as to generate a second-stage image in which one level of noise has been removed from a third-stage image in which one level of noise has been added to the second-stage image. Therefore, the information processing device 100 can enable appropriate image generation.

The information processing device 100 according to the embodiment also includes a providing unit 135. The providing unit 135 provides a service using the diffusion model learned by the learning unit 133.

In this way, the information processing device 100 according to the embodiment can provide appropriate services by providing services using a diffusion model.

In addition, in the information processing device 100 according to the embodiment, the providing unit 135 provides a service to a user using a transformed image generated by the diffusion model.

In this way, the information processing device 100 according to the embodiment can provide the user with an appropriate service by providing the user with a service that uses a transformed image generated by a diffusion model.

In addition, in the information processing device 100 according to the embodiment, the providing unit 135 provides the user with an image search service using the converted image.

In this way, the information processing device 100 according to the embodiment can provide the user with an appropriate search service by providing the user with an image search service using the converted image.

In addition, in the information processing device 100 according to the embodiment, the providing unit 135 transmits the results of the image search using the converted image to the terminal device 10 used by the user.

In this way, the information processing device 100 according to the embodiment can provide the user with appropriate search results by transmitting the results of an image search using the converted image to the terminal device 10 used by the user.

In addition, in the information processing device 100 according to the embodiment, the providing unit 135 transmits a similar image that is similar to the converted image to the terminal device 10 used by the user.

In this way, the information processing device 100 according to the embodiment can provide the user with appropriate search results for similar images by transmitting similar images that are similar to the converted image to the terminal device 10 used by the user.

In addition, in the information processing device 100 according to the embodiment, the acquisition unit 131 acquires a target image to be converted from a user. The provision unit 135 provides the user with a service that uses a converted image generated using a grayscale image of the target image and a diffusion model.

In this way, the information processing device 100 according to the embodiment can provide the user with an appropriate service by acquiring a target image to be converted from the user and providing the user with a service using a converted image generated using a grayscale image of the acquired target image and a diffusion model.

In addition, in the information processing device 100 according to the embodiment, the acquisition unit 131 acquires specification information for specifying features from the user. The provision unit 135 provides the user with a service that uses the converted image generated using the specification information.

In this way, the information processing device 100 according to the embodiment can provide the user with an appropriate service by acquiring specification information that specifies characteristics from the user and providing the user with a service that uses a converted image generated using the acquired specification information.

The information processing device 100 according to the embodiment also includes a processing unit 134. The processing unit 134 generates a converted image using the diffusion model learned by the learning unit 133.

In this way, the information processing device 100 according to the embodiment can generate an appropriate image by generating a transformed image using a diffusion model.

6. Hardware Configuration
The information processing device 100 and the terminal device 10 according to the above-described embodiment are realized by a computer 1000 having a configuration as shown in Fig. 8, for example. The information processing device 100 will be described below as an example. Fig. 8 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which a calculation device 1030, a primary storage device 1040, a secondary storage device 1050, an output I/F (Interface) 1060, an input I/F 1070, and a network I/F 1080 are connected by a bus 1090.

The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The arithmetic device 1030 is realized, for example, by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like.

The primary storage device 1040 is a memory device such as a RAM (Random Access Memory) that primarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device in which data used by the arithmetic device 1030 for various calculations and various databases are registered, and is realized by a ROM (Read Only Memory), a HDD (Hard Disk Drive), a SSD (Solid State Drive), a flash memory, or the like. The secondary storage device 1050 may be an internal storage device or an external storage device. The secondary storage device 1050 may be a removable storage medium such as a USB memory or a SD (Secure Digital) memory card. The secondary storage device 1050 may be a cloud storage device (online storage device), a NAS (Network Attached Storage), a file server, or the like.

The output I/F 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various types of information, such as a display, projector, printer, etc., and is realized by a connector conforming to a standard such as USB (Universal Serial Bus), DVI (Digital Visual Interface), or HDMI (registered trademark) (High Definition Multimedia Interface). The input I/F 1070 is an interface for receiving information from various input devices 1020, such as a mouse, keyboard, keypad, button, scanner, etc., and is realized by a USB, etc.

In addition, the output I/F 1060 and the input I/F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. That is, the output device 1010 and the input device 1020 may be wireless devices.

In addition, the output device 1010 and the input device 1020 may be integrated, such as a touch panel. In this case, the output I/F 1060 and the input I/F 1070 may also be integrated as an input/output I/F.

The input device 1020 may be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

The network I/F 1080 receives data from other devices via the network N and sends it to the computing device 1030, and also transmits data generated by the computing device 1030 to other devices via the network N.

The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output I/F 1060 and the input I/F 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

For example, when the computer 1000 functions as the information processing device 100, the arithmetic unit 1030 of the computer 1000 executes a program loaded onto the primary storage device 1040 to realize the functions of the control unit 130. The arithmetic unit 1030 of the computer 1000 may also load a program acquired from another device via the network I/F 1080 onto the primary storage device 1040 and execute the loaded program. The arithmetic unit 1030 of the computer 1000 may also cooperate with other devices via the network I/F 1080 and use the functions and data of a program by calling them from other programs of the other devices.

[7. Other]
Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

The above-described embodiments and variations can be combined as appropriate to the extent that they do not cause inconsistencies in the processing content.

The above-mentioned "section, module, unit" can be read as "means" or "circuit." For example, an acquisition unit can be read as an acquisition means or an acquisition circuit.

REFERENCE SIGNS LIST 1 Information processing system 100 Information processing device 121 Learning data storage unit 122 Model information storage unit 130 Control unit 131 Acquisition unit 132 Generation unit 133 Learning unit 134 Processing unit 135 Providing unit 10 Terminal device 50 Information providing device N Network

Claims (15)

an acquisition unit that acquires a learning grayscale image that is a learning image obtained by capturing a predetermined object and a label that indicates a feature of the predetermined object in the learning image;
a learning unit that learns a diffusion model that converts a noise image, which is an image including noise, so as to approximate the learning image based on the learning grayscale image and the label acquired by the acquisition unit;
a provision unit that provides a user with an image search service using a transformed image generated by the diffusion model trained by the learning unit;
An information processing device comprising: The acquisition unit is
The information processing apparatus according to claim 1 , further comprising: acquiring the label indicating the feature including at least one of a color, a shape, and an atmosphere of the predetermined object in the learning image. The acquisition unit is
obtaining the label indicating the color of the predetermined object in the learning image;
The learning unit is
The information processing apparatus according to claim 1 , further comprising: learning the diffusion model for converting the one noise image so as to approximate the learning image including the predetermined target of the color indicated by the label. The acquisition unit is
acquiring the one noise image;
The learning unit is
The information processing apparatus according to claim 1 , further comprising: learning the diffusion model that converts the one noise image acquired by the acquisition unit so as to approach the learning image. The acquisition unit is
Obtaining a plurality of step images by adding noise to the learning image in a stepwise manner;
The learning unit is
The information processing apparatus according to claim 4 , wherein the diffusion model is trained using the plurality of stage images acquired by the acquisition unit. The learning unit is
The information processing apparatus according to claim 5 , wherein the diffusion model is trained to generate a step image that is an image in which noise is removed by one step from one of the plurality of step images. The acquisition unit is
acquiring the plurality of stage images including a first stage image and a second stage image obtained by adding one stage noise to the first stage image;
The learning unit is
The information processing apparatus according to claim 5 , wherein the diffusion model is trained to generate the first-stage image by removing noise from the second-stage image by one stage. The acquisition unit is
acquiring the plurality of stage images including a third stage image in which one stage noise is added to the second stage image;
The learning unit is
The information processing apparatus according to claim 7 , wherein the diffusion model is trained to generate the second stage image by removing noise from the third stage image by one stage. The providing unit is
The information processing apparatus according to claim 1 , further comprising: a terminal device that transmits a result of an image search using the converted image to the user. The providing unit is
The information processing apparatus according to claim 1 , further comprising: a processor configured to transmit an image similar to the converted image to a terminal device used by the user. The acquisition unit is
Obtaining a target image to be converted from the user;
The providing unit is
The information processing apparatus according to claim 1 , further comprising: providing the user with a service using the converted image generated using the grayscale image of the target image and the diffusion model. The acquisition unit is
acquiring specification information specifying a characteristic from the user;
The providing unit is
The information processing apparatus according to claim 1 , further comprising: a service that uses the converted image generated using the specification information and is provided to the user. a processing unit that generates a transformed image by using the diffusion model trained by the training unit;
The information processing apparatus according to claim 1 , further comprising: 1. A computer-implemented information processing method, comprising:
an acquisition step of acquiring a learning grayscale image in which a learning image, which is an image of a predetermined object, is grayscaled, and acquiring a label indicating a feature of the predetermined object in the learning image;
a learning process for learning a diffusion model that converts a noise image, which is an image including noise, so as to approximate the learning image based on the learning grayscale image acquired by the acquisition process and the label;
a providing step of providing an image search service to a user using a transformed image generated by the diffusion model trained in the training step;
13. An information processing method comprising: An acquisition step of acquiring a learning grayscale image in which a learning image, which is an image of a predetermined object, is grayscaled, and acquiring a label indicating a feature of the predetermined object in the learning image;
a learning step of learning a diffusion model that converts a noise image, which is an image containing noise, so as to approximate the learning image based on the learning grayscale image acquired by the acquisition step and the label;
a providing step of providing a user with an image search service using a transformed image generated by the diffusion model trained by the learning step;
An information processing program characterized by causing a computer to execute the above.

Images