JP7619576B2 - Information processing device and information processing method - Google Patents

Description

The present invention relates to an information processing device and an information processing method.

Conventionally, there is known technology for generating captions (also called caption text, hereinafter referred to as "video description text") from videos. For example, a technology is known in which a video captured by a surveillance camera is input to a multi-layer neural network that outputs elements contained in an image as words, and a description of the video is generated.

JP 2018-101317 A

However, the above-mentioned conventional technology simply inputs video captured by a surveillance camera into a multi-layer neural network that outputs elements contained in the image as words, and generates a description of the video, so it is not necessarily possible to generate a variety of video descriptions that correspond to the frame images of interest.

The present application aims to provide an information processing device and information processing method that can generate a variety of video descriptions according to a frame image of interest.

The information processing device according to the present application includes a video generation unit that generates a training video based on an image and text dataset including a pair of a captured image and an image description that is a text that describes the content of the captured image, an extraction unit that extracts training frame features that are features of each of a plurality of frame images that constitute the training video, a determination unit that determines training weights that are weights corresponding to each of a plurality of frame images that constitute the training video, and a model generation unit that generates a text generation model that is a machine learning model trained to generate training video descriptions that are text that describes the content of the training video, the training video descriptions having features that correspond to the training frame features weighted by the training weights, based on the training frame features and the training weights.

The information processing device according to the present application also includes an acquisition unit that acquires a sentence generation model that is a machine learning model trained to generate a learning video description that is a sentence that describes the content of a learning video, the learning video description having features corresponding to the learning frame features weighted by the learning weights, based on learning frame features that are features of each of a plurality of frame images that constitute a learning video generated based on an image and text dataset including a pair of a captured image and an image description that is a sentence that describes the content of the captured image, and learning weights that are weights corresponding to each of the plurality of frame images that constitute the learning video; an extraction unit that extracts target frame features that are features of each of a plurality of frame images that constitute a target video that is a video to be processed; a determination unit that determines target weights that are weights corresponding to each of the plurality of frame images among the plurality of frame images that constitute the target video, such that the weight corresponding to a designated frame image designated by a user is greater than the weights corresponding to other frame images other than the designated frame image; and a sentence generation unit that inputs the target frame features weighted by the target weights to the sentence generation model based on the target frame features and the target weights to generate a target video description that is a sentence that describes the content of the target video.

According to one aspect of the embodiment, it is possible to generate a variety of video descriptions according to the frame image of interest.

FIG. 1 is a diagram illustrating an example of the configuration of an information processing apparatus according to an embodiment. FIG. 2 is a diagram illustrating an example of information processing related to the pre-learning method according to the embodiment. FIG. 3 is a diagram illustrating an example of information processing related to the first additional learning method according to the embodiment. FIG. 4 is a diagram for explaining a method of weighting a frame feature amount by a weight according to the embodiment. FIG. 5 is a diagram illustrating an example of information processing related to the second additional learning method according to the embodiment. FIG. 6 is a diagram for explaining a method for calculating a similarity according to the embodiment. FIG. 7 is a diagram illustrating an example of information processing related to the inference method according to the embodiment. FIG. 8 is a diagram illustrating a conditional generative adversarial network (CGAN) that is an example of a sentence generation model according to the embodiment. FIG. 9 is a diagram illustrating a conditional variational autoencoder (CVAE) which is an example of a sentence generation model according to the first modified example. FIG. 10 is a diagram showing a conditional diffusion model, which is an example of a sentence generation model according to the second modified example. FIG. 11 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device.

Below, a detailed description will be given of a form for implementing the information processing device and information processing method according to the present application (hereinafter, referred to as an "embodiment") with reference to the drawings. Note that the information processing device and information processing method according to the present application are not limited to this embodiment. Furthermore, the same components in each of the following embodiments are given the same reference numerals, and duplicated descriptions will be omitted.

(Embodiment)
1. Introduction
Conventionally, a technique for generating a video description, which is a text that explains the contents of a video, from a video is known. For example, a machine learning model that generates a video description from a video is known. Here, it is known that the content of the video description generated by the machine learning model that generates a video description from a video varies depending on which part of the video is focused on.

In recent years, technology related to generative models has become known. A generative model is a model of the data generation process. A generative model is a machine learning model that can learn from training data and generate data similar to the training data. Technology related to conditional generative models has also become known. A conditional generative model is a machine learning model that can change the data generation process by changing the conditions, and generate diverse, high-quality data.

Here, the conditions in a conditional generative model refer to conditions that must be satisfied by the features of the target data generated by the conditional generative model. In other words, the conditions here refer to conditions related to the type and attributes of the target data generated by the conditional generative model (e.g., the type and attributes of features appearing in the target data). For example, if the target data generated by the conditional generative model is an image, the conditions may be information indicating the attributes and type of objects contained in the image. Specifically, by inputting a condition vector, which is a vector input as a condition to the conditional generative model, into the conditional generative model as input information, data having features corresponding to the condition vector can be generated. For example, the condition vector may be a vector corresponding to information indicating the condition.

For example, one example of a conditional generative model is the Conditional GAN (CGAN) (Reference: Paper title "Conditional Generative Adversarial Nets", <Internet> https://arxiv.org/pdf/1411.1784.pdf (Retrieved February 16, 2023)). CGAN is also called a conditional generative adversarial network, and is a machine learning model that has been improved to allow conditions to be given to GAN (generative adversarial network), which generates specific data from noise.

Another example of a conditional generative model is the Conditional Variational Auto Encoder (CVAE) (reference: Paper title: "Semi-supervised Learning with Deep Generative Models", <Internet> https://proceedings.neurips.cc/paper/2014/file/d523773c6b194f37b938d340d5d02232-Paper.pdf (searched February 16, 2023)). CVAE is also called a conditional variational autoencoder, and is a machine learning model that has been improved to allow conditions to be given to a VAE (variational autoencoder), which generates data according to a latent representation.

The Diffusion Model (reference: Paper title: "Denoising Diffusion Probabilistic Models", <Internet> https://arxiv.org/pdf/2006.11239.pdf (searched February 16, 2023)) is known as an example of a conditional generative model. The Diffusion Model learns the process of gradually restoring data from noise. The Diffusion Model is generally called the diffusion model, but in practical applications it is possible to generate data with conditions given to it.

Other examples of conditional generative models include GLIDE (reference: paper title “GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models”, <Internet> https://arxiv.org/pdf/2112.10741.pdf (searched February 16, 2023)), DALL-E 2 unCLIP (reference: paper title “Hierarchical Text-Conditional Image Generation with CLIP Latents”, <Internet> https://arxiv.org/pdf/2204.06125.pdf (searched February 16, 2023)), Imagen (reference: paper title “Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding”, <Internet> https://arxiv.org/pdf/2205.11487.pdf (searched February 16, 2023)), and Parti (reference: paper title “Scaling Autoregressive Models for "Content-Rich Text-to-Image Generation", <Internet> https://arxiv.org/pdf/2206.10789.pdf (Retrieved February 16, 2023) is known.

The information processing device according to this embodiment inputs weighted frame features as conditions to a conditional generative model based on weights corresponding to each of a plurality of frame images constituting a video and frame features that are features of each of a plurality of frame images constituting a video. The information processing device also generates a video description having features corresponding to the weighted frame features. As a result, the information processing device can control which part of the video (which frame image) to emphasize in generating a video description by using the weighted frame features corresponding to each frame image as a condition to be given to the conditional generative model. The information processing device can also generate a variety of video descriptions according to the frame image of interest. The information processing device can also use the weighted frame features corresponding to each frame image as a condition to be given to the conditional generative model, making it possible to reflect the time-series information of the video in natural language generation.

2. Configuration of information processing device
An example of the configuration of an information processing device 100 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the information processing device 100 according to an embodiment. The information processing device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

(Communication unit 110)
The communication unit 110 is realized by a network interface card (NIC), an antenna, etc. The communication unit 110 is connected to various networks via wired or wireless communication, and transmits and receives information to and from other information processing devices other than the information processing device 100, for example.

(Memory unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. Specifically, the storage unit 120 stores various data. For example, the storage unit 120 stores learning data used for learning the sentence generation model. The storage unit 120 also stores various programs. For example, the storage unit 120 stores information related to the sentence generation model generated by the model generation unit 134.

(Control unit 130)
The control unit 130 is a controller, and is realized, for example, by a central processing unit (CPU) or a micro processing unit (MPU) executing various programs stored in a storage device inside the information processing device 100 using a RAM as a working area. The control unit 130 is also a controller, and is realized, for example, by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

The control unit 130 has a video generation unit 131, an extraction unit 132, a determination unit 133, a model generation unit 134, an acquisition unit 135, and a sentence generation unit 136 as functional units, and may realize or execute the information processing actions described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 1, and may be other configurations that perform the information processing described below. Also, each functional unit indicates a function of the control unit 130, and does not necessarily have to be physically distinct.

(Video Generation Unit 131)
The video generator 131 generates a learning video based on an image and text data set that includes a pair of a captured image and an image caption that is a sentence that explains the content of the captured image.

(Extraction Unit 132)
The extraction unit 132 extracts learning frame features, which are features of each of a plurality of frame images constituting the learning moving image. The extraction unit 132 also extracts target frame features, which are features of each of a plurality of frame images constituting the target moving image, which is a moving image to be processed.

(Determination unit 133)
The determination unit 133 determines learning weights, which are weights corresponding to each of a plurality of frame images constituting the learning video. Also, the determination unit 133 determines target weights, which are weights corresponding to each of a plurality of frame images constituting the target video, such that the weight corresponding to a designated frame image designated by a user among the plurality of frame images constituting the target video is made larger than the weights corresponding to other frame images other than the designated frame image.

(Model Generation Unit 134)
The model generation unit 134 generates a sentence generation model, which is a machine learning model trained to generate a training video description, which is a sentence that describes the content of the training video, based on the training frame features and the training weights, and has features corresponding to the training frame features weighted by the training weights. Here, the sentence generation model may be a conditional generative model. For example, the sentence generation model may be a conditional generative adversarial network (CGAN), a conditional variational autoencoder (CVAE), or a conditional diffusion model.

The model generation unit 134 also generates a pre-trained sentence generation model by pre-training the sentence generation model. Next, the model generation unit 134 generates an additionally trained sentence generation model by additionally training the pre-trained sentence generation model. The model generation unit 134 stores information about the generated additionally trained sentence generation model in the storage unit 120. Note that, hereinafter, the additionally trained sentence generation model may be simply referred to as a "sentence generation model".

(Acquisition unit 135)
The acquisition unit 135 acquires a sentence generation model, which is a machine learning model trained to generate a training video description, which is a sentence that describes the content of a training video, having features corresponding to the training frame features weighted by the training weights, based on training frame features that are features of each of a plurality of frame images constituting a training video generated based on an image and sentence dataset including a pair of a captured image and an image description that is a sentence that describes the content of the captured image, and training weights that are weights corresponding to each of a plurality of frame images constituting the training video. Specifically, the acquisition unit 135 acquires the sentence generation model generated by the model generation unit 134. For example, the acquisition unit 135 acquires information on the sentence generation model from the storage unit 120.

(Sentence generation unit 136)
The sentence generation unit 136 inputs the target frame feature weighted by the target weight to a sentence generation model based on the target frame feature and the target weight, and generates a target moving image description, which is a sentence that describes the content of the target moving image. Generate a statement.

[3. Pre-learning methods]
A pre-learning method for a sentence generation model according to an embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of information processing related to the pre-learning method according to an embodiment. Here, pre-learning refers to learning of a model that is performed as a preparation before the model is fully trained by the first additional learning (see Fig. 3) or the second additional learning (see Fig. 5) described later.

As shown in Figure 2, in the pre-learning stage, (1) image features are extracted from each frame image constituting a video included in a video sentence dataset that includes pairs of videos and video descriptions. (2) No weighting is applied to each frame image constituting a video. (3) The image features extracted from each frame image are input as conditions into a sentence generation model, which is a conditional generation model, and the sentence generation model is trained to generate video descriptions having features corresponding to the image features.

Specifically, the extraction unit 132 may acquire a video text dataset #1 including a pair of a captured video for pre-learning (hereinafter, may be referred to as "pre-learning video #1") and a video description corresponding to the captured video for pre-learning (hereinafter, may be referred to as "pre-learning video description #1"). For example, the extraction unit 132 may acquire the video text dataset #1 from an external information processing device via the communication unit 110.

Next, the extraction unit 132 may extract image features of each of the multiple frame images constituting the pre-learning video #1 from each of the multiple frame images constituting the pre-learning video #1 included in the video sentence dataset #1 (step S11). For example, the image features may be a multidimensional vector. In FIG. 2, for simplicity, a case in which there are three frame images constituting the pre-learning video #1 is described, but the number of frame images constituting the pre-learning video #1 may be four or more. In FIG. 2, a first frame image corresponding to the start time of the pre-learning video #1, a second frame image corresponding to a time between the start time and the end time of the pre-learning video #1, and a third frame image corresponding to the end time of the pre-learning video #1 are arranged in chronological order. For example, the extraction unit 132 extracts a feature vector V11 from the first frame image. In addition, the extraction unit 132 extracts a feature vector V12 from the second frame image. In addition, the extraction unit 132 extracts a feature vector V13 from the third frame image. Next, the extraction unit 132 may acquire a vector (V11, V12, V13) of the set of feature vectors V11 to V13 as pre-learning frame feature #1, which is the image feature of each of the multiple frame images that make up the pre-learning video #1.

For example, the extraction unit 132 may extract image features from each of the multiple frame images constituting the video, using any known technology capable of extracting image features from an image. For example, the extraction unit 132 may include an image encoder and extract image features using the image encoder. For example, the extraction unit 132 may include a convolutional neural network (CNN) as the image encoder. Then, the extraction unit 132 may extract image features from each frame image using the CNN. Furthermore, for example, the extraction unit 132 may be equipped with ResNet (Residual Network) (Kaiming He et al., 2015), AlexNet (Krizhevsky et al., 2012), VGGNet (Simonyan et al., 2014), GoogLeNet (Szegedy et al., 2014), SENet (Squeeze-and-Excitation Networks) (Jie Hu et al., 2018), EfficientNet (Tan et al., 2019), or ZFNet (Matthew et al., 2013) developed for object recognition as an image encoder. The extraction unit 132 may extract image features from each frame image using ResNet, AlexNet, VGGNet, GoogLeNet, SENet, EfficientNet, or ZFNet. For example, the extraction unit 132 may include Faster R-CNN (Shaoqing Ren et al., 2015), YOLO (You Look Only Onse) (Joseph Redmon et al., 2015), or SSD (Single Shot MultiBox Detector) (Wei Liu., 2015), which are developed for object detection, as an image encoder. The extraction unit 132 may extract image features from each frame image using Faster R-CNN, YOLO, or SSD.

In this way, the extraction unit 132 extracts pre-learning frame features #1, which are features of each of the multiple frame images that make up a captured video included in video text dataset #1, which includes a pair of a captured video (in the example of Figure 2, pre-learning video #1) and a video description, which is a sentence that explains the content of the captured video (in the example of Figure 2, pre-learning video description #1).

Next, the model generation unit 134 may input the pre-learning frame feature #1 extracted by the extraction unit 132 to the sentence generation model M1 (step S12). For example, the model generation unit 134 may generate a condition vector #1 based on the pre-learning frame feature #1. Next, the model generation unit 134 may combine the generated condition vector #1 with a noise vector (also called a random vector). For example, the model generation unit 134 may use a linear conversion process to adjust the size of the condition vector #1 and the noise vector so that they are the same. Next, the model generation unit 134 may combine the condition vector #1 and the noise vector by adding each element of the condition vector #1 to each element of the noise vector. Alternatively, the model generation unit 134 may combine the condition vector #1 and the noise vector by multiplying each element of the condition vector #1 by each element of the noise vector. Next, the model generation unit 134 may input the combined condition vector #1 and the noise vector as input information to the sentence generation model M1, which is a conditional generation model.

Then, the model generation unit 134 may acquire the video description generated by the sentence generation model M1 in response to the input of the combined condition vector #1 and the noise vector, and output as output information from the sentence generation model M1 (step S13). The model generation unit 134 may train the sentence generation model M1 so as to generate a video description having a feature corresponding to the condition vector #1 based on the condition vector #1. For example, the model generation unit 134 may train the sentence generation model M1 using backpropagation (error backpropagation method) or the like so as to reduce the error between the video description output from the sentence generation model M1 and the pre-training video description #1 included in the video description dataset #1. In this way, the model generation unit 134 may train the sentence generation model M1 so as to generate a video description having a feature corresponding to the pre-training frame feature #1 based on the pre-training frame feature #1.

In this way, the model generation unit 134 generates a pre-trained sentence generation model M1, which is a machine learning model that has been pre-trained to generate a video description (pre-training video description #1 in Figure 2) having features corresponding to the pre-training frame feature #1, based on the pre-training frame feature #1.

[4. First additional learning method]
A first additional learning method according to the embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of information processing related to the first additional learning method according to the embodiment. The first additional learning refers to full-scale model learning performed after the pre-learning described in Fig. 2.

As shown in FIG. 3, in the first additional learning stage, (1) a known video generation model is used to generate a video from images included in an image-sentence dataset including a pair of images (still images) and image captions. Hereinafter, the image (included in the image-sentence dataset) from which the video is generated may be referred to as the "original image". The generated video includes the original image as a frame. (2) Among the frame images constituting the generated video, the frame image corresponding to the original image is set as a focus point, and weights corresponding to each frame image constituting the video are determined. In addition, image features are extracted from each frame image constituting the generated video, and the image features extracted from each frame image are weighted by the weights corresponding to each frame image. (3) The weighted image features are input as conditions to a pre-trained sentence generation model M1, which is a conditional generation model, and the pre-trained sentence generation model M1 is re-trained to generate a video caption having features corresponding to the weighted image features, thereby generating a first additionally trained sentence generation model M2.

Specifically, the video generating unit 131 may generate the learning video #2 based on an image and sentence data set #2 including a pair of a captured image (hereinafter, sometimes referred to as "image #2") and an image description (hereinafter, sometimes referred to as "image description #2") that is a sentence that explains the contents of the captured image. For example, the video generating unit 131 may acquire the image and sentence data set #2 from an external information processing device via the communication unit 110. Next, the video generating unit 131 may acquire a first video generation model M21, which is a machine learning model that generates a video from an image. For example, the first video generation model M21 may be a known machine learning model that generates a video from an image (Reference: "Generating Videos with Scene Dynamics", Carl Vondrick et al., 2016, <Internet> https://arxiv.org/pdf/1609.02612.pdf (searched on February 16, 2023)). For example, the video generator 131 may acquire the first video generation model M21 from the storage unit 120, which stores information about the first video generation model M21 in advance. Next, the video generator 131 may input image #2 included in the image-sentence dataset #2 to the first video generation model M21, and generate a learning video #2 from image #2 that includes image #2 in a frame (step S21).

In this way, the video generation unit 131 uses the first video generation model M21, which is a machine learning model that generates videos from images, to generate a learning video #2 that includes a captured image in a frame from a captured image (image #2 in FIG. 3) included in the image-sentence dataset #2. Hereinafter, image #2 that was the source from which learning video #2 was generated may be referred to as the "original image #2."

The extraction unit 132 may extract image features of each of the multiple frame images constituting the learning video #2 from each of the multiple frame images constituting the learning video #2 generated by the video generation unit 131 (step S22). The method by which the extraction unit 132 extracts image features from each frame image may be the same as the method of extracting image features from each frame image in the pre-learning described in FIG. 2. In the following, explanations that overlap with FIG. 2 will be omitted. In FIG. 3, for simplicity, a case will be described in which there are three frame images constituting the learning video #2, but the number of frame images constituting the learning video #2 may be four or more. In FIG. 3, a first frame image corresponding to the start time of the learning video #2, a second frame image corresponding to a time between the start time and the end time of the learning video #2, and a third frame image corresponding to the end time of the learning video #2 are arranged in chronological order. For example, the extraction unit 132 extracts a feature vector V21 from the first frame image. The extraction unit 132 also extracts a feature vector V22 from the second frame image. The extraction unit 132 also extracts a feature vector V23 from the third frame image. Next, the extraction unit 132 may acquire a vector (V21, V22, V23) of the set of feature vectors V21 to V23 as learning frame feature #2, which is the image feature of each of the multiple frame images that make up learning video #2.

In this way, the extraction unit 132 extracts learning frame features #2, which are features of each of the multiple frame images that make up learning video #2.

The determination unit 133 may determine weights corresponding to each of the multiple frame images constituting the learning video #2 generated by the video generation unit 131 (step S22). Either the process of extracting image features by the extraction unit 132 or the process of determining weights by the determination unit 133 may be performed first, or may be performed simultaneously by the extraction unit 132 and the determination unit 133.

For example, the determination unit 133 may determine weights corresponding to each of the multiple frame images constituting the learning video #2 such that the weight of the frame image corresponding to the original image #2 is greater than the weights corresponding to the other frame images other than the frame image corresponding to the original image #2. For example, the determination unit 133 may determine weights corresponding to each of the multiple frame images based on the value of a convex function such as a Gaussian function (also called a normal distribution) or a part of a circle, in which the periphery of the maximum value is differentiable. Note that the determination unit 133 may determine weights using any function, not limited to a Gaussian function or a part of a circle, as long as the periphery of the maximum value is differentiable. For example, the determination unit 133 may set a value corresponding to the maximum value of a function whose periphery of the maximum value is differentiable as the weight of the frame image corresponding to the original image #2. Furthermore, the determination unit 133 may set a value corresponding to the periphery of the maximum value of a function whose periphery of the maximum value is differentiable as the weight of the frame image other than the frame image corresponding to the original image #2.

In FIG. 3, the determination unit 133 may determine weights corresponding to each of the multiple frame images using the value of a Gaussian function with the horizontal axis representing the playback time of the video and the vertical axis representing the weight. For example, the determination unit 133 may determine the time corresponding to the average value of the Gaussian function as the playback time of the second frame image corresponding to the original image #2. The determination unit 133 may also determine the value of the time corresponding to the average value of the Gaussian function, "1.0", as the weight #22 of the second frame image corresponding to the original image #2. The determination unit 133 may also determine the time corresponding to a value smaller than the average value of the Gaussian function as the playback time of the first frame image. The determination unit 133 may also determine the value of the time corresponding to a value smaller than the average value of the Gaussian function as the weight #21 of the first frame image. The determination unit 133 may also determine the time corresponding to a value larger than the average value of the Gaussian function as the playback time of the third frame image. Furthermore, the determination unit 133 may set "0.8", which is the value at the time corresponding to a value greater than the average value of the Gaussian function, as weight #23 for the third frame image. For example, the determination unit 133 may obtain a vector (weight #21, weight #22, weight #23) = (0.8, 1.0, 0.8) of the set of weight #21 for the first frame image to weight #23 for the second frame image as learning weight #2, which is the weight corresponding to each of the multiple frame images constituting learning video #2.

In this way, the determination unit 133 determines the learning weights #2 (in FIG. 3, (weights #21, #22, #23) = (0.8, 1.0, 0.8)) corresponding to each of the multiple frame images constituting the learning video #2 so that the weight (in FIG. 3, weight #22 of the second frame image, "1.0") corresponding to the captured image (original image #2 in FIG. 3) is greater than the weights corresponding to the other frame images other than the captured image (in FIG. 3, weight #21 of the first frame image, "0.8" and weight #23 of the third frame image, "0.8"). Also, in this way, the determination unit 133 determines the learning weights #2 that correspond to each of the multiple frame images constituting the learning video #2.

The model generation unit 134 may weight the learning frame feature #2 extracted by the extraction unit 132 by the learning weight #2 determined by the determination unit 133. The model generation unit 134 may generate a weighted learning frame feature #2', which is the learning frame feature #2 weighted by the learning weight #2 determined by the determination unit 133. In FIG. 3, the model generation unit 134 may generate a feature vector V21' weighted by the weight #21 by multiplying each element of the feature vector V21 by "0.8", which is the weight #21 corresponding to the first frame image constituting the learning video #2. The model generation unit 134 may generate a feature vector V22' weighted by the weight #22 by multiplying each element of the feature vector V22 by "1.0", which is the weight #22 corresponding to the second frame image constituting the learning video #2. Furthermore, the model generation unit 134 may generate a feature vector V23' weighted by weight #23 by multiplying each element of the feature vector V23 by "0.8", which is weight #23 corresponding to the third frame image constituting the learning video #2. In this manner, the model generation unit 134 may generate weighted learning frame feature #2'. In FIG. 3, the model generation unit 134 may generate (weight #21, weight #22, weight #23) * (V21, V22, V23) = (weight #21 * V21, weight #22 * V22, weight #23 * V23) = (V21', V22', V23') as the weighted learning frame feature #2'.

Then, the model generation unit 134 may input the weighted learning frame feature #2' to the pre-trained sentence generation model M1 (hereinafter, sometimes abbreviated as "sentence generation model M1") (step S23). For example, the model generation unit 134 may generate a condition vector #2 based on the weighted learning frame feature #2'. Then, the model generation unit 134 may combine the generated condition vector #2 with a noise vector. Note that the method by which the model generation unit 134 combines the condition vector #2 with the noise vector may be the same as the method by which the condition vector #1 with the noise vector is combined in the pre-training described in FIG. 2. In the following, the description that overlaps with FIG. 2 will be omitted. Then, the model generation unit 134 may input the combined condition vector #2 and noise vector as input information to the sentence generation model M1, which is a conditional generation model.

Next, the model generation unit 134 may acquire the video description generated by the sentence generation model M1 in response to the input of the combined condition vector #2 and the noise vector, and output as output information from the sentence generation model M1 (hereinafter, may be referred to as "learning video description #2") (step S24). The model generation unit 134 may retrain the sentence generation model M1 so as to generate a video description having a feature corresponding to the condition vector #2 based on the condition vector #2. For example, the model generation unit 134 may retrain the sentence generation model M1 using backpropagation (error backpropagation method) or the like so as to reduce the error between the learning video description #2 output from the sentence generation model M1 and the image description #2 (image description corresponding to the original image #2) included in the image sentence data set #2. In this way, the model generation unit 134 may retrain the sentence generation model M1 so as to generate a video description having a feature corresponding to the weighted learning frame feature #2' based on the weighted learning frame feature #2'.

In this way, the model generation unit 134 generates a sentence generation model M2 by re-training the pre-trained sentence generation model M1 to generate training video description #2, which is a sentence that explains the content of the training video, based on training frame feature #2 and training weight #2, and has features corresponding to training frame feature #2' weighted by training weight #2.

FIG. 4 is a diagram for explaining a method of weighting frame features by weights according to an embodiment. The frame features may have values corresponding to each pixel of the image. In the example shown in FIG. 4, for simplicity, a case will be explained in which the pixels of the image are represented by a 3×3 matrix. In this case, the frame features may be represented by a 3×3 matrix. Also, for simplicity, the weight value is set to "3". In this case, the model generation unit 134 generates weighted frame features by multiplying each element of the frame features (each element of the 3×3 matrix) by the weight value "3".

[5. Second additional learning method]
A second additional learning method according to the embodiment will be described with reference to Fig. 4. Fig. 5 is a diagram showing an example of information processing related to the second additional learning method according to the embodiment. The second additional learning refers to full-scale model learning performed after the pre-learning described in Fig. 2. Fig. 5 differs from Fig. 3 in that the model generation unit 134 re-learns the pre-trained sentence generation model M1 by the second additional learning instead of the first additional learning.

As shown in FIG. 5, in the second additional learning stage, (1) a known video generation model is used to generate a video from an image description included in an image-sentence dataset including a pair of an image (still image) and an image description. Hereinafter, the image (included in the image-sentence dataset) corresponding to the image description from which the video is generated may be referred to as the "original image". (2) The similarity between each frame image constituting the generated video and the original image is calculated, and the calculated similarity is set as the weight corresponding to each frame image constituting the video. In addition, image features are extracted from each frame image constituting the generated video, and the image features extracted from each frame image are weighted by the weight corresponding to each frame image. (3) The weighted image features are input as conditions to the pre-trained sentence generation model M1, which is a conditional generation model, and the pre-trained sentence generation model M1 is re-trained to generate a video description having features corresponding to the weighted image features, thereby generating a second additionally trained sentence generation model M3.

Specifically, the video generation unit 131 may generate the learning video #3 based on an image and sentence data set #3 including a pair of a captured image (hereinafter, sometimes referred to as "image #3") and an image description (hereinafter, sometimes referred to as "image description #3") that is a sentence that explains the content of the captured image. For example, the video generation unit 131 may acquire the image and sentence data set #3 from an external information processing device via the communication unit 110. Next, the video generation unit 131 may acquire a second video generation model M31, which is a machine learning model that generates a video from a sentence. For example, the second video generation model M31 may be a known machine learning model that generates a video from a sentence (Reference: "CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers", Wenyi Hong et al, 2022), <Internet> https://arxiv.org/pdf/2205.15868.pdf (searched on February 16, 2023)). For example, the video generation unit 131 may acquire the second video generation model M31 from the storage unit 120 that stores information about the second video generation model M31 in advance. Next, the video generation unit 131 may input the image description #3 included in the image text data set #3 to the second video generation model M31, and generate a learning video #3 from the image description #3 (step S31).

In this way, the video generation unit 131 generates training video #3 from image description #3 included in image sentence dataset #3 using the second video generation model M31, which is a machine learning model that generates videos from text. In the following, the image description #3 that was the source for generating training video #3 and the corresponding image #3 (image #3 included in image sentence dataset #3) may be referred to as the "original image #3."

The extraction unit 132 may extract image features of each of the multiple frame images constituting the learning video #3 from each of the multiple frame images constituting the learning video #3 generated by the video generation unit 131 (step S32). As in FIG. 3, the method in which the extraction unit 132 extracts image features from each frame image may be the same as the method in which image features are extracted from each frame image in the pre-learning described in FIG. 2. In the following, explanations that overlap with FIG. 2 will be omitted. In FIG. 5, for simplicity, a case in which there are three frame images constituting the learning video #3 will be described, but the number of frame images constituting the learning video #3 may be four or more. In FIG. 5, a first frame image corresponding to the start time of the learning video #3, a second frame image corresponding to a time between the start time and the end time of the learning video #3, and a third frame image corresponding to the end time of the learning video #3 are shown arranged in chronological order. For example, the extraction unit 132 extracts a feature vector V31 from the first frame image. The extraction unit 132 also extracts a feature vector V32 from the second frame image. The extraction unit 132 also extracts a feature vector V33 from the third frame image. Next, the extraction unit 132 may acquire a vector (V31, V32, V33) of the set of feature vectors V31 to V33 as learning frame feature #3, which is the image feature of each of the multiple frame images that make up learning video #3.

In this way, the extraction unit 132 extracts learning frame features #3, which are features of each of the multiple frame images that make up learning video #3.

The determination unit 133 may determine weights corresponding to each of the multiple frame images constituting the learning video #3 generated by the video generation unit 131 (step S32). As in FIG. 3, the process of extracting image features by the extraction unit 132 and the process of determining weights by the determination unit 133 may be performed first, or may be performed simultaneously by the extraction unit 132 and the determination unit 133.

For example, the determination unit 133 may determine weights corresponding to each of the multiple frame images based on the similarity between each of the multiple frame images constituting the learning video #3 and the original image #3. In FIG. 5, the determination unit 133 calculates the similarity #31 between the first frame image and the original image #3 to be "0.1". Then, the determination unit 133 may set the calculated value of similarity #31, "0.1", as the weight #31 of the first frame image. The determination unit 133 also calculates the similarity #32 between the second frame image and the original image #3 to be "0.7". Then, the determination unit 133 may set the calculated value of similarity #32, "0.7", as the weight #32 of the second frame image. The determination unit 133 also calculates the similarity #33 between the third frame image and the original image #3 to be "0.2". Next, the determination unit 133 may set the calculated value of similarity #33, "0.2", as the weight #33 of the third frame image. For example, the determination unit 133 may obtain a vector (weight #31, weight #32, weight #33) = (similarity #31, similarity #32, similarity #33) = (0.1, 0.7, 0.2) of the set of weights #31 of the first frame image to #33 of the second frame image as learning weights #3, which are weights corresponding to each of the multiple frame images that make up the learning video #3.

In this way, the determination unit 133 sets information regarding the similarity (in FIG. 5, (similarity #31, similarity #32, similarity #33) = (0.1, 0.7, 0.2)) between each of the multiple frame images (in FIG. 5, the first to third frame images) that make up the learning video #3 and the captured image (in FIG. 5, original image #3) as learning weights #3 (in FIG. 5, (weights #31, weights #32, weights #33) = (0.1, 0.7, 0.2)) corresponding to each of the multiple frame images. In this way, the determination unit 133 determines learning weights #3 that are weights that correspond to each of the multiple frame images that make up the learning video #3.

The model generation unit 134 may weight the learning frame feature #3 extracted by the extraction unit 132 by the learning weight #3 determined by the determination unit 133. The model generation unit 134 may generate a weighted learning frame feature #3', which is the learning frame feature #3 weighted by the learning weight #3 determined by the determination unit 133. In FIG. 5, the model generation unit 134 may generate a feature vector V31' weighted by the weight #31 by multiplying each element of the feature vector V31 by "0.1", which is the weight #31 corresponding to the first frame image constituting the learning video #3. The model generation unit 134 may generate a feature vector V32' weighted by the weight #32 by multiplying each element of the feature vector V32 by "0.7", which is the weight #32 corresponding to the second frame image constituting the learning video #3. Furthermore, the model generation unit 134 may generate a feature vector V33' weighted by weight #33 by multiplying each element of the feature vector V33 by "0.2", which is weight #33 corresponding to the third frame image constituting the learning video #3. In this manner, the model generation unit 134 may generate weighted learning frame feature #3'. In FIG. 5, the model generation unit 134 may generate (weight #31, weight #32, weight #33) * (V31, V32, V33) = (weight #31 * V31, weight #32 * V32, weight #33 * V33) = (V31', V32', V33') as the weighted learning frame feature #3'.

Then, the model generation unit 134 may input the weighted learning frame feature #3' to the pre-trained sentence generation model M1 (hereinafter, sometimes abbreviated as "sentence generation model M1") (step S33). For example, the model generation unit 134 may generate a condition vector #3 based on the weighted learning frame feature #3'. Then, the model generation unit 134 may combine the generated condition vector #3 with a noise vector. Note that the method by which the model generation unit 134 combines the condition vector #3 with the noise vector may be the same as the method by which the condition vector #1 with the noise vector is combined in the pre-training described in FIG. 2. In the following, the description that overlaps with FIG. 2 will be omitted. Then, the model generation unit 134 may input the combined condition vector #3 and noise vector as input information to the sentence generation model M1, which is a conditional generation model.

Then, the model generation unit 134 may acquire the video description generated by the sentence generation model M1 in response to the input of the combined condition vector #3 and the noise vector, and output from the sentence generation model M1 as output information (hereinafter, may be referred to as "learning video description #3") (step S34). The model generation unit 134 may retrain the sentence generation model M1 so as to generate a video description having a feature corresponding to the condition vector #3 based on the condition vector #3. For example, the model generation unit 134 may retrain the sentence generation model M1 using backpropagation (error backpropagation method) or the like so as to reduce the error between the learning video description #3 output from the sentence generation model M1 and the image description #3 (image description corresponding to the original image #3) included in the image sentence data set #3. In this way, the model generation unit 134 may retrain the sentence generation model M1 so as to generate a video description having a feature corresponding to the weighted learning frame feature #3' based on the weighted learning frame feature #3'.

In this way, the model generation unit 134 generates a sentence generation model M3 by re-training the pre-trained sentence generation model M1 to generate training video description #3, which is a sentence that explains the content of training video #3, based on training frame feature #3 and training weight #3, and has features corresponding to training frame feature #3' weighted by training weight #3.

FIG. 6 is a diagram for explaining a method of calculating similarity according to an embodiment. The frame image and original image #3 have pixel values corresponding to each pixel of the image. In the example shown in FIG. 6, for simplicity, a case where the pixels of the image are expressed as a 3×3 matrix will be explained. The left side of FIG. 6 shows one of the multiple frame images constituting the learning video #3. The right side of FIG. 6 shows the original image #3. At this time, the determination unit 133 may calculate the cosine similarity between the one frame image and the original image #3 as the similarity between the one frame image and the original image #3. For example, the determination unit 133 may calculate the cosine similarity between the one frame image and the original image #3 according to the following formula (1).

For example, the determination unit 133 determines the cosine similarity between one frame image and the original image #3 in accordance with the above formula (1) as follows: {(1.2*0.2)+(2.4*7.2)+((-2.3)*0.9)+(0.8*(-2.4))+((-1.3)*(-3.9))+((-1.2)*(-3.6))+(2.0*6.0)+((-3.2)*9.6)+(0.3*1.9)}/{1.2 ² +2.4 ² +((-2.3)) ² +0.8 ² +((-1.3)) ² +((-1.2)) ² +2.0 ² +((-3.2)) ² +0.3 ² } ^1/2 {0.2 ² +7.2 ² + 0.9 ² + ((-2.4)) ² + ((-3.9)) ² + ((-3.6)) ² + 6.0 ^{2 +} 9.6 ² + 1.9 ² } ^1/2 = 0.05.

6. Inference Method
The inference method according to the embodiment will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of information processing related to the inference method according to the embodiment. In the inference stage, the first additionally trained sentence generation model M2 described in FIG. 3 or the second additionally trained sentence generation model M3 described in FIG. 5 is used to generate a target video description, which is a sentence that describes the contents of a target video that is a video to be processed. FIG. 7 describes a case where the sentence generation unit 136 generates a target video description using the first additionally trained sentence generation model M2 (hereinafter, sometimes abbreviated as "sentence generation model M2"). Note that the sentence generation unit 136 may generate a target video description using the second additionally trained sentence generation model M3 (hereinafter, sometimes abbreviated as "sentence generation model M3") instead of the sentence generation model M2.

As shown in FIG. 7, in the inference stage, (1) image features are extracted from each frame image constituting the target video. (2) A frame image of interest (hereinafter sometimes referred to as a "designated frame image") is specified by the user, and weights corresponding to each of the multiple frame images are determined so that the weight corresponding to the designated frame image is maximized. (3) The image features extracted from each frame image are weighted by the weight corresponding to each frame image. The weighted image features are input as conditions to sentence generation model M2, which is a conditional generation model. (4) A description of the target video is generated by sentence generation model M2.

Specifically, the extraction unit 132 may acquire the target video #4, which is a video to be processed. For example, the extraction unit 132 may acquire the target video #4 from the information processing device used by the user via the communication unit 110. Next, the extraction unit 132 may extract image features of each of the multiple frame images constituting the target video #4 from each of the multiple frame images constituting the target video #4 (step S41). As in FIG. 3 and FIG. 5, the method in which the extraction unit 132 extracts image features from each frame image may be the same as the method in which image features are extracted from each frame image in the pre-learning described in FIG. 2. In the following, explanations that overlap with FIG. 2 will be omitted. In FIG. 7, for simplicity, a case in which the frame images constituting the target video #4 are three will be described, but the number of frame images constituting the target video #4 may be four or more. FIG. 7 shows a first frame image corresponding to the start time of the target video #4, a second frame image corresponding to a time between the start time and end time of the target video #4, and a third frame image corresponding to the end time of the target video #4, all arranged in chronological order. For example, the extraction unit 132 extracts a feature vector V41 from the first frame image. The extraction unit 132 also extracts a feature vector V42 from the second frame image. The extraction unit 132 also extracts a feature vector V43 from the third frame image. Next, the extraction unit 132 may acquire a vector (V41, V42, V43) of the set of feature vectors V41 to V43 as target frame feature #4, which is the image feature of each of the multiple frame images that make up the target video #4.

In this way, the extraction unit 132 extracts target frame features #4, which are features of each of the multiple frame images that make up target video #4, which is the video to be processed.

The determination unit 133 may also acquire a frame image designated by the user from among the multiple frame images constituting the target video #4 (hereinafter, may be referred to as "designated frame image #4") and the target video #4. For example, the determination unit 133 may acquire the designated frame image #4 and the target video #4 from an information processing device used by the user via the communication unit 110. Next, the determination unit 133 may determine weights corresponding to each of the multiple frame images constituting the target video #4 (step S41). Note that, as in FIGS. 3 and 5, the process of extracting image features by the extraction unit 132 and the process of determining weights by the determination unit 133 may be performed first, or may be performed simultaneously by the extraction unit 132 and the determination unit 133.

For example, the determination unit 133 may determine weights corresponding to each of the multiple frame images such that the weight of the frame image corresponding to the designated frame image #4 among the multiple frame images constituting the target video #4 is greater than the weights corresponding to the other frame images other than the frame image corresponding to the designated frame image #4. For example, the determination unit 133 may determine weights corresponding to each of the multiple frame images based on the value of a convex function such as a Gaussian function or a part of a circle, the periphery of which is differentiable around the maximum value. In FIG. 7, the determination unit 133 may determine weights corresponding to each of the multiple frame images using the value of a Gaussian function with the horizontal axis representing the playback time of the video and the vertical axis representing the weight, as in FIG. 3. For example, the determination unit 133 may determine the time corresponding to the average value of the Gaussian function as the playback time of the second frame image corresponding to the designated frame image #4. In addition, the determination unit 133 may determine the weight #42 of the second frame image corresponding to the designated frame image #4 to be "1.0", which is the value of the time corresponding to the average value of the Gaussian function. The determination unit 133 may also determine the time corresponding to a value smaller than the average value of the Gaussian function as the playback time of the first frame image. The determination unit 133 may also determine "0.8", which is the value of the time corresponding to a value smaller than the average value of the Gaussian function, as the weight #41 of the first frame image. The determination unit 133 may also determine the time corresponding to a value larger than the average value of the Gaussian function, as the playback time of the third frame image. The determination unit 133 may also determine "0.8", which is the value of the time corresponding to a value larger than the average value of the Gaussian function, as the weight #43 of the third frame image. For example, the determination unit 133 may obtain a vector (weight #41, weight #42, weight #43) = (0.8, 1.0, 0.8) of the set of weights #41 of the first frame image to #43 of the second frame image as target weight #4, which is the weight corresponding to each of the multiple frame images constituting the target video #4.

In this way, the determination unit 133 determines target weights #4, which are weights corresponding to each of the multiple frame images that make up the target video #4, such that the weight corresponding to the designated frame image #4 designated by the user (in FIG. 7, weight #42 of the second frame image, "1.0") is greater than the weights corresponding to the other frame images other than the designated frame image (in FIG. 7, weight #41 of the first frame image, "0.8", and weight #43 of the third frame image, "0.8"). Also, in this way, the determination unit 133 determines target weights #4, which are weights corresponding to each of the multiple frame images that make up the target video #4.

The sentence generation unit 136 may weight the target frame feature #4 extracted by the extraction unit 132 by the target weight #4 determined by the determination unit 133. The sentence generation unit 136 may generate a weighted target frame feature #4', which is the target frame feature #4 weighted by the target weight #4 determined by the determination unit 133. In FIG. 7, the sentence generation unit 136 may generate a feature vector V41' weighted by the weight #41 by multiplying each element of the feature vector V41 by "0.8", which is the weight #41 corresponding to the first frame image constituting the target video #4. The sentence generation unit 136 may generate a feature vector V42' weighted by the weight #42 by multiplying each element of the feature vector V42 by "1.0", which is the weight #42 corresponding to the second frame image constituting the target video #4. Furthermore, the sentence generation unit 136 may generate a feature vector V43' weighted by weight #43 by multiplying each element of the feature vector V43 by "0.8", which is weight #43 corresponding to the third frame image constituting the target video #4. In this manner, the sentence generation unit 136 may generate weighted target frame feature #4'. In FIG. 7, the sentence generation unit 136 may generate (weight #41, weight #42, weight #43) * (V41, V42, V43) = (weight #41 * V41, weight #42 * V42, weight #43 * V43) = (V41', V42', V43') as the weighted target frame feature #4'.

The acquisition unit 135 may also acquire the sentence generation model M2. For example, the acquisition unit 135 may acquire the sentence generation model M2 from the storage unit 120, which stores information about the sentence generation model M2.

In this way, the acquisition unit 135 acquires a sentence generation model M2, which is a machine learning model trained to generate training video description #2, which is a sentence explaining the content of the training video #2, having features corresponding to the training frame feature #2 weighted by the training weight #2, based on the training frame feature #2, which is a feature of each of the multiple frame images constituting the training video #2 generated based on the image and sentence dataset #2 including a pair of a captured image (in FIG. 7, image #2 described in FIG. 3) and an image description (in FIG. 7, image description #2 described in FIG. 3), which is a sentence explaining the content of the captured image, and based on the training weight #2, which is a weight corresponding to each of the multiple frame images constituting the training video #2.

The sentence generation unit 136 may input the weighted target frame feature #4' to the sentence generation model M2 acquired by the acquisition unit 135 (step S42). For example, the sentence generation unit 136 may generate a condition vector #4 based on the weighted target frame feature #4'. Next, the sentence generation unit 136 may combine the generated condition vector #4 with a noise vector. Note that the method by which the sentence generation unit 136 combines the condition vector #4 with the noise vector may be the same as the method by which the condition vector #1 with the noise vector is combined in the pre-learning described in FIG. 2. In the following, descriptions that overlap with FIG. 2 will be omitted. Next, the sentence generation unit 136 may input the combined condition vector #4 and noise vector as input information to the sentence generation model M2, which is a conditional generation model.

Next, the sentence generation unit 136 may acquire the video description generated by the sentence generation model M2 in response to the input of the combined condition vector #4 and the noise vector, and output as output information from the sentence generation model M2 (hereinafter, this may be referred to as "target video description #4") (step S43).

In this way, based on the target frame feature #4 and the target weight #4, the sentence generation unit 136 inputs the target frame feature #4' weighted by the target weight to the sentence generation model M2 to generate the target video description #4, which is a sentence that explains the content of the target video #4.

[7. Example of a sentence generation model]
Fig. 8 is a diagram showing a conditional generative adversarial network (CGAN) as an example of a sentence generation model according to an embodiment. As shown in Fig. 8, the sentence generation model may be a conditional generative adversarial network including a generator network G1 and a classifier network D1. Fig. 8 describes the first additional learning described in Fig. 3 or the second additional learning described in Fig. 5.

The generator network G1 shown in FIG. 8 may be a machine learning model suitable for generating text, which is time-series data. For example, the generator network G1 may be a recurrent neural network (RNN), a gated recurrent unit (GRU), a long short-term memory (LSTM), a transformer (Ashish Vaswani et al., 2017), a bidirectional encoder representations from transformers (BERT) based on a transformer, a generative pre-training3 (GPT-3), or a text-to-text transfer transformer (T5).

First, the model generation unit 134 may train a conditional generative adversarial network including the generator network G1 and the discriminator network D1 by the pre-learning described in FIG. 2. For example, the model generation unit 134 may generate a pre-learning adversarial condition vector V50, which is a condition vector based on the pre-learning frame feature #50. Next, the model generation unit 134 may train the generator network G1 to generate a video description having features corresponding to the pre-learning adversarial condition vector V50 when the pre-learning adversarial condition vector V50 and a noise vector are input to the generator network G1 as input information. For example, the model generation unit 134 may train the generator network G1 using backpropagation (error backpropagation method) or the like so that the error between the video description output from the generator network G1 and the pre-learning video description #50 included in the video description dataset #50 is reduced. In this way, the model generation unit 134 may generate a pre-trained generator network G1 that has been trained in advance to generate video description text having features that correspond to the pre-training frame feature #50, based on the pre-training frame feature #50.

In addition, the model generation unit 134 may train the classifier network D1 so that when the pre-learning hostile condition vector # and the pre-learning video description #50 are input as input information to the classifier network D1, the classifier network D1 outputs information (e.g., the number "1") indicating that the pre-learning video description #50 is a true video description and is a video description corresponding to the pre-learning hostile condition vector V50 as output information. In addition, the model generation unit 134 may adjust the size of the pre-learning hostile condition vector V50 and the pre-learning video description #50 so that they are the same, for example, using a linear transformation process. Next, the model generation unit 134 may combine the pre-learning hostile condition vector V50 and the pre-learning video description #50, and input the combined pre-learning hostile condition vector V50 and the pre-learning video description #50 as input information to the classifier network D1. Furthermore, the model generation unit 134 may train the classifier network D1 so that when the pre-training adversarial condition vector V50 and the fake video description generated by the generator network G1 are input as input information to the classifier network D1, the classifier network D1 outputs information (e.g., the number "0") other than information indicating that the fake video description generated by the generator network G1 is a true video description and corresponds to the pre-training adversarial condition vector V50 as output information. In this way, the model generation unit 134 may generate a pre-trained classifier network D1 that has been trained in advance based on the pre-training frame feature #50.

The model generation unit 134 may also retrain the pre-trained generator network G1 (hereinafter sometimes abbreviated as "generator network G1") and the pre-trained discriminator network D1 (hereinafter sometimes abbreviated as "discriminator network D1") by the first additional learning described in FIG. 3 or the second additional learning described in FIG. 5.

In FIG. 8, the model generation unit 134 may generate a first training adversarial condition vector V51 (hereinafter, may be abbreviated as "first adversarial condition vector V51"), which is a condition vector based on the training frame feature weighted by the training weight. Next, the model generation unit 134 may retrain the generator network G1 so as to generate a video description (training video description #51 in FIG. 8) having features corresponding to the first adversarial condition vector V51 when the first adversarial condition vector V51 and the noise vector N1 are input to the generator network G1 as input information. For example, the model generation unit 134 may retrain the generator network G1 using backpropagation (error backpropagation method) or the like so as to reduce the error between the training video description #51 output from the generator network G1 and the image description #51 (image description corresponding to the original image #51) included in the image description dataset #51. In this manner, the model generation unit 134 may retrain the generator network G1 so as to generate video description sentences having features corresponding to the weighted training frame features based on the weighted training frame features. In this manner, the model generation unit 134 may generate a first additionally trained or second additionally trained generator network G1.

In addition, when the first hostile condition vector V51 and the learning video description #51, which is a false video description generated by the generator network G1, are input as input information to the classifier network D1, the model generation unit 134 may retrain the classifier network D1 so that the learning video description #51 is a true video description and is a video description corresponding to the first hostile condition vector V51, and other information (e.g., the number "0") other than the information indicating that the learning video description #51 is a true video description and is a video description corresponding to the first hostile condition vector V51 is output as output information. In addition, the model generation unit 134 may generate a second learning hostile condition vector V52 (hereinafter, sometimes abbreviated as "second hostile condition vector V52") that is a condition vector based on the pre-learning frame feature #52, which is a feature of each of the multiple frame images constituting the pre-learning video #52 included in the video text data set #52 including the pair of the pre-learning video #52 and the pre-learning video description #52. Next, the model generation unit 134 may retrain the classifier network D1 so that, when the second adversarial condition vector V52 and the pre-learning video description #52 are input as input information to the classifier network D1, the classifier network D1 outputs information indicating that the pre-learning video description #52 is a true video description and is a video description corresponding to the second adversarial condition vector V52 (e.g., the number "1") as output information. In this manner, the model generation unit 134 may generate the first additionally trained or second additionally trained classifier network D1.

The sentence generation unit 136 may also generate an adversarial condition vector V53, which is a condition vector based on the target frame features weighted by the target weight. Next, the model generation unit 134 may input the adversarial condition vector V53 and the noise vector N2 as input information to the generator network G1 to generate a target video description, which is a sentence that describes the contents of the target video. In this way, the sentence generation unit 136 may input the target frame features weighted by the target weight to the first additionally trained or second additionally trained generator network G1 based on the target frame features and the target weight to generate a target video description, which is a sentence that describes the contents of the target video.

8. Modifications
The process according to the embodiment described above may be implemented in various different forms other than the above embodiment.

8-1. First Modified Example
In the above-described embodiment, the sentence generation model is a conditional generative adversarial network including a generator network and a classifier network. In a first modified example, the sentence generation model is a conditional variational autoencoder including an encoder and a decoder.

Figure 9 is a diagram showing a conditional variational autoencoder (CVAE), which is an example of a sentence generation model related to the first modified example. As shown in Figure 9, the sentence generation model may be a conditional variational autoencoder including an encoder EN1 and a decoder DE1. Figure 9 explains the pre-learning described in Figure 2.

First, the model generation unit 134 may train a conditional variational autoencoder including the encoder EN1 and the decoder DE1 by the pre-learning described in FIG. 2. For example, the model generation unit 134 may generate a pre-learning variational condition vector V60 (hereinafter, may be abbreviated as "variational condition vector V60"), which is a condition vector based on the pre-learning frame feature #60. Next, the model generation unit 134 may train the encoder EN1 so that, when the variational condition vector V60 and the pre-learning video description #60 are input to the encoder EN1 as input information, the encoder EN1 outputs the mean vector μ and variance vector σ in the multivariate normal distribution as output information. In addition, the model generation unit 134 may determine a latent vector z, which is a sample that follows a multivariate normal distribution based on the mean vector μ and variance vector σ. In addition, the model generation unit 134 may introduce a random variable ε obtained by randomly sampling from a standard normal distribution, and use this to determine the latent vector z. Next, the model generation unit 134 may train the decoder DE1 so that when the latent vector z and the variation condition vector V60 are input to the decoder DE1 as input information, the decoder DE1 outputs a video description (pre-learning video description #60 in FIG. 9) having a feature corresponding to the variation condition vector V60 as output information. For example, the model generation unit 134 may train the encoder EN1 and the decoder DE1 using backpropagation or the like so that the error between the pre-learning video description #60 input to the encoder EN1 and the video description output from the decoder DE1 is reduced. In this way, the model generation unit 134 may generate a pre-trained decoder DE1 that has been trained in advance to generate a video description having a feature corresponding to the pre-learning frame feature #60 based on the pre-learning frame feature #60.

The model generation unit 134 may also retrain the pre-trained decoder DE1 (hereinafter sometimes abbreviated as "decoder DE1") by the first additional learning described in FIG. 3 or the second additional learning described in FIG. 5.

For example, the model generation unit 134 may determine a latent vector z, which is a sample that follows a multivariate normal distribution based on the mean vector μ and variance vector σ output from the encoder EN1. The model generation unit 134 may also generate a training variation condition vector V61 (hereinafter, sometimes abbreviated as "variation condition vector V61"), which is a condition vector based on training frame features weighted by training weights. Next, the model generation unit 134 may retrain the decoder DE1 so that, when the latent vector z and the variation condition vector V61 are input to the decoder DE1 as input information, a video description having a feature corresponding to the variation condition vector V61 is output as output information. For example, the model generation unit 134 may retrain the decoder DE1 using backpropagation (error backpropagation method) or the like so that the error between the training video description #61 output from the decoder DE1 and the image description #61 (image description corresponding to the original image #61) included in the image description dataset #61 is reduced. In this manner, the model generation unit 134 may retrain the decoder DE1 so as to generate video description text having features corresponding to the weighted training frame features based on the weighted training frame features. In this manner, the model generation unit 134 may generate a first additionally trained or second additionally trained decoder DE1.

The sentence generation unit 136 may also generate a variational condition vector V62, which is a condition vector based on the target frame features weighted by the target weight. Next, the model generation unit 134 may input the latent vector z and the variational condition vector V62 as input information to the decoder DE1 to generate a target video description, which is a sentence that describes the contents of the target video. In this way, the sentence generation unit 136 may input the target frame features weighted by the target weight to the first additionally trained or second additionally trained decoder DE1 based on the target frame features and the target weight to generate a target video description, which is a sentence that describes the contents of the target video.

8-2. Second Modification
In the above-described embodiment, a case has been described in which the sentence generation model is a conditional generative adversarial network including a generator network and a discriminator network. In the first modified example, a case has been described in which the sentence generation model is a conditional variational autoencoder including an encoder and a decoder. In the second modified example, a case in which the sentence generation model is a conditional diffusion model will be described.

Figure 10 is a diagram showing a conditional diffusion model, which is an example of a sentence generation model related to the second modified example. As shown in Figure 10, the sentence generation model may be a conditional diffusion model. Figure 10 explains the first additional learning described in Figure 3 or the second additional learning described in Figure 5.

FIG. 10 shows a plurality of noise-added video descriptions in which noise is added stepwise to an initial video description x ₀ as an example of data used in the learning process of the conditional diffusion model. The model generation unit 134 uses learning data including the plurality of noise-added video descriptions shown in FIG. 10 to train the conditional diffusion model. In FIG. 10, the initial video description x ₀ is at stage #0 in terms of the addition of noise. That is, it is a video description to which no noise is added. The model generation unit 134 gradually adds Gaussian noise to the initial video description x ₀ , and in the process (diffusion process) of finally obtaining pure Gaussian noise x _T , generates a noise-added video description x _t-1 in which minute noise is added several times to the initial video description x _0. The noise-added video description x _t-1 is at stage #t-1 in terms of the addition of noise. That is, it is a video description to which noise is added t-1 stages to the initial video description x ₀ . Next, the model generation unit 134 generates a noise-added video description _xt in which minute noise is added to the noise-added video description _xt-1 . The noise-added video description _xt is a video description in which one more level of noise is added to the noise-added video description _xt-1 . The noise-added video description _xt is a stage #t in terms of noise addition. That is, it is a video description in which t levels of noise are added to the initial video description _x0 . For example, the noise-added video description _xt is generated by a noise addition process that adds noise to the noise-added video description _xt-1 . q( _xt | _xt-1 ) shown in FIG. 10 indicates the transition probability of transitioning from the noise-added video description _xt-1 to the noise-added video description _xt .

Next, the model generation unit 134 gradually removes Gaussian noise from the pure Gaussian noise x _T , and in the process (reverse diffusion process) of finally obtaining a video description x ₀ without added noise, the model generation unit 134 trains a conditional diffusion model that removes minute noise from the noisy video description x _t to generate a noisy video description x _t-1 . For example, the model generation unit 134 trains a conditional diffusion model so that the model generation unit 134 receives the noisy video description x t as input and outputs the noisy video description x _t-1 obtained by removing noise from the previous process, that is, the noisy video description x _t by one step. p _θ (x _{t -1} | x _t , V71) shown in FIG. 10 indicates the transition probability of transitioning from the noisy video description x _t-1 to the noisy video description x _t . Also, p _θ (x _t-1 | x _t , V71) is the output of a neural network having a parameter θ determined by learning. In this way, the model generation unit 134 trains a conditional diffusion model, which is a machine learning model that takes pure Gaussian noise _xT as input and gradually removes the noise, thereby ultimately generating a video description _x0 to which no noise has been added.

First, the model generation unit 134 may train a conditional diffusion model by the pre-learning described in FIG. 2. For example, the model generation unit 134 may generate a pre-learning diffusion condition vector V70 (hereinafter, may be abbreviated as "diffusion condition vector V70"), which is a condition vector based on the pre-learning frame feature #70. Next, the model generation unit 134 may train the conditional diffusion model so that when the diffusion condition vector V70 is input as input information to the conditional diffusion model, the mean vector and variance vector in the multivariate normal distribution to which the noise vector follows are output as output information. For example, the model generation unit 134 may train the conditional diffusion model so that when the diffusion condition vector V70 and a pure Gaussian noise vector are input as input information to the conditional diffusion model, the model generation unit 134 may train the conditional diffusion model so as to generate a video description having characteristics corresponding to the diffusion condition vector V70. For example, the model generation unit 134 may use backpropagation or the like to train the conditional diffusion model so as to reduce the error between the video description output from the conditional diffusion model and the pre-training video description #70 included in the video description dataset #70. In this way, the model generation unit 134 may generate a pre-trained conditional diffusion model that has been trained in advance based on the pre-training frame feature #70 to generate a video description having features corresponding to the pre-training frame feature #70.

The model generation unit 134 may retrain the pre-trained conditional diffusion model (hereinafter, may be abbreviated as "conditional diffusion model") by the first additional learning described in FIG. 3 or the second additional learning described in FIG. 5. For example, the model generation unit 134 may generate a learning diffusion condition vector V71 (hereinafter, may be abbreviated as "diffusion condition vector V71") which is a condition vector based on the learning frame feature weighted by the learning weight. Next, the model generation unit 134 may train the conditional diffusion model so that when the diffusion condition vector V71 which is a condition vector based on the learning frame feature weighted by the learning weight is input as input information to the conditional diffusion model, the model generation unit 134 outputs the mean vector and the variance vector in the multivariate normal distribution to which the noise vector follows as output information. For example, the model generation unit 134 may retrain the conditional diffusion model so that when the diffusion condition vector V71 and pure Gaussian noise are input as input information to the conditional diffusion model, the model generation unit 134 generates a video description having characteristics corresponding to the diffusion condition vector V71. For example, the model generation unit 134 may use backpropagation or the like to retrain the conditional diffusion model so as to reduce an error between the training video description #71 output from the conditional diffusion model and the image description #71 (image description corresponding to the original image #71) included in the image description dataset #71. In this manner, the model generation unit 134 may retrain the conditional diffusion model so as to generate a video description having features corresponding to the weighted training frame features based on the weighted training frame features. In this manner, the model generation unit 134 may generate a first additionally trained or second additionally trained conditional diffusion model.

The sentence generation unit 136 also generates a video description by estimating a noise vector using a trained conditional diffusion model (hereinafter sometimes abbreviated as "conditional diffusion model") and removing the noise vector from the noised video description feature. For example, the sentence generation unit 136 may generate a diffusion condition vector V72, which is a condition vector based on the target frame feature weighted by the target weight. Next, the model generation unit 134 may input the diffusion condition vector V72 and a pure Gaussian noise vector as input information to the conditional diffusion model to generate a target video description, which is a sentence that describes the contents of the target video. In this way, the sentence generation unit 136 may input the target frame feature weighted by the target weight to the first additionally trained or second additionally trained conditional diffusion model based on the target frame feature and the target weight to generate a target video description, which is a sentence that describes the contents of the target video.

9. Effects
As described above, the information processing device 100 according to the embodiment includes a video generating unit 131, an extracting unit 132, a determining unit 133, and a model generating unit 134. The video generating unit 131 generates a learning video based on an image and sentence data set including a pair of a captured image and an image description that is a sentence that describes the content of the captured image. The extracting unit 132 extracts learning frame features that are features of each of a plurality of frame images that constitute the learning video. The determining unit 133 determines learning weights that are weights corresponding to each of a plurality of frame images that constitute the learning video. The model generating unit 134 generates a sentence generation model that is a machine learning model trained to generate a learning video description that is a sentence that describes the content of the learning video and has features corresponding to the learning frame features weighted by the learning weight, based on the learning frame features and the learning weights.

As a result, the information processing device 100 can control which part of the video (which frame image) to emphasize when generating a video description by using weighted frame features corresponding to each frame image as a condition to be given to the conditional generative model. The information processing device 100 can also generate a variety of video descriptions according to the frame image of interest. The information processing device 100 can also generate a variety of video descriptions according to the frame image of interest, which can contribute to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Build resilience, innovate and innovate." The information processing device 100 can also reflect the time-series information of the video in natural language generation by using weighted frame features corresponding to each frame image as a condition to be given to the conditional generative model.

The video generation unit 131 also generates a learning video including captured images in frames from captured images included in the image-sentence dataset using a first video generation model, which is a machine learning model that generates videos from images. The determination unit 133 determines learning weights corresponding to each of the multiple frame images constituting the learning video such that the weight corresponding to the captured image is greater than the weights corresponding to the other frame images other than the captured image.

This enables the information processing device 100 to generate a video description that places more importance on the captured image that was the source of generating the video than on the other frame images among the multiple frame images that make up the video.

The video generation unit 131 also generates training videos from the image captions included in the image-sentence dataset using a second video generation model, which is a machine learning model that generates videos from text. The determination unit 133 sets information regarding the similarity between each of the multiple frame images constituting the training video and the captured image as training weights corresponding to each of the multiple frame images.

This enables the information processing device 100 to generate a video description that places emphasis on frame images that have a high similarity to the captured image corresponding to the image description from which the video was generated, rather than frame images that have a low similarity to the captured image corresponding to the image description from which the video was generated.

The extraction unit 132 also extracts pre-training frame features, which are features of each of a plurality of frame images constituting the captured video included in a video text dataset including a pair of a captured video and a video description, which is a text that describes the content of the captured video. The model generation unit 134 generates a pre-trained text generation model, which is a machine learning model that has been trained in advance to generate a video description having features corresponding to the pre-training frame features, based on the pre-training frame features, and generates a text generation model by re-training the pre-trained text generation model to generate a training video description, which is a text that describes the content of the training video, based on the training frame features and the training weights, and which has features corresponding to the training frame features weighted by the training weights.

This enables the information processing device 100 to generate video description text that has characteristics corresponding to the frame features.

The information processing device 100 further includes an acquisition unit 135 and a sentence generation unit 136. The acquisition unit 135 acquires a sentence generation model, which is a machine learning model trained to generate a learning video description, which is a sentence that describes the content of the learning video, having features corresponding to the learning frame features weighted by the learning weight, based on the learning frame features that are features of each of the multiple frame images constituting the learning video generated based on an image and sentence dataset including a pair of a captured image and an image description that is a sentence that describes the content of the captured image, and the learning weights that are weights corresponding to each of the multiple frame images constituting the learning video. The extraction unit 132 extracts target frame features that are features of each of the multiple frame images constituting the target video, which is a video to be processed. The determination unit 133 determines target weights that are weights corresponding to each of the multiple frame images such that the weight corresponding to the designated frame image designated by the user among the multiple frame images constituting the target video is larger than the weight corresponding to other frame images other than the designated frame image. The sentence generation unit 136 inputs the target frame features weighted by the target weights into a sentence generation model based on the target frame features and the target weights, and generates a target video description, which is a sentence that explains the contents of the target video.

As a result, the information processing device 100 can control which part of the video (which frame image) to emphasize when generating a video description by using weighted frame features corresponding to each frame image as a condition to be given to the conditional generative model. The information processing device 100 can also generate a variety of video descriptions according to the frame image of interest. The information processing device 100 can also generate a variety of video descriptions according to the frame image of interest, which can contribute to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Build resilience, innovate and innovate." The information processing device 100 can also use weighted frame features corresponding to each frame image as a condition to be given to the conditional generative model, which can enable the time-series information of the video to be reflected in natural language generation.

The sentence generation model is a conditional adversarial generative network including a generator network and a classifier network, and includes a generator network trained to output a learning video description as output information when a first adversarial condition vector, which is a condition vector based on training frame features weighted by training weights, and a noise vector are input as input information to the generator network, and a second adversarial condition vector, which is a condition vector based on pre-training frame features that are features of each of a plurality of frame images constituting a captured video included in a video text dataset that includes a pair of a captured video and a video description that is a text that explains the content of the captured video, and a noise vector, which is a condition vector based on pre-training frame features that are features of each of a plurality of frame images constituting a captured video included in a video text dataset that includes a pair of a captured video and a video description that is a text that explains the content of the captured video, A machine learning model including: a classifier network trained to output, as output information, information indicating that a video description is a true video description and that it corresponds to a second adversarial condition vector when an image description is input to the classifier network as input information; and a classifier network trained to output, as output information, information other than information indicating that a training video description is a true video description and that it corresponds to the first adversarial condition vector when a training video description, which is a false video description generated by a generator network, and a first adversarial condition vector and a training video description, are input to the classifier network as input information.

This allows the information processing device 100 to use a conditional generative adversarial network to generate a variety of video descriptions that correspond to the frame image of interest.

The sentence generation model is a conditional variational autoencoder including an encoder and a decoder, and is a machine learning model including an encoder trained to output a mean vector and a variance vector in a multivariate normal distribution as output information when a first variational condition vector, which is a condition vector based on pre-training frame features that are features of each of a plurality of frame images constituting a captured video included in a video text dataset including a pair of a captured video and a video description that is a text that explains the content of the captured video, and the video description are input as input information to the encoder, and a decoder trained to output the training video description as output information when a latent vector, which is a sample that follows a multivariate normal distribution based on the mean vector and the variance vector, and a second variational condition vector, which is a condition vector based on the training frame features weighted by the training weight, are input as input information to the decoder.

This allows the information processing device 100 to use a conditional variational autoencoder to generate a variety of video descriptions that correspond to the frame image of interest.

The text generation model is a conditional diffusion model, and is a machine learning model that estimates the noise vector using the conditional diffusion model trained to output, as output information, the mean vector and variance vector in the multivariate normal distribution to which the noise vector follows when a noisy video description, which is a training video description including a noise vector, and a diffusion condition vector, which is a condition vector based on training frame features weighted by training weights, are input to the conditional diffusion model as input information, and generates a training video description by removing the noise vector from the noisy video description.

This allows the information processing device 100 to use the conditional diffusion model to generate a variety of video descriptions that correspond to the frame image of interest.

10. Hardware Configuration
Moreover, the information processing device 100 according to the embodiment described above is realized by a computer 1000 having a configuration as shown in Fig. 11, for example. Fig. 11 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, a HDD 1400, a communication interface (I/F) 1500, an input/output interface (I/F) 1600, and a media interface (I/F) 1700.

The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400, and controls each component. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000, etc.

HDD 1400 stores programs executed by CPU 1100 and data used by such programs. Communication interface 1500 receives data from other devices via a specified communication network and sends it to CPU 1100, and transmits data generated by CPU 1100 to other devices via the specified communication network.

The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input/output interface 1600. The CPU 1100 acquires data from the input devices via the input/output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input/output interface 1600.

The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 130. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via a specified communication network.

Although several embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be embodied in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the forms described in the disclosure section of the invention.

11. Other
Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

For example, in the above embodiment, the information processing device 100 has the video generation unit 131, the extraction unit 132, the determination unit 133, the model generation unit 134, the acquisition unit 135, and the sentence generation unit 136 as functional units. However, each unit can be distributed and configured in a separate device. For example, the information processing device 100 can have the acquisition unit 135 and the sentence generation unit 136 as functional units as an information processing device that generates sentences. In addition, an information processing device other than the information processing device 100 (hereinafter, referred to as a "generation device") can have the video generation unit 131, the extraction unit 132, the determination unit 133, and the model generation unit 134 as functional units as an information processing device that generates a sentence generation model. At this time, the information processing device 100 and the generation device may be connected to various networks by wire or wirelessly and may transmit and receive information to each other. For example, the information processing device 100 may receive information about the sentence generation model generated by the generation device from the generation device.

The above-described embodiments and variations can be combined as appropriate to the extent that they do not cause inconsistencies in the processing content.

REFERENCE SIGNS LIST 100 Information processing device 110 Communication unit 120 Storage unit 130 Control unit 131 Video generation unit 132 Extraction unit 133 Determination unit 134 Model generation unit 135 Acquisition unit 136 Text generation unit

Claims (10)

A video generator that generates learning videos based on an image and text data set including a pair of a captured image and an image description text that explains the content of the captured image;
An extraction unit that extracts learning frame features that are features of each of a plurality of frame images that constitute the learning video;
A determination unit that determines learning weights corresponding to each of the plurality of frame images constituting the learning video such that a weight corresponding to the captured image is greater than weights corresponding to other frame images other than the captured image ; and
a model generation unit that generates a sentence generation model, which is a machine learning model trained to generate a training video description, which is a sentence explaining the content of the training video, based on the training frame features and the training weights, the training video description having features corresponding to the training frame features weighted by the training weights;
An information processing device comprising: The video generation unit is
generating the learning video including the captured image in a frame from the captured image included in the image-sentence dataset using a first video generation model that is a machine learning model that generates a video from an image;
The information processing device according to claim 1 . The video generation unit is
Using a second video generation model that is a machine learning model that generates videos from text, the training video is generated from the image caption included in the image and text dataset;
The determination unit is
information regarding a similarity between each of a plurality of frame images constituting the learning video and the captured image is set as the learning weight corresponding to each of the plurality of frame images;
The information processing device according to claim 1 . The extraction unit is
Extracting pre-learning frame features which are features of each of a plurality of frame images constituting the captured video included in a video text dataset including a pair of a captured video and a video description text which is a text explaining the content of the captured video;
The model generation unit
Based on the pre-training frame features, a pre-trained sentence generation model is generated, which is a machine learning model that has been pre-trained to generate the video description having features corresponding to the pre-training frame features, and based on the training frame features and the training weights, a training video description is a sentence that explains the content of the training video, and the training video description has features corresponding to the training frame features weighted by the training weights, thereby generating the sentence generation model.
The information processing device according to claim 1 . an acquisition unit that acquires a sentence generation model, which is a machine learning model trained to generate training video descriptions, which are sentences explaining the content of a training video, based on training frame features, which are features of each of a plurality of frame images constituting a training video generated based on an image and text dataset including a pair of a captured image and an image description, which is a sentence explaining the content of the captured image, and training weights, which are weights corresponding to each of a plurality of frame images constituting the training video, and which has features corresponding to the training frame features weighted by the training weights;
an extraction unit that extracts target frame features that are features of each of a plurality of frame images that constitute a target moving image that is a moving image to be processed;
a determination unit that determines target weights, which are weights corresponding to each of a plurality of frame images constituting the target moving image, such that a weight corresponding to a designated frame image designated by a user is made larger than weights corresponding to other frame images other than the designated frame image;
A sentence generation unit that inputs the target frame feature weighted by the target weight to the sentence generation model based on the target frame feature and the target weight, and generates a target video description sentence that is a sentence that explains the content of the target video;
An information processing device comprising: The sentence generation model is a conditional generative adversarial network including a generator network and a classifier network;
The generator network is trained to output the training video description as output information when a first adversarial condition vector, which is a condition vector based on the training frame features weighted by the training weights, and a noise vector are input to the generator network as input information;
The classifier network is trained to output, as output information, information indicating that the video description is a true video description and corresponds to the second adversarial condition vector, when a second adversarial condition vector, which is a condition vector based on pre-learning frame features that are features of each of a plurality of frame images constituting the captured video included in a video text dataset including a pair of a captured video and a video description that is a text that explains the content of the captured video, and the video description are input as input information to the classifier network, and the classifier network is trained to output, as output information, information indicating that the video description is a true video description and corresponds to the second adversarial condition vector, when the first adversarial condition vector and the training video description, which is a false video description generated by the generator network, are input as input information to the classifier network,
6. The information processing device according to claim 1 or 5. the sentence generation model is a conditional variational autoencoder including an encoder and a decoder;
The encoder is trained to output a mean vector and a variance vector in a multivariate normal distribution as output information when a first variational condition vector, which is a condition vector based on pre-learning frame features that are features of each of a plurality of frame images constituting the captured video included in a video text data set including a pair of a captured video and a video description that is a text that explains the content of the captured video, and the video description are input as input information to the encoder;
a decoder that is trained to output the training video description as output information when a latent vector that is a sample following the multivariate normal distribution based on the mean vector and the variance vector, and a second variational condition vector that is a condition vector based on the training frame features weighted by the training weights are input to the decoder as input information.
6. The information processing device according to claim 1 or 5. The sentence generation model is a conditional diffusion model,
a machine learning model that generates the training video description by estimating the noise vector using the conditional diffusion model that has been trained to output, as output information, a mean vector and a variance vector in a multivariate normal distribution that the noise vector follows when a noisy video description that is the training video description including a noise vector and a diffusion condition vector that is a condition vector based on the training frame features weighted by the training weight are input as input information to the conditional diffusion model, and removing the noise vector from the noisy video description;
6. The information processing device according to claim 1 or 5. An information processing method implemented by a program executed by an information processing device, comprising:
A video generation process for generating learning videos based on an image and text data set including a pair of a captured image and an image description text that explains the content of the captured image;
An extraction step of extracting learning frame features which are features of each of a plurality of frame images constituting the learning video;
A determination step of determining learning weights corresponding to each of the plurality of frame images constituting the learning video such that the weight corresponding to the captured image is greater than the weights corresponding to other frame images other than the captured image ;
a model generation process for generating a sentence generation model, which is a machine learning model trained to generate a training video description, which is a sentence explaining the content of the training video, based on the training frame features and the training weights, the training video description having features corresponding to the training frame features weighted by the training weights;
An information processing method comprising: An information processing method implemented by a program executed by an information processing device, comprising:
an acquisition process for acquiring a sentence generation model, which is a machine learning model trained to generate training video description, which is a sentence explaining the content of the training video, based on training frame features, which are features of each of a plurality of frame images constituting the training video, generated based on an image and text dataset including a pair of a captured image and an image description, which is a sentence explaining the content of the captured image, and training weights, which are weights corresponding to each of a plurality of frame images constituting the training video, and which has features corresponding to the training frame features weighted by the training weights;
an extraction step of extracting target frame features which are features of each of a plurality of frame images constituting a target moving image which is a moving image to be processed;
a determining step of determining target weights, which are weights corresponding to each of a plurality of frame images constituting the target moving image, such that a weight corresponding to a designated frame image designated by a user is made larger than weights corresponding to other frame images other than the designated frame image;
A sentence generation process of inputting the target frame feature weighted by the target weight to the sentence generation model based on the target frame feature and the target weight to generate a target video description sentence which is a sentence explaining the content of the target video;
An information processing method comprising:

Images