JP7643638B2 - Training data generation device, training data generation method, and program - Google Patents

Description

The present invention relates to a technology for generating training data used to train a summary model for generating summary text of a video from the video.

In recent years, online meetings have become more common, and many videos of presentations at these meetings have been made available on the Internet.

Generally, presentation videos are long, so to understand the content one must watch the video for a long time. Therefore, there is a demand to understand the content of presentation videos in a short amount of time.

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

In order to quickly grasp the contents of a presentation video, it is possible to use a neural network model (called a summary model) to generate text (summary text) that represents a summary of the presentation video.

However, in the case of presentation videos, the amount of correct answer data (training data) used to train the summary model was small, and it was not possible to generate a summary model with sufficient accuracy using only the correct answer data that was collected. This issue is not limited to presentation videos, but can occur with any video for which summaries are to be generated.

The present invention has been made in consideration of the above points, and aims to provide a technology that makes it possible to generate training data for training a summary model that generates summary text from video.

According to the disclosed technology, there is provided a training data generation device for generating a training data set for training a summary model that generates a summary text for a video, the device comprising:
The present invention provides a training data generation device including: a training data generation unit configured to generate at least one further training data set from an original training data set having a first text, the first text being text extracted from an image in the video, a second text being text extracted from an audio in the video, and a ground-truth summary text of the video.

The disclosed technology provides technology that enables the generation of training data for training a summarization model that generates summary text from video.

FIG. 1 is a diagram showing a basic process flow for creating a summary text from a presentation video. FIG. 1 is a configuration diagram of a summary generation device 100. 4 is a flowchart for explaining the operation of the summary generating device 100. FIG. 2 is a configuration diagram of a summary model learning device 200. FIG. 1 illustrates a configuration for pre-training a summary model. 4 is a flowchart for explaining the operation of the summary model learning device 200. FIG. 13 is a diagram showing an example of input to a summary model and output from the summary model in pre-learning. FIG. 11 is a diagram for explaining a process of extracting an image from a moving image. FIG. 1 is a diagram for explaining extraction of text from an image. FIG. 1 is a diagram for explaining extraction of text from speech. FIG. 2 is a diagram showing an example of input to a summary model and output from the summary model in learning. FIG. 4 is a diagram showing the configuration of a data extension unit 400. 10 is a flowchart for explaining the operation of the data expansion unit 400. FIG. 13 is a diagram illustrating an example of data division. FIG. 13 is a diagram for explaining learning using a divided learning data set. FIG. 2 illustrates an example of a hardware configuration of the apparatus. FIG. 13 is a diagram showing the effect of learning paper data in advance. FIG. 13 is a diagram showing the effect of pre-learning slide summaries. FIG. 13 is a diagram showing the effect of training the original training data set together with a training data set obtained by division.

Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applicable is not limited to the following embodiment.

Both the summary generation device 100 and the summary model learning device 200 described below provide certain improvements over conventional techniques, such as generating summaries from papers, and represent an advancement in the technical field relating to techniques for generating summaries from videos.

The data augmentation unit 400 (training data generation device 400) described below provides certain improvements over conventional techniques such as manually generating summaries and represents an advancement in the technical field relating to techniques for learning summary models for generating summary text for videos.

In the following, a presentation video is used as the video for which a summary is to be generated, but this is just an example. The technology according to the present invention can be applied to any video, not just presentation videos.

(Overview of the embodiment)
In recent years, online conferences have become more common, and many videos of presentations at conferences and other events have been made public. Generally, presentation videos are long, so there is a demand for the content to be understood in a short time. In order to understand the content of a presentation video in a short time, it is desirable to be able to generate a summary of the presentation video.

Therefore, in this embodiment, we describe a technology for generating summary text corresponding to a presentation video.

<Example of presentation video>
As an example, as disclosed in "https://slideslive.com/38928967/predicting-depression-in-screening-interviews-from-latent-categorization-of-interview-prompts" (searched February 27, 2022) and "https://videolectures.net/" (searched February 27, 2022), a typical presentation video consists of an image of a slide containing the content of the presentation, an image of the presenter, and the voice of the presenter. Note that in many cases, the image of the presenter is not displayed.

<Basic process flow for creating summary text from a presentation video>
The basic process flow for creating a summary text from a presentation video will be described with reference to Fig. 1. In the following description, for convenience, the presentation video may be referred to as a "video" and the summary text may be referred to as a "summary".

First, from the video for which a summary is to be created, (A) presentation slides, (B) images extracted from the video, and (C) audio are prepared as input data for the summary generation unit 130.

It is assumed that the presentation slides in (A) are separate files from the video. Also, a summary can be generated if at least one of the three input data (A), (B), and (C) is available. However, to generate a more accurate summary, it is preferable to have three input data (A), (B), and (C), or two input data (A) and (C), or two input data (B) and (C).

Next, the input data converted into text by image recognition/voice recognition is input to the summary generation unit 130, which outputs the summary text. The summary generation unit 130 is a functional unit included in the summary generation device 100 described below.

About summary generation technology
In this embodiment, the summary generator 130 uses a neural network model (called a summary model) to generate a summary from text.

Any summarization model can be used as long as it inputs text and outputs summary text, but in this embodiment, as an example, we use a model based on BART disclosed in non-patent document 1.

BART is a model consisting of an encoder and a decoder. By using a trained model, when text is input to the encoder, a summary text is output from the decoder.

<About the assignment>
Although there have been conventional technologies that input text and output a summary, there has been no technology that outputs a summary from multimodal input data. In other words, there has been no conventional technology that appropriately generates a summary text from a video that includes audio and images (slide images, etc.), such as a presentation video.

If we were to divide the above issues into more specific issues from the perspective of the implementation form, they could be divided into issues 1 to 3 as follows.

Challenge 1: The cost of creating training data containing correct summary text to be used when training a summarization model to generate summaries for videos is high.

Challenge 2: There is no summary generation technology that uses a summary model that extracts audio and images from video and uses these as input to output summary text.

Issue 3: Even if it were possible to collect correct summary text from an external server, etc. to use when training a summary model for generating summaries for videos, the amount of data would be small, resulting in a small amount of training data and making it impossible to generate an accurate summary model.

Below, we will explain the configuration and operation of a summary generation device 100 that generates a summary from a presentation video, and a summary model learning device 200 that generates (learns) a summary model used in the summary generation device 100. The technology described below solves problems 1 to 3 above.

(Configuration and Operation of Summary Generation Device 100)
Fig. 2 shows a configuration diagram of a summary generation device 100 according to this embodiment. As shown in Fig. 2, the summary generation device 100 has an image processing unit 110, an audio processing unit 120, a summary generation unit 130, and a summary model DB (database) 140. A trained summary model is stored in the summary model DB 140. Note that the DB in this specification may also be called a memory unit or a storage unit.

The operation flow of the summary generation device 100 shown in Figure 2 will be explained with reference to the flowchart of Figure 3.

Audio information and image information are extracted from the video for which a summary is to be created, and in S101, the image information is input to the image processing unit 110, and the audio information is input to the audio processing unit 120. Note that in the example of Figure 2, it is assumed that the functional unit for extracting audio information and image information (particularly image information) from the video is external to the summary generation device 100, but the functional unit may be provided inside the summary generation device 100.

In S102, the image processing unit 110 uses image recognition technology to extract text from the image. In addition to the text, the image processing unit 110 may also extract associated auxiliary information (such as the color of the text in the slide).

In S103, the voice processing unit 120 uses voice recognition technology to extract text from the voice. Note that the order of the processes in S102 and S103 may be reversed, or S102 and S103 may be executed simultaneously.

The text extracted in S102 and the text extracted in S103 are input to the summary generation unit 130. In S104, the summary generation unit 130 generates a summary from the text extracted in S102 and the text extracted in S103 using the summary model read from the summary model DB 140. As explained in the section on learning the summary model, in addition to the text, information to which one, more than one, or all of the following features of character arrangement, image, and audio features are added may be used as input to the summary model. The actual substance of the "summary model" is data consisting of functions and weight parameters that constitute a neural network. In S104, the summary generation unit 130 outputs the generated summary.

As described above, by using both audio and image information obtained from a video, a high-quality summary can be generated.

The processing in the functional units that extract audio and image information from video, the image processing unit 110, and the audio processing unit 120, is the same as the processing in the learning data input unit 220, the image processing unit 230, and the audio processing unit 240 of the summary model learning device 200 described below, so detailed processing of these will be explained in the explanation of the summary model learning device 220.

The summary generation device 100 of this embodiment solves the above-mentioned problem 2, and realizes a summary generation technology using a summary model that extracts audio and images from video and outputs summary text using these as input. The summary model is trained by the summary model training device 200 described below.

(Configuration and operation of summary model learning device)
Fig. 4 shows an example of the configuration of a summary model learning device 200 in this embodiment. As shown in Fig. 4, the summary model learning device 200 has a data acquisition unit 210, a learning data input unit 220, an image processing unit 230, an audio processing unit 240, a summary model learning unit 250, a data extension unit 400, a model setting unit 270, a summary model DB 280 that stores previously trained summary models, and a summary model DB 290 that stores summary models currently being trained.

In this embodiment, when training the summary model, a summary model is created that has been trained in advance on a large number of summaries of papers that are considered to have a high similarity in content to the presentation, and the summary model is fine-tuned using a small amount of presentation summary data. This makes it possible to achieve high accuracy even with a small amount of correct summary data for the presentation video.

Note that performing pre-learning as described above is one method of solving problem 3. Problem 3 can also be solved by using further learning data generated by the data extension unit 400 described later without performing pre-learning. Performing pre-learning and using further learning data generated by the data extension unit 400 described later may be combined.

The configuration shown in FIG. 4 shows a configuration in the case where the above-mentioned pre-learning is performed, but learning may be performed using the learning data generated by the data extension unit 400 without performing pre-learning. In addition, learning may be performed using the learning data generated by the data extension unit 400 for a summary model that has been pre-learned.

The configuration for pre-learning is shown in Figure 5. As shown in Figure 5, the configuration for pre-learning includes a summary model pre-learning unit 310 and a summary model DB 320 that stores summary models during pre-learning.

A summary model pre-training device (a device separate from the summary model training device 200) may be configured having the summary model pre-training unit 310 and the summary model DB 320, or the summary model pre-training unit 310 and the summary model DB 320 may be included within the summary model training device 200.

The flow of operations of the summary model learning device 200 and the summary model pre-learning unit 310 will be described with reference to the flowchart in Figure 6. Detailed processing will be described later.

S201 and S202 are processes in the configuration for pre-learning shown in Figure 5. In S201, pre-learning data is input to the summary model pre-learning unit 310. The pre-learning data is, for example, the text of a paper related to the presentation and a summary of that paper (correct answer data).

In S202, the summary model pre-training unit 310 uses the input data to train (pre-train) a summary model. The pre-trained summary model is stored in the summary model DB 280 in the summary model training device 200.

S203 to S207 are processes in the summary model learning device 200 shown in Figure 4. In the input process of S203, access information (e.g., the URL where the paper and presentation video are published) is input to the data acquisition unit 210. The data acquisition unit 210 uses the access information to acquire learning data, for example from a server on the network, and inputs it to the learning data input unit 220. The learning data is, for example, a presentation video related to the paper and a correct summary text corresponding to the video. In S203, the learning data input unit 220 further performs a process of separating the presentation video into image information and audio information, inputs the image information to the image processing unit 230, inputs the audio information to the audio processing unit 240, and inputs the correct summary to the summary model learning unit 250.

The image information input by the learning data input unit 220 to the image processing unit 230 may be a slide image or the like that is a separate file from the presentation video, or a slide image or the like extracted from the presentation video. In either case, the image may be expressed as an "image in the video" or an "image related to the video". In either case, text can be extracted from the "image in the video" or the "image related to the video" by image recognition processing.

In the following explanation, it is assumed that the image information input to the image processing unit 230 is a slide image, etc. extracted from a presentation video.

In S204, the image processing unit 230 uses image recognition technology to extract text from the image. In addition to the text, the image processing unit 230 may extract accompanying auxiliary information (such as the color of the text in the slide), character arrangement features, image features, and the like.

In S205, the voice processing unit 120 uses voice recognition technology to extract text from the voice. The voice processing unit 120 may extract voice features in addition to the text. Note that the order of the processes of S204 and S205 may be reversed, or S204 and S205 may be executed simultaneously.

The text extracted in S204 and the text extracted in S205 are input to the summary model learning unit 250. The correct summary is also input to the summary model learning unit 250.

Here, the model setting unit 270 reads out a pre-trained summary model from the summary model DB 280, and the pre-trained summary model is stored in the summary model DB 290. The following learning (fine tuning) is performed using the parameters in this pre-trained summary model as initial values.

In S206, the summary model learning unit 250 uses the summary model read from the summary model DB 290 to generate a summary from the text extracted in S204 and the text extracted in S205, and learns the summary model (updates parameters) so as to minimize the error between the generated summary and the correct summary.

Once the learning is completed, the summary model learning unit 250 stores the learned summary model in the summary model DB 140 of the summary generation device 100.

Note that in the above example, pre-learning is performed to fine-tune the pre-trained learning model, but as mentioned above, pre-learning is not required. Processing may be started from S203 in FIG. 6 without performing pre-learning. When pre-learning is not performed, the initial values of the parameters of the summary model may be random values or values other than random values.

Below, the processing contents of each step from S201 to S207 are explained in more detail.

(S201, S202: Pre-learning)
A detailed example of pre-learning performed by the summary model pre-learning unit 310 shown in Fig. 5 will be described. In the pre-learning, a summary model is trained using text from a field related to the field of the presentation video to be summarized (called related field text) and its correct summary. The related field text is, for example, a paper text (text of the body of the paper), text of slides, etc.

Figure 7 shows an example of input to and output from the summary model when a research paper text is used as the related field text. As mentioned above, the summary model in this embodiment is a model consisting of an encoder and a decoder.

As shown in Figure 7, the main text of a paper is input to the encoder, and summary text is output from the decoder. A summary model is trained to minimize the error between the output summary text and the correct summary text. When slide text is used as input, the processing is the same as when paper text is used.

When text is input to the encoder, the token sequence of the text is first converted into a fixed dimension vector of d dimensions, and then converted into summary text through the encoder-decoder.

An example of input paper text is shown below.

"We assume familiarity with basic notions of graph theory (see, for instance, 1]) and with elementary notions of polyhedral combinatorics (see, for instance, 6]).", "Our graphs will be undirected and simple (no loops and no multiple edges).", "As usual, K n denotes the complete graph with n vertices; K n;m denotes the complete bipartite graph with n + m vertices and nm edges.", "Let G be a graph; G is connected if for every pair of distinct vertices there exists a path in G joining them; G is twoconnected if for every vertex v of G, the graph G ?", "v is connected; G is planar if it can be embedded in the plane.", "A subgraph H of a G is spanning if the vertex sets of H and G are the same.", "Subdivision of an edge uv of G consists of edge uv, and adding a new vertex w and the two edges uw and vw; w is called subdivision vertex.", "In automatic graph drawing the following problem arises: nd in a complete graph with weights on its edges a two-connected planar spanning subgraph with weight as Partially supported by DFG-Grant JU204/7-1 Forschungsschwerpunkt \" E ziente Algorithmen f ur diskrete Probleme und ihre Anw…"
An example of the output (or summary text, which is the correct answer data) for the above input is shown below.

"The problem of finding a two-connected planar spanning subgraph of maximum weight in a complete edge-weighted graph is important in automatic graph drawing.", "We investigate the problem from a polyhedral point of view."
On presentation video sites, slide files may be available as separate files from the video. Slide files often contain both the slide data (slide text) and a summary of the slide (summary text). In such cases, a summary model can be pre-trained by using the slide text as input to the encoder-decoder and the summary text as the correct answer.

An example of input slide text is shown below.

"[["ssn"], ["MASTERS", "IN", "AUTOMOTIVE"], ["ENGINEERING"], ["Karthiek", "Nagaraj"], ["PRESENTED", "AT", "IRIS", ",", "DEPARTMENT", "OF", "MECHANICAL", "ENGINEERING"], ["SSN"], ["WHY", "AUTOMOBILE", "ENGINEERING", "?"], ["Its", "scope", "is", "irrefutable", "and", "job", "prospects", "are", "very", "strong", "in", "any", "part", "of", "the", "world", ".", "Also", "the", "prospect", "of", "returning", "to", "India", "to", "work", "is", "bright", "as", "the", "indian", "automotive", "industry", "is", "making", "tremendous", "progress", "."], [">", "It", "is", "a", "stream", "which", "blends", "passion", "for", "vehicles", "and", "technical", "knowledge", ",", "thus", "making", "it", "all", "the", "more", "interesting", "."], ["It", "is", "an", "interdisciplinary", "field", "which", "encompasses", "mechanical", "engineering", ",", "electrical", "and", "electronics", "engineering", "and", "software", "engineering", ".", "This", "again", "adds", "to", "the", "interest", "factor", "."], ["A", "multitude", "of", "research", "options", "are", "on", "offer", ",", "especially", "in", "hybrid", "powertrains", "and", "fuel", "cells", "."], ["PRESENTED", "AT", "IRIS", ",", "DEPARTMENT", "OF", "MECHANICAL", "ENGINEERING"], ["2"], ["SSN"], ["KEY", "AREAS", "OF", "AUTOMOTIVE", "ENGINEERING"], ["Vehicle", "Propulsion", "~", "Internal", "combustion", "engines"], ["Powertrain", "dynamics", "and", "control"], ["Vehicle", "dynamics", "~", "Handling", "response"], ["~", "Advanced", "transmission"], ["systems"], ["~", "Hybrid", "propulsion", "systems"], ["~", "Terrain", "modelling"], ["~", "Fuel", "cells"], ["~", "Drivetrain", "control", "systems"], ["~", "NVH", "modelling"], ["Automotive", "body", "structures", "~", "Material", "selection"], ["Automotive", "safety", "~", "Active", "and", "passive", "safety"], ["systems"], ["~", "Crash", "worthiness"], ["~", "Human", "factor", "engineering"], ["and","
An example of the output (or a slide summary, which is the correct answer data) for the above input is shown below.

"A guide to Masters in Automotive Engineering at International Destinations"
(S203: Input process of summary model learning device 200)
Next, a detailed example of the processing by the data acquisition unit 210 and the processing by the learning data input unit 220 in the summary model learning device 200 shown in FIG. 4 will be described.

The data acquisition unit 210 accesses, for example, a presentation video site on the Internet and acquires the presentation video and a correct answer summary corresponding to the video from the site. An example of a site from which such videos and summaries can be acquired is "https://aclanthology.org/" (searched on February 27, 2022).

As described above, by obtaining the presentation video and its summary from a server on the network, learning data can be created without having to manually create a summary, thereby solving the aforementioned problem 1.

The learning data input unit 220 processes the presentation video acquired by the data acquisition unit 210 to separate it into image information and audio information, and inputs the image information to the image processing unit 230 and the audio information to the audio processing unit 240.

The image information is not limited to a specific image, but here it is assumed that the image information is a slide image in a presentation video.

Referring to Figure 8, an example of the process of extracting an image from a presentation video by the learning data input unit 220 is described.

S203(1-1):
The learning data input unit 220 extracts images from the presentation video every k seconds, where k is a predetermined real number greater than 0. The upper part of Fig. 8 shows six images extracted every k seconds.

S203(1-2):
The learning data input unit 220 compares the images extracted in S203(1-1) in order for each time, and if the similarity between the t-th image and the t-1-th image is equal to or greater than a threshold, it determines that these images are the same image. Note that any method may be used to determine the similarity between images. FIG. 8 shows an example of the similarity between each pair of six images.

S203(1-3):
The learning data input unit 220 repeats S203(1-1) and S203(1-2) to extract a set of different images. In Fig. 8, image 1, image 4, and image 6 are shown as a set of different images when the threshold value is 25. The obtained set of images is input to the image processing unit 230.

(S204: Image processing)
Next, a detailed example of image processing executed by the image processing unit 230 will be described. The image processing unit 230 performs OCR (Optical Character Recognition) processing on the set of different images input from the learning data input unit 220, and acquires text, character color, character size, character position information, and the like from each image in the set of different images, as shown in Fig. 9. Note that the acquired information may be only text.

(S205: Audio processing)
Next, a detailed example of the voice processing executed by the voice processing unit 240 will be described. As shown in Fig. 10, the voice processing unit 240 performs a voice recognition process on the voice input from the learning data input unit 220, and obtains text of the voice recognition result.

(S206: Learning process)
Next, a detailed example of the learning process executed by the summary model learning unit 250 will be described. The summary model learning unit 250 combines the text obtained by the image processing unit 230 and the text obtained by the voice processing unit 240, and inputs the combined text to the summary model. The summary model learning unit 250 learns the summary model so that the error between the summary text output from the summary model and the correct summary text is minimized. For input to the summary model, information obtained by adding character arrangement features, image features, character size, color information, etc. obtained by the image processing unit 230 to the combined text may be used. Information obtained by adding voice features obtained by the voice processing unit 240 to the combined text may also be used.

The initial state of the above summary model is the summary model pre-trained in S202. However, as described above, it is possible not to perform pre-training, so the initial state of the above summary model does not have to be the summary model pre-trained in S202. If pre-training is not performed, learning may be performed using further training data generated by the data expansion unit 400 described later.

An example of input to and output from the summary model is shown in Figure 11. As mentioned above, the summary model in this embodiment is a model consisting of an encoder and a decoder.

As shown in Figure 11, the encoder receives the text combined by [SEP], character size, and color information, and the decoder outputs summary text. The summary model is trained to minimize the error between the output summary text and the correct summary text.

When inputting text to the encoder, the text token sequence is first converted into a fixed dimension vector of d dimensions, and then converted into summary text through the encoder-decoder. Also, character size and color information may not be required in the input.

The text obtained by the voice processing unit 240 may be called ASR (Automatic Speech Recognition) text, and the text obtained by the image processing unit 240 may be called OCR text.

An example of ASR text is shown below.

"So to put in context to put my presentation in the context, I will, I would like to begin with the word decision support or decision-making. And first ask the question who, or what is making decisions and obviously we get two branches here. One is that we have a human decision maker who makes a decision and all of us are decision makers and then we are also talking about the decision systems. So robots."
An example of OCR text is shown below: The example below is an example of text obtained from a slide image disclosed in "http://videolectures.net/site/normal_dl/tag=1005123/icml2015_schmidt_time_framework_01.pdf" (retrieved February 26, 2022).

“Structured sparsity sparsity is widely used in signal processing, machine learning, and statistics (compressive sensing, sparse linear regression, etc.) Examples of sparsity….”
Below is an example of the summary text (or its correct answer) that is output when the ASR text and the OCR text are combined and input to the summary model.

“Decision Support is a discipline concerned with human decision making: it aims to provide methods and tools that support, rather than replace, people in making difficult decisions. One of the widely used decision-support approaches rely on decision models, which are developed in the decision process and used to evaluate and analyze decision alternatives. In this lecture, we shall present the method DEX (Decision EXpert), which was heavily influenced by ideas from Artificial Intelligence. DEX is a hierarchical, qualitative, rule-based, multi-criteria modeling method, particularly suitable for solving classification decision problems. DEX combines traditional approaches with those from expert systems and machine learning. DEX is supported by the software called DEXi and has been used in hundreds of real-world decision-making studies. The presentation will be illustrated by recent applications in the areas of electric energy production, food safety and health care.”
(Configuration and Operation of Data Expansion Unit 400)
The following describes one technique for solving problem 3, which is a technique for automatically generating an additional training dataset.

Figure 12 shows the configuration of the data expansion unit 400 in the summary model learning device 200 shown in Figure 4. As shown in Figure 12, the data expansion unit 400 has a learning data generation unit 410, an important sentence extraction unit 420, and a task information assignment unit 430. Note that the data expansion unit 400 may be a functional unit within the summary model learning device 200, or may be a separate device external to the summary model learning device 200. When the data expansion unit 400 is within the summary model learning device 200, the summary model learning device 200 may be called the learning data generation device 400. When the data expansion unit 400 is a separate device external to the summary model learning device 200, the separate device may be called the learning data generation device 400.

The flow of operation of the data expansion unit 400 (learning data generation device 400) shown in Fig. 12 will be described with reference to the flowchart in Fig. 13. In S301, the ASR text obtained by speech processing, the OCR text obtained by image processing, and the corresponding correct summary text are input to the learning data generation unit 410.

In S302, the data division unit 410 performs a learning data generation process (which may also be called a data division process) on the input data. In S302, an important sentence extraction process is also performed by the important sentence extraction unit 420. Note that the important sentence extraction unit 420 may be included in the learning data generation unit 410.

In S303, the task information assignment unit 430 assigns task information to the generated learning data set, and in S304, outputs the learning data set to which the task information has been assigned. The output data is input to the summary model learning unit 250 and used to learn the summary model. The processing of each of the above steps is explained in more detail below.

(S301: Input, S302: Data division)
Data consisting of "OCR text, ASR text, and correct summary text" for one presentation video is input to the learning data generation unit 410. A data set for learning is called a learning data set.

Based on the above input data, the training data generation unit 410 generates the following five training data sets as shown in FIG. 14. Note that (1) is the original training data set. Since each training data set represents a task, the training data set may be called a task. Note that the following five are examples, and in addition to the original training data set, at least one further training data set may be generated. In addition to the following, (6) OCR text, OCR key sentences, and (7) ASR text, ASR key sentences may be generated.

(1) OCR text, ASR text, correct summary text (2) OCR text, correct summary text (3) ASR text, correct summary text (4) OCR text, ASR key sentence (5) ASR text, OCR key sentence Both the ASR key sentence and the OCR key sentence are examples of pseudo-correct information. Both the ASR key sentence and the OCR key sentence are created by the key sentence extraction unit 420. An example of a method for creating these key sentences is described below.

Regarding the ASR key sentences, the key sentence extraction unit 420 extracts the ASR key sentences by matching the summary text with the ASR text. For example, the key sentence extraction unit 420 extracts, from the ASR text, a portion that is highly similar to the summary text as the ASR key sentence.

Regarding OCR important sentences, the important sentence extraction unit 420 extracts OCR important sentences by matching the summary text with the OCR text. For example, the important sentence extraction unit 420 extracts, from the OCR text, a portion that is highly similar to the summary text as an OCR important sentence.

Any method can be applied as a matching method for extracting ASR/OCR important sentences, but it is also possible to use a method used in creating data for extractive summarization, such as the method described in Fine-tune BERT for Extractive Summarization (https://arxiv.org/pdf/1903.10318v2.pdf, retrieved February 27, 2022).

(S303: Task information assignment)
The task information assigning unit 430 assigns identification information (which may be called a label) for identifying a task to each learning data set generated by the learning data generating unit 410. The identification information is a special token. In the above examples (1) to (5), for example, identification information such as [task0] is assigned as follows.

(1) [task0] OCR text, ASR text, correct summary text (2) [task1] OCR text, correct summary text (3) [task2] ASR text, correct summary text (4) [task3] OCR text, ASR key sentence (5) [task4] ASR text, OCR key sentence (S304: Output, (and learning))
Each task (each learning data set) to which identification information has been assigned in S303 is output to the summary model learning unit 250.

The summary model training unit 250 trains the summary model using each training data set with identification information. The training method for each training data set is the same as the training method in S206 described above. However, here, as shown in FIG. 15, text with the above identification information is used for input to the decoder. FIG. 15 shows an example of training for task (2) out of the five tasks above. Such training is performed for each of (1) to (5).

This allows us to increase the amount of training data and generate more accurate summary models.

(Hardware configuration example)
The summary generation device 100, the summary model learning device 200, and the training data generation device 400 can all be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud. Hereinafter, the summary generation device 100, the summary model learning device 200, and the training data generation device 400 will be collectively referred to as the "devices."

That is, the device can be realized by using hardware resources such as a CPU and memory built into a computer to execute a program corresponding to the processing performed by the device. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

Figure 16 is a diagram showing an example of the hardware configuration of the computer. The computer in Figure 16 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., which are interconnected by a bus BS.

The program that realizes the processing on the computer is provided by a recording medium 1001, such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 via the drive device 1000 into the auxiliary storage device 1002. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

When an instruction to start a program is received, the memory device 1003 reads out and stores the program from the auxiliary storage device 1002. The CPU 1004 realizes functions related to the light touch maintenance device 100 in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to a program. The input device 1007 is composed of a keyboard and mouse, buttons, a touch panel, etc., and is used to input various operational instructions. The output device 1008 outputs the results of calculations.

(Effects of the embodiment)
As described above, the technology according to the present embodiment makes it possible to appropriately generate summary text from videos that include audio and images, such as presentation videos. It also makes it possible to automatically generate additional training data for training a summary model that generates summary text from videos.

In particular, in this embodiment, the accuracy of the summary model can be improved by performing pre-learning or data augmentation (generating additional training data by data division).

Below, we explain the effects based on the experimental results when pre-learning was performed, and when data division was performed. In the following, ROUGE-1, ROUGE-2, and ROUGE-L are used as evaluation indices, and are abbreviated as R1, R2, and RL, respectively.

Figure 17 shows the effect of pre-learning paper data. For comparison, "ASR+OCR" shows the evaluation results when no paper data is pre-learned. "+Paper Summaries (300,000)" and "+Paper Summaries (500,000)" show the evaluation results when 300,000 and 500,000 paper summaries were pre-learned, respectively. As shown in Figure 17, it can be seen that accuracy is improved by pre-learning paper data.

Figure 18 shows the effect of learning the slide summaries in advance. For comparison, "ASR+OCR(4096)" shows the evaluation results when the slide summaries are not learned in advance. "+slideshare" shows the evaluation results when the slide summaries are learned in advance. As shown in Figure 18, it can be seen that accuracy is improved by learning the slide summaries in advance.

Figure 19 shows the effect of training the original learning dataset together with a further learning dataset obtained by splitting. For comparison, "ASR+OCR(4096)" shows the evaluation results when only the original learning dataset was trained. "ASR+OCR(4096)+extend" shows the evaluation results when the original learning dataset was trained together with a further learning dataset obtained by splitting. As shown in Figure 19, it can be seen that accuracy is improved by training the original learning dataset together with the learning dataset obtained by splitting.

(Additional Note)
Regarding the above embodiment, the following supplementary items are further disclosed.
(Additional Note 1)
A training data generation device for generating a training data set for training a summary model for generating a summary text for a video, comprising:
Memory,
at least one processor coupled to the memory;
Including,
The processor,
A training data generation device that generates at least one further training dataset from an original training dataset having a first text, the first text being text extracted from an image in the video, a second text, the second text being text extracted from an audio in the video, and a correct summary text of the video.
(Additional Note 2)
The processor generates, as the further training data set, a training data set that includes the first text and does not include the second text, or a training data set that includes the second text and does not include the first text.
(Additional Note 3)
The processor generates, as the further training data set, a training data set including either the first text or the second text, and important sentences obtained by matching either the first text or the second text with the correct summary text.
(Additional Note 4)
The training data generation device according to claim 1, wherein the processor assigns identification information to the further training data set for identifying a task performed by the further training data set.
(Additional Note 5)
1. A computer-implemented training data generation method for use as a training data generation device for generating a training data set for training a summary model for generating a summary text for a video, comprising:
a training data generation step of generating at least one further training data set from an original training data set having a first text, the first text being text extracted from an image in the video, a second text being text extracted from an audio in the video, and a correct summary text of the video.
(Additional Note 6)
A non-transitory storage medium storing a program executable by a computer to perform a training data generation process for generating a training data set for training a summary model that generates a summary text for a video, the non-transitory storage medium comprising:
The learning data generation process includes:
A non-transitory storage medium for generating at least one further training dataset from an original training dataset having a first text, the first text being text extracted from images in the video, a second text being text extracted from audio in the video, and a ground-truth summary text of the video.

Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention as described in the claims.

100 Summary generation device 110 Image processing unit 120 Audio processing unit 130 Summary generation unit 140 Summary model DB
200 Summary model learning device 210 Data acquisition unit 220 Learning data input unit 230 Image processing unit 240 Audio processing unit 250 Summary model learning unit 270 Model setting unit 280 Summary model DB
290 Summary Model DB
310 Summary model pre-learning unit 320 Summary model DB
400 Data expansion unit 410 Learning data generation unit 420 Key sentence extraction unit 430 Task information assignment unit 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU
1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims (6)

A training data generation device for generating a training data set for training a summary model for generating a summary text for a video, comprising:
a training data generation unit configured to generate at least one further training data set from an original training data set having a first text, the first text being text extracted from an image in the video, a second text being text extracted from an audio in the video, and a correct summary text of the video. 2. The training data generation device according to claim 1, wherein the training data generation unit generates, as the further training data set, a training data set that includes the first text and does not include the second text, or a training data set that includes the second text and does not include the first text. 2. The training data generation device according to claim 1, wherein the training data generation unit generates, as the further training data set, a training data set including either the first text or the second text, and important sentences obtained by matching either the first text or the second text with the correct summary text. The training data generation device according to claim 1 , further comprising: a task information assigning unit that assigns, to the further training data set, identification information for identifying a task performed by the further training data set. 1. A computer-implemented training data generation method for use as a training data generation device for generating a training data set for training a summary model for generating a summary text for a video, comprising:
a training data generation step of generating at least one further training data set from an original training data set having a first text, the first text being text extracted from an image in the video, a second text being text extracted from an audio in the video, and a correct summary text of the video. A program that causes a computer to function as each component of a training data generation device described in any one of claims 1 to 4.

Images