JP7652252B2 - Image processing device, image processing method, and program - Google Patents

Description

This disclosure relates to processing of video data.

A technology has been proposed for generating a video digest from video. Patent Document 1 discloses a highlight extraction device that creates a learning data file from training video prepared in advance and video of important scenes specified by the user, and detects important scenes from a target video based on the learning data file.

JP 2008-022103 A

In digest videos of sports and the like, a single scene is often composed of multiple implicitly determined images. For example, in a digest video of a soccer game, a single goal scene will often be shown with footage of the player's shot from the moment he scores the goal, taken with a camera capturing an overhead view of the entire field, followed by footage of the goal scene taken from a different angle, or footage of the player's goal celebration. Therefore, when creating a digest video, it is necessary to detect multiple short images related to a single scene.

One objective of the present disclosure is to provide a video processing device capable of detecting, with high accuracy, multiple short videos that constitute important scenes contained in raw video.

In one aspect of the present invention, a video processing device includes:
a video acquisition means for acquiring raw video and digest video;
a matching portion detection means for detecting a matching portion where the contents of the raw video and the digest video match;
a grouping means for grouping matching portions whose temporal distance is equal to or less than a predetermined value as a single important scene;
a classification means for classifying the matching portions in each group obtained by the grouping into different classes and classifying the material video into the classes;
an event determination means for determining a key event based on the frequency of appearance of each class included in the raw video;
Equipped with.

In another aspect of the present invention, a video processing method includes:
Obtain raw footage and digest footage,
Detecting a portion where the contents of the raw video and the digest video match;
Matching parts whose temporal distance is less than a predetermined value are grouped as the same important scene ,
In each group obtained by the grouping, the matching portions are classified into different classes, and the material video is classified into the classes;
A key event is determined based on the frequency of appearance of each class contained in the raw video .

In yet another aspect of the invention, a program comprises:
Obtain raw footage and digest footage,
Detecting a portion where the contents of the raw video and the digest video match;
Matching parts whose temporal distance is less than a predetermined value are grouped as the same important scene ,
In each group obtained by the grouping, the matching portions are classified into different classes, and the material video is classified into the classes;
The computer executes a process for determining a key event based on the frequency of appearance of each class contained in the raw video .
In still another aspect of the present invention, a video processing device includes:
An image acquisition means for acquiring an input image;
an important scene detection means for detecting an important scene from the input video;
a first partial image extraction means for extracting, from the input image, a temporally continuous area including at least a portion following the important scene as a first partial image;
a related event detection means for detecting a related event from the first partial video by using a related event detection model trained to detect a related event related to the important scene from a partial video including at least a part following the important scene;
Equipped with.

This disclosure makes it possible to detect multiple short videos that constitute important scenes contained in raw footage with high accuracy.

1 shows a schematic configuration of a digest video generating device. An example of a digest video is shown below. Here is an example of the elements that make up a goal scene. 1 illustrates a method for detecting multiple images that compose one important scene. FIG. 2 is a block diagram showing a hardware configuration of a digest video generation device. FIG. 2 is a block diagram showing a functional configuration of a digest video generating device. 13 is a flowchart of a digest video generation process. 1 shows an example of a configuration for learning an important scene detection unit. 13 shows an example of a configuration for learning of the related event detection unit. 1 shows a training data generating device according to a second embodiment. 1 shows a process for comparing a raw video with a digest video. The process of grouping matching parts is shown below. Illustrates learning and inference of a video recognition model. FIG. 2 is a block diagram showing a hardware configuration of the training data generating device. FIG. 2 is a block diagram showing a functional configuration of the training data generating device. 13 is a flowchart of a learning data generation process. FIG. 11 is a block diagram showing a functional configuration of a video processing device according to a third embodiment. 13 is a flowchart of a process performed by a video processing device according to a third embodiment. FIG. 13 is a block diagram showing a functional configuration of a video processing device according to a fourth embodiment. 13 is a flowchart of a process performed by a video processing device according to a fourth embodiment.

Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings.
First Embodiment
[Digest video generating device]
1 shows a schematic configuration of a digest video generating device. The digest video generating device 100 is connected to a raw video database (hereinafter, "database" is also referred to as "DB") 2. The raw video DB 2 stores various raw videos, i.e., moving images. The raw videos may be, for example, videos of television programs broadcast by a broadcasting station, or videos distributed via the Internet or the like. The raw videos may or may not include audio.

The digest video generating device 100 generates and outputs a digest video using a portion of the material video stored in the material video DB 2. The digest video is a video in which scenes in which some event occurs in the material video are linked in chronological order. As described below, the digest video generating device 100 detects important scenes from the material video using an important scene detection model trained by machine learning, and links the important scenes in chronological order to generate a digest video. The important scene detection model is a model that detects sections of important scenes from the material video, and can be, for example, a deep learning model using a neural network.

Figure 2 shows an example of a digest video. In the example of Figure 2, the digest video generating device 100 extracts important scenes A to D contained in the raw video and connects them in chronological order to generate a digest video. Note that important scenes extracted from the raw video may be used repeatedly in the digest video depending on their content. An important scene is composed of multiple frame images corresponding to a scene in which some event occurred in the raw video. An important scene is defined by its start point and end point. Note that an important scene may be defined by the length of the important scene instead of the end point.

[Basic Principles]
First, the basic principle of the digest video generating device according to this embodiment will be described. In sports digest videos, one important scene is often composed of multiple short videos that are implicitly determined. For example, in a soccer goal scene, a video of the player's shot to the goal is often shown with a camera that shoots the entire field from a bird's-eye view (hereinafter referred to as a "bird's-eye view camera"), followed by a video of the player's goal celebration, or a video of the goal scene shot with a camera at a different angle.

Figure 3 shows an example of the elements that make up a goal scene. In this example, one goal scene is made up of images 1 to 3. Image 1 is an image of the goal scene captured by an overhead camera, image 2 is an image of the goal scene captured by a different camera, and image 3 is an image of the player's goal celebration.

In a typical digest video generation method, an important scene detection model is trained by treating all the parts used in the digest video as important scenes, and important scenes are detected from the raw video by inference using the model. For this reason, as shown in Figure 3, when an important scene is detected from a certain raw video A using a trained important scene detection model, video 1 shot with an overhead camera may be detected, but video 3 of a player's goal performance may not be detected. In other words, the above important scene detection model may only be able to detect some of the multiple components that make up one important scene.

Therefore, in this embodiment, the multiple elements (images) that make up one important scene are captured in a hierarchical structure, and two-stage processing is performed to detect multiple images from the raw video. Figure 4 shows a schematic diagram of a method for detecting multiple images that make up one important scene. First, the digest video generation device 100 detects important scenes from the raw video using an important scene detection model. In Figure 4, the important scene detection model detects one important scene 51 from raw video A. In this example, important scene 51 is a goal-scoring scene captured by an overhead camera. The important scene event detected in this way is also called a "key event."

Next, the digest video generation device 100 detects related events related to the important scene based on the detected important scene. Specifically, the digest video generation device 100 cuts out video around the detected important scene and detects related events contained in the cut-out video. The detection of related events is performed using a related event detection model that has been trained in advance. That is, a partial video cut out around the important scene (hereinafter also referred to as "important scene partial video") is input to the related event detection model, and the related event detection model infers and outputs related events.

In the example of Figure 4, important scene partial video 52a is extracted from material video A based on important scene 51 detected from material video A. Important scene partial video 52b is also extracted from material video B, which is synchronized in time with material video A, based on important scene 51. Important scene partial videos 52a and 52b are then input into a related event detection model to detect related events. Specifically, video of a player's goal performance is detected from important scene partial video 52a, and video of a goal scene captured by another camera is detected from important scene partial video 52b.

In this manner, in this embodiment, an important scene is first detected from the raw footage, and then related events are detected from the partial footage surrounding the important scene, so that multiple images related to the important scene can be extracted from the raw footage. Thus, one important scene can be composed of a video of a key event corresponding to the important scene and a video of a related event. In the example of FIG. 4, raw footage A and B can be used to include in the digest video an image of one important scene, including a video of the goal scene captured by an overhead camera, a video of the goal scene captured by another camera, and a video of the player's goal performance.

[Hardware configuration]
5 is a block diagram showing a hardware configuration of the digest video generation device 100. As shown in the figure, the digest video generation device 100 includes an interface (IF) 11, a processor 12, a memory 13, a recording medium 14, and a database (DB) 15.

IF11 inputs and outputs data to and from external devices. Specifically, interface 11 receives raw footage from raw footage DB2 or the like that is the basis for creating a digest video, and outputs the generated digest video.

The processor 12 is a computer such as a CPU (Central Processing Unit) and controls the entire digest video generating device 100 by executing a pre-prepared program. Specifically, the processor 12 executes the digest video generating process described below.

The memory 13 is composed of a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 13 is also used as a working memory while the processor 12 is executing various processes.

The recording medium 14 is a non-volatile, non-temporary recording medium such as a disk-shaped recording medium or a semiconductor memory, and is configured to be detachable from the digest video generating device 100. The recording medium 14 records various programs executed by the processor 12. When the digest video generating device 100 executes various processes, the programs recorded on the recording medium 14 are loaded into the memory 13 and executed by the processor 12.

The database 15 temporarily stores raw footage input via the IF 11, generated digest footage, etc. The digest footage generating device 100 may also include an input unit such as a keyboard or mouse for the creator to enter instructions or input, and a display unit such as a liquid crystal display.

[Functional configuration]
6 is a block diagram showing the functional configuration of the digest video generation device 100. The digest video generation device 100 includes a video acquisition unit 21, an important scene detection unit 22, a first video cut-out unit 23, a related event detection unit 24, a second video cut-out unit 25, and a video combination unit 26.

The video acquisition unit 21 acquires input video from the outside and outputs it to the important scene detection unit 22, the first video extraction unit 23, and the second video extraction unit 25. The input video is raw video that is the basis of the digest video. The input video may be so-called switching video that includes video shot by multiple cameras. In addition to a certain input video, the video acquisition unit 21 may also input one or more videos (e.g., video shot by another camera) that are time-synchronized with the input video.

The important scene detection unit 22 detects important scenes from the input video. An important scene is a scene in the input video in which a noteworthy event occurs, such as one that is used in a digest video. The important scene detection unit 22 performs inference using a pre-trained important scene detection model to detect important scenes from the input video. The important scene detection unit 22 outputs information about the detected important scenes, such as time information about the important scenes in the input video, to the first video extraction unit 23 and the second video extraction unit 25.

The first video cutout unit 23 cuts out a video around an important scene from the input video as a partial video. Specifically, the first video cutout unit 23 cuts out a temporally continuous area including an important scene from the input video as an important scene partial video. The important scene partial video includes the important scene and one or both of a portion before and a portion after the important scene in terms of time, and corresponds to the first partial video. For example, the first video cutout unit 23 cuts out an area including a portion of a predetermined time width before the important scene in the input video, the important scene, and a portion of a predetermined time width after the important scene as the important scene partial video.

The first video cutout unit 23 may perform section detection from the input video around the important scene and cut out a partial video at the boundary of the detected video section. Section detection is a process of detecting the scene switching position (boundary position) in the input video. This allows the start and end points of the important scene partial video to match the scene switching position in the input video. In this case, the section detection process can use a method of detecting a section based on the change amount of the color histogram or the change amount of the sound of the input video, or a deep learning model that finds the start and end points. If a deep learning model that finds the start and end points (for example, BMN (Boundary-Matching Network)) is used, the start and end points of the section can be detected even in places where the color or sound in the input video does not change significantly. The first video cutout unit 23 outputs the cut out important scene partial video to the related event detection unit 24.

In addition, when multiple input videos that are synchronized in time are input to the video acquisition unit 21, the first video extraction unit 23 may extract important scene portion videos from each of the multiple input videos.

The related event detection unit 24 detects related events from the input important scene partial video. Specifically, the related event detection unit 24 uses a previously trained related event detection model to infer the important scene partial video and detects related events. The related event detection unit 24 detects one important scene as a key event and detects one or more related events related to the key event. In the example of FIG. 3, the related event detection unit 24 detects a goal scene captured by an overhead camera as an important scene (key event) and detects a goal scene captured by another camera and a goal performance scene of a player as a related event. The learning of the related event detection unit 24 will be described later. The related event detection unit 24 outputs information on the detected related events, such as time information on each related event in the input video, to the second video cut-out unit 25.

The second video cut-out unit 25 cuts out partial video of each event from the input video. Specifically, the second video cut-out unit 25 cuts out a portion of an important scene from the input video as a key event partial video based on the time information of the important scene input from the important scene detection unit 22. The second video cut-out unit 25 also cuts out a portion of a related event from the input video as a related event partial video based on the time information of each related event input from the related event detection unit 24. The key event partial video and the related event partial video correspond to the second partial video. The second video cut-out unit 25 then outputs the key event partial video and one or more related event partial videos to the video combination unit 26.

The second video clipping unit 25 may detect sections from the input video based on the important scenes detected by the important scene detection unit 22 and the related events detected by the related event detection unit 24, and clip the key event partial video and the related event partial video at the boundaries of the detected video sections. In this case, the section detection process may use a method for detecting sections based on the amount of change in the color histogram or the amount of change in the audio of the input video, or a deep learning model for finding the start and end points.

The video combining unit 26 combines the key event partial video and the related event partial video input from the second video extraction unit 25 to generate and output a digest video. The video combining unit 26 basically combines the key event partial video and the related event partial video in the order indicated by their time information. However, the video combining unit 26 may combine multiple related event partial videos in a predetermined order. For example, the video combining unit 26 may combine soccer goal scenes so that a goal scene from a separate camera is always followed by a scene of the player's goal celebration.

In the above configuration, the video acquisition unit 21 is an example of a video acquisition means, the important scene detection unit 22 is an example of an important scene detection means, the first video cut-out unit 23 is an example of a first partial video extraction means, and the related event detection unit 24 is an example of a related event detection means. The second video cut-out unit 25 is an example of a second partial video extraction means, and the video combination unit 26 is an example of a video combination means.

[Digest video generation process]
7 is a flowchart of the digest video generation process by the digest video generation device 100. This process is realized by the processor 12 shown in FIG. 5 executing a program prepared in advance and operating as each element shown in FIG.

First, the video acquisition unit 21 acquires an input video (step S21). The input video may be a single video or multiple videos synchronized in time. Next, the important scene detection unit 22 detects an important scene from the input video (step S22). Next, the first video cut-out unit 23 cuts out the surroundings of the important scene from the input video and outputs the important scene partial video (step S23).

Next, the related event detection unit 24 detects related events from the important scene partial video (step S24). Next, the second video extraction unit 25 extracts, from the input video, a key event partial video corresponding to the important scene and a related event partial video corresponding to the related event (step S25). Next, the video combination unit 26 combines the key event partial video and the related event partial video to generate a digest video (step S26). Then, the process ends.

[Learning of Important Scene Detection Unit]
Next, the learning of the important scene detection unit 22 will be described. FIG. 8 shows an example of a configuration for learning the important scene detection unit 22. The important scene detection unit 22 is connected to the learning unit 28. The important scene detection unit 22 and the learning unit 28 receive learning data prepared in advance. Specifically, the learning data includes a learning input video and correct answer data. The correct answer data is data such as time information indicating a section of an important scene in the learning input video. The learning data is generated, for example, by using a digest video prepared in advance. Specifically, the learning input video is compared with the digest video prepared in advance, and data indicating a section of the learning input video that matches the digest video is set as correct answer data.

During learning, the learning input video of the learning data is input to the important scene detection unit 22, and the correct answer data is input to the learning unit 28. The important scene detection unit 22 detects important scenes from the learning input video using an important scene detection model, and outputs the important scenes to the learning unit 28. As the important scene detection model, for example, a CNN (Convolutional Neural Network) can be used. The learning unit 28 optimizes the important scene detection model based on the important scenes detected by the important scene detection unit 22 and the correct answer data. In this way, learning of the important scene detection unit 22 is performed.

[Learning of related event detection unit]
Next, the learning of the associated event detection unit 24 will be described. Fig. 9 shows an example of a configuration for learning of the associated event detection unit 24. The associated event detection unit 24 is connected to a learning unit 29. Pre-prepared learning data is input to the associated event detection unit 24 and the learning unit 29. Specifically, the learning data includes learning input video and correct answer data. The correct answer data is data such as time information indicating a section of an associated event related to an important scene (key event).

The related event detection unit 24 detects related events using a related event detection model. In one example, a model that executes an image classification task is used as the related event detection model, and may be a model that uses, for example, CNN or 3D CNN that uses time information. In another example, an action recognition model may be used as the related event detection model. In either case, the related event detection unit 24 may determine that the input video is always one of the classes, or may determine that a portion where the score at the time of inference is smaller than a predetermined threshold is a non-important scene.

In the learning process of the related event detection unit 24, image matching is performed between digest footage edited for news programs and the like and the raw footage before editing, and learning is performed by classifying the sections where the two continuously match into classes. In this process, parts of the raw footage that are not used as digest footage are classified as non-important scene classes, and classes that are used as digest footage are classified into several classes to generate learning data and perform learning.

In addition, in the learning process of the related event detection unit 24, image matching may be performed between a digest video edited for a news program or the like and the raw video before editing, and a partial video clip cut out from a few seconds of video around the point where the two continuously match may be used as input to generate the above learning data and perform learning.

In addition, the learning of the related event detection unit 24 may involve image matching between digest footage edited for a news program or the like and raw footage before editing, performing section detection processing based on the parts where the two consecutively match, and generating the above-mentioned learning data using partial footage cut out from the detected section as input for learning. In this case, the section detection processing may involve a method of detecting sections based on the amount of change in the color histogram or the amount of change in the audio of the input video, or a deep learning model that determines the start and end points.

Second Embodiment
The second embodiment relates to a method for generating learning data used for learning the important scene detection unit 22 and the related event detection unit 24 in the first embodiment.

[Learning data generation device]
10 shows a training data generation device 200 according to the second embodiment. A raw video and a digest video are input to the training data generation device 200. The training data generation device 200 generates and outputs training data using the input raw video and digest video.

[Basic Principles]
The basic principle of the training data generating device 200 according to the second embodiment will be described. A raw video and a digest video are input to the training data generating device 200. As shown in FIG. 11, the training data generating device 200 first compares the raw video with the digest video. Specifically, the training data generating device 200 compares an already created digest video such as a news video with raw videos A and B, and detects matching portions of the videos. In this example, the raw video B is a video shot by a different camera from the video included in the raw video A. In the example of FIG. 11, four matching portions P1 to P4 are detected from the raw video A, and two matching portions P5 to P6 are detected from the raw video B.

Next, as shown in Fig. 12, the training data generation device 200 groups the detected matching locations based on their positions on the time axis. Specifically, the training data generation device 200 groups matching locations whose distance on the time axis is closer than a predetermined value into the same group, and matching locations whose distance on the time axis is farther than the predetermined value into another group. In the example of Fig. 12, the training data generation device 200 groups matching locations P1, P5, and P2 into group 1, matching locations P3 and P4 into group 2, and matching location P6 into group 3 based on the distance on the time axis.

Next, as shown in Fig. 13(A), the training data generation device 200 trains a video recognition model for each group by treating the video of the matching parts as a separate class. In the example of Fig. 13(A), the video recognition model is trained by treating the matching parts P1, P5, and P2 grouped in group 1 as class 1, class 2, and class 3, respectively.

Next, the training data generation device 200 inputs the raw videos A and B into the trained video recognition model to perform inference, and detects key events and related events based on the inference results. Specifically, the training data generation device 200 classifies the raw videos into a plurality of classes using the video recognition model. Then, the training data generation device 200 determines that, among the classes belonging to that group, the class that appears least frequently in the classification results is the key event, and determines the other classes as related events.

The reason why the class with the lowest frequency of occurrence is determined to be the key event is as follows. Normally, raw video is composed of characteristic video that is included in the digest video as an important scene, and general video that is commonly included in other scenes. Therefore, the learning data generation device 200 estimates the class that appears least frequently in the raw video among the multiple classes in each group as the characteristic video, and estimates the other classes as general video. The learning data generation device 200 then determines the class corresponding to the characteristic video as the key event, and determines the classes corresponding to the other general videos as related events.

In the example of Figure 13 (B), the class that appears least frequently in the raw video is class 1. Therefore, the learning data generation device 200 determines class 1 to be a key event, and classes 2 and 3 to be related events. In this way, learning data can be generated by assigning flags of key events and related events to the raw video as correct answer data.

[Hardware configuration]
14 is a block diagram showing a hardware configuration of the training data generation device 200. As shown in the figure, the training data generation device 200 includes an interface (IF) 211, a processor 212, a memory 213, a recording medium 214, and a database (DB) 15.

IF211 inputs and outputs data between external devices. Specifically, interface 211 receives raw footage and digest footage, and outputs the generated learning data.

The processor 212 is a computer such as a CPU, and controls the entire learning data generation device 200 by executing a pre-prepared program. Specifically, the processor 212 executes the learning data generation process described below.

The memory 213 is composed of ROM, RAM, etc. The memory 213 is also used as a working memory while the processor 212 is executing various processes.

The recording medium 214 is a non-volatile, non-temporary recording medium such as a disk-shaped recording medium or a semiconductor memory, and is configured to be detachable from the learning data generation device 200. The recording medium 214 records various programs executed by the processor 212. When the learning data generation device 200 executes various processes, the programs recorded on the recording medium 214 are loaded into the memory 213 and executed by the processor 212.

The database 215 temporarily stores raw footage, digest footage, and the like input through the IF 211. The learning data generation device 200 may also include an input unit such as a keyboard or mouse for the creator to enter instructions or input, and a display unit such as a liquid crystal display.

[Functional configuration]
15 is a block diagram showing the functional configuration of the training data generation device 200. The training data generation device 200 includes a video acquisition unit 221, a video matching unit 222, a grouping unit 223, a model learning unit 224, an inference unit 225, an event determination unit 226, and a training data generation unit 227.

The video acquisition unit 221 acquires raw video and digest video. The raw video is the source video for generating learning data. The digest video is video generated for purposes such as news footage. The video acquisition unit 221 outputs the raw video and digest video to the video matching unit 222, and outputs the raw video to the inference unit 225 and the learning data generation unit 227.

As shown in Fig. 11, the video matching unit 222 matches the raw video with the digest video and detects matching parts of the raw video with the digest video. The video matching unit 222 outputs the detected matching parts to the grouping unit 223.

As shown in FIG. 12, the grouping unit 223 groups the input matching locations based on their positions on the time axis. Specifically, the grouping unit 223 includes matching locations whose distance on the time axis is closer than a predetermined value in the same group, and includes matching locations whose distance on the time axis is farther than a predetermined value in another group. The grouping unit 223 outputs the grouping result to the model learning unit 224.

As shown in FIG. 13(A), the model learning unit 224 learns a video recognition model that recognizes and classifies the material video. Specifically, for each group generated by the grouping unit 223, the model learning unit 224 learns a video recognition model that recognizes and classifies each matching portion contained in the group as a separate class. The video recognition model can be an image recognition model or an image classification model using deep learning. The trained video recognition model obtained by training can classify the material video into multiple classes contained in the group. The model learning unit 224 outputs the trained video recognition model to the inference unit 225.

The inference unit 225 uses the video recognition model learned by the model learning unit 224 to perform inference on the raw video as shown in FIG. 13(B) and classify the raw video into multiple classes. The inference unit 225 then outputs the inference result to the event determination unit 226.

The event determination unit 226 detects events from the raw video based on the inference result by the inference unit 225. Specifically, the event determination unit 226 determines the class with the lowest occurrence frequency in the inference result as a key event, and determines other events as related events. The event determination unit 226 outputs the determination result to the learning data generation unit 227.

The learning data generation unit 227 generates learning data in which the determination results of key events and related events are used as correct answer data. Specifically, the learning data generation unit 227 generates and outputs learning data in which a flag indicating a key event and/or related event is added to the raw video. For example, learning data in which a key event flag is added to the raw video can be generated and used for training the important scene detection unit 22 in the first embodiment. Furthermore, learning data in which a related event flag is added to the raw video can be generated and used for training the related event detection unit 24 in the first embodiment.

In the above configuration, the video acquisition unit 221 is an example of a video acquisition means, the video matching unit 222 is an example of a matching part detection means, and the grouping unit 223 is an example of a grouping means. In addition, the model learning unit 224 is an example of a learning means, the inference unit 225 is an example of a classification means, the event determination unit 226 is an example of an event determination means, and the learning data generation unit 227 is an example of a learning data generation means.

[Learning data generation process]
Fig. 16 is a flowchart of a learning data generation process by the learning data generation device 200. This process is realized by the processor 212 shown in Fig. 14 executing a program prepared in advance and operating as each element shown in Fig. 15.

First, the video acquisition unit 221 acquires the raw video and the digest video (step S41). Next, the video matching unit 222 detects parts of the raw video that match the digest video (step S42). Next, the grouping unit 223 groups the detected matching parts that are close in time (step S43).

Next, the model learning unit 224 learns a video recognition model by treating each of the matching parts included in one group obtained by grouping as a separate class (step S44). Next, the inference unit 225 infers the raw video using the trained video recognition model and classifies the raw video into a plurality of classes (step S45). Next, the event determination unit 226 determines the events in the group based on the inference result by the inference unit 225 (step S46). Specifically, the event determination unit 226 determines the class that appears least frequently in the raw video as a key event, and determines the other classes as related events.

Next, the learning data generation device 200 determines whether or not all groups obtained in step S43 have been processed (step S47). If not all groups have been processed (step S47: No), the processes of steps S44 to S47 are repeated for another group. Then, when all groups have been processed (step S47: Yes), the learning data generation unit 227 generates and outputs learning data in which flags are added to locations of key events and/or related events in the material video (step S48). The learning data generation process then ends.

[Modification]
Next, a description will be given of modifications of the second embodiment. The following modifications can be applied to the second embodiment in appropriate combination.

(Variation 1)
In the above-described second embodiment, the model learning unit 224 learns a video recognition model, and the inference unit 225 performs inference on the material video using the trained video recognition model, thereby classifying the material video into a plurality of classes. Alternatively, the material video may be classified into a plurality of classes without training a model.

As a specific method, the training data generation device 200 first uses a pre-trained video recognition model to convert the matching parts detected by the video matching unit 222 into features as separate classes. For example, each matching part can be input to a deep learning network that constitutes the video recognition model, and features obtained from the layer before the final layer can be used.

Next, the training data generation device 200 generates features for the raw video using the same method as above. The training data generation device 200 then compares the features of each frame image of the raw video with the features of each class corresponding to each of the above matching locations, and determines the class with the highest similarity (closest feature distance) as the class to which the frame image belongs. In this way, the training data generation device 200 classifies each frame image of the raw video into a plurality of classes, and determines the class with the lowest frequency of occurrence in the raw video to be the key event. In this case, the training data generation device 200 also functions as first and second feature extraction means.

(Variation 2)
In the above example, the event determination unit 226 determines the class that appears least frequently in the material video as the key event based on the inference result of the material video using the video recognition model. Alternatively, the event determination unit 226 may determine the class that appears least frequently in the material video and that is always present in the matching portion detected by the video matching unit 222 as the key event. This makes it possible to reliably detect the class included in the digest video as the key event.

Third Embodiment
Next, a third embodiment of the present disclosure will be described. Fig. 17 is a block diagram showing a functional configuration of a video processing device according to the third embodiment. As shown in the figure, a video processing device 70 includes a video acquisition unit 71, an important scene detection unit 72, a first partial video extraction unit 73, and a related event detection unit 74.

Figure 18 is a flowchart of processing by the video processing device 70. The video acquisition means 71 acquires an input video (step S71). The important scene detection means 72 detects important scenes from the input video (step S72). The first partial video extraction means 73 extracts a temporally continuous area including important scenes from the input video as a first partial video (step S73). Then, the related event detection means 74 detects related events related to the important scenes from the first partial video (step S74).

According to the video processing device 70 of the third embodiment, important scenes and related events related to those important scenes can be detected from the input video.

Fourth Embodiment
Next, a fourth embodiment of the present disclosure will be described. Fig. 19 is a block diagram showing a functional configuration of a video processing device according to the fourth embodiment. As shown in the figure, a video processing device 80 includes a video acquisition unit 81, a matching portion detection unit 82, and a grouping unit 83.

Figure 20 is a flowchart of processing by the video processing device 80. The video acquisition means 81 acquires raw video and digest video (step S81). The matching portion detection means 82 detects matching portions where the content of the raw video and digest video matches (step S82). Then, the grouping means 83 groups matching portions whose temporal distance is less than a predetermined value as the same important scene (step S83).

According to the fourth embodiment of the video processing device 80, sections of the raw video that match the digest video can be grouped as the same important scene.

Some or all of the above embodiments may also be described as, but are not limited to, the following notes:

(Appendix 1)
An image acquisition means for acquiring an input image;
an important scene detection means for detecting an important scene from the input video;
a first partial image extraction means for extracting, from the input image, a temporally continuous area including the important scene as a first partial image;
a related event detection means for detecting a related event related to the important scene from the first partial video;
A video processing device comprising:

(Appendix 2)
The video processing device described in Appendix 1, wherein the first partial image extraction means extracts an area in the input video that includes the important scene and at least one of a portion temporally before and a portion temporally after the important scene as the first partial image.

(Appendix 3)
3. The video processing device according to claim 1, wherein the first partial video extraction means detects a section of the input video and cuts out the first partial video at a boundary position of the detected section.

(Appendix 4)
The image acquisition means acquires another input image that is synchronized in time with the input image,
The image processing device according to any one of claims 1 to 3, wherein the first partial image extraction means extracts a temporally continuous area including the important scene from the other input image as the first partial image.

(Appendix 5)
a second partial image extraction means for extracting, from the input image, an area corresponding to the important scene and an area corresponding to the related event as a second partial image;
a video combining means for combining the second partial videos to generate a digest video;
5. The image processing device according to claim 1, further comprising:

(Appendix 6)
6. The video processing device according to claim 5, wherein the second partial video extraction means detects a section of the input video and extracts the second partial video at a boundary position of the detected section.

(Appendix 7)
Get the input video,
Detecting important scenes from the input video;
extracting a temporally continuous area including the important scene from the input video as a first partial video;
A video processing method for detecting a relevant event related to the important scene from the first partial video.

(Appendix 8)
Get the input video,
Detecting important scenes from the input video;
extracting a temporally continuous area including the important scene from the input video as a first partial video;
A recording medium having recorded thereon a program for causing a computer to execute a process for detecting a relevant event related to the important scene from the first partial video.

(Appendix 9)
a video acquisition means for acquiring raw video and digest video;
a matching portion detection means for detecting a matching portion where the contents of the raw video and the digest video match;
a grouping means for grouping matching portions whose temporal distance is equal to or less than a predetermined value as a single important scene;
A video processing device comprising:

(Appendix 10)
a learning means for learning a video recognition model that classifies the matching portions in each group obtained by the grouping into different classes and classifies the video into the classes;
a classification means for classifying the raw video into the classes using the video recognition model obtained by learning;
an event determination means for determining a key event based on the frequency of appearance of each class included in the raw video;
10. The image processing device according to claim 9, comprising:

(Appendix 11)
a first feature extraction means for extracting features of each of the matching portions by classifying the matching portions into different classes in each group obtained by the grouping;
a second feature extraction means for extracting a feature of each frame image of the raw video;
a classification means for classifying each frame of the raw video into the class based on the feature amount of each of the matching portions and the feature amount of each frame of the raw video;
an event determination means for determining a key event based on the frequency of appearance of each class included in the raw video;
10. The image processing device according to claim 9, comprising:

(Appendix 12)
12. The video processing device according to claim 10, wherein the event determination means determines a class having a lowest occurrence frequency as a key event.

(Appendix 13)
12. The video processing device according to claim 10, wherein the event determination means determines a class that appears least frequently and is included in the matching portion as a key event.

(Appendix 14)
The video processing device according to any one of claims 10 to 13, wherein the event determination means determines a class other than the class determined to be the key event as an associated event.

(Appendix 15)
15. The video processing device according to claim 14, further comprising: a learning data generating means for generating learning data to which flags corresponding to the key event and the related event are added.

(Appendix 16)
Obtain raw footage and digest footage,
Detecting a portion where the contents of the raw video and the digest video match;
A video processing method for grouping matching portions whose temporal distance is equal to or less than a predetermined value as the same important scene.

(Appendix 17)
Obtain raw footage and digest footage,
Detecting a portion where the contents of the raw video and the digest video match;
A recording medium having recorded thereon a program for causing a computer to execute a process of grouping matching portions whose temporal distance is equal to or less than a predetermined value as the same important scene.

Although the present disclosure has been described above with reference to embodiments and examples, the present disclosure is not limited to the above embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

Reference Signs List 12, 212 Processor 21, 221 Video acquisition unit 22 Important scene detection unit 23 First video extraction unit 24 Related event detection unit 25 Second video extraction unit 26 Video combination unit 100 Digest video generation device 200 Learning data generation device 222 Video matching unit 223 Grouping unit 224 Model learning unit 225 Inference unit 226 Event determination unit 227 Learning data generation unit

Claims (12)

a video acquisition means for acquiring raw video and digest video;
a matching portion detection means for detecting a matching portion where the contents of the raw video and the digest video match;
a grouping means for grouping matching portions whose temporal distance is equal to or less than a predetermined value as a single important scene;
a classification means for classifying the matching portions in each group obtained by the grouping into different classes and classifying the material video into the classes;
an event determination means for determining a key event based on the frequency of appearance of each class included in the raw video;
A video processing device comprising: A learning means for learning an image recognition model that classifies an image into the classes,
The image processing device according to claim 1 , wherein the classification means classifies the raw image into the classes using the image recognition model obtained by learning. a first feature extraction means for extracting a feature of the matching portion;
a second feature extraction means for extracting a feature of each frame image of the raw video;
Equipped with
The video processing device according to claim 1 , wherein the classifying means classifies each frame of the raw video into the class based on a feature amount of each of the matching portions and a feature amount of each frame of the raw video. The video processing device according to claim 2 or 3, wherein the event determination means determines the class with the lowest occurrence frequency as a key event. The video processing device according to claim 2 or 3, wherein the event determination means determines the class that appears least frequently and is included in the matching portion as a key event. The video processing device according to claim 2 or 3, wherein the event determination means determines classes other than the class determined to be the key event as related events. The video processing device according to claim 6, further comprising a learning data generating means for generating learning data to which flags corresponding to the key events and the related events are assigned. Obtain raw footage and digest footage,
Detecting a portion where the contents of the raw video and the digest video match;
Matching parts whose temporal distance is less than a predetermined value are grouped as the same important scene,
In each group obtained by the grouping, the matching portions are classified into different classes, and the material video is classified into the classes;
The video processing method includes determining a key event based on the frequency of appearance of each class contained in the raw video. Obtain raw footage and digest footage,
Detecting a portion where the contents of the raw video and the digest video match;
Matching parts whose temporal distance is less than a predetermined value are grouped as the same important scene,
In each group obtained by the grouping, the matching portions are classified into different classes, and the material video is classified into the classes;
A program that causes a computer to execute a process of determining a key event based on the frequency of appearance of each class contained in the raw video. An image acquisition means for acquiring an input image;
an important scene detection means for detecting an important scene from the input video;
a first partial image extraction means for extracting, from the input image, a temporally continuous area including at least a portion following the important scene as a first partial image;
a related event detection means for detecting a related event from the first partial video by using a related event detection model trained to detect a related event related to the important scene from a partial video including at least a part following the important scene;
A video processing device comprising: The video processing device according to claim 10, further comprising a second partial video extraction means for extracting a portion of the important scene from the input video as a key event partial video based on time information of the important scene, and extracting a portion of the related event from the input video as a related event partial video based on time information of the related event. The video processing device according to claim 11, further comprising a video combining means for combining the key event partial video and the related event partial video.

Images