JP7662033B2 - Image processing device, image processing method, and program - Google Patents

Description

This disclosure relates to processing of video data.

A technology has been proposed for generating a video digest from video. Patent Document 1 discloses a highlight extraction device that creates a learning data file from training video prepared in advance and video of important scenes specified by the user, and detects important scenes from a target video based on the learning data file.

JP 2008-022103 A

Images contained in digest videos of sports and the like include images that are characteristic of digest videos and other general images. In conventional digest video generation devices, learning was performed by generating learning data based on whether or not an image was used in the digest video. In other words, learning data was generated and learning was performed by treating all parts of the raw video that were used in the digest video as correct answers. In this case, general images were included in both the correct and incorrect classes of the learning data, which was one of the factors that reduced the accuracy of detecting important events from raw video.

One objective of the present disclosure is to enable the generation of a digest video device that can detect important events from raw video with high accuracy.

According to one aspect of the present disclosure, there is provided a video processing device comprising:
an information acquisition means for acquiring a raw video and information on a section of the raw video used for a digest video;
a video search means for searching the raw video for an element similar to an element included in the section used in the digest video;
an importance calculation means for calculating the importance of each element based on the frequency of appearance of each element in the raw video and in a section used in the digest video;
Equipped with
The importance level has a higher value the higher the frequency of appearance of the element in the section used in the digest video is relative to the frequency of appearance of other elements, and the lower the frequency of appearance of the element in the raw video is, the higher the importance level has .

In another aspect of the present disclosure, a video processing method includes:
Acquire a raw video and information on a section of the raw video used in the digest video;
Searching for an element similar to an element included in the section used in the digest video in the raw video;
Calculating the importance of each element based on the frequency of appearance of each element in the raw video and in the section used in the digest video ;
The importance level has a higher value the higher the frequency of appearance of the element in the section used in the digest video is relative to the frequency of appearance of other elements, and the lower the frequency of appearance of the element in the raw video is, the higher the importance level has .

In yet another aspect of the invention, a program comprising:
Acquire a raw video and information on a section of the raw video used in the digest video;
Searching for an element similar to an element included in the section used in the digest video in the raw video;
Calculating the importance of each element based on the frequency of appearance of each element in the raw video and in the section used in the digest video ;
The computer is caused to execute processing such that the importance becomes a higher value the higher the frequency of appearance of the element in the section used in the digest video is relative to the frequency of appearance of other elements, and the higher the importance becomes the lower the frequency of appearance of the element in the raw video .

This disclosure makes it possible to generate a digest video device that can detect important events from raw video with high accuracy.

1 shows a schematic configuration of a digest video generating device. An example of a digest video is shown below. An example of labeling raw footage when generating training data is shown below. 1 shows an overall configuration of a training data generating device according to the present embodiment. FIG. 2 is a block diagram showing a hardware configuration of the training data generating device. FIG. 2 is a block diagram showing a functional configuration of the training data generating device. An example of a calculation formula for importance is shown below. 13 is a flowchart of a learning data generation process performed by the learning data generation device. FIG. 11 is a diagram illustrating a specific example of a learning data generation process. FIG. 11 is a diagram illustrating a specific example of a learning data generation process. FIG. 11 is a diagram illustrating a specific example of a learning data generation process. An example of generated training data is shown below. 1 shows a configuration for learning a digest video generation device. FIG. 13 is a block diagram showing a configuration of a modified example of the digest video generation device. FIG. 11 is a block diagram showing a functional configuration of a video processing device according to a second embodiment. 10 is a flowchart of a process performed by a video processing device according to a second embodiment.

Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings.
First Embodiment
[Digest video generating device]
First, as a premise, the digest video generating device will be described. FIG. 1 shows a schematic configuration of the digest video generating device. The digest video generating device 200 is connected to a material video database (hereinafter, "database" will also be referred to as "DB") 2. The material video DB 2 stores various material videos, that is, moving images. The material videos may be, for example, videos of television programs broadcast by a broadcasting station, or videos distributed via the Internet or the like. The material videos may or may not include audio.

The digest video generation device 200 generates and outputs a digest video using a portion of the material video stored in the material video DB 2. The digest video is a video in which scenes in which some event occurs in the material video are linked in chronological order. The digest video generation device 200 detects event scenes from the material video using a digest video generation model trained by machine learning, and links multiple event scenes in chronological order to generate a digest video. The digest video generation model is a model that detects event scenes from the material video, and can be, for example, a deep learning model using a neural network.

Figure 2 shows an example of a digest video. In the example of Figure 2, the digest video generation device 200 extracts event scenes A to D included in the material video and connects them in chronological order to generate a digest video. Note that the event scenes extracted from the material video may be used repeatedly in the digest video depending on their contents. An event scene is composed of one or more frame images corresponding to a scene in which some event occurs in the material video. An event scene is defined by its start point and end point. Note that an event scene may be defined using the length of the event scene instead of the end point.

[Learning data generation device]
(Basic Principles)
Next, a basic principle of the training data generation device according to this embodiment will be described. The training data generation device is a device that generates training data used for training the digest video generation device 200 (digest video generation model) described above.

In sports digest videos, important events that are detected as event scenes are often composed of multiple, implicitly determined short videos. For example, in a soccer goal scene, a video of the player's shot from the moment he scores the goal is often shown taken by a camera that takes an overhead view of the entire field (hereafter referred to as an "overhead camera"), followed by a video of the player's goal celebration, or a video of the goal scene taken by a camera at a different angle.

The images contained in the digest video include images that are characteristic of the digest video and other universal images. Universal images are images that are not only contained in the digest video, but also in scenes other than the digest video.

Normally, in training a digest video generation device, training data is generated based on whether or not a certain part was used in the digest video, and a digest video generation model is trained using this training data. In other words, all parts of the raw video that were used in the digest video were treated as correct answers, and training data was generated to train the digest video generation model. In this case, common images were included in both the correct and incorrect classes of the training data, which was one of the factors that reduced accuracy when detecting important event scenes, such as soccer goal scenes, from raw video of the entire game.

In this embodiment, the learning data generation device assumes that footage that appears frequently in places used in the digest video but appears relatively infrequently in places not used in the digest video is footage that is characteristic of an event scene, and determines a "characteristic element" that defines this characteristic footage. A characteristic element is an element that is included in a section used in the digest video, i.e., an event scene. Here, a characteristic element may be, for example, a single frame image of a video, or multiple consecutive frame images. Furthermore, the characteristic element may be one type, or two or more types.

Figure 3 shows an example of labeling of material video when generating training data. When learning a digest video generation model, a correct answer label is added to the material video to generate training data. In Figure 3, the event scene 7 shown with diagonal lines is a scene included in the existing digest video among the material video. That is, the event scene 7 is a part where the material video matches the existing digest video used in the news, etc. In the case of the soccer material video shown in Figure 3, the event scenes detected are a goal scene captured by an overhead camera, a goal scene captured by a camera behind the goal, a goal performance scene of a player, and the like. Specifically, the event scene 7a in Figure 3 is a goal scene captured by an overhead camera, the event scene 7b is a goal scene captured by a camera behind the goal, and the event scene 7c is a goal performance scene of a player. In addition, the scenes 8 before and after the event scenes 7a to 7c are scenes not included in the digest video.

When generating training data, correct labels are assigned to event scenes detected from raw footage. As shown in Figure 3, the normal labeling method assigns the label "1" to frame images of event scenes and the label "0" to frame images of other scenes.

In contrast, in the labeling method 1 of this embodiment, a value between "0" and "1" is assigned to each event scene according to the degree to which the event scene is distinctive. This "degree of distinctiveness" corresponds to the "importance" described below. In the example of FIG. 3, the image captured by the camera behind the goal is considered to be a particularly distinctive soccer goal scene, so the label "1" is assigned to event scene 7b, which is a goal scene captured by the camera behind the goal. The label "0.5" is assigned to event scene 7a, which is a goal scene captured by an overhead camera, and event scene 7c, which is a goal performance scene of a player. The label "0" is assigned to other scenes.

In addition, in labeling method 2 of this embodiment, the event scene 7b, which is a goal scene captured by a camera behind the goal, is assigned the label "1", and other scenes are assigned the label "0".

In this way, in this embodiment, the learning data is labeled not simply based on whether the scene is included in the digest video, but based on the degree to which it is distinctive in the digest video. This makes it possible to reduce the influence of generic images in the sections used in the digest video. Therefore, by learning using this learning data, it is possible to obtain a highly accurate digest video generation device.

(Overall composition)
Fig. 4 shows the overall configuration of the training data generation device of this embodiment. The training data generation device 100 generates training data from raw video and digest video. The raw video is video that is the source of the training data. The digest video is existing digest video used in news programs, etc. On the other hand, the training data is data in which a correct answer label is added to the raw video, as shown in Fig. 3.

(Hardware configuration)
5 is a block diagram showing a hardware configuration of the training data generation device 100. As shown in the figure, the training data generation device 100 includes an interface (IF) 11, a processor 12, a memory 13, a recording medium 14, and a database (DB) 15.

IF11 inputs and outputs data between external devices. Specifically, interface 11 receives raw video that is the source of the learning data and existing digest video, and outputs the generated learning data.

The processor 12 is a computer such as a CPU (Central Processing Unit) and executes a pre-prepared program to control the entire learning data generation device 100. Specifically, the processor 12 executes the learning data generation process described below.

The memory 13 is composed of a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 13 is also used as a working memory while the processor 12 is executing various processes.

The recording medium 14 is a non-volatile, non-temporary recording medium such as a disk-shaped recording medium or a semiconductor memory, and is configured to be detachable from the learning data generation device 100. The recording medium 14 records various programs executed by the processor 12. When the learning data generation device 100 executes various processes, the programs recorded on the recording medium 14 are loaded into the memory 13 and executed by the processor 12.

The database 15 temporarily stores raw footage, digest footage, and the like input through the IF 11. The learning data generation device 100 may also include an input unit such as a keyboard or mouse for an operator to enter instructions or input data, and a display unit such as a liquid crystal display.

(Functional configuration)
6 is a block diagram showing a functional configuration of the training data generation device 100. The training data generation device 100 includes a video acquisition unit 21, a video search unit 22, an importance calculation unit 23, and a training data generation unit 24.

The video acquisition unit 21 acquires the raw video and the digest video from the outside. The video acquisition unit 21 compares the raw video with the digest video, and generates information on the section in the raw video that matches the digest video, i.e., the section used in the digest video (hereinafter also referred to as the "digest video use section"). The video acquisition unit 21 then outputs the raw video and information on the digest video use section in the raw video to the video search unit 22. The video acquisition unit 21 also outputs the raw video to the learning data generation unit 24.

In addition, if the information on the section in which the digest video is used can be obtained from outside, the video acquisition unit 21 may acquire the raw video and the information on the section in which the digest video is used from outside and output them to the video search unit 22.

The video search unit 22 searches the raw video for elements similar to elements included in the digest video use section. Here, the "element" may be one frame image or multiple consecutive frame images. For example, the video search unit 22 clusters the frame images included in the digest video use section to extract multiple elements. In this case, each element may be a frame image included in each cluster obtained by clustering one frame image or multiple frame images. Furthermore, each element may be the central feature of each cluster obtained by clustering one frame image or multiple frame images.

Alternatively, the video search unit 22 may detect shot change positions from the digest video and extract footage from individual cameras as elements.

Then, the video search unit 22 searches the material video for elements similar to each element, and outputs the elements to the importance calculation unit 23. Note that the video search unit 22 may search the entire material video from start to finish to detect each element, or may search sections of the material video other than the section in which the digest video is used to detect each element.

Instead of searching for each element from the raw video, the video search unit 22 may classify the raw video into each element. The search method and classification method for each element may be a method that requires learning, or a method that does not require learning (e.g., unsupervised learning or a method that uses an existing model).

The importance calculation unit 23 calculates the importance of each element based on the frequency of occurrence of each element in the raw video and in the digest video usage section. Here, "importance" is a value reflecting whether each element is an element that characterizes a scene included in the digest video, and corresponds to the "degree to which each event scene is distinctive" mentioned above. The larger the importance value, the more distinctive the scene is. The importance calculation unit 23 then calculates the importance of each element and outputs it to the learning data generation unit 24.

Figure 7 shows an example of a formula for calculating importance. Importance s(x) can be defined, for example, by equation (1) or equation (2) shown in Figure 7 (A) using functions a(x) and b(x). By using equation (1) or equation (2), importance s(x) becomes a value that increases when both functions a(x) and b(x) become large.

Figure 7 (B) shows examples of functions a(x) and b(x). Function a(x) is the ratio (called the "first ratio") of the number of elements x in the section used for the digest video to the total number of elements in the section used for the digest video. The function a(x) takes a larger value the greater the frequency of occurrence of element x in the section used for the digest video is relative to the frequency of occurrence of other elements. Function b(x) is the ratio (called the "second ratio") of the number of elements x in the section used for the digest video to the number of elements x in the raw video. Function b(x) takes a larger value the greater the frequency of occurrence of element x in the section used for the digest video is relative to the number of elements x in the raw video.

Figure 7 (C) shows another example of a function. Function a(x) is the same as in Figure 7 (B). Meanwhile, function b'(x) is used in place of function b(x). Function b'(x) is a value obtained by subtracting from 1 the ratio of the number of elements x in the source video to the total number of elements in the source video (called the "third ratio"). The lower the frequency of occurrence of element x in the source video, the larger the value of function b'(x).

The learning data generating unit 24 generates and outputs learning data by labeling the material video based on the importance of each element. Specifically, the learning data generating unit 24 determines the value of the correct label for each element similar to an element included in the digest video use section in the material video as shown in FIG. 3 based on the importance calculated by the importance calculating unit 23, and performs labeling. In one example, the learning data generating unit 24 assigns a label to the element with the highest importance among the multiple elements as "1" and a value obtained by normalizing the importance of the other elements as a label. In another example, the learning data generating unit 24 assigns a label to a predetermined number of elements in descending order of importance as "1", and assigns a label to the other elements as "0". In this way, as shown in FIG. 3, learning data is generated in which an appropriate label is assigned to each element similar to an element included in the digest video use section.

In the above example, the learning data generation unit 24 generates learning data by labeling the raw video based on the importance of each element calculated by the importance calculation unit 23. Alternatively, the importance calculation unit 23 may output the calculated importance of each element directly to an external device. In this case, the external device may generate learning data in the same way as the learning data generation unit 24, or may use the importance of each element for other purposes.

In the above configuration, the video acquisition unit 21 is an example of an information acquisition means, the video search unit 22 is an example of a video search means, the importance calculation unit 23 is an example of an importance calculation means and an output means, and the learning data generation unit 24 is an example of a learning data generation means.

(Learning data generation process)
8 is a flowchart of a training data generation process by the training data generation device 100. This process is realized by the processor 12 shown in FIG. 5 executing a program prepared in advance and operating as each element shown in FIG.

First, the video acquisition unit 21 acquires the raw video and information on the section in the raw video where the digest video is used (step S21). In one example, the video acquisition unit 21 acquires the raw video and the digest video, detects a section where the raw video and the digest video match, and acquires information on the section where the digest video is used. In another example, the video acquisition unit 21 acquires the raw video and information on the section where the digest video is used from outside.

Next, the video search unit 22 determines whether all elements included in the digest video using section have been processed (step S22). If not all elements have been processed (step S22: No), the video search unit 22 determines whether the end of the material video has been processed (step S23). If the end of the material video has not been processed (step S23: No), the video search unit 22 searches for elements in the material video that are similar to elements included in the digest video (step S24). Specifically, the video search unit 22 first detects multiple elements included in the digest video using section by clustering the videos in the digest video using section, and searches the material video for an element similar to one of the elements.

When the video search unit 22 has performed this search up to the end of the material video (step S23: Yes), it executes steps S23 to S24 for other elements included in the digest video using section. When all elements included in the digest video using section have been processed in this manner (step S22: Yes), the importance calculation unit 23 calculates the importance of each element based on the frequency of appearance of each element in the material video and the digest video using section (step S25). Specifically, the importance calculation unit 23 calculates the importance of each element using a formula such as that shown in FIG. 7.

Next, the learning data generation unit 24 assigns a correct answer label to the section corresponding to each element of the raw video based on the importance of each element, and generates learning data (step S26). Then, the learning data generation process ends.

(Specific example)
Next, a specific example of the learning data generation process will be described. Assume that material videos A and B and a digest video are given as shown in Fig. 9(A). The video acquisition unit 21 matches material videos A and B with the material videos, and detects a matching section M in the material videos as shown in Fig. 9(B). The matching section corresponds to a section in which the digest video is used.

Next, as shown in Figure 10 (A), the video search unit 22 clusters the videos of each matching section M to obtain multiple clusters. In the example of Figure 10 (A), videos P1 to P9 of the matching section are obtained from material videos A and B, and the video search unit 22 clusters the videos P1 to P9 to obtain five clusters as shown in Figure 10 (B). Note that each cluster corresponds to an individual element. In Figure 10 (B), cluster 1 includes videos P1 and P3, cluster 2 includes videos P4 and P7, cluster 3 includes video P6, cluster 4 includes video P8, and cluster 5 includes videos P2, P5, and P9.

Next, the video search unit 22 learns a classification model with each cluster shown in Figure 10 (B) as one class. This classification model classifies the input material video into classes 1 to 5.

Next, as shown in FIG. 11, the video search unit 22 inputs the material video A into the trained classification model 31 and classifies the material video A into five classes. Note that the classification results of the classification model 31 may include images that do not resemble any of the classes. Note that the material video B is also classified using the classification model 31 in the same manner.

Next, the importance calculation unit 23 calculates the frequency of occurrence of each element in the material video and the digest video use section based on the classification result, and calculates the importance of each element using the calculation formula shown in Figure 7. The learning data generation unit 24 then labels the section of each element in the material video using the importance of each element, and generates learning data. As a result, learning data is generated in which correct labels are assigned to the sections of each element in material video A, as shown in Figure 12, for example.

[Learning of digest video generation device]
Next, a method for training the digest video generation device (digest video generation model) using the training data generated by the training data generation device 100 will be described. Fig. 13 shows a configuration for training the digest video generation device 200. The digest video generation device 200 is connected to a training unit 41. The training data generated by the training data generation device 100 is input to the digest video generation device 200 and the training unit 41. Specifically, the training data includes a material video and a correct answer label assigned to a section of each element of the material video, as exemplified in Fig. 12.

During learning, the raw video of the learning data is input to the digest video generation device 200, and the correct answer label is input to the learning unit 41. The digest video generation device 200 detects event scenes from the raw video using the digest video generation model and outputs them to the learning unit 41. The learning unit 41 optimizes the digest video generation model based on the event scenes detected by the digest video generation model and the correct answer labels. In this way, the digest video generation device 200 learns using the learning data generated by the learning data generation device 100.

The learning data generated by the learning data generation device 100 of this embodiment is labeled not simply based on whether it is used in the digest video, but based on the importance of whether it is an element that characterizes the scene used in the digest video. Therefore, the digest video generation device 200 that learns using the learning data reduces the influence of general images included in the sections used in the digest video, making it possible to generate appropriate digest videos.

[Modification of digest video generating device]
As described above, instead of training the digest video generation device 200 using the training data generated by the training data generation device 100, the digest video generation device may be configured using the classification model 31 generated by the video search unit 22 of the training data generation device 100.

Figure 14 is a block diagram showing the configuration of a digest video generation device 200x relating to a modified example. The digest video generation device 200x comprises a classification model 31, an importance calculation unit 32, an event scene selection unit 33, and a video combination unit 34. Material video X is input to the classification model 31. The material video X is video that is the source of the digest video creation. The classification model 31 classifies the material video X into each element included in the digest video usage section, and outputs the result to the importance calculation unit 32. The importance calculation unit 32 calculates the importance of each element included in the material video X, and outputs the result to the event scene selection unit 33.

The event scene selection unit 33 selects event scenes that make up the digest video based on the importance of each element, and outputs the selected scenes to the video combination unit 34. For example, the event scene selection unit 33 selects, from the raw video X, a section that corresponds to an element whose importance is equal to or greater than a predetermined value as an event scene. The video combination unit 34 combines the scenes selected by the event scene selection unit 33 in chronological order to generate the digest video.

Second Embodiment
Next, a second embodiment of the present disclosure will be described. Fig. 15 is a block diagram showing a functional configuration of a video processing device according to the second embodiment. As shown in the figure, a video processing device 70 includes an information acquisition unit 71, a video search unit 72, and an importance calculation unit 73.

Figure 16 is a flowchart of processing by the video processing device 70. The information acquisition means 71 acquires the raw video and information on the section of the raw video used in the digest video (step S71). The video search means 72 searches the raw video for elements similar to elements contained in the section used in the digest video (step S72). The importance calculation means 73 calculates the importance of each element based on the frequency of appearance of each element in the raw video and in the section used in the digest video (step S73).

According to the second embodiment of the video processing device 70, it is possible to calculate the importance of elements in the raw video that are similar to elements contained in the section used in the digest video.

Some or all of the above embodiments may also be described as, but are not limited to, the following notes:

(Appendix 1)
an information acquisition means for acquiring a raw video and information on a section of the raw video used for a digest video;
a video search means for searching the raw video for an element similar to an element included in the section used in the digest video;
an importance calculation means for calculating the importance of each element based on the frequency of appearance of each element in the raw video and in a section used in the digest video;
A video processing device comprising:

(Appendix 2)
2. The video processing device according to claim 1, further comprising an output unit for outputting the importance of each element.

(Appendix 3)
2. The video processing device according to claim 1, further comprising: a learning data generating means for labeling sections of each element in the raw video based on the importance of each element, and generating learning data.

(Appendix 4)
The image processing device according to any one of claims 1 to 3, wherein the element is one frame image or a plurality of frame images.

(Appendix 5)
The image processing device according to any one of claims 1 to 3, wherein the elements are frame images included in each cluster obtained by clustering one or more frame images.

(Appendix 6)
The image processing device according to any one of claims 1 to 3, wherein the element is a central feature in each cluster obtained by clustering one or more frame images.

(Appendix 7)
The video processing device according to any one of claims 1 to 6, wherein the importance calculation means calculates the importance based on a first ratio which is a ratio of the number of elements in the section used for the digest video to the number of all elements in the section used for the digest video, and a second ratio which is a ratio of the number of elements in the section used for the digest video to the number of elements in the raw video.

(Appendix 8)
The video processing device according to any one of claims 1 to 6, wherein the importance calculation means calculates the importance based on a value obtained by subtracting from 1 a first ratio, which is a ratio of the number of one element in a section used in the digest video to the number of all elements in the section used in the digest video, and a third ratio, which is a ratio of the number of the one element in the raw video to the number of all elements in the raw video.

(Appendix 9)
Acquire a raw video and information on a section of the raw video used in the digest video;
Searching for an element similar to an element included in the section used in the digest video in the raw video;
The video processing method calculates the importance of each element based on the frequency of appearance of each element in the raw video and in a section used in the digest video.

(Appendix 10)
Acquire a raw video and information on a section of the raw video used in the digest video;
Searching for an element similar to an element included in the section used in the digest video in the raw video;
A recording medium having recorded thereon a program for causing a computer to execute a process of calculating the importance of each element based on the frequency of appearance of each element in the raw video and in the section used in the digest video.

Although the present disclosure has been described above with reference to embodiments and examples, the present disclosure is not limited to the above embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

REFERENCE SIGNS LIST 12 Processor 21 Video acquisition unit 22 Video search unit 23 Importance calculation unit 24 Learning data generation unit 31 Classification model 32 Importance calculation unit 33 Event scene selection unit 34 Video combination unit 41 Learning unit 100 Learning data generation device 200 Digest video generation device

Claims (10)

an information acquisition means for acquiring a raw video and information on a section of the raw video used for a digest video;
a video search means for searching the raw video for an element similar to an element included in the section used in the digest video;
an importance calculation means for calculating the importance of each element based on the frequency of appearance of each element in the raw video and in a section used in the digest video;
Equipped with
The importance level has a higher value when the frequency of appearance of the element in the section used in the digest video is higher than the frequency of appearance of other elements, and the importance level has a higher value when the frequency of appearance of the element in the raw video is lower . The video processing device according to claim 1, further comprising an output means for outputting the importance of each element. The video processing device according to claim 1, further comprising a learning data generating means for labeling sections of each element in the raw video based on the importance of each element, and generating learning data. The image processing device according to any one of claims 1 to 3, wherein the element is one frame image or multiple frame images. The image processing device according to any one of claims 1 to 3, wherein the elements are frame images included in each cluster obtained by clustering one or more frame images. The image processing device according to any one of claims 1 to 3, wherein the element is a central feature in each cluster obtained by clustering one or more frame images. The video processing device according to any one of claims 1 to 6, wherein the importance calculation means calculates the importance based on a first ratio, which is a ratio of the number of elements in the section used in the digest video to the number of all elements in the section used in the digest video, and a second ratio, which is a ratio of the number of elements in the section used in the digest video to the number of elements in the raw video. The video processing device according to any one of claims 1 to 6, wherein the importance calculation means calculates the importance based on a value obtained by subtracting from 1 a first ratio, which is the ratio of the number of elements in the section used in the digest video to the number of all elements in the section used in the digest video, and a third ratio, which is the ratio of the number of elements in the raw video to the number of all elements in the raw video. Acquire a raw video and information on a section of the raw video used in the digest video;
Searching for an element similar to an element included in the section used in the digest video in the raw video;
Calculating the importance of each element based on the frequency of appearance of each element in the raw video and in the section used in the digest video ;
a video processing method in which the importance level has a higher value when the frequency of appearance of the element in the section used in the digest video is higher than the frequency of appearance of other elements, and the importance level has a higher value when the frequency of appearance of the element in the raw video is lower . Acquire a raw video and information on a section of the raw video used in the digest video;
Searching for an element similar to an element included in the section used in the digest video in the raw video;
Calculating the importance of each element based on the frequency of appearance of each element in the raw video and in the section used in the digest video ;
The program causes a computer to execute a process in which the importance level is increased when the frequency of appearance of the element in the section used in the digest video is higher than the frequency of appearance of other elements, and is increased when the frequency of appearance of the element in the raw video is lower .

Images