JP7700561B2 - Learning video data editing program, learning video data editing method, and information processing device - Google Patents

Description

The present invention relates to a video data editing program for learning, etc.

In recent years, learning models such as neural networks (NNs) have been used for image recognition. When using a learning model to perform image recognition, supervised machine learning is performed using training data as a preparatory step. In the following explanation, supervised machine learning will be referred to simply as machine learning.

For example, the training data contains pairs of image data and correct labels that indicate the classification results of this image data. In machine learning, a process is carried out to adjust the parameters in the learning model so that the output when image data is input into the learning model approaches the correct label.

In general, the accuracy of a learning model can be improved by using more training data, so it is preferable to prepare a large amount of training data. In preparing the training data, workers check image data that can be used as training data and set correct labels that correspond to the image data. There is also a conventional technology that deletes image data that is deemed unnecessary based on the features of multiple image data, and narrows down the image data to be used as training data.

JP 2020-135494 A JP 2020-154720 A

In the conventional technology described above, image data is deleted based on feature values, but it is difficult to accurately determine whether image data is unnecessary for learning data using feature values alone. As a result, the final decision on whether image data is unnecessary or not is made by workers, which places a heavy burden on workers.

Therefore, there is a need to enable workers to easily check information about unnecessary data.

In one aspect, the present invention aims to provide a learning video data editing program, a learning video data editing method, and an information processing device that enable workers to easily check information about unnecessary data.

In the first proposal, the computer is made to execute the following process. The computer extracts a first feature amount and a second feature amount from a plurality of image frames in a time series included in the video data. The computer executes a process for each of the plurality of image frames to place points corresponding to the image frames on a two-dimensional plane on the visible first display screen based on the first feature amount and the second feature amount corresponding to the image frames. When the computer receives a designation of an area on the two-dimensional plane, it assigns information to the first image frame corresponding to the points included in the area and to the second image frame not included in the area. The computer assigns information of the first image frame and information of the second image frame to a seek bar that is set on the visible second display screen and that displays a plurality of image frames in a time series.

It is possible to make it easier for workers to check information about unnecessary data.

FIG. 1 is a diagram for explaining the process of an information processing device according to the present embodiment. FIG. 2 is a functional block diagram showing the configuration of an information processing apparatus according to the present embodiment. FIG. 3 is a diagram illustrating an example of a data structure of the video DB. FIG. 4 is a diagram showing an example of the data structure of the image frame management table. FIG. 5 is a flowchart showing a processing procedure of the information processing device according to the present embodiment. FIG. 6 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing apparatus of the embodiment.

Below, examples of the educational video data editing program, educational video data editing method, and information processing device disclosed in the present application are described in detail with reference to the drawings. Note that the present invention is not limited to these examples.

FIG. 1 is a diagram for explaining the processing of an information processing device according to this embodiment. When an instruction to analyze video data is received, the information processing device according to this embodiment analyzes image frames in time series contained in the video data to extract a first feature amount and a second feature amount from each image frame. Each image frame is assigned a frame number in time series.

The information processing device displays screen information 10 having a two-dimensional plane 10a on a screen that can be viewed by a user. For example, the vertical axis of the two-dimensional plane 10a corresponds to a first feature amount, and the horizontal axis corresponds to a second feature amount. The information processing device plots dots corresponding to each image frame on the two-dimensional plane 10a based on the first feature amount and the second feature amount extracted from each image frame. In the example shown in Figure 1, the dots are indicated by circles.

The information processing device sets a threshold value _Th1 on the vertical axis and a threshold value _Th2 on the horizontal axis. Threshold values _Th1 and _Th2 are pre-specified values. The information processing device identifies a region 11 in which the first feature amount is less than the threshold value _Th1 and the second feature amount is less than the threshold value Th2 from among the entire region of the _two- dimensional plane 10a.

The information processing device sets a flag not to be deleted for each image frame that corresponds to the dots included in area 11 among the dots included in the entire area of the two-dimensional plane 10a. In the following description, an image frame for which a flag not to be deleted is set will be referred to as the "first image frame" where appropriate. In the example shown in Figure 1, the image frame corresponding to the black circle is the first image frame.

The information processing device sets a flag to be deleted for each image frame that corresponds to dots not included in area 11 among the dots included in the entire area of the two-dimensional plane 10a. In the following description, an image frame for which a flag to be deleted is set will be referred to as a "second image frame" where appropriate. In the example shown in FIG. 1, the image frame corresponding to the white circle is the second image frame. The information processing device displays the frame number corresponding to the second image frame in area 10b of the screen information 10.

When the user presses the apply button B1 included in the screen information 10, the information processing device displays the screen information 20 on a screen that is visible to the user.

The screen information 20 includes a seek bar 21 and a display area 22. In the seek bar 21, a position on the seek bar 21 can be associated with each frame number of multiple image data in time series. The left end of the seek bar 21 corresponds to the frame number of the first image data of the video data, and the frame number increases as the position on the seek bar 21 moves further to the right. The right end of the seek bar 21 corresponds to the frame number of the last image data of the video data.

The seek bar 21 visually displays the area on the seek bar 21 that corresponds to the frame number of the second image frame. In the following description, the area on the seek bar 21 that corresponds to the frame number of the second image frame is referred to as the "exclusion target area". In the example shown in FIG. 1, exclusion target areas L1 to L10 are set on the seek bar 21. The position of the exclusion target area on the seek bar 21 corresponds to the position of the second image data in the video data. For each exclusion target area, the information processing device displays in area 23 the start frame number and end frame number of the consecutive frame numbers included in the exclusion target area. Note that the area on the seek bar 21 other than the exclusion target areas L1 to L10 corresponds to the frame numbers of the first image frame.

The user can move pointer 21A on seek bar 21. Image data of the frame number corresponding to the position of pointer 21A specified by the user is displayed in display area 22. For example, the user can check the second image frame to be deleted by moving pointer 21A to a position corresponding to the area to be excluded.

As described above, the information processing device according to this embodiment extracts second image data to be deleted based on the features of multiple image data included in the video data, and displays the extracted second image data to be deleted on a seek bar 21 that allows the user to grasp the position within the video data. This allows the user to easily check information related to the second image data to be deleted.

The information processing device also generates screen information 10 in which dots corresponding to each image frame are plotted on a two-dimensional plane 10a based on the first feature amount and the second feature amount extracted from each image frame, and visually displays the screen information 10. This allows the user to easily understand the relationship between the feature amount of the second image frame to be deleted and the threshold value.

Next, an example of the configuration of an information processing device that executes the process shown in FIG. 1 will be described. FIG. 2 is a functional block diagram showing the configuration of an information processing device according to this embodiment. As shown in FIG. 2, this information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a memory unit 140, and a control unit 150.

The communication unit 110 is connected to an external device or the like via a wired or wireless connection, and transmits and receives information to and from the external device or the like. For example, the communication unit 110 is realized by a NIC (Network Interface Card) or the like. The communication unit 110 may be connected to a network (not shown). The communication unit 110 may acquire the video DB 141 from the external device via the network.

The input unit 120 is an input device that inputs various information to the information processing device 100. The input unit 120 corresponds to a keyboard, a mouse, a touch panel, etc. For example, a user may operate the input unit 120 to input an instruction to analyze video data.

The display unit 130 is a display device that displays information output from the control unit 150. The display unit 130 corresponds to a liquid crystal display, an organic EL (Electro Luminescence) display, a touch panel, or the like. For example, the display unit 130 displays the screen information 10 and 20 described in FIG. 1.

The storage unit 140 has a video DB (Data Base) 141 and an image frame management table 142. The storage unit 140 is realized, for example, by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk.

Video DB 141 is a database that holds video data. For example, video data has multiple image frames in a time series, and each image frame is assigned a frame number. Figure 3 is a diagram showing an example of the data structure of a video DB. As shown in Figure 3, this video DB 141 associates frame numbers with image frames.

The image frame management table 142 holds various information related to image frames. FIG. 4 is a diagram showing an example of the data structure of the image frame management table. As shown in FIG. 4, this image frame management table 142 associates a frame number, a first feature amount, a second feature amount, and a flag.

The frame number is a number set for an image frame and is a number that identifies each image frame. The first feature amount and the second feature amount are features extracted by analyzing the image frame. For example, the first feature amount corresponds to the average luminance. The second feature amount corresponds to the variance of the luminance. The first feature amount and the second feature amount may be other image features.

The flag is information indicating whether the corresponding image frame is a second image frame to be deleted. For example, if the flag is "off", it indicates that the corresponding image frame is a first image frame that is not to be deleted. If the flag is "on", it indicates that the corresponding image frame is a second image frame to be deleted. The initial value of the flag is "off".

Returning to the explanation of FIG. 2, the control unit 150 has an extraction unit 151 and a screen information generation unit 152. The control unit 150 is realized, for example, by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The control unit 150 may also be executed by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

When the extraction unit 151 receives an analysis instruction from the input unit 120, it extracts a first feature amount and a second feature amount from each image frame registered in the video DB 141. The extraction unit 151 associates the frame number of the image frame with the first feature amount and the second feature amount, and registers them in the image frame management table 142. The extraction unit 151 sets the flag to the initial value "off."

The screen information generating unit 152 generates the screen information 10, 20 described in FIG. 1, and outputs the screen information 10, 20 to the display unit 130 for display. An example of the processing of the screen information generating unit 152 is described below.

The screen information generating unit 152 identifies the first and second features corresponding to each image frame based on the image frame management table 142, and plots dots corresponding to each image frame on the two-dimensional plane 10a.

The screen information generating unit 152 sets a threshold value _Th1 on the vertical axis of the two-dimensional plane 10a and a threshold value _Th2 on the horizontal axis. The threshold values _Th1 and _Th2 may be preset values or may be received from the input unit 120 operated by the user. The screen information generating unit 152 identifies a region 11 in which the first feature amount is less than the threshold value _Th1 and the second feature amount is less than the threshold value Th2 from among the entire region of the _two -dimensional plane 10a.

The screen information generating unit 152 identifies each image frame (frame number of the image frame) that corresponds to the dots not included in the area 11 among the dots included in the entire area of the two-dimensional plane 10a. The screen information generating unit 152 sets the flag of the frame number that corresponds to the dots not included in the area 11 to "ON" in the image frame management table 142.

The screen information generating unit 152 may set different colors for dots included in the area 11 and dots not included in the area 11 among the dots shown on the two-dimensional plane 10a of the screen information 10. The screen information generating unit 152 sets a list of frame numbers whose flags are "ON" in the image frame management table 142 in the area 10b of the screen information 10. The screen information generating unit 152 generates the screen information 10 by the above processing, and outputs it to the display unit 130 for display.

The user refers to the screen information 10 displayed on the display unit 130. The user may change the thresholds Th ₁ and Th ₂ by operating the input unit 120. When the screen information generating unit 152 receives a change to the thresholds Th ₁ and Th ₂ , it updates the area 11 on the two-dimensional plane 10a, and updates the flags in the image frame management table 142 based on whether or not a dot is included in the area 11. This updates the color, etc. of each dot shown on the two-dimensional plane 10a of the screen information 10.

When the screen information generating unit 152 detects that the apply button B1 of the screen information 10 has been pressed by a user operating the input unit 120, it outputs the screen information 20 to the display unit 130 for display.

The screen information generating unit 152 refers to the image frame management table 142 to identify the frame numbers for which the flag is set to "on" and identifies the start frame number and the end frame number of the exclusion target area. The screen information generating unit 152 sets the exclusion target area on the seek bar 21 based on the start frame number and the end frame number of the exclusion target area. In the example shown in FIG. 1, the screen information generating unit 152 sets the exclusion target areas L1 to L10 on the seek bar 21. The screen information generating unit 152 also sets the start frame number and the end frame number of the consecutive frame numbers included in the exclusion target area in the area 23 of the screen information 20.

The user refers to the screen information 20 displayed on the display unit 130. The user operates the input unit 120 to move the pointer 21A on the seek bar 21 to the desired position. When the screen information generating unit 152 detects the movement of the pointer 21A, it identifies the frame number corresponding to the position of the pointer 21A, retrieves the image frame corresponding to the identified frame number from the video DB 141, and displays it in the display area 22.

When the screen information generating unit 152 detects that the play button has been pressed by a user operating the input unit 120, it moves the pointer 21A on the seek bar 21 from the left end to the right end, and sequentially displays the image frames with the frame numbers corresponding to the position of the pointer 21A in the display area 22.

Next, an example of the processing procedure of the information processing device according to this embodiment will be described. FIG. 5 is a flowchart showing the processing procedure of the information processing device according to this embodiment. As shown in FIG. 5, the extraction unit 151 of the information processing device 100 receives an analysis instruction from the input unit 120 (step S101).

The extraction unit 151 analyzes all image frames of the video data, extracts the first feature amount and the second feature amount, and registers them in the image frame management table 142 (step S102).

The screen information generating unit 152 of the information processing device 100 plots dots corresponding to each image frame on a two-dimensional plane based on the first feature amount and the second feature amount (step S103). The screen information generating unit 152 identifies image frames to be deleted depending on whether or not dots are included in an area identified based on the threshold values _Th1 and _Th2 (step S104).

The screen information generating unit 152 updates the flag in the image frame management table 142 (step S105). The screen information generating unit 152 displays the screen information 10 on the display unit 130 (step S106).

The screen information generating unit 152 receives a press of the Apply button B1 (step S107). The screen information generating unit 152 sets an exclusion target area on the seek bar 21 based on the image frame management table 142 (step S108).

The screen information generating unit 152 causes the display unit 130 to display the screen information 20 (step S109). The screen information generating unit 152 causes the display area 22 to display the image frame with the frame number corresponding to the pointer position on the seek bar 21 (step S110).

Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 extracts second image data to be deleted based on the feature quantities of multiple image data included in the video data, and displays the extracted second image data to be deleted on a seek bar 21 that allows the user to grasp the position within the video data. This allows the user to easily check information related to the second image data to be deleted.

The user can move the pointer 21A on the seek bar 21. The information processing device 100 displays, in the display area 22, image data of the frame number corresponding to the position of the pointer 21A specified by the user. For example, the user can check the second image frame to be deleted by moving the pointer 21A to a position corresponding to the area to be excluded.

The information processing device 100 generates screen information 10 in which dots corresponding to each image frame are plotted on a two-dimensional plane 10a based on the first feature amount and the second feature amount extracted from each image frame, and visually displays the screen information 10. This allows the user to easily understand the relationship between the feature amount of the second image frame to be deleted and the threshold value.

In addition, the information processing device 100 may generate learning data by deleting the second image frame from the video DB 141 and use the data for machine learning of the learning model.

Next, an example of the hardware configuration of a computer that realizes the same functions as the information processing device 100 shown in the above embodiment will be described. FIG. 6 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

As shown in FIG. 6, computer 200 has a CPU 201 that executes various types of arithmetic processing, an input device 202 that accepts data input from a user, and a display 203. Computer 200 also has a communication device 204 that transmits and receives data to and from external devices, etc., via a wired or wireless network, and an interface device 205. Computer 200 also has a RAM 206 that temporarily stores various types of information, and a hard disk device 207. Each of devices 201 to 207 is connected to a bus 208.

The hard disk device 207 has an extraction program 207a and a screen information generation program 207b. The CPU 201 also reads out each of the programs 207a and 207b and expands them in the RAM 206.

The extraction program 207a functions as the extraction process 206a. The screen information generation program 207b functions as the screen information generation process 206b.

The processing of the extraction process 206a corresponds to the processing of the extraction unit 151. The processing of the screen information generation process 206b corresponds to the processing of the screen information generation unit 152.

Note that each of the programs 207a and 207b does not necessarily have to be stored in the hard disk device 207 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a and 207b.

The following notes are further provided with respect to the embodiments including the above examples.

(Supplementary Note 1) A first feature amount and a second feature amount are extracted from a plurality of image frames in a time series included in the video data;
performing a process for each of the plurality of image frames to arrange points corresponding to the image frames on a two-dimensional plane on a first display screen that is visible based on the first feature amount and the second feature amount corresponding to the image frames;
When a designation of an area of the two-dimensional plane is accepted, information is added to a first image frame corresponding to a point included in the area and to a second image frame not included in the area;
A learning video data editing program that causes a computer to execute a process of adding information about the first image frame and information about the second image frame to a seek bar that is set on a visible second display screen and that displays the multiple image frames in chronological order.

(Appendix 2) The learning video data editing program described in Appendix 1, characterized in that the process of adding information adds information to the first image frame indicating that it is not to be deleted, and adds information to the second image frame indicating that it is to be deleted.

(Appendix 3) The learning video data editing program described in Appendix 1 is characterized in that, when displaying the first display screen, the program further executes a process of displaying the color of the dots corresponding to the first image frame and the color of the dots corresponding to the second image frame in different colors.

(Appendix 4) The learning video data editing program described in Appendix 1, further comprising a process for displaying an image frame corresponding to a position selected on the seek bar on the second display screen.

(Supplementary Note 5) Extracting a first feature amount and a second feature amount from a plurality of image frames in a time series included in the video data;
performing a process for each of the plurality of image frames to arrange points corresponding to the image frames on a two-dimensional plane on a first display screen that is visible based on the first feature amount and the second feature amount corresponding to the image frames;
When a designation of an area of the two-dimensional plane is accepted, information is added to a first image frame corresponding to a point included in the area and to a second image frame not included in the area;
A method for editing educational video data, characterized in that a computer executes a process of adding information about the first image frame and information about the second image frame to a seek bar that is set on a visible second display screen and that displays the multiple image frames in chronological order.

(Appendix 6) The method for editing learning video data described in Appendix 5 is characterized in that the process of adding information adds information to the first image frame indicating that it is not to be deleted, and adds information to the second image frame indicating that it is to be deleted.

(Appendix 7) The method for editing learning video data described in Appendix 5, further comprising the step of displaying the first display screen in a color different from the color of the dots corresponding to the first image frame and the color of the dots corresponding to the second image frame.

(Appendix 8) The method for editing learning video data described in Appendix 5, further comprising the step of executing a process for displaying an image frame corresponding to a position selected on the seek bar on the second display screen.

(Supplementary Note 9) An extractor that extracts a first feature amount and a second feature amount from a plurality of image frames in a time series included in video data;
an information processing device comprising: a screen information generating unit that performs a process for each of the multiple image frames to place points corresponding to the image frames on a two-dimensional plane on a visible first display screen based on the first feature amount and the second feature amount corresponding to the image frames; and, when a designation of an area of the two-dimensional plane is accepted, adds information to the first image frame corresponding to points included in the area and a second image frame not included in the area; and a seek bar that is set on a visible second display screen, the seek bar displaying the multiple image frames in chronological order, and adds information about the first image frame and information about the second image frame to the seek bar.

(Supplementary Note 10) The information processing device described in Supplementary Note 9, characterized in that the screen information generating unit assigns information to the first image frame indicating that it is not subject to deletion, and assigns information to the second image frame indicating that it is subject to deletion.

(Appendix 11) The information processing device described in Appendix 9 is characterized in that the screen information generating unit further executes a process of displaying the color of the dots corresponding to the first image frame and the color of the dots corresponding to the second image frame in different colors when displaying the first display screen.

(Appendix 12) The information processing device described in Appendix 9, characterized in that the screen information generating unit further executes a process to display an image frame corresponding to a position selected on the seek bar on the second display screen.

REFERENCE SIGNS LIST 100 Information processing device 110 Communication unit 120 Input unit 130 Display unit 140 Storage unit 141 Video DB
142 Image frame management table 150 Control unit 151 Extraction unit 152 Screen information generation unit

Claims (6)

extracting a first feature amount and a second feature amount from a plurality of image frames in a time series included in the video data;
performing a process for each of the plurality of image frames to arrange points corresponding to the image frames on a two-dimensional plane on a first display screen that is visible based on the first feature amount and the second feature amount corresponding to the image frames;
setting a first threshold value on a first axis of the two-dimensional plane, setting a second threshold value on a second axis of the two-dimensional plane, and visually displaying an area of the two-dimensional plane that is less than the first threshold value and less than the second threshold value;
When a designation of an area of the two-dimensional plane is accepted, information is added to a first image frame corresponding to a point included in the area and to a second image frame not included in the area;
A learning video data editing program that causes a computer to execute a process of adding information about the first image frame and information about the second image frame to a seek bar that is set on a visible second display screen and that displays the multiple image frames in chronological order. The video data editing program for learning according to claim 1, characterized in that the process of adding information adds information to the first image frame indicating that it is not to be deleted, and adds information to the second image frame indicating that it is to be deleted. The video data editing program for learning according to claim 1, further comprising a process for displaying the dots corresponding to the first image frame and the dots corresponding to the second image frame in different colors when the first display screen is displayed. The learning video data editing program according to claim 1, further comprising a process for displaying an image frame corresponding to a position selected on the seek bar on the second display screen. extracting a first feature amount and a second feature amount from a plurality of image frames in a time series included in the video data;
performing a process for each of the plurality of image frames to arrange points corresponding to the image frames on a two-dimensional plane on a first display screen that is visible based on the first feature amount and the second feature amount corresponding to the image frames;
setting a first threshold value on a first axis of the two-dimensional plane, setting a second threshold value on a second axis of the two-dimensional plane, and visually displaying an area of the two-dimensional plane that is less than the first threshold value and less than the second threshold value;
When a designation of an area of the two-dimensional plane is accepted, information is added to a first image frame corresponding to a point included in the area and to a second image frame not included in the area;
A method for editing educational video data, characterized in that a computer executes a process of adding information about the first image frame and information about the second image frame to a seek bar that is set on a visible second display screen and that displays the multiple image frames in chronological order. an extracting unit that extracts a first feature amount and a second feature amount from a plurality of time-series image frames included in the video data;
an information processing device which performs a process for each of the multiple image frames to place points corresponding to the image frames on a two-dimensional plane on a visible first display screen based on the first feature amount and the second feature amount corresponding to the image frames, sets a first threshold value on a first axis of the two-dimensional plane, sets a second threshold value on a second axis of the two-dimensional plane, and visually displays an area of the two-dimensional plane that is less than the first threshold value and less than the second threshold value, and upon receiving specification of an area of the two-dimensional plane, adds information to a first image frame corresponding to points included in the area and a second image frame that is not included in the area, and a screen information generation unit which adds information about the first image frame and information about the second image frame to the seek bar which displays the multiple image frames in chronological order.

Images