JP7528706B2 - Learning model generation device, learning model, behavior recognition device, learning data generation device, and learning data generation method - Google Patents

Description

The present disclosure relates to a learning model generation device, a learning model, a behavior recognition device, a learning data generation device, and a learning data generation method.

Patent Document 1 discloses a behavior recognition device that uses deep learning techniques such as Convolutional Neural Network (CNN) to recognize human behavior from video.

JP 2019-191981 A

Machine learning such as deep learning requires a large amount of training data for machine learning. In order to obtain a large amount of training data, a technique called data augmentation is used to increase the amount of training data. Data augmentation is also called data padding. Examples of data augmentation processing for still image data include rotating the image, flipping it upside down, and flipping it left-right.

However, even if conventional data augmentation processing, which is applied to still image data, is applied to video data, sufficient learning effects may not be expected. For this reason, a new data augmentation method suitable for video data is desired.

One aspect of the present disclosure is a learning model generation device. The disclosed learning model generation device includes a processing device configured to execute a learning data generation process including a second video data generation process for generating second video data different from first video data in which a plurality of image data are arranged in chronological order, and a machine learning process for generating a learning model for behavior recognition using the first video data and the second video data as learning data, and the second video data is configured such that the plurality of image data included in the first video data are arranged in reverse chronological order.

Another aspect of the present disclosure is a learning model. The disclosed learning model is a learning model for behavior recognition that is machine-learned using learning data, and the learning data includes one or more first video data in which multiple image data are arranged in chronological order, and second video data in which the multiple image data included in the first video data are arranged in reverse chronological order.

Another aspect of the present disclosure is a behavior recognition device. The disclosed behavior recognition device is configured to recognize the behavior of an object in video data as test data using the learning model.

Another aspect of the present disclosure is a training data generation device. The disclosed training data generation device includes a processing device configured to execute a training data generation process including a second video data generation process for generating second video data different from a first video data in which a plurality of image data are arranged in chronological order, and the second video data is configured such that the plurality of image data included in the first video data are arranged in reverse chronological order.

Another aspect of the present disclosure is a training data generation method. The disclosed training data generation method includes generating second video data different from one or more first video data in which a plurality of image data are arranged in chronological order, and the second video data is configured such that the plurality of image data included in the first video data are arranged in reverse chronological order.

According to this disclosure, data expansion suitable for video is performed.

FIG. 1 is a schematic configuration diagram of a system including a learning model generation device and an action recognition device. FIG. 2 is a configuration diagram of the behavior recognition device. FIG. 3 is a flowchart of the behavior recognition process. FIG. 4 is an explanatory diagram of a method for generating test data. FIG. 5 is a diagram showing an example of a screen displayed on the display device. FIG. 6 is a configuration diagram of a learning model generating device. FIG. 7 is a flowchart of the learning data generation process. FIG. 8 is an explanatory diagram of a method for creating a short video. FIG. 9 is an explanatory diagram of a method for generating first video data and second video data from a short video. FIG. 10 is a table showing the experimental results. FIG. 11 is a diagram showing the results under conditions 1, 2, and 3 in the first experiment. FIG. 12 shows the results of conditions 4, 5, and 6 in the first experiment. FIG. 13 is a diagram showing the results under conditions 1, 2, and 3 in the second experiment. FIG. 14 shows the results of conditions 4, 5, and 6 in the second experiment.

[Description of the embodiment of the present disclosure]

(1) A learning model generation device according to an embodiment includes a processing device configured to execute a learning data generation process including a second video data generation process for generating second video data different from first video data in which multiple image data are arranged in chronological order, and a machine learning process for generating a learning model for behavior recognition using the first video data and the second video data as learning data. The second video data is configured such that the multiple image data included in the first video data are arranged in reverse order to the chronological order. In this case, second video data arranged in reverse order to the first video data is obtained. In other words, the video data is data-extended.

(2) The learning data generation process further includes a first video data generation process for generating a plurality of pieces of first video data from original video data, the first video data having a first time length shorter than the time length of the original video data, and each of the plurality of pieces of first video data generated by the first video data generation process preferably overlaps in time with other pieces of first video data included in the plurality of pieces of first video data. In this case, more first video data can be generated from the original video data, and data expansion is performed.

(3) It is preferable that the overlap time during which each of the plurality of first video data overlaps with other first video data included in the plurality of first video data is 50% or more and 80% or less of the first time length. In this case, the overlap time length is appropriate.

(4) The learning model according to the embodiment is a learning model for behavior recognition that is machine-learned using learning data, and it is preferable that the learning data includes one or more first video data in which multiple image data are arranged in chronological order, and second video data in which the multiple image data included in the first video data are arranged in reverse chronological order. In this case, the data-expanded learning data results in a more appropriate learning model.

(5) The behavior recognition device according to the embodiment is configured to recognize the behavior of an object in video data as test data using the learning model described in (4). In this case, the accuracy of behavior recognition is improved.

(6) A learning data generation device according to an embodiment includes a processing device configured to execute a learning data generation process including a second video data generation process for generating second video data different from first video data in which a plurality of image data are arranged in chronological order, and the second video data is configured such that the plurality of image data included in the first video data are arranged in a reverse order to the chronological order. In this case, second video data arranged in a reverse order to the first video data is obtained. In other words, the video data is data-extended.

(7) A learning data generation method according to an embodiment includes generating second video data different from one or more first video data in which multiple image data are arranged in chronological order, and the second video data is configured such that the multiple image data included in the first video data are arranged in a reverse order to the chronological order. In this case, second video data arranged in a reverse order to the first video data is obtained. In other words, the video data is data-extended.

The computer program according to the embodiment causes a processor to execute at least one of a learning data generation process, a machine learning process, and a behavior recognition process. The computer program is stored in a computer-readable, non-transitory storage medium.

[Details of the embodiment of the present disclosure]

Figure 1 shows a system 1 for behavior recognition. Behavior recognition is the recognition of the behavior of an object shown in video data. The object shown in the video data is, for example, a person or an object. Behavior recognition is also the classification of video data.

Behavior recognition is performed, for example, in a factory or the like, using video data of the behavior of workers captured by an imaging device 30. In this case, behavior recognition classifies the behavior performed by the worker. This makes it possible to measure the time required for the worker to perform a specific task. Furthermore, by classifying the behavior of the worker, it is possible to understand whether the worker is performing the task at the appropriate time. Furthermore, by classifying the behavior of each of multiple workers, it is possible to understand whether there are differences in the behavior of each worker.

The system 1 shown in FIG. 1 includes a behavior recognition device 20 and one or more imaging devices 30 connected to the behavior recognition device 20. The imaging device 30 is installed so as to capture an image of an object such as a worker. The imaging device 30 outputs video data. The video data is a video composed of a plurality of image data (image frames) arranged in a time series. Hereinafter, the video data output from the imaging device 30 is referred to as "original video data." The behavior recognition device 20 classifies the behavior of the worker based on the original video data captured by the imaging device 30. The classified behaviors include, for example, a specific first task such as processing a product, and a second task that is a task other than the first task.

As shown in FIG. 2, the behavior recognition device 20 is configured by a computer including a processing device 21 and a storage device 22 connected to the processing device 21. The computer configuring the behavior recognition device 20 further includes an interface 25.

The processing device 21 provided in the behavior recognition device 20 has, for example, at least one of a GPU (Graphics Processing Unit) and a CPU (Central Processing Unit). The storage device 22 has, for example, a primary storage device and a secondary storage device. The primary storage device is, for example, a RAM. The secondary storage device is, for example, a hard disk drive (HDD) or a solid state drive (SSD). The interface 25 is used to connect to external devices such as the learning model generation device 10, the imaging device 30, and the display device 40. The behavior recognition device 20 is connected to the external devices, for example, via a network not shown.

The computer constituting the behavior recognition device 20 executes the behavior recognition processing 210. The storage device 22 included in the behavior recognition device 20 stores a computer program 24 for causing the computer to function as the behavior recognition device 20. In other words, the storage device 12 stores the computer program 24 for causing the computer to execute the behavior recognition processing 210. The computer program 24 comprises program code for causing the computer to execute the behavior recognition processing 210. The processing device 21 reads out and executes the computer program 24 from the storage device 22. The behavior recognition processing 210 will be described later.

The storage device 22 can store original video data 220 and test data 221. The original video data 220 is acquired from the imaging device 30 and stored in the storage device 22. The test data 221 is video data generated from the original video data 220 stored in the storage device 22. The behavior recognition device 20 executes behavior recognition processing 210 on the test data 221. The behavior recognition processing 210 is executed using a learning model 225 constructed by machine learning. In the behavior recognition processing 210, the test data 221 is input to the learning model 225.

The storage device 22 can store the learning model 225 used in the behavior recognition process 210. Specifically, the storage device 22 stores parameters constituting the learning model 225. The learning model 225 according to the embodiment is configured, for example, by a neural network. As an example, the learning model 225 is configured by a three-dimensional convolutional neural network (3D Convolutional Neural Network; 3D-CNN) . The 3D-CNN can accept three-dimensional data having two-dimensional spatial information and one- dimensional time information as input data. In the embodiment, in the behavior recognition device 20, the three-dimensional data input to the learning model 225 configured by the 3D-CNN is the above-mentioned test data 221. The 3D-CNN classifies the behavior of an object (such as a worker) reflected in the input test data 221.

The behavior recognition device 20 outputs the task classification result by the behavior recognition processing 210 as the behavior recognition result. The task classification result is displayed, for example, on the display device 40. The display by the display device 40 will be described later.

In a 2D Convolutional Neural Network (2D-CNN), for example, input data for a color still image is composed of [Height, Width, Channels]. "Height" indicates the vertical position of the image. "Width" indicates the horizontal position of the image. "Channel" indicates the color of the image, for example, RGB (red, green, blue). On the other hand, the input data for a 3D-CNN is the same as the input data for a 2D-CNN, with time added. The input data for a 3D-CNN is composed of, for example, [Height, Width, Time, Channels].

Figure 3 shows the procedure of the behavior recognition process 210. First, the processing device 21 of the behavior recognition device 20 reads the original video data 220 stored in the storage device 22, and generates multiple test data 221 from the read original video data 220 (step S21). The original video data 220 has a time length of, for example, several minutes, several tens of minutes, several hours, or several tens of hours.

Figure 4 shows a method of generating test data 221 from original video data 220. Original video data 220 is divided into multiple short videos. The time length of each short video is, for example, 60 seconds. For example, if original video data 220 has a time length of one hour, 60 60-second short videos are generated. One test data 221 is generated from one short video. A 60-second short video is composed of multiple image data for 60 seconds. The test data 221 is extracted from the short video at equal intervals and in chronological order. Here, as an example, 32 pieces of image data are extracted from the short video. Video data consisting of a set of the extracted 32 pieces of image data becomes one piece of test data 221. In the test data 221, the 32 pieces of image data are arranged in chronological order. Note that the procedure of extracting multiple image data from the short video at equal intervals and in chronological order may be omitted, and each short video may be used as test data 221.

Returning to FIG. 3, the processing device 21 of the behavior recognition device 20 provides each test data 221 as an input to the learning model 225. When the test data 221, which is video data, is input, the learning model 225 according to the embodiment outputs a result of classifying the behavior of the object shown in the test data 221 (step S22). In the embodiment, as an example, the learning model 225 classifies whether the work performed by the worker shown in the test data 221 is a specific first work such as processing a product, or a second work that is a work other than the first work. The classification is performed for each test data 221. When the test data 221 is generated from a 60-second short video, the classification is performed on the worker's behavior every minute. Here, the types of behavior classified are the first work and the second work described above.

The processing device 21 of the behavior recognition device 20 outputs the task classification result (step S23). The task classification result is output, for example, as a display on the screen 400 by the display device 40 connected to the behavior recognition device 20.

Figure 5 shows an example of a screen 400 displayed by the display device 40. The screen 400 has a classification result display section 405 that shows the classification results of the work along a time axis. In the classification result display section 405 of Figure 5, white areas indicate the first work, and black areas indicate the second work. The screen 400 also has a date display section 401 that shows the date the work was performed, a first work display section 402 that shows the time when the first work was performed and the number of times it was performed, and a legend display section 403. By displaying the screen 400, a user such as a manager of the work can understand the time, number of times, timing, etc. when the first work was performed.

The system 1 shown in FIG. 1 includes a learning model generation device 10 that generates a learning model 225 used by the behavior recognition device 20. The learning model 225 generated by the learning model generation device 10 is provided to the behavior recognition device 20 via a network.

As shown in FIG. 6, the learning model generation device 10 according to the embodiment is configured by a computer including a processing device 11 and a storage device 12 connected to the processing device 11. The computer including the learning model generation device 10 further includes an interface 15.

The processing device 11 provided in the learning model generation device 10 has, for example, at least one of a GPU (Graphics Processing Unit) and a CPU (Central Processing Unit). The storage device 12 has, for example, a primary storage device and a secondary storage device. The primary storage device is, for example, a RAM. The secondary storage device is, for example, a hard disk drive (HDD) or a solid state drive (SSD). The interface 15 is used to connect to external devices such as the behavior recognition device 20 and the imaging device 30. The learning model generation device 10 is connected to the external devices, for example, via a network not shown.

The computer constituting the learning model generation device 10 executes a learning data generation process 110 that generates learning data used to generate a learning model. Therefore, the computer constituting the learning model generation device 10 is also a learning data generation device. In addition, the computer constituting the learning model generation device 10 executes a machine learning process 115 that generates a learning model using the learning data.

The storage device 12 included in the learning model generation device 10 stores a computer program 14 for causing a computer to function as the learning model generation device 10 or a learning data generation device. In other words, the storage device 12 stores a computer program for causing a computer to execute a learning data generation process 110. The storage device 12 also stores a computer program 14 for causing a computer to execute a machine learning process 115. The computer program 14 includes program code for causing the processing device 11 to execute the learning data generation process 110 and the machine learning process 115. The processing device 11 reads out and executes the computer program 14 from the storage device 12. The learning data generation process 110 and the machine learning process 115 will be described later.

The storage device 12 can store original video data 120 and learning data 121. The original video data 120 is acquired from the imaging device 30 and stored in the storage device 12. The learning data 121 is generated from the original video data 120 stored in the storage device 12 by a learning data generation process 110. The learning data 121 is also called teacher data. The learning data according to the embodiment includes first video data 121A and second video data 121B. The first video data 121A is generated by a first video data generation process 111 included in the learning data generation process 110. The second video data 121B is generated by a second video data generation process 112 included in the learning data generation process 110.

Figure 7 shows the procedure of the learning data generation process 110. First, the processing device 11 of the learning model generation device 10 acquires the original video data 120 by reading the original video data 120 stored in the storage device 12 (step S11). The processing device 11 creates multiple short videos from the original video data 120 (step S12). In step S12, as shown in Figure 8, multiple short videos are created so that a predetermined overlap time occurs between each short video. By creating overlaps, many short videos can be created from the original video data 120. In other words, data expansion is possible by overlapping video data. By performing data expansion so that overlaps occur between video data, efficient learning is possible without reducing the action recognition accuracy (task classification accuracy).

The duration of each short video is preferably 30 seconds or more, more preferably 40 seconds or more, and even more preferably 50 seconds or more. If the duration of the short videos is sufficiently long, the differences between overlapping short videos will be significant. Therefore, longer duration short videos are more suitable for data expansion.

Furthermore, the duration of each short video is preferably 5 minutes or less, more preferably 3 minutes or less, and even more preferably 2 minutes or less. If the duration of the short video is too long, the number of short videos obtained from the original video data 120 will be reduced, which is detrimental to machine learning. Therefore, it is advantageous for the duration of the short video not to be too long.

The duration of each short video is preferably within the range of 30 seconds or more and 5 minutes or less. As mentioned above, the lower limit of this range is more preferably 40 seconds or more, and even more preferably 50 seconds or more. As mentioned above, the upper limit of this range is more preferably 3 minutes or less, and even more preferably 2 minutes or less.

The overlap time between each short video is preferably 5 seconds or more, more preferably 20 seconds or more, and even more preferably 30 seconds or more. A longer overlap time is advantageous because it increases the number of short videos that can be obtained from the original video data 120.

The overlap time of each short video is preferably 50 seconds or less, more preferably 45 seconds or less, and even more preferably 40 seconds or less. If the overlap time is too long, the difference between the overlapping short videos will be small, which is disadvantageous for data expansion.

The overlap time of each short video is preferably within the range of 5 seconds or more and 50 seconds or less. As mentioned above, the lower limit of this range is more preferably 20 seconds or more, and even more preferably 30 seconds or more. As mentioned above, the upper limit of this range is more preferably 45 seconds or less, and even more preferably 40 seconds or less.

The ratio of overlap time to the duration of each short video is preferably 10% or more, more preferably 30% or more, and even more preferably 50% or more. A large ratio of overlap time is advantageous because it increases the number of short videos that can be obtained from the original video data 120.

The ratio of overlap time to the duration of each short video is preferably 80% or less, more preferably 75% or less, and even more preferably 70% or less. If the ratio of overlap time is too large, the difference between the overlapping short videos will be small, which is disadvantageous for data expansion.

The ratio of overlap time to the duration of each short video is preferably in the range of 10% or more and 80% or less. As mentioned above, the lower limit of this range is more preferably 30% or more, and even more preferably 50% or more. As mentioned above, the upper limit of this range is more preferably 75% or less, and even more preferably 70% or less.

For example, if the duration of a short video is 60 seconds, the overlap time can be one of 5 seconds, 10 seconds, 15 seconds, 20 seconds, 25 seconds, and 30 seconds.

Figure 8 shows an example where the duration of the short video is 60 seconds and the overlap time is 30 seconds. In Figure 8, the first short video is video ranging from 0 [s] to 60 [s] of the original video data. The second short video is video ranging from 30 [s] to 90 [s] of the original video data. The third short video is video ranging from 60 [s] to 120 [s] of the original video data.

As shown in FIG. 4, if the original video data is simply divided, only two 60-second short videos can be obtained from 120 seconds of original video data. However, as shown in FIG. 8, if an overlap time of 30 seconds is provided, three 60-second short videos can be obtained from 120 seconds of original video data 120, which is advantageous.

Returning to FIG. 7, the processing device 11 labels each short video with a type of work (step S13). That is, a type of work is associated with each short video and stored in the storage device 12. The type of work here is either a first work or a second work. The type of work labeled with the short video is used as the type of work associated with the first video data 121A and the second video data 121B created from the short video.

The processing device 11 extracts multiple image data from the short video at equal intervals and in chronological order (step S14). For example, as shown in FIG. 9, 32 pieces of image data are extracted from a 60-second short video consisting of multiple pieces of image data for 60 seconds. The set of 32 extracted pieces of image data becomes the first video data 121A for learning. By thinning out the image data in this way, even if the time length of the first video data 121A is extended, an increase in the amount of data in the first video data 121A can be suppressed. In the first video data 121A, the 32 pieces of image data have a time length of 60 seconds (first time length) and are arranged in chronological order.

In the learning data generation process 110, the process up to step S14 corresponds to the first video data generation process 111. Each of the generated multiple first video data 121A overlaps in time with other first video data 121A included in the multiple first video data 121A. For example, each first video data 121A has a time length of 60 seconds, and the overlap time with other first video data 121A is 30 seconds. Note that in the learning data generation process 110, the procedure of extracting multiple image data at equal intervals and in chronological order from the short video may be omitted, and each short video may be used as the first video data 121A.

Furthermore, the processing device 11 extracts a plurality of image data from the short video at equal intervals and in reverse order to the chronological order (step S15). For example, as shown in FIG. 9, 32 pieces of image data are extracted in reverse order to the first video data 121A from a 60-second short video composed of a plurality of image data for 60 seconds. The set of 32 extracted pieces of image data becomes the second video data 121B for learning. By thinning out the image data in this way, even if the time length of the second video data 121B is extended, the increase in the amount of data of the second video data 121B can be suppressed. In the second video data 121B, the 32 pieces of image data have a time length of 60 seconds (first time length) and are arranged in reverse order to the chronological order. In other words, the second video data 121B played back is equivalent to the first video data 121A played back in reverse.

In the learning data generation process 110, the process of step S15 corresponds to the second video data generation process 112. Each of the generated second video data 121B overlaps in time with other second video data 121B included in the plurality of second video data 121B. For example, each second video data 121B has a time length of 60 seconds, and the overlap time with other second video data 121B is 30 seconds. Note that in the learning data generation process 110, the second video data 121B may be generated not from a short video but by rearranging the first video data 121A in reverse order.

In this embodiment, not only the first video data 121A but also the second video data 121B in the reverse order to the first video data 121A is created as learning data, so the amount of learning data can be increased. In addition, the first video data 121A and the second video data 121B each have overlapping times, so the amount of learning data is further increased.

The learning model generation device 10 executes the machine learning process 115 using the first video data 121A and the second video data 121B as learning data (teacher data). The machine learning process 115 optimizes the learning model 225 for behavior recognition such as task classification. This completes the generation of the learning model 225. In the learning data generation process 110, data expansion is performed to generate the second video data 121B, so the number of learning data is large. This allows for efficient learning. The generated learning model 225 is transmitted to the behavior recognition device 20.

Here, the learning model (particularly, 3D-CNN) to which the video data is input does not capture the movement characteristics of the video data, but merely selects image data (image frames) necessary for identification (classification) from the input video data. Therefore, the direction in which an object (such as a worker) in the video data faces and the order of its movements do not have much effect on behavior recognition. In this embodiment, focusing on this viewpoint, data expansion of the learning data is performed by creating second video data 121B in the reverse order to the first video data 121A. As described later, by using the second video data 121B, efficient learning is possible without reducing the behavior recognition accuracy (task classification accuracy).

Figure 10 shows the results of an experiment in which the actions of a worker performing a first task and a second task were classified by the behavior recognition device 20 using the learning model according to the embodiment. The experiments were a first experiment conducted on a first day, and a second experiment conducted on a second day different from the first day. Each of the first and second experiments was conducted under a number of different conditions. The multiple conditions were condition 1, condition 2, condition 3, condition 4, condition 5, and condition 6.

In condition 1, only multiple pieces of first video data 121A that were not overlapped with each other and whose image data were arranged in chronological order were used as the learning data. In condition 2, multiple pieces of first video data 121A that were not overlapped with each other and whose image data were arranged in chronological order, and multiple pieces of second video data 121B that were not overlapped with each other and whose image data were arranged in reverse chronological order were used as the learning data. In condition 3, the first video data 121A and the second video data 121B were used as in condition 2. However, in condition 3, the overlap time between the first video data 121A and the overlap time between the second video data 121B were each set to 5 s.

Condition 4 is the same as condition 3, except that the overlap time is set to 20 seconds. Condition 5 is the same as condition 3, except that the overlap time is set to 25 seconds. Condition 6 is the same as condition 3, except that the overlap time is set to 30 seconds.

In FIG. 10, TP indicates a true positive, which means that the task was correctly classified as the first task. FP indicates a false positive, which means that the task was incorrectly classified as the first task. FN indicates a false negative, which means that the task was overlooked as the first task. TN indicates a true negative, which means that the task was correctly classified as the second task.

Precision is calculated by the formula TP/(TP+FP). Recall is calculated by the formula TP/(TP+FN). F-measure is calculated by the formula (2 x precision x recall)/(precision + recall). F-measure is the harmonic mean of precision and recall.

For condition 1, the average F-value for the first and second experiments was 0.598. For condition 1, the average F-value for the first and second experiments was 0.672. Comparing condition 1 and condition 2, condition 2 has a better average F-value. Therefore, it can be seen that it is advantageous to use the reversed second video data 121B as learning data.

In the case of condition 3, the average F-value of the first and second experiments was 0.871. Comparing condition 2 and condition 3, condition 3 had a better average F-value. Therefore, it can be seen that it is advantageous to use the second video data 121B, which has overlapping time, as learning data.

In the case of condition 4, the average F-value of the first and second experiments was 0.882. In the case of condition 5, the average F-value of the first and second experiments was 0.897. In the case of condition 5, the average F-value of the first and second experiments was 0.934. It can therefore be seen that the longer the overlap time, the better the F-value. In particular, the F-value of condition 5, in which the overlap time was 30 seconds and the ratio of the overlap time to the respective durations (first duration) of the first video data 121A and the second video data 121B was 50%, was the best.

Figures 11 and 12 show the task classification results by visual inspection under conditions 1, 2, 3, 4, 5, and 6 in the first experiment, and the task classification results by the learning model. Figures 13 and 14 show the task classification results by visual inspection under conditions 1, 2, 3, 4, 5, and 6 in the second experiment, and the task classification results by the learning model. In the classification results shown in Figures 11 to 14, the horizontal axis is time. In Figures 11 to 14, white areas indicate classification as the first task, and black areas indicate classification as the second task.

When observing the focus points 721, 722, 723, 724, 725, and 726 shown in Figures 11 and 12, in the first experiment, the classification results by the learning model were closer to the classification results by visual inspection in the order of conditions 1, 2, 3, 4, 5, and 6, and were therefore better.

Furthermore, when observing the focus points 731, 732, 733, 734, 735, and 736 shown in Figures 13 and 14, it can be seen that in the second experiment as well, the classification results based on the learning model are closer to the classification results based on visual inspection in the order of conditions 1, 2, 3, 4, 5, and 6, and are therefore better.

According to the experimental results shown in Figures 10 to 14, it is more preferable that the ratio of overlap time to the duration (first duration) of each of the first video data 121A and the second video data 121B is 50% or more. As mentioned above, if the ratio of overlap time is too large, the difference between the overlapping short videos will be small. For this reason, it is preferable that the ratio of overlap time to the duration (first duration) of each of the first video data 121A and the second video data 121B is in the range of 50% or more and 80% or less.

It should be noted that the embodiments disclosed herein are illustrative in all respects and are not restrictive. The scope of the present invention is indicated by the claims, not by the meaning described above, and is intended to include all modifications within the scope of the claims and meanings equivalent thereto.

1: System 10: Learning model generation device 11: Processing device 12: Storage device 14: Computer program 15: Interface 20: Behavior recognition device 21: Processing device 22: Storage device 24: Computer program 25: Interface 30: Imaging device 40: Display device 110: Learning data generation process 111: First video data generation process 112: Second video data generation process 115: Machine learning process 120: Original video data 121: Learning data 121A: First video data 121B: Second video data 210: Behavior recognition process 220: Original video data 221: Test data 225: Learning model 400: Screen 401: Date display section 402: First task display section 403: Legend display section 405: Classification result display section 721: Focus point 722: Focus point 723: Focus point 724 : Focus point 725 : Focus point 726 : Focus point 731 : Focus point 732 : Focus point 733 : Focus point 734 : Focus point 735 : Focus point 736 : Focus point

Claims (6)

A learning data generation process including a second image data generation process for generating second image data different from the first image data in which a plurality of image data are arranged in chronological order;
a machine learning process for generating a learning model for behavior recognition using the first video data and the second video data as learning data;
a processor configured to execute
The learning data generation process further includes a first video data generation process for generating a plurality of first video data from original video data, the first video data having a first time length shorter than a time length of the original video data;
each of the plurality of first video data generated by the first video data generation process overlaps in time with other first video data included in the plurality of first video data;
The second video data is configured such that the plurality of image data included in the first video data are arranged in a reverse order to the chronological order. The learning model generation device according to claim 1 , wherein an overlap time during which each of the plurality of first video data overlaps with other first video data included in the plurality of first video data is greater than or equal to 50% and less than or equal to 80 % of the first time length. A learning model for activity recognition, which is machine-learned using learning data,
The learning data is
a plurality of first video data in which a plurality of image data are arranged in chronological order, the plurality of first video data being overlapped in time by generating a plurality of the first video data having a first time length shorter than a time length of the original video data from original video data;
second image data in which the plurality of image data included in the first image data are arranged in a reverse order to the chronological order;
Including,
causing a computer to function in such a way that, when video data is given as an input, the computer outputs a recognition result of an object's behavior in the video data;
Learning model. An action recognition device configured to recognize an action of an object in video data as test data using the learning model according to claim 3 . a processing device configured to execute a learning data generation process including a second video data generation process for generating second video data different from first video data in which a plurality of image data are arranged in chronological order;
The learning data generation process further includes a first video data generation process for generating a plurality of first video data from original video data, the first video data having a first time length shorter than a time length of the original video data;
each of the plurality of first video data generated by the first video data generation process overlaps in time with other first video data included in the plurality of first video data;
the second video data is configured such that the plurality of image data included in the first video data are arranged in a reverse order to the chronological order. The method includes a first step of generating a plurality of first image data in which a plurality of image data are arranged in chronological order, and a second step of generating a second image data different from the plurality of first image data,
The first step includes generating a plurality of first video data from original video data, the first video data having a first time length shorter than a time length of the original video data;
Each of the plurality of first image data generated in the first step overlaps in time with other first image data included in the plurality of first image data,
the second video data is configured such that the plurality of image data included in the first video data are arranged in a reverse order to the chronological order.

Images