JP7798700B2 - Learning system, worker action recognition system, and recognition model learning method - Google Patents

Description

The present invention relates to a learning system, a worker action recognition system, and a method for learning a recognition model.

The long tail is a fundamental problem often faced in recognition work using artificial intelligence. This problem occurs when there is significantly less training data for some categories than for other categories. Recognizing worker activities in industry is also prone to uneven training data. In industry, some activities are more risky than others. For example, climbing onto a roof is much riskier than tightening screws, and if performed frequently, it can lead to an accident. As some activities involve high risks, it is extremely difficult to collect the same amount of training data for all activities, resulting in a long-tail dataset.

Creating an activity recognition model using such a long-tail dataset results in biased performance. Specifically, it shows very good performance for categories with sufficient training data and very poor performance for categories with little training data. This is because traditionally, recognition models are updated by assigning equal weight to all training samples, resulting in more learning from categories with more data. The simplest way to solve this problem is to collect more data for activities with little data. However, this is not realistic because categories with little data typically represent high-risk activities that cannot be repeated many times, such as climbing onto a roof. There is a need to recognize such dangerous activities with high accuracy and provide support to prevent worker accidents.

There has been previous research that attempts to solve the long tail without collecting more data. Two common approaches to the long tail are data resampling and weighted loss. Data resampling samples training samples with unequal sampling probability, generally with a higher probability for categories with fewer training samples. Weighted loss assigns a weight to each sample, which is used to increase or decrease the update from each sample. The weight is generally higher for categories with fewer training samples, with the aim of balancing the amount of update from all categories.

Patent document 1 uses a weighted loss, and other methods that use data resampling are also known. However, most of these methods have been proposed for image recognition tasks. Furthermore, a method that introduces a frame stacking technique to tackle long-tail video recognition problems is also known. In this method, two categories are randomly sampled, and frames are sampled and stacked from each category based on their frequency in the training data. Next, a recognition model is trained using soft labels to achieve balanced learning from the two categories.

US Patent Application Publication No. 2019/0232964

Patent Document 1 and other known techniques assume that the provided video has been appropriately trimmed, i.e., segmented in the time domain, during inference. However, in reality, it is entirely conceivable that appropriate trimming will not be performed. For example, if a video contains both high-risk actions and portions of low-risk actions, the recognition system will likely recognize only the low-risk actions and ignore the risky actions. However, in order to prevent potential accidents in industry, it is extremely important to recognize high-risk actions in videos that have been improperly trimmed in this way.

A learning system according to a first aspect of the present invention comprises a learning video generation unit that combines two videos, a main category video that is a video belonging to a main category included in a behavioral category, and a secondary category video that is a video belonging to a secondary category that is included in the behavioral category and is different from the main category, at a connection point and trims the videos to include the connection point to generate a learning video; a learning unit that uses the learning video to train a recognition model that outputs the main category for the input learning video; and a performance evaluation unit that evaluates the prediction results by the recognition model to generate evaluation results, and the secondary category is selected based on the risk levels of the main category and the secondary category in the behavioral category and the evaluation results.
A worker action recognition system according to a second aspect of the present invention comprises a recognition model generated by the aforementioned learning system, a detector for detecting a worker, a tracker for tracking the worker, and an extractor for extracting a candidate area, which is an area in which the worker is present, using the detector and the tracker, and the recognition model receives as input an image of the candidate area.
A worker action recognition system according to a third aspect of the present invention comprises a recognition model generated by the aforementioned learning system, a detector for detecting a worker's hand, a tracker for tracking the worker's hand, and an extractor for extracting a candidate area, which is an area in which the worker is present, using the detector and the tracker, and the recognition model receives as input an image of the candidate area.
A fourth aspect of the present invention provides a method for training a recognition model that outputs a category corresponding to an input video, and includes the steps of: combining two videos, a main category video that is a video belonging to a main category included in an action category, and a secondary category video that is a video belonging to a secondary category included in the action category and different from the main category, at a connection point, and trimming the videos to include the connection point to generate a training video; using the training video, training the recognition model to output the main category for the input training video; and evaluating the prediction results by the recognition model to generate an evaluation result, wherein the secondary category is selected based on the risk levels of the main category and the secondary category in the action category and the evaluation result.

The present invention provides a recognition model that can recognize high-risk behavior even in videos that are insufficiently trimmed.

1 is a diagram illustrating a configuration of a learning system according to a first embodiment. Conceptual diagram explaining segmented video and composite video FIG. 10 is a diagram showing an example of an action label group. FIG. 10 is a diagram showing an example of a history of decisions made by a main category decision unit; A diagram showing an example of statistical information An example of risk information A diagram showing an example of evaluation results Learning system hardware configuration diagram Flowchart showing the operation of the relative difficulty calculation unit Flowchart showing the operation of the determination unit Flowchart showing the operation of the learning video generation unit A diagram showing a specific example of the operation of the learning video generation unit Flowchart showing the operation of the performance evaluation unit 1 is a diagram illustrating a configuration of a learning system according to a second embodiment. 10 is a flowchart showing the operation of the learning video generation unit in the second embodiment. 1. A configuration diagram of a learning system according to a third embodiment. 10 is a flowchart showing the operation of the main category determination unit in the third embodiment. FIG. 13 is a diagram showing an example of gaze parameters in a modified example of the third embodiment. 10 is a diagram showing the configuration of a learning system according to a fourth embodiment. A diagram explaining the GUI Worker motion recognition system configuration diagram

-First embodiment-

The first embodiment of the learning system will be described below with reference to Figures 1 to 13.

Figure 1 is an overall configuration diagram of learning system 1. Learning system 1 has, as its functions, a main category determination unit 103, a subcategory determination unit 106, a learning video generation unit 107, a performance evaluation unit 113, a learning unit 114, and a risk information generation unit 118. Learning system 1 also has information such as a segmented video group 101, an action label group 102, statistical information 119, risk information 104, a ratio parameter 105, and an evaluation result 117. Before explaining the elements that make up learning system 1, we will explain the classification of videos in this embodiment.

Figure 2 is a conceptual diagram illustrating segmented videos and composite videos in this embodiment. Video V100 is a 40-second video, i.e., a video clip. Video V100 includes multiple scenes, such as a worker walking scene from 0 to 10 seconds, a scene of tightening screws from 10 to 20 seconds, a scene of working on a roof from 20 to 30 seconds, and a scene of creating a record from 30 to 40 seconds. Each scene includes only one type of human action. In this embodiment, human actions are classified as "action labels." Action labels include, for example, walking, running, tightening screws, hitting with a hammer, climbing onto the roof, and creating a record.

In this embodiment, video V100 is trimmed to include only one scene, i.e., extracted in the time domain, and is referred to as a "divided video." In contrast, video that has been incompletely divided and trimmed to include multiple scenes is referred to as a "composite video." In other words, a composite video includes multiple actions performed by a person. For example, video V101, which is video V100 from 0 to 10 seconds, video V102, which is video V103, which is video V103 from 20 to 30 seconds, and video V104, which is video V104 from 30 to 40 seconds, are all divided videos. In contrast, video V111, which is video V100 from 0 to 13 seconds, video V112, which is video V112 from 5 to 20 seconds, and video V113, which is video V100 from 20 to 45 seconds, are all composite videos. Note that in this embodiment, trimming a video means extracting a video, i.e., a video clip, in the time domain, as shown in Figure 2. Let's return to Figure 1 and continue the explanation.

The segmented video group 101 is a collection of segmented videos that have been created in advance. The action label group 102 is a collection of action labels that have been created in advance and correspond to each video included in the segmented video group 101.

The main category determination unit 103 determines the main category of the video to be included in the composite video generated by the training video generation unit 107. The main category determination unit 103 may determine the main category by referring to the action label group 102, or may determine the main category by referring to the risk information 104. The main category determination unit 103 outputs information on the determined main category to the subcategory determination unit 106, the training video generation unit 107, the loss calculation unit 111, and the performance evaluation unit 113. However, for convenience of illustration, arrows from the main category determination unit 103 to the loss calculation unit 111 and the performance evaluation unit 113 are not shown in Figure 1. The detailed operation of the main category determination unit 103 will be described later.

The subcategory determination unit 106 determines the subcategory of the video to be included in the training video generated by the training video generation unit 107. The subcategory determination unit 106 outputs the determined subcategory to the training video generation unit 107. The subcategory determination unit 106 receives as input the main category determined by the main category determination unit 103, the ratio parameter 105, the risk information 104, and the performance of the recognition model 108 output by the performance evaluation unit 113. The training video generation unit 107 generates a training video, which is a composite video, based on the main category determined by the main category determination unit 103 and the subcategory determined by the subcategory determination unit 106. The training video generation unit 107 outputs the generated training video to the recognition model 108.

The learning unit 114 includes a recognition model 108, a prediction result 110, a loss calculation unit 111, an update unit 112, and model parameters 109. The learning unit 114 updates the model parameters 109 of the recognition model 108 using the training videos generated by the training video generation unit 107. The training video generation unit 107 inputs the training unit 114, which is unique to this embodiment. However, since the training process of a neural network that outputs labels for input videos is widely known, the configuration and processing of the training unit 114 will be explained briefly.

The recognition model 108 is any video recognition model, such as Transformers, Long Short Term Memory (LSTM), or 3D-CNN (3D Convolutional Neural Network). The model parameters 109 are the weights and biases of the model. When a training video is input from the training video generation unit 107, the recognition model 108 outputs a prediction result 110. This prediction result 110 is compared by both the loss calculation unit 111 and the performance evaluation unit 113 with the correct category information of the training video, i.e., the main category output by the main category determination unit 103.

The loss calculation unit 111 calculates the loss using, for example, a cross-entropy loss function. However, the loss calculation unit 111 may also use other loss functions such as focus loss, mean squared error loss, and binary cross-entropy loss. The update unit 112 updates the model parameters 109 using the loss calculated by the loss calculation unit 111. The update unit 112 may use, for example, a backpropagation method. The performance evaluation unit 113 calculates the performance of the recognition model 108 using the prediction result 110 and feeds it back to the subcategory determination unit 106. The performance evaluation unit 113 calculates the performance of the recognition model 108 for a predetermined number of batches or a predetermined number of epochs. The learning process by the learning unit 114 is repeated until the loss reaches a minimum value or the number of iterations reaches a maximum value.

Figure 3 is a diagram showing an example of the action label group 102. The action label group 102 is made up of multiple records, and each record has fields for identifier 201 and action label 202. "VC001" and the like written in the identifier 201 column are identifiers of videos included in the divided video group 101. "A1", "A2", and the like written in the action label 202 column are action labels. In the example shown in Figure 3, it is shown that the action label of video "VC001" is "A1".

Figure 4 shows an example of the history of decisions made by the main category determination unit 103. The example shown in Figure 4 shows that the first time the main category determination unit 103 makes a decision, "A4" is determined as the main category, the second time "A2" is determined, and the third time "A7" is determined. Note that this information may be sent each time the main category determination unit 103 makes a decision, or may be sent all at once after multiple decisions have been made.

Figure 5 is a diagram showing an example of statistical information 119. Statistical information 119 may be statistics related to videos included in segmented video group 101, or may be statistics for a predetermined period, such as the past year, regardless of whether the video is included in segmented video group 101. Statistical information 119 is made up of multiple records, and each record has fields for action label 1191, number of accidents 1192, and total number 1193. Action label 1191 has the same meaning as action label 1022. Number of accidents 1192 is the number of accidents that occurred in the task corresponding to action label 1191 of the same record. Total number 1193 is the total number of tasks corresponding to action label 1191 of the same record. That is, in statistical information 119, number of accidents 1192 and total number 1193 are actually entered as numerical values.

Since statistical information 119 is used to evaluate the relative risk between action labels 1191, various criteria can be used as long as the criteria are consistent across one piece of statistical information 119. For example, total number 1193 may be the number of videos in which the action corresponding to action label 1191 was filmed, or the number of times the action corresponding to action label 1191 was performed regardless of whether it was filmed or not. Furthermore, total number 1193 may be information indicating the length of time the action corresponding to action label 1191 was performed, for example, a number in minutes indicating the total time it was performed.

Figure 6 is a diagram showing an example of risk information 104. Risk information 104 is composed of multiple records, each of which has fields for action label 1041 and risk index 1042. Action label 1041 is the same as action label 202, etc. The risk index 402 is a numerical value indicating the magnitude of risk of the action represented by action label 401 in the same record. The risk index 402 can be defined and calculated in any way as long as it allows comparison between action labels. For example, the risk information generation unit 118 may calculate the risk index 402 using statistical information 119, or a human may determine the risk index 402 based on the results of a survey. For example, the risk information generation unit 118 may use a value between 0 and 1 obtained by dividing the number of accidents 501 in statistical information 119 by the total number 502 for each action label as the risk index 402.

Figure 7 shows an example of the evaluation result 117 by the performance evaluation unit 113. The evaluation result 117 consists of multiple records, each of which has an action label and performance field. In the example shown in Figure 7, classification accuracy is used as performance, but various other metrics such as accuracy, f-measure, and error may also be used.

Figure 8 is a hardware configuration diagram of the learning system 1. The learning system is composed of one or more arithmetic units 40. The arithmetic unit 40 includes a CPU 41, which is a central processing unit; a ROM 42, which is a read-only storage device; a RAM 43, which is a read/write storage device; an input/output device 44, which is a user interface; and a communication device 45. The CPU 41 performs the various calculations described above by expanding the programs stored in the ROM 42 into the RAM 43 and executing them. In other words, the main category determination unit 103, sub-category determination unit 106, learning video generation unit 107, performance evaluation unit 113, loss calculation unit 111, update unit 112, and risk information generation unit 118 are realized by the CPU 41.

The divided video group 101, action label group 102, statistical information 119, risk information 104, ratio parameter 105, and evaluation result 117 may be stored in RAM 43 or in a non-volatile storage device (not shown). The communication device 45 is used for communication with other arithmetic devices 40. If the learning system 1 is composed of a single arithmetic device 40, the arithmetic device 40 does not need to be equipped with a communication device 45.

Instead of the combination of CPU 41, ROM 42, and RAM 43, the arithmetic unit 40 may be realized by a rewritable logic circuit, such as a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). Furthermore, instead of the combination of CPU 41, ROM 42, and RAM 43, the arithmetic unit 40 may be realized by a combination of different configurations, such as a combination of CPU 41, ROM 42, RAM 43, and FPGA.

The main category determination unit 103 operates in the following two ways. The first operation is determining the correct frequency. In this case, the main category determination unit 103 randomly extracts one correct divided video from the divided video group 101, and reads out the action label corresponding to that correct divided video from the action label group 102. In this case, since videos are extracted randomly from the divided video group 101, the proportion of video categories included in the divided video group 101 is directly proportional to the proportion of extracted categories. The more videos a category contains in the divided video group 101, the more likely it is to be selected. In other words, in this case, it can be said that categories are selected with directly proportional frequency.

The second operation is inverse frequency determination. In this case, the main category determination unit 103 first reads the entire action label group 102 and calculates the presence ratio of action labels for videos included in the segmented video group 101. The main category determination unit 103 then selects an action label with a probability corresponding to the inverse of the presence ratio. Therefore, in inverse frequency determination, categories with fewer videos included in the segmented video group 101 are more likely to be selected. Inverse frequency determination is effective for long-tail recognition problems.

Figure 9 is a flowchart showing the operation of the relative difficulty calculation unit 703. First, in step S301, the relative difficulty calculation unit 703 reads the necessary information. The information read in this step is the risk information 104, the main category determined by the main category determination unit 103, the evaluation result 117 calculated by the performance evaluation unit 113, and the ratio parameter 105. In the following step S302, the relative difficulty calculation unit 703 initializes the variable i to 1 and proceeds to step S303. This variable i indicates the action category, and changing the value of variable i changes the action category indicated by action label Ai. In step S303, the relative difficulty calculation unit 703 calculates the relative difficulty d(n, i) between action label An and action label Ai. The relative difficulty d(n, i) is calculated using the following equation 1.

d(n, i)=POW(Rn/Ri, γ)*POW(ACCi/ACCn, 1-γ)... (Formula 1)

However, the function POW in Equation 1 represents the exponentiation of the second argument to the first argument; for example, POW(10,2) represents 10 to the square, which is 100. Also, in Equation 1, Rn represents the risk index for action label An, Ri represents the risk index for action label Ai, ACCi represents the evaluation value for action label Ai, and ACCn represents the evaluation value for action label An. γ is a ratio parameter 105, a preset value between 0 and 1.

Note that action label An is the action label of the main category, and action label Ai is the action label of the category corresponding to variable i. If the risk index of action label An is greater than the risk index of action label Ai, Rn/Ri is greater than 1. ACCi/ACCn is the bias of category n relative to category i in model performance. As γ increases, the weight of the risk ratio at relative difficulty d increases, and as γ decreases, the weight of the risk ratio at relative difficulty d decreases.

In the following step S304, the relative difficulty calculation unit 703 increments the variable i, i.e., increases the value of variable i by "1", and proceeds to step S305. In step S305, the relative difficulty calculation unit 703 determines whether the value of variable i exceeds the total number C of action labels. If the relative difficulty calculation unit 703 determines that the value of variable i exceeds the total number C of action labels, it proceeds to step S306, and if it determines that the value of variable i does not exceed the total number C of action labels, it returns to step S303. In step S306, the relative difficulty calculation unit 703 outputs the relative difficulty d calculated in step S303 up to this point to the determination unit 704, and ends the processing shown in FIG. 9. In steps S302 to S305, the relative difficulty is calculated by changing the variable i from 1 to C, so the relative difficulty d output in step S306 is d(n,1), d(n,2), d(n,3), ..., d(n,C).

Figure 10 is a flowchart showing the operation of the determination unit 704. First, in step S311, the determination unit 704 reads the relative difficulty d calculated by the relative difficulty calculation unit 703. In the following step S312, the determination unit 704 identifies the category number of the relative difficulty having the maximum value from the multiple relative difficulty levels d that have been read. In the following step S313, the determination unit 704 outputs the category number identified in step S312 as the number of the subcategory. In the following step S314, the determination unit 704 outputs the maximum relative difficulty and ends the processing shown in Figure 10.

Figure 11 is a flowchart showing the operation of the learning video generation unit 107. The processing of Figure 11 will be described below with reference to Figure 12, which shows a specific example of the operation of the learning video generation unit 107. The learning video generation unit 107 first reads the necessary information in step S321. The information read in this step is the main category and sub-category. In the following step S322, the learning video generation unit 107 randomly selects one video in the main category from the divided video group 101. However, if step S322 has been executed previously, the processing of this step may be omitted and a video in the same main category as the previous time may be selected.

In the following step S323, the learning video generation unit 107 randomly selects one video in the sub-category from the divided video group 101. In the following step S323, the learning video generation unit 107 simply combines the two videos selected in steps S322 and S323. Simple combining here refers to the process of joining the beginning of one video to the end of the other video to create a single video. This will be explained in detail with reference to Figure 12.

In Figure 12, the symbol Vp indicates the main category video selected in step S322. The symbol Vs indicates the sub-category video selected in step S323. For the sake of explanation below, the length of the main category video Vp is referred to as Tp, and the length of the sub-category video Vs is referred to as Ts. In step S324, the sub-category video Vs is placed after the main category video Vp, or the main category video Vp is placed after the sub-category video Vs. In either case, the length of the simply combined videos is Tp + Ts. Whether the main category video Vp or the sub-category video Vs is placed after the other is determined randomly.

In the following step S325, the learning video generation unit 107 identifies the switching timing Tw. The switching timing Tw is the timing at which the main category video Vp and the sub-category video Vs switch in the simply combined video Vc. The switching timing Tw can be indicated, for example, by the time from the beginning of the simply combined video Vc or the time from the end of the simply combined video Vc. In this embodiment, the switching timing Tw is indicated by the time from the beginning of the simply combined video Vc. When the sub-category video Vs is placed after the main category video Vp to form the simply combined video Vc, the switching timing Tw is Tp, which is the length of the main category video Vp. When the main category video Vp is placed after the sub-category video Vs to form the simply combined video Vc, the switching timing Tw is Ts, which is the length of the sub-category video Vs.

In the following step S326, the learning video generation unit 107 determines a front length L1, which is the length of the video before the switch, and a back length L2, which is the length of the video after the switch. The front length L1 and back length L2 may be determined randomly or may be specified in advance by the operator. Furthermore, instead of directly determining or specifying the front length L1 and back length L2, the sum of the front length L1 and back length L2, or the ratio of the front length L1 to the back length L2, may be determined or specified. In the following step S326, the learning video generation unit 107 extracts the video from time "Tw-L1" to time "Tw+L2" in the simply combined video Vc as a learning video, and ends the processing in FIG. 11. As shown in FIG. 12, the learning video VT has a length of L1+L2, and is a video in which the main category video Vp and the sub-category video Vs switch at time L1.

The processing performed by the learning video generation unit 107 described above can be summarized as follows: The learning video generation unit 107 joins two videos, a main category video Vp and a sub-category video Vs, which is a video belonging to a sub-category different from the main category, at a joining point at the end of one of the videos, and then trims the video to include the joining point to generate a learning video.

Figure 13 is a flowchart showing the operation of the performance evaluation unit 113. In step S331, the performance evaluation unit 113 reads the prediction result 110 from the recognition model 108. As mentioned above, the performance evaluation unit 113 is given information on the main category for each training video VT in advance. In the following step S332, the performance evaluation unit 113 initializes a variable i to 1. This variable i indicates the action category, and changing the value of the variable i changes the action category indicated by the action label Ai.

In the following step S333, the performance evaluation unit 113 evaluates the prediction result 110. Specifically, the performance evaluation unit 113 determines whether the prediction result 110 matches the action label Ai for the training video VT for which prediction by the recognition model 108 has completed and all main categories are Ai. In the following step S334, the performance evaluation unit 113 calculates an evaluation value ACCi for the action label Ai of the recognition model 108. The evaluation value ACCi takes a value between 0 and 1 and is calculated using the following equation 2.

ACCi = GOOD_VTi/ALL_VTi (Formula 2)

where GOOD_VTi is the number of training videos whose main category is Ai that the recognition model 108 correctly determined to be category Ai, and ALL_VTi is the number of training videos whose main category is Ai. In the following step S335, the performance evaluation unit 113 increments the variable i, i.e., increases the value of the variable i by "1", and proceeds to step S336. In step S336, the performance evaluation unit 113 determines whether the value of the variable i exceeds the total number C of action labels. If the performance evaluation unit 113 determines that the value of the variable i exceeds the total number C of action labels, it proceeds to step S337, and if it determines that the value of the variable i does not exceed the total number C of action labels, it returns to step S333. In step S337, the performance evaluation unit 113 outputs the total "C" evaluation values calculated in step S334, i.e., ACC1, ACC2, ..., ACCC, as the evaluation result 117, and ends the processing shown in FIG. 13.

According to the first embodiment described above, the following advantageous effects can be obtained.
(1) The learning system 1 includes a learning video generation unit 107 that generates a learning video by joining two videos, a main category video that is a video belonging to a main category included in an action category, and a secondary category video that is a video belonging to a secondary category that is included in an action category but different from the main category, at a joining point that is the end of one of the videos and trimming the video to include the joining point; a learning unit 114 that uses the learning video to train a recognition model 108 that outputs a main category for an input learning video; and a performance evaluation unit 113 that evaluates a prediction result 110 by the recognition model 108 and generates an evaluation result 117. The secondary category is selected based on the risk levels of the main category and secondary category in the action category and the evaluation result 117. Therefore, it is possible to provide a recognition model 108 that can recognize high-risk behavior even in videos that are insufficiently trimmed.

(2) The learning system 1 includes a subcategory determination unit 106 that determines, as a subcategory, the behavioral category that is most often mistakenly determined by the recognition model 108 for the determined main category. This improves the accuracy of the recognition model 10.

(3) The learning system 1 is equipped with a main category determination unit 103 that determines an activity category as a main category more frequently the higher the risk in the activity category. This reduces errors in high-risk activity categories made by the recognition model 108.

(4) The subcategory determination unit 106 determines the subcategory based on the risk information 104 indicating the risk of each category included in the behavior category and the evaluation result 117, as shown in Equation 1.

(5) The learning system 1 includes a risk information generation unit 118 that calculates risk information 104 based on past statistics. Therefore, the user does not need to manually generate risk information 104.

(6) The subcategory determination unit 106 determines a subcategory based on the risk information 104, the evaluation result 117, and a ratio parameter 105 represented by the variable γ. The ratio parameter 105 determines the ratio of the influence that the risk information 104 and the evaluation result 117 have on determining the subcategory, as shown in Equation 1. Therefore, by appropriately setting the value of the ratio parameter 105, it is possible to achieve a balance between risk and evaluation.

(7) The performance evaluation unit 113 evaluates the performance of the recognition model 108 for each predetermined number of batches.

(8) The training method of the recognition model 108 is a training method of the recognition model 108 that outputs a category corresponding to an input video, and includes: combining two videos, a main category video that is a video belonging to a main category included in an action category, and a sub-category video that is a video belonging to a sub-category that is included in an action category and different from the main category, at a connection point, and trimming the videos to include the connection point to generate a training video; using the training video, training the recognition model 108 to output a main category for the input training video; and evaluating the prediction result 110 by the recognition model 108 to generate an evaluation result 117, where the sub-category is selected based on the risk levels of the main category and sub-category in the action category and the evaluation result 117.

- Second embodiment -
A second embodiment of the learning system will be described with reference to Figures 14 and 15. In the following description, the same components as in the first embodiment are designated by the same reference numerals, and differences will be mainly described. Points that are not specifically described are the same as in the first embodiment. In this embodiment, the main difference from the first embodiment is the processing of the learning video generation unit.

Figure 14 is a configuration diagram of a learning system 1A in the second embodiment. Learning system 1A includes a learning video generation unit 107A instead of the learning video generation unit 107 in the first embodiment. In the second embodiment, only the operation of the learning video generation unit 107A differs from the first embodiment.

Figure 15 is a flowchart showing the operation of the learning video generation unit 107A in the second embodiment. Figure 15 differs from Figure 11 in that steps S323A and S323B have been added between steps S323 and S324. The differences between Figure 15 and Figure 11 are explained below. Points not specifically explained are the same as those in the first embodiment.

In step S323A, the learning video generation unit 107A determines whether the relative difficulty d between the main category and the sub-category is greater than a predetermined threshold. This relative difficulty is calculated by the relative difficulty calculation unit 703. If the learning video generation unit 107A determines that the relative difficulty d is greater than the predetermined threshold THRE, it proceeds to step S324 and performs the same processing as in the first embodiment. If the learning video generation unit 107A determines that the relative difficulty d is equal to or less than the predetermined threshold THRE, it proceeds to step S323B. In step S323B, the learning video generation unit 107A outputs the video of the main category selected in step S322 as the learning video, and ends the processing shown in FIG. 15.

By increasing the value of the threshold THRE, the amount of calculation and time required for learning can be reduced. However, if the threshold THRE value is too large, there will be too few training videos that combine videos from multiple categories, reducing the robustness of the recognition model 108 against imperfect trimming. Therefore, the operator must carefully set the threshold THRE value.

According to the second embodiment described above, the following effects can be obtained.
(9) The learning video generation unit 107 generates learning videos when the rate at which learning videos of the main category are determined to be sub-categories is higher than a predetermined threshold in the recognition performance of the recognition model 108, i.e., when a positive judgment is made in S323A of Fig. 15. Therefore, when the performance of the recognition model 108 is sufficient, the processing load can be reduced by omitting the generation of learning videos.

-Third embodiment-
A third embodiment of a learning system will be described with reference to Figures 16 to 18. In the following description, the same components as in the first embodiment are designated by the same reference numerals, and differences will be mainly described. Points that are not specifically described are the same as in the first embodiment. In this embodiment, the main difference from the first embodiment is the operation of the main category determination unit.

During training, videos with high-risk action labels are sampled more frequently than videos with low-risk action labels, because risky actions are more difficult for the model to learn due to data limitations. Training risky actions more frequently allows the recognition model 108 to learn risky actions in a balanced manner, regardless of whether they are high or low risk. This is also desirable because it makes the model more robust to insufficient trimming of risky categories.

Figure 16 is a configuration diagram of a learning system 1B in the third embodiment. In this embodiment, the operation of the main category determination unit 103A differs from that in the first embodiment. In addition, risk information 104 is provided not only to the subcategory determination unit 106 but also to the main category determination unit 103A. The main category determination unit 103A reads not only the action label group 102, but also the risk information 104 and gaze parameters 1801.

Figure 17 is a flowchart showing the operation of the main category determination unit 103A in the third embodiment. First, the main category determination unit 103A reads the necessary information in step S351. In the following step S352, the main category determination unit 103A calculates the video frequency fi for each category using the action label group 102. The video frequency fi is the ratio of videos in category Ai included in the segmented video group 101, and is the value obtained by dividing the number of videos in category Ai by the total number of videos included in the segmented video group 101.

Specifically, the main category determination unit 103A calculates f1, which is the value obtained by dividing the number of videos in category A1 by the total number of videos included in the segmented video group 101, f2, which is the value obtained by dividing the number of videos in category A2 by the total number of videos included in the segmented video group 101, and so on, for all categories. In the following step S353, the main category determination unit 103A calculates the sampling index SIi for each category. The sampling index SIi is calculated using the following equation 3.

SIi = POW (Ri, λ) * POW (1/fi, 1-λ) ... (Formula 3)

In Equation 3, Ri is the risk index for category Ai, λ is the gaze parameter 1801, and fi is the video frequency fi for category Ai. In the following step S354, the main category determination unit 103A calculates the index sum SI_SUM, which is the sum of the sampling indices SIi for each category. Since the index sum SI_SUM is the sum of the sampling indices SIi for all categories, it can be calculated using the following Equation 4.

SI_SUM = SI1+SI2+...+SIC...(Formula 4)

In the following step S355, the main category determination unit 103A calculates the normalized sampling index NSIi for each category. The normalized sampling index NIi is the value obtained by dividing each sampling index SIi by the sum of the indices SI_SUM, and can be calculated using the following equation 5:

NSIi = SIi/SI_SUM (Formula 5)

In the following step S356, the main category determination unit 103A determines the main category based on the normalized sampling index calculated in step S355, and then ends the processing shown in Figure 17.

According to the above-described third embodiment, the following effects can be obtained.
(10) The main category determination unit 103A determines the main category based on the risk of each category included in the behavior category, the frequency of existence of each behavior category in the divided moving image group 101, which is the population of main category moving images, and the variable λ, which is the gaze parameter 1801. The gaze parameter 1801 determines the ratio of the influence on the determination of the main category of the risk of each category included in the behavior category and the frequency of existence of each behavior category in the divided moving image group 101. Therefore, the main category can be determined taking into account the risk of each category.

(Modification of the third embodiment)
In the third embodiment, the gaze parameter 1801 represented by λ in Equation 3 is a single value. However, the gaze parameter 1801 may be set to a different value for each category.

Figure 18 shows the gaze parameters 1801 in this modified example. As shown in this figure, in this modified example, the value of the gaze parameters 1801 is set for each category.

This modified example provides the following advantages.
(11) As shown in Fig. 18, the gaze parameter 1801 is set for each category included in the behavior category, so that the optimum value of the gaze parameter 1801 can be set for each category.

-Fourth embodiment-
A fourth embodiment of the learning system will be described with reference to Figure 19. In the following description, the same components as those in the first embodiment are designated by the same reference numerals, and differences will be mainly described. Points that are not specifically described are the same as those in the first embodiment. This embodiment differs from the first embodiment mainly in that it includes a video for evaluation.

Figure 19 is a configuration diagram of a learning system 1C in the fourth embodiment. In addition to the configuration in the first embodiment, the learning system 1C has an evaluation video group 2201, an evaluation label group 2202, a recognition model 2203, and an evaluation prediction result 2204. In the first embodiment, the performance evaluation unit 113 generated the evaluation result 117 using the prediction result 110 output by the recognition model 108, but in this embodiment, the evaluation result 117 is generated using the evaluation prediction result 2204. The evaluation video group 2201 is a composite video created in advance. The evaluation video group 2201 is not particularly limited as long as it is a collection of composite videos, but it is not allowed to be completely identical to the collection of training videos created by the training video generation unit 107 and input to the recognition model 108. The evaluation label group 2202 is the action label of the main category of each video included in the evaluation video group 2201.

In this embodiment, the training videos used to generate the recognition model 108 are different from the videos included in the evaluation video group 2201, so the model can be evaluated using videos different from the videos used to generate the model.

According to the above-described fourth embodiment, the following advantageous effects can be obtained.
(12) The performance evaluation unit 113 evaluates the evaluation prediction result 2204 obtained by inputting a video different from the training video used for training the recognition model 108, i.e., a video included in the evaluation video group 2201. This prevents overlearning.

(GUI)
A GUI (Graphical User Interface) for the first to fourth embodiments will be described with reference to FIG. 20. FIG. 20 shows the initial state of the GUI. To start the machine learning process, necessary information must be entered into the GUI. The information required to start the machine learning process can be divided into five parts: first, basic information; second, risk value; third, category determination method; fourth, main category video determination method; and fifth, rating frequency. The GUI accepts input for these five parts in sequence.

The first information to be entered is basic information 2301, including the number of epochs 2302, the storage location of training data 2303, and the storage location of validation data 2304. The number of epochs 2302 specifies the stopping point for repeated training of the recognition model 108. The locations where the respective data sets can be accessed are entered in 2303 and 2304. The second setting, the risk value, has two options: manual input 2401 and automatic calculation 2402. If manual input 2401 is selected, a dialog box for manual input is displayed. If automatic calculation 2402 is selected, a dialog box is displayed prompting the user to enter an input field for data to be used for automatic calculation, for example, the storage location of statistical information 119.

In the third category determination method, one of random 2501, frequency criterion 2502, and risk criterion 2503 is selected. If random 2501 is selected, the main category determination unit 103 performs the first operation, positive frequency determination. If frequency criterion 2502 is selected, the main category determination unit 103 performs the second operation, inverse frequency determination. If risk criterion 2503 is selected, a dialog box for setting the value of the gaze parameter 1801, represented by the variable λ, is displayed. This dialog box may accept numerical input, or may display a slider for setting the value. Once the value of the gaze parameter 1801 is set, the main category determination unit 103 determines the main category using the method of the third embodiment.

When determining the fourth main category video, either normal 2601 or selective 2602 can be selected. If normal 2601 is selected, a dialog is displayed for setting the value of the ratio parameter 105 represented by the variable γ. This dialog may accept numerical input, or may display a slider for setting the value. It may also accept input of the values of L1 and L2 shown in Figure 12. If selective 2602 is selected, a dialog is displayed for setting the value of the ratio parameter 105 represented by the variable γ, the values of L1 and L2, and the threshold value THRE.

In the fifth evaluation frequency determination, either epoch 2701 or batch 2702 is selected. In addition, count 2703 is also input. In the example shown in Figure 20, epoch 2701 is selected and "1" is input as count 2703, so evaluation by the performance evaluation unit 113 is performed for each epoch, i.e., evaluation result 117 is output. Once all five items have been input, learning of the recognition model 108 begins.

(Worker motion recognition system)
A worker action recognition system that uses the recognition model 108 generated in the first to fourth embodiments described above will be described with reference to Fig. 21. Fig. 21 is a configuration diagram of a worker action recognition system 3000. The worker action recognition system 3000 includes a video acquisition unit 3001, a detector 3002 that detects a worker, a tracker 3003 that tracks the worker, an extractor 3004 that uses the detector 3002 and the tracker 3003 to extract a candidate area where the worker is present, the recognition model 108, and an output unit 3005.

The video acquisition unit 3001 may be a communications interface or storage medium reader that acquires captured video files, or a video capture device that includes an image sensor, such as a web camera. The detector 3002, tracker 3003, and extractor 3004 can use known configurations, and are realized, for example, by a CPU 41, ROM 42, and RAM 43, as in the computing device 40 shown in Figure 8. The output unit 3005 is a communications interface, a video output device such as an LCD display, or a display adapter that outputs a video signal to a video output device.

The video acquired by the video acquisition unit 3001 is sent to the detector 3002. The detector 3002 detects a worker, i.e., a human, from the input video and outputs the position information to the tracker 3003. The tracker 3003 uses the position information output from the detector 3002 as an initial value to track the position of the worker in the video. The extractor 3004 uses the output of the detector 3002 and the position of the worker tracked by the tracker 3003 to extract a spatial region of the video and output it to the recognition model 108. The recognition model 108 performs recognition processing on the spatial region of the video extracted by the extractor 3004 and outputs an action category. The output unit 3005 outputs the action category information output by the recognition model 108.

Note that detector 3002 and tracker 3003 may detect and track only the worker's hands, instead of detecting and tracking the worker's entire body. In this case, extractor 3004 extracts an area including the worker's entire body based on the position of the tracked hands. In other words, even in this case, the output of extractor 3004 does not change.

The worker action recognition system 3000 described above has the following advantages.
(13) The worker action recognition system 3000 includes the recognition model 108 generated by the learning system 1, a detector 3002 that detects a worker, a tracker 3003 that tracks the worker, and an extractor 3004 that extracts a candidate area where the worker is present using the detector 3002 and the tracker 3003. The recognition model 108 receives an image of the candidate area. Therefore, the worker action recognition system 3000 can recognize high-risk behavior.

In each of the above-described embodiments and variations, the functional block configurations are merely examples. Several functional configurations shown as separate functional blocks may be configured together, or a configuration shown in a single functional block diagram may be divided into two or more functions. Furthermore, some of the functions of each functional block may be provided by other functional blocks.

In the above-described embodiments and variations, the program is stored in ROM 42 (not shown), but the program may also be stored in a non-volatile storage device (not shown). Furthermore, the arithmetic device 40 may be equipped with an input/output interface (not shown), and the program may be loaded from another device when necessary via the input/output interface and a medium available to the arithmetic device 40. Here, the medium refers to, for example, a storage medium that is detachable from the input/output interface, or a communication medium, i.e., a wired, wireless, optical, or other network, or a carrier wave or digital signal that propagates through such a network. Furthermore, some or all of the functions realized by the program may be realized by a hardware circuit or FPGA.

The above-described embodiments and variations may be combined with each other. While various embodiments and variations have been described above, the present invention is not limited to these. Other embodiments conceivable within the technical scope of the present invention are also included within the scope of the present invention.

1, 1A, 1B, 1C: Learning system 101: Divided video group 102: Action label group 103, 103A: Main category determination unit 104: Risk information 106: Subcategory determination unit 107, 107A: Learning video generation unit 108: Recognition model 113: Performance evaluation unit 114: Learning unit 117: Evaluation result 118: Risk information generation unit 119: Statistical information 703: Relative difficulty calculation unit

Claims (14)

a learning video generation unit that combines two videos, a main category video that is a video that belongs to a main category included in a behavior category, and a sub-category video that is a video that is a video that is included in the behavior category and belongs to a sub-category that is different from the main category, at a connection point, and trims the videos to include the connection point, thereby generating a learning video;
a learning unit that uses the learning video to learn a recognition model that outputs the main category for the input learning video;
a performance evaluation unit that evaluates a prediction result by the recognition model and generates an evaluation result;
The sub-category is selected based on the risk levels of the main category and the sub-category in the behavior category and the evaluation results. 2. The learning system according to claim 1,
The learning system further comprises a subcategory determination unit that determines, as the subcategory, the behavior category that the recognition model most often erroneously determines as the determined main category. 2. The learning system according to claim 1,
The learning system wherein the learning video generation unit generates the learning videos when the recognition performance of the recognition model determines that the learning videos of the main category are the sub-category at a rate higher than a predetermined threshold. 2. The learning system according to claim 1,
The learning system further comprises a main category determination unit that determines the behavior category as a main category with a higher frequency as the risk in the behavior category increases. 3. The learning system according to claim 2,
The subcategory determination unit is a learning system that determines the subcategory based on risk information indicating the risk of each category included in the behavior category and the evaluation result. 5. The learning system according to claim 4,
the main category determination unit determines the main category based on a risk of each category included in the behavior category, a frequency of existence of each category of the behavior category in a divided video group that is a population of the main category videos, and a gaze parameter;
A learning system in which the attention parameter determines the ratio of the influence that the risk of each category included in the behavioral category and the frequency of occurrence of each category of the behavioral category in the divided video group have on determining the main category. 6. The learning system according to claim 5,
The learning system further comprises a risk information generation unit that calculates the risk information based on past statistics. 6. The learning system according to claim 5,
the sub-category determination unit determines the sub-category based on the risk information, the evaluation result, and a ratio parameter;
A learning system in which the ratio parameter determines the ratio of influence that the risk information and the evaluation result have on determining the subcategory. 2. The learning system according to claim 1,
The performance evaluation unit is a learning system that evaluates prediction results obtained by inputting a video different from the learning video used for learning by the recognition model. 2. The learning system according to claim 1,
The performance evaluation unit is a learning system that evaluates the performance of the recognition model for each predetermined number of batches. 7. The learning system according to claim 6,
A learning system in which the gaze parameters are set for each category included in the behavior category. A recognition model generated by the learning system according to claim 1;
a detector for detecting a worker;
a tracker for tracking the worker;
an extractor that extracts a candidate area where the worker is present using the detector and the tracker;
The recognition model is a worker action recognition system that inputs an image of the candidate area. A recognition model generated by the learning system according to claim 1;
a detector for detecting the worker's hand;
a tracker for tracking the worker's hand;
an extractor that extracts a candidate area where the worker is present using the detector and the tracker;
The recognition model is a worker action recognition system that inputs an image of the candidate area. A method for learning a recognition model that outputs a category corresponding to an input video,
A main category video is a video that belongs to a main category included in a behavior category, and a sub-category video is a video that belongs to a sub-category that is included in the behavior category and is different from the main category. Two videos are joined at a joining point, and the two videos are trimmed to include the joining point, thereby generating a learning video.
training the recognition model using the training video so as to output the main category for the input training video;
evaluating a prediction result by the recognition model to generate an evaluation result;
A method for learning a recognition model, wherein the sub-category is selected based on the risk levels of the main category and the sub-category in the behavior category and the evaluation result.